If this job doesn't kill me, I might live till I'm old

        It’s been one of those days - again!

        It’s feeling a lot like ground hog day.

        A strange problem, corrupted our production AIX server        and we are restoring the OS from a mksysb (just the AIX naming for the root disk backup). I am hoping that the recovery will restore all the security information, which we were able to save from the old system. What exactly went wrong on it is still to be determined. Right now, it’s all hands on deck to get the system back.

        Plans were made to switch to the DR (Disaster Recovery) site but we are not sure that the other systems will be able to continue operating on this side of the great divide, while the core system is sitting in the DR computer room.

        First signs that things were not normal? There were none. A customer support agent came storming in waving arms wildly about his head, babbling about transactions not going through. When I investigated I found that I could not login (switch user) to the application users. It popped up an error that I’ve only ever seen when the system was installed the first time and you forgot to configure the five profile files to the correct values.

        Then it got worse. No one could establish new sessions to the system which is bad. All the automation is on an external system that launches it through the application users. So, the little problem suddenly got much larger.

        I started checking the system. Almost all the files in the /dev file system were deleted.

        Ah-ha.. that’s why I can’t SSH into it. Manually create the random device files and SSH out works, but not in. Great! Neither the memory, kmem or pty device files where there… then the Titanic feeling set in.. there’s no net0 driver, nor any of the disk and rhdisk pairs…

        That’s where we are now. Restoring another systems OS. Hoping the security information will transfer correctly from the backups we made and make the amount of work to recover less.

        Now, to get the rest of the banks systems to get their act together so we can fully switch components over to the DR site.

        Another late, late night… no golf (I’m really upset about that) and a ruined week… best laid plans and all.