Skip to main content

If this job doesn't kill me, I might live till I'm old

        It’s been one of those days - again!



        It’s feeling a lot like ground hog day.



        A strange problem, corrupted our production AIX server        and we are restoring the OS from a mksysb (just the AIX naming for the root disk backup). I am hoping that the recovery will restore all the security information, which we were able to save from the old system. What exactly went wrong on it is still to be determined. Right now, it’s all hands on deck to get the system back.


        Plans were made to switch to the DR (Disaster Recovery) site but we are not sure that the other systems will be able to continue operating on this side of the great divide, while the core system is sitting in the DR computer room.


        First signs that things were not normal? There were none. A customer support agent came storming in waving arms wildly about his head, babbling about transactions not going through. When I investigated I found that I could not login (switch user) to the application users. It popped up an error that I’ve only ever seen when the system was installed the first time and you forgot to configure the five profile files to the correct values.


        Then it got worse. No one could establish new sessions to the system which is bad. All the automation is on an external system that launches it through the application users. So, the little problem suddenly got much larger.


        I started checking the system. Almost all the files in the /dev file system were deleted.


        Ah-ha.. that’s why I can’t SSH into it. Manually create the random device files and SSH out works, but not in. Great! Neither the memory, kmem or pty device files where there… then the Titanic feeling set in.. there’s no net0 driver, nor any of the disk and rhdisk pairs…


        That’s where we are now. Restoring another systems OS. Hoping the security information will transfer correctly from the backups we made and make the amount of work to recover less.


        Now, to get the rest of the banks systems to get their act together so we can fully switch components over to the DR site.


        Another late, late night… no golf (I’m really upset about that) and a ruined week… best laid plans and all.

Labels:

Comments

Post a Comment

Popular posts from this blog

Acts Online – Home

For thoes South Africans, that feel that they are continuously taken advantage of, by business, government, and well-meaning but misinformed family members and friends, here is an online resource where you can have a look at the different Acts on the South African books. Acts Online – Home : ‘via Blog this.'
Post a Comment
Read more

Manta ray breached

This is the first time I've seen a manta ray breaching. Nature channels are full of whales, dolphins and sharks, to name just a few, but never the devil fish!
Post a Comment
Read more

In Case of Emergency

In the world we live in, it has become customary to complain about your life. Well, that is what it would seem to be, to any intelligent race that lands on the planet. Just open your ears and listen to the people around you. It’s hard to find anyone upbeat about anything anymore and should you be upbeat, then they lay the beat down on you. For as sure as tomorrow will come, they will have a stupendous amount of reasons why you shouldn’t be happy. Now, I ask you: “Why the hell not?” Why shouldn’t you be happy? Ignore the stress and worries for the day, because you can always worry tomorrow. If tomorrow doesn’t come, then at least you didn’t waste your today on worrying about something that would not come. I know I am a worry wart. It seems to be expected. At work, everyone passes their worries on to you, then seem surprised when you just listen, nod wisely, and carry on. Why take on someone else’ burdens when you have so many of your own to carry. Grab your coat. Grab your hat. Leave ...
Post a Comment
Read more