Skip to main content

If this job doesn't kill me, I might live till I'm old

        It’s been one of those days - again!



        It’s feeling a lot like ground hog day.



        A strange problem, corrupted our production AIX server        and we are restoring the OS from a mksysb (just the AIX naming for the root disk backup). I am hoping that the recovery will restore all the security information, which we were able to save from the old system. What exactly went wrong on it is still to be determined. Right now, it’s all hands on deck to get the system back.


        Plans were made to switch to the DR (Disaster Recovery) site but we are not sure that the other systems will be able to continue operating on this side of the great divide, while the core system is sitting in the DR computer room.


        First signs that things were not normal? There were none. A customer support agent came storming in waving arms wildly about his head, babbling about transactions not going through. When I investigated I found that I could not login (switch user) to the application users. It popped up an error that I’ve only ever seen when the system was installed the first time and you forgot to configure the five profile files to the correct values.


        Then it got worse. No one could establish new sessions to the system which is bad. All the automation is on an external system that launches it through the application users. So, the little problem suddenly got much larger.


        I started checking the system. Almost all the files in the /dev file system were deleted.


        Ah-ha.. that’s why I can’t SSH into it. Manually create the random device files and SSH out works, but not in. Great! Neither the memory, kmem or pty device files where there… then the Titanic feeling set in.. there’s no net0 driver, nor any of the disk and rhdisk pairs…


        That’s where we are now. Restoring another systems OS. Hoping the security information will transfer correctly from the backups we made and make the amount of work to recover less.


        Now, to get the rest of the banks systems to get their act together so we can fully switch components over to the DR site.


        Another late, late night… no golf (I’m really upset about that) and a ruined week… best laid plans and all.

Labels:

Comments

Post a Comment

Popular posts from this blog

Acts Online – Home

For thoes South Africans, that feel that they are continuously taken advantage of, by business, government, and well-meaning but misinformed family members and friends, here is an online resource where you can have a look at the different Acts on the South African books. Acts Online – Home : ‘via Blog this.'
Post a Comment
Read more

Manta ray breached

This is the first time I've seen a manta ray breaching. Nature channels are full of whales, dolphins and sharks, to name just a few, but never the devil fish!
Post a Comment
Read more

Suicide

Suicide Shocking! Misunderstood. Illness. Deviation. Problem. Many words describe what first flows through your mind when you hear the word. In this week I had been confronted with the reality of it, or rather and attempt at it. A family member of a colleague had been under treatment for bi-polar disorder and had taken an overdose of her medication. Luckily the attempt was just that and the quick actions of people in this persons life saved it. I remember going ice cold. Like a part of the arctic was pushed into my core. Several years ago, I had suffered from severe depression. To the point where taking my own life was not something to be feared, but sought. Anything to just feel again. It is hard to describe to anyone the feeling of not feeling. It’s much the same as explaining the concept of a world where God is not present to someone that devoutly believes. You just cannot fathom the depth of despair when you cannot feel joy again. Everyone is used to these emotions in their live...
Post a Comment
Read more