Death of the old... servers.

I have written in the past about the set of old old servers that I maintain. It is not always easy to keep things that went out of warranty 6 years ago up and running. This weekend we had something called a "teidenn" basically they turned off the power to the entire building to preform some maintenance on the power system of the building. When the power gets turned off, so do all the servers. So from about 7 am a co-worker and I began powering off servers. This was the easy part. We went our seperate ways and waited for the Building manager to call us back and say the power was back on. When he did that is when the fun began. We arrived and there were still 2 guys from the power firm working on 1 plug in our building. I made myself busy by switching on machines and logging into the console server and trying to get them to start. The list of problems evolved from there beginning with the fact that I had left the root password list at the office. With most of the machines needing disc maintenance and/or fsck. It was not a good start. After the acquisition of the passwords most things rolled along nicely with a few exceptions, One server that held a few servers specialized functions would not start, completely dead. To complicate matters further there was no "easy" replacement for that server and I was stuck as for what to do with it. Another server was extremely fickle and kept pushing an impossible amount of errors to the terminal. A few break commands and a quick re-mount of the /usr slice and it was back in action. All in all we were there till about 8 pm and when I left there was still one server down. I skipped rugby practice and went to the "Data Center" and tried my best to get the server up and running up again to no avail. A whole day wasted for just one server. ERRRRRR I do not mind this is my job, but every year we say .... "dont worry we will be out of here by next year" and we never are. Next year I hope that we are right, but we probably wont be. E./