| ★ wanayoo — archive 1999 http://geek.com/geekside/gs015.htm | Nouvelle recherche | Portail wanayoo |
|
|
|
|
||
 |
|||||||
 |
Tales from the Geekside | ||||||
|
Resources
|
 |
author: Rob Hughes April Fools Back in 1997, April 1, Boston was hit with a massive snowstorm. It was quite late in the year for that sort of thing. I called the special number we have at my company for emergency announcements, and my CFO said that we did not have to come into work if we thought that the roads were too bad; but if we did come in, we got a comp day. Great, I thought, and went back to sleep. I awoke with a start to the frantic call of a co-worker. First I thought, "What is this guy doing at work today?" As I shook off the sleep, I realized he lives right next door to the office and had come in to check on some stuff. The salespeople that made it in couldn't run our contact management/order entry software. Apparently, something was wrong with our main SQL server. It was an HP Netserver running Windows NT 3.51 and Microsoft SQL Server 6.0. It had been running without a problem for several months. It was RAIDed up the wazoo with a few RAID 1s and an external cabinet housing five 2 GB hard drives in a RAID 5 configuration. The Netserver was a Pentium 166 with 128 MB of RAM. I spent a lot of time on the phone with my co-worker. Talking all day, my voice was hoarse. First, we did some basic troubleshooting, and we found that the server was on, and up, but the databases were not responding. Why not? Well, it seemed as though some drives weren't showing up in Disk Administrator. I had him check the drives. It was the RAID 5 in the external cabinet--they were off. He tried to turn them on . . . no dice. That was odd. We checked all the obvious stuff. The power cord was fine and the outlet had power. The cabinet had two power supplies that were supposedly redundant. However, whatever happened to the cabinet wasn't letting it turn on. I had my co-worker who was in call HP while I ate some lunch at home. HP couldn't get anyone out to fix the box that late in the day, especially with all that snow. We decided to try to get our backup box up and running. We came to realize that the box choked sometime that morning. Luckily, we dump our SQL logs on the hour and dump the database fully every night. We checked the dump from the night before. It went fine. We found that the server croaked sometime between 10 and 11 A.M., just enough time for the salespeople that did make it in to do a whole bunch of transactions. Great. Now, not only did I have to restore the database from a dump file, but I also had to apply hourly transaction logs to it. The good news was that we didn't lose much. Yes, we could have had a backup server running Octopus or something and not lost anything, but we didn't. At about 4 P.M., I went outside and headed for work. I was meeting a co-worker and a consultant there who was familiar with restoring SQL databases. I had restored SQL databases but I hadn't ever had to apply transaction logs before. As I walked out to the car, I decided that the company would be welcome. The snow had stopped, and the sky was pretty sunny at this point. Unfortunately, part of me was still in my bed, so my energy was wavering as I shoveled my car out of 18 inches of snow. I got into my rear wheel drive monster and entertained myself by spinning around corners as I headed to work. The roads were actually mostly plowed. I got to work and we started copying the files to our backup SQL server. We hadn't ever needed to use it before, but now was the time. It was much slower and less robust than the Netserver, but it had plenty of drive space and I wasn't complaining. The database copy went fine, and then we applied the transaction logs. It all went well, and users would be able to sell the next day. Fine, but what the heck was wrong with that external cabinet? HP came in the next day with new power supplies. I watched the HP guy operate, and through his smooth machinations, he determined that the problem was not the power supplies. The box still would not turn on. Great, I have redundant power supplies and my cabinet still loses power? It turned out that the faulty piece was the module that controlled the redundant power supplies. It just burned up and croaked. I learned that redundancy is not always as it seems, and I was certainly glad that I had the backup server ready. That HP storage cabinet sure played a great April Fool's joke on us.
Home | Welcome
| Survey | Newsletters
| Site Map | Contact
Us |
|||||
