Mmm, cheesy!

I work for the most unbelievably mickey mouse organization (no offense to Disney, they are not A mickey mouse organization they are THE mickey mouse organization).

So the day started with me checking a new website prior to a demo. See the web guy asked me to check it 2 days ago and I generated 10 or 15 corrections. Then yesterday when I found another error, I discovered that things I figured were one piece of code (because they are exactly the same every where) are really coded on every page. So where I had checked the "go to" menu once, I really should have gone to each of the 10 main pages from every page. That means to test those menus alone I should have made roughly 400 page hits. Silly me I thought it was coded smarter than that.

So 400+ page hits and 10 or so errors later, they go to do their demo. Blissful quiet for a while.

Then shortly after lunch the place goes crazy as the network generates a broadcast message that one of the servers has lost power and will shutdown in 5 minutes. What the F***. I go in to check it out (it's not my job, but no one else but the admin seemed to care). Seems it was intentional.

The bossman wants to increase the size of our server room. Makes sense, it's small and pretty jam packed with machines. He wanted to increase the office space too, but they shot him down on that.

So I think I've mentioned that the lawyers next door are going out of business (gosh that sounds bizarre). Anyway the Ancient & Venerable Bank of Baltimore is going to take over at least some of that office space to move people back in who are in another building. But the boss got a morsel of that space (4 or 6 feet maybe) to add to the server room.

So far, so good - except that the wiring for the room is all along the wall that will be removed ... OOPS. So today they were in moving the wiring. So they wired up a new spot. Unplugged each server cabinet and moved the plug to the new one. The plan being that the UPS would hold it long enough to be moved.

I went into the room discovered the plan. Then noticed an error on one of my servers. IP Conflict. Now I do not do servers. I am a database programmer, but I do not ignore messages like that. Run for boss, he says: reboot the server, it will go away. Sure enough message disappears. I go back to my desk (STUPID, STUPID, STUPID - Why do I always believe them!).

So a $20 million client calls and can't get to the web site. DOH! Seems pulling the power from the UNIX box (web server), which did not have a UPS, caused it to reboot, spaz and try to pull the wrong IP address (from my server). It had no idea who it was and the website completely choked.

GRRR. Took a good 40 minutes to get everything back up and running. Only after calling our IP guy to fix the UNIX after he'd been up most the night working on the search engine for the demo.

I love my job.