Episode 33 BOFH 2004
It seems the power supply of our webserver crapped itself and, in an excellent example of superior technology, grounded the phase for the rack - which in turn tripped the breaker and took out all the non-redundant powered systems within.
The PFY is on the job in a flash having seen exactly this type of thing many times before in kit from this vendor. (It's always reassuring to note that when a company merges with a company that in turn merged with the _original_ hardware manufacturer, they still manage to incorporate the worst design features from the original product.)
But I digress.
"When's the website going to be up?" the Boss asks, sputniking around me madly.
"Not sure," I respond. "It'll depend on how long the engineer takes to get here."
"Can't you fix it?"
"I could, but it would void the warranty - me not being a certified engineer and all."
"Well, have you called it in?"
"Just getting round to it," I say, firing up the systems page on the browser to get the serial number. >scratchey< >scratch< “Righto, Sorted."
"Where are you going now?!" the Boss gasps as I get up.
"Just getting a coffee and some lunch."
"But it's only 11:30?!"
"Yeah but I am planning on waiting on the line till they answer..."
..later that day outside the Boss' office..
"So how important is it that we get the website back online?" I ask.
"Very!!" the boss blurts.
"I told them it was a severity three call."
"That's good isn't it?"
"Not really. Watering their pot plants is Sev-2, so I doubt they'll be here today..."
"What?! Ring them back, change the severity!"
"Well, I could try calling, but it's 4pm now and their call desk closes at five."
"THAT ISN'T SERVICE!"
"No, but it's only a web server."
"Every minute the site's down we're losing money!" he gasps.
"No we're not. We don't do any business over the web so we're not losing money! Had it been the Stocks trading server at this time of day though, it'd be a completely different story and half the company would be banging on your door.."
"But the web's a valuable customer interface!"
"If you're Amazon, or Sendit, but not if you're us. We're a web nothing! Baby seals get more hits!"
Somewhere in the recesses of my mind I realise I may have stepped over a line here - if only in good taste.
"I..." the Boss says, slightly deterred, "... but it's core business!"
"Nah, that's just what the website company said. In reality, it could disappear without anyone noticing."
"I think that's an overstatement!"
"How much traffic does the website do on the average day?" I ask the PFY.
clickety< >tap< >tap< >click<
"Hmm, about a meg. Including the index rebuild," The PFY responds.
"And the index rebuild generates?"
"Just under a meg."
"So in effect, the loss of our website would have as much impact..."
"As Karen Carpenter on a Salad Bar," the PFY says, joining me over this side of the line.
"Well I still want it up!" the Boss snaps, liking the feel of an executive decision.
"But the only way to get it up today would be to borrow the spare supply out of another server," the PFY suggests. "A more important server. We don't generally like to do it beca.."
"JUST DO IT!"
"GET IT DONE!"
Five minutes and one large >clack< from the powerboard later.
"What the hell's happened?" the Boss blurts.
"The only similar power supply was in the Stocks and Tradings server, so I whipped that out to test if it would work, but the problem seems to be on the motherboard so it's cooked another supply," the PFY gabbles.
"YOU USED THE REDUNDANT SUPPLY FROM THE STOCKS TRADING SERVER!" the Boss gasps. "No."
"No it seems it wasn't redundant after all."
"What are you telling me?" the Boss asks, paling slightly.
"You can take the power supply out and the machine will work ok."
"..for about five minutes, then it overheats because the cooling fan in the power supply is missing from the machine," the PFY finishes.
"THAT'S BLOODY CRAP REDUNDANCY!" the Boss shouts.
"Sort of. You see, there's a fan unit somewhere that you remove when you install the redundant power supply."
"Then put it back in the machine!"
"There's the problem, no-one knows where it is, and there's less than a minute left."
"So what do we do?"
"I'm going to check the stock exchange website to see what the company's holding in its system and get a printout up to the traders. You might want to grab the newspaper."
"To check stock prices?"
"Nah. The job pages. I foresee another crap redundancy on the horizon..." ®;;;;;;;