r/talesfromtechsupport • u/bippyz Screaming Makes The Server Work • Sep 02 '12
It's raining RAM, hallelujah!
This is long, but it's how I became a real BOFH.
Many moons ago, when I was a literal PFY (damn acne), my office decided to expand and bought a building across the street. As a wee PFY stuck doing a lot of desktop crap, I volunteered to help set up the new server room in the hopes of getting a better job (it worked). This meant I was making multiple trips across a four lane, two-way road, with a dolly.
At the time, there was no stop light or sign in the middle, where we were, but there were ones on either end of the block. Officially we were supposed to walk to the end of the block, wait for the light to change, cross, and walk back. Bitch, please!
I'm trundling back and forth with server parts, monitors, whatever, and delivering them to the brand new control center. Suddenly I see one of the windows BOFHs dead ass sprinting towards me, his arms filled with backup tapes. He's a big guy, used to be a linebacker in college, but well gone to seed now. This man does not run. Ron (not his real name) saunters at best.
"Ron, man, what's up?" I ask.
"BIP! Ditch that load and bring the dolly back!"
I don't question him, but it's here I notice something off... Ron's wet. "Should I get the bigger one?" I ask carefully. I was avoiding it because I was a 120lb kid at the time (god I miss that) and I couldn't muscle it loaded.
"Yes! Good thinking!" and he's gone.
I go bad to switch carts and my shoes (dress shoes to sneakers, finance company). This was very smart, as when I get back to the new building, there is a funny feel to the lobby. The guard waves me through and I get up to the server room... And it's wet.
All over.
Its raining in the damn server room and the real windows PFYs are just standing around. The servers were off the floor, but I quickly noticed they we still on, and shout, "I'm shutting them down!"
To this day I have no idea why the other guys stood there with their thumbs up their butts, but I turned the servers off and started hauling them out of the room. Even though I was the juniorest PFY, and not even a server admin at the time, I started bossing them around. "Herp, go find Manager and tell her what happened! Derp, take apart the racks, we can put them up back in the old building!"
By the time BOFH comes back, he is relieved to see people are doing things, and he and I start triage. This is dead, this is okay, etc etc. Since I'm half his size, Ron takes the servers back and forth across the street, leaving me to get maintenance to turn off the water (they didn't want to because they could only do it for the whole building). We turn off the water, and the manager finally shows up. She is thrilled we've got it under control, takes the status, and tells me what happened.
Upstairs is the cafeteria. And by upstairs, what I mean is directly above this room. And the morons cut the water line. She goes on to say that my call to shut the water off is probably going to save us millions, shook my hand, and then said she was drafting me. She said she'd email everyone what was up, and tell my boss. Oh and I'll get double OT.
Once the water was off, we were able to clean up and save most of the servers. Seven PM rolls around and we're still slogging away, when one of the Suits shows up.
"Why is the server still off?"
I'm seriously wet, trying to decide if this joker is screwing with me. "Which server?" I ask, stalling for time.
"The trade server!"
We labeled them all on the outside, so I take a quick look and see the trade server was, naturally, the one under the flow. "I'm sorry, that one was ruined. But Ron has his PFY restoring it from backup across the street. Should be done tonight, and we can process the trades." I gesture at the server.
Now we have a backup server that was actually doing the work, albeit slowly, and we didn't lose any trades. But the end of day processing, where we do 'night' work was dead.
"It's unplugged!" he shouts and starts to push past me.
I'm five foot nothing, a hundred twenty nothing. I jump up and hold up my hands, causing him to smack into them. "Sir! The sever is wet! You will electrocute yourself!" The power was also off, but let's not get into that. I forgot at the time.
"Shut up, girl! You don't know how important this is!"
Ron had sort of ignore this up until now, but I clearly heard him say WTF and he came over as I was explaining I did understand, but the server was not getting plugged back in.
Suit swatted my hands, and Ron, all 6'8" and over 200lbs of him, steps up. "Touch her, or call her girl, and I'll let you plug the server in," he snaps.
The absurdity of that sentence gave Suit pause, and Ron lays it the saga. "Well why didn't she say that the server got wet?" complains the suit. We both stared at him as he ranted that I didn't tell him the servers were went, and he storms off.
We finally get it all cleaned up, I get home around one, with a 'take the day off' credit, so it's a while before I think to check my work email. When I do, I have a note to call HR. Suit complained. I forward the message to Ron and his manager, as well as mine, and say I want Ron there in the meeting. When I roll in to work, it's all over. Ron told his manager what happened, she went to HR, and Suit sent me a formal apology. Ron's manager got me transferred to her team and hired me as a real BOFH not six months later.
tl;dr Money still works when it's wet! So should electronics!
24
u/PoliteSarcasticThing chmod -x chmod Sep 02 '12
Best story I've read in a long time. Well done.