Sweaty Spice, the other Spice Girl

A conversation with a colleague about ginormous "oops" moments caused me to recollect one such case.

The Setup

In the mid 'oughts, I worked at a medium sized business that was an agglomeration of several smaller companies assembled into a sort of a hash. It was a pretty good place. Large enough that I didn't worry about my paycheck clearing, but small enough that there weren't rigid policies.

This included IT. I was a dyed in the wool Mac person, and the local team was cool with me using a Mac. I was (and am still) competent enough to self maintain, and really it just worked better for me in what I was doing. Use the best tool for the job and all.

(Well, except for Powerpoint X, that program sucked ass).

We had a sort of wild west common file server that all the groups sans HR used for shared documents. Engineering, manufacturing, marketing (where I sat) and Finance put all our files in this one giant file server that was just an SMB share. Of course, we were supposed to not put any sensitive files in there (hence the separate HR servers) but for the most part it just worked.

The problem

Our WiFi sucked. It was an old building, an old IBM call center in fact (as evidenced by the token ring coaxial wiring in the offices) laid out to make it difficult to get good wireless network access.

So, to do any real pushing of data, you would plug into the ethernet, and get that Fast Ethernet goodness (circa 2005, 100 megabit was pretty sweet indeed).

One day, I camped out in a conference room with a goal to reorganize product management's documents, policies, and properly structure the data to make it easier/more intuitive to navigate and find things. After a couple of hours of moving, uploading and crafting a directory structure that was perfection.

Confident that I had it just as I wanted it, I went to delete the old, chaotic structure to save a gigabyte or so of storage. I dropped to the terminal in OS-X, and typed that fateful command rm -Rf <directory>. Simple, I do it ALL the time.

But I made a typo. Instead of deleting the old product management directory tree, it was the equivalent of

rm -Rf *

If you don't know what that means, well it will systematically go through all the directories that are in the current directory, and delete ALL the files and directories underneathe it and since I was at the root directory, it would erase the whole disk.

Of course, this shouldn't have been possible, but our mid sized company, grown through acquisitions and agglomerization meant that we had pretty loose access controls, and no granularity. It was a free for all, and we trusted the users to not do dumb things.

ha ha ha ha

Watching my simple one line command start to erase the whole server, I pulled the network cable out, put on my best hang-dog face, and walked down to the IT team's offices to explain what I had done.

The Aftermath

I explained what had happened, what was affected, and how I completely fucked up. It was difficult, but it was the right thing to do. Sure, our IT team wasn't sophisticated enough to have great logging, especially since as a Mac user, I wasn't technically logged into the AD domain. But screwups this big need to be owned up to.

The IT folks were totally cool. They understood, and they even laughed with me at it. They pulled the tape from the prior night's backup, and in a couple hours had restored the files.

I no longer work there, but I presume that they have gotten religion, RBAC, and a lot more internal security.

Final note

As this place was a bit wild west. we had a "hidden" SMB share, a server that had a terabyte or so of MP3 files on it, and the cool kids knew how to find it, so we always had great tunes playing in our offices. I am sure that later IT audits found and disappeared that server, in the interest of not hosting pirated music.

Ah well, all's well that ends well.

You’ve successfully subscribed to Sweaty's Corner
Welcome back! You’ve successfully signed in.
Great! You’ve successfully signed up.
Your link has expired
Success! Check your email for magic link to sign-in.