Divert the power to the shields. 'I'm givin' her all she's got, Captain!'

Always check the roof – you never know what might (not) be lurking there

Who, Me? August is now just a memory, but hey – console yourself that Christmas is just around the corner. Or simply grab a caffeinated Monday beverage and take delight in another's pain courtesy of The Register's regular Who, Me? column.

Today's tale comes from a reader we'll call "Jack", who spent the early part of this century working on a project to unify the land-based communications for some parts of the US military.

The project, Jack recalled, included networking, video, audio, phones and so on. "The system," he told us, "employed a mixture of Sun Solaris Unix servers and Microsoft Windows 2000 servers (and workstations)."

The data centre took up an entire floor of one of the larger buildings on the US base. "The non-classified server room was about the size of a football (US, not soccer) field, and a smaller classified server room. Help Desk and NOC (network operations control) were also located there."

Jack, who had been originally hired as a Windows administrator, got his first inkling that all might not be well with the project when he studied the reams of documentation supplied by engineers for the setting up of Microsoft's finest. It lacked TCP/IP settings, which, as anyone who has had the misfortune to sit in front of the install wizard of the operating system knows, are a tad important.

But hey ho, Jack was (and is) a professional and made do with what he had.

"After a year or so," he told us, "all the hardware/software was installed and configured; and about 50,000 end users were live on the system." The eventual plan was to have 450,000 users on the thing.

Then, one summer weekend (and it is always the weekend, isn't it?), the power to all that gear abruptly failed.

Now, there are three important things to consider here. Firstly, the DC was located in a US state notable for hot summer days and warm summer nights. Secondly, power was naturally backed up by a generator that was seated on the roof: "It was quite large, having powered all the previous mainframes and terminals."

And thirdly: "It was strongly discouraged [for us to] actually go to the roof of the building to SEE the generator system."

As seasoned readers have probably already guessed: "It was NOWHERE NEAR large enough to run all of the servers, NOC and HD workstations, and other power needs."

While the battery-backed UPS units shrieked in their death throes, a hurried shutdown ensued. Being a government operation, "no one had ever thought of setting up a Shutdown Order Document (and the companion Start Up Document)," said Jack, before acidly observing: "Those documents became a priority after all of this."

Jack called his boss at home to agree which servers absolutely had to stay up. With the batteries under the floor wheezing their last in less than 20 minutes, tough decisions had already been taken by the team.

Eventually the gang got things powered down to the point where the asthmatic generator could handle the load. A lucky few got delegated to dealing with irate users calling in (those that could – a large chunk of the phone system had fallen victim to desperate power-saving measures) and, said Jack, "all was well in server land."

Alas, all was most definitely NOT well with the chillers responsible for keeping the server room cool. "For some unknown reason, NONE of the chillers were even set up to be powered by the generator," said Jack.

"The server room started warming up from the usual meat-locker temperature. Felt kind of good for a bit, except [for] the annoying little bell in our heads that it was going to get too warm. And – it did."

Even with the huge reduction in powered-on equipment, the temperature still climbed into the 90s (Fahrenheit – which is the 30-somethings in Celsius) until, after a day or so (which must have seemed much longer), power was finally restored.

The bonus day turned out to be handy as it gave the team time to devise a plan for bring all those abruptly terminated Windows and Unix servers back to life. Sunday (and into the night) was spent nervously pressing the "On" button and watching services spin up.

"Upper Management," said master-of-understatement Jack, "realized that [a] much larger generator and considerable rewiring was needed."

Even then, it still took a year to get the thing set up. "We were," said Jack, "quite fortunate not to have a repetition of the 'Event'."

While Jack, now semi-retired, has long departed the base, the moral of the story is one that could apply to oh-so-many projects today, government or private: "Poor planning never ends well."

Ever seen the best (or worst) made plans crumble at their first encounter with reality? Maybe some of those plans were yours? Drop Who, Me? a line and confess all to The Register readership. Discretion assured. ®

Similar topics

Broader topics

Other stories you might like

  • A discounting disaster averted at the expense of one's own employment
    I know what this process needs: Microsoft Access!

    Who, Me? A tale of discounts and process improvement via the magic of Excel, Access and a fair bit of electronic duct tape we imagine. Welcome to Who, Me?

    "James" is the Regomized reader of record today, and continues the theme of running the risk of doing a job just that little bit too well with an ancedote from the end of the last century involving his first job out of university, at a certain telecommunications giant.

    The job involved a process of calculating the discount received by big customers (the ones with multiple branches). "For the life of me I can't remember what the main DB was called," he told us, "but it was the old style green writing on a black screen that took forever to download the necessary data."

    Continue reading
  • In IT, no good deed ever goes unpunished
    When being helpful can mean being shown the door

    Who, Me? Going above and beyond in IT can sometimes lead to also going directly out of the door, as one Register reader found when discovering that sometimes efficiencies can be less than rewarding.

    A reader Regomised as "Will" told of us his days working at a now-defunct company that produced large telephone switches. In those days whenever a major software revision occurred, customers were expected to send in their configurations and Will's group would merge them into the latest and greatest. A new load would then be returned to the customers.

    It was not a fun process, not least because of constant hardware and software failures during the merge process. "When I first started, there was a constant grumble about how unreliable the machine used for the merging was," Will told us.

    Continue reading
  • An early crack at network management with an unfortunate logfile
    It's a backronym, right?

    Who, Me? Come with us on a journey back to the glory days of Visual Basic 6, misplaced enthusiasm and an unfortunate naming incident. Welcome to Who, Me?

    Today's tale comes from a reader Regomised as "Stephen", who was working in the IT department of a Royal Air Force base. "My duties were many," he told us, "from running daily backups of an ancient engineering system using (I kid you not) reel-to-reel tapes to swapping out misbehaving printers."

    This being the early 2000s, his boss loaded up our hero with more tasks. He could change printers and tapes, so Visual Basic (and its bedfellow, Access) should present no problem.

    Continue reading
  • What do you do when all your source walks out the door?
    Where the phrase 'don't put all your eggs in one basket' originates

    Who, Me? Who has got your back up? Forget comments in code, what do you do when all your source has been packed into the trunk of a family sedan? Welcome to Who, Me?

    Today's story, from a reader Regomised as "Al", concerns his time at a company in the 1980s. The company was working on a project to replace thousands of ageing "dumb" terminals with PCs. "The Great PC Invasion and Distributed Computing Revolution were under way," Al observed.

    "The company had hired a collection of experienced PC and minicomputer programmers who were led by a management team of Mainframe Gods (as they viewed themselves)."

    Continue reading

Biting the hand that feeds IT © 1998–2022