Software guy smashes through the Somebody Else's Problem field to save the day

Stay in your lane? Or check for a shoddy connection job?

On Call A warning from the past in today's On Call. Helpfulness is not always rewarded with a pat on the back and a slap-up meal on expenses. Sometimes the Somebody Else's Problem field* is best left alone.

Our tale comes from a reader Regomised as Derek and concerns his time working for a multinational with plants at multiple locations in the UK. He was part of the application support team and toiled under a 24/7 call-out arrangement. A cry for help had to be answered within the hour.

One of the plants would start production at 0400 on Monday mornings, but started work four hours earlier to make sure things were up to speed. It had two main buildings. One was an office unit, housing the comms and server rooms. The other had an equipment room, with switches and patch panels as well as an operations room with client PCs and expensively large monitors.

The two buildings were connected by a fibre-optic cable that ran in a culvert beneath the yard, through which large trucks (Derek told us they could weigh as much as 38 tonnes) would rumble.

"The plant required the applications to be running," Derek told us. "No manual overrides so no system = no production.

You can probably see where this is going.

Sure enough, at 0130 one Monday morning the call came in. Nothing was working.

"I reluctantly dragged myself from my warm bed to drive the 10 miles to the plant," recalled Derek, "to be greeted by the sombre production team."

It wasn't looking good. All the screens were showing "connection error" messages and a simple ping test showed that the network between the two buildings was down. Derek's heart sank as he pondered the cause. The culvert running under the yard – the one over which those heavy trucks passed over and over again – had probably been stoved in, severing the connection.

Worse, for reasons known only to management, the infrastructure team responsible didn't provide out-of-hours support. Sure – the applications team was always on the end of the phone. The rest? Only 8 to 5.

What to do? At this point Derek could have played the "Not my job" card and left production hanging. Instead, he decided to see if there was anything he could do to help.

He went to the equipment room where, sure enough, the fibre connection light on the switch was dark. "So I followed the cable from the rack," he said, "and found it went to a wall-mounted box where it converted from the patch cable to the main multicore cable that ran under the yard..."

Hang on. One of the connectors looked like it was sitting a bit lower than its partner. Derek gave it a push and twist to seat it properly and hey presto! The plant burst into life.

"I suspect the connector had never been fully home since being installed about five years previously," he told us, "it had been hanging, but had finally dropped to the point where the connection broke."

Derek was the hero of the hour, and the recipient of many beer tokens from a grateful production team.

His boss, however, was less than impressed. Not with the slipshod connection, but with Derek's helpful assistance. Instead he found himself receiving a dressing down: "That's not your job!" snapped the boss.

"Such is life," said a now-retired Derek.

Derek's story reminded your correspondent of a fibre connection in a previous life where contractors left a mere 2cm of cable outside of the shielding. 2cm that proved all too attractive for sharp rodent teeth. Sadly, fiddling with the connectors didn't resolve the rat-induced hardware trauma.

But that's another story.

Have you ever stepped out of your comfort zone to fix somebody else's problem? Were you garlanded with praise or berated for your efforts? Let us know, with an email to On Call. ®

*Somebody Else's Problem (SEP) field: documented in Douglas Adams' Life, The Universe and Everything and concerns something being invisible because it is, er, somebody's else's problem. Not to be confused with the Steve Jobs Reality Distortion Field.

Similar topics

Other stories you might like

  • US won’t prosecute ‘good faith’ security researchers under CFAA
    Well, that clears things up? Maybe not.

    The US Justice Department has directed prosecutors not to charge "good-faith security researchers" with violating the Computer Fraud and Abuse Act (CFAA) if their reasons for hacking are ethical — things like bug hunting, responsible vulnerability disclosure, or above-board penetration testing.

    Good-faith, according to the policy [PDF], means using a computer "solely for purposes of good-faith testing, investigation, and/or correction of a security flaw or vulnerability."

    Additionally, this activity must be "carried out in a manner designed to avoid any harm to individuals or the public, and where the information derived from the activity is used primarily to promote the security or safety of the class of devices, machines, or online services to which the accessed computer belongs, or those who use such devices, machines, or online services."

    Continue reading
  • Intel plans immersion lab to chill its power-hungry chips
    AI chips are sucking down 600W+ and the solution could be to drown them.

    Intel this week unveiled a $700 million sustainability initiative to try innovative liquid and immersion cooling technologies to the datacenter.

    The project will see Intel construct a 200,000-square-foot "mega lab" approximately 20 miles west of Portland at its Hillsboro campus, where the chipmaker will qualify, test, and demo its expansive — and power hungry — datacenter portfolio using a variety of cooling tech.

    Alongside the lab, the x86 giant unveiled an open reference design for immersion cooling systems for its chips that is being developed by Intel Taiwan. The chip giant is hoping to bring other Taiwanese manufacturers into the fold and it'll then be rolled out globally.

    Continue reading
  • US recovers a record $15m from the 3ve ad-fraud crew
    Swiss banks cough up around half of the proceeds of crime

    The US government has recovered over $15 million in proceeds from the 3ve digital advertising fraud operation that cost businesses more than $29 million for ads that were never viewed.

    "This forfeiture is the largest international cybercrime recovery in the history of the Eastern District of New York," US Attorney Breon Peace said in a statement

    The action, Peace added, "sends a powerful message to those involved in cyber fraud that there are no boundaries to prosecuting these bad actors and locating their ill-gotten assets wherever they are in the world."

    Continue reading

Biting the hand that feeds IT © 1998–2022