Updating in production, like a boss

The pain in Spain comes mainly from a lack of training


Who, Me? Testing in production has always been a thing, sometimes by accident and sometimes because the powers that be cannot be bothered with multiple environments. And sometimes things go wrong. Welcome to Who, Me?

Our tale, from a reader Regomised as Nicolás, takes us back through the decades and to the glory days of Microsoft SQL Server 7.

SQL Server 7, which turned up in 1998, was a major rewrite of the old Sybase engine that had underpinned previous versions of Microsoft's database. Not so fragrant and fresh, however, were the other tools Nicolás used in his Madrid day job: Visual Basic 6 and that terror of programmers everywhere at the time, Crystal Reports.

At this point we'd normally go off on a historical tangent regarding the reporting tool, which came bundled with the likes of Visual Studio and Borland Delphi, but the trauma of having to change data sources programmatically still haunts this hack decades after the event. However, it was not Crystal Reports, nor was it Visual Basic 6 that lay at the heart of this week's metaphorical locking of the brakes in front of a looming truck. It was SQL Server.

Nicolás and his manager had been tasked with fixing problems with some brand spanking new in-house accounting software. “To say that the previous coding was utter crap is an understatement,” he muttered. Inexplicably, the duo were denied a development and test environment, or even space for backups.

“Keep up the good work,” exhorted the management as the pair glumly looked at the task.

They got cracking. “I was the database expert,” Nicolás modestly told us, as well as being the resident VB6 guru. The latter meant that he was elbow-deep dealing with what he described as “seriously crap VB6 code (forms would duplicate and crash)” when his colleague decided to have a crack at some data modifications.

Nicolás, although more experienced, was the junior partner in terms of job titles and so his colleague did not trouble him with the ins and outs of the task. It was just an update. On the production database server. Without backups.

Looking back, Nicolás told us: “To this day I don’t really know what went through their head, but those modifications were intended to fix some inconsistencies in the database (one [insert expletive here] had added a small amount of fixes to hide the previous software rounding errors!)

“Since we were not having rounding errors, those small adjustments had to go.”

Alas, the adjustments were not the only thing to go.

“Oh Dios … la he jodido.”

Nicolás looked up from his work. What had been, er, screwed up?

There are two types of database programmer in the world. Those who have missed a critical filter in the WHERE clause of an UPDATE, and those that will do so at some later point in their career.

Nicolás’s colleague had skipped from the latter camp to the former, and done so in production and without backups. Every record in the table had been inadvertently updated.

While the need for anonymity forbids us from revealing the purpose of the database, the cock-up would make for national news if it could not be fixed. Nicolás played the only card available to him: “The database is going down for important performance updates, please do not use the software.”

There is a happy ending to the story: he was able to reconstruct the borked data from the content of other tables during what we imagine was a very sweaty-palmed SQL session and bring the production system back online, the users none the wiser.

“It required so many filters it took hours,” he said, “not to mention carefully checking data consistency to make sure it made sense.”

Job done, he took himself home to soothe his rattled nerves. Perhaps with an adult beverage or two. It could, after all, have been much, much worse – a DELETE might have been issued rather than an update.

Then it would have had to be a "capacity optimisation" instead of the "performance updates."

We all know never to test or develop in production. But sometimes management is reluctant to pay for all those extra environments. Tell us about your moment of expense justification with an email to Who, Me? ®

Similar topics

Broader topics


Other stories you might like

  • Suspected phishing email crime boss cuffed in Nigeria
    Interpol, cops swoop with intel from cybersecurity bods

    Interpol and cops in Africa have arrested a Nigerian man suspected of running a multi-continent cybercrime ring that specialized in phishing emails targeting businesses.

    His alleged operation was responsible for so-called business email compromise (BEC), a mix of fraud and social engineering in which staff at targeted companies are hoodwinked into, for example, wiring funds to scammers or sending out sensitive information. This can be done by sending messages that impersonate executives or suppliers, with instructions on where to send payments or data, sometimes by breaking into an employee's work email account to do so.

    The 37-year-old's detention is part of a year-long, counter-BEC initiative code-named Operation Delilah that involved international law enforcement, and started with intelligence from cybersecurity companies Group-IB, Palo Alto Networks Unit 42, and Trend Micro.

    Continue reading
  • Broadcom buying VMware could create an edge infrastructure and IoT empire
    Hypervisor giant too big to be kept ticking over like CA or Symantec. Instead it can wrangle net-connected kit

    Comment Broadcom’s mooted acquisition of VMware looks odd at face value, but if considered as a means to make edge computing and the Internet of Things (IoT) more mature and manageable, and give organizations the tools to drive them, the deal makes rather more sense.

    Edge and IoT are the two coming things in computing and will grow for years, meaning the proposed deal could be very good for VMware’s current customers.

    An Ethernet switch that Broadcom launched this week shows why this is a plausible scenario.

    Continue reading
  • Ex-spymaster and fellow Brexiteers' emails leaked by suspected Russian op
    A 'Very English Coop (sic) d'Etat'

    Emails between leading pro-Brexit figures in the UK have seemingly been stolen and leaked online by what could be a Kremlin cyberespionage team.

    The messages feature conversations between former spymaster Richard Dearlove, who led Britain's foreign intelligence service MI6 from 1999 to 2004; Baroness Gisela Stuart, a member of the House of Lords; and Robert Tombs, an expert of French history at the University of Cambridge, as well as other Brexit supporters. The emails were uploaded to a .co.uk website titled "Very English Coop d'Etat," Reuters first reported this week.

    Dearlove confirmed his ProtonMail account was compromised. "I am well aware of a Russian operation against a Proton account which contained emails to and from me," he said. The Register has asked Baroness Stuart and Tombs as well as ProtonMail for comment. Tombs declined to comment.

    Continue reading
  • As Microsoft's $70b takeover of Activision nears, workers step up their organizing
    This week: Subsidiary's QA staff officially unionize, $18m settlement disputed, and more

    Current and former Activision Blizzard staff are stepping up their organizing and pressure campaigns on execs as the video-game giant tries to close its $68.7bn acquisition by Microsoft.

    Firstly, QA workers at Raven Software – a studio based in Wisconsin that develops the popular first-person shooter series Call of Duty – successfully voted to officially unionize against parent biz Activision. Secondly, a former employee appealed Activision's proposed $18 million settlement with America's Equal Employment Opportunity Commission regarding claims of "sex-based discrimination" and "harassment" of female staff at the corporation. 

    Finally, a group of current and ex-Activision employees have formed a Worker Committee Against Sex and Gender Discrimination to try and improve the company's internal sexual harassment policies. All three events occurred this week, and show how Activision is still grappling with internal revolt as it pushes ahead for Microsoft's takeover. 

    Continue reading
  • Nvidia shares tumble as China lockdown, Russia blamed for dent in outlook
    Sure, stonking server and gaming sales, but hiring and expenses to slow down, too

    Nvidia exceeded market expectations and on Wednesday reported record first-quarter fiscal 2023 revenue of $8.29 billion, an increase of 46 percent from a year ago and eight percent from the previous quarter.

    Nonetheless the GPU goliath's stock slipped by more than nine percent in after-hours trading amid remarks by CFO Colette Kress regarding the business's financial outlook, and plans to slow hiring and limit expenses. Nvidia stock subsequently recovered a little, and was trading down about seven percent at time of publication.

    Kress said non-GAAP operating expenses in the three months to May 1 increased 35 percent from a year ago to $1.6 billion, and were "driven by employee growth, compensation-related costs and engineering development costs."

    Continue reading
  • Millions of people's info stolen from MGM Resorts dumped on Telegram for free
    Meanwhile, Twitter coughs up $150m after using account security contact details for advertising

    Miscreants have dumped on Telegram more than 142 million customer records stolen from MGM Resorts, exposing names, postal and email addresses, phone numbers, and dates of birth for any would-be identity thief.

    The vpnMentor research team stumbled upon the files, which totaled 8.7 GB of data, on the messaging platform earlier this week, and noted that they "assume at least 30 million people had some of their data leaked." MGM Resorts, a hotel and casino chain, did not respond to The Register's request for comment.

    The researchers reckon this information is linked to the theft of millions of guest records, which included the details of Twitter's Jack Dorsey and pop star Justin Bieber, from MGM Resorts in 2019 that was subsequently distributed via underground forums.

    Continue reading
  • DuckDuckGo tries to explain why its browsers won't block some Microsoft web trackers
    Meanwhile, Tails 5.0 users told to stop what they're doing over Firefox flaw

    DuckDuckGo promises privacy to users of its Android, iOS browsers, and macOS browsers – yet it allows certain data to flow from third-party websites to Microsoft-owned services.

    Security researcher Zach Edwards recently conducted an audit of DuckDuckGo's mobile browsers and found that, contrary to expectations, they do not block Meta's Workplace domain, for example, from sending information to Microsoft's Bing and LinkedIn domains.

    Specifically, DuckDuckGo's software didn't stop Microsoft's trackers on the Workplace page from blabbing information about the user to Bing and LinkedIn for tailored advertising purposes. Other trackers, such as Google's, are blocked.

    Continue reading
  • Despite 'key' partnership with AWS, Meta taps up Microsoft Azure for AI work
    Someone got Zuck'd

    Meta’s AI business unit set up shop in Microsoft Azure this week and announced a strategic partnership it says will advance PyTorch development on the public cloud.

    The deal [PDF] will see Mark Zuckerberg’s umbrella company deploy machine-learning workloads on thousands of Nvidia GPUs running in Azure. While a win for Microsoft, the partnership calls in to question just how strong Meta’s commitment to Amazon Web Services (AWS) really is.

    Back in those long-gone days of December, Meta named AWS as its “key long-term strategic cloud provider." As part of that, Meta promised that if it bought any companies that used AWS, it would continue to support their use of Amazon's cloud, rather than force them off into its own private datacenters. The pact also included a vow to expand Meta’s consumption of Amazon’s cloud-based compute, storage, database, and security services.

    Continue reading

Biting the hand that feeds IT © 1998–2022