Building disaster into the network: how UK.gov does IT

Contracting out the blame...


Analysis The Department of Work & Pensions may ultimately derive some lessons from its IT disaster last week, but it's doubtful that it will spot the most important one, far less take the necessary corrective action. The problem, you see, is that its IT strategy is now largely out of its control, and that it is essentially deskilling itself as far as IT strategy is concerned, and abdicating responsibility for the consequences. This has particular relevance across the Government, because on the one hand the overall budget depends on IT working sufficiently to achieve savings and staff reductions, while on the other IT is being outsourced to the private sector as a matter of policy.

The strategy depends on the heavy investment in IT paying off, that success lies in the hands of outside companies, and last week's IT disaster could not have, should not have happened if the standard disciplines and safeguards of network management had been in place. The inquisition will presumably establish whose hand pressed the self-destruct button, but that is not really what's important - the button should not have been available to be pressed, and the DWP should have been in the position to know and understand why the button could not have been pressed. IT professionals looking at what happened must however surely conclude that the DWP effectively did not have anything in place that could be dignified by the term network management, and that if it even knew what the expression means it must have placed its faith entirely in the contractor, EDS.

With any justification? Well... The Register has been leaked sufficient information for us to have built up a reasonable picture of the disaster's genesis, and it's very simple. A small group of machines at the DWP was running a pilot XP installation in preparation for rolling out XP across the whole organisation. An update that was intended to be deployed only to this group of machines was, because of a bad or corrupt Group Policy, deployed to the entire network as well. According to one DWP source, this changed local policy in such a way that clients were denied network access. It wasn't immediately clear to the centre what had happened, and even when it was fixing the issue without the presence of a network was complex.

The DWP has somewhere between 80-100,000 PCs, and admitted to 80 per cent having been hit by the problem, which extended from Monday into Friday afternoon, when the DWP said things were getting back to normal. The DWP argues that the matter has been much overblown, and that there was little disruption to benefits payments; in which case, one is tempted to argue, why do you need getting on for 100,000 XP machines when you can manage so well without them? But although it is difficult to see how EDS, the main IT service provider for the DWP with according to one staffer "full access and control for all of our key systems and complete monopoly where decisions are made", could not have been in control of the finger on the button, we really do need to take a closer look at that button, before it can switch off the whole country.

It is, as Lib Dem MP Richard Allan commented in his blog on Saturday, no more than good practice to roll out updates to networks in small batches in order to minimise the damage that could be done if something went wrong. This was not of course an intentional rollout of an update to the entire network, but that simply makes it worse because, if a group policy action was not intended to apply to the whole network, then what on earth was it doing being pointed at it? And how is it possible to do something like that by accident? (Or, come to it, at all?)

Good IT practice surely dictates that the pilot XP system should have been thoroughly walled off from the operational network, and that the accounts and policies used to manage it should have been quite distinct from those of the main network. As this manifestly could not have been the case, we are left with virtually the sole conclusion that the DWP's network policies are badly broken, and quite possibly being operated by the class of IT administrator who thinks he's god.

Which is OK, but wise IT admins who think they're god make sure they've got a few failsafe policies in place so that they can't destroy the whole network, company, department or Government if they accidentally push the wrong button. One of the ways you can do this is to construct rational and limited-sized groups for different roles and purposes, and another is for you to not be driving the network while you have vastly more admin privileges than you need for the particular jobs you're doing. Is this basic network management training? Yes, we fear it is.

Meanwhile, the DWP is probably deriving precisely the wrong lesson from the fact that its operations were, at least after a fashion, able to continue during the crisis. Having the PCs disabled meant that new cases and amended cases couldn't be processed, but existing payments could be made, so an outage for a couple of days needn't have a long term impact. Or so they think. The DWP is not however where it intends to be with its PC network yet. It currently has under construction something it terms its Desktop Office Infrastructure (DOI). According to in-house techies this is proving a major headache to implement, but in principle it's intended to be The Way Forward. The use of networked PCs is intended to produce efficiencies and savings by allowing the DWP to cut staff - but if this goes ahead and it's still possible for a single finger to destroy the PC network, what kind of damage would the DWP's operations then sustain? As you push more of your critical assets out onto the things most likely to break, you should be taking a great deal of care to construct systems to stop it breaking, and to build resilience into the network.

The point to bear in mind here is that although we've been hearing about the joys of sharing information on PC networks and the wonders of single console network administration since the ark, we tend to forget that it's something that's only just starting to arrive on a large scale, particularly in government. Sure, for years it's been possible for imbeciles in charge of network configurations that are too dumb to live to break little bits, but it's only just becoming possible for them to paralyse whole departments, across the country. The UK Government actually has quite a lot of these networks going into place right now, but is smitten by the brochureware on the joys of centralised network management. It does not grasp (this isn't in the brochures) that if you don't do the management properly then you're building a self-destruct capability in, and for so long as it concentrates on haggling over prices and delivery without having the capability to understand the nuts and bolts, it won't be able to grasp how dangerous the weapons it's building are.

By now the DWP and the rest of the Government's departments are beginning to categorise last week's crisis as just one of those things, and an isolated problem that has been overcome. That's if they're still thinking about it at all. But at the very least they should be asking, could it happen again, could it happen to me? Under what circumstances if any, ministers should be asking, would it be possible for any single person in the organisation to roll out an executable, a patch, a change in settings, anything, to the whole network at once? And if that's not theoretically possible, what safeguards are in place to ensure that it really isn't possible, and that it can't become possible either through error or deliberate sabotage? Most of them won't know the answer, nor will they readily be able to find out what it is, and if told it's all perfectly secure by their consultants or contractors, they'll happily believe that.

Ignorance oughtn't to be an excuse, but these days, regrettably, it often is. Outsourcing means you always have somebody to blame, and so what if you're so ignorant of the technology and the issues that you're utterly incapable of specifying a project properly (and thus, utterly incapable of having an IT project work)? Never mind, you can always blame the contractor when it goes wrong - but it's your network, your department, and sooner or later that flock of chickens is going to come home to roost. ®

Related links:

DWP kills 60k+ PCs in Windows XP upgrade lash-up

Civil servants sacked over Net porn

Public sector has IT companies over a barrel

CSA boss falls on sword over £456m IT system fiasco


Other stories you might like

  • Demand for PC and smartphone chips drops 'like a rock' says CEO of China’s top chipmaker
    Markets outside China are doing better, but at home vendors have huge component stockpiles

    Demand for chips needed to make smartphones and PCs has dropped "like a rock" – but mostly in China, according to Zhao Haijun, the CEO of China's largest chipmaker Semiconductor Manufacturing International Corporation (SMIC).

    Speaking on the company's Q1 2022 earnings call last Friday, Zhao said smartphone makers currently have five months inventory to hand, so are working through that stockpile before ordering new product. Sales of PCs, consumer electronics and appliances are also in trouble, the CEO said, leaving some markets oversupplied with product for now. But unmet demand remains for silicon used for Wi-Fi 6, power conversion, green energy products, and analog-to-digital conversion.

    Zhao partly attributed sales slumps to the Ukraine war which has made the Russian market off limits to many vendors and effectively taken Ukraine's 44 million citizens out of the global market for non-essential purchases.

    Continue reading
  • Colocation consolidation: Analysts look at what's driving the feeding frenzy
    Sometimes a half-sized shipping container at the base of a cell tower is all you need

    Analysis Colocation facilities aren't just a place to drop a couple of servers anymore. Many are quickly becoming full-fledged infrastructure-as-a-service providers as they embrace new consumption-based models and place a stronger emphasis on networking and edge connectivity.

    But supporting the growing menagerie of value-added services takes a substantial footprint and an even larger customer base, a dynamic that's driven a wave of consolidation throughout the industry, analysts from Forrester Research and Gartner told The Register.

    "You can only provide those value-added services if you're big enough," Forrester research director Glenn O'Donnell said.

    Continue reading
  • D-Wave deploys first US-based Advantage quantum system
    For those that want to keep their data in the homeland

    Quantum computing outfit D-Wave Systems has announced availability of an Advantage quantum computer accessible via the cloud but physically located in the US, a key move for selling quantum services to American customers.

    D-Wave reported that the newly deployed system is the first of its Advantage line of quantum computers available via its Leap quantum cloud service that is physically located in the US, rather than operating out of D-Wave’s facilities in British Columbia.

    The new system is based at the University of Southern California, as part of the USC-Lockheed Martin Quantum Computing Center hosted at USC’s Information Sciences Institute, a factor that may encourage US organizations interested in evaluating quantum computing that are likely to want the assurance of accessing facilities based in the same country.

    Continue reading
  • Bosses using AI to hire candidates risk discriminating against disabled applicants
    US publishes technical guide to help organizations avoid violating Americans with Disabilities Act

    The Biden administration and Department of Justice have warned employers using AI software for recruitment purposes to take extra steps to support disabled job applicants or they risk violating the Americans with Disabilities Act (ADA).

    Under the ADA, employers must provide adequate accommodations to all qualified disabled job seekers so they can fairly take part in the application process. But the increasing rollout of machine learning algorithms by companies in their hiring processes opens new possibilities that can disadvantage candidates with disabilities. 

    The Equal Employment Opportunity Commission (EEOC) and the DoJ published a new document this week, providing technical guidance to ensure companies don't violate ADA when using AI technology for recruitment purposes.

    Continue reading
  • How ICE became a $2.8b domestic surveillance agency
    Your US tax dollars at work

    The US Immigration and Customs Enforcement (ICE) agency has spent about $2.8 billion over the past 14 years on a massive surveillance "dragnet" that uses big data and facial-recognition technology to secretly spy on most Americans, according to a report from Georgetown Law's Center on Privacy and Technology.

    The research took two years and included "hundreds" of Freedom of Information Act requests, along with reviews of ICE's contracting and procurement records. It details how ICE surveillance spending jumped from about $71 million annually in 2008 to about $388 million per year as of 2021. The network it has purchased with this $2.8 billion means that "ICE now operates as a domestic surveillance agency" and its methods cross "legal and ethical lines," the report concludes.

    ICE did not respond to The Register's request for comment.

    Continue reading
  • Fully automated AI networks less than 5 years away, reckons Juniper CEO
    You robot kids, get off my LAN

    AI will completely automate the network within five years, Juniper CEO Rami Rahim boasted during the company’s Global Summit this week.

    “I truly believe that just as there is this need today for a self-driving automobile, the future is around a self-driving network where humans literally have to do nothing,” he said. “It's probably weird for people to hear the CEO of a networking company say that… but that's exactly what we should be wishing for.”

    Rahim believes AI-driven automation is the latest phase in computer networking’s evolution, which began with the rise of TCP/IP and the internet, was accelerated by faster and more efficient silicon, and then made manageable by advances in software.

    Continue reading
  • Pictured: Sagittarius A*, the supermassive black hole at the center of the Milky Way
    We speak to scientists involved in historic first snap – and no, this isn't the M87*

    Astronomers have captured a clear image of the gigantic supermassive black hole at the center of our galaxy for the first time.

    Sagittarius A*, or Sgr A* for short, is 27,000 light-years from Earth. Scientists knew for a while there was a mysterious object in the constellation of Sagittarius emitting strong radio waves, though it wasn't really discovered until the 1970s. Although astronomers managed to characterize some of the object's properties, experts weren't quite sure what exactly they were looking at.

    Years later, in 2020, the Nobel Prize in physics was awarded to a pair of scientists, who mathematically proved the object must be a supermassive black hole. Now, their work has been experimentally verified in the form of the first-ever snap of Sgr A*, captured by more than 300 researchers working across 80 institutions in the Event Horizon Telescope Collaboration. 

    Continue reading
  • Shopping for malware: $260 gets you a password stealer. $90 for a crypto-miner...
    We take a look at low, low subscription prices – not that we want to give anyone any ideas

    A Tor-hidden website dubbed the Eternity Project is offering a toolkit of malware, including ransomware, worms, and – coming soon – distributed denial-of-service programs, at low prices.

    According to researchers at cyber-intelligence outfit Cyble, the Eternity site's operators also have a channel on Telegram, where they provide videos detailing features and functions of the Windows malware. Once bought, it's up to the buyer how victims' computers are infected; we'll leave that to your imagination.

    The Telegram channel has about 500 subscribers, Team Cyble documented this week. Once someone decides to purchase of one or more of Eternity's malware components, they have the option to customize the final binary executable for whatever crimes they want to commit.

    Continue reading

Biting the hand that feeds IT © 1998–2022