Go on, feast your eyes on... HMRC's backend: 4,000 IT staff, its hookup with AWS and more

UK taxman's digital boss talks living with Amazon

OpenInfra Days UK 2019 The British tax man loves the cloud, but anyone who thinks public infrastructure can be run by a skeleton crew should think again: HMRC has no fewer than 4,000 IT staff who deliver around 140 digital services.

... people were really reluctant to do on-call because of the problems we had

The government agency is also a big believer in agile development, employing 55 scrum teams across the country, all dedicated to the old DevOps mantra: we build it, we run it.

These and other details of HMRC's backend were revealed by Andrew Sheppard, interim head of digital operations, during a session at the Open Infrastructure Days UK – an event organised by the local Open Infrastructure community and supported by the OpenStack Foundation.

Sheppard bemoaned the fact that the tax-slurper has to compete for talent with the private sector – where salaries are much, much higher. As a solution, the agency has launched an apprenticeship scheme – which can take a PFY with A levels and make them a valuable member of the platform team in less than two years.

HMRC photo, Gov.uk

HMRC boss defends shift to AWS, says they got 50% knocked off


In 2017, HMRC moved from two colocation environments, where it was running OpenStack, into AWS – leading to the demise of a small British data centre business along the way. The main driver behind this shift was said to be cost savings; the same year, Sir Jonathan Thompson, chief executive and permanent secretary of HMRC, claimed AWS was working out 50 per cent cheaper than Azure.

"We only have one infrastructure engineer who's on call at any one time, backed up by an on-call manager," Sheppard said on stage. "Since we moved into AWS, there isn't much for them to do – you can see now that the rota gets filled in way ahead [of time], because it's easy money for people, whereas prior to that, people were really reluctant to do on-call because of the problems we had."

HMRC's core platform staff consist of the operations team that provides "all of the collaborative tooling"; the platform operations team that provides "the capability and the expertise for the service teams to get on with writing the microservices," which are mostly done in Scala; the build and deploy team that owns the CI/CD pipeline; the telemetry team that provides dashboards for the service teams; the infrastructure team that is responsible for the vast majority of platform infrastructure; and the four-person platform security team, also known as DevSecOps.

"Even within the IT department of HMRC, I still have discussions with people to try to ensure that they understand that public cloud is safe, depending on how you architect it. There's still a very on-premises mindset; I'm sure that's not just HMRC," Sheppard said.

At the core of HMRC's operations is the Multi-Channel Digital Tax Platform – this Platform-as-a-Service has been live for more than five years, surviving three major iterations. It operates on a "typical microservices architecture" with 850 microservices in production, all of them stateless, and MongoDB in the backend – apart from "classic" services that run on Oracle.

"I'd like to say that none of our server images are older than seven days, but we do end up with some outliers that go a bit stale, so we're still working on improving that," Sheppard said.

All the workloads are spread across three AWS availability zones in the UK – which means three different data centres in London.

A part of the platform has been reserved for external testing. "We have lots of third-party software providers like Sage and Intuit. We provide this as a live-like sandbox environment for them to come in, because we know they are actually external customers. We support it Monday to Friday, 9 to 5, to test against their APIs."

The tax collector also maintains a special "snowflake" area for services that were architected for the agency's previous, now-defunct platform. "We rebranded them and called them the 'classic' services because we got sick of referring to them as legacy services, or crappy old services," Sheppard explained.

Recently, HMRC's platform was integrated with tech from the "new" government gateway, also hosted on AWS, after GOV.UK killed funding for the controversial digital identity system Verify. "The final physical remnants of the old one were shredded the week before the last, so that old platform that Atos and DWP built and ran for many years now literally no longer exists," Sheppard said.

But the main question remains: has moving to AWS actually made a positive difference for the taxman? According to Sheppard: "31 January, the end of the self-assessment filing period, busiest day of the year – it used to be a big deal, and to us it is becoming a BAU [business as usual] event. It's a business problem to solve, to get rid of that peak, and it is a very difficult thing for them to solve, but at the moment we've got that hyperscale [platform-as-a-service] and it can deal with that."

As for apprenticeships, in two years HMRC's in-house academy has been through 11 PFYs: two left because it turned out they didn't like to code, the remaining nine are still making their way through the programme, with two having already moved to the core platform team. Considering that all the investment it took was the salary of a part-time trainer and funding for the apprenticeships, HMRC seems to have found a reasonably effective, low-cost way to tackle the skills shortage. ®

Other stories you might like

  • Microsoft Azure to spin up AMD MI200 GPU clusters for 'large scale' AI training
    Windows giant carries a PyTorch for chip designer and its rival Nvidia

    Microsoft Build Microsoft Azure on Thursday revealed it will use AMD's top-tier MI200 Instinct GPUs to perform “large-scale” AI training in the cloud.

    “Azure will be the first public cloud to deploy clusters of AMD's flagship MI200 GPUs for large-scale AI training,” Microsoft CTO Kevin Scott said during the company’s Build conference this week. “We've already started testing these clusters using some of our own AI workloads with great performance.”

    AMD launched its MI200-series GPUs at its Accelerated Datacenter event last fall. The GPUs are based on AMD’s CDNA2 architecture and pack 58 billion transistors and up to 128GB of high-bandwidth memory into a dual-die package.

    Continue reading
  • New York City rips out last city-owned public payphones
    Y'know, those large cellphones fixed in place that you share with everyone and have to put coins in. Y'know, those metal disks representing...

    New York City this week ripped out its last municipally-owned payphones from Times Square to make room for Wi-Fi kiosks from city infrastructure project LinkNYC.

    "NYC's last free-standing payphones were removed today; they'll be replaced with a Link, boosting accessibility and connectivity across the city," LinkNYC said via Twitter.

    Manhattan Borough President Mark Levine said, "Truly the end of an era but also, hopefully, the start of a new one with more equity in technology access!"

    Continue reading
  • Cheers ransomware hits VMware ESXi systems
    Now we can say extortionware has jumped the shark

    Another ransomware strain is targeting VMware ESXi servers, which have been the focus of extortionists and other miscreants in recent months.

    ESXi, a bare-metal hypervisor used by a broad range of organizations throughout the world, has become the target of such ransomware families as LockBit, Hive, and RansomEXX. The ubiquitous use of the technology, and the size of some companies that use it has made it an efficient way for crooks to infect large numbers of virtualized systems and connected devices and equipment, according to researchers with Trend Micro.

    "ESXi is widely used in enterprise settings for server virtualization," Trend Micro noted in a write-up this week. "It is therefore a popular target for ransomware attacks … Compromising ESXi servers has been a scheme used by some notorious cybercriminal groups because it is a means to swiftly spread the ransomware to many devices."

    Continue reading
  • Twitter founder Dorsey beats hasty retweet from the board
    As shareholders sue the social network amid Elon Musk's takeover scramble

    Twitter has officially entered the post-Dorsey age: its founder and two-time CEO's board term expired Wednesday, marking the first time the social media company hasn't had him around in some capacity.

    Jack Dorsey announced his resignation as Twitter chief exec in November 2021, and passed the baton to Parag Agrawal while remaining on the board. Now that board term has ended, and Dorsey has stepped down as expected. Agrawal has taken Dorsey's board seat; Salesforce co-CEO Bret Taylor has assumed the role of Twitter's board chair. 

    In his resignation announcement, Dorsey – who co-founded and is CEO of Block (formerly Square) – said having founders leading the companies they created can be severely limiting for an organization and can serve as a single point of failure. "I believe it's critical a company can stand on its own, free of its founder's influence or direction," Dorsey said. He didn't respond to a request for further comment today. 

    Continue reading
  • Snowflake stock drops as some top customers cut usage
    You might say its valuation is melting away

    IPO darling Snowflake's share price took a beating in an already bearish market for tech stocks after filing weaker than expected financial guidance amid a slowdown in orders from some of its largest customers.

    For its first quarter of fiscal 2023, ended April 30, Snowflake's revenue grew 85 percent year-on-year to $422.4 million. The company made an operating loss of $188.8 million, albeit down from $205.6 million a year ago.

    Although surpassing revenue expectations, the cloud-based data warehousing business saw its valuation tumble 16 percent in extended trading on Wednesday. Its stock price dived from $133 apiece to $117 in after-hours trading, and today is cruising back at $127. That stumble arrived amid a general tech stock sell-off some observers said was overdue.

    Continue reading

Biting the hand that feeds IT © 1998–2022