Go on, feast your eyes on... HMRC's backend: 4,000 IT staff, its hookup with AWS and more

UK taxman's digital boss talks living with Amazon

OpenInfra Days UK 2019 The British tax man loves the cloud, but anyone who thinks public infrastructure can be run by a skeleton crew should think again: HMRC has no fewer than 4,000 IT staff who deliver around 140 digital services.

... people were really reluctant to do on-call because of the problems we had

The government agency is also a big believer in agile development, employing 55 scrum teams across the country, all dedicated to the old DevOps mantra: we build it, we run it.

These and other details of HMRC's backend were revealed by Andrew Sheppard, interim head of digital operations, during a session at the Open Infrastructure Days UK – an event organised by the local Open Infrastructure community and supported by the OpenStack Foundation.

Sheppard bemoaned the fact that the tax-slurper has to compete for talent with the private sector – where salaries are much, much higher. As a solution, the agency has launched an apprenticeship scheme – which can take a PFY with A levels and make them a valuable member of the platform team in less than two years.

In 2017, HMRC moved from two colocation environments, where it was running OpenStack, into AWS – leading to the demise of a small British data centre business along the way. The main driver behind this shift was said to be cost savings; the same year, Sir Jonathan Thompson, chief executive and permanent secretary of HMRC, claimed AWS was working out 50 per cent cheaper than Azure.

"We only have one infrastructure engineer who's on call at any one time, backed up by an on-call manager," Sheppard said on stage. "Since we moved into AWS, there isn't much for them to do – you can see now that the rota gets filled in way ahead [of time], because it's easy money for people, whereas prior to that, people were really reluctant to do on-call because of the problems we had."

HMRC's core platform staff consist of the operations team that provides "all of the collaborative tooling"; the platform operations team that provides "the capability and the expertise for the service teams to get on with writing the microservices," which are mostly done in Scala; the build and deploy team that owns the CI/CD pipeline; the telemetry team that provides dashboards for the service teams; the infrastructure team that is responsible for the vast majority of platform infrastructure; and the four-person platform security team, also known as DevSecOps.

"Even within the IT department of HMRC, I still have discussions with people to try to ensure that they understand that public cloud is safe, depending on how you architect it. There's still a very on-premises mindset; I'm sure that's not just HMRC," Sheppard said.

At the core of HMRC's operations is the Multi-Channel Digital Tax Platform – this Platform-as-a-Service has been live for more than five years, surviving three major iterations. It operates on a "typical microservices architecture" with 850 microservices in production, all of them stateless, and MongoDB in the backend – apart from "classic" services that run on Oracle.

"I'd like to say that none of our server images are older than seven days, but we do end up with some outliers that go a bit stale, so we're still working on improving that," Sheppard said.

All the workloads are spread across three AWS availability zones in the UK – which means three different data centres in London.

A part of the platform has been reserved for external testing. "We have lots of third-party software providers like Sage and Intuit. We provide this as a live-like sandbox environment for them to come in, because we know they are actually external customers. We support it Monday to Friday, 9 to 5, to test against their APIs."

The tax collector also maintains a special "snowflake" area for services that were architected for the agency's previous, now-defunct platform. "We rebranded them and called them the 'classic' services because we got sick of referring to them as legacy services, or crappy old services," Sheppard explained.

Recently, HMRC's platform was integrated with tech from the "new" government gateway, also hosted on AWS, after GOV.UK killed funding for the controversial digital identity system Verify. "The final physical remnants of the old one were shredded the week before the last, so that old platform that Atos and DWP built and ran for many years now literally no longer exists," Sheppard said.

But the main question remains: has moving to AWS actually made a positive difference for the taxman? According to Sheppard: "31 January, the end of the self-assessment filing period, busiest day of the year – it used to be a big deal, and to us it is becoming a BAU [business as usual] event. It's a business problem to solve, to get rid of that peak, and it is a very difficult thing for them to solve, but at the moment we've got that hyperscale [platform-as-a-service] and it can deal with that."

As for apprenticeships, in two years HMRC's in-house academy has been through 11 PFYs: two left because it turned out they didn't like to code, the remaining nine are still making their way through the programme, with two having already moved to the core platform team. Considering that all the investment it took was the salary of a part-time trainer and funding for the apprenticeships, HMRC seems to have found a reasonably effective, low-cost way to tackle the skills shortage. ®