Software

Devops

How does Monzo keep 1,600 microservices spinning? Go, clean code, and a strong team

Well-known software development principles count for more than technology choices


QCon London Software engineers from digital bank Monzo told developers at the QCon event in London how and why it runs its banking systems on 1,600 microservices.

Monzo's session at QCon was in stark contrast to Monday's presentation in which Sam Newman warned that a microservices architecture is a "last resort". Senior engineers Matt Heath and Suhail Patel described how microservices work well for the bank, founded in 2015 and now with over 4 million customers.

As a new business that hoped to grow substantially, Monzo had a requirement for a technology platform that was extensible, scalable, resilient and secure. The idea was to start with a few basic banking services, and then be able to add more as time and resources allowed.

Monzo was convinced early on of the value of a distributed system. The bank did not want a single, big resilient system with a complex failover that you hope never has to run.

"If you don't exercise those failover modes, how can you know that they work reliably?" said Heath. They started with Mesos for cluster management, but by 2016 they switched to Kubernetes (K8s) as the "emerging market leader".

One of the goals was to abstract the complexity of infrastructure. "We think all the complexities about scaling of infrastructure, making sure that services are provisioned and databases are available, should be dealt with by a specific team, so that engineers can focus on the product," said Patel. The systems run on Amazon Web Services (AWS).

Matt Heath and Suhail Patel present at QCon London

K8s has not been completely pain-free. In 2017, Monzo "had quite a large outage, because a problem with K8s and how it interacted with etcd and linkerd, due to a combination of different bugs that were quite hard to test," said Heath.

Monzo picked Cassandra as the database because it scales horizontally (meaning you can simply add more hardware to scale, rather than having to migrate to a bigger system).

On the coding side: "We use Go as our primary programming language," said Heath. "It's quite simple, it's statically typed, and it makes it easy for us to get people on board." Go has a backwards-compatibility guarantee, meaning that when a new version appears, you can simply recompile existing code, getting the benefits of updates to features such as the garbage collector. Go is also well suited to strict policies about error handling.

"We have static analysis to make sure that you are not papering over errors," said Patel.

Banking systems are well suited to a modular approach. There is a requirement to link to many different systems such as BACS, CHAPS, Visa, Mastercard, Apple Pay and Google Pay. "Adding those things as separate systems allows us to keep them simpler," said Heath. Monzo builds integrations as much as possible in-house, rather than using third-party implementations, to get more control over resilience and performance (and likely saving money in the long term as well). They even built their own chat system, used internally and for support.

Monzo has also built its own tools for interacting with AWS and K8s, such as one called Shipper, which can deploy or rollback an individual service. Shipper can deploy directly from a pull request, which represents an update to code maintained in a Git repository.

Each Monzo microservice runs in a Docker container. "One of our biggest decisions was our approach to writing microservices," said Patel. There is a shared core library, which is available in every service; this is essentially copied in every container, though the build process will strip out unused code. This means that "engineers are not rewriting core abstractions like marshalling of data". It also enables metrics for every service so that after deployment it immediately shows up in a dashboard with analysis of CPU usage, network calls and so on. Automated alerting will identify degraded services.

Monzo has an extensive shared library available within every microservice

A lot of thought goes into the interface or API that each service exposes. The team favours writing many small services, each dedicated to a single purpose, rather than fewer more complex services. "Why do we have such granularity? We want to minimise the risk of change," said Patel. "For example, if we want to change the way contactless payments work, we're not affecting the chip and PIN system."

How do developers work on their code, given that running 1,600 microservices is not going to work on a laptop? "You are running a subset," said Heath. "We have an RPC filter that can detect you are trying to send a request to a downstream that isn't currently running, it can compile it, start it, and then send the request to it."

Why do microservices work for Monzo, whereas in some cases they add complexity without delivering much benefit? The Register has attended many QCon events, and while software development trends have changed from year to year, some things have remained consistent. One is that the way developers interact with each other in a team (and with management) counts for more than whatever development methodology they espouse. Another is that an incremental approach wins over occasional large changes. "An iterative process is generally what we take to heart at Monzo," said Heath, "both from an infrastructure perspective but also from a product perspective. By making small changes frequently we make sure we are going in the right direction."

Another recurrent QCon theme is the advantage of simplicity over complexity. Taken as a whole, what Monzo's system does is highly complex, but it has designed its systems in such as way as to divide that complexity into smaller, simpler pieces, and abstract it away from developers working on the code. "You don't need to know how 1,600 services work," said Heath. The hard task of managing K8s is delegated to specialists.

Monzo has also standardised "a small set of technology choices", said Heath, so that "as a group, we can collectively improve those tools". This could be frustrating for developers who have different technology preferences, but must help substantially with collaboration since everyone learns the same set of tools. "Code needs to be readable to other humans," said Patel. "We optimise code for readability. One of our engineering principles is not to optimise [performance] unless it is a bottleneck."

What Monzo presented at QCon seemed to be a strong template for software development and deployment in the case where you have a complex system with many components, and need to be able to respond quickly when requirements change or features are added.

Heath and Patel made a great case for the value of microservices. Note, though, that Monzo uses a lot of custom, in-house tools and libraries that are not easy to replicate. Further, many of the principles they presented – like writing clean, readable and disciplined code, focusing on a few carefully chosen pieces of technology, and taking an incremental approach – are winners in any software architecture. ®

Send us news
55 Comments

Chat among yourselves: New EU law may force the big IM platforms to open up

Send an iMessage to Facebook, and we'll talk

The European Parliament's new Digital Markets Act, adopted as a draft law this week, could compel big platforms owned by large firms including Apple, Google, and Facebook to make their tech interoperable.

Among other things, this might mean forcing the tech vendors' messaging apps to allow communication with other services.

If the EU deems a company to be what it calls a "gatekeeper", it could impose "structural or behavioural remedies" – compelling the largest outfits to allow interoperability, or imposing fines. The Act would also restrict what companies could do with personal data – not the first time it's tried.

Continue reading

Sweden asks EU to ban Bitcoin mining because while hydroelectric power is cheap, they need it for other stuff

Lighting and warming homes in winter, or ransoming encrypted files and buying drugs? Hmmm

The directors general of Sweden's Financial Supervisory Authority and Environmental Protection Agency have called upon both the EU and Sweden's government to ban cryptocurrency mining.

Continue reading

The rocky road to better Linux software installation: Containers, containers, containers

Let's be real: Everyone is trying to catch up with Apple

Analysis Linux cross-platform packaging format Flatpak has come under the spotlight this week, with the "fundamental problems inherent in [its] design" criticised in a withering post by Canadian software dev Nicholas Fraser.

Fraser wrote in a blog published on 23 November that "these are not the future of desktop Linux apps," citing a litany of technical, security and usability problems. His assertions about disk usage and sharing of runtimes between apps were hotly disputed by Will Thompson, director of OS at Endless OS Foundation a day later in a post titled: "On Flatpak disk usage and deduplication," but there is no denying it is horribly inefficient.

Most people don't care about that any more, one could argue. But they should.

Continue reading

EU needs more cybersecurity graduates, says ENISA infosec agency – pointing at growing list of master's degree courses

Skills gap needs filling somehow

The EU needs more cybersecurity graduates to plug the political bloc's shortage of skilled infosec bods, according to a report from the ENISA online security agency.

The public sectors of EU countries should "support a unified approach" to infosec-focused higher education, it says, addressing an issue that is by no means unique to the bloc.

In a new report titled "Addressing the EU Cybersecurity Skills Shortage and Gap Through Higher Education", academics Jason Nurse and Konstantinos Adamos, together with ENISA's Athanasios Grammatopoulos and Fabio Di Franco, said the European Union needs to get more students signing up for cybersecurity degrees.

Continue reading

Nuclear fusion firm Pulsar fires up a UK-built hybrid rocket engine

A win in the rocketry world: 'Flames came out of the right end'

UK nuclear fusion outfit Pulsar Fusion has fired up a chemical rocket engine running on a combination of nitrous oxide oxidiser, high-density polyethylene fuel and oxygen.

The acceptance tests of the UK-built rocket were conducted at COTEC, a UK Ministry of Defence site at Salisbury Plain in southern England.

We spoke to CEO of the company, Richard Dinan, in 2018, when he discussed the prospects for fusion power, and the use of the technology for space travel as well as electricity generation. In 2020 he was showing off an ion thruster with plasma running at several million degrees and particles fired at speeds over 20km per second.

Continue reading

Bad news for Tencent: Chinese companies steer employees away from Weixin or WeChat

Middle Kingdom's internet giant: It's a switch to enterprise apps. Try ours?

Managers of large Chinese state-run companies have told employees to delete, shutdown and discontinue use of Tencent messaging app Weixin for work purposes, citing potential security breaches, according to the Wall Street Journal.

The news outlet named China Mobile, China Construction Bank and China National Petroleum among nine companies that confirmed the communication policy change, although none have officially gone on record.

Employees have reportedly also been warned to beware Weixin's sister app, WeChat. No details were given regarding what communication tools personnel were directed to use instead.

Continue reading

Privacy Sandbox saga continues: UK watchdog extracts more commitments from Google over ad tech

Roll up, roll up. Come and be the CMA-approved trustee to keep an eye on the Chocolate Factory's antics

The torrid tale of Google's Privacy Sandbox took another turn today with the UK's Competitions and Markets Authority (CMA) saying it has "secured improved commitments" from the ad giant over the cookie crushing tech.

The CMA's claims come in the wake of yesterday's call by the UK's data watchdog, the Information Commissioner's Office (ICO), for Google and co to sort out the privacy risks posed by ads. The ICO continues to work with the CMA to review the plans of the Mountain View gang.

The investigation by the competition regulator kicked off in January amid worries that Google's intention to change its Chrome browser and phase out third party cookies in favour of a so-called Privacy Sandbox would, in fact, strengthen the megacorp's grip on the online ad ecosystem.

Continue reading

Government-favoured child safety app warned it could violate the UK's Investigatory Powers Act with message-scanning tech

Redesigned SafeToNet feature highlights tech law mess

A company repeatedly endorsed by ministers backing the UK's Online Safety Bill was warned by its lawyers that its technology could breach the Investigatory Powers Act's ban on unlawful interception of communications, The Register can reveal.

SafeToNet, a content-scanning startup whose product is aimed at parents and uses AI to monitor messages sent to and from children's online accounts, had to change its product after being warned that a feature developed for the government-approved app would break the law.

SafeToNet was hailed this week by senior politicians as an example of "new tech in the fight against online child abuse," having previously featured in announcements from the Department for Digital, Culture, Media and Sport over the past 12 months.

Continue reading

Reviving a classic: ThinkPad modder rattles tin to fund new motherboard for 2008's T60 and T61 series of laptops

When vendors don't update old models, someone must step up

The range of Thinkpads you can modernise is getting wider. XyTech is trying to crowdfund a new mainboard for the 2008 T60/T61 so fans can upgrade the much-loved noughties laptop.

"The goal is to recreate the TP experience as much as possible, while incorporating the latest CPUs and technology," XyTech's Xue Yao writes. "As the motherboard is not from [Lenovo], it will require quite a bit of hands-on from the user to get the best experience out of the machine. It will be as stable as any other computer motherboard but will not have original TP software support and features."

XyTech is not alone. CnMod is another small Chinese business that updates teenaged – and by laptop standards, that's positively geriatric – ThinkPads. The replacement motherboards come from cottage-industry scale manufacturers on the forums at 51NB.com. They offer replacement motherboards for various classic ThinkPads, including the X200, X201 and X62, updating them with modern processors, memory and storage. There's also the X330, which combines the classic keyboard of the X220 with the faster mainboard of an X230.

Continue reading

You forced me to use this fancypants app and now you're asking for a printout?

'Just take the meds, Mr Sloper, and enjoy your holiday'

Something for the Weekend, Sir? I could just do with some popcorn right now.

I am loitering among the sick and deranged. The selfish fools decided to pile into the chemist's at 9am, the very moment I sensibly chose to visit. Half of them seem to be loitering around the entrance, jabbing urgently at their smartphones and muttering to themselves.

The popcorn? It will not cure my ailment but, despite research from the Rotterdam School of Management that claims otherwise, popcorn would enhance my user experience (UX) of waiting in the queue.

Continue reading

<abbr title="Bastard Operator from Hell">BOFH</abbr>: What if International Bad Actors designed the vaccine to make us watch more Steven Seagal movies?

Pipe down – Nicolas Cage could be listening

Episode 21 I've got nothing against conspiracy theories in general because if they didn't exist the PFY would probably have to join a book club or a sewing circle. But even the PFY will admit there's a limit, and at lunch today we think we found it ...

"So let me get this straight," I say. "The vaccine contains tiny … robots …"

"Nanobots," the bloke across the table from me chips in.

Continue reading