'Our hosted pools are under attack by abusers': Azure DevOps enjoys a midweek TITSUP*

Crypto-mining jerks at least partly to blame

Engineers around the world have been given a few hours off today, assuming they are using Microsoft's Azure DevOps Hosted Pools, as the platform buckled under a second wave of attacks.

Issues first cropped up in the early hours of the morning (UK time), afflicting "all geographies" according to Microsoft and was described delicately as "an event" by the engineers attempting to put things back in order.

Over the course of five hours Microsoft's team battled to deal with "the incident" before declaring the issue as "fully mitigated" just as UK workers fired up their workstations to kick off the day's pipeline party.

Sadly, the respite was not to last and by 15:06 UK (14:06 UTC) Ed Glas posted a refreshingly blunt update to the effect that all was not well. "Our hosted pools are under attack by abusers," he said. "We knocked down the first wave of abuse but they came back with another attack. We are working to block this attack."

The exact nature of the attack is unclear at the moment, although Microsoft's wobbly infrastructure allowed a host of the company's services to come crashing down on 1 April, an event attributed to "an anomalous surge in DNS queries."

Uh huh.

Nick, a Register-reading tech boffin, got in touch regarding the problem, and remarked that his outfit had seen build times jump by between 20 and 50 per cent. Other users have been similarly affected and taken their woes to social media.

Experiencing the same issue. It's taking an hour or more to acquire an agent... See also: https://t.co/mZ7ETgP8aQ

— Richard (@badsyntax) April 7, 2021

The service has been a little up and down of late. A shame, because the likes of Ubuntu, Windows and macOS virtual machines courtesy of Microsoft are convenient alternatives to self-hosting.

When they work.

A glimpse at the history shows that the last week has not been a good one for Azure DevOps. As well as this morning's whoopsie, there was also "degradation" yesterday. One user, attempting to use Ubuntu, grumbled that the pipelines for the UK at least "were essentially down all afternoon & evening."

The Register understands that the current issues can be laid, at least partially, at the doorstep of opportunistic crypto-mining.

Free CI/CD (including hosted agents) has been a thing at Azure DevOps since 2018, but Microsoft has spent this year dealing with abuses of its largesse, first by fiddling with the free tier in public projects and, last month, being more selective about the free grant in private projects.

We have contacted Microsoft for more details, and will update should an explanation be forthcoming. ®

* Tottering Infrastructure Topples Software User Programming