NHS reply-all meltdown swamped system with half a billion emails

Accenture blamed for system swamp

The NHS reply-all email fail last year involved 500 million emails being sent across the health service's network in just 75 minutes.

A test message sent on 14 November to what an unfortunate "senior associate ICT delivery facilitator" thought was a local distribution list she had created instead went to all 850,000 people with an NHSmail email account.

The blank message, sent early in the morning with a subject line that simply read "test", was sent to a distribution list called "CroydonPractices". Around 80 irritated folk promptly hit "reply" to demand they be removed from the list – which was when the meltdown began.

"NHSmail's Dynamic Distribution List (DDL) functionality allows administrators to create distribution lists using a range of options and rules," said an official report into the meltdown, which is contained in papers being submitted to the NHS Digital board tomorrow by James Hawkins, NHS Digital's director of programmes. The organisation's chief operating officer, Rob Shaw, approved the paper, which has been seen by The Register.

The local admin selected the "only in my organisation" rule, which she thought would restrict the distribution list to her South London clinical commissioning group.

"A software configuration error meant that the system applied an 'All England' rule rather than one including only the administrator's organisation," continued the report on the snafu. "The administrator would not have known that this had occurred."

Sources contacted The Register at the time to say their emails had been delayed by up to three hours at midday. The report revealed that half a billion emails crossed the NHS network between 0829 and 0945 that day, against the usual traffic volume of three to five million emails per day. It also claimed that the service "did not crash at any point", though it confessed to "significant service delays for the majority of the day".

The NHS report appeared to blame IT outsourcing outfit Accenture for not having failsafes in place that would have prevented the cockup, stating that one of the NHSmail system's design requirements was that "strict controls must be in place to limit the volume of any one email sent by an individual user or local administrator".

"This functionality is still to be delivered by Accenture," the report noted stiffly, adding: "The ability to create DDLs of similar form will remain disabled until NHS Digital is satisfied this has been delivered."

We've contacted Accenture for comment. ®