Australian prisoner-tracking system brought down by 3PAR defects

‘A number of enterprises’ escalated mystery storage problem to HPE HQ

Defects in HPE 3PAR storage area networks caused “a series of abnormal outages” that resulted in systems to track prisoners on release from incarceration in the Australian State of New South Wales (NSW) becoming unavailable.

Prisoner tracking devices and software in the state are provided by a British company called Buddi, which won a tender to provide what the NSW government labelled a “new state-of-the-art electronic monitoring system” designed to “strengthen round-the-clock surveillance of a small group of around 55 parolees and high-risk Extended Supervision Order (ESO) offenders who are already on GPS monitoring in the community.”

Buddi’s offering sees prisoners fitted with tracking anklets so they can be monitored in real time. That equipment was also “… fitted to 430 lower-risk offenders including home detainees who are on in-home curfew monitoring, so they can be subject to GPS spot location checks.”

Buddi’s software runs in GovDC, a pair of data centres that host NSW government infrastructure.

An Australian company, ac3, is accredited to provide infrastructure-as-a-service inside GovDC data centres and to work with the NSW Department of Justice, which awarded the tender. Buddi selected ac3 to run the tracking service.

But ac3’s HPE 3PAR storage for the Buddi service fell over at least four times.

But documents obtained by The Register using a freedom of information request revealed 96 minutes of downtime in October 2015, two hours of downtime in January 2016, five hours in September 2016 and 12 minutes in November 2016.

The NSW Department of Corrections told us that “System outages and faults are infrequent and technical solutions are provided accordingly. “

“When there is a system outage there is no indication on the worn device that this is the case, offenders are not aware that they are not actively monitored at the time.”

A document provided by the NSW Department of Justice attributed the issue to “Compatibility issues with HP 3PAR” that “have had global visibility in the market”.

“ac3 took immediate mitigation action to reduce the risk until a final fix was identified by HP,” the documents say. But the issue required “additional mitigations” that “included alternative storage and implementing multiple back-up systems.”

“The issue was escalated to global executive management at HP with a number of enterprises seeking a resolution to the 3PAR defect.”

Ac3 declined to comment on the outages, the nature of the 3PAE defects or their remediation efforts.

The Register understands that the issues may be similar to the problems that caused prolonged and embarrassing outages at the Australian Taxation Office (ATO). We further understand that DXC’s final report on the root cause of those outages is in the hands of the ATO.

We asked HPE if the ac3 and ATO faults are related. The company has not responded. Nor has it detailed the nature of the fault that felled ac3.

There’s some good news for the vendor: the NSW Justice Department told us “HP has now resolved the issue and the storage sub-system has been monitored closely and verified as stable.”

The nature of the fault, however, remains obscure. ®