Opayo, the payment processor formerly known as Sage Pay, has not been having a great Thursday, as an unknown systems failure took its systems offline - preventing customers from accepting any payments.
The biz is no small beans: it provides payment processing for companies including Europcar, Krispy Kreme, easyJet, Bunches, Office Shoes, Murco Petroleum, and even provides the technology behind the BBC's Children In Need donation app.
The trouble started in the early hours, with Opayo's status reports on the glitch beginning at 04:20 UTC. Initially, the issue appeared to be limited to the company's Live Payment Gateway - a bad part of the system to go down right before shops begin to open - but it soon spread.
Opayo's customers soon started complaining of payment failures, an inability to use the e-comms system, and being locked out of MySagePay - a portal which has yet to undergo the same rebranding as the company operating it. By 07:13 UTC, Opayo thought it had the problem licked.
"Our technology teams have resolved the issue as of 8:13am [07:13 UTC]," the company posted to its status portal. "Please accept our apologies for the disruption to your service."
This was soon followed by a retraction, warning roughly an hour later of "further connectivity issues" which continued through the morning. "Though transactions are processing," Opayo admitted, "this is not consistent and some customers may see limited volume to no transactions."
The problem spread further, too: "Our MySagePay log in is also impacted," an update at 10:00 UTC admitted, "and our test platforms too" - the latter possibly explaining why finding the root cause of the flaw and fixing it was taking so long.
By 11:30 UTC, Opayo's engineers claimed to be "seeing a strong reduction in the connectivity issues."
That was enough for some of the company's customers to tell their customers that the issue was resolved and payments once more flowing correctly - though at the time of writing some, such as Brighton-based gift specialist Creased Cards, continued to report problems and recommended using PayPal as an alternative payment method.
Expressing his frustrations on Twitter, web developer Matt Freeman didn't hold back: "When Opayo/SagePay has downtime its always massive," he wrote. "They are true clowns, from API design (OMG its so bad compared to Adyen, Stripe, PayPal, anything out there), to tech support, to half-assed docs, no hugops for them as there won't be real post-mortem and lesson won't be learnt."
Thus far, Opayo hasn't announced what caused the extended outage - though it came at a time when a number of Australian banking services had been knocked offline by issues at Akamai, seemingly linked to its Prolexic platform for Distributed Denial of Service (DDoS) protection.
Opayo has been contacted for comment, but at the time of writing had not provided a response. ®