What started as a Saturday afternoon nap for SagePay turned into a three-day snooze fest, angering retailers that were as of last night still struggling to process sales in a peak shopping week.
The online payment service, which boasts about letting retailers get on with the business of retailing while it manages multi-channel payments on their behalf, went for a lie-down at around 16.30GMT on 18 November, customers told us. Sage Tweeted this later in the day.
We are currently experiencing issues with MySagePay and the Live Gateway. Our investigations are underway to establish the root cause and resolve the issue.— SagePay Support (@SagePaySupport) November 18, 2017
We will provide another update within the next hour.
The company issued updates in the next hours confirming transactions were still not going through as normal. "We apologise for the impact to processing, a full root cause analysis will be completed when normal services resumes,” it said.
It is not yet clear what wiped the old Protx network (Sage slurped Protx in 2006) off the face of the internet – a source speculated a DDoS which remains unconfirmed – but it returned on Saturday at nearly 10pm GMT “hiding behind Akamai but under a different IP,” our source claimed.
One SagePay business specialising in e-commerce store hosting told us it saw “tickets flood in to our support desk and customer frantically calling about lost business during huge marketing events”.
"Black Friday shopping week" (yes, it was new to us as well) had begun the day before the outage hit and is set to conclude this coming Friday, the day after Thanksgiving in the US, 24 November.
El Reg was further told by the customer that SagePay had switched back to its original IPs later that night after Akamai announced its IP space, but performance and the ability to use the payment system “suffered badly” until mid-day Sunday. Then Sage wrote this:
Transactions have been restored to a normal service and have been stable for a sustained period. We’ll continue to monitor. Root cause analysis is underway, we will take any steps necessary to avoid a further reoccurrence. Please accept our apologies for any impact caused.— SagePay Support (@SagePaySupport) November 19, 2017
The SagePay status page has since carried a green tick to indicate no further service interruptions, but some customers flocked to Twitter to disagree.
Mmm, your system monitor says "System Normal", your twitter status everything is back to normal, but when I speak to your support they're not in agreement. Neither are lots of others looking at the comments.— Daniel Phillips (@kinboshi) November 20, 2017
We're seeing sites currently having intermittent issues with Sagepay.
@SagePaySupport #sagepay 3 days now, when are you going to get this working, or at least acknowledge that there are issues. #lying to your customers is not the way forward we have had no payment gateway for 34 days now pic.twitter.com/GXWbh1o1xA— Latex Leather & Lace (@LatexLeather) November 20, 2017
1:14 PM here and just had another Sagepay not available message. Rang SagePay an hour ago and they insisted the problem was fixed last night. I suggested they take a look at Twitter and stop lying and was hung up on. What lousy service!— WoodenToyShop.co.uk (@WoodenToyShopUK) November 20, 2017
Online card processing by SagePay is having UKwide technical issues which may restrict credit cards payments on our website today - but the Paypal option is still working just fine. Plus you can call the shop to place your order by phone, too. 01316671296.— brewstore (@brewstore) November 20, 2017
Payment processing company SagePay is currently having issues that have been affecting our website - if you have tried to order & are unsure if your payment has been successful, please contact us on email@example.com. We will help in any way we can. Thank you.— Noble Collection UK (@NobleCollUK) November 20, 2017
Another customer opined that akamai.prolexic.com settings added by SagePay on Saturday were causing the issue – and suggested a potential workaround, which apparently worked for some but not for others. We have asked Akamai for comment.
We can confirm that https://t.co/bir6yX5jDN (which sagepay added over the weekend) is the source of the problem. Using a host file and ssh forwarding to a server that doesn't route through that host (e.g. Linode) solves the issue.— Duncan Ball (@Duncan_ThePixel) November 20, 2017
For others the problems persisted into Monday evening.
Still unable to process any payments via Sagepay...... Busiest week of the shopping year. Pathetic!— Tony Fairhurst (@tonywallsauce) November 20, 2017
The web hosting firm said his retail customers were unable to process transactions for around 24 hours, "after spending tens of thousands on marketing".
This morning, there were still isolated reports from customers saying they had issues with the service, though many said their services had swung back into action.
Sorry, we are having technical difficulties taking card payments through the website/sagepay. :confounded::broken_heart: We can still take #paypal and payments over the phone! (Select the pay by phone option during checkout and give us a ring :phone:)— Bureau Direct (@bureaudirect) November 21, 2017
Hi Matt - Our integration to sage is server direct API, it has taken 3 days for the service to start working. Late last night we finally started to see card transactions happen. What we are most annoyed about is the way Sage has handled this situation. #awful— Tony Fairhurst (@tonywallsauce) November 21, 2017
Sage did not answer questions about what took the service down but instead sent us a prepared statement:
"After a disruption we have now returned to a normal service. If any customers are still experiencing issues we are encouraging them to get in touch and working round the clock to support them. Our customers are our first priority and we will continue to monitor."
As El Reg has always said, outages will happen, but it's the way a cloud company deals with them that sets the grownups apart. We've again asked the kids at Sage if they will answer technical questions about the outage. ®