AWS Aurora PostgreSQL versions vanish from the mega-cloud for days, leaving customers in the dark
Disappeared databases since returned with no memory of what happened, speculation of bug abounds
Several versions of Amazon Web Services' Aurora PostgreSQL went missing earlier this week, prompting concern among AWS customers.
Greg Clough, a software engineer who uses the American super-cloud, noticed that AWS Aurora PostgreSQL v10.12, v10.13, and v11.8 vanished from every AWS region on Monday without explanation. And in some areas, v11.7 evaporated as well.
From what we can tell, subscribers' data was not affected by the disappearing act, rather the software could not be deployed. If you wanted to migrate to one of the affected versions, you couldn't. And if you wanted to perform an operation, such as restoring a snapshot, using one of the vanished builds, you'd run into an error long the lines of
cannot find version 10.13 for aurora-postgresql.
Concerned about their well-being, he put up a blog post in the hope of finding out what happened.
Lost time with these database instances will never be recovered but with some explanation, closure and healing may be possible. The most likely explanation is that an AWS insider abducted the PostgreSQL code to squash a rather undesirable bug.
In v10.12, Clough said, "NOT EXISTS" queries fail to perform as expected. In fact, they do the opposite because they ignore the "NOT" operator.
"One workaround is to add 'AND <column> is NOT NULL' to all affected queries," he explained in his internet memo. "I have no information if this also affects v10.13 or v11.8, but I suspect it does as these versions were all pulled from the catalogue at the same time."
PostgreSQL on Amazon Aurora Serverless goes live. Check the price tag before diving in, thoughREAD MORE
The Register understands that the bug appears to be specific to Aurora PostgreSQL as the issue has not been spotted in the open-source PostgreSQL codebase or RDS PostgreSQL.
In the AWS Aurora forum, other customers also raised the alarm about the unexplained absence of certain versions of Aurora PostgreSQL code.
We've asked an AWS spokesperson whether anyone is investigating the incident and whether any suspects in the mysterious disappearance have been identified. Its PR team said they'll have an explanation for us soon.
However, widespread panic appears to have been averted. Earlier this morning, PostgreSQL v10.12, v10.13, and 11.7 returned to AWS Aurora as silently as they'd left, to some AWS regions at least.
That leaves only v11.8 unaccounted for. We hope for its safe return, perhaps with an explanation. ®