Even Facebook struggles: Zuck's titanic database upgrade hits numerous legacy software bergs

The Social Network™ has spent years trying to hop from MySQL 5.6 to 8.0 and still isn’t done


Facebook has had all sorts of no fun trying to migrate from MySQL 5.6 to version 8.0.

A post from the social network’s engineering team reveals that Facebook's last MySQL upgrade – to version 5.6 – took "more than a year".

Moving to version 8.0 has taken "a few years so far".

Some of the reasons for the slow rollout will sound familiar – such as continued work on legacy software, even as new infrastructure was being implemented.

Facebook had custom code – over 1700 patches in its in-house branch of MySQL 5.6. Even as it ported them to 8.0, the company was adding more custom features to version 5.6 that also needed to be dragged into the future.

Other parts of Facebook legacy code wouldn't work with MySQL 8.0, and therefore "required a deprecation and migration path forward".

"Upgrading from 5.6 to 8.0 skipped over 5.7 entirely. APIs that were active in 5.6 would have been deprecated in 5.7 and possibly removed in 8.0, requiring us to update any application using the now-removed APIs," wrote Facebook software engineer Herman Lee and production engineering manager Pradeep Nayak.

Complicating matters further was Facebook's use of the MyRocks project, an effort that lets MySQL use RocksDB as its storage backend, and which Facebook was developing during its migration to MySQL 5.6.

Jumping from version 5.6 to 8,0 – and skipping version 5.7 – meant Facebook could not upgrade servers in place, so had to use logical dump and restore to build a new server.

"However, for very large mysqld instances, this can take many days on a live production server and this fragile process will likely be interrupted before it can complete. For these large instances, we had to modify our backup and restore systems to handle the rebuild," Lee and Nayak wrote.

Facebook tracked the porting process of its custom patches using … wait for it … spreadsheets – and recorded discrepancies as they went.

"Discrepancies on porting status would inevitably arise due to the large number of patches we needed to sift through and these notes helped us resolve them," the pair wrote.

For migration, Facebook hatched a plan to create secondary instances of MySQL databases, then copy data from version 5.6. Once that was done, the secondary version 8.0 instance would be copied to the primary.

To get that done at Facebook scale, "we needed to build new software infrastructure", the post states matter-of-factly.

We had to modify our backup and restore systems to handle the rebuild

Other issues that cropped up included increased memory usage, file format incompatibilities, and conflicts between newly adopted keywords and "popular table column names and aliases used in application queries".

The tone used by the post's authors suggests the project has made steady progress, with glitches to be overcome rather than setbacks that derailed progress or left project timelines in tatters.

The post ends by saying the apps Facebook has moved are benefiting from their new database environment.

"Overall, the new version greatly expands on what we can do with MySQL @ Facebook," Lee and Nayak conclude.

Whether that's good for the rest of us is, as ever, the question Facebook struggles to answer. ®

Similar topics


Other stories you might like

  • Google sours on legacy G Suite freeloaders, demands fee or flee

    Free incarnation of online app package, which became Workplace, is going away

    Google has served eviction notices to its legacy G Suite squatters: the free service will no longer be available in four months and existing users can either pay for a Google Workspace subscription or export their data and take their not particularly valuable businesses elsewhere.

    "If you have the G Suite legacy free edition, you need to upgrade to a paid Google Workspace subscription to keep your services," the company said in a recently revised support document. "The G Suite legacy free edition will no longer be available starting May 1, 2022."

    Continue reading
  • SpaceX Starlink sat streaks now present in nearly a fifth of all astronomical images snapped by Caltech telescope

    Annoying, maybe – but totally ruining this science, maybe not

    SpaceX’s Starlink satellites appear in about a fifth of all images snapped by the Zwicky Transient Facility (ZTF), a camera attached to the Samuel Oschin Telescope in California, which is used by astronomers to study supernovae, gamma ray bursts, asteroids, and suchlike.

    A study led by Przemek Mróz, a former postdoctoral scholar at the California Institute of Technology (Caltech) and now a researcher at the University of Warsaw in Poland, analysed the current and future effects of Starlink satellites on the ZTF. The telescope and camera are housed at the Palomar Observatory, which is operated by Caltech.

    The team of astronomers found 5,301 streaks leftover from the moving satellites in images taken by the instrument between November 2019 and September 2021, according to their paper on the subject, published in the Astrophysical Journal Letters this week.

    Continue reading
  • AI tool finds hundreds of genes related to human motor neuron disease

    Breakthrough could lead to development of drugs to target illness

    A machine-learning algorithm has helped scientists find 690 human genes associated with a higher risk of developing motor neuron disease, according to research published in Cell this week.

    Neuronal cells in the central nervous system and brain break down and die in people with motor neuron disease, like amyotrophic lateral sclerosis (ALS) more commonly known as Lou Gehrig's disease, named after the baseball player who developed it. They lose control over their bodies, and as the disease progresses patients become completely paralyzed. There is currently no verified cure for ALS.

    Motor neuron disease typically affects people in old age and its causes are unknown. Johnathan Cooper-Knock, a clinical lecturer at the University of Sheffield in England and leader of Project MinE, an ambitious effort to perform whole genome sequencing of ALS, believes that understanding how genes affect cellular function could help scientists develop new drugs to treat the disease.

    Continue reading

Biting the hand that feeds IT © 1998–2022