Netflix reveals massive migration to new mix of microservices, asynchronous workflows and serverless functions

Goes deep on Docker and adopts ‘strangler fig’ pattern to replace legacy platform


Netflix has revealed it’s built a new media ingestion and distribution platform and expects to spend much of 2021 on a migration from what it describes as a “large and complicated legacy system”.

As detailed in the streaming firm’s tech blog, the new platform is called “Cosmos” and is the fourth generation of a tool that is used to “process incoming media files from our partners and studios to make them playable on all devices.”

The previous generation of the platform, called “Reloaded”, has been in use for seven years but was showing its age.

“When Reloaded was designed, we were a small team of developers operating a constrained compute cluster, and focused on one use case: the video/audio processing pipeline,” wrote Netflix senior software engineer Frank San Miguel. “As time passed the number of developers more than tripled, the breadth and depth of our use cases expanded, and our scale increased more than tenfold, the monolithic architecture significantly slowed down the delivery of new features.”

'Strangler fig' lets the new system grow around the old one and eventually replace it completely

“We could no longer expect everyone to possess the specialized knowledge that was necessary to build and deploy new features. Dealing with production issues became an expensive chore that placed a tax on all developers because infrastructure code was all mixed up with application code. The centralized data model that had served us well when we were a small team became a liability.”

Netflix is famously all-in on cloud and microservices, two technologies suggested as paragons of agility. Yet the quotes above do read rather like an admission the company nonetheless accrued technical debt and has a legacy system to replace.

San Miguel’s post describes Cosmos services as “not a microservice but there are similarities.”

His definition of a microservice is “an API with stateless business logic which is autoscaled based on request load … provides strong contracts with its peers while segregating application data and binary dependencies from other systems.”

DaaS-appearing trick: Netflix teases desktops-as-a-service product

READ MORE

“A Cosmos service retains the strong contracts and segregated data/dependencies of a microservice, but adds multi-step workflows and computationally intensive asynchronous serverless functions.” San Miguel wrote.

The developer said that in a typical Cosmos service “clients send requests to a Video encoder service API layer. A set of rules orchestrate workflow steps and a set of serverless functions power domain-specific algorithms. Functions are packaged as Docker images and bring their own media-specific binary dependencies (e.g. Debian packages). They are scaled based on queue size, and may run on tens of thousands of different containers. Requests may take hours or days to complete.”

Cosmos has subsystems, among them one named “Stratum - a serverless layer called for running stateless and computational-intensive functions.”

The post explains a fair bit about how Cosmos prioritises workloads and allocates resources, and how Netflix’s colossal AWS rig is configured to make sure developers and apps have the power they need when they need it.

San Miguel also reveals that Netflix started work on Cosmos in 2018, has used it in production since 2019 and now uses it for around 40 services. The company has chosen the “strangler fig” migration pattern as it “which lets the new system grow around the old one and eventually replace it completely.”

“2021 will be a big year for Cosmos as we move the majority of work from Reloaded into Cosmos, with more developers and much higher load,” San Miguel concludes. “We plan to evolve the programming model to accommodate new use cases. Our goals are to make Cosmos easier to use, more resilient, faster and more efficient.” ®


Other stories you might like

  • Twitter founder Dorsey beats hasty retweet from the board
    As shareholders sue the social network amid Elon Musk's takeover attempt

    Twitter has officially entered the post-Dorsey age: its founder and two-time CEO's board term expired Wednesday, marking the first time the social media company hasn't had him around in some capacity.

    Jack Dorsey announced his resignation as Twitter chief exec in November 2021, and passed the baton to Parag Agrawal while remaining on the board. Now that board term has ended, and Dorsey has stepped down as expected. Agrawal has taken Dorsey's board seat; Salesforce co-CEO Bret Taylor has assumed the role of Twitter's board chair. 

    In his resignation announcement, Dorsey – who co-founded and is CEO of Block (formerly Square) – said having founders leading the companies they created can be severely limiting for an organization and can serve as a single point of failure. "I believe it's critical a company can stand on its own, free of its founder's influence or direction," Dorsey said. He didn't respond to a request for further comment today. 

    Continue reading
  • Snowflake stock drops as some top customers cut usage
    You might say its valuation is melting away

    IPO darling Snowflake's share price took a beating in an already bearish market for tech stocks after filing weaker than expected financial guidance amid a slowdown in orders from some of its largest customers.

    For its first quarter of fiscal 2023, ended April 30, Snowflake's revenue grew 85 percent year-on-year to $422.4 million. The company made an operating loss of $188.8 million, albeit down from $205.6 million a year ago.

    Although surpassing revenue expectations, the cloud-based data warehousing business saw its valuation tumble 16 percent in extended trading on Wednesday. Its stock price dived from $133 apiece to $117 in after-hours trading, and today is cruising back at $127. That stumble arrived amid a general tech stock sell-off some observers said was overdue.

    Continue reading
  • Amazon investors nuke proposed ethics overhaul and say yes to $212m CEO pay
    Workplace safety, labor organizing, sustainability and, um, wage 'fairness' all struck down in vote

    Amazon CEO Andy Jassy's first shareholder meeting was a rousing success for Amazon leadership and Jassy's bank account. But for activist investors intent on making Amazon more open and transparent, it was nothing short of a disaster.

    While actual voting results haven't been released yet, Amazon general counsel David Zapolsky told Reuters that stock owners voted down fifteen shareholder resolutions addressing topics including workplace safety, labor organizing, sustainability, and pay fairness. Amazon's board recommended voting no on all of the proposals.

    Jassy and the board scored additional victories in the form of shareholder approval for board appointments, executive compensation and a 20-for-1 stock split. Jassy's executive compensation package, which is tied to Amazon stock price and mostly delivered as stock awards over a multi-year period, was $212 million in 2021. 

    Continue reading

Biting the hand that feeds IT © 1998–2022