Microservices - Netflix

According to Wikipedia Netflix is currently one of the ten biggest internet companies. They started around 2000 with a conventional architecture. Later they performed a radical switch towards a microservice architecture lead by priorities best fitting to their fast-changing market.

Background

Netflix was founded in 1997 by Marc Randolph and Reed Hastings as a DVD rental company. 2000 they began to offer one customers videos based on the videos this customer rented so far. In 2007 they started to stream videos and in 2012 they began to produce their own content. Currently Netflix is available in about 190 countries and provides 250 million hours of video per day for around 98 million customers. 2011 Netflix started Open Connect as a program to work with internet service providers (ISP) as well as a technology including hard and software to provide contend in an optimized way. The Open Connect appliances are installed at the ISP data centers caching the content all around the world.

Netflix earns money with a simple three type subscription model. Customers pay a yearly or monthly subscription fee. They can consume as much content as they like to. The subscription types are differentiated based on video quality and numbers of parallel viewers. In 2018 Netflix made about 15.7 billion $ revenue and had 865 million $ net income with 7,100 employees.

2008 Netflix faced a four days outage because of a database crush. This was the trigger to change the at that time monolithic infrastructure into a service-orientated modern infrastructure. The main goal was a more reliable fault-tolerant infrastructure. The change was led by 3 ordered priorities.

  1. Innovation
  2. Reliability
  3. Efficiency

Innovation was intentionally set before reliability. This led to lose coupling because tight coupling hinders innovation. Tight coupling means a lot of strong dependencies resulting in difficult and rare (often only quarterly) deployments. Based on Conway’s law loose coupling works best with end-to-end ownership or DevOps. Central organizations tend to monolithic architectures. The teams are now responsible for all tasks from the first design steps over production and support to the retirement of a microservice. There is no longer a need for dedicated central testing, operations and support organizations.

In order to pave the way for a microservice architecture, Netflix defined some principles.

This transition had to take place while at the same time keeping the current production business up. For a time, this doubled the effort and cost. Additional problems with data replication and implementing new features twice occurred.

Technology

This diagram shows a simplified picture of the Netflix architecture with focus on the areas not related to the content provisioning.

me

Netflix is accessible via a bunch of different devices with dedicated clients. These clients are implemented as rich clients with Node.js supported by React and RxJS. The elastic load balancer of the Amazon webservice (AWS) cloud is used to connect these devices with Netflix infrastructure. Zuul is a gateway service providing service discovery, load balancing, security, monitoring and resilience. The main principle is non-blocking request and response management. It is built in Java using Netty for asynchronous processing and RxJava for filter chains.

The API gateway is entry point and responsible for the service coordination. As it calls a lot of code not implemented by the same team it evolves into a kind of monolith. The nodes are server components for which the client teams are responsible. They are used to overcome the monolithic character of the API Gateway. Each client team can easily implement their logic in the same language Node.js using all the API functionalities of the next service layer they need. Netflix business logic is built in cascading microservice.

The business logic is implemented in cascading services. As a main infrastructure component Hystrix tracks calls of different service for errors and latency to prevent cascading errors that could compromise the entire system. Netflix is currently replacing it with Resilience4j. Data are mainly stored in Cassandra and MySQL. Both are used for data that need to be stored for a long time. Additionally, different in-memory databases are used for various purposes.

The Open Connect Appliances are specially designed servers with a Netflix software running on it. They are directly installed at the ISP nodes to provide the Netflix content. They do not store customer data. Due to them information about health, performance and content consumption are sent to the central cache control service on a regular basis. The control service collects the information from all OCAs and prepares it for the monitoring and steering of content requests. The steering service is one of the services used when a content request arrives. Based on the information stored by the Control service the Steering Service decides from which OCA the device should request the streaming.

Culture

Company culture is a vital concern for Netflix. They are very proud of it and expect their employees to live according to its guidelines. There are several points in this culture which are very different from other companies. The employees have unlimited vacation days, no expense-approval is needed and there are no yearly performance meetings. The staff is expected to act grown-up, self-responsible people, who are aware what is good for the company and themselves. The main principles are freedom and responsibility.

On the other hand, Netflix lays emphasis on working with the most capable people who know to exploit their full potential. From time to time the managers are required to ask themselves hypothetically, what they would do if an employee wants to leave and whether they would like to hold him or her. If they decide against the employer in question, he or she will be dismissed and will get a generous severance package. This situation creates uncertainty and in combination with the strong work ethic of the coworkers - therefore pressure.

Cloud usage and sharing are important parts of the Netflix culture. Netflix takes pride in not having an own data center. The organization is actively contributing to several open source projects and is often offering frameworks and components developed at Netflix as open source. An important cornerstone of Netflix’ success is its testing effort. They regularly perform chaos test in production. They invest in Chaos Monkey developed as a special component to randomly switch off components and servers. The goal is to enhance stability and resilience of the overall system.

Lessons learned

In his article Mastering Chaos – A Netflix Guide to Microservices John Evans explains, that microservices are not clean and easy small pieces of software, but rather abstractions with a complex internal structure. They may have their own database perhaps with caching. Mostly they are using several internal or external libraries. They are at least dependent on infrastructure services, but usually also on other services. In fact, they are not as independent as often said.

In his article The Paved PaaS To NodeJS Microservices (JavaScript @ Netflix) Yunong Xiao outlines the benefits of a standardization in microservice development. He lists these topics:

In an interview lead by Chris Anderson, the Netflix CEO and chairman Reed Hasting answered the question how Netflix culture has supported the great success with self-produced content as follows: “It is very big on freedom and responsibility. I pride myself on making as few decisions as possible in a quarter. And we are getting better and better at that. There are sometimes I am going a hole quarter without making any decisions.” It is this culture together with the frequently asked question “What comes next?”, that drives innovation and in the end the IT architecture.

Copyright 2021, last updated 11.01.2021

Next post: Microservices - Spotify

References

Wikipedia, List of largest Internet companies, 11.08.2020, https://en.wikipedia.org/wiki/List_of_largest_Internet_companies, found 12.08.2020

Josh Evans [Netflix], Mastering Chaos - A Netflix Guide to Microservices, 22.02.2017, https://www.youtube.com/watch?v=CZ3wIuvmHeM, found 21.08.2019

Neil Hunt [Netflix], How AWS Supports Deployment of New Features and Tools, 02.12.2016, https://www.youtube.com/watch?v=SorHbAiZ918, found 21.08.2019

Mayukh Nair [Medium], How Netflix works: the (hugely simplified) complex stuff that happens every time you hit Play, 17.10.2017, https://medium.com/refraction-tech-everything/how-netflix-works-the-hugely-simplified-complex-stuff-that-happens-every-time-you-hit-play-3a40c9be254b, found 21.08.2019

Author unknown [Netflix], Shareholder Letter Q1 2019, 24.06.2019, https://s22.q4cdn.com/959853165/files/doc_financials/quarterly_reports/2019/q1/FINAL-Q1-19-Shareholder-Letter.pdf, found 09.09.2019

Narendra L [TechDummies], NETFLIX System design - software architecture for Netflix, 24.08.2018, https://www.youtube.com/watch?v=psQzyFfsUGU, found 21.08.2019

Arthur Gonigberg, Zuul's Journey to Non-Blocking, 02.10.2017, https://www.youtube.com/watch?v=2oXqbLhMS_A, found 21.08.2019

Author unknown [Netflix], Netflix Culture, release date unknown, https://jobs.netflix.com/culture, found 21.08.2019

Reed Hastings [Chairman and CEO of Netflix] interviewed by Chris Anderson, How Netflix changed entertainment -- and where it's headed, 12.07.2018, https://www.youtube.com/watch?v=LsAN-TEJfN0, found 21.08.2019

Timothy Stenovec, One Reason For Netflix’s Success — It Treats Employees Like Grownups, 06.12.2017, https://www.huffpost.com/entry/netflix-culture-deck-success_n_6763716, found 21.08.2019

Ruslan Meshenberg, GOTO 2016 - Microservices at Netflix Scale: Principles, Tradeoffs & Lessons Learned, 02.09.2016, https://www.youtube.com/watch?v=57UK46qfBLY, found 04.09.2019