Microservices - Uber

Uber started 2009 with a standard monolithic architecture and switched to a microservices architecture in 2015. This switch was time-consuming and difficult. On its way Uber faced many of the challenges associated with the usage of microservices.

Background

Uber was founded in 2009 as UberCab by Garrett Camp and Travis Kalanick. It started 2010 offering only luxury black cars. 2011 Uber expanded outside the US to Paris and in 2012 to London. Beyond that Uber starts to deliver ice cream in the summer of 2012. In 2013 Uber allowed drivers to use their own cars instead of the luxury limousines in combination with a much lower price. In the same year Uber expanded to Asia. Ever since Uber is constantly expanding its business all around the world. In 2014 Uber launched carpooling and food delivery services in the US. In December 2015 Uber reached 1 billion rides. 6 month later it was 2 billion rides and in May 2017 5 billion rides.

Uber makes a profit of about 20-25% per each ride. This is Uber’s main income. The other 75-80% of the fare are passed to the driver. The fare for each ride is limited by Uber.

Technology

Just like many other companies Uber started with a monolithic system using Python for the application servers and PostgreSQL as database. This system was focused on the business model of the first days. It limited further growth. Therefore, Uber decided 2014/2015 to rewrite its system. Main drivers for this decision where a multiple system meltdown in 2014 and a DB outage in 2015. Both downtimes together lead to strong focus on reliability.

This diagram shows a simplified picture of the Uber architecture with focus on the areas not related to the geo components.

The Web Application Firewall (WAF) is the first access point for all incoming traffic. Each vehicle in use sends a location every four seconds to the server. A classical load balancer is used to distribute the incoming requests.

The WebSocket protocol allows a contiguously open bidirectional connection between the device app and the disco service. This component is implemented in Node.js because Node.js is very well supporting asynchronous processing. The supply service provides all information about a supplier (e.g. a driver with a car with 3 seats etc.). The demand service provides all information about a demand (e.g. there is a group of three people with one child). Both are as well written in Node.js.

Inside the bigger Disco area, the geo servers are used to hold all current locations of the vehicles and calculate the routes as well as estimated arrival times for the demands. These services are using the Google S2 library. This library facilitates geo locating and calculation of distances by converting every latitude and longitude information into a three-dimensional spherical model with an identifier for each cell.

Long term data are stored in a MySQL database with a NoSQL schema-less extension. For short term data Cassandra is used. Redis is used for caching and queuing. Logging events are written into Kafka queues and are backed-up into Hadoop. Beneath the operation systems Uber drives different analytical systems.

Culture

The culture of Uber was not as good as to be expected for a modern internet company. It escalated in the years 2014 to 2017. In June 2017 founder and CEO Travis Kalanick resigned because of complaints of employees and investors about the bad company-culture and some scandals. There were some evidences about bad behavior and sexism. In August 2017 Dara Khosrowshahi was recruited as a new CEO. Up to now he is working to improve the bad internal and external image of Uber.

Lessons learned

2016 Matt Ranney from Uber talked at GOTO 2016 conference in his presentation What I Wish I Had Known Before Scaling Uber to 1000 Services about lessons learned using microservices. At this time, he estimated that Uber has between 800 and 1,000 microservices. Among other topics he mentioned:

It is difficult to change a microservice. It would be better but complicated, if it would be immutable or at least just addable.
A distributed system is much more complex to handle than a monolithic system. It can be very difficult to find errors.
Sometimes a new microservice will be added instead of fixing a problem in an existing microservice.
Even if microservices are independent from each other, interfaces need to be defined according to data types and semantic content. This could be difficult for hundreds of microservices and pear to pear connections.
It is difficult to define the right number of repositories to store the microservice code in.
There should be minimal performance requirements and latency controls for each microservice.
It is difficult to test microservices with similar load and data as in production. When testing new microservices in production it is difficult to separate them from normal production.
Chaos testing (e.g. randomly switching off components or services) is essential for distributed systems but difficult to realize and often developers don’t like it.
Use of and contribution to open source components/projects should be carefully decided.
The freedom of the developer teams to use their own tools, languages and methods has some disadvantages:

The decision for a tool many be driven by intransparent not every time rational factors.
To avoid working together with another team one team can (politicly) decide to build something new.
You can get to religious discussions about the best tools or languages between the teams.
It is hard to share code fragments from one service for another.
It is difficult to change teams. It is difficult for an employee to change into a new team.
Some kind of agreement over all teams for logging/monitoring (incl. performance measurement, dashboarding and tracing of requests) is necessary.

Two years later 2018 Prabhu Krishnamoorthy stated in Building Reliable Microservices @ Uber: An Introduction that Uber has more than:

4,000 microservices
20 mobile apps
500 web apps
10,000 dependencies to internal or open source libraries
10 languages are used
10 different build tools
12,000 repositories to hold code and other assets

Next post: Microservices - Recommendation

References

Narendra L [TechDummies], UBER System design | OLA system design | uber architecture | amazon interview question, 11.08.2018, https://www.youtube.com/watch?v=umWABit-wbk, found 19.07.2019

Matt Ranney [Uber], What I Wish I Had Known Before Scaling Uber to 1000 Services, 28.09.2016, https://www.youtube.com/watch?v=kb-m2fasdDY, found 19.07.2019

Prabhu Krishnamoorthy [Uber], Building Reliable Microservices @ Uber: An Introduction, 22.10.2018, https://www.youtube.com/watch?v=kb-m2fasdDY, found 19.07.2019

Matt Ranney [Uber], What Comes after Microservices?, 20.10.2017, https://www.youtube.com/watch?v=kb-m2fasdDY, found 21.08.2019

Dara Khosrowshahi, Uber’s new cultural norms, 07.11.2017, https://www.youtube.com/watch?v=kb-m2fasdDY, found 21.08.2019

Edoardo Zorzoli, What Uber’s Story Can Teach Us About Company Culture, 28.06.2017, https://wearebeem.com/what-the-uber-story-can-teach-us-about-company-culture/, found 21.08.2019

Stackshare author unknown, Stack History: A Timeline of Uber's Tech Stack Evolution, https://stackshare.io/stack-history-timeline-uber-tech-stack-evolution release date unknown, found 25.07.2019

Ryan McKillen, What is the technology stack behind Uber?, 06.03.2012, https://www.quora.com/What-is-the-technology-stack-behind-Uber/answer/Ryan-McKillen?ref=stackshare, found 25.07.2019

Reza Shiftehfar, Uber’s Big Data Platform: 100+ Petabytes with Minute Latency, 17.10.2018, https://eng.uber.com/uber-big-data-platform/?ref=stackshare, found 25.07.2019

Rene Schmidt, Project Mezzanine: The Great Migration, 28.07.2019, https://eng.uber.com/mezzanine-migration/?ref=stackshare, found 25.07.2019

Lucie Lozinski, The Uber Engineering Tech Stack - Part I: The Foundation, 19.07.2016, https://eng.uber.com/tech-stack-part-one/?ref=stackshare, found 25.07.2019

Lucie Lozinski, The Uber Engineering Tech Stack - Part II: The Edge and Beyond, 21.07.2016, https://eng.uber.com/tech-stack-part-two/?ref=stackshare, found 25.07.2019

Matt Ranney, Scaling Uber’s Real-time Market Platform, 14.07.2015, https://www.infoq.com/presentations/uber-market-platform/, found 26.07.2019