Microservices - Spotify

Spotify is a company with a very interesting culture. Culture is an important part of its business and is often seen as an ideal way of working agile. It is one of the big players which started directly with a microservice architecture. Spotify's success shows how microservices and a strong agile culture support each other.

Background

2006 Daniel Ek and Martin Lorentzon founded Spotify. The Spotify desktop application was launched in October 2008 in Sweden with about 2,000 monthly active users (MAU). One year later they reached 3.2 million MAUs. This success was pushed by a very fast streaming technology and the good coverage of Sweden with fast internet access at this time. In October 2009 Spotify launched their mobile app supporting listening to playlists offline.

2010 the music industry forced Spotify to cap their free desktop application in a way that it could only be used some hours per week or for a limited amount of songs. Based on this the amount of MAUs stagnated and gone slightly down to 7.2 million MAUs. In the same year Spotify was launched in the US. 2011 a partnership was launched between Facebook and Spotify.

In summer 2013 the switch of many people from desktop PCs to mobile phones brought Spotify second time MAUs to a stagnation and small downturn at a level of 24.7 million MAUs. In October 2017 Spotify reacts with free mobile app allowing the user to listen playlist exclusively in shuffle mode. At this particular time there was no other app providing something similar. This increased the number of MAUs in February 2015 to 63 million.

During the next years Spotify spent much effort in AI technologies to suggest users new songs and playlist that meet their preferences. Based on all the stored playlist data it was possible to create dedicated playlists for specific moods or situations and to recommend newly released songs matching the playlists of a user. In 2018 they reached 125 MAUs and went public. Currently they have 191 million MAUs.

In 2016 Spotify migrated its content data (1.5 billion files with 3 Peta Byte (PB)) from AWS to Google cloud platform (GCP). In the same year they started to move their entire IT from their own four data centers to the Google cloud platform (GCP). As Ramon van Alteren described in his article Spotify's Journey to the Cloud the decision was driven by two main reasons:

They had to migrate:

The last two numbers are a matter of peculiar interest. Spotify has got more data processing jobs than microservices. Most likely they are used to prepare analytic data. This is a complex world beneath the microservice architecture. They have far more client and analytical data (100PB) than content data (3PB). In December 2016 Spotify switched 100% of the users to the services on GCP. The first data center was closed in October 2017 the last one in 2018.

Spotify offers different subscriptions. The first is for free whereas the others cost a monthly fee and provide the full Spotify functionality. Currently there are subscriptions for single persons, families and students. Spotify states that it pays 70% to the artists and rights holders for the played audio files. In 2018 Spotify had 191 million MAUs and 87 million subscribers, the revenue amounted to 6 billion $ and the net income to 88 million $ with 4165 employees.

Technology

In the beginning, Spotify was mainly written in PHP but it was switched to Java because people recognized, that more often production incidences were caused by PHP components as by Java components. Currently Spotify is using exclusively Java in their production systems. Therefore, it is easier for developers to move from one team to another. In the background, e.g. for analytics, they are using different languages.

The following diagram shows the layered approach of Spotify’s microservice architecture.

me

The whole traffic from devices passes an Access Points. These points perform authentication and authorization and route the requests to a feature service. These Feature Services correspond to the Feature Partitions of the devices. They are calling nested Frontend Services which use a lot of centrally provided Infrastructure Services.

The Apollo framework is used to ease microservice development. Nearly all microservices are built based on Apollo. Spotify provides it as open source.

System-Z is used as a kind of repository to provide information about all parts of the system. It covers components (e.g. microservices, data stores, pipelines and libraries) and systems storing information about the microservice itself, APIs, dependencies, ownerships and alerting. It also provides real-time deployment as well as monitoring information and supports a lot of tasks around the microservice management.

Helios is used as a container orchestration management tool.

Culture

Spotify works with full stack vertical teams. If a new team is created, it gets a mission quite different from other teams’ missions to make sure there is no big overlap in the team’s tasks. Teams are fully responsible for development and operations. This is facilitated by several infrastructure tools more or less needed by all teams. With creating manageable infrastructure services for the common way to implement tasks like monitoring, Spotify motivates the teams to do it in the common way. For all other ways the team must build the infrastructure itself. Reusable code is normally part of a library.

Spotify tries to avoid dependencies between teams as much as possible. In his article Backend Infrastructure Gösta Forsum phrases it as follows: “All the Spotify code is available to all developers in a transparent code model. This means that all code in the Spotify client, Spotify backend and Spotify infrastructure is available to all the developers at Spotify to read or change. If a squad [team] is blocking on some other squad [team] to make a change in some code, they always have the option to go ahead and make the change themselves.”

During the migration to GCP the data center teams were heavily involved in the preparation process. Beneath other topics the question how their job will look like after migration was solved. These teams in the end were advocates for the cloud migration.

Lessons learned

Beginning with its own data centers Spotify discovered that using the Google infrastructure better suited its business strategy and helped to focus on its core business. Additionally, it benefited from some of the Google technologies which are part of Google’s cloud offering.

Apart from the microservice architecture Spotify has one of the biggest if not the biggest Hadoop installation with 20,000 data processing jobs. This is in parallel to the microservices architecture an entire ‘world’ in itself with massive complex dependencies. All these jobs and dependencies must be maintained like the microservices in a repository. Most likely there are as well dependencies between microservices and Hadoop jobs.

Copyright 2021, last updated 11.01.2021

Next post: Microservices - Uber

References

Gustav Söderström, A Brief History Of Spotify, 10.01.2019, https://www.youtube.com/watch?v=jTM7ZCKEUGM, found 21.08.2019

Author unknown [Wikipedia], https://en.wikipedia.org/wiki/Spotify, found 02.09.2019

Kevin Goldsmith [Spotify], Microservices at Spotify, 23.12.2015, "https://www.youtube.com/watch?v=7LGPeBgNFuU, found 21.08.2019

Peter Måhlén [Spotify], Modelling Microservices at Spotify, 23.02.2016, https://www.youtube.com/watch?v=7XDA044tl8k, found 17.07.2019

Gösta Forsum [Spotify], Backend Infrastructure, 15.03.2013, https://labs.spotify.com/2013/03/15/backend-infrastructure-at-spotify/, found 19.07.2019

Peter-Mark Verwoerd [Google], Max Charas [Google], Ramon van Alteren [Spotify], Josh Baer [Spotify], Spotify's Journey to the Cloud, 26.07.2018, https://www.youtube.com/watch?v=5aBORQim-KM, found 27.08.2019

Nick Heath, Switching clouds: What Spotify learned when it swapped AWS for Google's cloud, 21.10.2020, https://www.techrepublic.com/article/switching-clouds-what-spotify-learned-when-it-swapped-aws-for-googles-cloud/, found 28.08.2019

Henrik Kniberg [Spotify], Spotify engineering culture (part 1), 27.04.2014, https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/, found 21.08.2019

Henrik Kniberg [Spotify], Spotify engineering culture (part 2), 20.09.2014, https://labs.spotify.com/2014/09/20/spotify-engineering-culture-part-2/, found 21.08.2019