Faster image building thanks to Uber's Makisu

With the exponential growth of FlashDrive, numbers of images built in the last weeks increased at the same time and we needed to find a better solution than regular docker building to reduce building and deployment time.

Uber successfully migrated to Docker in 2017 and build hundreds of images a day for about 3000 micro-services services millions of customers worldwide.

To achieve that level of scalability, Uber engineering team has to build a tool that provides speed and reliability for Docker building images.

While Makisu is initially made for Apache Mesos and Kubernetes orchestrator, we had only very minor updates to make it compatible with FlashWarm, FlashDrive's proprietary orchestrator system.

Makisu :

• requires no elevated privileges, making the build process portable.
• uses a distributed layer cache to improve performance across a build cluster.
• provides flexible layer generation, preventing unnecessary files in images.
• is Docker-compatible, supporting multi-stage builds and common build commands.

One major issue with deploying a tool like Makisu in a multi tenant architecture like FlashDrive is that we needed to eliminate the requirement to alter the DockerFile to make it usable by Makisu.

We achieved that by adding a step before the build process that, altering the DockerFile updated by our customers to make it compatible with Makisu without changing the build logic or order.

Specifically the builder will add the appropriate #!COMMIT flags to the DockerFile allowing Makisu to know where it should generate a new layer during the process.

If you need more informations about Makisu, please check this blog post from Uber's Engineering team or the Github repository of Makisu.