Build Docker Images 10x Faster: Use Layer Caching with Harness CI Cloud

According to the CNCF 2022 annual survey, 44% of respondents use containers for nearly all applications and business segments, and another 35% say they use containers for at least a few production applications. The 2023 StackOverflow Developer Survey found that Docker is a top-used tool by 56% of professional developers.

When developers build Docker images on their workstations, previously-built layers are always available from the filesystem. However, Continuous Integration (CI) environments are intentionally ephemeral to ensure consistency and reproducibility. Implementing Docker layer caching in CI pipelines often requires developers to manage cloud storage buckets, where cached layers are stored between pipeline executions, as well as adding the necessary parameters to Docker build commands. In Harness CI Cloud, enabling Docker layer caching is as simple as checking a box in your Docker image build step.

How Docker Layers are Built

Dockerfiles consist of instructions (such as COPY, RUN, ADD, and FROM), and each instruction results in a layer that is cached within the filesystem. The sequence of these instructions determines whether a layer is reused from the cache or if it must be rebuilt. If a layer has changed since the previous build, that layer and all following layers are rebuilt.

Here is an example Dockerfile for a Ruby application. Each line in the file is an instruction that results in a layer in the Docker image.

In this example, files other than Gemfile and Gemfile.lock (the files that manage dependencies) have changed between builds. Green layers are pulled from the cache, red layers are rebuilt.

The ADD instruction adds all files in the current directory into the resulting layer. Therefore, if any file in the build context changes, Docker rebuilds this layer and the following RUN layer, which installs dependencies. Even if the dependencies didn’t change, Docker rebuilds the RUN layer whenever the ADD layer changes. This is because Docker always rebuilds all subsequent layers that follow a changed layer. This can introduce unnecessary inefficiency and increase build times.

Knowing that the ADD layer changes frequently, here’s the same Dockerfile, rearranged to take advantage of Docker layer caching.

Again, in this example, Gemfile and Gemfile.lock have not changed between builds.

By adding Gemfile and Gemfile.lock as a separate instruction, before installing dependencies, Docker rebuilds the dependencies layer only when these files change. With these instructions, Docker reuses the previously-built layers from the cache, saving valuable developer time.

For optimal Docker layer caching, arrange the instructions in your Dockerfiles from least changed to most changed.

For more information on how to write Dockerfiles that make the most of layer caching, see Docker’s Optimizing builds with cache management documentation.

Docker Layer Caching in Harness CI Pipelines

In Harness CI Cloud, enabling Docker layer caching is as simple as checking a box in your Docker image build step.

In this demo, see how layer caching reduces build time of the Grafana Docker image from seven minutes to just twenty seconds, reducing the build time by 10x.

Try Docker Layer Caching Yourself

To reproduce the above demo in your Harness account, follow these steps.

1. Sign up for your free Harness account
2. Docker layer caching with Harness Cloud is currently behind a feature flag, contact Harness Support to enable the feature in your Harness account
3. Follow this guide
4. The first Docker image build will take about seven minutes
5. When committing a change to the ENTRYPOINT script, the Docker build will only take about twenty seconds

Caching Docker layers is just one of many Harness CI tools to help speed up your pipelines:

• Run builds up to 4x faster using Test Intelligence
• Build Arm Docker images 5x faster
• Avoid rebuilding dependencies between pipelines with Cache Intelligence

If you’d like to learn more, schedule a demo to see how Harness can revolutionize your software development process.