Development Containers, or DevContainers, represent a paradigm shift in software development methodology, encapsulating comprehensive development environments within containerized configurations. This approach addresses the imperative need for consistency across diverse development environments by packaging essential tools, libraries, and configurations within self-contained units. Developers can utilize DevContainers to seamlessly share and replicate their development setups, fostering collaboration and enabling a hassle-free onboarding process for new team members. This enhances efficiency, reduces setup time, and mitigates compatibility issues, ultimately streamlining the development workflow.

The DevContainers CLI, denoted as @devcontainers/cli, is a robust command-line interface integral to managing and manipulating development containers. This CLI empowers developers to orchestrate containerized environments efficiently, providing granular control over configurations and dependencies.

I leverage the DevContainers CLI for constructing images and incorporating features essential to my daily development tasks. Recently, I encountered an unexpected behavior, requiring a dedicated effort to troubleshoot and resolve the issue. In this post, I aim to share the insights gained and the solutions devised during this experience.

The conventional method employed in CI/CD pipelines typically involves:

devcontainer build \
    --workspace-folder "some-path" \
    --image-name "name:tag" \
    --platform linux/amd64,linux/arm64 \
    --push

In most cases, this approach functions seamlessly and effectively fulfills its purpose. Yet, when your container configuration uses DevContainer features, and you intend to push the resultant image into a registry, unforeseen complications may arise.

Let’s consider an example: suppose you’re developing in C++ and want to create a base image that automatically installs your preferred VSCode extensions. You utilize the mcr.microsoft.com/devcontainers/cpp image as the foundation and then introduce your specific customizations. Your container configuration might resemble the following:

{
    "name": "Sample container",
    "image": "mcr.microsoft.com/vscode/devcontainers/cpp",
    "customizations": {
        "vscode": {
            "extensions": [
                "akiramiyakoda.cppincludeguard",
                "xaver.clang-format",
                "github.copilot",
                "GitHub.copilot-chat"
            ]
        }
    }
}

Notably, the base image is built with the git feature.

Now, let’s build the image and push it to the registry:

devcontainer build \
  --image-name "ghcr.io/username/codespaces/my-cpp-image:latest" \
  --workspace-folder . --log-level debug \
  --push

You will see something similar to this (I have omitted irrelevant lines for the sake of brevity):

[3 ms] @devcontainers/cli 0.56.1. Node.js v20.9.0. linux 6.5.0-14-generic x64.
[3 ms] Start: Run: docker buildx version
[38 ms] github.com/docker/buildx v0.12.1 30feaa1
[38 ms] 
[211 ms] Start: Run: docker inspect --type image mcr.microsoft.com/vscode/devcontainers/cpp
[1786 ms] Start: Run: docker inspect --type image mcr.microsoft.com/vscode/devcontainers/cpp
[2960 ms] Start: Run: docker buildx build --push --build-arg _DEV_CONTAINERS_BASE_IMAGE=mcr.microsoft.com/vscode/devcontainers/cpp --target dev_containers_target_stage -t vsc-t-072e9ca346c27581bb269b320965889adf4c4ab1737e2bf2ca6c5874df42c80a-features -f /tmp/devcontainercli-volodymyr/container-features/0.56.1-1707128869943/Dockerfile.extended /tmp/devcontainercli-volodymyr/empty-folder
[+] Building 2.7s (5/5) FINISHED                                                                                                                                                                                            
...
------
 > exporting to image:
------
ERROR: failed to solve: failed to push vsc-t-072e9ca346c27581bb269b320965889adf4c4ab1737e2bf2ca6c5874df42c80a-features: server message: insufficient_scope: authorization failed

The CLI disregards the name specified in the --image-name option, opting instead for its autogenerated label. Compounding the issue, it pushes the resulting image under this autogenerated name.

The apparent solution will be to remove the --push option and see what happens:

[4 ms] @devcontainers/cli 0.56.1. Node.js v20.9.0. linux 6.5.0-14-generic x64.
[4 ms] Start: Run: docker buildx version
[36 ms] github.com/docker/buildx v0.12.1 30feaa1
[36 ms] 
[215 ms] Start: Run: docker inspect --type image mcr.microsoft.com/vscode/devcontainers/cpp
[1514 ms] Start: Run: docker inspect --type image mcr.microsoft.com/vscode/devcontainers/cpp
[2481 ms] Start: Run: docker buildx build --load --build-arg _DEV_CONTAINERS_BASE_IMAGE=mcr.microsoft.com/vscode/devcontainers/cpp --target dev_containers_target_stage -t vsc-t-072e9ca346c27581bb269b320965889adf4c4ab1737e2bf2ca6c5874df42c80a-features -f /tmp/devcontainercli-volodymyr/container-features/0.56.1-1707129710060/Dockerfile.extended /tmp/devcontainercli-volodymyr/empty-folder
[+] Building 3.9s (6/6) FINISHED                                                                                                                                                                                            
...
[6736 ms] Start: Run: docker tag vsc-t-072e9ca346c27581bb269b320965889adf4c4ab1737e2bf2ca6c5874df42c80a-features ghcr.io/username/codespaces/my-cpp-image:latest
{"outcome":"success","imageName":["ghcr.io/username/codespaces/my-cpp-image:latest"]}

Upon completing the image build, the CLI tags the image with the name we passed in the command line. docker image ls ghcr.io/username/codespaces/my-cpp-image confirms that the image exists, and we can now run docker push to push it to the registry.

Now, let’s do something more complex. Suppose you are developing on both your Intel PC and an ARM MacBook, desiring the availability of your image for both architectures (linux/amd64 and linux/arm64). DevContainers CLI seemingly has you covered — you just pass the --platform linux/amd64,linux/arm64 argument in the command line and anticipate a seamless process. Or perhaps, not quite as straightforward as expected.

devcontainer build \
  --image-name "ghcr.io/username/codespaces/my-cpp-image:latest" \
  --workspace-folder . --log-level debug \
  --platform linux/amd64,linux/arm64

This will result in an error: docker exporter does not currently support exporting manifest lists.

A quick Google search leads to this GitHub issue, but regrettably, the information provided here is not helpful.

DevContainer CLI supports the --output option, allowing you to override the default behavior of loading built images into the local Docker registry. It supports the same options as docker buildx build‘s --output parameter. Looking at the image option, we see:

The image exporter writes the build result as an image or a manifest list. When using docker driver the image will appear in docker images. 

Let us give it a try:

devcontainer build \
  --image-name "ghcr.io/username/codespaces/my-cpp-image:latest" \
  --workspace-folder . \
  --platform linux/amd64,linux/arm64 \
  --output type=image

But it fails:

ERROR: Multi-platform build is not supported for the docker driver.
Switch to a different driver, or turn on the containerd image store, and try again.
Learn more at https://docs.docker.com/go/build-multi-platform/

The recommended solution is to create a new builder that utilizes the docker-container driver.

docker buildx create --driver=docker-container --name build-container
docker buildx use build-container

Let’s retry the build:

[4 ms] @devcontainers/cli 0.56.1. Node.js v20.9.0. linux 6.5.0-14-generic x64.
[4 ms] Start: Run: docker buildx version
[39 ms] github.com/docker/buildx v0.12.1 30feaa1
[39 ms] 
[247 ms] Start: Run: docker inspect --type image mcr.microsoft.com/vscode/devcontainers/cpp
[1665 ms] Start: Run: docker inspect --type image mcr.microsoft.com/vscode/devcontainers/cpp
[2617 ms] Start: Run: docker buildx build --platform linux/amd64,linux/arm64 --output type=image --build-arg _DEV_CONTAINERS_BASE_IMAGE=mcr.microsoft.com/vscode/devcontainers/cpp --target dev_containers_target_stage -t vsc-t-072e9ca346c27581bb269b320965889adf4c4ab1737e2bf2ca6c5874df42c80a-features -f /tmp/devcontainercli-volodymyr/container-features/0.56.1-1707131698995/Dockerfile.extended /tmp/devcontainercli-volodymyr/empty-folder
[+] Building 1.0s (7/7) FINISHED                                                                                                                                                                                            
...
[3944 ms] Start: Run: docker tag vsc-t-072e9ca346c27581bb269b320965889adf4c4ab1737e2bf2ca6c5874df42c80a-features ghcr.io/username/codespaces/my-cpp-image:latest
Error response from daemon: No such image: vsc-t-072e9ca346c27581bb269b320965889adf4c4ab1737e2bf2ca6c5874df42c80a-features:latest
{"outcome":"error","message":"Command failed: docker tag vsc-t-072e9ca346c27581bb269b320965889adf4c4ab1737e2bf2ca6c5874df42c80a-features ghcr.io/username/codespaces/my-cpp-image:latest","description":"An error occurred building the container."}

The CLI tried to tag the built image with the name we passed in the command line, but Docker could not find the image it had built.

Returning to the description of the image exporter:

When using docker driver the image will appear in docker images.

However, given that we used the docker-container driver, this approach does not yield the desired outcome.

There is another exporter, docker:

The docker export type writes the single-platform result image as a Docker image specification tarball on the client. Tarballs created by this exporter are also OCI compatible.

The default image store in Docker Engine doesn’t support loading multi-platform images. You can enable the containerd image store, or push multi-platform images is to directly push to a registry, see registry.

We obviously cannot push our multi-platform image to the registry because of the tagging mechanism employed by DevContainer CLI. The remaining viable option is to enable the containerd image store. Fortunately, this is easy because the instructions are clear.

Let’s apply the changes, restart the Docker daemon, and retry the build with --output type=image:

$ devcontainer build --image-name "ghcr.io/username/codespaces/my-cpp-image:latest" --workspace-folder . --log-level debug --platform linux/amd64,linux/arm64 --output type=docker
[4 ms] @devcontainers/cli 0.56.1. Node.js v20.9.0. linux 6.5.0-14-generic x64.
[4 ms] Start: Run: docker buildx version
[40 ms] github.com/docker/buildx v0.12.1 30feaa1
[40 ms] 
[43 ms] Start: Run: docker inspect --type image mcr.microsoft.com/vscode/devcontainers/cpp
[1097 ms] Start: Run: docker inspect --type image mcr.microsoft.com/vscode/devcontainers/cpp
[2073 ms] Start: Run: docker buildx build --platform linux/amd64,linux/arm64 --output type=docker --build-arg _DEV_CONTAINERS_BASE_IMAGE=mcr.microsoft.com/vscode/devcontainers/cpp --target dev_containers_target_stage -t vsc-t-072e9ca346c27581bb269b320965889adf4c4ab1737e2bf2ca6c5874df42c80a-features -f /tmp/devcontainercli-volodymyr/container-features/0.56.1-1707132741473/Dockerfile.extended /tmp/devcontainercli-volodymyr/empty-folder
[+] Building 97.2s (8/8) FINISHED                                                                                                                                                                                           ...
[99607 ms] Start: Run: docker tag vsc-t-072e9ca346c27581bb269b320965889adf4c4ab1737e2bf2ca6c5874df42c80a-features ghcr.io/username/codespaces/my-cpp-image:latest
{"outcome":"success","imageName":["ghcr.io/username/codespaces/my-cpp-image:latest"]}

It now works; we can run docker push to push the built image to the registry.

Bonus: how to set up GitHub Actions to build a DevContainer image:

      - name: Expose GitHub Runtime
        uses: crazy-max/[email protected]

      - name: Set up Docker
        uses: crazy-max/[email protected]
        with:
          daemon-config: |
            {
              "features": {
                "containerd-snapshotter": true
              }
            }

      - name: Set up QEMU
        uses: docker/[email protected]

      - name: Set up Docker Buildx
        uses: docker/[email protected]

      - name: Log in to GitHub Docker Registry
        uses: docker/[email protected]
        with:
          registry: https://ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Install @devcontainers/cli
        run: npm install -g @devcontainers/cli

      - name: Build image
        run: |
          devcontainer build \
            --workspace-folder '...' \
            --platform linux/amd64,linux/arm64 \
            --image-name=<image-name>:<tag1> \
            --image-name=<image-name>:<tag2> \
            --output type=docker \
            --cache-from type=gha \
            --cache-to type=gha,mode=max

      - name: Publish image
        run: docker push --all-tags '<image-name>'
Using DevContainer CLI to Build Multi-Platform Images with Embedded Features
Tagged on:             

2 thoughts on “Using DevContainer CLI to Build Multi-Platform Images with Embedded Features”

Leave a Reply

Your email address will not be published. Required fields are marked *