I wanted to play with Docker swarm on a local machine to test a couple of scenarios. The goal was to run three manager nodes, and three worker nodes. I did not want to use a virtual machine to run five nodes on my computer, and I decided to use LXD. When using LXC or LXD containers, I usually try to use Alpine Linux for its small size, unless there are specific requirements.

First, I initialized the swarm on my local machine:

Then I creates two manager nodes, and three worker nodes. To create a container, which will run Docker, you need to set security.nesting=true.

OK, here it goes:

Now, docker node ls shows something like this:

However, when I tried to deploy my stack into the swarm, I faced the problem: Docker was unable to deploy any services to LXC nodes because of the following error:

Error response from daemon: cgroups: cannot find cgroup mount destination: unknown.

To make sure that this is not related to the way I have configured my containers (e.g., it’s not an issue with AppArmor, etc), I have configured another container, this time it was Ubuntu-based:

That worked: some services got deployed to the Ubuntu worker. This means that the problem was somewhere in the Alpine 🙁

I started to dig deeper.

When starting docker (rc-service docker start), I noticed mount: permission denied errors:

OK, let us see what mount | grep cgroup shows:

…and what are the subdirectories in /sys/fs/cgroup/:

We see that cpu, cpuacct, net_cls, net_prio are not mounted. And indeed, if you try to mount any of them, you will get an error:

OK, now let us see how Ubuntu handles that:

We see that it combines net_cls and net_prio into a single thing, and does the same to cpu and cpuacct.

No problem, let us go back to Alpine and add these mounts:

mount gave no “permission denied” errors; however, docker is still unable to launch any containers:

When looking at /etc/init.d/cgroups, I saw the following piece of code:

However, I did not see /sys/fs/cgroup/openrc in the mount list. And indeed, if I try to mount it manually, it fails with infamous “permission denied” error.

There was one unanswered question, and then another one that gave me a clue:

So, we do not have name=openrc there, nor do we have separate cpu, cpuacct, net_cls, and net_prio (and now it makes it clear to me why Ubuntu used cpu,cpuacct and net_cls,net_prio.

OK, instead of

I tried

…and it worked!

I intentionally did not change paths under /sys/fs/cgroup in order not to break OpenRC’s cgroup-release-agent.sh.

Success!

So, what are the changes? After cgroups start, we need to run the following piece of code:

For the sake of simplicity I decided not to parse /proc/1/cgroups

OK, now let us create a service that runs these commands:

Save this as /etc/init.d/cgroups-patch, then

and then reboot.

Once the container is up, docker run -it --rm alpine ash works.

How to Run Docker in Alpine Container in LXC/LXD
Tagged on:                 

Leave a Reply

Your email address will not be published. Required fields are marked *