Debian 10: some docker containers are not starting

I just updated several of my boxes to Debian 10 Buster, including one virtual server I have to build and update Docker images for CI purposes.

Suddenly the containers for CentOS6 using centos:centos6 image refused to start for no reason. It didn’t matter if I tried to run sh or bash (for example with docker run -ti -u root --rm centos:centos6 /bin/bash). No error from the command docker at all. Just the prompt again.

The exact same Docker image worked fine on my openSUSE Leap 15.1.

I finally found the problem by looking at dmesg:

kernel: bash[3455] vsyscall attempted with vsyscall=none ip:ffffffffff600400 cs:33 sp:7ffc65288fb8 ax:ffffffffff600400 si:7ffc65289f81 di:0
kernel: bash[3455]: segfault at ffffffffff600400 ip ffffffffff600400 sp 00007ffc65288fb8 error 15

It seems that the new kernel at Debian 10 does not emulate vsyscalls by default (details at bug 852620), and that is needed by CentOS6 containers.

Fixing it is easy:

1. Modify the kernel parameters: at /etc/default/grub add vsyscall=emulate to GRUB_CMDLINE_LINUX_DEFAULT (if for some reason you want it at recovery mode, then GRUB_CMDLINE_LINUX as well)
2. Run update-grub
3. Reboot.

This will reenable the emulation and will allow you to start CentOS6 containers (and any other containers requiring vsyscalls).

If you are using other distributions (or a different kernel) and modifying the kernel parameters, updating grub and rebooting does not help, maybe emulation was just no compiled (see this and this comments at Arch Linux bugtracker).

How to verify if that is your case will depend on your distribution and kernel.

Several ideas:

1. Check /proc/config.gz
2. Check /boot/config-$(uname -r)
3. Install the kernel headers and look for the .config file at the root of the folder where headers are.
4. If you compiled on your own and options 1 and 2 are not working, look for the .config file at the root of the folder where the sources you used are.

And to fix it, you will need to either recompile your kernel, either install a new one with support for vsyscall emulation. But that’s out of the scope of this post.