Blog of Raivo Laanemets

Docker udev access race

On

I recently upgraded my Docker version to 1.8 and started seeing a warning message "Udev sync is not supported" when trying to start the Docker daemon. I am using the devicemapper storage backend and the daemon would not start.

To force the daemon to start, remove the current devicemapper data:

rm -r /var/lib/docker/devicemapper/

This will make Docker use the potentially dangerous combination with a known access race condition that can make your data disappear. This might be sometimes OK in development until a proper fix can be applied.

Enabling sync

Udev sync can be enabled by compiling dynamically-linked docker binary. Docker is distributed and installed as a static executable by default. You can check whether the installed docker is static or dynamic with the ldd utility. If the command

ldd `which docker`

answers

not a dynamic executable

then it is static while the output similar to

linux-vdso.so.1 (0x00007fff613e5000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc145519000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fc145315000)
libdevmapper.so.1.02 => /lib64/libdevmapper.so.1.02 (0x00007fc1450d9000)
libc.so.6 => /lib64/libc.so.6 (0x00007fc144d10000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc14576b000)
librt.so.1 => /lib64/librt.so.1 (0x00007fc144b08000)
libudev.so.0 => /lib64/libudev.so.0 (0x00007fc1448fa000)

indicates that it is dynamically linked. You can compile dynamic docker binary using the guide here. The guide requires some adjustment depending on your distro and the docker version. Do not blindly copy-paste those commands without knowing what they do.

With dynamic binary the docker daemon should now start up without complaining about udev sync.

Other backends

The issue can be avoided or worked around by using another storage backend. There are some to choose from: btrfs, zfs, aufs, overlayfs.

Btrfs

Btrfs backend assumes that filesystems are stored onto a disk with btrfs filesystem. This requires host machine /var/lib/docker to be on a btrfs filesystem. Of course, it also needs kernel support for the filesystem. My current kernel (custom built) lacks it and it was too unconvenient to create a new partition/install a disk for /var/lib/docker.

ZFS

ZFS backend is similar to the btrfs backend but uses the ZFS filesystem. It is said to be slower than btrfs backend but more stable given that ZFS is older and potentially more mature. This is by the description from the storage drivers documentation linked above.

Aufs

Aufs was a preferred choice for docker backend some time ago. It is not anymore, mostly because it has been rejected from kernel and it is unlikely that it will be ever merged. Aufs requires custom kernel and is distributed as a set of patches. For my machine, it would also require recompilation.

OverlayFS

OverlayFS is similar to Aufs except it is merged and developed into the kernel tree. It's considered to be simple and fast. This makes it probably the best choice in the future. However, it has one very nasty bug/missing feature: it does not support symlinks. This has been reported and aknowledged and is possibly fixed soon. There seems to be strong support for making this the default driver once it becomes more stable.

However, without symlinks I cannot run supervisor inside a docker and this is a deal breaker for me. It would also require kernel recompilation as my current one has no support for it and does not even include the required code (needs 3.18+).

Devicemapper with separate pool

I'm not sure if this would avoid the udev access race but it seems to be recommended for production setups at the moment. It is said to be much faster than devicemapper with loopback devices. The setup is described here.

My setup

For the moment, I have decided to keep using Docker with devicemapper loopback devices, at least on my main desktop machine. This might change once I'm finally forced to compile a new kernel (when Slackware 15 is released!). My main use case for Docker is running test environments. It's much more maintenance-free than running a separate VPS or even a dedicated machine per project. A Dockerfile also gives a clear recipe for building up the system from the clean state. This has avoided me quite many problems (like missing libraries and tools!) I might have faced in production.


Comments

No comments have been added so far.

Email is not displayed anywhere.
URLs (max 3) starting with http:// or https:// can be used. Use @Name to mention someone.