Primer on Linux container filesystems
Primer on Linux container filesystems
Building a container filesytem by hand
I spent the weekend building a toy Docker clone. One question going into this project was how each container gets its own filesystem. Let’s first reverse-engineer what Docker does and then replicate it ourselves.
I’ll start by starting a shell in a Docker container using Alpine image.
michal@michal-lg:~$ docker run -it --entrypoint /bin/sh --rm --name "alpine-container" alpine
/ # ls
bin dev etc home lib media mnt opt proc root run sbin srv sys tmp usr var
/ # hostname
a7cbf0aea1ad
/ # cd home && lsWe are in a separate filesystem and it’s pretty empty. Let’s make a file.
/ # echo -e "Hello there\nGeneral Kenobi" > /home/hello_there.txt
/ # cat /home/hello_there.txt
Hello there
General KenobiWhat do you think, can we access this file from the host?
…
…
…
We can - let’s find it!
Docker stores everything under /var/lib/docker so we can start looking from there from a second terminal:
root@michal-lg:/var/lib/docker# find -name hello_there.txt
./overlay2/1557145fe40a1595d090eeafa72c39a7b54cca4791ae9e3ffafabff06466125c/diff/home/hello_there.txt
./overlay2/1557145fe40a1595d090eeafa72c39a7b54cca4791ae9e3ffafabff06466125c/merged/home/hello_there.txt
root@michal-lg:/var/lib/docker# Curiously, we found the file twice in two different directories. Let’s see what else is in the diff and merged directories.
root@michal-lg:/var/lib/docker/.../diff# ls
home root
root@michal-lg:/var/lib/docker/.../merged# ls
bin dev etc home lib media mnt opt proc root run sbin srv sys tmp usr varThe diff directory only contains an empty root directory and a home directory with the file we created. Contents of merged exactly match the container’s filesystem.
Overlayfs
Docker uses overlayfs filesystem. Overlayfs lets us combine 2 file file trees, “lower” and “upper”, into a combined view “merged”. Docker calls the “upper” file tree “diff”, which is perhaps more fitting and I’ll be referring to it as such.
Usage of union filesystems like overlayfs comes from an interesting observation: Often we want to run multiple containers on a single host. Chances are, these containers might share the lower layer - be it Alpine, Ubuntu, or a more specialized one like golang.
By making the lower layer read-only, multiple containers can share it. Changes are only written to the upper layer. Let’s illustrate what happened when I earlier created hello_there.txt. I’m not expanding all folders to avoid clutter.
When we then create hello_there.txt in /home, it was written to diff and overlayfs constructed a combined view in merged.
What if we modify something in the lower layer? Let’s rename /bin/echo to /bin/echo.old.
/ # cd bin
/bin # mv echo echo.old
/bin # ls
...
chgrp echo.old gzip ln mount printenv setserial umount
...As mentioned, the lower layer is read-only so the only place where the file is actually modified is in diff.
root@michal-lg:/var/lib/docker/overlay2/.../diff/bin# ls -l
total 0
c--------- 1 root root 0, 0 Nov 11 23:06 echo
lrwxrwxrwx 1 root root 12 Sep 6 13:34 echo.old -> /bin/busyboxThere are two files! One for the no longer existing echo and one for echo.old. Overlayfs uses special whiteout files to deal with deletion of a file. When overlayfs sees this file, it knows not to include it in the merged view.
The second file is much less interesting, it’s the renamed echo, which turns out was just a symbolic link to busybox.
Creating container filesystem
Finally, let’s see how Docker uses Overlayfs under the hood to create a new filesystem.
First, let’s create temporary directories in /tmp/ where we’ll setup a container filesystem manually.
michal@michal-lg:/tmp$ mkdir -p /tmp/container-demo/{diff,merged,work}
michal@michal-lg:/tmp$ ls container-demo/
diff merged workWe’ve seen diff and merged before. work is used by overlayfs as a scratchpad for and we don’t need to care about it.
Next, download Alpine minirootfs for your CPU architecture and extract it to /tmp/container-demo/. I’ll rename the extracted folder to “alpine”.
michal@michal-lg:/tmp/container-demo$ ls
alpine diff merged workNow that everything is setup, we can mount the overlayfs filesystem.
michal@michal-lg:/tmp/container-demo$ sudo mount -t overlay overlay -o lowerdir=alpine,upperdir=diff,workdir=work mergedNow if we list contents of merged, we’ll see the Alpine file system:
michal@michal-lg:/tmp/container-demo/merged$ ls
bin etc lib mnt proc run srv tmp var
dev home media opt root sbin sys usrAnd if we create a file in there, it will be written to the diff folder.
michal@michal-lg:/tmp/container-demo/merged$ echo hello > hello.txt
michal@michal-lg:/tmp/container-demo/merged$ ls
bin etc home media opt root sbin sys usr
dev hello.txt lib mnt proc run srv tmp var
michal@michal-lg:/tmp/container-demo/merged$ ls ../diff/
hello.txtAs a final point, we can create a new shell process and set merged as its root directory. This is how Linux containers can only see their own filesystem. Any process spawned by this shell will inherit the root so it will also be contained to this filesystem.
michal@michal-lg:/tmp/container-demo$ sudo chroot merged /bin/sh
/ # ls
bin hello.txt media proc sbin tmp
dev home mnt root srv usr
etc lib opt run sys var
/ # cd ..
/ # A full implementation would take advantage of namespaces for additional isolation. You can learn more about those in Linux man-pages.
Conclusion
Containers are a black box for vast majority of engineers. You should now have a solid understanding of how things work under the hood!
If you are interested in a more complete implementation of Docker from scratch, consider checkout my Golang Docker clone that’s around 200 lines of code.
I hope you learned something new! If you did, consider subscribing and/or following me on LinkedIn.
You might also enjoy some of my other posts linked below. Perhaps my deep-dive into SQLite storage format, my implementation of MapReduce from scratch, or my series building an ML inference engine from scratch?
