basforge

Search titles, text and the “If it doesn’t work” cases.

    RU

    Docker: what it is, where it came from and what it costs

    Docker runs a program together with everything it needs — libraries, interpreter, config — as a separate process on the host’s kernel, with its own file system and its own network. The program arrives as a ready image and installs with one command, and it’s removed without leaving traces in the system. There’s no virtual machine: all containers share one kernel, so startup takes a fraction of a second, but the boundary between containers is thinner than between machines.

    What problem it solves

    A program on a server isn’t a single file. A PHP site needs an interpreter of the right version, a dozen extensions, system libraries and a config; a Python service needs its own packages from pip. While there’s one program, everything goes into the system. The trouble starts with the second one:

    • Dependencies clash. The old site needs PHP 7.4, the new one 8.4; one tool needs one version of libssl, another needs a different one. Both can’t go into one system.
    • “Works on my machine.” The developer’s laptop has some versions, the server has others, and the program behaves differently. The difference shows up only after the deploy.
    • The server can’t be reproduced. Over a year, two dozen packages were added by hand and fifty files in /etc were edited. None of it is written down anywhere, and a second server like it can’t be built.
    • Removing things is scary. A program’s packages pull in dependencies, configs stay behind, and a year later it’s unclear which installed thing is needed by whom.

    The classic answer is a virtual machine per program. The isolation is complete, but every machine has its own kernel and its own OS: gigabytes on disk, hundreds of megabytes of memory just for the system, boot in tens of seconds, updates in each one.

    Docker makes isolation cheaper. A program is built into an image: its files together with the whole user-space part of the system, from /bin/sh to the libraries. A running image is a container: an ordinary process on the host’s kernel, which the kernel shows only its own files, its own network and its own processes. There’s one kernel for all, so a container starts like a program, not like an OS, and uses as much memory as the program itself.

    Programs in the system

    • PHP 8.3 from Ubuntu packages — one version for everyone
    • an Ubuntu upgrade changes versions under running sites
    • removal leaves configs and dependencies behind
    • the server’s description is in the admin’s head

    Programs in containers

    • each service has its own version: php:7.4-apache and php:8.4-fpm side by side
    • a system upgrade doesn’t touch the services, they carry their own libraries
    • docker compose down and delete the directory — the service is gone
    • the server’s description is compose files in git

    Where it came from: from chroot to Docker

    Docker didn’t invent the container. Almost everything it stands on appeared in kernels earlier:

    1979  chroot in Unix V7          a process sees one directory, not the whole disk
    2000  jail in FreeBSD 4.0        its own root, its own network, its own users
    2002  mount namespace in Linux   the first "namespace": its own mount tree
    2005  Zones in Solaris 10        the OS is split into isolated zones
    2008  cgroups in Linux 2.6.24    the kernel accounts and limits CPU and memory of a process group
    2008  LXC                        tools that assemble a container from namespaces and cgroups
    2013  user namespace (3.8)       root in a container isn't root on the host

    By 2013 Linux had everything for isolation, but few people knew how to use it: an LXC container was a small system set up by hand that still had to be filled, and there was nothing to move it to another machine with.

    Docker grew inside dotCloud, a cloud platform that ran customers’ applications in LXC containers. In March 2013 the company’s founder, Solomon Hykes, showed the internal tool in a five-minute talk at PyCon, and the code was opened the same month. What was new wasn’t the kernel but everything around it:

    • the image — a program’s file system in layers that can be built from a recipe (Dockerfile), downloaded and run anywhere;
    • the registry — a shared image store: docker pull nginx, and a ready nginx on any machine with Docker;
    • one command, docker run, instead of assembling a container by hand.

    The metaphor comes from logistics: a standard-size shipping container that doesn’t care what’s inside, while the crane and the ship don’t care whose it is. Hence the whale carrying containers on the logo.

    Then things moved fast:

    2013  October: dotCloud renames itself Docker Inc.
    2014  Docker 0.9 — its own libcontainer instead of LXC; Docker 1.0 in June
    2014  Google opens Kubernetes; Docker buys Fig — the future Docker Compose
    2015  Open Container Initiative: the image and runtime format becomes a common standard, runc implements it
    2017  containerd goes to the CNCF, Docker's open source becomes the Moby project
    2020  Docker Hub introduces an image pull limit
    2021  Docker Desktop becomes paid for companies with 250+ employees or $10M+ revenue
    2022  Kubernetes 1.24 removes dockershim: clusters work through containerd directly
    2023  Compose v1 (hyphenated docker-compose) stops getting updates; the docker compose plugin remains

    What this history means for a server: the image format became the OCI standard, and an image built by Docker runs in Podman, containerd and Kubernetes. Docker Engine on Linux is open source (Apache 2.0) and free for everyone; the license only concerns Docker Desktop.

    How it works: image, container, registry

    Image

    nginx:1.29

    Read-only file system layers plus metadata: which command to run, which ports and variables. Every Dockerfile instruction is a new layer. Images with a common base store it on disk once.

    Container

    docker run

    An image plus a thin writable layer on top (overlayfs). Everything the program writes outside volumes lands in this layer and disappears with the container.

    Volume

    volumes:

    A directory outside the container: a Docker volume in /var/lib/docker/volumes/ or a host directory. Data lives here and survives a recreate.

    Registry

    docker.io, ghcr.io

    A server images are pulled from and pushed to. Docker Hub is the default registry: nginx means docker.io/library/nginx:latest.

    What happens on docker run:

    docker run -d -p 8080:80 nginx:1.29

    1. ClientdockerThe command in the terminal only sends an API request to the /var/run/docker.sock socket. Hence the rule: whoever can write to the socket is root on the host (docker.sock).
    2. DaemondockerdPulls the image if it’s missing, creates the network and volumes, writes firewall rules for -p.
    3. RuntimecontainerdStores images and watches over containers’ lifecycles. The same containerd runs in Kubernetes.
    4. StartruncCreates the process in new namespaces, puts it into a cgroup with limits and exits. From there the process runs on its own.

    an nginx process on the host’s kernel · its own files, network and PIDs

    Two kernel mechanisms provide the isolation:

    • namespaces — what a process sees. pid: its own process numbers; the program in the container is PID 1 and doesn’t see anyone else’s. net: its own interfaces and ports, so ten containers listen on port 80 without getting in each other’s way. mnt: its own file system from the image. Also uts (its own hostname), ipc and user.
    • cgroups — how much a process can take: memory, CPU share, number of processes. There are no limits by default.

    Plus filters on top: Docker takes most capabilities away from root in the container (changing the host’s network, loading kernel modules) and uses seccomp to forbid about fifty system calls.

    A Linux container runs only on a Linux kernel. Docker Desktop on Windows and macOS brings up a Linux virtual machine for it (on Windows — in WSL 2), and the containers run there.

    Container vs virtual machine

    Virtual machine

    KVM, Proxmox, VirtualBox
    • its own kernel and its own OS; the hypervisor isolates at the hardware level
    • starts in tens of seconds; the OS takes hundreds of MB of memory
    • an image is gigabytes
    • runs any OS: Windows on a Linux host
    • a guest kernel vulnerability doesn’t affect the host

    Container

    Docker, Podman, LXC
    • the host’s kernel; isolation by that kernel’s means
    • starts in a fraction of a second; uses as much memory as the program
    • an image is from a few MB (alpine — about 8 MB)
    • Linux on Linux only
    • a kernel vulnerability is shared by all containers and the host

    They aren’t rivals: your own server usually has both. A VPS from a provider is already a virtual machine, and Docker runs inside it. In a home lab, Proxmox splits the hardware into machines, and the services in them run in containers.

    Where it’s used

    The main case

    Your own server

    Every service is an /opt/<stack>/ directory with a compose.yaml. Proxy, bans, monitoring, sites — everything in containers; the server is rebuilt from git (Server configs in git).

    Development

    A database, Redis, a queue — docker compose up in the project directory, and the whole team has the same versions. A devcontainer in VS Code puts the whole development environment in a container.

    CI

    Tests and builds in a clean container on every run. The result is an image: GitHub Actions pushes it to a registry, and the server pulls it ready-made, with no build of its own.

    Trying out a program

    docker run --rm — run it, look around, the container is deleted. Nothing is left in the system except the image.

    Old software on a new system

    An app for PHP 7.4 or Python 2 runs in an image with its own system on a fresh Ubuntu, whose packages dropped those versions long ago.

    Kubernetes

    Images are the same OCI format. Docker isn’t needed there, but images for a cluster are usually built with it.

    Other ways to solve it

    How it isolates Daemon Service description
    Docker an application container: one process, an image from a registry dockerd as root compose.yaml
    Packages and systemd it doesn’t: all programs in one system — a systemd unit, configs in /etc
    Podman the same OCI containers, Docker-like commands none, a container is a child process; also runs as a regular user podman compose or Quadlet units in systemd
    LXC / Incus a system container: a whole OS with init and SSH inside incusd set up inside, like a regular server
    Virtual machines a hypervisor, its own kernel the hypervisor a disk image and setup inside
    Kubernetes, k3s OCI containers on many machines agents on every node YAML manifests
    If… Use
    one or two servers, three or more services, and you want to rebuild the server from files Docker and Compose
    one program that’s in the distribution’s packages and updates with it packages and systemd
    a root daemon isn’t allowed, or you want containers right in systemd Podman
    you need a “small machine” with several services and SSH login LXC / Incus
    a different kernel or OS, or untrusted third-party code a virtual machine
    many machines, and services move between them on their own Kubernetes; on small hardware — k3s

    What you get and what it costs

    What you get

    • The same everywhere. An image carries its own system, and postgres:17 behaves the same on a laptop and on a server.
    • Dependencies don’t clash. Two sites on different PHP versions, three databases of different versions — on one host.
    • Install and remove with one command. docker compose up -d and docker compose down; after removal no packages or configs are left in the system.
    • Update and roll back by tag. Change nextcloud:31 to 32 — pull and up -d; it didn’t take off — put the tag back, the old image is still on disk.
    • Ready images from the programs’ authors. Traefik, CrowdSec, Gitea, Vaultwarden — an official image with instructions instead of a build from source.
    • Startup in a fraction of a second, and density: a VPS with 2 GB of memory runs a dozen services that, in separate machines, wouldn’t have enough memory even for the OSes.

    What it costs

    • The boundary is thinner than a machine’s. The kernel is shared: a vulnerability in it is an escape from the container to the host. The docker group and access to docker.sock mean root (docker.sock).
    • Ports bypass the firewall. ports: - "8080:80" opens the port to the whole internet, and ufw doesn’t see it (Docker ports bypass the input chain).
    • System updates don’t reach containers. unattended-upgrades updates the host, while the OpenSSL inside an image stays the one the image was built with. Fresh fixes arrive only with a new image.
    • Data is lost silently. Everything written outside volumes disappears when the container is recreated — with no errors or warnings.
    • Disk space goes unnoticed. json-file logs without a limit, old images, build cache (Docker on Ubuntu).
    • No limits by default. One container can eat all of the host’s memory and CPU.
    • One more layer to debug. A minimal image has no ps, curl, ping, often not even bash; permissions on files in volumes depend on the uid inside the image (WordPress file owner).
    • Docker Hub and the license. Anonymous pulls are limited; Docker Desktop is paid in a large company.

    What you find out later: the hidden benefits

    These benefits aren’t on the first page of the documentation, yet on your own server they weigh no less than the obvious ones:

    • The data boundary is visible. Everything that needs backing up is the volumes and directories listed in volumes:. The rest is restored from the image and the compose file. A server backup comes down to /opt and database dumps (A backup that outlives the server).
    • The server’s description is the server. A compose file is documentation at the same time: which image, which ports, which variables, who shares a network with whom. Put it into git and you get a change history and rollback.
    • An OS upgrade doesn’t break services. Moving from Ubuntu 24.04 to 26.04 changes the system libraries, but not the ones inside images. The services move along with /opt.
    • A new version is tried next to the old one. A second container with the new tag on a different port or name; check it, switch over — the old one is still running.
    • The same interface for every service. Logs — docker logs for any of them, restart after a crash — restart: unless-stopped, liveness check — healthcheck. No systemd unit per service.
    • Networking by name. On a shared network a container finds its neighbour by service name: db:3306, no addresses. Traefik relies on this too: it takes routes from container labels (Traefik).
    • Moving to another server — copy /opt and the volume data, docker compose up -d. Every program’s version is the same, because it’s written in the tags.

    Fine points

    A throwaway container instead of an install

    Need a tool or a different language version once — run it in a container with the current directory inside:

    docker run --rm -it -v "$PWD":/work -w /work python:3.15 python script.py

    --rm deletes the container on exit, -v "$PWD":/work mounts the current directory, -w makes it the working one. The same goes for node, golang, or mariadb to check a dump — nothing installed in the system.

    What the container wrote to itself

    docker diff shows changes in the writable layer: A — added, C — changed, D — deleted.

    docker diff memos | grep memos_prod
    # A /var/opt/memos/memos_prod.db

    The database file is outside a volume — it will disappear on a recreate. Check every new service like this before real data shows up in it.

    Networking from inside a minimal container

    The image has neither curl nor dig — attach a separate container with tools to its network namespace:

    docker run --rm -it --network container:memos nicolaka/netshoot

    Inside are the same interfaces, address and DNS as memos has, plus the full kit: curl, dig, ss, tcpdump. memos itself stays untouched.

    Why stopping waits 10 seconds

    docker stop sends SIGTERM to the process with PID 1 and SIGKILL after 10 s. In the shell form (CMD python app.py) PID 1 is /bin/sh -c, which doesn’t pass the signal to the program: the program never gets it and dies from SIGKILL without closing connections and files. Write the exec form, CMD ["python", "app.py"], or turn on init: true in compose. 10 s is too little for databases under load — give them stop_grace_period: 2m (MariaDB in a container).

    Layer order in a Dockerfile

    A layer is rebuilt if it or any layer above it changed. So copy what rarely changes first, then the code:

    FROM python:3.15-slim
    WORKDIR /app
    COPY requirements.txt .
    RUN pip install --no-cache-dir -r requirements.txt
    COPY . .
    CMD ["python", "app.py"]

    A code change rebuilds only the last two layers; pip install comes from the cache. A .dockerignore next to the Dockerfile keeps .git, node_modules and .env out of the image — otherwise secrets travel to the registry with it. A multi-stage build (FROM … AS build, then COPY --from=build) leaves the compilers in an intermediate image: only the result goes into the final one.

    A tag isn’t a version

    A tag is a moving label: tomorrow nginx:1.29 will point to 1.29.3 instead of 1.29.2, and latest to anything at all. Only a digest pins an image exactly:

    image: traefik:v3.7@sha256:<digest>

    Get the digest from docker image inspect traefik:v3.7 --format '{{index .RepoDigests 0}}'. On your own server a tag with the minor version is usually enough; a digest is needed where the image must be exactly the one you tested.

    The final compose with every substitution

    docker compose config prints the file the way Docker sees it: with variables from .env substituted, paths expanded and defaults filled in. The first thing to look at when “the variable didn’t get substituted” (The $ sign).

    An image to another server without a registry

    docker save memos-custom:1 | gzip | ssh web2 'gunzip | docker load'

    The image goes over SSH in one stream; neither Docker Hub nor your own registry is needed.

    Another server’s Docker from your own console

    docker context create web1 --docker "host=ssh://example-user@web1"
    docker --context web1 ps

    docker commands run on web1 over SSH, with no API port open to the outside. The user on web1 is in the docker group, with everything that implies (docker.sock).

    Restarting dockerd without stopping containers

    By default, systemctl restart docker and a docker-ce package upgrade stop every container while the daemon restarts. With "live-restore": true in /etc/docker/daemon.json the containers keep running while the daemon is down, and the daemon picks them up after it starts. Doesn’t work with Swarm.

    Where the space went

    docker system df -v

    The output lists images, containers, volumes and build cache separately, each with how much space can be freed (RECLAIMABLE). docker image prune -a deletes images no container uses, and docker builder prune clears the build cache. Don’t run docker system prune blindly: it deletes all stopped containers, and with --volumes also volumes without containers (since Docker 23 — only anonymous ones). A stack stopped for a while loses what was in such volumes.

    Start order by readiness

    depends_on on its own waits only for the container to start, not for the database inside to be ready. Wait for the check:

    services:
      app:
        depends_on:
          db:
            condition: service_healthy
      db:
        image: mariadb:11.8
        healthcheck:
          test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
          interval: 10s

    And docker compose up -d --wait returns only when every check is green, and exits with an error if a service never came up. In an update script this stops the deploy right away.

    Where to start

    If it doesn’t work

    After docker compose up -d the data is gone: the database is empty, uploaded files have vanished

    Cause
    The data was in the container’s writable layer. When the container is recreated — a new image, a compose edit — that layer is deleted along with the old container.
    Fix
    Everything that must outlive the container goes into a volume or a host directory (volumes:). docker diff <container> shows what the container writes to itself: A and C lines outside volumes disappear on the next recreate.

    docker stop and docker compose down wait exactly 10 seconds every time

    Cause
    The process with PID 1 in the container doesn’t react to SIGTERM: the command runs through sh -c (the shell form of CMD), and the shell doesn’t pass the signal on. After 10 s Docker sends SIGKILL.
    Fix
    Use the exec form of CMD — ["python", "app.py"] — or init: true in compose: then PID 1 is tini, which passes signals on and reaps zombies.

    toomanyrequests: You have reached your pull rate limit

    Cause
    Docker Hub limits anonymous image pulls by the server’s address. On a shared address (NAT, CI) the limit runs out faster.
    Fix
    docker login with a free account raises the limit. Pull images that exist on GHCR or quay.io from there.

    exec format error right when the container starts

    Cause
    The image is built for a different architecture: amd64 on an ARM server or the other way around.
    Fix
    Check what the image is built for: docker image inspect <image> --format '{{.Architecture}}'. Use a tag with the right architecture or build your own: docker buildx build --platform linux/arm64.

    You updated an image, and the service stopped starting, though compose wasn’t touched

    Cause
    The latest tag (or a tag without a version) points to the new release after every publish. docker compose pull fetched a different major version.
    Fix
    Go back to the previous version by its number in the tag, and from then on write the version explicitly: traefik:v3.7, mariadb:11.8.

    The server stopped responding, and dmesg says Out of memory: Killed process

    Cause
    A container has no memory limit by default: one leaking service takes all of the host’s memory, and the OOM killer kills whatever it picks, sshd included.
    Fix
    Give every heavy service a limit: mem_limit: 512m in compose. Who eats how much — docker stats --no-stream.

    A container can’t find its neighbour by name: could not translate host name "db"

    Cause
    The containers are on different networks, or were started with docker run on the default bridge network: it has no name lookup.
    Fix
    Put both containers on one network of your own. Compose creates one for the stack itself; between stacks — a shared external network, such as proxy.

    The disk is full, though there isn’t much data in the volumes

    Cause
    Old images, build cache, stopped containers and json-file logs without a limit.
    Fix
    docker system df shows what takes how much; clean up with docker image prune -a and docker builder prune. Log limits — Docker on Ubuntu.