basforge

Search titles, text and the “If it doesn’t work” cases.

    RU
    ConfigLinuxsystemd

    systemd: delayed, one-by-one start of compose stacks

    On a weak server, containers that all start at once after boot get in each other’s way: the database doesn’t come up in time, and apps crash on connect. A systemd unit waits for Docker, pauses, and starts the stacks one at a time.

    What it’s made of

    Three parts:

    1. The stacks the script starts have restart: on-failure, not unless-stopped. Otherwise Docker brings them up itself, right away and all together, and the queue is pointless.
    2. The script — a list of stacks in the right order, with pauses.
    3. A systemd unit — runs the script after Docker starts.

    The unit

    /etc/systemd/system/docker-delayed-start.service
    [Unit]
    Description=Delayed Start for Docker Compose Projects
    After=docker.service
    Requires=docker.service

    After — starts after Docker. Requires — won’t start without Docker; if Docker is stopped, the unit stops too.

    [Service]
    Type=oneshot

    oneshot — “run and finish”. Its start timeout is off by default, so systemd won’t cut long pauses in the script short.

    ExecStartPre=/bin/sleep 60

    One minute while Docker brings up its own unless-stopped containers and the system settles after boot.

    ExecStart=/opt/scripts/start-all.sh

    The script itself. It needs the execute bit: chmod 755. It lives in /opt/scripts/, the place for server-wide scripts; keep a script that serves one stack in that stack’s directory.

    RemainAfterExit=yes

    Once done, the unit stays active (exited) — systemctl status shows it has run. A second systemctl start won’t run the script again.

    [Install]
    WantedBy=multi-user.target

    Starts on a normal boot, after systemctl enable.

    The script

    Order: protection first (CrowdSec), then the entry point (Traefik), then databases, then apps, everything else last.

    /opt/scripts/start-all.sh
    #!/bin/bash
    
    echo "Starting crowdsec..."
    cd /opt/crowdsec && docker compose up -d
    sleep 10
    
    echo "Starting reverse proxy..."
    cd /opt/traefik && docker compose up -d
    sleep 10

    cd … && — if the directory is missing, the step is skipped. Without &&, compose would run in the current directory, i.e. in the previous stack. Pauses follow the weight of the stack: protection and the proxy come up in seconds.

    echo "Starting database and cache..."
    cd /opt/mariadb && docker compose up -d
    sleep 50
    Key part

    On a slow disk, a database spends a few dozen seconds after start loading its buffer and checking its log — it gets a longer pause.

    echo "Starting nginx and WordPress..."
    cd /opt/project_root && docker compose up -d
    sleep 50
    # echo "Starting panel..."
    # cd /opt/panel && docker compose up -d
    
    echo "All containers started one by one!"

    The script is the list of what should be running. Comment out a stack you don’t need for now here, or it comes back after the next reboot.

    Installing

    Keep the unit in the server repository (host/systemd/, Server configs in git) and install it from there:

    install -m 644 /opt/host/systemd/docker-delayed-start.service /etc/systemd/system/docker-delayed-start.service
    systemctl daemon-reload
    systemctl enable docker-delayed-start.service

    Other ways to order the start

    Approach What you get Cost
    restart: unless-stopped everywhere, no script nothing to set up: Docker brings everything up after boot everything starts at once — fine on a fast disk with spare memory
    depends_on with condition: service_healthy “database, then app” order without pauses, by readiness works only inside one compose project
    on-failure + a unit with a script order and pauses across projects, in one file if the unit is disabled, stacks may not come up after a reboot
    a systemd unit per stack with After= on each other each stack has its own status in systemctl lots of files

    What restart policies do after a server reboot:

    Policy After a reboot
    always always comes up
    unless-stopped comes up unless it was stopped by hand
    on-failure[:N] comes up only if the container exited with an error; after a normal stop at shutdown it depends on the exit code — don’t rely on it
    no doesn’t come up

    If it doesn’t work

    Some sites didn’t come back after a reboot

    Cause
    Those stacks have restart: on-failure, and the unit is disabled or the script failed.
    Fix
    Check systemctl is-enabled docker-delayed-start.service, systemctl status … and journalctl -u docker-delayed-start -b.

    After a reboot, a service came up that should stay off

    Cause
    Its step in the script isn’t commented out.
    Fix
    Comment out the step in start-all.sh — the script is the list of what should be running.

    Edited the unit — nothing changed

    Cause
    systemd rereads units only on systemctl daemon-reload or at boot.
    Fix
    Run systemctl daemon-reload.

    The script started nothing, and systemctl status shows active (exited)

    Cause
    A second start doesn’t rerun a oneshot unit with RemainAfterExit=yes.
    Fix
    Use systemctl restart … or run the script by hand.

    The server moved to a fast disk

    Cause
    The pauses were tuned for a slow disk.
    Fix
    You can shorten or drop the pauses. They’re not required, but they don’t hurt either.

    The server changed roles and has hardly any stacks left

    Cause
    The start queue is no longer needed.
    Fix
    Disable the unit (systemctl disable) and give the stacks unless-stopped back. The script and unit can stay in the repository for later.

    How to check

    systemctl is-enabled docker-delayed-start.service       # enabled
    systemctl status docker-delayed-start.service           # active (exited)
    journalctl -u docker-delayed-start -b                   # the script's echo output for this boot
    docker ps --format 'table {{.Names}}\t{{.Status}}'      # did everything come up