basforge

Search titles, text and the “If it doesn’t work” cases.

    RU

    Secrets in Docker stacks: where to keep them and how not to leak them

    A secret is stored on the server as a single copy that only root can read: no copies in git, on the command line or in other containers’ environment. The default place is a file in /srv/secrets/<stack>/, attached as a Docker secret; if the app can’t do that, a .env with mode 600.

    Where a secret lives

    Place When Permissions
    file /srv/secrets/<stack>/<name>, inside the container — Docker secret /run/secrets/<name> the app can read the secret from a file directories root 700, files root 400
    /opt/<stack>/.env compose itself needs the value (labels, substitution), or the app reads only environment variables root 600
    .env.example in git variable names only, no values —
    compose file, git (even private), command line, a shared chat, /tmp never —

    /srv/secrets lives outside /opt. If /opt is mounted into a Docker management panel (Dockhand, Portainer) or ends up in a config archive, the secrets won’t be there.

    Setup

    1. A directory for secrets

      install -d -m 700 /srv/secrets /srv/secrets/<stack>

      Directories are root 700, files in them root 400. Docker runs as root, so mounting a file into a container still works.

    2. Move the secret into a file — off screen

      From a file:

      install -m 400 /opt/<stack>/token.txt /srv/secrets/<stack>/token
      wc -c < /srv/secrets/<stack>/token          # not 0 — and nothing on screen

      From a config line — through a variable, with no output and no trailing newline:

      key=$(sed -nE 's/^[[:space:]]*apiKey:[[:space:]]*"?([^"[:space:]]+)"?.*/\1/p' config.yml)
      printf '%s' "$key" > /srv/secrets/<stack>/key; unset key
      chmod 400 /srv/secrets/<stack>/key
    3. Attach it as a Docker secret

      services:
        traefik:
          environment:
            CF_DNS_API_TOKEN_FILE: /run/secrets/cf_api_token     # the app reads the file
          secrets:
            - cf_api_token
            - crowdsec_lapi_key
      secrets:
        cf_api_token:
          file: /srv/secrets/traefik/cf_api_token
        crowdsec_lapi_key:
          file: /srv/secrets/traefik/crowdsec_lapi_key

      Without Swarm this is a plain file mount at /run/secrets/<name>. The difference from a volume: the compose file shows right away that it’s a secret, and all secrets are described in one place.

    4. The app reads the secret from a file

      Method Example
      a *_FILE variable CF_DNS_API_TOKEN_FILE, MARIADB_ROOT_PASSWORD_FILE, POSTGRES_PASSWORD_FILE — the official MariaDB and Postgres images support this; Postgres only for POSTGRES_PASSWORD, POSTGRES_USER, POSTGRES_DB and POSTGRES_INITDB_ARGS
      a …File option in the config crowdsecLapiKeyFile in the CrowdSec plugin for Traefik (CrowdSec with Traefik behind Cloudflare)
      a users file Basic Auth in Traefik: basicauth.usersfile=/path instead of basicauth.users=…
    5. Basic Auth hash — as a users file

      A label gets its value when compose starts, so it can’t be a secret: the hash in basicauth.users=${APP_AUTH} can only come from .env. To get it out of there, use a users file:

      # Traefik: a directory with users files — one mount for every service
      services:
        traefik:
          volumes:
            - /srv/secrets/traefik/basicauth:/basicauth:ro
      # the service: the label points to a path inside the Traefik container
      - "traefik.http.middlewares.app-auth.basicauth.usersfile=/basicauth/app"

      The file /srv/secrets/traefik/basicauth/app holds user:hash lines in htpasswd format, $ unescaped.

    Passwords: new, for a command, leaked

    A new password goes straight where it belongs

    new=$(openssl rand -hex 20)      # not printed on screen
    # … apply new: to the database, to .env …
    unset new

    Keep passwords for people in a password manager (such as Vaultwarden) and passwords for services on the server. Those are two different places, and that’s how it should be.

    A password for a command — not on the command line

    An argument like -p'password' stays in shell history (~/.bash_history) and shows in the process list. If the password is already in the container’s environment, take it from there, inside the container:

    docker exec db sh -c 'exec mariadb-dump -u root -p"$MARIADB_ROOT_PASSWORD" --single-transaction app' | gzip > app.sql.gz

    The outer single quotes make the shell inside the container expand $MARIADB_ROOT_PASSWORD.

    A password leaked — change it off screen

    For MariaDB:

    new=$(openssl rand -hex 20)
    docker exec -e NEW="$new" db sh -c 'mariadb -u root -p"$MARIADB_ROOT_PASSWORD" -e "ALTER USER IF EXISTS root@localhost IDENTIFIED BY \"$NEW\"; ALTER USER IF EXISTS root@\"%\" IDENTIFIED BY \"$NEW\";"' \
      && sed -i "s/^MARIADB_ROOT_PASSWORD=.*/MARIADB_ROOT_PASSWORD='$new'/" .env && echo "changed"
    unset new; docker compose up -d     # so the container gets the new password

    The && makes .env change only if the database accepted the new password. Details in MariaDB in a container.

    Other ways to keep secrets

    Approach What you get Cost
    Docker secrets from /srv/secrets the secret is outside /opt, shows up in compose as a secret, mode 400 needs *_FILE support or reading from a file
    .env with mode 600 works with everything; labels can’t do without it sits in /opt next to the configs and travels with them into any archive
    encrypted secrets in git (sops, age) secrets are versioned, a server can be rebuilt from scratch the decryption key still has to live somewhere; one more tool in the chain
    a secrets manager (Vault, Infisical) rotation, audit, role-based access one more service to keep alive

    For a handful of your own servers the first two are enough: /srv/secrets for everything that can read files, .env 600 for the rest.

    If it doesn’t work

    A secret got into git — even a private repository

    Cause
    Git history keeps it forever. You can scrub the history, but it takes time and doesn’t guarantee no copy is left anywhere.
    Fix
    Treat the secret as leaked and change the value.

    A password ended up in a chat, a ticket or a screenshot

    Cause
    Everyone with access to the conversation has seen the value.
    Fix
    Change it so the new password never appears on screen. The MariaDB command is in the “Passwords” section above.

    .env with mode 644

    Cause
    Any user on the server can read it.
    Fix
    chmod 600 /opt/*/.env.

    The same secret variable sits in the .env of several stacks, but only one reads it

    Cause
    The key was copied “just in case” — say, the bouncer key into CrowdSec’s .env. When the key changes, the copies get forgotten.
    Fix
    Delete the extra copies.

    A script made backup copies of .env in /tmp

    Cause
    Every user on the server can read /tmp.
    Fix
    Make copies in /root and delete them right after.

    A key from a file is rejected, though it’s correct

    Cause
    The file ends with a newline: some programs read the key together with \n.
    Fix
    Write the file with printf '%s', not echo.

    Changed a secret’s path in compose — the container won’t be created

    Cause
    There’s no file at the new path yet, and without it compose doesn’t create the container.
    Fix
    Put the file in place first, then git pull and up -d in one command (The secret file comes before up -d).

    How to check

    find /srv/secrets -exec stat -c '%a %U %s %n' {} \;      # mode, owner, size — no contents
    stat -c '%a %n' /opt/*/.env                              # 600
    docker exec traefik sh -c 'test -s /run/secrets/cf_api_token && echo "token in place"'

    Plus a secrets search over what goes into git, with values masked (Server configs in git, step 5).