basforge

Search titles, text and the “If it doesn’t work” cases.

    RU

    Server configs in git: /opt as the repository

    Every compose file and config on the server lives in a private git repository whose root is /opt itself. Edit on your computer; on the server, only git pull. Secrets and data never leave the server: .gitignore is an allowlist, and before the first commit a check looks for secrets without printing them.

    How an edit reaches the server

    computer: edit → commit → push ──→ GitHub (private infra-<server>) ──→ server: cd /opt && git pull
                                                                                 docker compose up -d (changed stacks)

    Five rules — write them into the README of every such repository:

    1. Never git clean in /opt, especially -fdx: it deletes ignored files, and those are databases, certificates, acme.json, .env.
    2. Edit on your computer; on the server, only git pull. If you make an urgent edit on the server, commit it from there right away.
    3. Secrets live only on the server: /srv/secrets/<stack>/ or .env (600); git gets .env.example with names (Secrets in Docker stacks).
    4. .gitignore is an allowlist: a new file doesn’t get into git until it’s allowed by name.
    5. System files outside /opt (systemd, cron, logrotate) live in the repository’s host/ folder and are installed with a command from the README.

    There’s no standard place for stacks. Per FHS, /opt is for software installed as one folder and /srv is for data the server serves; both fit, and single-user servers often use ~/docker. Don’t put your files in /var/lib/docker: that’s Docker’s own storage for images and volumes. Pick one place on every server and don’t move it without a reason: the compose project name is the directory name, so a stack without name: gets new empty volumes after a move (The compose project name). Inside, one directory per stack; server-wide scripts go in /opt/scripts/, a script for one stack stays in that stack’s directory.

    Connect a server

    1. Inventory — paths only

      hostname; git --version; docker compose version
      docker compose ls -a                               # compose projects and their file paths
      find / -xdev \( -path /proc -o -path /sys -o -path /var/lib/docker -o -path /var/lib/containerd -o -path /snap -o -path /usr \) -prune \
        -o -type f \( -name 'docker-compose*.y*ml' -o -name 'compose*.y*ml' \) -print 2>/dev/null
      find /opt -maxdepth 3 -name .git                   # nested git clones — leave them out of the repository
      find /opt -maxdepth 3 -not -path '*/.git/*' -not -path '*/.git' | sort | head -400

      The inventory shows where the stacks are, which of them are separate repositories, and where data (db/, data/, *.db), keys and .env sit next to compose files.

    2. .gitattributes and .gitignore — an allowlist

      cd /opt
      printf '* text=auto eol=lf\n' > .gitattributes      # edits from Windows don't break line endings

      In .gitignore everything is denied, things are allowed by name, plus protection at any depth:

      /opt/.gitignore
      /*
      !/.gitignore
      !/.gitattributes
      !/README.md
      !/host/
      !/scripts/
      
      # Protection at any depth
      .env
      *.db
      *.db-*
      *.sqlite3*
      *.pem
      *.key
      *.txt
      acme.json
      
      # A stack: allow the folder, deny everything inside, allow the files you need
      !/memos/
      /memos/*
      !/memos/docker-compose.yaml
      !/memos/.env.example
      
      # A stack with a config subfolder
      !/crowdsec/
      /crowdsec/*
      !/crowdsec/docker-compose.yaml
      !/crowdsec/data/
      /crowdsec/data/*
      !/crowdsec/data/config/
      /crowdsec/data/config/*
      !/crowdsec/data/config/acquis.yaml
      !/crowdsec/data/config/acquis.d/
      !/crowdsec/data/config/profiles.yaml

      Line order matters: the last matching rule wins.

    3. .env.example — names only

      for e in */.env; do sed -nE 's/^(export[[:space:]]+)?([A-Za-z_][A-Za-z0-9_]*)=.*/\2=/p' "$e" > "$e.example"; done

      Only NAME= lines, no values and no comments: comments sometimes hold old values (#PASSWORD=old).

    4. Dry run

      git init -b main
      git add --dry-run .                                # what goes into git — read the list
      git ls-files --others --exclude-standard -z | xargs -0r du -k | sort -n | tail -5   # the largest files

      A large file among configs is almost always a generated list or data.

    5. Secrets check — without printing values

      The command reads the files headed for git and, for every suspicious line, prints the name and the kind of value: ${NAME} (comes from .env — good), a path, a placeholder, (empty) — or <value, N chars>, which means a real value is written in the line. The value itself is never printed, so you can show the output to anyone.

      git ls-files --others --exclude-standard -z | xargs -0r awk '
      FILENAME == ".gitignore" { next }
      {
        l = tolower($0)
        if (l !~ /token|secret|pass([^t]|$)|key|private|bearer|basicauth|htpasswd|credential|salt|jwt|apr1|argon2|\$2[aby]\$|:\/\/[^\/ ]+:[^@ ]+@/) next
        s = $0; sub(/^[ \t-]+/, "", s)
        if (s ~ /^#/) c = "  [comment]"; else { c = ""; sub(/[ \t]+#.*$/, "", s) }
        if (match(s, /[=:]/)) { k = substr(s, 1, RSTART); v = substr(s, RSTART + 1) }
        else if (match(s, /[ \t]/)) { k = substr(s, 1, RSTART - 1); v = substr(s, RSTART + 1) }
        else { k = s; v = "" }
        gsub(/^[ \t"]+|[ \t";]+$/, "", v)
        if (v == "") t = "(empty)"
        else if (v ~ /^\$\{[A-Za-z_][A-Za-z0-9_]*\}$/ || v ~ /^<[A-Z_]+>$/ || v ~ /^[.\/][A-Za-z0-9_.\/-]*$/) t = v
        else t = "<value, " length(v) " chars>"
        print FILENAME ":" FNR ": " k " " t c
      }'

      It catches secret words (token, secret, pass, key, credential…), hashes ($apr1$, $2y$, argon2) and a password in a URL (://user:pass@). pass([^t]|$) skips passthrough. Sample output:

      mariadb/docker-compose.yaml:29: MYSQL_ROOT_PASSWORD: <value, 20 chars>     ← a real password: move it to .env
      panel/docker-compose.yml:21: PANEL_CREDENTIALS: ${PANEL_CREDENTIALS}       ← good
      traefik/docker-compose.yaml:26: CF_DNS_API_TOKEN_FILE: /run/secrets/cf_api_token
      crowdsec/data/config/profiles.yaml:11: # <value, 112 chars>  [comment]  ← a stock comment
    6. A key for one repository

      REPO=infra-<server>
      ssh-keygen -t ed25519 -f ~/.ssh/infra_deploy -N "" -C "$REPO@vps"
      cat ~/.ssh/infra_deploy.pub         # → GitHub: repository → Settings → Deploy keys → Allow write access
      cat >> ~/.ssh/config <<'EOF'
      Host github-infra
        HostName github.com
        User git
        IdentityFile ~/.ssh/infra_deploy
        IdentitiesOnly yes
      EOF
      chmod 600 ~/.ssh/config
      ssh -T github-infra                 # Hi <user>/<repo>! You've successfully authenticated…

      On the first connection, compare GitHub’s key fingerprint with the one published on docs.github.com.

    7. The first commit — on the server

      cd /opt
      git config user.name "<name>"; git config user.email "<email>"
      git add .
      git status --short                  # the same list as in the dry run
      git commit -m "infra: stack configs, no data or secrets"
      git remote add origin "git@github-infra:<user>/$REPO.git"
      git push -u origin main

      On your computer — git clone. Then the repository README: the stacks, what’s where, what isn’t in git and why, the rules.

    How to set it up: repositories, first commit, list, key

    Choice What you get Cost
    one server — one repository paths match 1:1 (git pull puts files where Docker expects them); a key for one repository — a breached server doesn’t expose the others several repositories; shared bits are copied by hand
    one repository for all servers everything in one place the server needs a slice (subdirectories, sparse checkout); the key sees every server
    first commit on the server secrets don’t travel; history holds an exact snapshot from before any edits needs write access from the server
    copy the files to your computer no key on the server .env and keys travel along with the configs; /opt on the server isn’t a repository — git pull hits “files already exist”
    allowlist a new file with a secret can’t slip in by accident every new file has to be allowed
    denylist (*.env, data/) fewer lines every new database or key with an unexpected name ends up in git
    deploy key (SSH, one repository) access to exactly one repository a key per server
    GitHub token one for everything access to the whole account from every server

    If it doesn’t work

    fatal: remote error: … is not a valid repository name

    Cause
    An extra = when setting the name: REPO==infra-x instead of REPO=infra-x. Then git remote add gets the name =infra-x, starting with =. Running remote add again answers remote origin already exists.
    Fix
    git remote set-url origin git@github-infra:<user>/<repo>.git, check git remote -v, then push.

    git pull refuses: the file already exists

    Cause
    The file used to be in .gitignore (a config with a key, for example), and now it comes from git.
    Fix
    Save a copy, delete the file and run git pull — in one command. Keep the copy until you’ve checked.

    After removing a stack, git status shows deleted files

    Cause
    You ran rm -rf first and git pull second, so git sees the deleted files as uncommitted changes.
    Fix
    Do it in this order: git pull first — git removes its own files — then rm -rf of what’s left: .env, data.

    git pull conflicts over a generated file

    Cause
    An IP list or a cache got into git, a script updates it on the server, and the working copy is always dirty.
    Fix
    Exclude that file from git, and write in the README how to create it.

    On Windows a symlink turns into a text file with a path

    Cause
    Git stores a link (to a certificates directory, say) as a link, and a Windows clone turns it into a file.
    Fix
    Exclude such links from git and describe them in the README.

    A nested git clone inside /opt

    Cause
    A stack that is a repository of its own, with its own history.
    Fix
    Leave it out of the server repository.

    A migration script left copies of .env in /tmp

    Cause
    Every user on the server can read /tmp.
    Fix
    Delete them right away.

    After git pull the container doesn’t see the changes

    Cause
    The file is mounted into the container on its own, and git pull replaced it with a new one.
    Fix
    docker compose restart <service>, or better, mount a directory (Single-file mounts).

    A secret is already in git history

    Cause
    History keeps everything that went into a commit.
    Fix
    Change the value (Secrets in Docker stacks).

    How to check

    cd /opt && git status --short          # empty: server = repository
    git log --oneline -3                   # the latest commits match your computer

    And now and then — the secrets check from step 5 across the whole repository: git ls-files -z | xargs -0r awk '…'.