How an edit reaches the server
computer: edit → commit → push ──→ GitHub (private infra-<server>) ──→ server: cd /opt && git pull
docker compose up -d (changed stacks)
Five rules — write them into the README of every such repository:
- Never
git cleanin/opt, especially-fdx: it deletes ignored files, and those are databases, certificates,acme.json,.env. - Edit on your computer; on the server, only
git pull. If you make an urgent edit on the server, commit it from there right away. - Secrets live only on the server:
/srv/secrets/<stack>/or.env(600); git gets.env.examplewith names (Secrets in Docker stacks). .gitignoreis an allowlist: a new file doesn’t get into git until it’s allowed by name.- System files outside
/opt(systemd, cron, logrotate) live in the repository’shost/folder and are installed with a command from the README.
There’s no standard place for stacks. Per FHS, /opt is for software installed as one folder and /srv is for
data the server serves; both fit, and single-user servers often use ~/docker. Don’t put your files in
/var/lib/docker: that’s Docker’s own storage for images and volumes. Pick one place on every server and don’t move
it without a reason: the compose project name is the directory name, so a stack without name: gets new empty
volumes after a move (The compose project name). Inside, one directory per stack; server-wide scripts go in
/opt/scripts/, a script for one stack stays in that stack’s directory.
Connect a server
Inventory — paths only
hostname; git --version; docker compose version docker compose ls -a # compose projects and their file paths find / -xdev \( -path /proc -o -path /sys -o -path /var/lib/docker -o -path /var/lib/containerd -o -path /snap -o -path /usr \) -prune \ -o -type f \( -name 'docker-compose*.y*ml' -o -name 'compose*.y*ml' \) -print 2>/dev/null find /opt -maxdepth 3 -name .git # nested git clones — leave them out of the repository find /opt -maxdepth 3 -not -path '*/.git/*' -not -path '*/.git' | sort | head -400The inventory shows where the stacks are, which of them are separate repositories, and where data (
db/,data/,*.db), keys and.envsit next to compose files..gitattributes and .gitignore — an allowlist
cd /opt printf '* text=auto eol=lf\n' > .gitattributes # edits from Windows don't break line endingsIn
.gitignoreeverything is denied, things are allowed by name, plus protection at any depth:/opt/.gitignore /* !/.gitignore !/.gitattributes !/README.md !/host/ !/scripts/ # Protection at any depth .env *.db *.db-* *.sqlite3* *.pem *.key *.txt acme.json # A stack: allow the folder, deny everything inside, allow the files you need !/memos/ /memos/* !/memos/docker-compose.yaml !/memos/.env.example # A stack with a config subfolder !/crowdsec/ /crowdsec/* !/crowdsec/docker-compose.yaml !/crowdsec/data/ /crowdsec/data/* !/crowdsec/data/config/ /crowdsec/data/config/* !/crowdsec/data/config/acquis.yaml !/crowdsec/data/config/acquis.d/ !/crowdsec/data/config/profiles.yamlLine order matters: the last matching rule wins.
.env.example — names only
for e in */.env; do sed -nE 's/^(export[[:space:]]+)?([A-Za-z_][A-Za-z0-9_]*)=.*/\2=/p' "$e" > "$e.example"; doneOnly
NAME=lines, no values and no comments: comments sometimes hold old values (#PASSWORD=old).Dry run
git init -b main git add --dry-run . # what goes into git — read the list git ls-files --others --exclude-standard -z | xargs -0r du -k | sort -n | tail -5 # the largest filesA large file among configs is almost always a generated list or data.
Secrets check — without printing values
The command reads the files headed for git and, for every suspicious line, prints the name and the kind of value:
${NAME}(comes from.env— good), a path, a placeholder,(empty)— or<value, N chars>, which means a real value is written in the line. The value itself is never printed, so you can show the output to anyone.git ls-files --others --exclude-standard -z | xargs -0r awk ' FILENAME == ".gitignore" { next } { l = tolower($0) if (l !~ /token|secret|pass([^t]|$)|key|private|bearer|basicauth|htpasswd|credential|salt|jwt|apr1|argon2|\$2[aby]\$|:\/\/[^\/ ]+:[^@ ]+@/) next s = $0; sub(/^[ \t-]+/, "", s) if (s ~ /^#/) c = " [comment]"; else { c = ""; sub(/[ \t]+#.*$/, "", s) } if (match(s, /[=:]/)) { k = substr(s, 1, RSTART); v = substr(s, RSTART + 1) } else if (match(s, /[ \t]/)) { k = substr(s, 1, RSTART - 1); v = substr(s, RSTART + 1) } else { k = s; v = "" } gsub(/^[ \t"]+|[ \t";]+$/, "", v) if (v == "") t = "(empty)" else if (v ~ /^\$\{[A-Za-z_][A-Za-z0-9_]*\}$/ || v ~ /^<[A-Z_]+>$/ || v ~ /^[.\/][A-Za-z0-9_.\/-]*$/) t = v else t = "<value, " length(v) " chars>" print FILENAME ":" FNR ": " k " " t c }'It catches secret words (
token,secret,pass,key,credential…), hashes ($apr1$,$2y$,argon2) and a password in a URL (://user:pass@).pass([^t]|$)skipspassthrough. Sample output:mariadb/docker-compose.yaml:29: MYSQL_ROOT_PASSWORD: <value, 20 chars> ← a real password: move it to .env panel/docker-compose.yml:21: PANEL_CREDENTIALS: ${PANEL_CREDENTIALS} ← good traefik/docker-compose.yaml:26: CF_DNS_API_TOKEN_FILE: /run/secrets/cf_api_token crowdsec/data/config/profiles.yaml:11: # <value, 112 chars> [comment] ← a stock commentA key for one repository
REPO=infra-<server> ssh-keygen -t ed25519 -f ~/.ssh/infra_deploy -N "" -C "$REPO@vps" cat ~/.ssh/infra_deploy.pub # → GitHub: repository → Settings → Deploy keys → Allow write access cat >> ~/.ssh/config <<'EOF' Host github-infra HostName github.com User git IdentityFile ~/.ssh/infra_deploy IdentitiesOnly yes EOF chmod 600 ~/.ssh/config ssh -T github-infra # Hi <user>/<repo>! You've successfully authenticated…On the first connection, compare GitHub’s key fingerprint with the one published on docs.github.com.
The first commit — on the server
cd /opt git config user.name "<name>"; git config user.email "<email>" git add . git status --short # the same list as in the dry run git commit -m "infra: stack configs, no data or secrets" git remote add origin "git@github-infra:<user>/$REPO.git" git push -u origin mainOn your computer —
git clone. Then the repository README: the stacks, what’s where, what isn’t in git and why, the rules.
How to set it up: repositories, first commit, list, key
| Choice | What you get | Cost |
|---|---|---|
| one server — one repository | paths match 1:1 (git pull puts files where Docker expects them); a key for one repository — a breached server doesn’t expose the others |
several repositories; shared bits are copied by hand |
| one repository for all servers | everything in one place | the server needs a slice (subdirectories, sparse checkout); the key sees every server |
| first commit on the server | secrets don’t travel; history holds an exact snapshot from before any edits | needs write access from the server |
| copy the files to your computer | no key on the server | .env and keys travel along with the configs; /opt on the server isn’t a repository — git pull hits “files already exist” |
| allowlist | a new file with a secret can’t slip in by accident | every new file has to be allowed |
denylist (*.env, data/) |
fewer lines | every new database or key with an unexpected name ends up in git |
| deploy key (SSH, one repository) | access to exactly one repository | a key per server |
| GitHub token | one for everything | access to the whole account from every server |
If it doesn’t work
fatal: remote error: … is not a valid repository name
- Cause
- An extra
=when setting the name:REPO==infra-xinstead ofREPO=infra-x. Thengit remote addgets the name=infra-x, starting with=. Runningremote addagain answersremote origin already exists. - Fix
git remote set-url origin git@github-infra:<user>/<repo>.git, checkgit remote -v, thenpush.
git pull refuses: the file already exists
- Cause
- The file used to be in
.gitignore(a config with a key, for example), and now it comes from git. - Fix
- Save a copy, delete the file and run
git pull— in one command. Keep the copy until you’ve checked.
After removing a stack, git status shows deleted files
- Cause
- You ran
rm -rffirst andgit pullsecond, so git sees the deleted files as uncommitted changes. - Fix
- Do it in this order:
git pullfirst — git removes its own files — thenrm -rfof what’s left:.env, data.
git pull conflicts over a generated file
- Cause
- An IP list or a cache got into git, a script updates it on the server, and the working copy is always dirty.
- Fix
- Exclude that file from git, and write in the README how to create it.
On Windows a symlink turns into a text file with a path
- Cause
- Git stores a link (to a certificates directory, say) as a link, and a Windows clone turns it into a file.
- Fix
- Exclude such links from git and describe them in the README.
A nested git clone inside /opt
- Cause
- A stack that is a repository of its own, with its own history.
- Fix
- Leave it out of the server repository.
A migration script left copies of .env in /tmp
- Cause
- Every user on the server can read
/tmp. - Fix
- Delete them right away.
Old stacks kept “for reference” in /opt (_old, _backup) with databases and keys
- Cause
- A stopped stack looks harmless, but it leaks along with everything else in a breach.
- Fix
- Delete them or archive them off the server (A disabled stack doesn’t stay on the server “for reference”).
After git pull the container doesn’t see the changes
- Cause
- The file is mounted into the container on its own, and
git pullreplaced it with a new one. - Fix
docker compose restart <service>, or better, mount a directory (Single-file mounts).
A secret is already in git history
- Cause
- History keeps everything that went into a commit.
- Fix
- Change the value (Secrets in Docker stacks).
How to check
cd /opt && git status --short # empty: server = repository
git log --oneline -3 # the latest commits match your computer
And now and then — the secrets check from step 5 across the whole repository: git ls-files -z | xargs -0r awk '…'.