basforge

Search titles, text and the “If it doesn’t work” cases.

    RU

    Moving to a new server

    Put everything set up on levels 1–5 into git: /opt becomes a repository, and system files go into /opt/host. On an empty VPS at any host, one script brings up the server’s base, and a second one restores stacks from the Kopia backup: all of them, several, or one. Test it with a drill on a temporary VPS, and move when the time comes.

    Start froma server after level 5, “Backups”: Kopia snapshots /opt and /srv/secrets every night, and the storage password is in the password manager.

    What you get

    git (GitHub, private infra-web1)   configs and skeleton: compose, traefik.yml, Gatus, scripts, /opt/host
    Kopia backup                       the same plus the body: stacks’ data/, .env, acme.json, /srv/secrets
    
    empty VPS ── git clone ──▶ /opt ── bootstrap.sh ──▶ base of levels 1–5: user, SSH, ufw, updates,
                                                        swap, journal, Docker, bouncer, Kopia
                               Kopia ── restore.sh ──▶ all stacks, several or one — with their data
    drill: curl --resolve to the new address, delete the VPS   move: DNS to the new address, backup timers
    • You’ll needA GitHub account: the server’s private repository will live there
    • You’ll needAn empty VPS with Ubuntu 24.04 or 26.04 and root login — for the drill, later for the move; below its address is 203.0.113.20
    • You’ll needThe Kopia storage password from the password manager and access to the storage: SSH to backup1 (option A of level 5) or a computer with a browser for rclone (option B)
    • You’ll needAccess to the domain’s DNS — for the move

    Your values

    Filled into the commands and files of this and other recipes on the path; stored in this browser only.
    Domain
    example.orgplaceholder
    User
    example-userplaceholder
    Server name
    web1placeholder
    Backup server address
    198.51.100.20placeholder

    Without the script the commands show placeholder values: replace them with yours.

    Steps

    The same recipe, short: commands from the stages, files as one line. For a second server and for those who already know what each command does.

    /opt in git: an allowlist

    Why — Server configs in git

    /opt becomes a git repository: compose files, configs and scripts go into git — the server’s skeleton. Data, .env, acme.json and /srv/secrets stay in the Kopia backup, which holds both the skeleton and the body. This stage and the next two run on the old server.

    Repository
    Command What it does
    sudo git -C /opt init -b main /opt becomes a repository, branch main
    printf '* text=auto eol=lf\n' | sudo tee /opt/.gitattributes Linux line endings, even after an edit on Windows
    /opt/.gitignorean allowlist: everything is forbidden, files are allowed by name — the path’s stacks per the server profile
    # Белый список: в git только то, что разрешено поимённо. Порядок важен — побеждает последнее подходящее правило
    /*
    !/.gitignore
    !/.gitattributes
    !/host/
    !/scripts/
    
    # Секреты и базы — нигде, на любой глубине
    .env
    *.db
    *.db-*
    *.pem
    *.key
    acme.json
    id_ed25519*
    
    # Стек: каталог разрешён, внутри — только перечисленное
    !/traefik/
    /traefik/*
    !/traefik/docker-compose.yaml
    !/traefik/data/
    /traefik/data/*
    !/traefik/data/traefik.yml
    !/traefik/data/dynamic/
    
    !/crowdsec/
    /crowdsec/*
    !/crowdsec/docker-compose.yaml
    !/crowdsec/data/
    /crowdsec/data/*
    !/crowdsec/data/config/
    /crowdsec/data/config/*
    !/crowdsec/data/config/acquis.d/
    !/crowdsec/data/config/profiles.yaml
    
    !/whoami/
    /whoami/*
    !/whoami/docker-compose.yaml
    
    !/memos/
    /memos/*
    !/memos/docker-compose.yaml
    
    !/beszel/
    /beszel/*
    !/beszel/docker-compose.yaml
    !/beszel/.env.example
    
    !/gatus/
    /gatus/*
    !/gatus/docker-compose.yaml
    !/gatus/.env.example
    !/gatus/config/

    For a stack of your own outside the path, add a three-line block and its files, like memos.

    Variable names without values
    Command What it does
    for e in /opt/*/.env; do sudo sed -nE 's/^([A-Za-z_][A-Za-z0-9_]*)=.*/\1=/p' "$e" | sudo tee "$e.example" > /dev/null; done next to each .env, a .env.example with names only: it shows what to fill in
    sudo git -C /opt add --dry-run . what will go into git — read the list
    Done when
    Command What you should see
    sudo git -C /opt add --dry-run . | grep -E '\.env$|acme|\.db|id_ed25519|/data/' nothing except crowdsec/data/config/… and traefik/data/traefik.yml, traefik/data/dynamic/…
    sudo git -C /opt add --dry-run . | grep -c docker-compose.yaml the number of stacks: sudo docker compose ls -a shows as many
    Not like this — see “If it doesn’t work”, stage cases.

    System files and scripts — into /opt/host

    The base from levels 1–5 lives outside /opt: SSH, automatic updates, the journal, swap, daemon.json, the Kopia services. Copy these files into /opt/host/etc/ under the same paths, so both git and the backup pick them up. Next to them go the public keys you log in with and two scripts: bootstrap.sh builds the base from all this on an empty server, restore.sh brings the stacks back.

    Base files
    Command What it does
    cd / && sudo cp --parents etc/ssh/sshd_config.d/00-hardening.conf etc/apt/apt.conf.d/52unattended-upgrades-local etc/systemd/journald.conf.d/00-size.conf etc/sysctl.d/99-swappiness.conf etc/docker/daemon.json etc/systemd/system/kopia-* /opt/host/ system files from levels 1 and 5 — into /opt/host/etc/… under the same path
    sudo cp ~/.ssh/authorized_keys /opt/host/authorized_keys the public keys you log in with: a public key isn’t a secret

    When you edit a system file, edit its copy in /opt/host/etc/ too, or a new server gets the old version.

    /opt/host/bootstrap.shthe base of levels 1–5 on an empty Ubuntu; the server name is the first argument
    #!/bin/bash
    # Основа сервера по уровням 1–5 пути на пустой Ubuntu: имя, пользователь, SSH, фаервол, обновления, подкачка, журнал,
    # Docker, bouncer CrowdSec, Kopia. Запуск — под root: bash /opt/host/bootstrap.sh [имя]. Повторный запуск ничего не ломает
    set -eu
    
    name=${1:-web1}          # другое имя — для второго сервера: bash /opt/host/bootstrap.sh web2
    user=example-user
    zone=Europe/Berlin
    host=$(cd "$(dirname "$0")" && pwd)
    
    [ "$(id -u)" -eq 0 ] || { echo 'Запускаем под root' >&2; exit 1; }
    # Пароль root и вход root по SSH скрипт закрывает: без ключа пользователя на сервер будет не войти
    [ -s "$host/authorized_keys" ] || { echo "Нет $host/authorized_keys" >&2; exit 1; }
    
    export DEBIAN_FRONTEND=noninteractive
    
    # По часовому поясу наступает 4:00 ночной перезагрузки
    hostnamectl set-hostname "$name"
    sed -i '/^127\.0\.1\.1[[:space:]]/d' /etc/hosts
    echo "127.0.1.1 $name" >> /etc/hosts
    timedatectl set-timezone "$zone"
    
    apt-get update
    apt-get full-upgrade -y
    apt-get install -y ca-certificates curl gnupg sqlite3
    
    if ! id "$user" >/dev/null 2>&1; then
      adduser --disabled-password --comment '' "$user"
      usermod -aG sudo "$user"
      passwd "$user"         # пароль для sudo и консоли провайдера — в менеджер паролей
    fi
    install -d -m 700 -o "$user" -g "$user" "/home/$user/.ssh"
    install -m 600 -o "$user" -g "$user" "$host/authorized_keys" "/home/$user/.ssh/authorized_keys"
    
    # Системные файлы из git — на свои места: SSH, автообновления, журнал, подкачка, daemon.json, службы Kopia
    cp -r "$host/etc/." /etc/
    sshd -t
    systemctl restart ssh
    passwd -l root
    
    ufw default deny incoming
    ufw default allow outgoing
    ufw allow OpenSSH
    ufw allow 80,443/tcp
    ufw --force enable
    
    if ! swapon --show=NAME --noheadings | grep -qx /swapfile; then
      [ -f /swapfile ] || fallocate -l 2G /swapfile
      chmod 600 /swapfile
      mkswap /swapfile
      swapon /swapfile
    fi
    grep -q '^/swapfile ' /etc/fstab || echo '/swapfile none swap sw 0 0' >> /etc/fstab
    findmnt --verify         # ошибка в fstab останавливает загрузку — узнаём о ней сейчас
    sysctl --system >/dev/null
    systemctl restart systemd-journald
    
    # Docker и Kopia — из их репозиториев, bouncer — из репозитория CrowdSec
    for p in docker.io docker-compose docker-compose-v2 docker-doc podman-docker containerd runc; do
      if dpkg -s "$p" >/dev/null 2>&1; then apt-get remove -y "$p"; fi
    done
    install -m 0755 -d /etc/apt/keyrings
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
    chmod a+r /etc/apt/keyrings/docker.asc
    printf 'Types: deb\nURIs: https://download.docker.com/linux/ubuntu\nSuites: %s\nComponents: stable\nArchitectures: %s\nSigned-By: /etc/apt/keyrings/docker.asc\n' \
      "$(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")" "$(dpkg --print-architecture)" \
      > /etc/apt/sources.list.d/docker.sources
    curl -fsSL https://kopia.io/signing-key | gpg --dearmor --yes -o /etc/apt/keyrings/kopia-keyring.gpg
    echo 'deb [signed-by=/etc/apt/keyrings/kopia-keyring.gpg] http://packages.kopia.io/apt/ stable main' \
      > /etc/apt/sources.list.d/kopia.list
    curl -fsSL https://install.crowdsec.net | sh
    apt-get update
    apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin \
      kopia crowdsec-firewall-bouncer-nftables
    
    # daemon.json лёг раньше Docker, перезапуск — чтобы предел логов точно действовал
    systemctl restart docker
    docker network inspect proxy >/dev/null 2>&1 || docker network create proxy
    install -d -m 700 /srv/secrets
    # Таймеры бэкапа не включаем: снимок пустого сервера встал бы в хранилище последним
    systemctl daemon-reload
    
    echo "Основа $name готова. Вход — ssh $user@<адрес> из нового окна; это окно пока не закрываем"
    if [ -f /var/run/reboot-required ]; then echo 'Обновилось ядро: после проверки входа — sudo reboot'; fi
    /opt/scripts/restore.shstacks from the backup: all or the named ones
    #!/bin/bash
    # Стеки на новом сервере: файлы стека — из git, остальное (данные, .env, acme.json, /srv/secrets) — из последнего
    # снимка Kopia, потом запуск. sudo /opt/scripts/restore.sh — все стеки; sudo /opt/scripts/restore.sh memos gatus — названные
    set -eu
    cd /opt
    
    if [ $# -eq 0 ]; then
      set --
      for f in */docker-compose.yaml; do set -- "$@" "${f%%/*}"; done
    fi
    
    # Ключ bouncer'а лежит в /etc, вне бэкапа: в восстановленном CrowdSec меняем старый на новый
    bouncer() {
      for _ in $(seq 30); do
        if docker exec crowdsec cscli lapi status >/dev/null 2>&1; then break; fi
        sleep 2
      done
      docker exec crowdsec cscli bouncers delete firewall-bouncer >/dev/null 2>&1 || true
      key=$(docker exec crowdsec cscli bouncers add firewall-bouncer -o raw)
      sed -i "s#^api_key:.*#api_key: $key#" /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml
      chmod 600 /etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml
      systemctl restart crowdsec-firewall-bouncer
    }
    
    stack() {
      s=$1
      [ -f "$s/docker-compose.yaml" ] || { echo "$s: нет в /opt — сначала git pull" >&2; return 1; }
      # --skip-existing: файлы из git новее снимка, их не трогаем
      kopia snapshot restore --skip-existing "/opt/$s" "/opt/$s"
      case $s in
        memos)
          # Файлы самой базы бэкап не берёт, их место занимает дамп
          [ -f memos/data/memos_prod.db ] || cp memos/data/dump/memos.db memos/data/memos_prod.db ;;
        # База PostgreSQL — так же строкой: поднять только базу и залить дамп из data/dump/:
        # <стек>) (cd <стек> && docker compose up -d --wait db && docker compose exec -T db psql -U <пользователь> <база> < data/dump/<база>.sql) ;;
      esac
      (cd "$s" && docker compose up -d)
      if [ "$s" = crowdsec ]; then bouncer; fi
    }
    
    kopia snapshot restore --skip-existing /srv/secrets /srv/secrets
    
    # Traefik — первым, CrowdSec — вторым: остальным нужен прокси, а бан работает с первой минуты
    for s in traefik crowdsec; do
      case " $* " in *" $s "*) stack "$s" ;; esac
    done
    for s in "$@"; do
      case $s in traefik | crowdsec) ;; *) stack "$s" ;; esac
    done
    
    docker ps --format '{{.Names}}\t{{.Status}}'

    A PostgreSQL database comes back with one line in restore.sh, like the dump in level 5’s backup.sh: the example is in the comment at case.

    Permissions
    Command What it does
    sudo chmod 700 /opt/host/bootstrap.sh /opt/scripts/restore.sh only root runs and reads the scripts
    Done when
    Command What you should see
    sudo find /opt/host -type f | sort authorized_keys, bootstrap.sh and files under etc/: 00-hardening.conf, 52unattended-upgrades-local, 00-size.conf, 99-swappiness.conf, daemon.json, four kopia-*
    bash -n /opt/host/bootstrap.sh && bash -n /opt/scripts/restore.sh && echo ok ok: no syntax errors in the scripts
    Not like this — see “If it doesn’t work”, stage cases.

    The repository on GitHub

    Why — Server configs in git

    Create a private repository infra-web1 on GitHub — empty, no README. The access key is a deploy key: it opens this one repository, not the whole account. It needs write access: an urgent fix on the server is committed from there.

    Repository key
    Command What it does
    sudo ssh-keygen -t ed25519 -N '' -C infra-web1@web1 -f /root/.ssh/infra_deploy the key for git; root’s, since git in /opt runs with sudo
    sudo cat /root/.ssh/infra_deploy.pub the public key: in the repository — Settings, Deploy keys, Add, with “Allow write access” checked
    printf 'Host github-infra\n HostName github.com\n User git\n IdentityFile /root/.ssh/infra_deploy\n IdentitiesOnly yes\n' | sudo tee -a /root/.ssh/config a short name: git goes to GitHub with this key, not whichever comes first
    sudo ssh -T github-infra on first connect, check GitHub’s key fingerprint against docs.github.com; the answer is Hi <account>/infra-web1!
    First commit
    Command What it does
    sudo git -C /opt config user.name web1 && sudo git -C /opt config user.email example-user@web1 the author of commits made on the server
    sudo git -C /opt add . && sudo git -C /opt status --short the same list as in stage 1
    sudo git -C /opt commit -m "web1: stack configs and server base" the first commit
    sudo git -C /opt remote add origin git@github-infra:<account>/infra-web1.git the repository address; <account> is yours on GitHub
    sudo git -C /opt push -u origin main pushes

    On your computer, git clone: edit there from now on, and git pull on the server. More in Server configs in git.

    Done when
    Command What you should see
    sudo git -C /opt status --short nothing
    GitHub, repository infra-web1 stack directories, host/ and scripts/; not a single .env or acme.json
    Not like this — see “If it doesn’t work”, stage cases.

    New server: the base in one script

    From here on, work on the new VPS, at any host. For a drill, take a temporary one; below its address is 203.0.113.20. Commands run as root, without sudo. Keep the root window open until you’ve logged in as the user (Keep the old session open).

    Repository key — on the new server
    Command What it does
    ssh [email protected] log in; check the server’s key fingerprint in the provider console
    apt-get update && apt-get install -y git git; some images don’t have it
    ssh-keygen -t ed25519 -N '' -C infra-web1@new -f ~/.ssh/infra_deploy the new server’s own key: the old one stayed on the old server
    cat ~/.ssh/infra_deploy.pub the public key — into the Deploy keys of infra-web1, with “Allow write access”
    printf 'Host github-infra\n HostName github.com\n User git\n IdentityFile ~/.ssh/infra_deploy\n IdentitiesOnly yes\n' >> ~/.ssh/config the same short name as on the old server
    Base
    Command What it does
    git clone git@github-infra:<account>/infra-web1.git /opt the server’s skeleton — into /opt
    bash /opt/host/bootstrap.sh the base of levels 1–5; asks for the example-user password — save it in the password manager. A second server for some of the stacks — bash /opt/host/bootstrap.sh web2

    The script takes a few minutes. Log in from a second window, and only then close the root window.

    Log in as the user — in a second window
    Command What it does
    ssh [email protected] key login with a key from host/authorized_keys
    sudo reboot if the script said the kernel was updated

    If GitHub is unreachable, the whole /opt is in the backup. Then the order changes: hostnamectl set-hostname web1, Kopia with the three commands from stage 1 of level 5, storage access as in stage 5 below, kopia snapshot restore /opt /opt, and only then bash /opt/host/bootstrap.sh.

    Done when
    Command What you should see
    ssh [email protected] from your computer: Permission denied (publickey)
    sudo ufw status | head -1 Status: active
    sudo docker network ls --filter name=proxy -q | wc -l 1: the proxy network exists
    kopia --version 0.23.… or newer
    Not like this — see “If it doesn’t work”, stage cases.

    Storage and stacks from the backup

    Why — A backup that outlives the server

    Connect Kopia to the same storage as on level 5. The storage key is inside the backup, and the backup can’t be opened without the key, so the new server gets access of its own.

    A. Second server: the key — on the new server
    Command What it does
    sudo install -d -m 700 /srv/secrets/kopia the backup’s secrets directory
    sudo ssh-keygen -t ed25519 -N '' -C kopia@web1-new -f /srv/secrets/kopia/id_ed25519 the new server’s own SFTP key
    sudo cat /srv/secrets/kopia/id_ed25519.pub on backup1, add it as a restrict,command=… line in /srv/kopia/.ssh/authorized_keys, as in stage 2 of level 5; during a drill, leave the old key alone
    ssh-keyscan 198.51.100.20 | sudo tee /srv/secrets/kopia/known_hosts backup1’s keys
    sudo kopia repository connect sftp --host=198.51.100.20 --username=kopia --keyfile=/srv/secrets/kopia/id_ed25519 --known-hosts=/srv/secrets/kopia/known_hosts --path=/srv/kopia/web1 --override-hostname=web1 connects; asks for the storage password
    B. Google Drive
    Command What it does
    sudo apt-get install -y rclone && sudo rclone config the gdrive remote — the same answers as in stage 2 of level 5
    sudo kopia repository connect rclone --remote-path=gdrive:kopia-web1 --override-hostname=web1 connects; asks for the storage password

    --override-hostname=web1 — the old server’s snapshots are stored under its name. For web2 it’s web1 too: the stack comes from web1’s snapshots.

    Stacks
    Command What it does
    sudo /opt/scripts/restore.sh /srv/secrets and every stack: data from the latest snapshot, Traefik starts first
    sudo /opt/scripts/restore.sh traefik crowdsec memos or only the named ones: for a second server — Traefik, CrowdSec and the stack that moves

    The script takes a stack’s directory from the snapshot of the whole /opt, by a path inside the source.

    Check from your computer: --resolve sends the request to the new address while DNS still points to the old one. The certificate comes from the restored acme.json, so HTTPS works right away.

    Done when
    Command What you should see
    sudo docker ps --format '{{.Names}} {{.Status}}' the restored stacks are Up
    curl -s -o /dev/null -w '%{http_code}\n' --resolve memos.example.org:443:203.0.113.20 https://memos.example.org from your computer: 200
    sudo docker exec crowdsec cscli bouncers list firewall-bouncer, Last API pull seconds ago
    sudo kopia snapshot list --all | head -3 root@web1:/opt snapshots — the storage is connected
    Not like this — see “If it doesn’t work”, stage cases.

    The move, a second server, or a drill

    The new server is ready. Next comes one of three blocks.

    Drill — every six months and after big changes. Don’t enable the backup timers on the temporary server: its snapshots would land in storage next to the live server’s.

    Drill: remove the temporary server
    Command What it does
    VPS 203.0.113.20 — delete it in the host’s panel the temporary server is no longer needed
    key infra-web1@new — Delete in the Deploy keys of infra-web1 the deleted server’s key no longer opens the repository
    sudo sed -i '/kopia@web1-new/d' /srv/kopia/.ssh/authorized_keys option A, on backup1: the deleted server’s key no longer gets in

    Moving the whole server — the old server is gone, or you’re changing hosts. If the old one is still up, stop its stacks and take a fresh backup before stage 5, or anything written after the nightly snapshot stays on the old server: for f in /opt/*/docker-compose.yaml; do sudo docker compose -f "$f" stop; done, then sudo systemctl start kopia-backup.service.

    The move: address and backup
    Command What it does
    the domain’s A records — to 203.0.113.20 at the DNS provider service names lead to the new server; set the records’ TTL to 300 s a day before the move, and the change arrives within minutes
    sudo systemctl enable --now kopia-backup.timer kopia-verify.timer backup and check now run from the new server
    sudo systemctl disable --now kopia-backup.timer kopia-verify.timer on the old server, if it’s up: only the new one writes snapshots under the web1 name now
    key infra-web1@web1 — Delete in Deploy keys the old server no longer opens the repository

    In “Your values”, change the server address to the new one: the path’s recipes will pick it up.

    One stack to a second server — the old one can’t keep up. The second one is up as web2 (stages 4–5). So the data doesn’t diverge, stop the stack on web1 before stage 5 — cd /opt/memos && sudo docker compose stop — and take a backup: sudo systemctl start kopia-backup.service.

    Second server: its own repository and snapshots — on web2
    Command What it does
    sudo git -C /opt rm -r -q gatus beszel whoami web2 keeps only Traefik, CrowdSec and the stack that moved
    on GitHub — a private infra-web2; remove the infra-web1@new key from the Deploy keys of infra-web1 and add it to infra-web2 with “Allow write access” GitHub keeps a deploy key in one repository only; web2 goes to its own
    sudo git -C /opt remote set-url origin git@github-infra:<account>/infra-web2.git the short name github-infra stays, the repository is infra-web2
    sudo git -C /opt commit -am "web2: base and memos" && sudo git -C /opt push -u origin main web1’s history comes along: you can see where the stack came from
    sudo kopia repository disconnect disconnects from the web1 name
    the connect command from stage 5 without --override-hostname web2’s snapshots go under its own name
    echo '<ping-url>' | sudo tee /srv/secrets/kopia/hc-url > /dev/null web2 has its own web2-backup check in Healthchecks.io, as in stage 4 of level 5; <ping-url> is its address
    echo '<ping-url>' | sudo tee /srv/secrets/kopia/hc-verify-url > /dev/null and its own web2-verify for the restore check, as in stage 5 of level 5
    sudo systemctl enable --now kopia-backup.timer kopia-verify.timer web2’s backup on a schedule
    Second server: the stack leaves web1 — on web1
    Command What it does
    cd /opt/memos && sudo docker compose down memos on web1 is stopped and removed
    sudo git -C /opt rm -r memos && sudo git -C /opt commit -m "memos moved to web2" && sudo git -C /opt push the stack leaves the infra-web1 repository
    sudo rm -r /opt/memos the data is on web2 and in web1’s snapshots
    the A record for memos — to web2’s address the service name leads to the new server

    Gatus on web1 keeps checking memos over HTTPS — it doesn’t care which server the service runs on.

    Done when
    Command What you should see
    dig +short memos.example.org @1.1.1.1 move and second server: the new server’s address
    systemctl list-timers 'kopia-*' on the server that answers: two timers, both with NEXT filled in
    sudo kopia snapshot list --all drill, on web1: the latest snapshot is the nightly one, the temporary server made none
    Not like this — see “If it doesn’t work”, stage cases.

    If it doesn’t work

    Stage 1

    git add --dry-run lists .env, acme.json or a database file

    Cause
    A ! line in .gitignore allowed a whole directory, and the rules below no longer forbid anything in it.
    Fix
    For a directory, write three lines: !/stack/, /stack/* and the needed files by name — like the stacks in the stage 1 file.
    Stage 1

    A new stack doesn’t show up in git add --dry-run

    Cause
    It’s an allowlist: git doesn’t see a directory until .gitignore allows it.
    Fix
    Add a block for the stack to /opt/.gitignore: the directory, docker-compose.yaml, and .env.example if there is one.
    Stage 3

    ssh -T github-infra answers Permission denied (publickey)

    Cause
    The key isn’t in this repository’s Deploy keys, or ~/.ssh/config points to a different key.
    Fix
    Compare sudo cat /root/.ssh/infra_deploy.pub with the key in the repository settings, and check the IdentityFile line in /root/.ssh/config.
    Stage 3

    git push says ERROR: The key you are authenticating with has been marked as read only

    Cause
    The deploy key was added without “Allow write access”.
    Fix
    Delete the key in the repository settings and add it again with the box checked.
    Stage 4

    git clone says destination path '/opt' already exists and is not an empty directory

    Cause
    The host’s image already put something in /opt.
    Fix
    ls -A /opt shows what. Move it aside — mv /opt /opt.image && mkdir /opt — and clone again.
    Stage 4

    bootstrap.sh stopped with Нет /opt/host/authorized_keys

    Cause
    In stage 2 the keys weren’t copied or weren’t committed.
    Fix
    On the old server, run the stage 2 command with authorized_keys, commit and push; on the new one, git -C /opt pull and run the script again.
    Stage 4

    After the script, ssh [email protected] gives Permission denied (publickey)

    Cause
    host/authorized_keys has no key for the computer you log in from: it was copied from another machine.
    Fix
    In the root window that’s still open, append the ssh-ed25519 … line from this computer to /home/example-user/.ssh/authorized_keys, and on the old server to /opt/host/authorized_keys, with a commit.
    Stage 4

    curl … install.crowdsec.net or apt-get update fails and the script stops

    Cause
    The network or a repository is unreachable at that moment.
    Fix
    Run bash /opt/host/bootstrap.sh again: it skips what’s done or redoes it harmlessly.
    Stage 5

    restore.sh says there are no snapshots for /opt/…

    Cause
    Kopia looks up snapshots by server name, and the new server has a different name or connected without --override-hostname.
    Fix
    sudo kopia snapshot list --all shows the name in the snapshots. Reconnect with --override-hostname=web1, as in stage 5.
    Stage 5

    Traefik on the new server serves the TRAEFIK DEFAULT CERT certificate

    Cause
    acme.json wasn’t restored: the snapshot predates the certificate, or the file was excluded.
    Fix
    sudo ls -l /opt/traefik/data/acme.json — size above zero and mode 600. If it’s empty, Traefik issues certificates itself once DNS is switched.
    Stage 5

    sudo docker exec crowdsec cscli bouncers list shows firewall-bouncer with no Last API pull

    Cause
    CrowdSec didn’t come up within a minute, and the bouncer key wasn’t written.
    Fix
    sudo /opt/scripts/restore.sh crowdsec — the script replaces the key again.
    Stage 6

    After the DNS change, some requests still reach the old server

    Cause
    Resolvers keep the old record until its TTL expires.
    Fix
    Wait out the record’s TTL; don’t shut down the old server until dig +short memos.example.org @1.1.1.1 returns the new address.

    How to remove it

    Removing the repository from /opt doesn’t touch the stacks: only the git and base files go.

    sudo rm -rf /opt/.git /opt/.gitignore /opt/.gitattributes /opt/host /opt/scripts/restore.sh
    sudo rm /root/.ssh/infra_deploy /root/.ssh/infra_deploy.pub

    Delete the infra-web1@web1 key in the repository settings on GitHub, and the repository itself there too, under Settings.

    Next on the path