basforge

Search titles, text and the “If it doesn’t work” cases.

    RU

    A backup that outlives the server

    Every night the server dumps the memos database, and Kopia snapshots /opt and /srv/secrets, encrypts the snapshot and sends it to a second server over SFTP or to Google Drive. Healthchecks.io waits for a ping after each backup, and once a week the server restores the database from storage and checks it on its own.

    Start froma server after level 4, “Observability”: memos opens by name, Healthchecks.io has the ntfy integration.

    What you get

    every night at 03:00 ── memos dump, /opt, /srv/secrets ──▶ Kopia ── encrypts ──▶ backup1 over SFTP
                                                                                  or Google Drive
    Kopia ── ping after the snapshot ──▶ Healthchecks.io   no ping by 05:00 — alert to ntfy
    once a week ── 5 % of files from storage, the memos database is restored and checked
                ── ping ──▶ Healthchecks.io, a check of its own
    /opt/scripts/backup.sh, backup-verify.sh      backup and check
    /etc/systemd/system/kopia-*.timer             schedule
    /srv/secrets/kopia/                           ping URLs, SSH key for option A
    • You’ll needA password manager: it holds the storage password — without the password the backup can’t be read
    • You’ll needFor option A — a second server with SSH and free space, at another host or at home; below it’s backup1 at 198.51.100.20
    • You’ll needFor option B — a Google account and rclone on a computer with a browser, to grant Drive access once
    • You’ll needThe Healthchecks.io account and the ntfy topic from level 4

    Your values

    Filled into the commands and files of this and other recipes on the path; stored in this browser only.
    Server name
    web1placeholder
    Backup server address
    198.51.100.20placeholder

    Without the script the commands show placeholder values: replace them with yours.

    Steps

    The same recipe, short: commands from the stages, files as one line. For a second server and for those who already know what each command does.

    Kopia and the storage password

    Kopia encrypts a snapshot on the server, before sending it, with a key derived from the storage password. Storage sees only encrypted chunks, and only changes take up space: an unchanged file costs nothing in the tenth snapshot.

    Kopia from its repository
    Command What it does
    curl -s https://kopia.io/signing-key | sudo gpg --dearmor -o /etc/apt/keyrings/kopia-keyring.gpg the key that signs Kopia packages
    echo "deb [signed-by=/etc/apt/keyrings/kopia-keyring.gpg] http://packages.kopia.io/apt/ stable main" | sudo tee /etc/apt/sources.list.d/kopia.list the Kopia repository; http is fine: apt checks package signatures with the key
    sudo apt update && sudo apt install -y kopia sqlite3 Kopia, plus sqlite3 for the memos database dump
    sudo install -d -m 700 /srv/secrets/kopia the directory for backup secrets
    Password
    Command What it does
    openssl rand -base64 24 the storage password, 32 characters; printed once — save it to your password manager
    Done when
    Command What you should see
    kopia --version 0.23.… or newer
    password manager an entry with the Kopia storage password
    Not like this — see “If it doesn’t work”, stage cases.

    Storage: a second server or Google Drive

    The backup lives where whatever breaks the server can’t reach it: at another host or at home. Pick one option. A is a second server over SFTP: space is limited only by its disk. B is Google Drive through rclone: no second server needed, but the free 15 GB is shared with Gmail and Photos.

    Both options survive the server being deleted and a hosting outage, but not a break-in. The server is allowed to delete in storage, so whoever gets root on it can wipe the backups too. Storage the server can write to but can’t wipe is kopia server on a second server: Kopia server (RU).

    Option A. Create a key on web1 and a kopia user on backup1. The key only gets SFTP: command= in authorized_keys runs sftp-server instead of a shell, and restrict turns off tunnels and the terminal.

    A. The key — on web1
    Command What it does
    sudo ssh-keygen -t ed25519 -N '' -C kopia@web1 -f /srv/secrets/kopia/id_ed25519 a key without a passphrase: backups run at night, nobody is there to type it
    sudo cat /srv/secrets/kopia/id_ed25519.pub the public key — it goes on backup1
    A. The user — on backup1
    Command What it does
    sudo adduser --disabled-password --comment '' --home /srv/kopia kopia a user without a password, key login only
    sudo install -d -m 700 -o kopia -g kopia /srv/kopia/.ssh /srv/kopia/web1 the key directory and the storage directory
    sudo install -m 600 -o kopia -g kopia /dev/null /srv/kopia/.ssh/authorized_keys an empty authorized_keys with the right mode from the start
    echo 'restrict,command="/usr/lib/openssh/sftp-server" <key>' | sudo tee -a /srv/kopia/.ssh/authorized_keys web1’s key, for SFTP only; replace <key> with the ssh-ed25519 … line from web1
    sudo sed -i 's/^AllowUsers .*/& kopia/' /etc/ssh/sshd_config.d/00-hardening.conf && sudo systemctl reload ssh if backup1 was set up by level 1: adds kopia to AllowUsers
    A. The storage — on web1
    Command What it does
    ssh-keyscan 198.51.100.20 | sudo tee /srv/secrets/kopia/known_hosts backup1’s keys: Kopia checks the server against them on every login
    sudo kopia repository create sftp --host=198.51.100.20 --username=kopia --keyfile=/srv/secrets/kopia/id_ed25519 --known-hosts=/srv/secrets/kopia/known_hosts --path=/srv/kopia/web1 creates the storage and asks for the password twice — paste the one from stage 1

    Option B. rclone gets Drive access with the drive.file scope: it sees only the files it created itself, the rest of your Drive stays closed to the server. Signing in to Google needs a browser, so one command runs on your own computer.

    B. rclone and Drive access — on web1
    Command What it does
    sudo apt install -y rclone rclone: Kopia reaches Google Drive through it
    sudo rclone config a new remote — answer the questions as in the list below

    Answers in sudo rclone config: n for a new remote, name gdrive, type drive; client_id and client_secret empty; scope — drive.file; service_account_file empty; advanced config — n; browser sign-in on this machine — n. rclone prints a rclone authorize "drive" "…" command: run it on your computer with a browser, sign in to Google, and paste the printed token back into the server prompt. Shared drive — n, keep — y, quit — q.

    B. The storage — on web1
    Command What it does
    sudo rclone lsd gdrive: Drive answers; no files are listed — rclone hasn’t created any yet
    sudo kopia repository create rclone --remote-path=gdrive:kopia-web1 creates the storage in the kopia-web1 folder and asks for the password twice — paste the one from stage 1
    Done when
    Command What you should see
    sudo kopia repository status the storage is connected: web1, root and a storage line — SFTP or rclone
    Not like this — see “If it doesn’t work”, stage cases.

    What to back up: a database dump and the first snapshot

    Kopia takes /opt — every stack with its data/, as the server profile says — and /srv/secrets. Leave out the live memos database file: copied in the middle of a write, it may not open. Take a dump instead, with SQLite’s .backup: a whole copy even while memos writes. Leave out Traefik’s request log too: it’s big, and there’s nothing to restore from it.

    Snapshot rules
    Command What it does
    sudo kopia policy set --global --compression=zstd compression: configs and dumps shrink, images and archives don’t
    sudo kopia policy set /opt --add-ignore 'memos/data/memos_prod.db*' --add-ignore 'traefik/data/logs/' skips the live memos database and the request log
    Dump and first snapshot
    Command What it does
    sudo install -d -m 700 /opt/memos/data/dump the dump directory next to the memos data
    sudo sqlite3 /opt/memos/data/memos_prod.db ".backup '/opt/memos/data/dump/memos.db'" dumps the memos database
    sudo kopia snapshot create /opt /srv/secrets the first snapshot sends everything, later ones only changes
    Done when
    Command What you should see
    sudo kopia snapshot list --all root@web1:/opt and root@web1:/srv/secrets, each with a dated snapshot line
    Not like this — see “If it doesn’t work”, stage cases.

    On a schedule: timer and ping

    A systemd timer starts the backup at 03:00, an hour before the reboot after updates (level 1). The script takes the dump, makes the snapshot and pings Healthchecks.io: /start before the backup, the bare URL on success and /fail with the tail of the output on an error. A night without a ping is an alert too: the timer didn’t fire.

    In Healthchecks.io, “Add Check”: name web1-backup, Period — 1 day, Grace Time — 2 hours. The ntfy integration from level 4 is on for the new check — a tick on the check’s page. Copy the ping URL from there too.

    Ping URL
    Command What it does
    echo '<ping-url>' | sudo tee /srv/secrets/kopia/hc-url > /dev/null replace <ping-url> with a URL like https://hc-ping.com/…
    sudo chmod 400 /srv/secrets/kopia/hc-url read-only, root only
    sudo mkdir -p /opt/scripts the server’s script directory
    /opt/scripts/backup.shdump, snapshot, ping
    #!/bin/sh
    # Бэкап сервера: дампы баз, снимок Kopia, пинг в Healthchecks.io. Запускает kopia-backup.timer
    set -u
    
    ping=$(cat /srv/secrets/kopia/hc-url)
    curl -fsS -m 10 --retry 3 -o /dev/null "$ping/start"
    
    run() {
      # Дамп SQLite через .backup — целая копия и на ходу; файл базы, скопированный во время записи, бывает битым.
      # База PostgreSQL — так же строкой: docker exec <стек>-db pg_dump -U <пользователь> <база> > /opt/<стек>/data/dump/<база>.sql
      install -d -m 700 /opt/memos/data/dump &&
        sqlite3 /opt/memos/data/memos_prod.db ".backup '/opt/memos/data/dump/memos.db'" &&
        kopia snapshot create /opt /srv/secrets
    }
    
    if out=$(run 2>&1); then
      curl -fsS -m 10 --retry 3 -o /dev/null "$ping"
    else
      printf '%s\n' "$out" >&2
      printf '%s\n' "$out" | tail -c 10000 | curl -fsS -m 10 --retry 3 -o /dev/null --data-binary @- "$ping/fail"
      exit 1
    fi

    A service with a PostgreSQL database takes one more line in run(): pg_dump writes a dump to the stack’s data/dump/, and the live database files go into --add-ignore, like memos.

    /etc/systemd/system/kopia-backup.serviceruns the script
    [Unit]
    Description=Бэкап сервера в Kopia
    Wants=network-online.target
    After=network-online.target
    
    [Service]
    Type=oneshot
    ExecStart=/opt/scripts/backup.sh
    # Kopia и rclone ищут настройки в ~/.config: без User= systemd не задаёт HOME
    Environment=HOME=/root
    Nice=10
    IOSchedulingClass=idle
    /etc/systemd/system/kopia-backup.timerevery night at 03:00
    [Unit]
    Description=Бэкап сервера по расписанию
    
    [Timer]
    OnCalendar=*-*-* 03:00
    Persistent=true
    
    [Install]
    WantedBy=timers.target

    Persistent=true catches up on a miss: if the server was off at 03:00, the backup runs right after boot.

    Script and timer
    Command What it does
    sudo chmod 700 /opt/scripts/backup.sh the script runs, only root can read it
    sudo systemctl daemon-reload && sudo systemctl enable --now kopia-backup.timer the timer is on and survives a reboot

    How many snapshots to keep depends on how the data lives. A snapshot stays if any rule matches it: with the minimal scenario, after half a year you have the last 7 days, 4 weeks and 6 months. Pick one row and one command below.

    Scenario When it fits Snapshot Keeps Restore check
    Minimal services set up once and barely touched: something broke a month later — bring back the month-old snapshot daily 7 daily, 4 weekly, 6 monthly monthly
    Regular the data is used every day, losing a day hurts daily 14 daily, 8 weekly, 12 monthly weekly
    Active the data changes all day, losing even a few hours is expensive hourly 48 hourly, 30 daily, 12 weekly, 12 monthly weekly
    Minimal
    Command What it does
    sudo kopia policy set --global --keep-latest 7 --keep-hourly 0 --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --keep-annual 0 how many snapshots to keep
    Regular
    Command What it does
    sudo kopia policy set --global --keep-latest 14 --keep-hourly 0 --keep-daily 14 --keep-weekly 8 --keep-monthly 12 --keep-annual 0 how many snapshots to keep
    Active
    Command What it does
    sudo kopia policy set --global --keep-latest 48 --keep-hourly 48 --keep-daily 30 --keep-weekly 12 --keep-monthly 12 --keep-annual 0 how many snapshots to keep
    sudo sed -i 's/^OnCalendar=.*/OnCalendar=hourly/' /etc/systemd/system/kopia-backup.timer a backup every hour
    sudo systemctl daemon-reload && sudo systemctl restart kopia-backup.timer the timer with the new schedule

    With the active scenario, set the web1-backup check in Healthchecks.io to Period — 1 hour, Grace Time — 30 minutes. Kopia deletes extra snapshots itself after each backup, and scheduled storage maintenance frees the space they used — Kopia runs that on its own too.

    First run
    Command What it does
    sudo systemctl start kopia-backup.service a backup now, without waiting for the night
    Done when
    Command What you should see
    systemctl list-timers kopia-backup.timer NEXT shows the next 03:00, as set in the timer’s OnCalendar; with the active scenario — the start of the next hour
    sudo journalctl -u kopia-backup.service -n 3 Finished kopia-backup.service, no Failed
    Healthchecks.io, the web1-backup check up, last ping just now
    Not like this — see “If it doesn’t work”, stage cases.

    Restore — test it by doing

    Why — Test your server’s protection by action

    A backup nobody has ever restored from may turn out empty because of one typo in a path. So delete a memo and bring it back from storage. Then let the server do the same on a schedule.

    In memos, create a memo “Backup test”, run a backup with the stage 4 command, and delete the memo.

    Bring back the memos database
    Command What it does
    cd /opt/memos && sudo docker compose stop stops memos: don’t swap the database under it
    sudo kopia snapshot restore /opt/memos/data/dump /tmp/memos-restore the dump from the latest snapshot, into a temporary directory
    sudo cp /tmp/memos-restore/memos.db data/memos_prod.db the restored database in place of the live one
    sudo rm -f data/memos_prod.db-wal data/memos_prod.db-shm the old database’s journal: SQLite would apply it to the restored one
    sudo docker compose start && sudo rm -r /tmp/memos-restore starts memos and removes the temporary directory

    The “Backup test” memo is back. The check script does the same without stopping memos: it downloads 5 % of the files from storage, restores the dump into a temporary directory and asks SQLite whether the database is intact and has memos in it. An empty database is intact too: that’s the dump you get when backup.sh points at an old database path.

    The script pings a separate check in Healthchecks.io: a ping to web1-backup would reset its timer, and a missed nightly backup would go unnoticed. “Add Check”: name web1-verify, Period — 7 days, Grace Time — 1 day.

    The check’s ping URL
    Command What it does
    echo '<ping-url>' | sudo tee /srv/secrets/kopia/hc-verify-url > /dev/null <ping-url> is the URL of the web1-verify check
    sudo chmod 400 /srv/secrets/kopia/hc-verify-url read-only, root only
    /opt/scripts/backup-verify.sh5 % of files and the memos database from storage
    #!/bin/sh
    # Проверка бэкапа: часть файлов скачиваем из хранилища, базу memos восстанавливаем и проверяем. Запускает kopia-verify.timer
    set -u
    
    # Своя проверка web1-verify в Healthchecks.io: пинг в web1-backup сбросил бы её счётчик, и пропуск ночного бэкапа
    # остался бы незамеченным
    ping=$(cat /srv/secrets/kopia/hc-verify-url)
    curl -fsS -m 10 --retry 3 -o /dev/null "$ping/start"
    dir=$(mktemp -d)
    trap 'rm -rf "$dir"' EXIT
    
    # Целая база ещё не значит, что в ней данные: дамп базы, которой нет по пути из backup.sh, — пустая база,
    # и integrity_check отвечает ok
    check() {
      kopia snapshot verify --verify-files-percent=5 &&
        kopia snapshot restore /opt/memos/data/dump "$dir" &&
        test "$(sqlite3 "$dir/memos.db" 'PRAGMA integrity_check')" = ok &&
        notes=$(sqlite3 "$dir/memos.db" 'SELECT count(*) FROM memo') &&
        { [ "$notes" -gt 0 ] || { echo 'В дампе memos нет заметок'; false; }; }
    }
    
    if out=$(check 2>&1); then
      curl -fsS -m 10 --retry 3 -o /dev/null "$ping"
    else
      printf '%s\n' "$out" >&2
      printf '%s\n' "$out" | tail -c 10000 | curl -fsS -m 10 --retry 3 -o /dev/null --data-binary @- "$ping/fail"
      exit 1
    fi
    /etc/systemd/system/kopia-verify.serviceruns the check
    [Unit]
    Description=Проверка восстановления из Kopia
    Wants=network-online.target
    After=network-online.target
    
    [Service]
    Type=oneshot
    ExecStart=/opt/scripts/backup-verify.sh
    Environment=HOME=/root
    Nice=10
    IOSchedulingClass=idle
    /etc/systemd/system/kopia-verify.timerSundays at 05:30, after the 04:00 reboot
    [Unit]
    Description=Проверка восстановления по расписанию
    
    [Timer]
    OnCalendar=Sun *-*-* 05:30
    Persistent=true
    
    [Install]
    WantedBy=timers.target
    Check on a schedule
    Command What it does
    sudo chmod 700 /opt/scripts/backup-verify.sh the script runs, only root can read it
    sudo systemctl daemon-reload && sudo systemctl enable --now kopia-verify.timer a check once a week
    sudo systemctl start kopia-verify.service a check now; takes minutes — files download from storage

    With the minimal scenario the check runs monthly: set OnCalendar=*-*-01 05:30 in kopia-verify.timer, then daemon-reload and restart kopia-verify.timer again, and give web1-verify a Period of 31 days.

    If the server itself is gone, bring up a new one by level 1 with the same name, web1, and install Kopia with the stage 1 commands. Connect to the storage with the password from your password manager. With option A the new server has a key of its own: create it and known_hosts as in stage 2, and put the key on backup1 in place of the old one. With option B, go through sudo rclone config again.

    New server: data from storage
    Command What it does
    sudo kopia repository connect sftp --host=198.51.100.20 --username=kopia --keyfile=/srv/secrets/kopia/id_ed25519 --known-hosts=/srv/secrets/kopia/known_hosts --path=/srv/kopia/web1 option A: connects to the storage on backup1, asks for the password
    sudo kopia repository connect rclone --remote-path=gdrive:kopia-web1 option B: connects to the storage in Google Drive, asks for the password
    sudo kopia snapshot restore --skip-existing /srv/secrets /srv/secrets secrets back in place; --skip-existing leaves the new key alone: backup1 no longer accepts the old one
    sudo kopia snapshot restore /opt /opt the stacks with their data and dumps
    sudo cp /opt/memos/data/dump/memos.db /opt/memos/data/memos_prod.db the memos database from the dump

    Then bring the stacks up as on levels 2–4: the proxy network, then traefik, then the rest — cd /opt/<stack> && sudo docker compose up -d. The level 6 recipe Moving to a new server brings up the base and the stacks with one command each.

    Done when
    Command What you should see
    memos in the browser the “Backup test” memo is there
    systemctl list-timers 'kopia-*' two timers, both with NEXT filled in
    sudo journalctl -u kopia-verify.service -n 3 Finished kopia-verify.service, no Failed
    Healthchecks.io, the web1-verify check up, last ping just now
    Not like this — see “If it doesn’t work”, stage cases.

    If it doesn’t work

    Stage 1

    apt update says NO_PUBKEY or is not signed about packages.kopia.io

    Cause
    The key wasn’t written: curl didn’t download it, and gpg saved an empty file.
    Fix
    ls -l /etc/apt/keyrings/kopia-keyring.gpg should show a size above zero. Zero — run the stage’s first command again.
    Stage 2

    Option A: kopia repository create sftp says unable to authenticate

    Cause
    backup1 doesn’t let the kopia user in: the key isn’t in authorized_keys, .ssh has the wrong mode, or the user isn’t in AllowUsers.
    Fix
    The log on backup1 names the reason: sudo grep kopia /var/log/auth.log | tail -5. A not listed in AllowUsers line means the AllowUsers command was skipped.
    Stage 2

    Option A: the error knownhosts: key is unknown

    Cause
    known_hosts has a different address: --host has a name while ssh-keyscan asked by address, or the other way round.
    Fix
    Use the same value, 198.51.100.20, in --host and in ssh-keyscan, and write known_hosts again.
    Stage 2

    Option B: kopia repository create rclone says executable file not found

    Cause
    Kopia starts rclone itself, and it isn’t on the server.
    Fix
    sudo apt install -y rclone, then sudo rclone config from stage 2.
    Stage 2

    Option B: backups are slow, the log shows rateLimitExceeded

    Cause
    rclone talks to Google Drive with its shared client_id, and everyone who didn’t change it shares that request limit.
    Fix
    Create your own client_id following Making your own client_id in the rclone Google Drive docs and enter it in sudo rclone config.
    Stage 3

    sqlite3 says unable to open database

    Cause
    The memos database has a different name: that happens when compose sets MEMOS_DSN or demo mode is on.
    Fix
    sudo ls /opt/memos/data shows the *.db file name. Use it in the dump command, the --add-ignore rule and backup.sh.
    Stage 4

    sudo kopia snapshot create works by hand, but from the timer it fails saying the repository isn’t connected

    Cause
    The service runs without HOME: Kopia looks for its settings in ~/.config/kopia and finds nothing.
    Fix
    systemctl show kopia-backup.service -p Environment should print HOME=/root. If not, compare the unit with stage 4 and run sudo systemctl daemon-reload.
    Stage 4

    The web1-backup check in Healthchecks.io stays new

    Cause
    The ping doesn’t go out: hc-url has a stray space, a quote or a different URL.
    Fix
    sudo sh -c 'curl -fsS "$(cat /srv/secrets/kopia/hc-url)"' should print OK. On an error, write the URL again as in stage 4.
    Stage 5

    After the restore memos says database disk image is malformed or shows the wrong memos

    Cause
    memos_prod.db-wal and -shm from the old database stayed next to the restored one: SQLite applied the old journal to the new database.
    Fix
    Restore again: memos stopped, memos.db copied once more, both journal files deleted, then start.
    Stage 5

    On a new server kopia snapshot restore /opt says there are no snapshots

    Cause
    Kopia finds snapshots by user and host name, and the new server has a different name.
    Fix
    sudo kopia snapshot list --all shows the name the snapshots are under. Reconnect with --override-hostname=web1 or set the name with sudo hostnamectl set-hostname web1.

    How to remove it

    sudo systemctl disable --now kopia-backup.timer kopia-verify.timer
    sudo rm /etc/systemd/system/kopia-*.service /etc/systemd/system/kopia-*.timer
    sudo rm /opt/scripts/backup.sh /opt/scripts/backup-verify.sh
    sudo systemctl daemon-reload
    sudo kopia repository disconnect
    sudo rm -r /srv/secrets/kopia /opt/memos/data/dump

    The storage stays where it was: on backup1 sudo deluser --remove-home kopia removes it, in Google Drive — the kopia-web1 folder. Delete the web1-backup and web1-verify checks in Healthchecks.io too, or they send down.

    Next on the path