Kopia and the storage password
doneKopia encrypts a snapshot on the server, before sending it, with a key derived from the storage password. Storage sees only encrypted chunks, and only changes take up space: an unchanged file costs nothing in the tenth snapshot.
| Command | What it does |
|---|---|
curl -s https://kopia.io/signing-key | sudo gpg --dearmor -o /etc/apt/keyrings/kopia-keyring.gpg |
the key that signs Kopia packages |
echo "deb [signed-by=/etc/apt/keyrings/kopia-keyring.gpg] http://packages.kopia.io/apt/ stable main" | sudo tee /etc/apt/sources.list.d/kopia.list |
the Kopia repository; http is fine: apt checks package signatures with the key |
sudo apt update && sudo apt install -y kopia sqlite3 |
Kopia, plus sqlite3 for the memos database dump |
sudo install -d -m 700 /srv/secrets/kopia |
the directory for backup secrets |
| Command | What it does |
|---|---|
openssl rand -base64 24 |
the storage password, 32 characters; printed once — save it to your password manager |
| Command | What you should see |
|---|---|
kopia --version |
0.23.… or newer |
| password manager | an entry with the Kopia storage password |
Storage: a second server or Google Drive
doneThe backup lives where whatever breaks the server can’t reach it: at another host or at home. Pick one option. A is a second server over SFTP: space is limited only by its disk. B is Google Drive through rclone: no second server needed, but the free 15 GB is shared with Gmail and Photos.
Both options survive the server being deleted and a hosting outage, but not a break-in. The server is
allowed to delete in storage, so whoever gets root on it can wipe the backups too. Storage the server can
write to but can’t wipe is kopia server on a second server: Kopia server (RU).
Option A. Create a key on web1 and a kopia user on backup1. The key only gets SFTP: command= in
authorized_keys runs sftp-server instead of a shell, and restrict turns off tunnels and the terminal.
| Command | What it does |
|---|---|
sudo ssh-keygen -t ed25519 -N '' -C kopia@web1 -f /srv/secrets/kopia/id_ed25519 |
a key without a passphrase: backups run at night, nobody is there to type it |
sudo cat /srv/secrets/kopia/id_ed25519.pub |
the public key — it goes on backup1 |
| Command | What it does |
|---|---|
sudo adduser --disabled-password --comment '' --home /srv/kopia kopia |
a user without a password, key login only |
sudo install -d -m 700 -o kopia -g kopia /srv/kopia/.ssh /srv/kopia/web1 |
the key directory and the storage directory |
sudo install -m 600 -o kopia -g kopia /dev/null /srv/kopia/.ssh/authorized_keys |
an empty authorized_keys with the right mode from the start |
echo 'restrict,command="/usr/lib/openssh/sftp-server" <key>' | sudo tee -a /srv/kopia/.ssh/authorized_keys |
web1’s key, for SFTP only; replace <key> with the ssh-ed25519 … line from web1 |
sudo sed -i 's/^AllowUsers .*/& kopia/' /etc/ssh/sshd_config.d/00-hardening.conf && sudo systemctl reload ssh |
if backup1 was set up by level 1: adds kopia to AllowUsers |
| Command | What it does |
|---|---|
ssh-keyscan 198.51.100.20 | sudo tee /srv/secrets/kopia/known_hosts |
backup1’s keys: Kopia checks the server against them on every login |
sudo kopia repository create sftp --host=198.51.100.20 --username=kopia --keyfile=/srv/secrets/kopia/id_ed25519 --known-hosts=/srv/secrets/kopia/known_hosts --path=/srv/kopia/web1 |
creates the storage and asks for the password twice — paste the one from stage 1 |
Option B. rclone gets Drive access with the drive.file scope: it sees only the files it created
itself, the rest of your Drive stays closed to the server. Signing in to Google needs a browser, so one
command runs on your own computer.
| Command | What it does |
|---|---|
sudo apt install -y rclone |
rclone: Kopia reaches Google Drive through it |
sudo rclone config |
a new remote — answer the questions as in the list below |
Answers in sudo rclone config: n for a new remote, name gdrive, type drive; client_id and
client_secret empty; scope — drive.file; service_account_file empty; advanced config — n;
browser sign-in on this machine — n. rclone prints a rclone authorize "drive" "…" command: run it on
your computer with a browser, sign in to Google, and paste the printed token back into the server prompt.
Shared drive — n, keep — y, quit — q.
| Command | What it does |
|---|---|
sudo rclone lsd gdrive: |
Drive answers; no files are listed — rclone hasn’t created any yet |
sudo kopia repository create rclone --remote-path=gdrive:kopia-web1 |
creates the storage in the kopia-web1 folder and asks for the password twice — paste the one from stage 1 |
| Command | What you should see |
|---|---|
sudo kopia repository status |
the storage is connected: web1, root and a storage line — SFTP or rclone |
What to back up: a database dump and the first snapshot
doneKopia takes /opt — every stack with its data/, as the server profile says — and /srv/secrets. Leave
out the live memos database file: copied in the middle of a write, it may not open. Take a dump instead,
with SQLite’s .backup: a whole copy even while memos writes. Leave out Traefik’s request log too: it’s big,
and there’s nothing to restore from it.
| Command | What it does |
|---|---|
sudo kopia policy set --global --compression=zstd |
compression: configs and dumps shrink, images and archives don’t |
sudo kopia policy set /opt --add-ignore 'memos/data/memos_prod.db*' --add-ignore 'traefik/data/logs/' |
skips the live memos database and the request log |
| Command | What it does |
|---|---|
sudo install -d -m 700 /opt/memos/data/dump |
the dump directory next to the memos data |
sudo sqlite3 /opt/memos/data/memos_prod.db ".backup '/opt/memos/data/dump/memos.db'" |
dumps the memos database |
sudo kopia snapshot create /opt /srv/secrets |
the first snapshot sends everything, later ones only changes |
| Command | What you should see |
|---|---|
sudo kopia snapshot list --all |
root@web1:/opt and root@web1:/srv/secrets, each with a dated snapshot line |
On a schedule: timer and ping
doneA systemd timer starts the backup at 03:00, an hour before the reboot after updates (level 1). The script
takes the dump, makes the snapshot and pings Healthchecks.io: /start before the backup, the bare URL on
success and /fail with the tail of the output on an error. A night without a ping is an alert too: the
timer didn’t fire.
In Healthchecks.io, “Add Check”: name web1-backup, Period — 1 day, Grace Time — 2 hours. The ntfy
integration from level 4 is on for the new check — a tick on the check’s page. Copy the ping URL from there
too.
| Command | What it does |
|---|---|
echo '<ping-url>' | sudo tee /srv/secrets/kopia/hc-url > /dev/null |
replace <ping-url> with a URL like https://hc-ping.com/… |
sudo chmod 400 /srv/secrets/kopia/hc-url |
read-only, root only |
sudo mkdir -p /opt/scripts |
the server’s script directory |
#!/bin/sh
# Бэкап сервера: дампы баз, снимок Kopia, пинг в Healthchecks.io. Запускает kopia-backup.timer
set -u
ping=$(cat /srv/secrets/kopia/hc-url)
curl -fsS -m 10 --retry 3 -o /dev/null "$ping/start"
run() {
# Дамп SQLite через .backup — целая копия и на ходу; файл базы, скопированный во время записи, бывает битым.
# База PostgreSQL — так же строкой: docker exec <стек>-db pg_dump -U <пользователь> <база> > /opt/<стек>/data/dump/<база>.sql
install -d -m 700 /opt/memos/data/dump &&
sqlite3 /opt/memos/data/memos_prod.db ".backup '/opt/memos/data/dump/memos.db'" &&
kopia snapshot create /opt /srv/secrets
}
if out=$(run 2>&1); then
curl -fsS -m 10 --retry 3 -o /dev/null "$ping"
else
printf '%s\n' "$out" >&2
printf '%s\n' "$out" | tail -c 10000 | curl -fsS -m 10 --retry 3 -o /dev/null --data-binary @- "$ping/fail"
exit 1
fi
A service with a PostgreSQL database takes one more line in run(): pg_dump writes a dump to the stack’s
data/dump/, and the live database files go into --add-ignore, like memos.
[Unit]
Description=Бэкап сервера в Kopia
Wants=network-online.target
After=network-online.target
[Service]
Type=oneshot
ExecStart=/opt/scripts/backup.sh
# Kopia и rclone ищут настройки в ~/.config: без User= systemd не задаёт HOME
Environment=HOME=/root
Nice=10
IOSchedulingClass=idle
[Unit]
Description=Бэкап сервера по расписанию
[Timer]
OnCalendar=*-*-* 03:00
Persistent=true
[Install]
WantedBy=timers.target
Persistent=true catches up on a miss: if the server was off at 03:00, the backup runs right after boot.
| Command | What it does |
|---|---|
sudo chmod 700 /opt/scripts/backup.sh |
the script runs, only root can read it |
sudo systemctl daemon-reload && sudo systemctl enable --now kopia-backup.timer |
the timer is on and survives a reboot |
How many snapshots to keep depends on how the data lives. A snapshot stays if any rule matches it: with the minimal scenario, after half a year you have the last 7 days, 4 weeks and 6 months. Pick one row and one command below.
| Scenario | When it fits | Snapshot | Keeps | Restore check |
|---|---|---|---|---|
| Minimal | services set up once and barely touched: something broke a month later — bring back the month-old snapshot | daily | 7 daily, 4 weekly, 6 monthly | monthly |
| Regular | the data is used every day, losing a day hurts | daily | 14 daily, 8 weekly, 12 monthly | weekly |
| Active | the data changes all day, losing even a few hours is expensive | hourly | 48 hourly, 30 daily, 12 weekly, 12 monthly | weekly |
| Command | What it does |
|---|---|
sudo kopia policy set --global --keep-latest 7 --keep-hourly 0 --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --keep-annual 0 |
how many snapshots to keep |
| Command | What it does |
|---|---|
sudo kopia policy set --global --keep-latest 14 --keep-hourly 0 --keep-daily 14 --keep-weekly 8 --keep-monthly 12 --keep-annual 0 |
how many snapshots to keep |
| Command | What it does |
|---|---|
sudo kopia policy set --global --keep-latest 48 --keep-hourly 48 --keep-daily 30 --keep-weekly 12 --keep-monthly 12 --keep-annual 0 |
how many snapshots to keep |
sudo sed -i 's/^OnCalendar=.*/OnCalendar=hourly/' /etc/systemd/system/kopia-backup.timer |
a backup every hour |
sudo systemctl daemon-reload && sudo systemctl restart kopia-backup.timer |
the timer with the new schedule |
With the active scenario, set the web1-backup check in Healthchecks.io to Period — 1 hour, Grace Time —
30 minutes. Kopia deletes extra snapshots itself after each backup, and scheduled storage maintenance frees
the space they used — Kopia runs that on its own too.
| Command | What it does |
|---|---|
sudo systemctl start kopia-backup.service |
a backup now, without waiting for the night |
| Command | What you should see |
|---|---|
systemctl list-timers kopia-backup.timer |
NEXT shows the next 03:00, as set in the timer’s OnCalendar; with the active scenario — the start of the next hour |
sudo journalctl -u kopia-backup.service -n 3 |
Finished kopia-backup.service, no Failed |
Healthchecks.io, the web1-backup check |
up, last ping just now |
A backup nobody has ever restored from may turn out empty because of one typo in a path. So delete a memo and bring it back from storage. Then let the server do the same on a schedule.
In memos, create a memo “Backup test”, run a backup with the stage 4 command, and delete the memo.
| Command | What it does |
|---|---|
cd /opt/memos && sudo docker compose stop |
stops memos: don’t swap the database under it |
sudo kopia snapshot restore /opt/memos/data/dump /tmp/memos-restore |
the dump from the latest snapshot, into a temporary directory |
sudo cp /tmp/memos-restore/memos.db data/memos_prod.db |
the restored database in place of the live one |
sudo rm -f data/memos_prod.db-wal data/memos_prod.db-shm |
the old database’s journal: SQLite would apply it to the restored one |
sudo docker compose start && sudo rm -r /tmp/memos-restore |
starts memos and removes the temporary directory |
The “Backup test” memo is back. The check script does the same without stopping memos: it downloads 5 % of
the files from storage, restores the dump into a temporary directory and asks SQLite whether the database is
intact and has memos in it. An empty database is intact too: that’s the dump you get when backup.sh points
at an old database path.
The script pings a separate check in Healthchecks.io: a ping to web1-backup would reset its timer, and
a missed nightly backup would go unnoticed. “Add Check”: name web1-verify, Period — 7 days, Grace Time — 1 day.
| Command | What it does |
|---|---|
echo '<ping-url>' | sudo tee /srv/secrets/kopia/hc-verify-url > /dev/null |
<ping-url> is the URL of the web1-verify check |
sudo chmod 400 /srv/secrets/kopia/hc-verify-url |
read-only, root only |
#!/bin/sh
# Проверка бэкапа: часть файлов скачиваем из хранилища, базу memos восстанавливаем и проверяем. Запускает kopia-verify.timer
set -u
# Своя проверка web1-verify в Healthchecks.io: пинг в web1-backup сбросил бы её счётчик, и пропуск ночного бэкапа
# остался бы незамеченным
ping=$(cat /srv/secrets/kopia/hc-verify-url)
curl -fsS -m 10 --retry 3 -o /dev/null "$ping/start"
dir=$(mktemp -d)
trap 'rm -rf "$dir"' EXIT
# Целая база ещё не значит, что в ней данные: дамп базы, которой нет по пути из backup.sh, — пустая база,
# и integrity_check отвечает ok
check() {
kopia snapshot verify --verify-files-percent=5 &&
kopia snapshot restore /opt/memos/data/dump "$dir" &&
test "$(sqlite3 "$dir/memos.db" 'PRAGMA integrity_check')" = ok &&
notes=$(sqlite3 "$dir/memos.db" 'SELECT count(*) FROM memo') &&
{ [ "$notes" -gt 0 ] || { echo 'В дампе memos нет заметок'; false; }; }
}
if out=$(check 2>&1); then
curl -fsS -m 10 --retry 3 -o /dev/null "$ping"
else
printf '%s\n' "$out" >&2
printf '%s\n' "$out" | tail -c 10000 | curl -fsS -m 10 --retry 3 -o /dev/null --data-binary @- "$ping/fail"
exit 1
fi
[Unit]
Description=Проверка восстановления из Kopia
Wants=network-online.target
After=network-online.target
[Service]
Type=oneshot
ExecStart=/opt/scripts/backup-verify.sh
Environment=HOME=/root
Nice=10
IOSchedulingClass=idle
[Unit]
Description=Проверка восстановления по расписанию
[Timer]
OnCalendar=Sun *-*-* 05:30
Persistent=true
[Install]
WantedBy=timers.target
| Command | What it does |
|---|---|
sudo chmod 700 /opt/scripts/backup-verify.sh |
the script runs, only root can read it |
sudo systemctl daemon-reload && sudo systemctl enable --now kopia-verify.timer |
a check once a week |
sudo systemctl start kopia-verify.service |
a check now; takes minutes — files download from storage |
With the minimal scenario the check runs monthly: set OnCalendar=*-*-01 05:30 in kopia-verify.timer,
then daemon-reload and restart kopia-verify.timer again, and give web1-verify a Period of 31 days.
If the server itself is gone, bring up a new one by level 1 with the same name, web1, and install Kopia
with the stage 1 commands. Connect to the storage with the password from your password manager. With option
A the new server has a key of its own: create it and known_hosts as in stage 2, and put the key on backup1
in place of the old one. With option B, go through sudo rclone config again.
| Command | What it does |
|---|---|
sudo kopia repository connect sftp --host=198.51.100.20 --username=kopia --keyfile=/srv/secrets/kopia/id_ed25519 --known-hosts=/srv/secrets/kopia/known_hosts --path=/srv/kopia/web1 |
option A: connects to the storage on backup1, asks for the password |
sudo kopia repository connect rclone --remote-path=gdrive:kopia-web1 |
option B: connects to the storage in Google Drive, asks for the password |
sudo kopia snapshot restore --skip-existing /srv/secrets /srv/secrets |
secrets back in place; --skip-existing leaves the new key alone: backup1 no longer accepts the old one |
sudo kopia snapshot restore /opt /opt |
the stacks with their data and dumps |
sudo cp /opt/memos/data/dump/memos.db /opt/memos/data/memos_prod.db |
the memos database from the dump |
Then bring the stacks up as on levels 2–4: the proxy network, then traefik, then the rest —
cd /opt/<stack> && sudo docker compose up -d. The level 6 recipe Moving to a new server brings up the base
and the stacks with one command each.
| Command | What you should see |
|---|---|
| memos in the browser | the “Backup test” memo is there |
systemctl list-timers 'kopia-*' |
two timers, both with NEXT filled in |
sudo journalctl -u kopia-verify.service -n 3 |
Finished kopia-verify.service, no Failed |
Healthchecks.io, the web1-verify check |
up, last ping just now |
If it doesn’t work
apt update says NO_PUBKEY or is not signed about packages.kopia.io
- Cause
- The key wasn’t written:
curldidn’t download it, andgpgsaved an empty file. - Fix
ls -l /etc/apt/keyrings/kopia-keyring.gpgshould show a size above zero. Zero — run the stage’s first command again.
Option A: kopia repository create sftp says unable to authenticate
- Cause
- backup1 doesn’t let the
kopiauser in: the key isn’t inauthorized_keys,.sshhas the wrong mode, or the user isn’t inAllowUsers. - Fix
- The log on backup1 names the reason:
sudo grep kopia /var/log/auth.log | tail -5. Anot listed in AllowUsersline means theAllowUserscommand was skipped.
Option A: the error knownhosts: key is unknown
- Cause
known_hostshas a different address:--hosthas a name whilessh-keyscanasked by address, or the other way round.- Fix
- Use the same value,
198.51.100.20, in--hostand inssh-keyscan, and writeknown_hostsagain.
Option B: kopia repository create rclone says executable file not found
- Cause
- Kopia starts rclone itself, and it isn’t on the server.
- Fix
sudo apt install -y rclone, thensudo rclone configfrom stage 2.
Option B: backups are slow, the log shows rateLimitExceeded
- Cause
- rclone talks to Google Drive with its shared
client_id, and everyone who didn’t change it shares that request limit. - Fix
- Create your own
client_idfollowing Making your own client_id in the rclone Google Drive docs and enter it insudo rclone config.
sqlite3 says unable to open database
- Cause
- The memos database has a different name: that happens when compose sets
MEMOS_DSNor demo mode is on. - Fix
sudo ls /opt/memos/datashows the*.dbfile name. Use it in the dump command, the--add-ignorerule andbackup.sh.
sudo kopia snapshot create works by hand, but from the timer it fails saying the repository isn’t connected
- Cause
- The service runs without
HOME: Kopia looks for its settings in~/.config/kopiaand finds nothing. - Fix
systemctl show kopia-backup.service -p Environmentshould printHOME=/root. If not, compare the unit with stage 4 and runsudo systemctl daemon-reload.
The web1-backup check in Healthchecks.io stays new
- Cause
- The ping doesn’t go out:
hc-urlhas a stray space, a quote or a different URL. - Fix
sudo sh -c 'curl -fsS "$(cat /srv/secrets/kopia/hc-url)"'should printOK. On an error, write the URL again as in stage 4.
After the restore memos says database disk image is malformed or shows the wrong memos
- Cause
memos_prod.db-waland-shmfrom the old database stayed next to the restored one: SQLite applied the old journal to the new database.- Fix
- Restore again: memos stopped,
memos.dbcopied once more, both journal files deleted, thenstart.
On a new server kopia snapshot restore /opt says there are no snapshots
- Cause
- Kopia finds snapshots by user and host name, and the new server has a different name.
- Fix
sudo kopia snapshot list --allshows the name the snapshots are under. Reconnect with--override-hostname=web1or set the name withsudo hostnamectl set-hostname web1.
How to remove it
sudo systemctl disable --now kopia-backup.timer kopia-verify.timer
sudo rm /etc/systemd/system/kopia-*.service /etc/systemd/system/kopia-*.timer
sudo rm /opt/scripts/backup.sh /opt/scripts/backup-verify.sh
sudo systemctl daemon-reload
sudo kopia repository disconnect
sudo rm -r /srv/secrets/kopia /opt/memos/data/dump
The storage stays where it was: on backup1 sudo deluser --remove-home kopia removes it, in Google Drive —
the kopia-web1 folder. Delete the web1-backup and web1-verify checks in Healthchecks.io too, or they send down.