A deploy that quietly stalled is worse than a manual one: changes pile up while the server runs the old version. So
every script run sends a heartbeat, and a failure sends a fail ping with the reason. The check is separate from
the backup one: they run on different schedules.
In Healthchecks.io:
- “Add Check”: name
web1-deploy, Period — 10 minutes, Grace — 30 minutes. A short GitHub outage won’t become an alert, a stopped deploy will. - On the check’s page, under Notification Methods, turn on the ntfy integration from level 4.
- Copy the ping URL — it looks like
https://hc-ping.com/….
| Command | What it does |
|---|---|
sudo install -d -m 700 /srv/secrets/deploy |
directory for the deploy secret |
echo '<ping-url>' | sudo tee /srv/secrets/deploy/hc-url > /dev/null |
replace <ping-url> with the URL from Healthchecks.io |
sudo chmod 400 /srv/secrets/deploy/hc-url |
read-only, root only |
Do stages 1–3 in one go: the script sends the first ping.
| Command | What you should see |
|---|---|
sudo cat /srv/secrets/deploy/hc-url |
https://hc-ping.com/… |
The script fetches new commits from GitHub and merges them only by fast-forward: it doesn’t overwrite an uncommitted
change on the server or a commit GitHub doesn’t have — it stops with an alert. It finds the changed stacks with
git diff and brings them up with docker compose up -d --wait: the command waits until the containers start and
pass their healthcheck. If a stack doesn’t come up, the files go back to the previous commit, and the touched stacks
come up with them.
If another stack file next to the compose file changed — traefik.yml, for example — the stack is recreated: a
container doesn’t see a single-file mount after git replaces the file (Single-file mounts). The script doesn’t
apply changes in host/ and scripts/: install system files by hand after git, as on level 6.
#!/bin/bash
# Деплой из git: забирает новые коммиты infra-web1 и перезапускает изменённые стеки; стек не поднялся — откат
# всего коммита и тревога через Healthchecks.io. Запускает deploy.timer раз в 10 минут
set -u
cd /opt
ping=$(cat /srv/secrets/deploy/hc-url)
failed=.git/deploy-failed # коммит, который не поднялся: второй раз его не пробуем
hc() { curl -fsS -m 10 --retry 3 -o /dev/null "$@"; }
fail() {
printf '%s\n' "$1" >&2
printf '%s\n' "$1" | tail -c 10000 | hc --data-binary @- "$ping/fail"
exit 1
}
# Стек — к файлам из git: compose-файл есть — поднимаем, нет — стек убрали из git, останавливаем (data/ остаётся)
up() {
local s=$1
if [ ! -f "$s/docker-compose.yaml" ]; then
docker compose -p "$s" down
# Отдельный файл после git контейнер не видит: изменилось не только compose — пересоздаём
elif git diff --name-only "$old" "$new" -- "$s" | grep -qvx "$s/docker-compose.yaml"; then
(cd "$s" && docker compose up -d --wait --wait-timeout 180 --force-recreate)
else
(cd "$s" && docker compose up -d --wait --wait-timeout 180)
fi
}
# Тело — в функции: bash читает скрипт по ходу, а git может заменить и этот файл
main() {
# GitHub не ответил — молчим: короткий сбой не тревога, а долгий Healthchecks.io заметит по тишине
git fetch -q origin main || exit 0
old=$(git rev-parse HEAD)
new=$(git rev-parse origin/main)
if [ "$old" = "$new" ]; then hc "$ping"; exit 0; fi
if [ "$new" = "$(cat "$failed" 2>/dev/null)" ]; then fail "${new:0:7} не поднялся, ждём следующего коммита"; fi
# Правку на сервере не затираем: её сначала коммитят и отправляют на GitHub
dirty=$(git status --short --untracked-files=no)
[ -z "$dirty" ] || fail "В /opt правка без коммита:
$dirty"
git merge-base --is-ancestor "$old" "$new" ||
fail "На сервере коммиты, которых нет на GitHub: git -C /opt log --oneline origin/main..HEAD"
git merge -q --ff-only "$new" || fail "git merge не прошёл"
# Стек — первый каталог изменённого пути, у которого есть compose-файл до или после коммита.
# host/ и scripts/ — не стеки: системные файлы после git ставим руками
touched=
for s in $(git diff --name-only "$old" "$new" | cut -d/ -f1 | sort -u); do
git cat-file -e "$old:$s/docker-compose.yaml" 2>/dev/null ||
git cat-file -e "$new:$s/docker-compose.yaml" 2>/dev/null || continue
touched="$touched $s"
if ! out=$(up "$s" 2>&1); then
# reset --hard меняет только файлы из git: данные, .env и acme.json в git не входят (не git clean)
git reset -q --hard "$old"
for d in $touched; do up "$d" >/dev/null 2>&1; done
echo "$new" > "$failed"
fail "$s не поднялся на ${new:0:7}, вернули ${old:0:7}:
$out"
fi
done
rm -f "$failed"
echo "${new:0:7}:${touched:- без стеков}" | hc --data-binary @- "$ping"
}
main
| Command | What it does |
|---|---|
sudo chmod 700 /opt/scripts/deploy.sh |
only root runs and reads it |
sudo git -C /opt status --short |
empty except ?? scripts/deploy.sh: no uncommitted changes on the server |
sudo /opt/scripts/deploy.sh && echo ok |
no new commits — just a heartbeat to Healthchecks.io |
| Command | What you should see |
|---|---|
sudo /opt/scripts/deploy.sh && echo ok |
ok |
Healthchecks.io, the web1-deploy check |
up, last ping — just now |
A timer every 10 minutes
doneA systemd timer runs the script: a new run won’t overlap the previous one, and the output goes to the service journal. Push the script and the units to GitHub from the server — for the last time: from here on, edit only on your computer.
[Unit]
Description=Деплой стеков /opt из git
Wants=network-online.target
After=network-online.target docker.service
[Service]
Type=oneshot
ExecStart=/opt/scripts/deploy.sh
# git ищет настройки в ~: без User= systemd не задаёт HOME
Environment=HOME=/root
[Unit]
Description=Деплой из git каждые 10 минут
[Timer]
OnCalendar=*:0/10
[Install]
WantedBy=timers.target
| Command | What it does |
|---|---|
cd / && sudo cp --parents etc/systemd/system/deploy.service etc/systemd/system/deploy.timer /opt/host/ |
copies of the units — for bootstrap.sh, like the other system files |
sudo git -C /opt add scripts/deploy.sh host/etc/systemd/system |
the script and the units — into git |
sudo git -C /opt commit -m "web1: deploy from git on a timer" && sudo git -C /opt push |
to GitHub; before enabling the timer, or it will find an unpushed commit |
sudo systemctl daemon-reload && sudo systemctl enable --now deploy.timer |
the timer is on and survives a reboot |
On your computer — git pull: these commits belong there too.
From now on, an urgent fix on the server means an immediate commit and push from there, otherwise the deploy stops
with the “uncommitted change” alert. During the move (stage 6 of level 6), enable deploy.timer together with the
backup timers. Not during a drill: the temporary server would send heartbeats in place of the live one.
| Command | What you should see |
|---|---|
systemctl list-timers deploy.timer |
NEXT no more than 10 minutes away |
journalctl -u deploy.service -n 5 --no-pager |
after the next run — Finished deploy.service, no errors |
Test it for real: a change and an error from your computer
doneThe rest happens on your computer, in the infra-web1 clone. First, a change you can see from the outside: whoami
prints its name if one is set. In whoami/docker-compose.yaml, add to the whoami service:
environment:
WHOAMI_NAME: from-git| Command | What it does |
|---|---|
git commit -am "whoami: name from git" && git push |
the change goes to GitHub; the server takes it from there |
curl -s https://whoami.example.org | grep Name |
in 10 minutes: Name: from-git |
Now an error: in the same file, set the image to traefik/whoami:no-such-tag — that tag doesn’t exist and can’t be
pulled.
| Command | What it does |
|---|---|
git commit -am "whoami: nonexistent tag" && git push |
a commit that won’t come up |
curl -s https://whoami.example.org | grep Name |
in 10 minutes: a web1-deploy alert in ntfy, while whoami still answers Name: from-git |
git revert --no-edit HEAD && git push |
the fix as a new commit; in 10 minutes the check is up again |
Healthchecks.io shows what didn’t come up: the check’s log has the body of the fail ping with the docker compose
output. On the server, the same is in journalctl -u deploy.service.
| Command | What you should see |
|---|---|
curl -s https://whoami.example.org | grep Name |
Name: from-git |
Healthchecks.io, the web1-deploy check |
up; the log shows down with the reason and up after the revert |
Renovate: new image versions
doneRenovate finds the images in compose files and opens a pull request when a new version comes out: tag v3.8
instead of v3.7, or a new build of the same stable. Review what changed, merge — and the same script deploys it.
Every update shows in the history and rolls back with git revert.
Install the app: github.com/apps/renovate — Install, “Only select repositories” — infra-web1. A few minutes later
Renovate opens a “Configure Renovate” pull request with its own renovate.json and a list of the images it found.
Merge it, run git pull on your computer, and replace the file with this one:
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": ["config:recommended", "docker:pinDigests"],
"timezone": "Europe/Berlin",
"schedule": ["before 6am on monday"],
"packageRules": [
{
"description": "Новые сборки тех же тегов — одним PR в неделю",
"matchUpdateTypes": ["digest"],
"groupName": "digests"
}
]
}
docker:pinDigests adds a digest to the tag — stable@sha256:…. This way the image on the server is exactly the one
in git: a new build under the old tag arrives as a pull request and can’t quietly replace the image on pull. New
digests are grouped into one pull request a week: the window is Monday morning.
| Command | What it does |
|---|---|
git commit -am "renovate: digests, Monday window" && git push |
the config goes to GitHub; Renovate reads it from there |
The first one is “Pin dependencies”: digests for all images at once. If an image on the server hasn’t been pulled in a while, the digest will be newer, and after the merge the stacks come up on fresh images. Merge it when you have 10 minutes to watch the services.
| Command | What you should see |
|---|---|
GitHub, Issues in infra-web1 |
Dependency Dashboard: images from the compose files of every stack |
sudo grep -h 'image:' /opt/*/docker-compose.yaml |
after merging “Pin dependencies”: every image has @sha256:… |
If it doesn’t work
deploy.sh says cat: /srv/secrets/deploy/hc-url: No such file or directory
- Cause
- Stage 1 was skipped: the server has no ping URL.
- Fix
- Run the stage 1 commands, then the script again.
The web1-deploy check in Healthchecks.io stays new or goes down while the server is alive
- Cause
- The timer isn’t enabled, or
git fetchas root can’t reach GitHub: then the script stays silent, and Healthchecks.io sees silence. - Fix
systemctl list-timers deploy.timer— the timer is there;sudo git -C /opt fetch— the error shows right away.Permission denied (publickey)— check the key in Deploy keys, as in stage 3 of level 6.
An alert, and the check’s log says В /opt правка без коммита (an uncommitted change in /opt)
- Cause
- A file from git was edited on the server. The deploy doesn’t overwrite it and stays stopped until the change is dealt with.
- Fix
sudo git -C /opt diffshows the change. Need it — commit andpushfrom the server; don’t —sudo git -C /opt checkout -- <file>.
An alert, and the check’s log says На сервере коммиты, которых нет на GitHub (commits on the server that GitHub doesn’t have)
- Cause
- A commit was made on the server but not pushed, or its history diverged from GitHub.
- Fix
sudo git -C /opt push; if rejected —sudo git -C /opt pull --rebase && sudo git -C /opt push.
git push went through, and 20 minutes later nothing changed on the server
- Cause
- The script didn’t get to the stacks: the service journal says why.
- Fix
journalctl -u deploy.service -n 30 --no-pager. The branch on your computer ismain: the server pulls no other.
The alert keeps repeating, and the log says <commit> не поднялся, ждём следующего коммита (didn’t come up, waiting for the next commit)
- Cause
- The script remembers the commit that failed and doesn’t try it a second time.
- Fix
- Fix it in git and push a new commit or
git revert. If the failure was temporary — the image registry didn’t respond —sudo rm /opt/.git/deploy-failed: the next run tries again.
The Dependency Dashboard doesn’t list the images from the compose files
- Cause
- Renovate has no access to
infra-web1, or the compose file isn’t named the way it looks for. - Fix
- On GitHub — Settings, Applications, Renovate, Configure:
infra-web1is in the repository list. The file name isdocker-compose.yamlorcompose.yaml.
Monday has passed, and there are no pull requests with new digests
- Cause
- Renovate creates them only inside the
schedulewindow, and there may have been no new builds that week. - Fix
- The Dependency Dashboard lists what’s waiting for the window; ticking a line creates the pull request right away.
How to remove it
The stacks stay as they are; only the deploy and Renovate go away.
sudo systemctl disable --now deploy.timer
sudo rm /etc/systemd/system/deploy.service /etc/systemd/system/deploy.timer && sudo systemctl daemon-reload
sudo rm -r /srv/secrets/deploy
sudo git -C /opt rm -q scripts/deploy.sh host/etc/systemd/system/deploy.service host/etc/systemd/system/deploy.timer renovate.json
sudo git -C /opt commit -m "web1: no deploy from git" && sudo git -C /opt push
Delete the web1-deploy check in Healthchecks.io, or 40 minutes later it sends down. Renovate — on GitHub:
Settings, Applications, Renovate, Uninstall.