basforge

Search titles, text and the “If it doesn’t work” cases.

    RU

    Deploy without SSH

    Make changes only in git on your computer: every 10 minutes the server pulls new commits on its own and restarts the stacks that changed, and a stack that fails to come up is rolled back together with its commit. Renovate sends new image versions as pull requests — merge them, and the server deploys them itself. A failed deploy or a silent one raises an alert in ntfy through Healthchecks.io.

    Start froma server after level 6, “Migration”: /opt is the infra-web1 repository on GitHub, its clone is on your computer, and Healthchecks.io has the ntfy integration.

    What you get

    computer ── git push ──▶ GitHub (infra-web1) ◀── git fetch every 10 minutes ── web1: deploy.sh
    deploy.sh   changed stacks — docker compose up -d --wait; a stack didn't come up — the commit is rolled back
    Renovate ── pull request with a new image version ──▶ GitHub: review, merge — the server deploys it
    Healthchecks.io   a heartbeat from every run; a failed deploy or 40 minutes of silence — alert in ntfy
    
    /opt/scripts/deploy.sh          the deploy script, in git
    /srv/secrets/deploy/hc-url      ping URL of the web1-deploy check
    • You’ll needA clone of infra-web1 on your computer — stage 3 of level 6
    • You’ll needA Healthchecks.io account and an ntfy topic from level 4
    • You’ll needThe GitHub account that holds infra-web1: Renovate is installed there

    Your values

    Filled into the commands and files of this and other recipes on the path; stored in this browser only.
    Domain
    example.orgplaceholder
    Server name
    web1placeholder

    Without the script the commands show placeholder values: replace them with yours.

    Steps

    The same recipe, short: commands from the stages, files as one line. For a second server and for those who already know what each command does.

    A deploy check in Healthchecks.io

    Why — Metrics, uptime checks and alerts

    A deploy that quietly stalled is worse than a manual one: changes pile up while the server runs the old version. So every script run sends a heartbeat, and a failure sends a fail ping with the reason. The check is separate from the backup one: they run on different schedules.

    In Healthchecks.io:

    • “Add Check”: name web1-deploy, Period — 10 minutes, Grace — 30 minutes. A short GitHub outage won’t become an alert, a stopped deploy will.
    • On the check’s page, under Notification Methods, turn on the ntfy integration from level 4.
    • Copy the ping URL — it looks like https://hc-ping.com/….
    Ping URL
    Command What it does
    sudo install -d -m 700 /srv/secrets/deploy directory for the deploy secret
    echo '<ping-url>' | sudo tee /srv/secrets/deploy/hc-url > /dev/null replace <ping-url> with the URL from Healthchecks.io
    sudo chmod 400 /srv/secrets/deploy/hc-url read-only, root only

    Do stages 1–3 in one go: the script sends the first ping.

    Done when
    Command What you should see
    sudo cat /srv/secrets/deploy/hc-url https://hc-ping.com/…
    Not like this — see “If it doesn’t work”, stage cases.

    The deploy script

    Why — Server configs in git

    The script fetches new commits from GitHub and merges them only by fast-forward: it doesn’t overwrite an uncommitted change on the server or a commit GitHub doesn’t have — it stops with an alert. It finds the changed stacks with git diff and brings them up with docker compose up -d --wait: the command waits until the containers start and pass their healthcheck. If a stack doesn’t come up, the files go back to the previous commit, and the touched stacks come up with them.

    If another stack file next to the compose file changed — traefik.yml, for example — the stack is recreated: a container doesn’t see a single-file mount after git replaces the file (Single-file mounts). The script doesn’t apply changes in host/ and scripts/: install system files by hand after git, as on level 6.

    /opt/scripts/deploy.shpulls commits, brings up changed stacks, rolls back on failure
    #!/bin/bash
    # Деплой из git: забирает новые коммиты infra-web1 и перезапускает изменённые стеки; стек не поднялся — откат
    # всего коммита и тревога через Healthchecks.io. Запускает deploy.timer раз в 10 минут
    set -u
    cd /opt
    
    ping=$(cat /srv/secrets/deploy/hc-url)
    failed=.git/deploy-failed       # коммит, который не поднялся: второй раз его не пробуем
    
    hc() { curl -fsS -m 10 --retry 3 -o /dev/null "$@"; }
    
    fail() {
      printf '%s\n' "$1" >&2
      printf '%s\n' "$1" | tail -c 10000 | hc --data-binary @- "$ping/fail"
      exit 1
    }
    
    # Стек — к файлам из git: compose-файл есть — поднимаем, нет — стек убрали из git, останавливаем (data/ остаётся)
    up() {
      local s=$1
      if [ ! -f "$s/docker-compose.yaml" ]; then
        docker compose -p "$s" down
      # Отдельный файл после git контейнер не видит: изменилось не только compose — пересоздаём
      elif git diff --name-only "$old" "$new" -- "$s" | grep -qvx "$s/docker-compose.yaml"; then
        (cd "$s" && docker compose up -d --wait --wait-timeout 180 --force-recreate)
      else
        (cd "$s" && docker compose up -d --wait --wait-timeout 180)
      fi
    }
    
    # Тело — в функции: bash читает скрипт по ходу, а git может заменить и этот файл
    main() {
      # GitHub не ответил — молчим: короткий сбой не тревога, а долгий Healthchecks.io заметит по тишине
      git fetch -q origin main || exit 0
      old=$(git rev-parse HEAD)
      new=$(git rev-parse origin/main)
      if [ "$old" = "$new" ]; then hc "$ping"; exit 0; fi
      if [ "$new" = "$(cat "$failed" 2>/dev/null)" ]; then fail "${new:0:7} не поднялся, ждём следующего коммита"; fi
    
      # Правку на сервере не затираем: её сначала коммитят и отправляют на GitHub
      dirty=$(git status --short --untracked-files=no)
      [ -z "$dirty" ] || fail "В /opt правка без коммита:
    $dirty"
      git merge-base --is-ancestor "$old" "$new" ||
        fail "На сервере коммиты, которых нет на GitHub: git -C /opt log --oneline origin/main..HEAD"
      git merge -q --ff-only "$new" || fail "git merge не прошёл"
    
      # Стек — первый каталог изменённого пути, у которого есть compose-файл до или после коммита.
      # host/ и scripts/ — не стеки: системные файлы после git ставим руками
      touched=
      for s in $(git diff --name-only "$old" "$new" | cut -d/ -f1 | sort -u); do
        git cat-file -e "$old:$s/docker-compose.yaml" 2>/dev/null ||
          git cat-file -e "$new:$s/docker-compose.yaml" 2>/dev/null || continue
        touched="$touched $s"
        if ! out=$(up "$s" 2>&1); then
          # reset --hard меняет только файлы из git: данные, .env и acme.json в git не входят (не git clean)
          git reset -q --hard "$old"
          for d in $touched; do up "$d" >/dev/null 2>&1; done
          echo "$new" > "$failed"
          fail "$s не поднялся на ${new:0:7}, вернули ${old:0:7}:
    $out"
        fi
      done
    
      rm -f "$failed"
      echo "${new:0:7}:${touched:- без стеков}" | hc --data-binary @- "$ping"
    }
    
    main
    First run
    Command What it does
    sudo chmod 700 /opt/scripts/deploy.sh only root runs and reads it
    sudo git -C /opt status --short empty except ?? scripts/deploy.sh: no uncommitted changes on the server
    sudo /opt/scripts/deploy.sh && echo ok no new commits — just a heartbeat to Healthchecks.io
    Done when
    Command What you should see
    sudo /opt/scripts/deploy.sh && echo ok ok
    Healthchecks.io, the web1-deploy check up, last ping — just now
    Not like this — see “If it doesn’t work”, stage cases.

    A timer every 10 minutes

    A systemd timer runs the script: a new run won’t overlap the previous one, and the output goes to the service journal. Push the script and the units to GitHub from the server — for the last time: from here on, edit only on your computer.

    /etc/systemd/system/deploy.service
    [Unit]
    Description=Деплой стеков /opt из git
    Wants=network-online.target
    After=network-online.target docker.service
    
    [Service]
    Type=oneshot
    ExecStart=/opt/scripts/deploy.sh
    # git ищет настройки в ~: без User= systemd не задаёт HOME
    Environment=HOME=/root
    /etc/systemd/system/deploy.timer
    [Unit]
    Description=Деплой из git каждые 10 минут
    
    [Timer]
    OnCalendar=*:0/10
    
    [Install]
    WantedBy=timers.target
    Into git, then enable
    Command What it does
    cd / && sudo cp --parents etc/systemd/system/deploy.service etc/systemd/system/deploy.timer /opt/host/ copies of the units — for bootstrap.sh, like the other system files
    sudo git -C /opt add scripts/deploy.sh host/etc/systemd/system the script and the units — into git
    sudo git -C /opt commit -m "web1: deploy from git on a timer" && sudo git -C /opt push to GitHub; before enabling the timer, or it will find an unpushed commit
    sudo systemctl daemon-reload && sudo systemctl enable --now deploy.timer the timer is on and survives a reboot

    On your computer — git pull: these commits belong there too.

    From now on, an urgent fix on the server means an immediate commit and push from there, otherwise the deploy stops with the “uncommitted change” alert. During the move (stage 6 of level 6), enable deploy.timer together with the backup timers. Not during a drill: the temporary server would send heartbeats in place of the live one.

    Done when
    Command What you should see
    systemctl list-timers deploy.timer NEXT no more than 10 minutes away
    journalctl -u deploy.service -n 5 --no-pager after the next run — Finished deploy.service, no errors
    Not like this — see “If it doesn’t work”, stage cases.

    Test it for real: a change and an error from your computer

    The rest happens on your computer, in the infra-web1 clone. First, a change you can see from the outside: whoami prints its name if one is set. In whoami/docker-compose.yaml, add to the whoami service:

        environment:
          WHOAMI_NAME: from-git
    A change
    Command What it does
    git commit -am "whoami: name from git" && git push the change goes to GitHub; the server takes it from there
    curl -s https://whoami.example.org | grep Name in 10 minutes: Name: from-git

    Now an error: in the same file, set the image to traefik/whoami:no-such-tag — that tag doesn’t exist and can’t be pulled.

    An error and a rollback
    Command What it does
    git commit -am "whoami: nonexistent tag" && git push a commit that won’t come up
    curl -s https://whoami.example.org | grep Name in 10 minutes: a web1-deploy alert in ntfy, while whoami still answers Name: from-git
    git revert --no-edit HEAD && git push the fix as a new commit; in 10 minutes the check is up again

    Healthchecks.io shows what didn’t come up: the check’s log has the body of the fail ping with the docker compose output. On the server, the same is in journalctl -u deploy.service.

    Done when
    Command What you should see
    curl -s https://whoami.example.org | grep Name Name: from-git
    Healthchecks.io, the web1-deploy check up; the log shows down with the reason and up after the revert
    Not like this — see “If it doesn’t work”, stage cases.

    Renovate: new image versions

    Renovate finds the images in compose files and opens a pull request when a new version comes out: tag v3.8 instead of v3.7, or a new build of the same stable. Review what changed, merge — and the same script deploys it. Every update shows in the history and rolls back with git revert.

    Install the app: github.com/apps/renovate — Install, “Only select repositories” — infra-web1. A few minutes later Renovate opens a “Configure Renovate” pull request with its own renovate.json and a list of the images it found. Merge it, run git pull on your computer, and replace the file with this one:

    /opt/renovate.jsonin the repository root; edit it on your computer — the server gets it on its own
    {
      "$schema": "https://docs.renovatebot.com/renovate-schema.json",
      "extends": ["config:recommended", "docker:pinDigests"],
      "timezone": "Europe/Berlin",
      "schedule": ["before 6am on monday"],
      "packageRules": [
        {
          "description": "Новые сборки тех же тегов — одним PR в неделю",
          "matchUpdateTypes": ["digest"],
          "groupName": "digests"
        }
      ]
    }

    docker:pinDigests adds a digest to the tag — stable@sha256:…. This way the image on the server is exactly the one in git: a new build under the old tag arrives as a pull request and can’t quietly replace the image on pull. New digests are grouped into one pull request a week: the window is Monday morning.

    Renovate config
    Command What it does
    git commit -am "renovate: digests, Monday window" && git push the config goes to GitHub; Renovate reads it from there

    The first one is “Pin dependencies”: digests for all images at once. If an image on the server hasn’t been pulled in a while, the digest will be newer, and after the merge the stacks come up on fresh images. Merge it when you have 10 minutes to watch the services.

    Done when
    Command What you should see
    GitHub, Issues in infra-web1 Dependency Dashboard: images from the compose files of every stack
    sudo grep -h 'image:' /opt/*/docker-compose.yaml after merging “Pin dependencies”: every image has @sha256:…
    Not like this — see “If it doesn’t work”, stage cases.

    If it doesn’t work

    Stage 2

    deploy.sh says cat: /srv/secrets/deploy/hc-url: No such file or directory

    Cause
    Stage 1 was skipped: the server has no ping URL.
    Fix
    Run the stage 1 commands, then the script again.
    Stage 3

    The web1-deploy check in Healthchecks.io stays new or goes down while the server is alive

    Cause
    The timer isn’t enabled, or git fetch as root can’t reach GitHub: then the script stays silent, and Healthchecks.io sees silence.
    Fix
    systemctl list-timers deploy.timer — the timer is there; sudo git -C /opt fetch — the error shows right away. Permission denied (publickey) — check the key in Deploy keys, as in stage 3 of level 6.
    Stage 3

    An alert, and the check’s log says В /opt правка без коммита (an uncommitted change in /opt)

    Cause
    A file from git was edited on the server. The deploy doesn’t overwrite it and stays stopped until the change is dealt with.
    Fix
    sudo git -C /opt diff shows the change. Need it — commit and push from the server; don’t — sudo git -C /opt checkout -- <file>.
    Stage 3

    An alert, and the check’s log says На сервере коммиты, которых нет на GitHub (commits on the server that GitHub doesn’t have)

    Cause
    A commit was made on the server but not pushed, or its history diverged from GitHub.
    Fix
    sudo git -C /opt push; if rejected — sudo git -C /opt pull --rebase && sudo git -C /opt push.
    Stage 4

    git push went through, and 20 minutes later nothing changed on the server

    Cause
    The script didn’t get to the stacks: the service journal says why.
    Fix
    journalctl -u deploy.service -n 30 --no-pager. The branch on your computer is main: the server pulls no other.
    Stage 4

    The alert keeps repeating, and the log says <commit> не поднялся, ждём следующего коммита (didn’t come up, waiting for the next commit)

    Cause
    The script remembers the commit that failed and doesn’t try it a second time.
    Fix
    Fix it in git and push a new commit or git revert. If the failure was temporary — the image registry didn’t respond — sudo rm /opt/.git/deploy-failed: the next run tries again.
    Stage 5

    The Dependency Dashboard doesn’t list the images from the compose files

    Cause
    Renovate has no access to infra-web1, or the compose file isn’t named the way it looks for.
    Fix
    On GitHub — Settings, Applications, Renovate, Configure: infra-web1 is in the repository list. The file name is docker-compose.yaml or compose.yaml.
    Stage 5

    Monday has passed, and there are no pull requests with new digests

    Cause
    Renovate creates them only inside the schedule window, and there may have been no new builds that week.
    Fix
    The Dependency Dashboard lists what’s waiting for the window; ticking a line creates the pull request right away.

    How to remove it

    The stacks stay as they are; only the deploy and Renovate go away.

    sudo systemctl disable --now deploy.timer
    sudo rm /etc/systemd/system/deploy.service /etc/systemd/system/deploy.timer && sudo systemctl daemon-reload
    sudo rm -r /srv/secrets/deploy
    sudo git -C /opt rm -q scripts/deploy.sh host/etc/systemd/system/deploy.service host/etc/systemd/system/deploy.timer renovate.json
    sudo git -C /opt commit -m "web1: no deploy from git" && sudo git -C /opt push

    Delete the web1-deploy check in Healthchecks.io, or 40 minutes later it sends down. Renovate — on GitHub: Settings, Applications, Renovate, Uninstall.

    Next on the path