basforge

Search titles, text and the “If it doesn’t work” cases.

    RU

    Metrics, uptime checks and alerts

    Beszel shows what the server and its containers are busy with, Gatus opens your services over HTTPS once a minute, and Healthchecks.io waits outside for a heartbeat from the server: if it goes quiet, the server is down. All three send their alerts to your phone through ntfy.

    Start froma server after level 3, “Services”: Traefik gets certificates, memos opens by name, the proxy network exists.

    What you get

    Beszel agent ── host and containers ──▶ Beszel hub        graphs and thresholds, beszel.example.org
    Gatus ── every minute over HTTPS ─────▶ memos, beszel     status code and certificate expiry
    Beszel hub ── heartbeat every minute ─▶ Healthchecks.io   outside: no heartbeat for 6 minutes — alert
    alerts from all three ────────────────▶ ntfy.sh ──▶ phone
    /opt/beszel/               hub and agent: compose, .env, data/
    /opt/gatus/                checks: compose, .env, config/, data/
    /srv/secrets/ntfy/topic    the ntfy topic name
    • You’ll needA phone with the ntfy app (Android or iOS)
    • You’ll needAn account on healthchecks.io: the free plan is enough, it has 20 checks
    • You’ll needAn A record beszel.example.org pointing to the server address 203.0.113.10, like memos on level 3

    Your values

    Filled into the commands and files of this and other recipes on the path; stored in this browser only.
    Email
    [email protected]placeholder
    Domain
    example.orgplaceholder
    Server address
    203.0.113.10placeholder
    User
    example-userplaceholder
    Server name
    web1placeholder

    Without the script the commands show placeholder values: replace them with yours.

    Steps

    The same recipe, short: commands from the stages, files as one line. For a second server and for those who already know what each command does.

    An ntfy topic and your phone

    ntfy.sh is a public notification server with no sign-up: anyone who knows a topic name can read it and post to it. So the name is a random string, and you keep it as a secret. Your own ntfy server on the same VPS won’t do: when the server goes down, it goes down too, and the alert never arrives.

    Topic
    Command What it does
    sudo install -d -m 700 /srv/secrets/ntfy the secret’s directory, only root can enter it
    openssl rand -hex 16 | sudo tee /srv/secrets/ntfy/topic the topic name, 32 characters; printed so you can subscribe to it from your phone
    sudo chmod 400 /srv/secrets/ntfy/topic read-only, root only

    In the ntfy app on your phone, subscribe to the topic: the name is the line from the output, keep the server as ntfy.sh.

    Check
    Command What it does
    curl -s -o /dev/null -d 'Test from web1' ntfy.sh/$(sudo cat /srv/secrets/ntfy/topic) posts a notification to the topic
    Done when
    Command What you should see
    curl -s -o /dev/null -d 'Test from web1' ntfy.sh/$(sudo cat /srv/secrets/ntfy/topic) “Test from web1” on your phone within seconds
    Not like this — see “If it doesn’t work”, stage cases.

    Beszel: the hub and the agent

    The hub is a page with graphs at beszel.example.org, the agent collects metrics on the host. The agent talks to the hub through a socket in a shared directory: it has no TCP port.

    The name beszel.example.org lands in the public Certificate Transparency logs the moment Traefik gets its certificate, and scanners show up within minutes. That’s why the hub creates the user itself, from .env, on the first start: it never has a “Create account” page open to everyone.

    Directory and password
    Command What it does
    sudo mkdir -p /opt/beszel the stack directory
    sudo install -m 600 /dev/null /opt/beszel/.env an empty .env with mode 600 from the start: secrets go into a file that’s already closed
    echo "BESZEL_PASSWORD=$(openssl rand -hex 16)" | sudo tee -a /opt/beszel/.env the Beszel login password; the line is printed once — save the password in your password manager
    /opt/beszel/docker-compose.yamlthe hub behind Traefik and the agent on the host; the email is the Beszel login
    services:
      beszel:
        image: henrygd/beszel:latest
        container_name: beszel
        restart: unless-stopped
        environment:
          APP_URL: https://beszel.example.org
          USER_EMAIL: [email protected]
          USER_PASSWORD: ${BESZEL_PASSWORD}
          HEARTBEAT_URL: ${HEARTBEAT_URL:-}
        ports:
          - "127.0.0.1:8090:8090"
        volumes:
          - ./data/hub:/beszel_data
          - ./data/socket:/beszel_socket
        labels:
          - "traefik.enable=true"
          - "traefik.http.routers.beszel.rule=Host(`beszel.example.org`)"
          - "traefik.http.routers.beszel.entrypoints=https"
          - "traefik.http.services.beszel.loadbalancer.server.port=8090"
        networks:
          - proxy
    
      beszel-agent:
        image: henrygd/beszel-agent:latest
        container_name: beszel-agent
        restart: unless-stopped
        network_mode: host
        environment:
          LISTEN: /beszel_socket/beszel.sock
          HUB_URL: http://localhost:8090
          KEY: ${BESZEL_KEY:-}
          TOKEN: ${BESZEL_TOKEN:-}
        volumes:
          - ./data/agent:/var/lib/beszel-agent
          - ./data/socket:/beszel_socket
          - /var/run/docker.sock:/var/run/docker.sock:ro
    
    networks:
      proxy:
        external: true

    The hub is also published on 127.0.0.1:8090 (A host-only port goes on 127.0.0.1): the agent reaches it there, bypassing Traefik. If Traefik goes down, metrics keep coming in. The agent needs the Docker socket for container metrics, and that’s root-level access (docker.sock).

    Start the hub
    Command What it does
    cd /opt/beszel && sudo docker compose up -d beszel starts only the hub: there’s no key for the agent yet

    Open https://beszel.example.org and sign in: email [email protected], password from .env. Click “Add System”: name web1, “Host / IP” /beszel_socket/beszel.sock. Keep the dialog open: it shows the “Public Key” and the “Token” that go into .env.

    Agent key
    Command What it does
    printf 'BESZEL_KEY="%s"\nBESZEL_TOKEN=%s\n' '<key>' '<token>' | sudo tee -a /opt/beszel/.env > /dev/null appends the key and the token; replace <key> and <token> with the strings from the dialog
    cd /opt/beszel && sudo docker compose up -d starts the agent

    Click “Add System” in the Beszel dialog. Then two places in the UI:

    • Settings, Notifications, “Add URL”: ntfy://ntfy.sh/<topic>, the topic from sudo cat /srv/secrets/ntfy/topic. “Test URL” sends a test notification to your phone, then “Save Settings”.
    • The bell in the web1 row: turn on “Status” for 2 minutes, “CPU Usage”, “Memory Usage” and “Disk Usage”. Beszel suggests 80% for 10 minutes — keep that. “Status” here catches a dead agent: the hub lives on the same server, and when the whole server is down, nobody is left to say so. Stage 4 takes care of that.
    Done when
    Command What you should see
    sudo docker ps --format '{{.Names}} {{.Status}}' | grep beszel beszel Up … and beszel-agent Up …
    sudo ss -tlnH | grep 8090 one line, 127.0.0.1:8090
    https://beszel.example.org web1 is green, the traefik, memos and crowdsec containers have graphs
    Not like this — see “If it doesn’t work”, stage cases.

    Gatus: services over HTTPS

    Gatus opens URLs the way a browser does: by name, through Traefik, checking the certificate. The checks live in a file, and on the Migration level git picks them up along with the stack.

    Directories and topic
    Command What it does
    sudo mkdir -p /opt/gatus/config /opt/gatus/data the stack directories: config and check history
    sudo install -m 600 /dev/null /opt/gatus/.env an empty .env with mode 600
    echo "NTFY_TOPIC=$(sudo cat /srv/secrets/ntfy/topic)" | sudo tee -a /opt/gatus/.env > /dev/null the ntfy topic for Gatus, from the stage 1 file
    /opt/gatus/config/config.yamlwhat to check and where to send the alert
    storage:
      type: sqlite
      path: /data/data.db
    
    alerting:
      ntfy:
        topic: "${NTFY_TOPIC}"
        default-alert:
          failure-threshold: 3
          success-threshold: 2
          send-on-resolved: true
    
    endpoints:
      - name: memos
        url: https://memos.example.org
        interval: 60s
        conditions:
          - "[STATUS] == 200"
          - "[CERTIFICATE_EXPIRATION] > 336h"
        alerts:
          - type: ntfy
    
      - name: beszel
        url: https://beszel.example.org
        interval: 60s
        conditions:
          - "[STATUS] == 200"
          - "[CERTIFICATE_EXPIRATION] > 336h"
        alerts:
          - type: ntfy

    An alert goes out after three failures in a row: with a check every minute, that’s three minutes, so a container restart of half a minute doesn’t trigger it. send-on-resolved also sends “back up” after two successful checks. Traefik renews a certificate 30 days before it expires, so > 336h (14 days) fails only if renewal has been failing for more than two weeks.

    /opt/gatus/docker-compose.yamlthe Gatus page is only on 127.0.0.1:8081
    services:
      gatus:
        image: twinproduction/gatus:stable
        container_name: gatus
        restart: unless-stopped
        environment:
          NTFY_TOPIC: ${NTFY_TOPIC}
        volumes:
          - ./config:/config:ro
          - ./data:/data
        ports:
          - "127.0.0.1:8081:8080"

    Mount the config/ directory, not the single file: after sed -i or git pull, a file mounted on its own still looks old to the container (Single-file mounts).

    Start
    Command What it does
    cd /opt/gatus && sudo docker compose up -d starts Gatus
    sudo docker logs gatus 2>&1 | tail -5 the last lines of the log
    The page, from your computer
    Command What it does
    ssh -N -L 8081:127.0.0.1:8081 [email protected] a tunnel to the Gatus page while the window is open (SSH tunnels)

    Open http://localhost:8081 in your browser.

    Done when
    Command What you should see
    curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8081 200
    http://localhost:8081 through the tunnel memos and beszel are green
    Not like this — see “If it doesn’t work”, stage cases.

    A heartbeat from outside: Healthchecks.io

    A server can’t report that it’s down: Beszel and Gatus go quiet along with it. So the watchdog sits outside. The Beszel hub sends it a heartbeat every minute (HEARTBEAT_URL), and when the heartbeat is missing for longer than the limit, Healthchecks.io sends the alert.

    In Healthchecks.io:

    • On the Integrations tab, add ntfy: server https://ntfy.sh, the topic from sudo cat /srv/secrets/ntfy/topic.
    • “Add Check”: name web1, Period 1 minute, Grace Time 5 minutes. Copy the ping URL, https://hc-ping.com/…, from the check’s page.

    The five-minute margin covers the reboot after updates at 4:00 (level 1): without it, every such night would raise an alert.

    Heartbeat
    Command What it does
    echo "HEARTBEAT_URL=<ping-url>" | sudo tee -a /opt/beszel/.env > /dev/null the ping URL goes into the hub’s .env; replace <ping-url> with the URL from Healthchecks.io
    cd /opt/beszel && sudo docker compose up -d recreates the hub with the new variable

    The heartbeat confirms that the server is alive and can reach the internet. It doesn’t check whether your sites open from outside: Gatus reaches them from the server itself. For that you need an external HTTP check service.

    Done when
    Command What you should see
    Beszel, Settings, Heartbeat: “Test heartbeat” “Heartbeat sent successfully”
    Healthchecks.io, the web1 check up, last ping less than a minute ago
    Not like this — see “If it doesn’t work”, stage cases.

    Test the alerts by action

    Why — Test your server’s protection by action

    An alert that has never fired may be silent because of one wrong letter in the topic. So take down what it watches and wait for the notification.

    A service goes down
    Command What it does
    cd /opt/memos && sudo docker compose stop stops memos; Traefik answers 404 for its name
    sudo docker compose start after 5 minutes, once the alert has arrived, starts memos again
    The server goes quiet
    Command What it does
    cd /opt/beszel && sudo docker compose stop beszel the hub goes quiet, no heartbeat — to Healthchecks.io the server is down
    sudo docker compose start beszel after 7 minutes, once both alerts have arrived, starts the hub again
    Done when
    Command What you should see
    cd /opt/memos && sudo docker compose stop a Gatus alert about memos within 3–4 minutes; after start, a note that it answers again
    cd /opt/beszel && sudo docker compose stop beszel a Gatus alert about beszel within 3–4 minutes, down from Healthchecks.io around the sixth minute; after start, up
    Not like this — see “If it doesn’t work”, stage cases.

    If it doesn’t work

    Stage 1

    curl went through, but the phone stays silent

    Cause
    The app is subscribed to a different topic — one letter off — or to a different server.
    Fix
    Compare the topic in the app with sudo cat /srv/secrets/ntfy/topic; the server is ntfy.sh.
    Stage 2

    Beszel rejects the password from .env

    Cause
    The hub already ran without BESZEL_PASSWORD: USER_EMAIL and USER_PASSWORD create the user only on an empty database.
    Fix
    cd /opt/beszel && sudo docker compose down, sudo rm -r data/hub, check the line in .env, then sudo docker compose up -d beszel again.
    Stage 2

    web1 is red in the Beszel table

    Cause
    The agent isn’t running or runs without a key: BESZEL_KEY was added to .env, but the container wasn’t recreated.
    Fix
    sudo docker logs beszel-agent gives the reason. After editing .env, run sudo docker compose up -d: restart doesn’t pick up new variables.
    Stage 2

    “Test URL” in Beszel shows an error

    Cause
    The URL is written as https://ntfy.sh/…: Beszel picks the service by the scheme, and for ntfy it’s ntfy://.
    Fix
    ntfy://ntfy.sh/<topic>, the topic from sudo cat /srv/secrets/ntfy/topic.
    Stage 3

    gatus keeps restarting, sudo docker logs gatus shows a config parsing error

    Cause
    A condition without quotes: YAML reads a line that starts with [ as a list.
    Fix
    Put every condition in double quotes, as in the stage 3 file, then sudo docker compose restart.
    Stage 3

    memos is red on the Gatus page, but no alert arrives

    Cause
    The topic didn’t reach the container: .env was filled in after the start, or with a typo.
    Fix
    sudo docker inspect gatus --format '{{range .Config.Env}}{{println .}}{{end}}' | grep NTFY shows the topic inside the container. Empty — cd /opt/gatus && sudo docker compose up -d.
    Stage 4

    The web1 check in Healthchecks.io stays new

    Cause
    The hub runs without HEARTBEAT_URL: the line was added, but the container wasn’t recreated.
    Fix
    cd /opt/beszel && sudo docker compose up -d. In Beszel, Settings, Heartbeat shows the URL and interval and has “Test heartbeat”.
    Stage 4

    A down alert arrives at night, around 4:00

    Cause
    The server reboots after updates (level 1), and the margin runs out while the containers come up.
    Fix
    Raise the Grace Time of the web1 check in Healthchecks.io to 10 minutes.
    Stage 5

    memos has been down for 5 minutes, and no alert

    Cause
    Gatus doesn’t see the failure, or sees it but can’t send the alert.
    Fix
    Open the Gatus page through the tunnel from stage 3. memos is red — see the stage 3 case about alerts; green — config.yaml checks a different URL.

    How to remove it

    cd /opt/gatus && sudo docker compose down
    cd /opt/beszel && sudo docker compose down
    sudo rm -r /opt/gatus /opt/beszel /srv/secrets/ntfy

    Delete the web1 check in Healthchecks.io too, or it will send down in 6 minutes. Remove the topic subscription in the ntfy app on your phone.

    Next on the path