basforge

Search titles, text and the “If it doesn’t work” cases.

    RU

    New server: the first hour

    Turn an empty VPS into a server you can leave on the internet: your own user with key-only login, a firewall, updates with a nightly reboot, swap, a journal limit and Docker. The why is in the manuals, linked from every stage.

    Start froman empty VPS from the provider’s panel: Ubuntu 24.04 or 26.04, root login with the password from the provider’s email or the key from the panel.

    What you get

    login        example-user with a key; passwords and root over SSH are off
    outside      ufw: 22, 80 and 443 open, everything else dropped
    updates      security fixes every night, a reboot at 4:00 when needed
    memory       swap in /swapfile, 2 GB
    journal      500 MB at most; the clock is synchronized
    Docker       from Docker’s repository, container logs capped at 30 MB
    • You’ll needThe server address 203.0.113.10 and root access: the password from the provider’s email or the key from the panel
    • You’ll needAn SSH key on your computer (SSH keys); a second one on another device, in case you lose the first
    • You’ll needA login to the provider’s panel: it has the server’s web console (Provider console)

    Your values

    Filled into the commands and files of this and other recipes on the path; stored in this browser only.
    Server address
    203.0.113.10placeholder
    User
    example-userplaceholder
    Server name
    web1placeholder

    Without the script the commands show placeholder values: replace them with yours.

    Steps

    The same recipe, short: commands from the stages, files as one line. For a second server and for those who already know what each command does.

    First login: updates, hostname, time zone

    Why — First login to a server

    Before changing anything, open the server’s web console in the provider’s panel and log in as root. It bypasses the network and SSH: if you lock yourself out, it’s the only way back in (Provider console). This stage and the start of the next run as root, so there’s no sudo.

    Login
    Command What it does
    ssh [email protected] log in; check the server key fingerprint in the console: ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
    Updates
    Command What it does
    apt update && apt full-upgrade -y installs every update; full-upgrade also removes packages when the upgrade needs it
    cat /var/run/reboot-required 2>/dev/null && reboot reboots if the kernel was updated; log in again a minute later
    Hostname and time zone
    Command What it does
    hostnamectl set-hostname web1 the server’s name: shown in the prompt, the logs and emails
    sed -i 's/^127\.0\.1\.1.*/127.0.1.1 web1/' /etc/hosts the name in /etc/hosts, or every sudo says unable to resolve host
    grep -q '^127.0.1.1' /etc/hosts || echo '127.0.1.1 web1' >> /etc/hosts adds the line if it wasn’t there
    timedatectl set-timezone Europe/Berlin your zone, list them with timedatectl list-timezones; it decides when the nightly 4:00 reboot comes
    Done when
    Command What you should see
    cat /var/run/reboot-required 2>/dev/null || echo none none: no reboot needed
    hostname web1
    timedatectl | grep -E 'Time zone|synchronized' your zone and System clock synchronized: yes
    Not like this — see “If it doesn’t work”, stage cases.

    Your own user and key

    Why — First login to a server

    adduser asks for a password — make it long and keep it in a password manager. You won’t log in over SSH with it, but sudo and the provider’s console need it.

    User — as root
    Command What it does
    adduser example-user a user with a home directory; asks for a password
    Rights and key — as root
    Command What it does
    usermod -aG sudo example-user the right to use sudo
    rsync --archive --chown=example-user:example-user /root/.ssh /home/example-user/ the key from the provider’s panel goes to the user, with permissions; root has no key — see the next table
    Key from your computer — if root had none
    Command What it does
    ssh-copy-id [email protected] puts your key on the server while password login is still on; on Windows use the PowerShell command from SSH keys

    Keep the root window open (Keep the old session open). Continue in a second window.

    Log in as the user — second window
    Command What it does
    ssh [email protected] key login: only the key’s passphrase is asked
    Check sudo
    Command What it does
    sudo -v && sudo whoami sudo works: asks for example-user’s password and answers root
    Done when
    Command What you should see
    sudo whoami root after example-user’s password
    groups the list includes sudo
    Not like this — see “If it doesn’t work”, stage cases.

    Key-only login

    Why — sshd_config

    In the second window, as example-user. From this stage on, commands run with sudo.

    /etc/ssh/sshd_config.d/00-hardening.confkey-only login, and only for example-user; the 00- prefix makes it load before 50-cloud-init.conf
    PasswordAuthentication no
    KbdInteractiveAuthentication no
    PubkeyAuthentication yes
    
    PermitRootLogin no
    
    AllowUsers example-user
    
    MaxAuthTries 3
    LoginGraceTime 30
    
    X11Forwarding no
    AllowAgentForwarding no
    
    ClientAliveInterval 300
    ClientAliveCountMax 2
    Apply
    Command What it does
    sudo sshd -t && echo ok syntax check; no ok — don’t go on
    sudo systemctl restart ssh restart; open sessions survive

    Check the login from a third window, and only then close the root window. Before locking root’s password, log in to the provider’s console as example-user: after the lock, that’s the only way in there.

    Root password
    Command What it does
    sudo passwd -l root locks root’s password, the provider’s console included

    Run the first two checks from your computer, in a new window.

    Done when
    Command What you should see
    ssh -o PubkeyAuthentication=no -o PreferredAuthentications=password [email protected] Permission denied (publickey): no password accepted
    ssh [email protected] Permission denied (publickey): root can’t log in, even with a key
    sudo passwd -S root root L …: the password is locked
    Not like this — see “If it doesn’t work”, stage cases.

    Firewall

    Why — The ufw firewall
    Rules and enable
    Command What it does
    sudo ufw default deny incoming incoming is denied; takes effect on enable
    sudo ufw default allow outgoing outgoing is allowed
    sudo ufw allow OpenSSH SSH — before enabling, or enable closes it too
    sudo ufw allow 80,443/tcp the web: Let’s Encrypt checks and the HTTPS redirect arrive on 80
    sudo ufw --force enable enables it; --force skips the question about SSH sessions, the SSH rule is already there

    ufw doesn’t close ports that Docker publishes: they bypass its rules (Docker ports bypass the input chain). That’s why services further along the path have no ports:.

    Done when
    Command What you should see
    sudo ufw status verbose | head -3 Status: active and Default: deny (incoming), allow (outgoing), deny (routed)
    ssh [email protected] 'echo ok' from your computer, in a new window: ok
    Not like this — see “If it doesn’t work”, stage cases.

    Automatic updates and a reboot at 4:00

    Why — unattended-upgrades

    Ubuntu already installs security updates on its own, but a new kernel waits for a manual reboot. Your own file turns the reboot on and cleans out old kernels; leave 50unattended-upgrades from the package alone.

    /etc/apt/apt.conf.d/52unattended-upgrades-locala reboot at 4:00 when an update needs one, and old kernel cleanup
    Unattended-Upgrade::Automatic-Reboot "true";
    Unattended-Upgrade::Automatic-Reboot-Time "04:00";
    
    Unattended-Upgrade::Automatic-Reboot-WithUsers "true";
    
    Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";
    Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
    Unattended-Upgrade::Remove-Unused-Dependencies "true";
    
    Unattended-Upgrade::AutoFixInterruptedDpkg "true";

    Docker’s repository is left out of automatic updates on purpose: upgrading docker-ce restarts every container, so do it by hand.

    Done when
    Command What you should see
    apt-config dump | grep -E 'Automatic-Reboot(-Time)? ' Automatic-Reboot "true" and Automatic-Reboot-Time "04:00"
    sudo unattended-upgrade --dry-run --debug 2>&1 | grep 'Allowed origins' the list has noble-security, on 26.04 resolute-security
    systemctl list-timers 'apt-daily*' --no-pager apt-daily.timer and apt-daily-upgrade.timer with their next run time
    Not like this — see “If it doesn’t work”, stage cases.

    Swap, journal, time

    Why — Swap, journal and time
    2 GB of swap
    Command What it does
    sudo fallocate -l 2G /swapfile the swap file; on btrfs see “If it doesn’t work”
    sudo chmod 600 /swapfile process memory ends up in swap, so only root may read it
    sudo mkswap /swapfile formats the file as swap
    sudo swapon /swapfile turns it on
    echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab turn it on at boot
    sudo findmnt --verify checks /etc/fstab: an error there stops the boot
    echo 'vm.swappiness=10' | sudo tee /etc/sysctl.d/99-swappiness.conf the kernel swaps only when memory runs short
    sudo sysctl --system applies it without a reboot
    Journal settings directory
    Command What it does
    sudo mkdir -p /etc/systemd/journald.conf.d your settings go in a separate file that survives package upgrades
    /etc/systemd/journald.conf.d/00-size.confthe journal takes 500 MB at most and leaves at least 1 GB free on the disk
    [Journal]
    SystemMaxUse=500M
    SystemKeepFree=1G
    Apply
    Command What it does
    sudo systemctl restart systemd-journald the limit applies to new entries
    sudo journalctl --vacuum-size=500M deletes the excess right away

    The clock needs no setup: it synchronizes on its own, with timesyncd on 24.04 and chrony on 26.04. Only check that synchronization works.

    Done when
    Command What you should see
    swapon --show /swapfile file 2G
    cat /proc/sys/vm/swappiness 10
    journalctl --disk-usage 500 MB at most
    timedatectl show -p NTPSynchronized --value yes
    Not like this — see “If it doesn’t work”, stage cases.

    Docker from Docker’s repository

    Why — Docker on Ubuntu
    Docker’s repository
    Command What it does
    sudo apt-get remove -y $(dpkg --get-selections docker.io docker-compose docker-compose-v2 docker-doc docker-buildx podman-docker containerd runc | cut -f1) removes Ubuntu’s Docker if it’s there; a fresh server has nothing to remove
    snap list docker 2>/dev/null && sudo snap remove docker removes the snap Docker if it’s there
    sudo apt-get update && sudo apt-get install -y ca-certificates curl what’s needed to download the repository key
    sudo install -m 0755 -d /etc/apt/keyrings a directory for the key
    sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc the key apt uses to verify Docker’s packages
    sudo chmod a+r /etc/apt/keyrings/docker.asc everyone can read the key
    printf 'Types: deb\nURIs: https://download.docker.com/linux/ubuntu\nSuites: %s\nComponents: stable\nArchitectures: %s\nSigned-By: /etc/apt/keyrings/docker.asc\n' "$(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")" "$(dpkg --print-architecture)" | sudo tee /etc/apt/sources.list.d/docker.sources >/dev/null the Docker package source; the Ubuntu version and architecture fill themselves in
    sudo apt-get update apt learns about Docker’s packages
    sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin Engine and Compose; the service starts and enables itself
    /etc/docker/daemon.jsonthe log cap: three files of 10 MB per container
    {
      "log-driver": "json-file",
      "log-opts": {
        "max-size": "10m",
        "max-file": "3"
      }
    }
    Apply the cap
    Command What it does
    sudo systemctl restart docker the cap applies to containers created after the restart

    Don’t add the user to the docker group: through docker.sock it grants root without a password (docker.sock). That’s why sudo comes before docker too.

    Done when
    Command What you should see
    sudo docker run --rm hello-world | grep Hello Hello from Docker!
    docker compose version Docker Compose version v5.6.0 or newer
    sudo docker create --name logtest alpine >/dev/null && sudo docker inspect logtest --format '{{.HostConfig.LogConfig}}'; sudo docker rm logtest >/dev/null {json-file map[max-file:3 max-size:10m]}
    Not like this — see “If it doesn’t work”, stage cases.

    If it doesn’t work

    Stage 1

    Could not get lock /var/lib/dpkg/lock-frontend. It is held by process 1234 (unattended-upgr)

    Cause
    In the first minutes after boot the server installs security updates on its own.
    Fix
    Wait: while pgrep -x unattended-upgr >/dev/null; do sleep 5; done, then run the command again. Don’t delete the lock files.
    Stage 1

    During the upgrade a dialog says A new version of configuration file /etc/ssh/sshd_config is available

    Cause
    The provider or cloud-init changed the file, and the package doesn’t know whose version to keep.
    Fix
    Pick keep the local version currently installed: your settings go into a separate file in stage 3 anyway (sshd_config).
    Stage 1

    After a reboot the hostname is vm-4821 again

    Cause
    cloud-init sets the hostname from the provider’s settings on every boot.
    Fix
    echo 'preserve_hostname: true' > /etc/cloud/cloud.cfg.d/99-hostname.cfg, then hostnamectl set-hostname again.
    Stage 2

    example-user is not in the sudoers file

    Cause
    The user isn’t in the sudo group, or was added to it but the session is old: groups are read at login.
    Fix
    As root: usermod -aG sudo example-user, then log out and back in; check with groups example-user.
    Stage 2

    ssh example-user@… asks for the user’s password, not the key’s

    Cause
    The key isn’t in /home/example-user/.ssh/authorized_keys: root had none, so rsync copied nothing useful, or the client offered the wrong key.
    Fix
    From your computer, run ssh-copy-id from stage 2; ssh -v [email protected] 2>&1 | grep -E 'Offering|accepts' shows which key went out (SSH keys).
    Stage 3

    After restart ssh new sessions fail with Connection refused

    Cause
    The config has an error and sshd didn’t start. The open session lives on — it has a process of its own.
    Fix
    In the open session: sudo sshd -t names the line; fix it and run sudo systemctl restart ssh. No session left — use the provider’s console (Provider console).
    Stage 3

    Permission denied (publickey) for example-user, though the key worked before stage 3

    Cause
    AllowUsers has a different name: the user was created under a name other than the one in “Your values”, or there’s a typo.
    Fix
    In the open session fix the AllowUsers line in /etc/ssh/sshd_config.d/00-hardening.conf, run sudo sshd -t && sudo systemctl restart ssh, and log in from a new window (Keep the old session open).
    Stage 4

    After ufw enable new SSH sessions hang and time out

    Cause
    There was no SSH rule: the deny incoming policy closed port 22 as well.
    Fix
    In a session that’s still open, run sudo ufw allow OpenSSH. No session — the provider’s console, same command.
    Stage 5

    --dry-run installs nothing, /var/log/unattended-upgrades/ is empty

    Cause
    /etc/apt/apt.conf.d/20auto-upgrades says "0" or doesn’t exist — it happens with provider images.
    Fix
    sudo dpkg-reconfigure -plow unattended-upgrades; answering Yes recreates the file (unattended-upgrades).
    Stage 5

    The server rebooted at 4:00 UTC, not local time

    Cause
    Automatic-Reboot-Time follows the server’s time zone, and it’s still UTC.
    Fix
    sudo timedatectl set-timezone … from stage 1, or write the time in UTC.
    Stage 6

    swapon: /swapfile: swapon failed: Invalid argument or skipping - it appears to have holes

    Cause
    The file system is btrfs or an old XFS: fallocate makes a file with holes there, and the kernel rejects it.
    Fix
    sudo rm /swapfile, sudo dd if=/dev/zero of=/swapfile bs=1M count=2048 status=progress, then chmod, mkswap, swapon from stage 6. On btrfs use btrfs filesystem mkswapfile.
    Stage 6

    timedatectl show -p NTPSynchronized --value says no

    Cause
    Outbound UDP 123 is blocked — by the provider or an outbound firewall; on 26.04 TCP 4460 too: chrony gets its NTS keys over it.
    Fix
    Look at timedatectl timesync-status (24.04) or chronyc sources -v (26.04) and open the outbound ports (Swap, journal and time).
    Stage 7

    docker-ce : Conflicts: docker.io during install

    Cause
    Docker from the Ubuntu repository is installed, and the stage’s first command was skipped.
    Fix
    Run it and install again. Images and volumes in /var/lib/docker survive remove.
    Stage 7

    permission denied while trying to connect to the Docker daemon socket

    Cause
    The command ran without sudo: the user isn’t in the docker group, by design.
    Fix
    sudo docker …. The docker group is root without a password (docker.sock).

    How to remove it

    The base isn’t taken apart piece by piece: recreating the server in the provider’s panel is faster, and cloud-init gives you the same steps as a file at creation time (cloud-init). Only Docker comes off on its own — along with every image, container and volume:

    sudo apt-get purge -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
    sudo rm -r /var/lib/docker /var/lib/containerd
    sudo rm /etc/apt/sources.list.d/docker.sources /etc/apt/keyrings/docker.asc /etc/docker/daemon.json

    Next on the path