basforge

Search titles, text and the “If it doesn’t work” cases.

    RU

    Traefik and Let’s Encrypt: wildcard, ACME, a certificate for neighbours

    Request wildcard certificates (*.example.org) on the entry point, not on a router: then they don’t depend on any single service. A wildcard needs the DNS challenge, and the ACME email is written out literally. If someone else needs the certificate, cert-dumper exports it, and the consumer rereads the file on a schedule.

    Setup

    1. The wildcard goes on the entry point

      Traefik’s static config:

      traefik.yml
      entryPoints:
        http:
          address: ":80"
          http:
            redirections:
              entryPoint: { to: https, scheme: https }
        https:
          address: ":443"
          http:
            tls:
              certResolver: cloudflare

      The default resolver: routers with tls=true and no resolver of their own get the certificate from here.

              domains:
                - main: "example.org"
                  sans: ["*.example.org"]
                - main: "example.net"
                  sans: ["*.example.net"]
      Key part

      The wildcard goes here, not on a router. The certificates aren’t tied to any router: any service can be removed or renamed.

      certificatesResolvers:
        cloudflare:
          acme:
            email: [email protected]

      Write the address out: Traefik reads the static config itself, and ${EMAIL} goes through literally (traefik.yml doesn’t substitute variables).

            storage: acme.json

      Storage for the account and certificates. Mode 600, never in git, ideally in backups: lose it and you reissue everything, possibly hitting Let’s Encrypt limits.

            dnsChallenge:
              provider: cloudflare
              resolvers: ["1.1.1.1:53", "1.0.0.1:53"]

      The Cloudflare token is a Docker secret: CF_DNS_API_TOKEN_FILE: /run/secrets/cf_api_token (Secrets in Docker stacks). Token permissions — DNS edit for the needed zones only.

            # caServer: https://acme-staging-v02.api.letsencrypt.org/directory

      The Let’s Encrypt staging server, for experiments: no limits, but the certificates aren’t trusted.

    2. The service router — just tls=true

      - "traefik.http.routers.app-https.rule=Host(`app.example.org`)"
      - "traefik.http.routers.app-https.entrypoints=https"
      - "traefik.http.routers.app-https.tls=true"        # the certificate is the wildcard from storage
    3. A certificate for neighbours — cert-dumper

      If something other than Traefik needs the same certificate (nginx as a placeholder site, a mail server), export it from acme.json into files — for example, with traefik-certs-dumper:

      traefik-cert-dumper:
        image: ldez/traefik-certs-dumper
        entrypoint: sh -c 'apk add jq;
          while ! [ -e /data/acme.json ] || ! [ `jq ".[] | .Certificates | length" /data/acme.json` != 0 ]; do sleep 5; done &&
          traefik-certs-dumper file --version v2 --domain-subdir --clean=false --watch --source ./acme.json --dest /cert-dumps'
        volumes:
          - ./data:/data
          - ./cert-dumps:/cert-dumps
        working_dir: /data

      --domain-subdir puts the files into cert-dumps/<domain>/certificate.crt and privatekey.key; --watch updates them on renewal.

    4. nginx rereads the certificate on a schedule

      nginx reads the certificate at start and on reload; otherwise it keeps serving the old one until it expires. Once a week is plenty: renewal happens 30 days before expiry.

      /etc/cron.d/nginx-reload
      23 4 * * 1 root (docker exec nginx nginx -t && docker exec nginx nginx -s reload && echo reloaded) 2>&1 | logger -t nginx-reload

      Mount the certificates into nginx as a directory (./cert-dumps/example.org:/etc/nginx/ssl:ro), not as files: a single-file mount stays stale after an update (Single-file mounts).

    Traefik renews a Let’s Encrypt certificate (90 days) 30 days before it expires. There’s nothing to do for that, but do check the first renewal after any ACME change.

    A wildcard or a certificate per name

    Choice When Cost
    wildcard, DNS challenge many subdomains; you’d rather not expose their names a DNS provider API token on the server
    a certificate per subdomain, HTTP challenge (port 80) a couple of sites, DNS without an API every name lands in the public certificate logs for good
    TLS-ALPN challenge (port 443) port 80 is closed same, the names are public
    wildcard on the entry point whenever you use a wildcard —
    wildcard via tls.domains on a router — the certificates depend on one router: remove it and renewal stops

    Every issued certificate lands in the public Certificate Transparency logs (crt.sh). A separate certificate for admin.example.org is a public announcement that the admin panel exists. A wildcard doesn’t reveal subdomain names, unless separate certificates were issued for them before. Basic Auth hides the content, not the subdomain itself; an admin panel that shouldn’t be visible at all isn’t published — open it only from the internal network.

    With sniStrict: true in the TLS options (tls.options.default.sniStrict) Traefik doesn’t answer requests without a server name: scanners hitting the IP get a refusal, not a certificate full of domain names.

    If it doesn’t work

    Sites suddenly show Traefik’s self-signed certificate

    Cause
    The wildcard hung on the tls.domains of a router that was removed or renamed: renewal stopped, the certificate expired.
    Fix
    Move the wildcard to the entry point (step 1).

    The config says email: ${EMAIL_FOR_SSL}

    Cause
    Traefik doesn’t substitute variables in traefik.yml; the string goes through literally.
    Fix
    Write the address in. After an address change Traefik may register a new ACME account; certificates already issued stay valid.

    nginx serves an expired certificate while Traefik’s is fresh

    Cause
    nginx reads the certificate at start and on reload, and the file has changed since.
    Fix
    Schedule a reload (step 4).

    acme.json has permissions wider than 600

    Cause
    Traefik refuses to use it: permissions 644 for /acme.json are too open, please use 600.
    Fix
    chmod 600 acme.json.

    Experiments hit Let’s Encrypt rate limits

    Cause
    Every attempt went to the production server, which has issuance limits.
    Fix
    Work against the staging caServer until everything works.

    The certificate doesn’t renew

    Cause
    Most often the DNS token lacks permissions or isn’t mounted: /run/secrets/… is empty.
    Fix
    Check docker logs traefik 2>&1 | grep -i acme | tail -20, the token’s permissions and the secret file.

    How to check

    # expiry of the certificates from cert-dumper
    for d in example.org example.net; do printf '%s: ' "$d"; openssl x509 -enddate -noout -in /opt/traefik/cert-dumps/$d/certificate.crt; done
    # what the site serves
    echo | openssl s_client -connect app.example.org:443 -servername app.example.org 2>/dev/null | openssl x509 -noout -subject -enddate
    # ACME errors
    docker logs traefik 2>&1 | grep -iE 'acme|certificate' | grep -iE 'error|unable|fail' | tail -5

    After the first renewal the expiry date should move about 60 days forward.