basforge

Search titles, text and the “If it doesn’t work” cases.

    RU

    CrowdSec with Traefik behind Cloudflare: two bouncers and X-Forwarded-For

    CrowdSec blocks nothing by itself — bouncers do. Set up two: one on the host (nftables) and one in Traefik (a plugin, 403). Sites behind Cloudflare are covered only by the plugin, and only if you tell it correctly whom to trust in X-Forwarded-For.

    How a decision reaches the packet

    logs: SSH, nftables, Traefik access log, container logs ─→ CrowdSec: parsers → scenarios → decision
    community blocklists (CAPI) and subscriptions ──────────────────────────────────────────┘
                                             decisions → LAPI (127.0.0.1:8080)
         ├─→ crowdsec-firewall-bouncer on the host — nftables, input + forward hooks: packets from the address are dropped
         └─→ Traefik plugin — crowdsec@file middleware: HTTP 403 on routers that use it

    Setup

    1. CrowdSec in a container

      Publish LAPI only on 127.0.0.1 — the firewall bouncer on the host talks to it. The Traefik plugin only needs the shared proxy network.

      services:
        crowdsec:
          image: crowdsecurity/crowdsec:latest
          container_name: crowdsec
          restart: unless-stopped
          environment:
            # collections — only for sources you actually have
            COLLECTIONS: "crowdsecurity/linux crowdsecurity/sshd crowdsecurity/iptables crowdsecurity/base-http-scenarios crowdsecurity/http-cve crowdsecurity/traefik"
          volumes:
            - ./data/config:/etc/crowdsec                     # config, hub, LAPI/CAPI credentials
            - ./data/db:/var/lib/crowdsec/data                # decision database
            - /var/log:/var/log:ro                            # auth.log, nftables.log
            - /opt/traefik/data/logs:/opt/traefik/data/logs:ro
            - /var/run/docker.sock:/var/run/docker.sock:ro    # only if you have a source: docker
          ports:
            - "127.0.0.1:8080:8080"                           # LAPI — host only (firewall bouncer)
          networks: [proxy]                                   # shared network with Traefik — for the plugin
    2. Sources — data/config/acquis.d/*.yaml

      The acquis.yaml with a /does/not/exist line is a placeholder from the image. Put real sources into acquis.d/.

      # SSH
      filenames: [/var/log/auth.log]
      labels: { type: syslog }
      ---
      # Traefik access log (JSON)
      filenames: [/opt/traefik/data/logs/access.log]
      labels: { type: traefik }
      ---
      # nginx container logs — via docker.sock
      source: docker
      container_name: [nginx]
      labels: { type: nginx }
    3. Ban duration — profiles.yaml

      decisions:
        - type: ban
          duration: 30d

      The duration is written into a decision when it’s created, so an edit affects only new bans. After editing, restart: docker compose restart crowdsec.

    4. Bouncer on the host

      Install the crowdsec-firewall-bouncer package (nftables). cscli bouncers add <name> issues the key; the LAPI address is 127.0.0.1:8080. You need two hooks, input and forward: without forward the ban doesn’t cover containers with published ports (Docker ports bypass the input chain).

    5. Plugin in Traefik

      Static config traefik.yml:

      experimental:
        plugins:
          crowdsec-bouncer-traefik-plugin:
            moduleName: "github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin"
            version: "v1.8.1"

      The middleware goes into the dynamic config dynamic/crowdsec.yml:

      http:
        middlewares:
          crowdsec:
            plugin:
              crowdsec-bouncer-traefik-plugin:
                enabled: true
                crowdsecMode: stream                    # decisions are pulled every updateIntervalSeconds
                updateIntervalSeconds: 15
                updateMaxFailure: 10                    # after this many failures in a row — block everything
                crowdsecLapiHost: crowdsec:8080
                crowdsecLapiScheme: http
                crowdsecLapiKeyFile: /run/secrets/crowdsec_lapi_key   # the key is a Docker secret, not in the file
                forwardedHeadersTrustedIPs:             # proxies in front of Traefik: trust their X-Forwarded-For
                  - 10.0.0.0/8
                  - 172.16.0.0/12
                  - 192.168.0.0/16
                  # + Cloudflare ranges if the site is behind its proxy
                clientTrustedIPs:                       # who is NOT checked at all — your own networks only
                  - 10.0.0.0/8
                  - 172.16.0.0/12
                  - 192.168.0.0/16
                forwardedHeadersCustomName: "X-Forwarded-For"

      Attach it to a router first in the chain — then a banned client gets 403 before the password prompt:

      - "traefik.http.routers.app-https.middlewares=crowdsec@file,app-auth"

      Pass the key as a Docker secret in Traefik’s compose file:

      services:
        traefik:
          secrets: [crowdsec_lapi_key]
      secrets:
        crowdsec_lapi_key:
          file: /srv/secrets/traefik/crowdsec_lapi_key
    6. A site behind Cloudflare

      Through the Cloudflare proxy every request arrives from a Cloudflare address, and the visitor’s address rides in X-Forwarded-For.

      • Traefik: put the Cloudflare ranges, IPv4 and IPv6, into entryPoints.https.forwardedHeaders.trustedIPs (the list is at cloudflare.com/ips). Then Traefik logs the real address.
      • Plugin: the same ranges go into forwardedHeadersTrustedIPs. For requests from these addresses it takes the real IP from X-Forwarded-For and checks it; Cloudflare itself never gets banned.
      • Never put Cloudflare into clientTrustedIPs: that’s the list of clients the plugin doesn’t check at all.
      • You can’t spoof an address through Cloudflare: Cloudflare appends the real IP last, and the plugin reads the chain from right to left.

    The key is already written in crowdsec.yml

    If the key sits right in the file (crowdsecLapiKey: "…"), move it into a secret without showing it on screen:

    install -d -m 700 /srv/secrets /srv/secrets/traefik
    key=$(sed -nE 's/^[[:space:]]*crowdsecLapiKey:[[:space:]]*"?([^"[:space:]]+)"?.*/\1/p' /opt/traefik/data/dynamic/crowdsec.yml)
    printf '%s' "$key" > /srv/secrets/traefik/crowdsec_lapi_key; unset key   # printf — no trailing newline
    chmod 400 /srv/secrets/traefik/crowdsec_lapi_key
    wc -c < /srv/secrets/traefik/crowdsec_lapi_key                            # not 0

    How many bouncers, and where

    Setup What it covers When
    firewall bouncer only everything by IP: SSH, host services, containers (with the forward hook) a server with no domains and no CDN proxy, such as a backup store
    firewall bouncer + plugin plus sites behind Cloudflare — by the real address from XFF a server with domains, at least one behind a CDN proxy
    plugin only only HTTP routers with the middleware no access to the host firewall (rare)

    CrowdSec works even without its own sources: CAPI subscriptions, the community blocklists, arrive on their own. For a backup store, SSH logs and subscriptions are enough.

    On domains without a CDN the plugin duplicates the firewall: if the firewall bouncer sits on forward, a banned address never reaches Traefik. That does no harm, and for sites behind a CDN the plugin is required.

    Sometimes traffic has to go through Traefik as is, without decryption: a TCP router with tls.passthrough. HTTP middleware doesn’t apply to such a router, and the plugin doesn’t see it — only the firewall covers it, and it blocks the whole address: someone banned by mistake loses everything that goes through this server. Such traffic can’t pass the Cloudflare proxy — Cloudflare decrypts TLS itself, so these domains go without its proxy.

    If it doesn’t work

    Your own address is banned, but the site behind Cloudflare opens

    Cause
    Cloudflare ranges ended up in clientTrustedIPs, which is the list of clients the plugin doesn’t check. Every visitor comes through Cloudflare, so nobody is checked.
    Fix
    Move the ranges to forwardedHeadersTrustedIPs.

    The ban is there, but the site on a direct domain opens

    Cause
    Two things at once: the firewall bouncer sits only on input, and the plugin isn’t attached to this router. Most often the middleware hangs on a router with a different name: app-secure in the label, while the router is called app-https.
    Fix
    Give the firewall bouncer a forward hook (Docker ports bypass the input chain) and check the router name in every label (Middleware on a missing router).

    After editing crowdsec.yml, every site with the plugin went down

    Cause
    The plugin couldn’t read the key — the secret isn’t mounted into the container — and the middleware wasn’t created. Or updateMaxFailure ran out, and the plugin blocks everyone.
    Fix
    Roll back: put the previous crowdsec.yml back into dynamic/; Traefik rereads it immediately.

    CrowdSec finds nothing

    Cause
    A source points at a log that doesn’t exist: the service was removed, the path changed, the container was renamed.
    Fix
    Check cscli metrics show acquisition — how many lines each source has read — and fix the path in acquis.d/.

    cscli bouncers list shows lots of [email protected] entries

    Cause
    Each time the Traefik container is recreated it gets a new address, the plugin comes back with the same key, and CrowdSec adds an entry per address.
    Fix
    Nothing: the live entry has a fresh Last API pull, the rest are leftovers and do no harm. An entry with @ can’t be deleted by name: cscli answers auto-created and cannot be deleted. Leave the entry without @ alone: deleting it removes every entry at once and revokes the key — the plugin gets 403. Leftovers go with docker exec crowdsec cscli bouncers prune -d 168h --force: it deletes entries that haven’t contacted LAPI for a week; the key works while at least one entry is alive.

    Changed the ban duration, but old bans keep the old one

    Cause
    The duration is written into a decision when it’s created.
    Fix
    Nothing: old decisions run out their own time.

    You banned yourself

    Cause
    Your address ended up in decisions — through a scenario or a test ban.
    Fix
    Lift the ban from another address — your phone on mobile data — or through the hosting console: cscli decisions delete --ip <your IP>. That’s why test bans go on your phone’s address, not your work machine’s.

    How to check

    Test it for real, from your phone on mobile data — it has a different address, which ifconfig.me shows:

    docker exec crowdsec cscli decisions add --ip <IP> --duration 2m --reason test
    #  site behind Cloudflare        → 403 from the plugin
    #  site on a direct domain       → doesn't open at all (firewall bouncer)
    #  SSH from this address         → doesn't connect
    docker exec crowdsec cscli decisions delete --ip <IP>
    
    docker exec crowdsec cscli bouncers list     # both bouncers have a fresh "Last API pull"