basforge

Search titles, text and the “If it doesn’t work” cases.

    RU
    ManualHomelabLogs

    Traefik logs: CrowdSec, vector, Loki, rotation

    Traefik’s access log is the main source both for protection (CrowdSec) and for working out what happened (Loki and Grafana). It’s read directly or through vector, which splits the log by service and drops the noise. Every reader remembers where it stopped, and every file has exactly one rotation mechanism.

    Who writes and who reads

    Traefik writes two logs: its service log traefik.log, and the access log, one JSON line per request. Three readers consume the access log:

    access.log ─┬─→ CrowdSec (type: traefik)                       — attack scenarios → bans
                ├─→ vector → split/<service>.log → CrowdSec         — the same, per service and without noise
                └─→ Promtail → Loki → Grafana                       — search, graphs, investigation

    Promtail ships the logs of every container to Loki (through docker.sock:ro), plus the access log for full-text search. It keeps its positions in positions.yaml, on a volume.

    Setup

    1. Traefik writes two logs

      log:                                   # Traefik's service log
        level: INFO
        filePath: "/logs/traefik.log"
        maxSize: 10                          # MB — Traefik rotates this file itself
        maxBackups: 5
        maxAge: 30
        compress: true
      accesslog:                             # one JSON line per request
        format: json
        filePath: "/logs/access.log"         # no rotation of its own — needs logrotate
    2. CrowdSec reads the access log

      Directly, with the file acquis.d/traefik.yaml:

      filenames: [/opt/traefik/data/logs/access.log]
      labels: { type: traefik }

      With many services or noisy clients, put vector between Traefik and CrowdSec (next step), and CrowdSec reads split/*.log instead: each service gets its own file, and each one’s parsing is tuned separately.

    3. vector splits the log by service

      Split by request host, with noise dropped by a filter:

      data_dir: /var/lib/vector                       # read positions — on a volume (step 4)
      sources:
        traefik_raw:
          type: file
          include: [/logs/access.log]
          read_from: beginning                        # only for a file with no saved position
      transforms:
        parse_json:
          type: remap
          inputs: [traefik_raw]
          source: . = parse_json!(.message)
        global_filter:                                # drop noisy requests from monitoring agents
          type: filter
          inputs: [parse_json]
          condition: '!(.RequestHost == "monitor.example.org" && .RequestMethod == "POST" && starts_with(string!(.RequestPath), "/api/agents"))'
        split_services:
          type: route
          inputs: [global_filter]
          route:
            notes: '.RequestHost == "example.org"'
            monitor: '.RequestHost == "monitor.example.org"'
      sinks:
        notes_file:   { type: file, inputs: [split_services.notes],   path: /out/notes.log,   encoding: { codec: json } }
        monitor_file: { type: file, inputs: [split_services.monitor], path: /out/monitor.log, encoding: { codec: json } }
        fallback:     { type: file, inputs: [split_services._unmatched], path: /out/other.log, encoding: { codec: json } }

      Cut noisy requests before CrowdSec: scenarios like “too many requests in a short time” can ban your own monitoring agent.

    4. Read positions go on a volume

      Every reader remembers where it stopped: vector in data_dir, Promtail in positions.yaml, CrowdSec on its own. Positions must survive a container recreate, or the file gets read from the start after an image update:

      vector:
        volumes:
          - ./data/logs:/logs:ro
          - ./data/logs/split:/out
          - vector_data:/var/lib/vector               # positions — on a volume
      volumes:
        vector_data:

      If vector already runs without a volume, carry the positions over so it doesn’t reread the log:

      docker compose stop vector
      docker cp vector:/var/lib/vector/. /root/vector-ckpt/
      git pull                                         # compose with the volume
      docker compose create vector                     # new container, not started yet
      docker cp /root/vector-ckpt/. vector:/var/lib/vector/
      docker compose start vector && rm -rf /root/vector-ckpt
    5. Rotation — one mechanism per file

      File Who rotates it How
      traefik.log Traefik itself (maxSize, maxBackups, maxAge, compress) don’t add it to logrotate
      access.log logrotate rename plus the USR1 signal: Traefik reopens the file
      split/*.log (written by vector) logrotate copytruncate: vector keeps the files open and doesn’t take a reopen signal
      /etc/logrotate.d/traefik
      /opt/traefik/data/logs/access.log
      {
              rotate 2
              daily
              missingok
              notifempty
              compress
              delaycompress

      delaycompress — the .1 file isn’t compressed right away, so readers can finish it.

              postrotate
                docker kill --signal="USR1" traefik
              endscript
      }
      Key part

      After the rename, Traefik gets USR1 and opens a new access.log.

      /etc/logrotate.d/vector-split-logs
      /opt/traefik/data/logs/split/*.log
      {
          daily
          rotate 7
          maxsize 10M
          missingok
          notifempty
          compress
          delaycompress
          copytruncate
      }

      copytruncate: a copy goes to the archive and the file is truncated in place — vector keeps writing to the same file.

      Both files live in the server repository (host/logrotate.d/, Server configs in git).

    How many readers to run

    Setup When Cost
    CrowdSec reads access.log directly one or two sites, little noise everything in one pile
    vector splits into split/*.log, CrowdSec reads those many services; noisy clients (monitoring agents with frequent POSTs) one more container and rotation of its files
    plus Promtail, Loki and Grafana you need search and graphs memory: Loki is the heaviest (hundreds of MB)

    If it doesn’t work

    After a vector update, split/*.log has duplicates and CrowdSec bans for old events

    Cause
    Read positions lived in the container layer and were lost on recreate, so the whole access.log was read again.
    Fix
    Move data_dir to a volume (step 4).

    The splitter “works”, but the files are empty

    Cause
    It reads the wrong path — its own empty folder instead of Traefik’s logs — or it isn’t running.
    Fix
    Compare include with the volume and check docker compose ps.

    Some traefik.log lines go missing, files get mixed up

    Cause
    traefik.log is rotated by both Traefik itself and logrotate.
    Fix
    Remove it from logrotate.

    access.log grows forever

    Cause
    It has no rotation of its own.
    Fix
    Set up logrotate with the USR1 signal (step 5).

    A *.log glob in a CrowdSec source also picks up traefik.log

    Cause
    Service lines don’t parse as an access log and add noise to the parser metrics.
    Fix
    Name the files more precisely.

    CrowdSec stopped seeing events after rotation

    Cause
    postrotate didn’t run, and Traefik keeps writing to the old, renamed file.
    Fix
    Check ls -la /opt/traefik/data/logs/: Traefik should be writing to the new file. CrowdSec and Promtail follow a file by name and switch to the new one: the CrowdSec log shows Successfully reopened /logs/access.log.

    An attack disguised as a monitoring agent went unnoticed

    Cause
    The noise filter is too broad: CrowdSec never sees the dropped lines.
    Fix
    Filter narrowly: host, method and path together.

    How to check

    logrotate -d /etc/logrotate.conf 2>&1 | grep -iE 'error|traefik|split'    # dry run, changes nothing
    ls -la /opt/traefik/data/logs/ /opt/traefik/data/logs/split/              # sizes and freshness
    docker exec crowdsec cscli metrics show acquisition                       # lines per source keep growing