Who writes and who reads
Traefik writes two logs: its service log traefik.log, and the access log, one JSON line per request. Three
readers consume the access log:
access.log ─┬─→ CrowdSec (type: traefik) — attack scenarios → bans
├─→ vector → split/<service>.log → CrowdSec — the same, per service and without noise
└─→ Promtail → Loki → Grafana — search, graphs, investigation
Promtail ships the logs of every container to Loki (through docker.sock:ro), plus the access log for full-text
search. It keeps its positions in positions.yaml, on a volume.
Setup
Traefik writes two logs
log: # Traefik's service log level: INFO filePath: "/logs/traefik.log" maxSize: 10 # MB — Traefik rotates this file itself maxBackups: 5 maxAge: 30 compress: true accesslog: # one JSON line per request format: json filePath: "/logs/access.log" # no rotation of its own — needs logrotateCrowdSec reads the access log
Directly, with the file
acquis.d/traefik.yaml:filenames: [/opt/traefik/data/logs/access.log] labels: { type: traefik }With many services or noisy clients, put vector between Traefik and CrowdSec (next step), and CrowdSec reads
split/*.loginstead: each service gets its own file, and each one’s parsing is tuned separately.vector splits the log by service
Split by request host, with noise dropped by a filter:
data_dir: /var/lib/vector # read positions — on a volume (step 4) sources: traefik_raw: type: file include: [/logs/access.log] read_from: beginning # only for a file with no saved position transforms: parse_json: type: remap inputs: [traefik_raw] source: . = parse_json!(.message) global_filter: # drop noisy requests from monitoring agents type: filter inputs: [parse_json] condition: '!(.RequestHost == "monitor.example.org" && .RequestMethod == "POST" && starts_with(string!(.RequestPath), "/api/agents"))' split_services: type: route inputs: [global_filter] route: notes: '.RequestHost == "example.org"' monitor: '.RequestHost == "monitor.example.org"' sinks: notes_file: { type: file, inputs: [split_services.notes], path: /out/notes.log, encoding: { codec: json } } monitor_file: { type: file, inputs: [split_services.monitor], path: /out/monitor.log, encoding: { codec: json } } fallback: { type: file, inputs: [split_services._unmatched], path: /out/other.log, encoding: { codec: json } }Cut noisy requests before CrowdSec: scenarios like “too many requests in a short time” can ban your own monitoring agent.
Read positions go on a volume
Every reader remembers where it stopped: vector in
data_dir, Promtail inpositions.yaml, CrowdSec on its own. Positions must survive a container recreate, or the file gets read from the start after an image update:vector: volumes: - ./data/logs:/logs:ro - ./data/logs/split:/out - vector_data:/var/lib/vector # positions — on a volume volumes: vector_data:If vector already runs without a volume, carry the positions over so it doesn’t reread the log:
docker compose stop vector docker cp vector:/var/lib/vector/. /root/vector-ckpt/ git pull # compose with the volume docker compose create vector # new container, not started yet docker cp /root/vector-ckpt/. vector:/var/lib/vector/ docker compose start vector && rm -rf /root/vector-ckptRotation — one mechanism per file
File Who rotates it How traefik.logTraefik itself ( maxSize,maxBackups,maxAge,compress)don’t add it to logrotate access.loglogrotate rename plus the USR1signal: Traefik reopens the filesplit/*.log(written by vector)logrotate copytruncate: vector keeps the files open and doesn’t take a reopen signal/etc/logrotate.d/traefik /opt/traefik/data/logs/access.log { rotate 2 daily missingok notifempty compress delaycompressdelaycompress— the.1file isn’t compressed right away, so readers can finish it.postrotate docker kill --signal="USR1" traefik endscript }Key partAfter the rename, Traefik gets
USR1and opens a newaccess.log./etc/logrotate.d/vector-split-logs /opt/traefik/data/logs/split/*.log { daily rotate 7 maxsize 10M missingok notifempty compress delaycompress copytruncate }copytruncate: a copy goes to the archive and the file is truncated in place — vector keeps writing to the same file.Both files live in the server repository (
host/logrotate.d/, Server configs in git).
How many readers to run
| Setup | When | Cost |
|---|---|---|
CrowdSec reads access.log directly |
one or two sites, little noise | everything in one pile |
vector splits into split/*.log, CrowdSec reads those |
many services; noisy clients (monitoring agents with frequent POSTs) | one more container and rotation of its files |
| plus Promtail, Loki and Grafana | you need search and graphs | memory: Loki is the heaviest (hundreds of MB) |
If it doesn’t work
After a vector update, split/*.log has duplicates and CrowdSec bans for old events
- Cause
- Read positions lived in the container layer and were lost on recreate, so the whole
access.logwas read again. - Fix
- Move
data_dirto a volume (step 4).
The splitter “works”, but the files are empty
- Cause
- It reads the wrong path — its own empty folder instead of Traefik’s logs — or it isn’t running.
- Fix
- Compare
includewith the volume and checkdocker compose ps.
Some traefik.log lines go missing, files get mixed up
- Cause
traefik.logis rotated by both Traefik itself and logrotate.- Fix
- Remove it from logrotate.
access.log grows forever
- Cause
- It has no rotation of its own.
- Fix
- Set up logrotate with the
USR1signal (step 5).
A *.log glob in a CrowdSec source also picks up traefik.log
- Cause
- Service lines don’t parse as an access log and add noise to the parser metrics.
- Fix
- Name the files more precisely.
CrowdSec stopped seeing events after rotation
- Cause
postrotatedidn’t run, and Traefik keeps writing to the old, renamed file.- Fix
- Check
ls -la /opt/traefik/data/logs/: Traefik should be writing to the new file. CrowdSec and Promtail follow a file by name and switch to the new one: the CrowdSec log showsSuccessfully reopened /logs/access.log.
An attack disguised as a monitoring agent went unnoticed
- Cause
- The noise filter is too broad: CrowdSec never sees the dropped lines.
- Fix
- Filter narrowly: host, method and path together.
How to check
logrotate -d /etc/logrotate.conf 2>&1 | grep -iE 'error|traefik|split' # dry run, changes nothing
ls -la /opt/traefik/data/logs/ /opt/traefik/data/logs/split/ # sizes and freshness
docker exec crowdsec cscli metrics show acquisition # lines per source keep growing