Security Operations Centre

My homelab had identity, a reverse proxy and a firewall — and only the firewall produced any security telemetry. Authentik’s events stayed in its own database, Caddy wrote no access log at all, and 28 containers logged to disk with nothing reading them. Nothing needed installing; it was a sources gap, which is the kind that hides well because every service looks healthy.

This page is the result: what feeds the SOC, what it detects, how an alert reaches me, and what I still haven’t got right. The detection catalogue lists every rule with its live expression; the write-up covers the bugs found along the way — including two I introduced myself.

7 log sources centralised

35 detection rules evaluating

23 panels on the security dashboard

168h log retention — the honest ceiling on any baseline

The pipeline

Four stages, built in this order on purpose. Collection first, then detection, then the alert path, and the dashboard last — a single pane over sources that aren't shipping looks like coverage and behaves like decoration.

collect

firewall · host auth · syslog · reverse proxy · identity · containers · agent audit rsyslog · Promtail · exporters 🪵 Loki

Access logs are redacted at source — OAuth parameters, cookies and bearer headers never reach disk.

detect

🪵 Loki ruler · 17 log rules 📈 Prometheus · 18 metric rules

Thresholds derived from rotated raw logs, not from the log store, because a query longer than retention silently returns less.

notify

Loki ruler · Prometheus Alertmanager · split by severity 🔔 ntfy 📱 phone

Critical and warning route separately, so urgency is visible before the notification is opened.

investigate

📊 Grafana Homelab Security Overview · 23 panels 🔎 Explore · LogQL

Built last, deliberately. A dashboard over sources that aren't shipping is decoration.

What feeds it

The lab had identity, a reverse proxy and a firewall, and only the firewall produced security telemetry. This is what closing that gap actually required — the interesting column is the last one.

🧱 pfSense firewall

givesEvery filter decision at the perimeter — action, interface, direction, protocol, source and destination.

howfilterlog over syslog to rsyslog, parsed by Promtail into labels.

The only source producing security telemetry before this project.

🔑 Host authentication

givesSSH and sudo outcomes on the Debian host.

howauth.log, tailed by Promtail.

Source addresses are parsed at query time rather than indexed, so attribution costs no cardinality.

🚦 Reverse proxy access

givesEvery request to every service — virtual host, status, method, client address.

howCaddy JSON access log with a filter encoder, tailed by Promtail.

Did not exist before this project. The proxy fronts single sign-on, so redaction had to come first — and a response-header leak was found and closed after it had already written session tokens.

🪪 Identity provider

givesLogins, failures, authorisations, MFA changes, suspicious requests.

howA timer-driven exporter reading an allow-listed projection of the events table.

Its container stdout carries none of this — 30 minutes of output was health checks and zero login-shaped lines. Shipping container logs could never have delivered identity telemetry.

📦 Containers

givesstdout and stderr from every container on the host.

howPromtail reads Docker's json-file logs directly from disk.

Docker service discovery was refused on purpose: it needs the API socket mounted in, which is a host-root escalation path. The accepted cost is that streams carry container ids, not names.

🗒 System log

givesEverything the host reports that isn't authentication.

howsyslog, tailed by Promtail.

The catch-all for when you don't yet know what you're looking for.

🛡 Agent audit plane

givesWhat the AI agent running this lab attempted, and how it was judged.

howA reduced, identifier-free projection of the hash-chained ledger.

Four of the 35 rules watch the agent rather than the network.

Decisions worth defending

Each of these cost something. The reason is the point, not the choice.

No Docker socket in the log shipper Service discovery would give container names, but requires mounting the Docker API socket. Read-only does not help — the risk is the API, not the file. On a project about security posture, that is not a trade worth making for nicer labels.

Redact at the source, not in the pipeline The proxy sits in front of single sign-on, so OAuth parameters arrive on the URI. Redacting in the shipper would mean the secret is already on disk by the time anything filters it.

Baselines from rotated raw logs, not from the log store Retention is 168 hours. A query over 30 days does not error — it returns seven days. One rule shipped with a threshold a third of what it should have been because of exactly this.

Absence is a signal A source that stops reporting looks identical to a quiet one. A rule alerts on the absence of container logs, and the dashboard's per-source gauge uses explicit zero-valued queries so a dead source shows a bar at zero instead of vanishing.

Prove every detection with input you generate A brute-force rule had matched nothing in 30 days, which is indistinguishable from broken. Two deliberately failed logins settled it. Silence is not evidence.

What was actually proven

A detection that has never matched anything is indistinguishable from one that is broken. Each of these was demonstrated with input generated on purpose, not inferred from an absence of alerts.

Access-log redaction works verified A request carrying a unique canary in all six token parameters plus cookie and bearer headers. Zero canaries anywhere in the file; the control parameter survived, proving it redacts by name rather than blunting the whole query string.

The SSH brute-force rule can fire verified Two failed logins driven against the host as an invalid user — below any plausible ban threshold. Both reached the log store and the rule's own expression counted them.

The dashboard's error panel is not silently broken verified A real 404 generated against two virtual hosts. The filter caught both with the right host attached, so an empty panel now means no errors rather than a broken query.

Alert delivery reaches the phone verified End to end through Alertmanager and ntfy, including the resolved case, with severity routing verified on both paths.

Still not right

This section is the reason the rest is worth reading.

Container error baseline can only ever cover 7 days open A container's log file is deleted when the container is recreated, so unlike the host logs there is no long-lived raw source. Waiting does not fix it; only raising retention would.

Access-log lines written before the response-header fix have not been purged open Nothing new is being written, and the log store has no delete API configured, so they age out with retention.

A log filter shape that returns zero instead of erroring open Upstream behaviour in this version. Worked around by never writing that shape, and documented at the top of the rules file so the next rule does not repeat it.

Every rule, with its live expression and ATT&CK mapping: see the detection catalogue →