Monitoring answers “is it up?” Detection answers “is something wrong on purpose?” This is the catalogue of rules the lab actually evaluates — two over metrics, two over logs — each one shown with its real expression, the threshold that trips it, and the MITRE ATT&CK technique it maps to. Everything below is copied verbatim from the live config on the host (Prometheus alert rules and the Loki ruler); a rule that isn’t running doesn’t get a card.
Detection Engineering
Every alert rule the lab actually runs — the live LogQL and PromQL, what each one catches, and how it maps to MITRE ATT&CK. Copied from the config on the host, not a wish list.
4
live alert rules — two on metrics, two on logs
2
signal sources — Prometheus metrics · Loki logs
3
MITRE ATT&CK techniques mapped
5m
the window most rules evaluate over
ATT&CK coverage
ReconnaissanceCredential AccessImpact
Security detections
These fire on adversary behaviour. Each is mapped to the MITRE ATT&CK technique it represents — click the badge for the technique's own page.
SSHBruteForce
critical
sum(count_over_time({job="auth"} |= "Failed password" [5m])) > 10tripsMore than 10 failed SSH logins in a rolling 5-minute window — fires immediately (for: 0m).
Password-guessing against SSH: the single noisiest thing that happens to any internet-adjacent host, and the first stage of most account-takeover attempts.
FirewallBlockSpike
warning
sum(count_over_time({job="pfsense", action="block"} [5m])) > 500tripsMore than 500 firewall blocks in 5 minutes, sustained for two (for: 2m).
A burst of dropped packets at the perimeter — the shape of a port scan or recon sweep hitting the firewall. The action="block" label is parsed out of each filterlog line by Promtail.
Health & availability
These fire when the lab is hurting itself — but a stopped service or a flooded disk is also how the impact stage of an attack looks, so they double as tripwires.
InstanceDown
critical
up == 0tripsA scrape target unreachable for 10 continuous minutes (for: 10m) — long enough to ride over the nightly boot window and not cry wolf every morning.
A service, or the whole box, is gone. Primarily availability — but a service that's been stopped, killed, or cut off from the network is also exactly how the impact stage of an attack shows up.
DiskAlmostFull
warning
100 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100) > 85tripsRoot filesystem over 85% full for 15 minutes (for: 15m).
The classic silent killer: a full disk stops logging, databases, and backups. Catching it before 100% keeps the detection stack itself alive to detect anything else.
Health
How it was built →
Threat-hunting queries
Not every signal is an alert. These are interactive LogQL queries I run in Grafana Explore — the raw streams the alerts above are built on, and where a hunt starts before it ever becomes a rule.
{job="auth"} |= "Failed password"{job="pfsense"} |= "filterlog"{job="syslog"}From match to phone
Metric rules and log rules converge on one path. A number crossing a threshold or a log line matching a pattern becomes the same buzz in my pocket — with an all-clear when it clears.
📈 Prometheus · alert_rules
🪵 Loki ruler · security
rule matches
🔔 Alertmanager
ntfy
📱 phone
🔎 investigate
Every for: duration is deliberate. SSHBruteForce fires at 0m because a brute-force is real the instant it's counted; InstanceDown waits 10m so the nightly shutdown and morning boot never page me. send_resolved means the phone also buzzes with the all-clear.