Incidents

Every homelab blog shows the finished thing. This page is the other half: what broke, how I actually noticed, and what it taught me. Some of these are still open — an entry that sits at open for months is more useful than one quietly flipped to fixed.

The column that matters most is how it was found. “I happened to look” appears more than I’d like, and that’s the argument for better detection rather than better intentions.

17written up

13fixed

3still open

1accepted, not fixable

2026-08-25

The tests agreed with each other, and both were wrong

low fixed

what happened: The council dashboard has contract tests whose whole job is to stop the page reading a field the server does not send. They build a database from a fixture and compare it against what the page accesses. The fixture's schema was written by hand and had drifted from the real one — it was missing the message timestamp column, among others. A new query selecting that column would have failed against the fixture, been caught, and instead the whole activity feed was checked against a shape the lab's database does not have. Separately, the browser stand-in used to execute the page had a non-recursive `textContent` and a `querySelectorAll` that returned an empty list unconditionally, so every assertion about rendered text was reading an empty string.
how it was found: Writing a new test that expected the fixture to contain rows, and getting none.
fix: The fixture is now built through the application's own storage class, so it gets the production schema and its migrations for free, and a test compares it against the deployed database. Every insert names its columns instead of relying on position. The browser stand-in's text and selector behaviour are real enough to assert against, and an unsupported selector now throws rather than quietly matching nothing.
lesson: A test double is a claim about the real thing, and nobody checks it. When the double and the code drift together the suite goes green on a system that does not exist. Derive the double from the real definition, or test that it still matches.
still open because: Found while adding gates for a UI change, not while looking for it. The same pass found a colour check scoped to the stylesheet that had missed the inline SVG in the page header — a gate can be narrower than the thing it claims to cover.

2026-08-22

A mistyped path produced the same evidence as asking for none

medium fixed

what happened: The context builder collected repository and container evidence only when the target directory existed, with no branch for when it did not. Passing a misspelled directory therefore produced a byte-identical pack to passing no directory at all. The panel reviewed on no evidence, every stage passed, and nothing in the run recorded that evidence had been asked for and lost.
how it was found: The council reviewing its own source, on a request that happened to name the flag.
fix: The builder now states the outcome in every case — absent, not found, or collected — and those notes render into the pack the reviewers read. The directory check moved inside the collector, where a sibling collector already had one.
lesson: A silent skip and a deliberate omission look identical from the outside, and the difference only matters when something has gone wrong. Make the absence of an input as visible as the input.
still open because: The function had no test coverage at all, which is why it survived. Four of the five tests written afterwards fail against the old code.

2026-08-22

A run parked on a human scored itself as a complete panel

low fixed

what happened: Waiting on the human-mediated auditor is deliberately not treated as a degraded panel, because parking is recoverable and failing is not. The new run scoring read the absence of degradation as completeness, so a run that had heard from two of three reviewers recorded "every reviewer answered".
how it was found: Reading the scores of the first run that used them.
fix: Scoring now takes the set of reviewers still pending, and a parked run scores false naming who it waits on.
lesson: A distinction that is correct in one part of a system becomes a bug when another part reuses it as a proxy for something else. Absence of a bad signal is not presence of a good one.

2026-08-22

A dropped metric hid behind the fail-safe meant to protect the run

low fixed

what happened: Scores are sent best-effort so that an observability outage cannot fail a run whose result already exists. One score was malformed — a categorical value sent as a number instead of a label — and the server rejected it with a 400. The fail-safe swallowed that exactly as designed. Four scores landed, one vanished, and nothing said so.
how it was found: Counting the scores that arrived after the first real run.
fix: The emitter now separates the two cases. A rejected payload means this code sent something wrong and is reported with the response body; an unreachable server stays quiet.
lesson: "Never fail the caller" and "never say anything" are different promises, and conflating them turns a bug in your own request into silence. Swallow the outage, not the error.

2026-08-22

The interface opened with nowhere to draw and waited thirty-four minutes

low fixed

what happened: Resuming a run defaults to the terminal interface like every other invocation. Started with its output redirected, the interface did not fail — it rendered escape codes into the file and then blocked at the human approval gate on a keypress that could never arrive. The panel and the synthesis were already committed; from outside it looked like work in progress.
how it was found: Someone asking why it was taking so long.
fix: It now refuses to open an interface when its output is not a terminal, and refuses before creating the run record, so a run that never starts leaves nothing behind.
lesson: A program that blocks forever is worse than one that exits, because the first looks like progress. Check for the thing you need before doing the work that needs it.

2026-08-22

A dashboard served new data behind an old page

low fixed

what happened: The dashboard's service groups were reorganised. Its API returned the new layout immediately, but the rendered page still carried the old one, and a browser given old markup and new data rendered nothing useful.
how it was found: Trying to open the dashboard after changing it.
fix: Recreating the container rather than restarting it. A restart reuses the container's writable layer, so the framework's cached page survived; a recreate starts from a clean one.
lesson: "The config hot-reloads" can be true of one half of an application and false of the other. Verifying the API and stopping there is not verifying the thing the user opens.
still open because: The change had been checked against every proxied host except the one being edited.

2026-08-20

A spawned agent had a channel the orchestrator did not know about

medium open

what happened: During a review run, one of the agent subprocesses was deliberately barred from reading a config file so it would have to rely on its peer. Instead of asking its peer, it opened a direct message to a different session on the same host and asked there. Nothing about that exchange appeared in the run record, the event log, or the evidence chain.
how it was found: The message arrived in the other session and was noticed by a human reading it.
fix: Nothing yet. The request was declined, because answering it would have defeated the constraint the test existed to check.
lesson: The design assumes an agent reasons over a bounded context pack. That assumption holds for what it is *given* and not for what it can *reach*. A boundary that is not enforced is a convention, and a convention is not a control — which is the same lesson as the redaction module, arriving from the other direction.
still open because: Deliberately not overstated: the agent asked politely, escalated nothing, and the information was not sensitive. The defect is that the run record is incomplete, not that anything was compromised.

2026-08-20

Two reviewers asked each other the same question until the money ran out

low fixed

what happened: Both reviewers were denied the same fact. Each concluded, correctly, that it needed that fact and that the other reviewer might have it. They then asked each other for it every round until the hard turn cap stopped them — four rounds of paid calls producing no new information.
how it was found: Reading the transcript of a live run rather than only its final status.
fix: A question already put to an agent is never asked again. If every request in a round is a repeat, the conversation ends and the deadlock is recorded for the operator.
lesson: The turn cap was doing its job and that hid the problem — it bounded the cost without addressing the cause. A limit that stops a loop is not the same as detecting one.

2026-08-19

An agent leaked a credential from the project that built the redaction module

high fixed

what happened: While debugging a login failure, Claude printed raw HTTP response headers from an auth endpoint straight into the session. One carried a live session key. It then misdiagnosed that key as the gateway master key and rotated the wrong credential, and its repeated login tests left three stray admin keys in the key store.
how it was found: The agent caught and reported it itself, unprompted.
fix: All three keys revoked, master key rotated anyway. The real bug was never a password — see the redirect entry below.
lesson: That project contains a fail-closed redaction module written by the same agent a day earlier, and the debugging output did not go through it. Controls only work where they sit in the path. Good intentions are not a control, whether the operator is a person or a model.

Full write-up →

2026-08-19

A password that was never wrong

medium fixed

what happened: The gateway admin UI rejected every login attempt. The credentials were correct the entire time: the app rebuilds its post-login redirect from the request host and emitted http:// because TLS terminates at the reverse proxy, so the session cookie died on the scheme change and the UI bounced back to the login form.
how it was found: Testing the login endpoint directly instead of retyping the password.
fix: One environment variable telling the app its real external base URL.
lesson: Setting it in the env file alone did nothing — Compose forwards only the variables it names. A config value that exists but is not wired through looks identical to a config value that is wrong.

2026-08-18

The synthesised plan was never persisted

high fixed

what happened: The orchestrator binds a human approval to a hash of the exact plan. That plan lived only in memory, so a run that parked for approval — the normal path, not an edge case — came back with the pre-synthesis text, hashed differently, and could never satisfy its own approval.
how it was found: The first attempt to actually execute a change. Every unit test passed.
fix: Persist the plan on the run and restore it on resume.
lesson: The feature was broken in its single most common use case while the suite was green. Tests prove the parts behave; only running the whole thing proves the parts connect.

Full write-up →

2026-08-18

A vulnerability count that went up on an image that never changed

info accepted

what happened: After a full remediation pass, one Postgres image reported nine MORE fixable findings than before — on a byte-identical digest.
how it was found: Recomputing both scans from raw scanner output instead of trusting the summary.
fix: Nothing to fix. The image was identical; the vulnerability database had moved.
lesson: A scan delta is a measurement with a timestamp, not a score. Comparing two runs is only honest if you can point at something that did not change and show what it did.

Full write-up →

2026-08-18

An upgrade that fixed nothing

info open

what happened: Open WebUI was upgraded from a rolling tag to a pinned release. Different image, different digest, and exactly the same 148 fixable findings as before.
how it was found: Comparing per-image counts rather than the headline total.
fix: None available. The findings live in base layers the release does not address. It is now the single largest remaining contributor.
lesson: Pinning bought reproducibility, not security. Assuming a newer release means fewer vulnerabilities is a guess, and it was wrong here.

Full write-up →

2026-08-17

A backup that could not be restored

high fixed

what happened: A database volume was archived while its container kept running. The archiver failed mid-run with 'No such file or directory' as the database deleted its own parts during a merge. The resulting file looked like a backup and was not one.
how it was found: Reading the error output instead of only the exit code.
fix: Renamed the bad archive so it could never be mistaken for a restore point, stopped the stack, and retook it cold — 53,022 entries, zero errors.
lesson: Anything that writes continuously cannot be archived hot. The same applies to SQLite under WAL, which fails silently rather than loudly and is therefore worse.

Full write-up →

2026-08-17

A failing reviewer hidden behind a waiting one

medium fixed

what happened: In a multi-agent review, one reviewer hit its usage limit while another was waiting on a human. The stage reported only 'waiting' and looked benignly parked, while one of three reviewers had not run at all.
how it was found: The first live run. Mocks could not have produced it.
fix: Failures now outrank parking, and the parked agent is still named in the detail.
lesson: A review that looks complete and is not is the exact failure the whole design exists to prevent. Status precedence is a correctness property, not cosmetics.

Full write-up →

2026-07

Backups stopped for five days and nothing said anything

critical open

what happened: The scheduled backup silently stopped running. Five days passed with no archive and no alert.
how it was found: Building the watchdog that was supposed to catch it — it caught the live outage.
fix: A nightly workflow that checks the backup directory and pushes to my phone.
lesson: The monitor needed fixing before the thing it monitored. Absence of an alert is not evidence of health when nothing is checking for absence.
still open because: Still not fully resolved. The watchdog currently runs BEFORE the backup it checks, so it structurally cannot detect a missed one. Scheduled runs were also observed missing on two consecutive days in August. Listed as open because it is.

Full write-up →

2026-07

Patching that reported success while patching nothing

high fixed

what happened: Automated OS patching reported clean runs while applying no updates.
how it was found: Checking the thing being reported on, rather than the report.
fix: Repaired the reporting path first, then the patching itself.
lesson: A green dashboard is a claim about a system, not the system. Verify the claim against the thing it describes.

Full write-up →