Unnoticed for 63 days
The backups said “finished” and saved nothing
A configuration file gained a second set of cloud credentials without write access to the backup bucket. The backup script read that file, used those credentials, and every upload failed — quietly, because the script ran with --quiet. The log said “backup run finished” every night.
Found during a routine check of what the box would lose. Fixed the same hour; four more products' data added; the daily report now reads the last run's errors.
A job that finished is not a job that worked. Read its errors, and restore one backup to prove it.
Four headline numbers
Scanners counted as people
“22–69 visits a day” to the CV checker, “15 from search” to a developer tool, 279 clicks on a directory's contact buttons: all robots. Scanners send ordinary browser signatures and even fake Google referrers; single-page apps answer 200 to any path.
The rule now: a person loads the page's own scripts and images, never probes a service path on any of the sites, and is not part of a fleet using one signature from twenty addresses.
Before putting a number in front of anyone, look at five of the sessions behind it.
Off by ten times, twice
Estimates reported as measurements
“432 people read last week” — the database had been migrated and every row carried the migration's timestamp; the real figure was six. Later, “the video platform won't fit in memory” — an estimate of 380–500 MB; measured, it ran in 111 MB.
Run the thing and read the value. If it is an estimate, say “estimate”.
39 days
A link hidden under a photo
The directory's home page gained a line leading to its second city. The hero photo is a positioned element; the line was not, so the photo was painted over it. Every automated check reported the element as present and visible.
Visible means seen in a screenshot of a real phone-sized page, not reported by the DOM.
23 days of lost submissions
A redirect that turned forms into nothing
Old hostnames were redirected to new ones with a 301. A partner form still posted to an old hostname; a 301 turns a POST into a GET, so every submission arrived empty and was dropped.
Before redirecting a hostname, search everything served for it. APIs get 308.
One afternoon, whole catalogue down
A security fix that broke search
An audit flagged a storage bucket as publicly listable, so the agent removed listing. The directory's API listed that bucket anonymously to find each place's photos; search returned nothing and 405 of 406 pages went to 404 — and the pages kept the 404 cached after the fix.
Before closing an access path, find out who uses it.
288 of 367 questions
A quiz you could pass without reading
In a citizenship-test trainer the correct option was the longest one in 288 of 367 questions. Always picking the longest answer scored 78% and passed. A user said the answers felt too obvious; the measurement proved it.
For any quiz, measure the score of “always pick the longest” before shipping.
335 wrong words
Teaching a language that does not exist
A Luxembourgish flashcard deck had German forms and invented spellings among its words, and a quarter of its example sentences were wrong. The learner caught one in a lesson. Every word is now linked to the national dictionary and every example comes from it or passes a grammar check.
Generated content in a language you cannot read gets checked against an authority, word by word.
Eight times
Blaming the product for a broken test
The cron tester, the JSON tools, a game, the directory, the stats page: each time an automated check failed, the product was fine and the check was wrong — the wrong environment, a regex that matched a substring, a grep that skipped compressed logs.
When a check fails, first suspect the check. Then make it fail on purpose to see that it can.
One loud evening
A test that played sound in a family home
The machine the agent works from is in the owner's home, not a CI server. A test video with an audio tone played through the speakers. Nothing in the tool output showed it.
Fixtures carry no audio, browsers launch muted, every video is muted in markup. Three layers, because one is a typo.