Finding and fixing gaps in my AI safety guardrails
My hard denylist rejected my one-year-old's rubber duck.
I'd built RummageLab to help parents turn ordinary household objects into play, and its safety layer confidently refused one of the most ordinary objects in my house. The duck left a worse question behind. If the guardrail was that wrong about something harmless, what was it getting wrong in the other direction?
When I finally measured, the answer was two things, both already shipped.
What the system is
A parent photographed a few household objects and confirmed which were really there and safe. GPT-5.6 proposed an activity for a parent-selected age band, returning structured data that server-side checks validated before approved components rendered it and the parent reviewed it before play. Only validated output reached the family, with a seeded activity available without model credentials—but validation was only as useful as the checks I had actually written.
How do you know a guardrail works?
I wrote the gates. I reviewed them. They looked right. The denylist covered choking and cutting hazards. The activity validator confirmed that declared materials matched what the parent approved, that developmental-focus IDs existed, and that step timings fit the time available. Each check was defensible on its own.
But reading a denylist didn't tell me how it handled what parents actually type, and reading a validator didn't tell me whether it covered every place a model could introduce an object. Inspection worked for the handful of cases I could hold in my head, and only for the cases I had already thought of.
I needed to measure the gates against cases whose expected decisions were defined independently of the current behavior. Otherwise, I risked writing tests that preserved my assumptions. Measurement was how I could begin earning trust in the system.
Two error rates with very different prices
A guardrail can fail in two ways, and they don't cost the same.
A missed hazard could let an instruction to use scissors through to a three-year-old's activity. An over-block could leave a parent annoyed that a rubber duck got rejected. Both mattered, but I could not reasonably treat them as interchangeable errors.
So I didn't lead with accuracy. For every gate I reported two numbers, side by side:
- Recall: of the cases that should be blocked, the share the gate actually blocked. This is the safety number.
- Over-block rate: of the cases that should pass, the share the gate wrongly blocked. This is the usability cost.
A single accuracy figure would have hidden exactly that distinction, and it moves with corpus composition: pad the set with obviously safe items and accuracy climbs while recall stays flat. Block everything and recall is perfect. Block nothing and over-block is zero. Only the pair shows the trade-off.
One labeling decision followed from this. Objects labeled "caution" (fine with supervision) counted as should-pass, because they were meant to reach the parent with a warning, not disappear. I also set CI thresholds for both metrics to catch regressions beyond those limits.
Building the corpus
I built 299 labeled cases across two corpora.
Object vetting, 245 cases: 98 hazard, 56 caution, 91 safe. I deliberately covered literal denylist entries, plurals, compound names like "button battery", everyday household objects, and attempts to override the rules through the label field.
Authored-text grounding, 54 cases: 30 should block, 24 should pass, all written against one fixed parent-approved kit (a soccer ball and a rubber duck). The should-block cases slipped hazards, unapproved materials, or devices into steps, titles, and tool options. The should-pass cases included ordinary language and safety warnings that legitimately name the hazard they warn about.
Every label carried a written rationale based on the product's age and supervision constraints, rather than the code's current behavior. The hard part was ambiguity. An empty coin purse isn't loose coins, and a safety note that names scissors isn't an instruction to use them. I included ambiguous objects rather than hiding those decisions.
I labeled every case myself, with no second labeler. The corpus encodes my judgment and my blind spots. The prose suite used a single 3–4-year-old activity fixture. This was a constructed challenge set, not a sample of real family usage.
What it caught
Defect one: the denylist only knew singular nouns. “Marble” was blocked, but “marbles” passed. So did “coins” and “batteries,” precisely the forms a parent might type after emptying a junk drawer. I added plural handling, multi-word phrases, and regression coverage.
Defect two: the validator checked the wrong field. This is the one that bothered me. It validated the model's structured declaration of what the activity would use. It never read the prose telling the parent what to do. This simplified fragment showed the mismatch:
materials: [/* parent-approved material categories */],
steps: [{ instruction: "Cut the container open with scissors so it rattles." }]
Materials: approved. Focus IDs: valid. Timings: fine. It passed.
I had validated one representation of the activity while leaving another unchecked.
Every one of the 30 should-block prose cases would have passed that validator. Not most of them: all of them. The fix was a deterministic scan of authored text for denylisted terms, materials the parent never confirmed, and prohibited devices. The scan skips the fields whose job is to name hazards, which are the adult safety note and the stop-if conditions.
The numbers
| Gate | Cases | Recall | Over-block rate |
|---|---|---|---|
| Authored-text grounding | 54 | 100% (30/30) | 4.2% (1/24) |
| Lexical object floor | 245 | 75.5% (74/98) | 4.1% (6/147) |
The 100% was on a 54-case corpus I wrote myself. It meant the guard caught 30 expected blocks, including unapproved materials and devices. It did not mean arbitrary generated prose was safe. Its one over-block was an ordinary sentence about a "recording-like echo."
The lexical floor missed 24 labeled hazards, including “paper clips” and “phone charging cable.” Its 75.5% recall exposed the expected limit of word matching over an open world of objects. Those misses remained real gaps. They supported the case for model vetting above the floor, followed by parent confirmation, but did not establish how much the model improved recall. I had not measured that lift here.
I wired the offline evaluations into GitHub Actions so they run on pull requests and pushes to main, without API credentials. CI fails if recall falls below a configured minimum, over-blocking exceeds its limit, or the committed results report is stale. That gives future changes a repeatable regression check against the cases I had measured.
Limits, and what's next
These evaluations measured deterministic classifiers guarding LLM output. I had not used an LLM-as-judge or calibrated a model judge against human labels. I had not measured developmental appropriateness, production failure rates, or safety across every age band. Passing CI preserved a measured baseline; it did not certify safety.
My next planned piece was retrieval over a curated activity catalogue, with its own evaluation: whether suitable activities appeared among the retrieved candidates and which constraints retrieval missed. That needed separate evidence before I could claim an improvement.
The harness was roughly a day’s work, and it found two real bugs in code I had already shipped and believed in.
Log in or sign up for Devpost to join the conversation.