RummageLab

Rainy day, bored kid, junk drawer. Point RummageLab at three things on the counter and it spins up a safe, age-right science moment your toddler can actually do - parent-approved, ages 0 to 6.

Comment

The Story of RummageLab

Inspiration

Rainy season came early this year and refused to leave. Grey morning after grey morning stuck indoors with a baby, a toddler, and a full-time job — and every pressure valve equally unappealing: infinite snacks, more screen time, or expensive classes across town.

At the same time, I'd been itching to go deeper on modern AI tooling — pipelines, agent orchestration, scaffolding, git worktrees — and a hackathon weekend looked like a chance to solve both problems at once.

The dream workflow was honest about my life: type a /goal, send off a team of agents, and go cook breakfast for the wailing kids while their dad is still asleep. Come back, review, steer, repeat.

The dream product was equally simple: something that turns the junk drawer, the kitchen counter, and the backyard into short, age-right learning moments — things my kids can do with their hands, not another screen to stare at.

What it does

RummageLab turns everyday objects into safe, developmentally appropriate activities for ages 0–6. A parent picks an age band, then photographs or types what is on the table. On the live path, GPT-5.6 vets each object, and — only after the parent confirms every item — authors an activity fitted to the confirmed objects, the child's age band, and the weather. Under-threes get screen-free caregiver co-play scripts; threes-and-up get short guided quests that end with something the child noticed, made, heard, measured, or explained — not another chat transcript. A deterministic seeded path (Kitchen Sound Detectives) runs the full experience with no key configured, so the demo never depends on a live model call.

How I built it

I built RummageLab primarily with Codex 5.6 (Sol & Terra):

  • Codex 5.6 did the heavy early lifting: initial planning, architecture, scaffolding the Next.js app, the Zod schema layer, the runtime contracts, tests, and CI. Work ran as one-task-per-worktree branches with dedicated review passes before merging.
  • The orchestration pattern that emerged: a master orchestrator task fanning out subtasks mapped to git workflows, with dedicated reviewer agents on each feature branch before merge.

The runtime idea I'm proudest of: use live GPT-5.6 to author each activity
dynamically — age-appropriate, weather-dependent, built from parent-vetted
objects — while keeping a hard boundary around it. Nothing the model authors
reaches a child without passing a local hazard denylist, strict Zod plus
context re-validation (materials must be a subset of what the parent confirmed,
focus IDs must exist in the local catalogue, steps must fit the time window),
and explicit parent confirmation. The model composes; it never runs code, and
the app renders only prebuilt, non-recording components. Codex stays at build
time; GPT-5.6 is the only model in the learner runtime.

Challenges and what I learned

  1. AI orchestration is a skill of its own. Running a master orchestrator with subtasks mapped to git workflows — and dedicated reviewers per feature branch — took real iteration. Restarting agents with fresh, focused context beat letting any one context grow stale.
  2. Goal and loop engineering. Writing goals that agents can pursue for a long stretch without drifting, and knowing when to steer versus when to let a loop run, was half the project.
  3. Scaffold extensively, build in branches. Heavy upfront scaffolding plus worktree-isolated branches made parallel agent work merge cleanly instead of colliding.
  4. Vercel hosting for easy deployment.
  5. Don't use GPT-5.6 Ultra for everything. 💸 Gradiate the effort: start with high-effort planning, then drop to lower effort tiers for debugging and mechanical work. My token bill learned this lesson before I did.
  6. Kid safety has to be an axiom, not a feature. COPPA-adjacent design is genuinely tricky: no accounts, no stored photos, no child profiling, PII screening before any request. And it cuts both ways — my hard denylist confidently rejected the rubber duckie my one-year-old plays with incessantly. That forced a real design change: conservative automated vetting plus parental override, because the parent — not the model, and not my denylist — is the final authority on their own living room.

What's next

  • Finish the adaptive feedback loop: a personal model of family preferences (with the same no-PII stance — parent-approved tags only, never raw notes or child data) so subsequent activities follow what a family actually enjoys.
  • Grow RummageLab toward a K–8 audience — same everyday objects, bigger
    STEM ideas.

Built With

Share this project:

Updates

posted an update —

Finding and fixing gaps in my AI safety guardrails

My hard denylist rejected my one-year-old's rubber duck.

I'd built RummageLab to help parents turn ordinary household objects into play, and its safety layer confidently refused one of the most ordinary objects in my house. The duck left a worse question behind. If the guardrail was that wrong about something harmless, what was it getting wrong in the other direction?

When I finally measured, the answer was two things, both already shipped.

What the system is

A parent photographed a few household objects and confirmed which were really there and safe. GPT-5.6 proposed an activity for a parent-selected age band, returning structured data that server-side checks validated before approved components rendered it and the parent reviewed it before play. Only validated output reached the family, with a seeded activity available without model credentials—but validation was only as useful as the checks I had actually written.

How do you know a guardrail works?

I wrote the gates. I reviewed them. They looked right. The denylist covered choking and cutting hazards. The activity validator confirmed that declared materials matched what the parent approved, that developmental-focus IDs existed, and that step timings fit the time available. Each check was defensible on its own.

But reading a denylist didn't tell me how it handled what parents actually type, and reading a validator didn't tell me whether it covered every place a model could introduce an object. Inspection worked for the handful of cases I could hold in my head, and only for the cases I had already thought of.

I needed to measure the gates against cases whose expected decisions were defined independently of the current behavior. Otherwise, I risked writing tests that preserved my assumptions. Measurement was how I could begin earning trust in the system.

Two error rates with very different prices

A guardrail can fail in two ways, and they don't cost the same.

A missed hazard could let an instruction to use scissors through to a three-year-old's activity. An over-block could leave a parent annoyed that a rubber duck got rejected. Both mattered, but I could not reasonably treat them as interchangeable errors.

So I didn't lead with accuracy. For every gate I reported two numbers, side by side:

  • Recall: of the cases that should be blocked, the share the gate actually blocked. This is the safety number.
  • Over-block rate: of the cases that should pass, the share the gate wrongly blocked. This is the usability cost.

A single accuracy figure would have hidden exactly that distinction, and it moves with corpus composition: pad the set with obviously safe items and accuracy climbs while recall stays flat. Block everything and recall is perfect. Block nothing and over-block is zero. Only the pair shows the trade-off.

One labeling decision followed from this. Objects labeled "caution" (fine with supervision) counted as should-pass, because they were meant to reach the parent with a warning, not disappear. I also set CI thresholds for both metrics to catch regressions beyond those limits.

Building the corpus

I built 299 labeled cases across two corpora.

Object vetting, 245 cases: 98 hazard, 56 caution, 91 safe. I deliberately covered literal denylist entries, plurals, compound names like "button battery", everyday household objects, and attempts to override the rules through the label field.

Authored-text grounding, 54 cases: 30 should block, 24 should pass, all written against one fixed parent-approved kit (a soccer ball and a rubber duck). The should-block cases slipped hazards, unapproved materials, or devices into steps, titles, and tool options. The should-pass cases included ordinary language and safety warnings that legitimately name the hazard they warn about.

Every label carried a written rationale based on the product's age and supervision constraints, rather than the code's current behavior. The hard part was ambiguity. An empty coin purse isn't loose coins, and a safety note that names scissors isn't an instruction to use them. I included ambiguous objects rather than hiding those decisions.

I labeled every case myself, with no second labeler. The corpus encodes my judgment and my blind spots. The prose suite used a single 3–4-year-old activity fixture. This was a constructed challenge set, not a sample of real family usage.

What it caught

Defect one: the denylist only knew singular nouns. “Marble” was blocked, but “marbles” passed. So did “coins” and “batteries,” precisely the forms a parent might type after emptying a junk drawer. I added plural handling, multi-word phrases, and regression coverage.

Defect two: the validator checked the wrong field. This is the one that bothered me. It validated the model's structured declaration of what the activity would use. It never read the prose telling the parent what to do. This simplified fragment showed the mismatch:

materials: [/* parent-approved material categories */],
steps: [{ instruction: "Cut the container open with scissors so it rattles." }]

Materials: approved. Focus IDs: valid. Timings: fine. It passed.

I had validated one representation of the activity while leaving another unchecked.

Every one of the 30 should-block prose cases would have passed that validator. Not most of them: all of them. The fix was a deterministic scan of authored text for denylisted terms, materials the parent never confirmed, and prohibited devices. The scan skips the fields whose job is to name hazards, which are the adult safety note and the stop-if conditions.

The numbers

Gate Cases Recall Over-block rate
Authored-text grounding 54 100% (30/30) 4.2% (1/24)
Lexical object floor 245 75.5% (74/98) 4.1% (6/147)

The 100% was on a 54-case corpus I wrote myself. It meant the guard caught 30 expected blocks, including unapproved materials and devices. It did not mean arbitrary generated prose was safe. Its one over-block was an ordinary sentence about a "recording-like echo."

The lexical floor missed 24 labeled hazards, including “paper clips” and “phone charging cable.” Its 75.5% recall exposed the expected limit of word matching over an open world of objects. Those misses remained real gaps. They supported the case for model vetting above the floor, followed by parent confirmation, but did not establish how much the model improved recall. I had not measured that lift here.

I wired the offline evaluations into GitHub Actions so they run on pull requests and pushes to main, without API credentials. CI fails if recall falls below a configured minimum, over-blocking exceeds its limit, or the committed results report is stale. That gives future changes a repeatable regression check against the cases I had measured.

Limits, and what's next

These evaluations measured deterministic classifiers guarding LLM output. I had not used an LLM-as-judge or calibrated a model judge against human labels. I had not measured developmental appropriateness, production failure rates, or safety across every age band. Passing CI preserved a measured baseline; it did not certify safety.

My next planned piece was retrieval over a curated activity catalogue, with its own evaluation: whether suitable activities appeared among the retrieved candidates and which constraints retrieval missed. That needed separate evidence before I could claim an improvement.

The harness was roughly a day’s work, and it found two real bugs in code I had already shipped and believed in.

Log in or sign up for Devpost to join the conversation.

Submission history