Find the test an AI patch missed, then keep it as a regression test you can rerun in your browser or CI.
Run the test Gemini missed. The recorded patch opens at 4/4. One click adds the boundary case and takes it to 4/5. The walkthrough runs in your browser without an account or model download.
Reproduce three recorded repair trials. All three Gemini drafts passed the four examples they were shown. On cases set aside before the requests, checkout and ticket ordering each passed 7/8, and Unicode contact labels passed 4/4. The two misses involved unsafe integers. See the exact sources, results, and replay command.
The problem
The checkout had a free shipping option, but the function picked the paid quote and reordered the caller’s list. Gemini fixed both problems and passed all four examples it received. One input still broke the patch. A price outside JavaScript’s safe-integer range returned a quote when the requirement said to return null.
I built Local Pit Crew for developers reviewing AI-assisted changes to small JavaScript functions that make consequential decisions, such as choosing a shipping quote or deduplicating customer labels. The bench shows both what a function returned and whether it changed the data passed to it. That second check matters when the answer looks right but the caller’s data has changed.
How it works
Bring a synchronous JavaScript function and a few examples. Run the baseline, inspect the first unexpected result, review a replacement, and apply it in memory. A fresh QuickJS worker reruns the same examples against the new source. Add another example to probe a passing patch, then save the source and fixed examples as a replay.
A teammate can compare the replay with their working JavaScript file in the browser or run both versions in CI and produce a JUnit report. The comparison shows which examples were fixed, which regressed, and which still fail with a different result. The shareable summary contains check IDs and version hashes, with source and example values omitted.
How Firebase AI Logic is used
The on-device action uses Firebase AI Logic in Chrome with ONLY_ON_DEVICE to produce a source-linked repair hypothesis. The app shows the SDK’s reported inference source. For a second opinion, cloud patch, or new test suggestion, the app shows the exact prompt before the user sends it through a separate ONLY_IN_CLOUD action. Cloud requests use Firebase App Check. The on-device action never falls back automatically to cloud inference.
The video shows a saved tag-repair run. Chrome’s on-device hint pointed to the filter but missed why it dropped valid names. After an explicit Send, the cloud reviewer identified that inherited properties made those names look like duplicates. I reviewed and applied an authored correction, and the browser went from 3/5 to 5/5. Live on-device generation depends on Chrome’s model availability. If it times out, you can continue with the same source and examples or explicitly request cloud review.
Try it
Public Apache-2.0 source · Demo video
- Open the missed-test walkthrough and select Run the unsafe-price check. The new example expects
null. The recorded patch returns the"too-high"quote. - Select Choose a repair, then Fill reviewed safe-integer fix and Review pasted patch. Read the diff and select Apply in memory.
- Select Verify my counterexamples. The same five examples now pass. Save the checked replay to rerun it in the browser or CI.
For the Firebase path, open the video’s failing tag example in desktop Chrome. The browser runs the five checks and shows the first wrong return at 3/5. Select Ask Nano about this failure. With a supported model ready, look for the SDK’s inference-source label. Review cloud payload previews a second opinion without the local hypothesis. If a hint is available, Review second opinion with Nano hypothesis previews what you would share. The result puts that exact hypothesis beside Gemini’s opinion. Either action requires a deliberate Send this prompt to Google.
Test after a restart. While connected, open Runtime & setup → Save bench for offline use in the checkout walkthrough. When the save finishes, close Chrome and reopen the walkthrough offline. The verifier can finish the repair and rerun a downloaded replay. With Chrome’s model ready, Ask Nano about this failure also works offline.
To reproduce the same regression in CI, run these commands from the repository root with Node.js 24 or newer.
npm ci --ignore-scripts --prefix local-pit-crew/runner
npm run replay:function -- examples/checkout/gemini-draft.replay.json --compare-source examples/checkout/shipping-quote.js
The comparison reports 4/5 → 5/5, with only case-05 repaired and zero model calls.
During the submission period, I built the browser workbench, the Firebase diagnosis and review flows, and the browser-to-CI replay workflow. Codex helped with implementation, testing, synthetic repair pilots, and verifying the clean source archive.
Built With
- firebase-ai-logic
- firebase-app-check
- firebase-hosting
- gemini
- javascript
- node.js
- quickjs
- webassembly