-
-
The same 26 questions, bare model against the tool
-
Both rules computed, and the citation opens the provision underneath
-
Same behaviour in Russian, against the Labour Code of the Kyrgyz Republic
-
New York is not in the corpus, so the answer is the gap
Inspiration
Ask any assistant whether overtime is owed and it answers fluently, with a citation that looks right. Before writing a line of this project I measured that on the 26 questions now in the repository: with no sources, the model quoted words that exist in no provision in 16 of its answers, and one answer in twenty-six survived grading. The worst of them described a recordkeeping rule under 29 CFR 531.7 and quoted it - that section is marked [Reserved] and has no text at all.
Someone who is owed money cannot tell a real citation from a well-formed one. That is the problem this is built against.
What it does
It answers a question about pay and working time from three things: the law, the employment contract and the timesheet. Every claim names the provision it rests on, every quotation is matched against that provision in code before it is shown in quotation marks, and everything that could not be established appears under "Not verified" - in the same column, in the same type, as part of the answer rather than a disclaimer at the bottom.
It holds 530 provisions: the Fair Labor Standards Act, four parts of 29 CFR, seven sections of the California Labor Code, and all 265 articles of the Labour Code of the Kyrgyz Republic, in Russian. California Labor Code 226.7 is present in both of its versions - the one in force through 2026 and the one operative from January 2027 - because which applies depends on the date of the shift.
How I built it
Python 3.11 and the standard library. No framework, no vector database, nothing to install.
- Plan. The model proposes the phrases it would look up, in the register of the law: people ask about a "coffee break", the regulation is headed "Rest".
- Filter, then rank. Provisions outside the chosen jurisdictions or not in force on the date of the work are removed before ranking, so a version that takes effect in 2027 cannot win on wording for a shift worked in 2026. BM25 over what remains, with any provision the question names outright fetched directly - including one with no text.
- Draft. Claims are written, each naming one source id. A claim citing anything outside the shortlist never becomes an answer; it becomes a gap.
- Look again. Whatever the first pass could not settle becomes a query of its own. This second round took the graded run from 9 passes to 11.
- Verify. Each claim goes back to the model alone, with the text of the provision and nothing else: not the question, not the other claims, not the reasoning that produced it. A verifier shown the reasoning agrees with it.
- Check quotations in code, compute in Python. Hours and pay are worked out in hours.py and handed to the model as facts.
Planned with the Devpost Learn skill pack; devpost/scope.md, prd.md and spec.md are the documents that drove the build.
Challenges I ran into
Retrieval, and only retrieval. Two hundred sections of one regulation share their vocabulary, and lexical search cannot separate them. Two fixes were measured and thrown away: a model reranking the shortlist by title dropped 29 CFR 785.18, titled "Rest", from second place for a question about a coffee break and turned the answer into a refusal; handing the model the whole table of contents, 6,800 tokens a question, did no better than the search it was meant to replace.
The cheap bugs cost the most. The model writes "id:CALAB226.7" where the shortlist says "CALAB226.7", and treating that as an unknown source was throwing away correct claims. In Russian the answer was being cut off mid-JSON by the token budget, so every Russian question failed for a reason that had nothing to do with Russian.
Accomplishments that I'm proud of
Against the same 26 questions, the bare model with the same sources passes 13 and this passes 11 - but it shows no altered quotation as a quotation where the bare model shows three, cites nothing that was ruled out where the bare model does it four times, and reports the gap in all four questions that have no answer in the corpus, where the bare model reports two.
Right less often, wrong less dangerously.
What I learned
That grounding and honesty are different problems. Giving the model the sources cut invented quotations from 16 to 3 - and raised the number of times it cited something it had been told to ignore, from 1 to 4. More text in front of it meant more to reach for.
And that the instruction which fixes fabrication has a price. Told it may answer "not in the corpus", the model reported the gap in all four questions where that is the only honest answer - and also declined 12 of the 22 questions it could have answered. Both halves of that trade-off are only visible if you run both conditions, which is why the eval set carries controls.
Also: arithmetic does not belong to the model. Asked to work out a week's pay it went from the correct $637.00 to $757.36 inside one answer and presented the second figure as a correction. That computation now lives in Python, and the answer shows the working.
What's next for Chapter and Verse
A third search round, one per clause, so a contract review with six unlawful clauses enumerates all six instead of verifying one or two thoroughly. Then a way to add a jurisdiction without editing a file.
Honest limits
Retrieval finds about seven in ten of the provisions a question needs; when it misses, the tool reports a gap instead of inventing, but the answer is incomplete. It is a proof of concept: two corpora, frozen at a date. It shows provisions and arithmetic; it is not legal advice.
The corpora, the eval set and the sample documents were assembled on 24 September 2026, inside the submission period, and are in the repository with the scripts that built them. The pipeline, the interface, the graders and the video came after them, the same day.
Log in or sign up for Devpost to join the conversation.