Inspiration
The lack of a Kenyan citator.
In 2017 Kenya's Supreme Court held, in Francis Karioko Muruatetu & another v Republic, that the mandatory death sentence for murder is unconstitutional. Yet in the 2022 and 2023 versions of Kenya's official consolidated statutes, Penal Code s.204 still reads: "Any person convicted of murder shall be sentenced to death." Criminal defamation (s.194) was struck down in part in Okuta (2017), and the official text carries no note of that either. We examined 1,038 editorial remarks across the statute pages we collected. Not one mentions a court decision. They are all amendment history.
So a lawyer, a student, or an AI assistant reading the official text can be confidently wrong about the law. In the United States, citators like Shepard's and KeyCite exist to solve this problem. Kenya has none. And with lawyers worldwide being sanctioned for filing AI-hallucinated citations, a tool that checks whether a legal source says what it claims and is still good law sits squarely in the AI Safety & Governance track.
That's Hakiki (Swahili: verify).
What it does
Hakiki tells you whether a section of a Kenyan statute is still good law: in force, amended, repealed, or limited or struck down by a court. It works by linking court judgments to the specific sections they affected.
Its central design rule is never a simple yes/no. Courts often strike a section down only "to the extent that" it does something, and a section can have a history: read down by one court, then reversed on appeal. So instead of a red/green badge, Hakiki reports:
Penal Code s.204: limited by the Supreme Court in Muruatetu [2017]. Unconstitutional to the extent it makes the death sentence mandatory. Otherwise in force. [court's words quoted verbatim, with the paragraph and a link to the judgment]
Every status comes with the court's own words, quoted verbatim, and a timeline of every event that touched the section. Hakiki reports what the sources say; it never gives legal advice.
On top of this sits the filing checker: paste in a legal document, and Hakiki checks each citation in it three ways.
- Does the case exist? It is matched against our database of judgments, which is how hallucinated cases get caught.
- Does the quoted passage actually appear in it? The quote is compared with the judgment text, and a misquote is shown word by word.
- Is the cited section still good law? A citation that relies on a section a court has struck down or limited is flagged, with the court's words.
How we built it
1. Data. We collected statutes and judgments through Internet Archive snapshots of Kenya Law, stored byte-for-byte as WARC files: 8 Acts in 31 dated versions and 16,419 judgments from the Supreme Court, Court of Appeal and High Court. Each statute section carries a stable ID (an Akoma Ntoso eId, such as part_II__chp_XVIII__sec_204) that stays the same across versions, so we can follow one section through time.
2. A verified answer key. Before building any automation, we hand-built the ground truth: 19 events (what a court did to which section, with the operative words quoted verbatim) and 6 negatives (judgments that mention a section without affecting it). Each was audited blind, and a script checks that every quote appears in its source judgment.
3. Citation extraction. A regex grammar finds every citation of the form "section 204 of the Penal Code" or "Article 50(2) of the Constitution". It handles lists, ranges, "as read with", headnote formats, acronyms and Cap numbers. That produced 304,310 citations across 13,098 judgments. For the citations that don't name their law (a bare "section 39", or "the Act"), an LLM pass picks the law from the ones that judgment actually names. It isn't allowed to invent one, and every answer is validated in code. This resolved 41,382 citations.
We scored extraction on a held-out sample of 30 judgments that had never been used for tuning. Each was labelled independently by two annotators, blind to the extractor and to each other, and they agreed on 469 of 471 candidates:
$$ P = \frac{TP}{TP + FP} = \frac{410}{418} = 98.1\%, \qquad R = \frac{TP}{TP + FN} = \frac{410}{418} = 98.1\% $$
With the LLM pass, the cited law is identified correctly for 97.8% of held-out citations and wrongly for 1.0%. The rest are left unresolved rather than guessed.
4. Event classification. An LLM reads each candidate judgment and records what the court did to each cited section: declared it unconstitutional, read it down, upheld it, or reversed an earlier ruling. It also records the scope, total or partial. A quote is kept only if it appears verbatim in the judgment. A second-pass checker then asks whether each event is the court's own holding, on the right section. On blind-reviewed events, this raised precision from about 59% to about 80%.
5. Status resolver, API and app. A per-jurisdiction rules module combines a section's events into its current status, including direct reversals and later precedent from a higher court. A FastAPI service serves acts, provisions, search and the filing checks. The Next.js front end brings it together: search, a provision page with status, verbatim quotes and timeline, and the filing checker, demonstrated on clearly labelled synthetic filings.
Stack: Python, Postgres with pgvector on Neon, hybrid search (full-text plus embeddings, merged by reciprocal rank fusion), FastAPI, Next.js, Gemini and DeepSeek for the LLM passes, all cached to disk by input hash.
Challenges we ran into
Getting reliable data within the hackathon period. This was our biggest challenge. Kenya Law's live site returns HTTP 403 to automated clients, and its robots.txt blocks AI crawlers by name. So we worked from Internet Archive snapshots, which hold only about 10% of Kenya's judgments. Some statute versions were missing: we have 2 of the 13 listed versions of the Kenya Information and Communications Act. We saved key pages by hand where the archive lacked them. 286 judgments exist only as PDFs, with no text. Kenya Law's own metadata links judgments to legislation only at whole-Act level, and often not at all: Muruatetu doesn't link the Penal Code. So every citation had to be extracted from the judgment text itself. We also had to settle whether the data could be used at all; legal advice to the team confirmed it may be used as public domain, for non-commercial purposes.
The law is not binary. Muruatetu didn't kill s.204. It removed only the mandatory element, and follow-up directions in 2021 narrowed it further. The Sexual Offences Act s.8 was read down by the Court of Appeal, then reversed by the Supreme Court. Modelling this meant storing every event separately and deriving status from the sequence, not from a single flag.
LLMs that confidently guess. Our first event classifier had about 53% precision on a blind review. Courts that merely follow another ruling looked like rulings themselves, and ordinary appeals looked like reversals. Prompt revisions, verbatim-quote validation and a second-pass checker brought it to about 80%. One model's JSON mode doesn't enforce answer schemas at all, so every answer is validated in code.
Budgets and flaky infrastructure. A free-tier quota of about 500 requests a day would have taken about 15 days to finish citation resolution. We switched models and ran 8 parallel workers for about \$2. The connection then dropped repeatedly mid-run: workers crashed on cut-off replies and hung on database connections that had silently died. We added retries, TCP keepalives and statement timeouts, and made every step resumable from a content-addressed cache, so a restart never paid twice for the same answer.
Accomplishments that we're proud of
- A finding worth publishing on its own: Kenya's official consolidated statutes don't record court decisions that struck sections down. We show it with specific sections and dates.
- An end-to-end citator for Kenya, from raw archived web pages to a searchable app and a filing checker, built within a hackathon.
- Honest measurement. Every score comes from verified answer keys, including a held-out sample labelled twice, blind, with $\frac{469}{471} \approx 99.6\%$ agreement between annotators. Where a number is optimistic, because we tuned on that sample, we say so.
- Verbatim or nothing. No court's words are ever paraphrased into a status; a quote that can't be found character-for-character in the source is dropped.
- Scale on a shoestring: 16,419 judgments, 304,310 citations, and 41,382 resolved by an LLM, for a few dollars of API spend.
- A design that extends beyond Kenya: every ID carries its jurisdiction (
ke/act/cap-63/...), and status rules live in a per-jurisdiction module.
What we learned
- Build the answer key first. Having verified ground truth before writing the extractor let us catch real bugs, such as an Article wrongly attributed to the Elections Act in "article 178(1) as read with section 21(1) of the Elections Act". It also stopped us fooling ourselves.
- Tuning on your test set inflates your numbers. Our extraction recall looked like 98.5% on the sample we tuned on. The disjoint held-out sample gave us numbers we can actually defend.
- LLMs are good extractors but poor judges of legal effect unless you constrain them: fixed option lists, verbatim-quote checks, and a second, skeptical pass.
- Partial validity is the norm, not the edge case, in constitutional litigation. A citator that outputs red/green would be wrong in exactly the cases that matter most.
- Engineer for failure: caches keyed by content hash, idempotent loaders, and resumable, sharded jobs turned network outages from disasters into delays.
What's next for Hakiki: verify any Kenyan statute
- A human review queue, so legal experts can confirm or reject the events the model extracts. Each confirmed event joins the verified set and raises precision.
- Parliament's changes as events: turn more than 1,000 amendment notes into
amended_by_statute/repealed_by_statuteevents, so the timeline shows both courts and Parliament. - Full coverage: the Archive holds about 10% of judgments, so the next step is a proper data partnership with Kenya Law, plus more Acts beyond our current 8.
- Deeper filing checks: go beyond "does it exist and is it quoted right" to "does this case actually support the argument", a harder research problem we deliberately left out of the hackathon build.
- Beyond Kenya: the jurisdiction-scoped design means other common-law countries whose official texts lag behind their courts could be added with a new rules module.
Built With
- nextjs
- python