• 

  Screenshot of Indian Law 100: Legal AI Micro-Case Benchmark

Indian Law 100

Inspiration

Legal AI benchmarks often test whether a model can recall legal information or answer examination-style multiple-choice questions. That does not fully reflect what legal professionals need: applying the correct law to a specific set of facts, identifying the governing legal basis and avoiding plausible but legally outdated answers.

This problem is particularly important in India following the replacement of the Indian Penal Code, Code of Criminal Procedure and Indian Evidence Act with the Bharatiya Nyaya Sanhita (BNS), Bharatiya Nagarik Suraksha Sanhita (BNSS) and Bharatiya Sakshya Adhiniyam (BSA).

I wanted to build a benchmark that measures whether AI models can apply current Indian law to realistic legal scenarios rather than simply recall legal knowledge.

What it does

Indian Law 100 evaluates an AI model using 100 legal micro-cases covering 13 areas of Indian law.

The benchmark includes 40 easy, 40 medium and 20 hard cases. Several cases deliberately contain legal traps involving repealed statutes, IPC-to-BNS mapping, procedural changes, amendments, limitation periods and outdated precedents.

Each response is evaluated across three dimensions:

• Outcome (50%) – Did the model reach the correct legal conclusion?
• Legal basis (30%) – Did it identify the correct statutory provisions or legal principles?
• Reasoning (20%) – Was its legal reasoning sound?

The benchmark produces an overall score together with domain-wise performance, difficulty-wise performance, trap accuracy, runtime and API cost.

How I built it

I built the application using HTML, CSS and JavaScript, with PHP handling authenticated result submission and leaderboard generation.

The benchmark connects to OpenRouter, allowing users to test virtually any supported AI model using the same dataset and evaluation process.

Each case is sent independently to the selected model. The model's response is then graded by either an independent judge model or, if preferred, the same model.

Every answer is displayed alongside the expected answer and detailed scoring, allowing manual review and score overrides where necessary.

To make long benchmark runs practical, I added automatic progress recovery using IndexedDB, JSON export of complete benchmark sessions, authenticated server-side result storage and a public leaderboard for comparing model performance.

Challenges I faced

The biggest challenge was designing good benchmark questions.

Many legal questions are either too subjective to score consistently or so simple that nearly every model answers them correctly. I spent considerable time creating concise legal micro-cases that have objectively verifiable answers while still requiring genuine legal reasoning.

Another challenge was AI-based evaluation. Judge models do not always return perfectly structured outputs, so I implemented validation, repair logic and manual review tools to ensure grading remained reliable.

Finally, benchmarking large models can require hundreds of API calls. I therefore built automatic checkpointing and recovery so interrupted runs can resume without losing progress.

What I learned

Building the benchmark taught me that the dataset and evaluation methodology are at least as important as the software itself.

A carefully designed legal micro-case often reveals meaningful differences between models that score similarly on traditional legal benchmarks.

I also learned that transparency matters. Recording every answer, score, judge explanation and manual adjustment makes benchmark results far more trustworthy than publishing a single aggregate score.

What's next

I plan to expand the benchmark with additional legal micro-cases covering more areas of Indian law and, eventually, other jurisdictions.

Future versions will measure not only legal accuracy but also consistency, citation quality, hallucination rates and long-term model reliability.

My goal is to create an open, practical benchmark that helps developers, researchers and legal professionals evaluate how well AI systems apply real-world law—not just how well they answer legal exam questions.

Built With

• ai
• api
• css3
• html5
• indexeddb
• javascript
• json
• local
• openrouter
• php
• rest
• storage