Sherlock

Sherlock is an AI data analyst you can audit: it cleans messy CSVs with tested, reversable fixes, then answers any question, and every number links to the query that produced it.

Comment

Inspiration

I grew up on the Sherlock Holmes games, and my favourite part was always the mind palace: Sherlock steps inside his head, every clue he has collected is laid out in front of him, and he connects them into a deduction he can explain step by step.

I work with data, and my days don't feel like that. I love the end of the job: finding the story in the data and telling it. But first comes the part I dread. Opening a new CSV, poking around, fixing broken dates, hunting down N/A and -999, merging "NYC" and "new york " back together. It eats most of my time, and it's the least fun part. The most frustrating part is, while cleaning I don't know what to clean. Its like a "YOU DONT KNOW WHAT YOU DONT KNOW" situation. So being able to almost foresee how the data will look like if I were to apply a cleaning command is a game changer.

So I built the tool I wanted for myself: a mind palace for a dataset that does the cleaning and digging for me, and, like Sherlock, can always tell me how it knows.

What it does

You drop in a messy CSV. Sherlock finds what's wrong with it and fixes it, one issue at a time, and every fix can be undone. Then you can ask it anything, like "average fare by class" or "what drives survival?". Or you can manually clean with by typing cleaning commands in natural language into the chatbot. Every number in the answer is clickable and shows you the exact query behind it. When you're done, you can export everything as a Jupyter notebook that rebuilds your clean data from the original file.

How we built it

My one rule was: the AI decides, the code calculates, I verify I didn't want a model making up numbers, so the AI plans and writes code, and only code that actually ran puts numbers on the screen. The AI isn't even allowed to type a digit in its answers.

For cleaning, the AI suggests a few possible fixes for each problem. Each fix is tested on a copy of the data, checked by code, and kept only if Sherlock is confident enough:

$$ \text{confidence} = 0.6 \cdot \text{checks passed} + 0.2 \cdot \text{fixes agree} + 0.2 \cdot \text{second opinion} $$

If it isn't sure, it skips the fix or asks me. Nothing is ever deleted. Rows it can't fix are set aside with a reason.

It runs on NVIDIA Nemotron models through Nebius Token Factory: Nemotron 3 Ultra (550B) writes the fixes, Nemotron 3 Super (120B) answers questions, and Nemotron 3 Nano (30B), the small, fast one, works out what you’re asking. The back end is Python with DuckDB, the front end is React, and Tavily looks up what column names mean on the web, so the models work with data specific context.

Challenges we ran into

  • The AI kept getting creative with numbers. I had to make "no numbers in your own words" a hard rule enforced in code, not a polite request in the prompt.
  • Fixes that went too far. One test run set aside over 1,000 rows to fix a problem in just 4 of them. Another tried to merge "Austria" into "Australia". Each mistake became a new safety check.
  • Dates. Is 03/04/2025 March or April? When Sherlock can't tell, it asks instead of guessing.
  • Things outside my control. The token factory would go down momentarily, not sure if its on my end. I specially noticed it while running Claude automated user testing at night. Claude learned to wait a minute to try calling the models again. During the day I noticed the token factory server go down 5 seconds at a time maybe like once a day. The deprecation was announced about 3 hours before the submission deadline. I hope we are judged on the demo because this is too last minute to deal with. The live site will be up till Nov 7 as outlined by Nebius.

Accomplishments that I am proud of

  • On the benchmark files I ruined on purpose. Legit turned it to shreds. Sherlock recovered 97% of the damage on the Titanic-based set without breaking anything that was already right.
  • It answered 60 of 61 test questions, and no answer had a number without a source.. Getting it to source the correct data cells was the hardest part - sometimes it would get mixed up with similar numbers
  • The exported notebook really does rebuild the clean data from the raw file, every time. Oh man this one took one out of me. For 1 week, it was giving unordered executions.
  • I work at a wealth management firm and there are lots of numbers that get passes around from teams to teams and sometimes the "Cleaning Standard" vary team to team. A lot of back and forth happens between teams on who cleaned what part of the dataset and "what was the cleaning pandas code ran" questions. My agent automates a lot of their work and leaves audit trails on exactly what was done to the dataset.
  • I'd actually use it myself, its not perfect but respectable.

What I learned

The hardest and part was building the checks that decide when to trust it. I also learned I don't need an AI that's always right. I need one that shows its work so I can check it. If it didn't do the cleaning task correctly, I would know exactly where it went wrong and would just undo then do a targeted prompt so it would work fine the next time.

What's next for Sherlock

  • Massive Detection Corpus, right now we only have 24 deterministic fault detectors, this is the actual core of what makes the entire agent enterprise level. The bigger the corpus gets the better.
  • Bigger and wider files, and cleaning several columns at once so it's faster.
  • Smarter matching, so "ticket price" finds the fare column.
  • Reusable cleaning: run the same steps on next month's file automatically.
  • Joining files and connecting straight to databases and Google Sheets.
  • Accounts and share links, so you can keep your work and send an answer to a colleague with its proof attached.

Built With

Share this project:

Updates

Submission history