Inspiration
If you've ever worked with data, you've probably had this happen. You open a dashboard and revenue is up 100x overnight. For a second you're excited and your excitement reaches the moon but then comes crashing down when you realize one system started saving amounts in cents and nobody caught on to it. Nothing crashed. The pipeline ran fine. The numbers were just wrong.
The same data usually lives in a bunch of places as well such as a warehouse, a store database, an online shop, some spreadsheet on Google Drive, a folder of PDF invoices. Every one of them names things differently. Code has tests and CI, but data mostly doesn't. So, we wanted something that checks your data every time it moves and notifies you before a bad number ends up where it’s not supposed to be.
What we built
We built DataPulse, which connects to your sources (Postgres, MySQL, SQL Server, MongoDB, files, Google Sheets and Drive, REST APIs, even PDFs and scanned documents) and turns them into one clean table. Then it checks every new batch against a trusted baseline and tells you what broke, and whether it's bad enough to block.
How we built it
There were five of us and about two days of time and a lot of caffeine. DataPulse’s backend is Python and FastAPI, the dashboard is React and TypeScript, and the AI parts use Claude.
For unifying, the AI looks at a small sample and writes a mapping, like "divide amount_cents by 100" or "turn "$61.31" into a number." Then plain code applies that mapping to every row. We did this so the model isn’t touching every single record.
For checks, we use regular rules (missing columns, empty values, duplicates and so on) plus the Population Stability Index for numeric drift:
Our favorite part is the ML drift check. We train a random forest to guess whether each row came from the old data or the new data. If it can only guess (AUC around 0.5), nothing changed. If it can tell them apart easily, something shifted, and the model tells you which columns gave it away.
Challenges
The drift model flagged literally every batch at first. Turns out IDs and timestamps are always different between batches, so the model just learned "new IDs are bigger." Skipping those columns fixed it.
Getting tables out of PDFs without losing rows took way more tries than we expected.
The biggest one was scope. At one point we had an AI triage agent, an AI repair agent, Slack alerts, trend charts, and three levels of severity. Then we pictured someone opening a check with 300 issues and realized they'd just close the tab. So we cut a lot of stuff we'd spent hours on. Now there's a short AI summary at the top and a list split into blocking and not blocking. It was painful, but the product is way better for it.
And of course, five people in one repo for two days meant a lot of merge conflicts.
What we learned
Let the rules decide what passes and fails, and let AI explain things. The AI never gets the final say, which is what makes the results trustworthy. We also learned that deleting features is harder than building them, and usually the better call.
Built With
- css
- other
- python
- typescript
