CleanStack

AI-powered data cleaning with GitHub-style review. Upload messy files, get AI-generated rules reviewed by a 3-member AI committee, approve, and download production-ready clean data.

Comment

Inspiration

Data teams waste 60–80% of their time cleaning data before they can use it. Tools like Fivetran and Airbyte cost $500+/month and need a dedicated data engineering team. Small and mid-size businesses are stuck doing it manually in Excel.

I wanted to build something that gives any team — not just enterprises with big budgets — access to intelligent, automated data cleaning with full transparency and control.

What it does

CleanStack transforms raw, messy data files into clean, production-ready datasets using a three-stage AI pipeline.

  1. Upload — CSV, Excel, PDF, DOCX, JSON, XML (up to 100MB)
  2. AI Profiling — AWS Lambda analyzes data, computes a 0–100 quality score, detects nulls, duplicates, outliers, encoding issues, and PII
  3. Data PR — Claude Sonnet generates domain-aware cleaning rules. A 3-member AI committee (Safety Auditor, Statistician, Domain Validator) votes to approve or reject each rule by risk tier before anything runs
  4. Execute — Serverless Lambda applies approved transforms, re-profiles the output, and triggers schema drift alerts via SNS
  5. Auto-Clean — Up to 3 automated improvement passes with diminishing-returns detection

How I built it

Frontend / API: Next.js 15 App Router deployed on Vercel. All API routes are serverless functions.

Database: Amazon Aurora PostgreSQL via the RDS Data API (no direct TCP connections) — serverless access from Vercel. Multi-AZ, 2 instances, 7-day backup retention, deletion protection enabled.

Storage: Amazon S3 (SSE-KMS encrypted) for raw and processed files. Presigned PUT URLs for direct browser-to-S3 uploads.

Compute: AWS Lambda (Python 3.12) — three functions:

  • cleanstack-profiler — triggered by S3 PUT, parses 9 file formats, computes quality scores
  • cleanstack-executor — triggered by SQS, applies transforms with pandas, detects schema drift
  • cleanstack-drift — triggered by SNS, sends Slack alerts on schema changes

AI: Claude Sonnet 4.6 via Amazon Bedrock + Vercel AI SDK. Three use cases: rule generation (structured JSON), AI committee voting (3 parallel Bedrock calls), and conversational pipeline builder (streaming SSE).

Queue: Amazon SQS for decoupled executor invocation.

Alerts: Amazon SNS → Slack webhook for schema drift notifications.

Auth: Clerk (Vercel Marketplace). Reconciler: Amazon EventBridge fires every 5 minutes to recover stuck pipeline runs.

Accomplishments I'm proud of

  • Data PR concept — treating data cleaning like a code review, with full audit trail
  • AI committee — 3 independent AI personas voting on each rule by risk tier (LOW: 1/3 votes, MEDIUM: 2/3, HIGH: unanimous)
  • 17 industry domains tested — HR, finance, healthcare, legal, manufacturing, telecom, energy, SaaS and more — average quality improvement +18.1 points, zero failures across all 17
  • Auto-clean with diminishing returns — up to 3 automated passes that stop when improvement drops below 5%
  • Multi-format — same pipeline handles tabular data (CSV/Excel/JSON) and documents (PDF/DOCX) with domain-appropriate rules
  • True serverless — ~$5/month for an SME vs $500+/month for incumbent tools

Challenges I ran into

  • Aurora RDS Data API has specific requirements: UUID/TIMESTAMP type hints, serial queries inside transactions (no Promise.all), JSONB handling — all discovered and fixed
  • Lambda deployment size limit (250MB) — solved by patching the existing deployed zip rather than re-zipping from scratch
  • Clerk JWT tokens expire in 60 seconds — required careful session handling for long-running pipeline operations
  • PDF quality scoring — built a custom 0–100 model based on PII density, blank line ratio, encoding errors, and structural noise

What I learned

  • The "Data PR" metaphor resonates immediately with technical teams — they already understand the approval workflow concept
  • AI committee voting dramatically reduces false positives vs a single AI reviewer
  • Quality scoring must be domain-aware — 13/100 on a contract PDF and 70/100 on a CSV represent very different starting points and cleaning strategies

What's next

  • Stripe integration for self-serve upgrades (billing infrastructure live in Aurora — quota, plans, overage columns all reserved; one Stripe webhook away from charging)
  • Pipeline template marketplace (seeded with 4 public templates)
  • Webhook output destinations (Slack, custom endpoints)
  • Excel and XML transform support in executor
  • SOC 2 Type I compliance path

Built With

Share this project:

Updates