Site Diary AI

Dashboard
Demo video we uploaded and get result
we get material details
risk analysis
transcript

Inspiration

Construction sites in India run on foremen who know exactly what happened on site today — but most of them aren't comfortable typing in English, don't have time between managing labor and firefighting, and have zero patience for filling out a form on an app. So daily progress reports get skipped, backfilled from memory days later, or reduced to a one-line WhatsApp message that says nothing useful.

Meanwhile, project managers upstream are making schedule, payment, and resourcing calls on data that's stale, vague, or missing entirely. We kept coming back to the same realization: this isn't a tooling problem. Every construction-tech product we looked at assumes the user is a PM with a laptop. Almost none of them assume the user is a foreman with a basic phone who'd rather just say what happened than type it. That gap is what we set out to close.

What it does

Site Diary AI lets a foreman record a voice note — in Hindi, Hinglish, or a regional language — describing their day. No app to open, no form to fill, no English required.

Behind the scenes, the system:

Transcribes the audio
Translates and structures it into a clean daily progress report (work completed by location/task, labor present or absent, material status, issues raised)
Cross-checks that report against the project's schedule for that day, and automatically flags anything that's behind
Surfaces a clean, PM-facing summary with slippage and blockers at the top — not buried in a paragraph nobody reads
Reads a short confirmation back to the foreman in their own language, so they can trust the system actually understood them correctly, instead of it being a black box

You can try the deployed version here: (https://site-diary-ai-git-main-pranjals-projects-e2ff7a07.vercel.app/)

How we built it

We used a Python-first stack we were already comfortable with from prior RAG projects:

Whisper for speech-to-text, chosen for its handling of Hindi and code-switched speech
LangChain + Groq for translation, structured extraction, and the schedule cross-check logic
FastAPI to orchestrate the pipeline from audio in to structured report out
React for local prototyping of the pipeline, with the final demo deployed live on Vercel so judges can try it directly instead of just watching a video

For the MVP, we mocked a project schedule as a JSON file and skipped real telephony integration entirely — that's infrastructure we didn't have time to build in a day, and it wasn't the part we needed to prove. What mattered was proving the full loop end to end: voice in, structured and schedule-aware intelligence out — and getting it actually deployed and clickable, not just running locally.

Challenges we ran into

Code-switched speech is messy. Foremen don't speak pure Hindi or pure English — they mix both mid-sentence, and getting a clean, meaning-preserving transcript out of that (without losing specific quantities or locations) took more prompt iteration than we expected.
Deciding what not to build. It was tempting to reach for live telephony or a fancier dashboard, but with one day on the clock we had to be ruthless about scoping down to the smallest version that still proves the real idea.
Trust, not just accuracy. We realized early that transcription accuracy alone wouldn't be enough — if a foreman can't tell whether the system actually understood them, they won't keep using it. That's what pushed us to add the confirmation-readback step, which ended up being one of the more important design decisions in the project.
Getting it live in time. Deploying end-to-end on Vercel under deadline pressure, instead of leaving it as a local-only demo, meant working through environment and API-key configuration issues in the final hours.

Accomplishments that we're proud of

Getting a full end-to-end loop working and deployed live in a single day — voice note in, structured schedule-aware report out — instead of settling for a half-built or local-only demo.
The confirmation-readback feature. It wasn't in our original plan, but once we realized trust was the real adoption barrier — not just transcription accuracy — we prioritized building it, and it's the detail most people react to first.
Handling code-switched Hindi-English speech reasonably well without needing a custom-trained model, just careful prompt design on top of Whisper + Mistral.
Designing this around the person who's actually excluded by most construction-tech today, instead of building yet another PM dashboard.

What we learned

Real adoption problems in construction aren't usually about AI capability — they're about literacy, language, and friction. The smartest model in the world doesn't help if the person who needs it won't use it.
Ruthless scoping matters more than ambition under a tight deadline. We cut real telephony integration, text-to-speech, and a fancier dashboard early, which is exactly what let us finish a working, honest, deployed prototype instead of an unfinished ambitious one.
Trust has to be designed for explicitly — it doesn't come for free just because the transcription is accurate. The confirmation loop taught us that a good AI product sometimes needs a step that isn't about intelligence at all, just reassurance.

What's next for Site Diary AI

Real telephony/IVR integration, so a foreman can call a number instead of uploading a file — removing the last piece of friction between "having something to report" and it reaching the system.
Text-to-speech for the confirmation loop, so the readback is actually spoken back, not just displayed as text.
Support for real project schedules (MS Project/Primavera exports) instead of a mocked JSON file, so the schedule cross-check works on live projects out of the box.
Expanding beyond daily diaries to a running history — letting a PM ask questions like "how many days was Building B's shuttering behind this month" across weeks of logged reports.
Piloting on a real site to test the thing that matters most: whether foremen actually keep using it past week one.