SafeSiteAI

Inspiration

Construction loses money in three predictable, expensive ways ,bad designs that violate code, physical defects that go unnoticed during the build, and schedules that blow out because nobody connects the first two to the third. Every tool we found treats these as separate problems: a compliance checklist here, a site inspection app there, a scheduling tool somewhere else. Nobody closes the loop between them.

We wanted to build something that didn't just detect these three failure modes, but proved - visibly, in real time that finding a problem in one layer actually changes your risk forecast in another. Not a dashboard with three tabs. One connected system.

What it does

SafeSite AI is a three-layer construction intelligence platform:

Compliance Engine — upload a blueprint, and a vision-language model extracts spatial measurements (hallway widths, staircase widths, door widths), which get cross-referenced via RAG against the National Building Code of India 2016 to flag real code violations with citations.
Vision Engine — upload a site photo, and the system detects structural defects (spalling, exposed rebar, honeycombing) and audits PPE/site safety compliance against the SH17 taxonomy, zero-shot.
Foresight Engine — a Monte Carlo simulation (10,000 iterations) and linear-programming resource optimizer that recalculates your project's on-time completion probability automatically whenever the first two layers find something new.

That last part is the point of the whole project: in our own testing, a set of real defects flagged by the Vision Engine dropped our on-time probability from 93.7% to 2.7%, automatically, with no manual trigger because the layers are actually wired together, not just visualized next to each other.

How we built it

Backend: Python/FastAPI, Google Gemini for VLM inference, ChromaDB + bge-large-en-v1.5 for RAG retrieval over the NBC 2016 and IS 456 codes, SciPy for the Monte Carlo simulation and LP-based optimizer. Frontend: a dark-mode dashboard visualizing each layer's output plus the live feedback-loop state.

We scoped deliberately: rather than attempt fine-tuned point-cloud segmentation or trained defect-detection models, we leaned entirely on zero-shot VLM inference for Layers 1 and 2, with the architecture explicitly designed so a fine-tuned model (YOLOv11-seg, BIM-Net++) could be swapped in later without touching the rest of the system.

Once the individual pieces worked, we wrote an end-to-end smoke test and ran the actual pipeline real annotated blueprint, real construction photos of spalling and unsafe scaffolding end to end, repeatedly, until every layer produced correct output and the feedback loop measurably responded to new input.

Challenges we ran into

Getting each layer working individually was the easy part. Getting them to work reliably under real conditions surfaced a string of subtle bugs we had to chase down one at a time:

Our resource optimizer returned "infeasible" on its own default data traced to a unit mismatch comparing worker-days against a plain headcount in the LP constraint.
Our risk model would cliff straight to 0% on-time probability for almost any nonzero violation count, the delay multipliers scaled linearly against too tight an on-time threshold, so we recalibrated with square-root damping and a wider band to get a believable, non-binary risk curve.
Gemini kept returning truncated, unparseable JSON it turned out Gemini 2.5 Flash's internal reasoning budget was eating into our output token limit unpredictably. Switching to Gemini 2.0 Flash (no hidden reasoning step) and forcing strict JSON response mode fixed it for good, instead of just raising the token limit and hoping.
Malformed JSON from measurement values containing inch-mark quote characters (3'-0") breaking string escaping, also solved by forcing structured JSON output mode rather than trusting prompt instructions alone.

None of these were visible until we actually ran the full pipeline against real files rather than trusting that individually-tested components would compose correctly.

What we learned

The gap between "the architecture works on paper" and "the system works end-to-end on real data" is where almost all the real engineering time goes — and it's also where most projects quietly stop, leaving that gap for judges to discover live. We'd rather find it ourselves first. Every bug above only showed up because we insisted on running real inputs through the real pipeline instead of trusting each layer in isolation, and fixing root causes instead of symptoms (raising a token limit is a patch; switching to a model without a hidden reasoning budget is a fix).

We also learned to be upfront about scope. Zero-shot VLM inference instead of fine-tuned models isn't a shortcut we're hiding — it's a deliberate engineering call, with a clear line to what production would add.