Sharmik-Sevak

Inspiration

Construction sites are some of the most dynamic and high-risk work environments, yet the information meant to protect workers is often trapped inside safety manuals, permits, inspection checklists, toolbox talks, and static warning signs.

The problem is not that safety knowledge does not exist. The problem is that it rarely reaches the worker at the exact moment it is needed.

A worker may know that a ladder should be secured or that a crane has a dangerous swing radius, but traditional systems cannot warn them that this specific ladder is unstable, that a suspended load is moving behind them, or that the electrical panel in front of them has no active lockout permit.

Most existing construction-vision systems focus on fixed CCTV feeds and basic PPE detection. We wanted to go further and explore what construction safety could look like if AI could see from the worker’s own point of view, understand what was happening over time, predict danger before an accident occurred, and provide immediate hands-free guidance.

That idea became ShramikSevak — Digital Risk Intelligence and Safety Hazard Tracking Interface.

Our guiding framework is:

SEE → UNDERSTAND → PREDICT → GUIDE → ACT

What it does

ShramikSevak is an edge-first multimodal safety copilot designed for construction workers using smart glasses.

The proposed system observes the job site from the worker’s first-person perspective and combines:

real-time object and hazard detection,
worker and machinery tracking,
depth and proximity estimation,
human-pose and activity understanding,
short-horizon trajectory prediction,
construction-document retrieval,
multilingual voice guidance,
and automated supervisor escalation.

Instead of merely detecting that a forklift or crane is present, ShramikSevak aims to determine whether it is moving toward the worker, whether their trajectories may intersect, how serious the danger is, and what the worker should do next.

For example, the system could issue warnings such as:

“Caution. Vehicle approaching from behind. Step left.”

“This ladder appears unsecured. Stabilise it before climbing.”

“Panel E-17 is marked energised. No active lockout permit was found.”

A companion supervisor dashboard provides:

live site and worker status,
active hazard alerts,
incident timelines,
AI-generated explanations,
relevant safety-rule evidence,
risk-zone visualisations,
and emergency escalation information.

One of the central ideas behind ShramikSevak is the use of predictive risk cones. Rather than waiting for a violation or collision to occur, the system estimates forward-looking danger zones such as a forklift’s projected path, a crane’s swing radius, a ladder’s possible fall envelope, or a worker-machine trajectory intersection.

To reduce alert fatigue, the system is also designed to suppress repeated warnings, require hazards to persist before escalating, and choose the smallest useful intervention based on severity.

How we built it

For Stage 1, we focused on designing the complete product architecture and creating a polished concept demonstration that communicates how the system would behave in a real construction environment.

The concept demo was built as a cinematic simulation rather than a working production system. It uses construction-site video footage with scripted, time-synchronised hazard overlays, warnings, risk readouts, and dashboard alerts. This allowed us to demonstrate the intended worker experience and supervisor workflow clearly while keeping the prototype honest about its current stage. :contentReference[oaicite:1]{index=1}

The concept interface uses:

Next.js
TypeScript
Tailwind CSS
Framer Motion
timeline-driven scenario configuration
construction-site footage with synchronised HUD overlays
a simulated worker smart-glasses view
a simulated supervisor command centre

The proposed production architecture is hierarchical and edge-first.

A fast perception path would continuously run detection, tracking, depth estimation, pose analysis, and geometric safety rules on NVIDIA edge hardware. A slower multimodal reasoning path would only be activated when the fast path identifies a suspicious or ambiguous event.

The planned technical stack includes:

Meta smart glasses
an Android edge gateway
NVIDIA Jetson-class hardware
NVIDIA DeepStream and TensorRT
real-time object detection and multi-object tracking
monocular depth estimation
human-pose estimation
vision-language models for activity reasoning
retrieval-augmented generation over project documents
a safety knowledge and workflow layer
a FastAPI-based backend
PostgreSQL, Redis, and a vector database
a Next.js supervisor dashboard

We designed four core Stage 2 prototype scenarios:

Dynamic machine collision warning
A worker enters the projected path of a forklift or excavator, and the system predicts the trajectory intersection before issuing a warning.
Working-at-height risk
The system detects an unsecured ladder or unguarded edge, checks relevant PPE, and retrieves the applicable safety procedure.
Procedure-aware tool use
The system identifies the tool and activity and checks only the protective equipment required for that specific operation.
Fall and emergency response
A worker falls and remains motionless, prompting the system to ask for acknowledgement before automatically notifying the supervisor with location and supporting evidence.

Challenges we ran into

One of the biggest challenges was avoiding the temptation to turn the project into another generic helmet-detection system.

PPE detection is useful, but it does not fully capture the complexity of construction safety. A worker may be wearing every required piece of equipment and still be standing inside a crane’s swing radius or approaching an energised panel without authorisation.

This forced us to think beyond single-frame object detection and design a system that combines perception, temporal reasoning, site documentation, and physical risk prediction.

Another challenge was balancing technical ambition with real-time feasibility. Running a large multimodal model on every video frame would be too slow and computationally expensive. We therefore designed a fast-path/slow-path architecture in which lightweight perception operates continuously while deeper multimodal reasoning is invoked only for significant events.

We also had to think carefully about alert fatigue. A system that speaks every time it sees a tool, helmet, vehicle, or warning sign would quickly become unusable. The Alert Policy Engine was introduced to prioritise hazards, suppress repeated notifications, and escalate only when the risk becomes meaningful.

Smart glasses also introduce serious privacy concerns. We addressed this by designing the system around edge processing, limited incident-clip retention, bystander face blurring, role-based access, and storing structured hazard records instead of continuous video wherever possible.

Finally, the Stage 1 timeline required us to communicate a complex Physical AI system clearly without misrepresenting a concept simulation as a completed product. We therefore separated the current scripted demonstration from the planned Stage 2 functional prototype.

Accomplishments that we're proud of

We are proud that ShramikSevak is designed as more than a computer-vision dashboard.

The system completes the full loop:

SEE → UNDERSTAND → PREDICT → GUIDE → ACT

It does not only identify objects. It attempts to understand worker activity, estimate where danger is heading, retrieve the rule that applies to the exact situation, communicate a short actionable warning, and trigger the appropriate workflow.

We are also proud of the predictive-risk approach. Dynamic risk cones make the system proactive rather than reactive and provide a clear, explainable reason for every alert.

Another major achievement was designing the system to remain useful in low-connectivity environments. Critical perception, risk evaluation, and voice warnings are intended to run locally at the edge, while cloud systems are used only for deeper analysis, reporting, and cross-site intelligence.

We also created a complete Stage 1 package consisting of:

a detailed written proposal,
a 10-slide visual deck,
a two-minute concept video,
a worker point-of-view simulation,
a supervisor dashboard,
a technical architecture,
a four-scenario prototype plan,
and measurable Stage 2 targets.

Our target metrics include:

more than 20 FPS fast-path processing,
local critical-warning latency under 500 ms,
spoken guidance within 1.5 seconds,
full availability of critical functions offline,
and evidence-linked explanations for every alert.

These remain prototype targets to be evaluated during development rather than results already achieved.

What we learned

This project taught us that building useful Physical AI is not simply a matter of choosing the largest vision-language model.

A strong real-world system requires several specialised components working together:

fast perception,
temporal reasoning,
physical geometry,
domain knowledge,
human-centred interaction,
and reliable workflow execution.

We learned that context is often more important than detection accuracy alone. Detecting a grinder is not enough; the system must understand how it is being used, which PPE is required for that specific task, whether the worker is authorised, and which safety procedure governs the activity.

We also learned how important latency and offline resilience are in safety-critical systems. A warning that arrives several seconds late—or fails because the network is unavailable—has little value.

Another important lesson was that explainability cannot be added at the end. Supervisors need to know why an alert occurred, what evidence supported it, what rule was matched, and what action the system recommended.

Most importantly, we learned that the best worker-facing AI should not overwhelm the user. In a dangerous moment, the correct response is usually not a long explanation. It is a short, clear instruction delivered at the right time.

What's next for ShramikSevak

The next step is to convert the Stage 1 concept into a working Stage 2 prototype.

We plan to begin with the four defined scenarios: machine-collision prediction, working-at-height risk, procedure-aware tool use, and fall-response escalation.

Development will proceed in stages:

build and optimise the real-time detection and tracking pipeline,
integrate depth estimation and trajectory prediction,
implement risk-cone generation,
connect the worker feed to NVIDIA edge compute,
ingest sample construction documents into a retrieval system,
generate evidence-backed safety responses,
add multilingual speech input and spoken warnings,
integrate the smart-glasses hardware provided to shortlisted teams,
and measure latency, recall, false alerts, power use, and offline behaviour.

Beyond the hackathon, ShramikSevak could expand to include BIM and digital-twin integration, wearable and equipment sensors, multi-worker coordination, site-wide risk heatmaps, cross-site safety analytics, and predictive insights across entire construction fleets.

Our long-term vision is to move construction safety from passive observation and after-the-fact reporting toward proactive, worker-centred Physical AI.

ShramikSevak does not wait for an accident to become a report. It acts while the accident can still be prevented.