Inspiration

A mid-sized Indian construction project — say a 200-unit residential tower — generates roughly 4,000–6,000 discrete documents over its lifecycle: structural and MEP drawings, BOQs, RA (Running Account) bills, vendor POs, RFIs, DPRs (Daily Progress Reports), inspection checklists, NOC filings, and an unbroken stream of WhatsApp messages between the site engineer, the PMC, and forty-odd vendors. None of this lives in one place. Drawings sit in AutoCAD exports on someone's laptop. BOQs live in Excel. Contracts are PDFs in an email thread from eight months ago. Site photos are on the site engineer's phone, uploaded (if at all) to a shared drive with no metadata.

Existing construction software — ERP suites, BIM viewers, project management dashboards — largely accepts this fragmentation as given. They provide a place to store each artifact type but do nothing to connect them. A project manager who wants to know "why is the third-floor slab pour delayed" still has to manually cross-reference the schedule, the concrete PO, the site photos, and a WhatsApp thread with the ready-mix vendor. The software has not reduced the cognitive load of running the project; it has digitized the filing cabinet.

The result is systemic: 9 out of 10 large infrastructure projects in India run over schedule or over budget, and the root cause is almost never technical. It is coordination failure — information existing somewhere in the system but never reaching the right person at the right time.

This gap exists for two structural reasons. First, construction data is irreducibly heterogeneous — a drawing, a contract clause, and a drone orthophoto require entirely different extraction techniques, and most tools were built to handle exactly one of these well. Second, until recently, no available technology could reason across modalities and act on that reasoning without a human manually stitching the pieces together. Retrieval-augmented systems could search; they could not decide. Vision models could classify a photo; they could not connect it to a BOQ line item and flag a cost overrun.

Foundation models finally make it feasible to build software that reads a contract clause, cross-references it against a procurement PO, looks at a drone image of the same block, and concludes that the project is 11 days away from a liquidated-damages trigger. That capability — not a marginally better chatbot — is what makes it possible to build a Construction Operating System that behaves like a competent AI Project Engineer rather than a searchable archive. This is the moment to build the operating layer construction has been missing.

Problem Statement

Construction project information is structurally fragmented across three axes:

Modality — drawings (vector PDFs, DWG), contracts (text), schedules (XER/MPP), imagery (JPEG/drone orthomosaics), tabular data (BOQ, procurement), and unstructured communication (email, WhatsApp).
System — a single project routinely spans 8–15 disconnected tools, each with its own access model and data schema.
Time — project state evolves daily; a drawing revision on Monday invalidates a procurement decision made on Friday, but no system detects this.

The core problem is information fragmentation without state ownership. No system in a typical construction project owns a live, unified representation of "where the project actually stands." Instead, project state is reconstructed, manually and repeatedly, by humans pulling from disconnected sources every time a decision is needed.

This fragmentation produces four compounding failure modes:

Delay propagation goes undetected until it is expensive. A two-day delay in rebar delivery is knowable the moment the vendor confirms a revised dispatch date, but it typically surfaces to the Project Manager only during the weekly review — by which point downstream trades have already been scheduled around the original date, and the delay has propagated.
Cost overruns hide in reconciliation gaps. BOQ quantities, actual site consumption, and vendor billing are tracked in three different systems (or spreadsheets) that are reconciled manually, usually monthly. Discrepancies — over-ordering, wastage, billing mismatches — are caught weeks after the money has moved.
Procurement and RFIs are reactive, not anticipatory. Purchase requests get raised when someone notices a shortage, not when the schedule and current inventory make the shortage mathematically inevitable. RFIs get filed after work has stalled at a drawing ambiguity, not before the crew reaches that point on site.
Institutional knowledge does not compound. Every project re-learns the same lessons — this vendor is chronically late in monsoon months, this subcontractor under-quotes and pads change orders — because there is no persistent, queryable memory that carries forward across projects.

None of these are solved by digitizing documents faster or building one more dashboard. They require a system that continuously ingests, unifies, reasons over, and acts upon the full multimodal state of a construction project — a system that maintains continuous situational awareness and can reason about causality, why a task is at risk, not just that it is.

Proposed Solution

Nirmaan OS is not a dashboard and not a document chatbot. It is an agentic AI layer that sits above existing project tools and behaves like a tireless AI Project Engineer — a system that maintains a live, structured model of the project, continuously ingests new information as it arrives, predicts risk before it materializes, and — within permissions set by the human Project Manager — takes the next action itself, whether that is drafting an RFI, raising a purchase request, or flagging a stakeholder.

The distinction matters architecturally. A RAG chatbot answers the question it is asked. Nirmaan OS decides which questions need answering in the first place, because it is watching the project state change and comparing it against what the schedule and contracts say should be happening. The system is built around a perceive → reason → act → verify loop, not a query → retrieve → respond loop.

1. Unified Project Memory

A hybrid store combining a construction Knowledge Graph (entities: activities, drawings, WBS nodes, vendors, POs, clauses, sensors), a vector store for semantic retrieval, and a time-series store for schedule and progress signals. Every ingested artifact — a drawing, a contract clause, a site photo, an email — is resolved into entities and relationships in the Knowledge Graph, and simultaneously embedded into the vector store for semantic recall. The graph captures the structure a construction project actually has (a BOQ line item belongs to a WBS task, which depends on a drawing revision, which is contracted to a vendor, who has a delivery history) — relationships that a flat document index cannot represent. This is why plain RAG is insufficient: the answer to "why is Task 4.2 at risk" is not in any single document, it is in the relationship between three documents and a delivery log.

2. Document Intelligence

A pipeline of OCR, layout parsing, table extraction, and drawing symbol recognition that converts unstructured PDFs, DWGs, and scanned contracts into structured, graph-linked entities. A construction-specific NER identifies domain entities like IS 456, M30 concrete, pile P-217, clause 8.7 LD — distinctions that generic document AI consistently fails to make.

3. Multimodal Site Understanding

Vision-language models applied to site photographs, drone orthophotos, and CCTV frames to detect progress, PPE compliance, material staging, and drawing-versus-actual deviations. These visual signals are cross-referenced against the BOQ and schedule within the same reasoning pass, so a discrepancy between "what the photo shows" and "what the schedule claims" is itself a first-class signal the system can act on — not a separate output shown in a different tab.

4. Predictive Risk Engine

A time-series and graph-conditioned forecasting layer that predicts delay probability per WBS node, ranks causal drivers, and simulates recovery scenarios. When a risk is flagged, a root-cause attribution step traces it to the responsible upstream factor (late material, drawing ambiguity, vendor non-performance) rather than surfacing an undifferentiated red flag. A constraint-based optimizer then proposes ranked recovery options with cost and time trade-offs.

5. Agentic Execution Layer

A multi-agent runtime where a supervisor agent decomposes incoming triggers — a new risk flag, a stakeholder query, an incoming email — into subtasks routed to specialized agents (RFI, Procurement, Schedule, Compliance, Reporting), each with a narrow, well-defined toolset and a critic that validates output before execution. Actions that change project state (raising a purchase request, sending an RFI) pass through a human approval gate by default. The system is designed for supervised autonomy, not unattended execution, because the cost of an incorrect purchase order or a wrongly-worded RFI is real.

6. Project Health Score

A continuously updated composite metric spanning schedule, cost, quality, safety, procurement, and contractual exposure. It surfaces the why behind every movement, giving leadership an honest, real-time read on project status without waiting for the next review meeting.

7. Natural Language Interface

A single surface where "Why is Block B slipping?" returns a reasoned answer traced back to specific documents, images, and schedule nodes — with grounded citations. The same interface accepts instructions to act: "Draft an RFI for the discrepancy in Drawing S-204" triggers the appropriate agent workflow.

Hackathon MVP

The hackathon demonstration will focus on a working vertical slice through the full system, scoped to one realistic mid-rise project with a seeded dataset of 200+ artifacts. The following capabilities will be demonstrated end-to-end:

Document Ingestion and Knowledge Graph Generation. A representative set of construction documents — drawings, contracts, BOQs, procurement records, and site photographs — will be ingested through the Document Intelligence pipeline. The system will extract entities, resolve them, and populate the Knowledge Graph and vector store in real time.

GraphRAG-Powered Question Answering. Natural language queries such as "Which tasks are affected if the steel vendor delivery slips by a week?" will be answered through hybrid graph traversal and semantic retrieval, with grounded citations pointing to specific source documents, clauses, and images.

Delay Prediction with Root Cause Attribution. A synthetic-augmented schedule will demonstrate the Predictive Risk Engine detecting delay risk from leading indicators — procurement lead time shifts combined with schedule proximity — before the task is formally marked delayed. The system will surface the causal chain, not just the probability.

Autonomous RFI and Purchase Request Generation. At least one full draft-to-approval cycle will be demonstrated: the system detects an issue (a drawing ambiguity or a forecasted material shortage), drafts the appropriate document (RFI or purchase request) with correct references and quantities pre-filled, routes it through the critic/verifier, and presents it to the Project Manager for approval.

Live Project Health Dashboard. A web console will display the Project Health Score, risk feed, and pending approval inbox. During the demo, newly injected events will visibly update the score and trigger agent actions, demonstrating the continuous perceive-reason-act loop rather than a static dashboard.

Grounded Citations. Every answer and every drafted artifact will include traceable citations to source documents, drawing revisions, contract clauses, and Knowledge Graph nodes — demonstrating the zero-hallucination commitment on all executed actions.

This scope is designed to demonstrate the architectural differentiation of Nirmaan OS — the Knowledge Graph, the agentic loop, the multimodal fusion, and the human approval boundary — within a realistic hackathon timeline.

System Architecture

The system is organized into five layers: ingestion, understanding, Unified Project Memory, reasoning and agents, and action — with a human approval interface sitting between reasoning and action for any state-changing operation.

High-Level System Architecture

flowchart TB
    subgraph Sources["Data Sources"]
        A1["Drawings DWG/PDF"]
        A2["Contracts & BOQs"]
        A3["Schedules XER/MPP"]
        A4["Site Photos / Drone"]
        A5["Emails / WhatsApp / RFIs"]
        A6["ERP / Procurement"]
        A7["IoT / CCTV"]
    end

    subgraph Ingest["Ingestion & Normalization"]
        B1["Connectors"]
        B2["OCR + Layout Parser"]
        B3["Drawing Parser"]
        B4["Schedule Parser"]
        B5["Vision Encoder"]
    end

    subgraph Memory["Unified Project Memory"]
        C1[("Knowledge Graph")]
        C2[("Vector Store")]
        C3[("Time-Series Store")]
        C4[("Object Store")]
    end

    subgraph Reasoning["Reasoning Layer"]
        D1["GraphRAG Retriever"]
        D2["LLM Reasoner"]
        D3["Forecasting Engine"]
        D4["Optimization Solver"]
    end

    subgraph Agents["Agentic Execution Layer"]
        E1["RFI Agent"]
        E2["Procurement Agent"]
        E3["Schedule Agent"]
        E4["Compliance Agent"]
        E5["Reporting Agent"]
        E6["Orchestrator"]
    end

    subgraph Surface["User Surfaces"]
        F1["Web Console"]
        F2["Mobile / Field"]
        F3["Email / WhatsApp Bots"]
        F4["API for ERP/BIM"]
    end

    Sources --> Ingest --> Memory --> Reasoning --> Agents --> Surface
    Agents -.tool calls.-> Sources

Why this layering exists: understanding and memory are kept separate from reasoning so that the Knowledge Graph and vector store remain a stable, queryable source of truth regardless of which agent or model version is reasoning over them. This lets the agent layer evolve (new tools, better models) without re-architecting how project data is stored.

Data Flow

sequenceDiagram
    participant S as Site / Office Source
    participant ING as Ingestion Service
    participant EXT as Extraction Workers
    participant KG as Knowledge Graph
    participant VDB as Vector Store
    participant ORCH as Agent Orchestrator
    participant ACT as Action Executor
    participant PM as Project Manager

    S->>ING: New document, photo, or update
    ING->>EXT: Route by type (OCR / VLM / structured parser)
    EXT->>KG: Write extracted entities & relationships
    EXT->>VDB: Write embedded chunks
    ORCH->>KG: Query current project state
    ORCH->>VDB: Semantic retrieval for context
    ORCH->>ORCH: Multi-agent reasoning & planning
    ORCH->>ACT: Trigger action (RFI / PR / alert)
    ACT->>PM: Present draft for approval
    PM-->>ACT: Approve, edit, or reject
    ACT->>KG: Commit state update
    ACT->>S: Notify relevant stakeholders

Agent Architecture

flowchart TB
    ORCH["Orchestrator Agent"]
    ORCH --> PLN["Planner: task decomposition"]
    ORCH --> MEM["Working Memory"]
    ORCH --> CRIT["Critic / Verifier"]

    PLN --> W1["RFI Agent"]
    PLN --> W2["Procurement Agent"]
    PLN --> W3["Schedule Agent"]
    PLN --> W4["Compliance Agent"]
    PLN --> W5["Reporting Agent"]

    W1 --> T["Tool Registry"]
    W2 --> T
    W3 --> T
    W4 --> T
    W5 --> T

    T --> T1["KG Query"]
    T --> T2["Vector Search"]
    T --> T3["Schedule API"]
    T --> T4["ERP API"]
    T --> T5["Email / WhatsApp"]
    T --> T6["Doc Generator"]

    W1 --> CRIT
    W2 --> CRIT
    W3 --> CRIT
    W4 --> CRIT
    W5 --> CRIT
    CRIT -->|approve| EXEC["Execute Action"]
    CRIT -->|revise| PLN

The Critic/Verifier agent exists specifically to catch hallucinated entities (a BOQ line item number that does not exist, a vendor name that is misattributed) before an action reaches the human approval gate. It re-checks every factual claim in a draft action against the Knowledge Graph and blocks ungrounded generations from execution.

Document Intelligence Pipeline

flowchart LR
    D["Document"] --> C["Classifier: contract / BOQ / drawing / RFI"]
    C --> L["Layout Parser"]
    L --> O["OCR"]
    L --> TB["Table Extractor"]
    L --> SY["Drawing Symbol Detector"]
    O --> CH["Semantic Chunker"]
    TB --> NRM["Schema Normalizer"]
    SY --> SG["Symbol Graph"]
    CH --> NER["Construction NER"]
    NER --> LNK["Entity Linker → KG"]
    NRM --> LNK
    SG --> LNK
    CH --> EMB["Embeddings → Vector Store"]

Multimodal Understanding Pipeline

flowchart LR
    IMG["Site Photo / Drone / CCTV"] --> META["EXIF + Geo + Time"]
    IMG --> VLM["Vision-Language Model"]
    VLM --> DET["Object + Activity Detection"]
    VLM --> CAP["Structured Caption"]
    DET --> ALIGN["Align to WBS + Drawing"]
    CAP --> ALIGN
    META --> ALIGN
    ALIGN --> DIFF["Drawing vs Actual Delta"]
    ALIGN --> PROG["Progress Estimate"]
    DIFF --> KG[("Knowledge Graph")]
    PROG --> TS[("Time-Series Store")]

Delay Prediction Pipeline

flowchart TB
    S1["Baseline Schedule"] --> FE["Feature Builder"]
    S2["Actual Progress Time-Series"] --> FE
    S3["Procurement Lead Times"] --> FE
    S4["RFI Aging"] --> FE
    S5["Weather + Manpower"] --> FE
    S6["KG: dependencies + clauses"] --> FE

    FE --> TSF["Time-Series Forecaster"]
    FE --> GNN["Graph Neural Reasoner"]
    TSF --> RISK["Delay Probability per WBS"]
    GNN --> RISK
    RISK --> CAU["Causal Attribution"]
    CAU --> OPT["Recovery Optimizer"]
    OPT --> REC["Ranked Recovery Plans"]
    REC --> AG["Agents / PM Review"]

Workflow Automation Pipeline

flowchart LR
    EV["Event: e.g. drawing revised"] --> TR["Trigger Rules + LLM Intent"]
    TR --> PL["Plan Generation"]
    PL --> SIM["Dry-Run Simulation"]
    SIM --> APR{"Autonomy Policy"}
    APR -->|Auto| DO["Execute via Tools"]
    APR -->|Human-in-loop| RV["Route to PM Approval"]
    RV --> DO
    DO --> LOG["Audit Log + KG Update"]

Knowledge Graph Integration

flowchart TB
    subgraph Entities
        P["Project"] --- W["WBS Activity"]
        W --- DR["Drawing"]
        W --- BQ["BOQ Item"]
        BQ --- PO["Purchase Order"]
        PO --- V["Vendor"]
        W --- RFI["RFI"]
        DR --- CL["Contract Clause"]
        W --- PH["Site Photo"]
        W --- IR["Inspection Report"]
    end

    Q["User / Agent Query"] --> QP["Query Planner"]
    QP --> GR["Graph Retriever"]
    QP --> VR["Vector Retriever"]
    GR --> FUSE["Hybrid Fusion"]
    VR --> FUSE
    FUSE --> CTX["Grounded Context"]
    CTX --> LLM["LLM Reasoner"]
    LLM --> ANS["Cited Answer / Action"]

AI Pipeline

The end-to-end flow follows a consistent path from raw artifact to executed action:

Documents → Extraction → Knowledge Graph → Vector Store → Reasoning → Agents → Actions

Documents. Connectors continuously pull from email, drives, ERP, scheduling tools, drone platforms, and CCTV. Each artifact carries provenance metadata (source, author, timestamp, project, WBS hint) and is treated as an event, not a static file.
Extraction. A type router dispatches to specialized parsers: layout-aware OCR for contracts, table extractors for BOQs, DWG parsers for drawings, XER/MPP parsers for schedules, vision-language model encoders for imagery. A construction-specific NER identifies entities like IS 456, M30 concrete, pile P-217, clause 8.7 LD.
Knowledge Graph. Extracted entities are linked into a typed graph — WBS activities connected to drawings, BOQ items, POs, RFIs, and clauses. This graph is the spine of the system. It captures relationships that flat vector search cannot: a newly mentioned vendor is matched to an existing vendor record rather than duplicated, and every entity preserves the relational structure a construction project actually has.
Vector Store. Chunks, captions, and structured summaries are embedded for semantic recall. Every vector carries a graph-node ID, enabling GraphRAG — retrieval that expands from a seed node along typed relationships instead of relying on cosine similarity alone. The Knowledge Graph answers "what depends on this"; the vector store answers "what is semantically related to this." Both are necessary because construction queries range from precise structural lookups to open-ended natural language questions.
Reasoning. For any query or triggered event, a query planner decides what to retrieve, how far to traverse, and which forecasting or optimization module to invoke. An LLM reasoner composes the final answer or action plan, always grounded in retrieved citations. This is why agents come after reasoning: the context must be assembled, verified, and grounded before any specialized agent acts on it.
Agents. Domain agents (RFI, Procurement, Schedule, Compliance, Reporting) subscribe to graph and time-series events. Each has a scoped tool registry and a critic that validates output before execution.
Actions. Agents draft RFIs, raise purchase requests, update schedules, notify stakeholders on email and WhatsApp, and log every action back into the Knowledge Graph as a first-class event — closing the loop.

Core Features

F1 · Unified Project Memory

Purpose: Single source of truth spanning documents, imagery, schedules, and communications.
Inputs: All project artifacts and system exports.
AI Models: OCR + layout parser, construction NER, embedding model, entity linker.
Output: Typed Knowledge Graph + vector index + time-series store.
Business Impact: Eliminates cross-tool search; every downstream feature is grounded here.

F2 · Drawing Understanding

Purpose: Turn drawings into queryable objects (rooms, grids, MEP elements, revisions).
Inputs: PDF / DWG drawings and revision histories.
AI Models: Symbol detection CNN + VLM captioning + revision diff LLM.
Output: Structured drawing entities linked to WBS and clauses.
Business Impact: Detects revision impacts before they reach site.

F3 · Contract & BOQ Intelligence

Purpose: Extract obligations, milestones, LD clauses, and BOQ line items.
Inputs: Contracts, addenda, BOQ spreadsheets.
AI Models: Layout parser, clause classifier, table extractor, obligation LLM.
Output: Obligation register linked to activities and vendors.
Business Impact: Contractual exposure becomes computable, not tribal knowledge.

F4 · Multimodal Site Understanding

Purpose: Convert site photos and drone imagery into progress and compliance signals.
Inputs: Photos, drone orthomosaics, CCTV frames with geo/time metadata.
AI Models: VLM (captioning + VQA), object detection, drawing-vs-actual aligner.
Output: Progress percentage, safety flags, deviation reports per WBS node.
Business Impact: Replaces subjective DPR entries with evidence-based progress.

F5 · Delay Prediction & Root Cause Analysis

Purpose: Forecast delay probability per activity and attribute causes.
Inputs: Schedule baseline, actuals, procurement lead times, RFI aging, weather.
AI Models: Time-series forecaster + Graph Neural reasoner over dependency graph.
Output: Ranked at-risk activities with causal breakdown.
Business Impact: Shifts management from reactive to predictive.

F6 · Recovery Plan Optimization

Purpose: Suggest schedule and resource re-sequencing to recover slippage.
Inputs: Current schedule state, resource pool, constraints, cost weights.
AI Models: Constraint optimizer + LLM narrative generation.
Output: 3–5 ranked recovery scenarios with cost/time trade-offs.
Business Impact: Converts a two-week planning exercise into a two-minute review.

F7 · Autonomous RFI Generation

Purpose: Draft RFIs the moment ambiguity or mismatch is detected.
Inputs: Drawing deltas, site photo deviations, clause ambiguity.
AI Models: LLM with RFI schema + critic agent.
Output: Draft RFI with citations, routed to the correct discipline lead.
Business Impact: Cuts RFI turnaround from days to hours.

F8 · Autonomous Purchase Request Generation

Purpose: Raise purchase requests and flag lead-time risk before critical-path impact.
Inputs: BOQ quantities, PO status, vendor lead times, activity schedule.
AI Models: Forecaster + planner LLM + ERP tool calls.
Output: Draft purchase request with correct quantities and suggested vendor, pending approval.
Business Impact: Prevents material-driven delays — the single largest delay category.

F9 · Meeting Intelligence

Purpose: Turn review meetings into structured actions.
Inputs: Audio transcripts or written meeting notes.
AI Models: ASR + speaker diarization + summarization + action extraction.
Output: Minutes with action items linked to WBS nodes, follow-up nudges.
Business Impact: Eliminates the "who was supposed to do this?" problem.

F10 · Natural Language Interface

Purpose: Ask the project anything; instruct it to act.
Inputs: Text or voice queries.
AI Models: GraphRAG + LLM reasoner + tool router.
Output: Cited answers or executed actions with audit trail.
Business Impact: Zero training curve for site engineers and executives.

F11 · Project Health Score

Purpose: Continuous composite health metric with drill-down.
Inputs: All subsystem outputs.
AI Models: Weighted scoring + LLM explanation.
Output: Live score with explainable movers.
Business Impact: Replaces monthly PMC reports with real-time visibility.

Technical Innovation

Nirmaan OS is architecturally different from existing construction technology on six dimensions:

GraphRAG over flat retrieval. Construction data is fundamentally relational — a delay's cause is rarely in the document that reports the delay, it is in the dependency chain connecting a drawing revision, a vendor's delivery record, and a downstream task. Asking "why is Block B slipping?" walks from Block B → activities → predecessors → RFIs → drawing revisions → vendor POs — a path a vector search cannot reproduce. Pure vector retrieval flattens this structure and loses it. The Knowledge Graph preserves it.
Agentic workflows over single-shot Q&A. A chatbot waits to be asked. Nirmaan OS's agents subscribe to events and act. The system produces artifacts (RFIs, purchase requests, meeting minutes, recovery plans) — not just responses. It decides what deserves attention, not just how to answer what it is asked.
Multimodal reasoning fused into one state. Drawings, imagery, tabular BOQs, and text are fused at the entity level, not concatenated at the prompt level. A drone image of pile-cap P-217 is a graph neighbor of the corresponding drawing and BOQ row. A discrepancy between visual evidence and schedule claims is itself a first-class signal.
Construction-specific domain grounding. A construction-specific ontology (WBS, IS codes, contract clauses, MEP taxonomies) and NER prevent the generic-LLM failure mode of confidently mislabeling domain entities. Generic LLMs do not reliably distinguish an RA bill from an RFI, or understand that a BOQ variance beyond a contractual threshold triggers a different workflow than a minor one.
Forecasting and optimization, not just retrieval. Delay prediction uses time-series and graph-conditioned models; recovery uses constraint optimization. LLMs explain these outputs; they do not fabricate them. These are capabilities orthogonal to language modeling, integrated as tools the agents call rather than approximated by prompting.
Autonomous execution with critic-guarded safety. Every agent action passes through a critic and an autonomy policy (auto / human-in-loop / advisory) before touching ERP, email, or schedules — with a full audit trail. Unattended autonomy in procurement or contractual communication carries real financial and legal risk; encoding the oversight boundary as a first-class policy is what makes agentic systems safe to deploy in a domain where mistakes have physical consequences.

Demo Scenario

The following end-to-end scenario will be demonstrated live to illustrate the full perceive-reason-act loop:

Trigger: A steel vendor confirms a revised dispatch date that pushes rebar delivery by four days.

Perceive. The ingestion service picks up the vendor's revised delivery confirmation. The Document Intelligence pipeline extracts the new date, resolves the vendor to an existing Knowledge Graph node, and updates the procurement record.

Reason. The Predictive Risk Engine detects that the revised lead time pushes the rebar delivery past the scheduled dependency for the third-floor slab pour. The time-series forecaster recomputes delay probability for the affected WBS node. The Graph Neural reasoner propagates the impact downstream, identifying three additional tasks at risk: formwork for the fourth floor, MEP rough-in for the third floor, and the concrete pour milestone. A root-cause attribution trace confirms the upstream factor: vendor delivery slip for PO-ST-0047.

Act. The Procurement Agent drafts a purchase request for an alternate vendor with available stock, pre-filled with the correct BOQ quantities. The RFI Agent drafts a clarification request to the structural consultant regarding whether partial rebar delivery can support a phased pour. The Reporting Agent generates a stakeholder update summarizing the delay chain, affected milestones, and recommended recovery options. All three drafts are critic-verified against the Knowledge Graph for factual accuracy.

Verify. The Project Manager receives the three draft actions in the approval inbox on the web console. The Project Health Score has already updated to reflect the new risk. The Project Manager reviews each draft, approves two, edits one, and rejects none. Approved actions execute: the purchase request routes to the alternate vendor, the RFI routes to the consultant, and the stakeholder update dispatches via email. All state changes are committed back to the Knowledge Graph, and the audit trail is logged.

Total elapsed time from vendor email to Project Manager review: under 90 seconds.

How We Built It

Scope for the hackathon. A working vertical slice on one real-shaped mid-rise project: ingest 200+ artifacts, populate the Knowledge Graph, run three agents (RFI, Procurement, Reporting), demonstrate delay prediction on a synthetic-augmented schedule, and expose a unified web console.

Backend. Python (FastAPI) for services; Celery + Redis for async ingestion and agent jobs. LangGraph for agent orchestration with explicit state machines, chosen because deterministic state machines are more debuggable than free-form loops. Services are deployed as independently scalable microservices so the multimodal vision pipeline (GPU-bound) scales separately from the text extraction pipeline (CPU-bound).

Frontend. Next.js + React with a project console: Knowledge Graph explorer, document viewer with citation highlights, delay heatmap, and agent action inbox.

Databases. Neo4j for the construction Knowledge Graph (native support for the relationship-heavy queries the agents rely on). Qdrant for vector embeddings with payload filtering by graph-node ID. TimescaleDB for schedule actuals and sensor streams. PostgreSQL for transactional metadata and audit logs. S3-compatible object store for raw artifacts.

AI Stack. Frontier LLM (via API) for reasoning; a smaller open model (Llama-class) fine-tuned on construction NER for cost-efficient extraction. Vision-language model for site and drone imagery captioning and VQA. PaddleOCR / Surya for OCR; layout parsers for structured PDFs. Sentence-Transformers for embeddings; BGE reranker for retrieval refinement. N-BEATS / Prophet for time-series baselines; a Graph Neural Network (PyTorch Geometric) over the dependency graph for delay propagation. OR-Tools for recovery-plan optimization.

Agents. Orchestrator + planner + critic pattern. Tool registry with typed schemas (Knowledge Graph query, vector search, ERP mock, email/WhatsApp mock, document generator). Autonomy policy declared per action type.

Infrastructure. Docker + Kubernetes (single-node k3s for the demo). OpenTelemetry for tracing every agent step end-to-end. Terraform for reproducible environments.

Deployment. Cloud-hosted for the demo, with a scripted seed dataset and a "replay" mode that streams events across a simulated project timeline. All approval-gated actions routed through a sandboxed simulation layer for the hackathon environment rather than live vendor systems.

Challenges We Expect

Construction data heterogeneity. Drawing formats, BOQ templates, and contract structures vary significantly across firms and even across projects within the same firm. No two projects use the same WBS structure or drawing conventions. We mitigate with an adaptive schema layer and few-shot extractors rather than hard-coded parsers.

Multimodal fusion. Reconciling a vision model's progress estimate with a schedule's claimed progress requires careful calibration. The two signals disagree by default, and naive fusion risks amplifying noise rather than surfacing genuine discrepancies. We treat the mismatch as signal and fall back on VLM landmark matching and human-in-loop confirmation when geo-metadata is missing.

Reasoning over incomplete information. Real projects have gaps — a missing DPR, an unlogged vendor call — and the system needs to reason usefully around missing data rather than either hallucinating to fill gaps or refusing to act. Agents must know when to ask rather than assume.

Hallucination prevention in generated actions. A drafted RFI referencing a non-existent BOQ line item or drawing revision is worse than no draft at all. Every answer is grounded via GraphRAG citations, and the critic/verifier step blocks ungrounded generations from execution.

Retrieval latency under agentic loops. Graph traversal + vector search + LLM reasoning must stay sub-3 seconds. We cache subgraphs, pre-compute embeddings, and stream LLM outputs.

Knowledge graph maintenance. Entity resolution (is "ABC Steel Traders" the same vendor as "ABC Steel Pvt Ltd" mentioned elsewhere?) is a genuinely hard, ongoing problem. We treat the ontology as versioned code with migrations, not a one-time setup task.

Forecasting under sparse history. Early in a project, there is not enough historical data for the forecasting model to be confident. The system degrades gracefully to rule-based heuristics and surfaces confidence intervals rather than producing overconfident predictions.

Agent orchestration reliability. Preventing loops, duplicate actions, and race conditions requires deterministic state machines and idempotent tools — more evaluation infrastructure than the hackathon timeline naturally affords.

Accomplishments We Aim For

Ingest and unify a real mid-rise project's document set into a queryable Knowledge Graph during the hackathon window.
Sub-3-second grounded answers to natural-language questions with citations to specific drawings, clauses, and images.
Autonomous generation of at least three artifact types (RFI, purchase request, meeting minutes) with critic-verified quality.
Delay prediction on a seeded schedule with explainable root causes traced through the Knowledge Graph.
Live Project Health Score that responds to injected events in real time during the demo.
Zero-hallucination guarantee on execution: no action leaves the system without a grounded citation trail.

What We Learned

Building the extraction pipeline surfaced how much of "understanding" a construction document is really about resolving references — a BOQ line item is meaningless without knowing which drawing revision and which contract clause it ties back to. We underestimated, going in, how much of the system's value would come from entity resolution and relationship modeling rather than from the language model's raw comprehension ability. The Knowledge Graph turned out to be the product. The LLM is a reasoning surface; the agents are execution muscle; but the graph is what makes both trustworthy.

We also learned to be deliberate about where autonomy should stop. It was tempting to let the system execute actions directly, but the moments where a draft RFI referenced a slightly wrong clause number made clear why a verification and human-approval boundary is not a compromise — it is a requirement for the system to be trustworthy enough to actually use on a real project. The right question is never "should the AI act on its own?" but "for this action type, what is the correct human oversight?"

Finally, working across modalities clarified that fusion is a genuine research problem, not a plumbing one: a vision model's progress estimate and a schedule's claimed progress are two different, imperfect measurements of the same underlying reality, and treating a mismatch as signal rather than noise was one of the more valuable design decisions we made.

Future Roadmap

Horizon	Direction
Near-term	Native connectors for Primavera P6, SAP, Procore-class tools, Zoho, and WhatsApp Business.
Near-term	BIM (IFC) ingestion so the Knowledge Graph inherits geometry, not just documents.
Mid-term	IoT + sensor fusion (concrete maturity, crane telemetry, batching plant feeds) into the time-series layer.
Mid-term	Meta Smart Glasses / AR integration for site engineers to query drawings and log observations hands-free.
Mid-term	Predictive procurement — the system negotiates lead times with vendors autonomously within pre-approved envelopes.
Long-term	Full digital twin synchronization: drone flights and IoT updates keep the twin live; the twin becomes the substrate for simulation.
Long-term	Autonomous scheduling — the system proposes, simulates, and (within policy) commits schedule updates.
Long-term	Cross-project learning — a portfolio-scale model that transfers risk patterns from completed projects to new ones.

Tech Stack

Layer	Technology	Purpose
Frontend	Next.js, React	Console, document viewer, KG explorer
Visualization	Deck.gl, Mapbox, D3	Spatial + graph + timeline views
API	FastAPI, gRPC	Service interfaces
Orchestration	LangGraph, Celery, Redis	Agent state machines, async jobs
LLM Reasoning	Frontier LLM API + fine-tuned Llama-class	Reasoning + domain extraction
Vision-Language	Open VLM for captioning / VQA	Site & drone imagery understanding
OCR & Layout	PaddleOCR, Surya, layout parsers	Contract and drawing parsing
Embeddings	Sentence-Transformers, BGE reranker	Semantic retrieval
Knowledge Graph	Neo4j	Construction ontology + GraphRAG
Vector Store	Qdrant	Payload-filtered semantic search
Time-Series	TimescaleDB	Schedule actuals, sensor data
Forecasting	N-BEATS, Prophet, PyTorch Geometric (GNN)	Delay + risk prediction
Optimization	Google OR-Tools	Recovery-plan generation
Object Store	S3-compatible	Raw artifact storage
RDBMS	PostgreSQL	Metadata, audit, autonomy policy
Infrastructure	Docker, Kubernetes (k3s), Terraform	Reproducible deployment
Observability	OpenTelemetry, Grafana	End-to-end agent tracing
Integrations	Email, WhatsApp Business, ERP / Primavera connectors	Field-facing surfaces

Novelty

Compared to traditional project management software, Nirmaan OS does not require the Project Manager to manually enter and cross-reference status. It derives project state from the same raw artifacts (drawings, site photos, vendor emails) the Project Manager already produces, and updates continuously rather than at manual check-in intervals. Traditional tools are systems of record; Nirmaan OS is a system of understanding and action.

Compared to document chatbots, Nirmaan OS does not wait for a question. It maintains an ongoing model of expected versus actual project trajectory and initiates action — drafting an RFI, raising a purchase request — without a human needing to notice the problem first. It reasons across a typed graph that connects documents to schedules, imagery, procurement, and contracts.

Compared to generic RAG systems, Nirmaan OS represents project knowledge as a graph of typed, construction-specific relationships rather than a flat index of document chunks. GraphRAG retrieval follows semantic and structural relationships, enabling multi-hop reasoning ("what else is affected if this vendor slips") that similarity search alone cannot answer reliably.

Compared to standalone forecasting tools, Nirmaan OS does not stop at a risk score. The forecaster is graph-conditioned — it knows why an activity is at risk because it can traverse to the RFI, drawing revision, or vendor PO responsible. Forecasting is one input to an agentic loop that also attributes root cause, proposes optimized recovery options, and drafts the resulting action — closing the gap between prediction and execution.

The composite — graph + multimodal + agentic + domain-grounded — is the novelty. No individual capability is unique; the system is.

Conclusion

Construction does not need another place to store documents — it needs software that carries the weight of tracking, cross-referencing, and reacting to project reality, so that project managers can spend their attention on judgment calls instead of information archaeology. Nirmaan OS is built on the premise that an AI system capable of perceiving project state across modalities, reasoning over its true relational structure, and acting within a governed approval boundary can absorb that weight.

If software transforms construction the way it transformed every other industry, it will not be through better filing cabinets. It will be through systems that understand. Nirmaan OS is a step toward that future — beginning with a working slice at Kaya AI India Hackathon 2026, and scaling toward the operating layer for how India builds.

Built With

celery
computervision
docker
fastapi
graphrag
knowledgegraph
kubernetes
langgraph
llms
multiagentai
natural-language-processing
neo4j
next.js
ocr
opentelemetry
postgresql
python
pytorch
qdrant
rag
react
redis
terraform
timescaledb