RFIBot

Autonomous agent researching construction RFIs across specs, photos, email. Detects contradictions, drafts responses, routes approvals, saves 663 hours per project, zero manual work.

Comment

Inspiration

The construction industry loses over $31 Billion annually due to delays, rework, and communication bottlenecks stemming from the Request for Information (RFI) workflow. On a typical commercial project, answering a single RFI requires a human project manager or engineer to spend hours or days cross-referencing dense written specifications, architectural CAD drawings, past project decisions, and physical site photographs. A single missed detail can halt a job site, delay trades, or lead to multi-million dollar structural errors. i set out to ask: What if an AI agent could autonomously investigate an RFI across specs, blueprints, and site photos, synthesizing a verified, auditable response in under 40 seconds?

What it does

RFIBOT is an autonomous AI construction engineer designed to automate the complete RFI investigation lifecycle. When a construction manager submits a field question such as "What material should be used for interior walls in Zone A according to Section 3.2, and does our current installation match?" RFIBOT executes a multi-step investigation loop: Decomposes Query Intent: Breaks the RFI into required verification targets across specifications, CAD details, field images, and historical logs. Executes Multi-Modal Search: Queries project specifications, searches architectural drawing sheets, and pulls active site photographs. Computer Vision Verification: Analyses field photographs using multimodal vision to verify whether physical installations match project drawings. Precedent Cross-Checking: Audits past project RFIs to check if a prior design variance or exception was granted. Confidence & Risk Evaluation: Calculates an aggregate evidence score to determine if human intervention is required. Automated State Execution: Drafts an auditable answer with exact source citations, routes the package to the designated Lead Architect for sign-off, and updates the project RFI tracker.

How I built it

RFIBOT is built on a Google-first architecture, utilising the latest Google Cloud and AI ecosystem to demonstrate a true, multi-step agentic workflow rather than a simple chatbot: Google Agent Development Kit (ADK): Drives the core decision loop, tool routing, dynamic goal decomposition, and asynchronous state management. Gemini 3.5 Flash: Serves as the intelligence engine, executing high-speed function calling and multi-modal vision analysis on site photographs. Vertex AI Search (Agent Builder): Ingests and performs layout parsing across unstructured architectural drawings, specifications, and historical RFI logs. Firebase & Cloud Storage: Manages real-time application state streaming, session history, audit trail logging, and secure image/document hosting. Cloud Run: Hosts our Python ADK FastAPI backend API server container.Next.js 14 & Tailwind CSS: Powers a real-time terminal dashboard visualizing the agent's thought stream, retrieved citations, confidence score, and human-in-the-loop approval routing. The Math Behind Evidence Confidence To prevent dangerous construction hallucinations, RFIBOT computes an Aggregate Confidence Score ($C_{\text{RFI}}$) across $N$ retrieved evidence sources using a weighted alignment function:$$C_{\text{RFI}} = \sum_{i=1}^{N} w_i \cdot S(E_i, Q) \cdot (1 - \Delta_{\text{conflict}})$$Where:$w_i$ is the authoritative weight of the source type (e.g., $w_{\text{spec}} = 0.4$, $w_{\text{drawing}} = 0.3$, $w_{\text{photo}} = 0.2$, $w_{\text{history}} = 0.1$).$S(E_i, Q) \in [0, 1]$ represents the semantic and visual alignment function between retrieved evidence $E_i$ and query $Q$.$\Delta_{\text{conflict}} \in [0, 1]$ represents a penalty factor triggered if tool outputs contradict one another (e.g., spec requires mold-resistant board, but site photo reveals standard drywall).If $C_{\text{RFI}} < 0.85$, RFIBOT flags the response as High Uncertainty and automatically escalates the ticket directly to the Lead Architect for manual review.

Challenges I ran into:

Cross-Modal Grounding: Teaching the agent to correlate a written requirement (e.g., "Section 3.2: 5/8-inch Mold-Resistant Board") with a spatial visual (Detail A-2.1) and a real-world photo (identifying green-faced drywall) required fine-tuning tool prompts and visual input framing. Deterministic Safety vs. Autonomy: Construction decisions carry real legal liability. I had to design strict safeguards to ensure the agent never invents specifications when document evidence is missing or ambiguous. Real-time Latency: Demonstrating multi-step execution (Spec Search $\rightarrow$ Drawing Lookup $\rightarrow$ Vision Analysis $\rightarrow$ Precedent Audit) within a crisp 30–60 second demo required optimizing asynchronous tool calls and leveraging the inference speed of Gemini 3.5 Flash. ##Accomplishments that am proud of: True Agentic Orchestration: I avoided simple prompt-chaining and built a dynamic graph agent that selects tools, handles evidence dependencies, and executes database side-effects autonomously. Multimodal Verification Engine: Successfully pairing computer vision on job-site photos with PDF specification text retrieval to confirm physical construction compliance in real time. Auditable Human-in-the-Loop Model: Creating a trust-first framework where architects receive full evidence chains with one-click approval workflows rather than trusting a black-box AI.

3What I have learnt:

Agents need structured bounds: Unrestricted LLMs hallucinate tolerances, but an agent constrained by strict tool execution contracts and confidence math provides enterprise-grade safety. Multimodal cross-referencing is a superpower: Combining visual field photos with text spec retrieval yields a level of verification that neither medium can achieve on its own. Human-in-the-Loop builds trust: Positioning AI as an autonomous investigator that proposes rather than blindly approves makes the technology immediately adoptable for conservative industries like construction.

What's next for RFIBot

I envision RFIBOT becoming the standard intelligence layer for global construction management. Next steps include: Deep Enterprise Integrations: Building native API plugins for Procore, Autodesk Construction Cloud (ACC), and Oracle Aconex to process RFIs directly within current construction inboxes.3D BIM Integration: Expanding vision capabilities from 2D photos and drawings to full 3D Building Information Models (IFC/Revit files).Multi-Agent Collaboration Networks: Deploying specialized sub-agents (e.g., Structural Agent, MEP Agent, Code Compliance Agent) that debate and cross-examine complex engineering changes before human sign-off.

Built With

Share this project:

Updates

Submission history