SiteLens AI Reality Inspector

AI-powered construction inspection grounded in real-world evidence.

Comment

SiteLens™ AI Reality Inspector — Project Story

Reality Is the Reference™
Nebius × NVIDIA Global AI Hackathon

1. Inspiration

Construction sites change every day. Field engineers and superintendents capture hundreds of photographs of foundations, concrete slabs, structural steel, installations, and unfinished work. Yet photographs alone rarely answer the questions that matter most:

  • What is actually visible?
  • What differs from the expected state?
  • What requires closer inspection?
  • What evidence supports each finding?

Many reality-capture workflows depend on accurate, up-to-date Building Information Models (BIM). On active construction sites, however, models can be incomplete, outdated, or difficult to access in the field. Photographs may remain scattered across phones and messaging applications until a discrepancy is discovered too late.

Generic multimodal AI introduces another challenge: convincing conclusions that are not supported by the available evidence. A model may miscount construction elements, infer measurements that cannot be established from a photograph, or make an unjustified claim about code compliance.

We wanted to explore a different approach.

What if reality itself could be the reference—even when a perfect BIM model is unavailable?

SiteLens was built around one principle:

AI does not replace the inspector. It helps the inspector see what matters, understand potential discrepancies, and decide what must be verified.

Our goal is to turn ordinary construction imagery into structured, traceable observations while preserving uncertainty and human responsibility.

2. What SiteLens Does

SiteLens transforms construction photographs and live device-camera frames into structured inspection observations through four stages:

SEE → COMPARE → UNDERSTAND → VERIFY

  1. SEE — Observe reality. A vision model identifies visible construction elements and produces structured observations, including spatial coordinates when supported by the image.
  2. COMPARE — Calculate differences. Deterministic TypeScript logic compares observed values against an expected-state reference.
  3. UNDERSTAND — Explain potential significance. A reasoning model interprets structured observations and comparison results to suggest possible risks and practical verification steps.
  4. VERIFY — Keep humans accountable. A human inspector reviews each candidate finding and decides whether to verify it, request further review, or reject it.

The distinction matters: an AI suggestion is not a verified finding, and a comparison result is not a construction compliance certificate.

Key Features

Multi-Photo Inspection

Operators can upload or select 1–6 photographs for an inspection group. SiteLens organizes the evidence, observations, comparison results, and findings, with attribution to the photographs supporting each observation.

Live Camera AI — Beta

A dedicated live-camera workspace samples frames from a device camera and runs the inspection pipeline. It can display spatial overlays when usable bounding-box coordinates are available and populate a live discrepancy panel.

Live inspection is an assistive workflow. Detection quality and latency depend on the scene, device, network, and model-serving conditions.

Deterministic Comparison

Rather than asking a language model to decide whether work is complete, SiteLens uses code to calculate the difference between observed and expected element counts:

$$ \Delta_{\text{elements}} = C_{\text{observed}} - C_{\text{expected}} $$

The comparison produces three principal outcomes:

  • MATCH: The available comparison indicates agreement with the reference.
  • ATTENTION: A difference exists that merits investigation.
  • UNDETERMINED: The evidence is insufficient for a reliable comparison.

These outcomes describe the comparison, not the overall safety or compliance of the construction work. A match does not prove that a structure is safe, and an attention result does not independently establish a defect.

Truthful Spatial Grounding

When the vision model supplies usable normalized coordinates, SiteLens maps them to the displayed image:

$$ x_{\text{canvas}} = x_{\text{norm}}W_{\text{painted}} + X_{\text{offset}} $$

$$ y_{\text{canvas}} = y_{\text{norm}}H_{\text{painted}} + Y_{\text{offset}} $$

Here, (W_{\text{painted}}) and (H_{\text{painted}}) represent the displayed image dimensions, while (X_{\text{offset}}) and (Y_{\text{offset}}) account for its position within the canvas.

When a finding cannot be reliably localized, SiteLens preserves it as unlocalized instead of inventing a bounding box. The interface should never imply more spatial precision than the evidence supports.

Four-Language Interface

SiteLens supports:

  • English (en)
  • French (fr)
  • Arabic (ar, right-to-left)
  • Simplified Chinese (zh-CN, left-to-right)

Language switching reuses available localized content instead of requiring new inference for every switch. The image-coordinate layer is isolated from text direction so that Arabic layout does not unintentionally mirror spatial overlays.

Human Verification and Project Isolation

AI-generated findings begin as unverified candidates. A human inspector can mark a finding as VERIFIED, NEEDS_REVIEW, or REJECTED. Findings also support an OPEN → ACKNOWLEDGED → CLOSED triage workflow.

Each project maintains its own captures, selections, and findings. Switching projects clears transient evidence from the previous workspace, while returning to a project restores its saved state.

3. How We Built It

SiteLens combines computer vision, deterministic comparison, language-model reasoning, and human review. Each stage has a distinct responsibility.

Architecture

       CONSTRUCTION PHOTOS / LIVE CAMERA
                       |
                       v
             [1] SEE — Vision
       MiniCPM-V-4_5 via Nebius
                       |
                       v
           [2] COMPARE — TypeScript
         Deterministic comparison logic
                       |
             MATCH / ATTENTION /
               UNDETERMINED
                       |
                       v
          [3] UNDERSTAND — Reasoning
       NVIDIA Nemotron via Nebius
                       |
                       v
             [4] VERIFY — Human
         Review and decision ledger

3.1 NVIDIA and Nebius Integration

Vision model: openbmb/MiniCPM-V-4_5

The vision stage processes image inputs and returns structured observations, including detected elements and spatial coordinates when available.

Reasoning model: nvidia/Nemotron-3-Ultra-550b-a55b

The reasoning stage receives structured observations and comparison results. It helps explain why a difference may matter and suggests what an inspector should physically verify.

Nebius Token Factory

Nebius provides the model-serving interface used to connect the application to the vision and reasoning models. This allows SiteLens to combine specialized model capabilities without making one model responsible for the entire inspection decision.

3.2 The Vision–Reasoning Boundary

A central architectural decision is to separate image interpretation from reasoning over comparison results.

The pipeline works as follows:

  1. MiniCPM-V processes image pixels and returns structured observations.
  2. The application validates those observations and calculates comparison outcomes in deterministic code.
  3. Nemotron receives structured text containing the observations and comparison results—not the original image pixels.
  4. The human inspector reviews the resulting candidate findings.

This separation is designed to prevent the reasoning model from silently replacing a deterministic comparison result with an unsupported conclusion. Automated tests exercise these trust boundaries, including cases where uncertain evidence must remain uncertain.

3.3 Efficient Live-Camera Processing

Repeatedly processing nearly identical camera frames wastes inference requests and resources. SiteLens includes several mechanisms to reduce unnecessary work.

Frame-difference filtering

A lightweight (32 \times 32) luminance representation is used to estimate whether a frame has changed significantly:

$$ D(F_t,F_{t-1}) = \frac{1}{N}\sum_{i=1}^{N} \left|Y_t(i)-Y_{t-1}(i)\right| $$

Here, (Y_t(i)) is the luminance value of pixel (i) in the current sampled frame, and (N) is the number of sampled pixels. Frames below a configured change threshold can be skipped during automatic processing, while manual snapshot capture remains available.

Vision-result reuse

When an operator freezes and saves a recent inspection result, the application can reuse the existing vision output when it is still valid, avoiding an unnecessary repeat vision request.

Finding deduplication

Repeated findings can be suppressed during a cooldown period. New findings or changes in severity can trigger an update rather than generating an endless stream of duplicate reports.

Together, these mechanisms aim to reduce redundant inference while keeping manual inspection available.

3.4 Generational Concurrency Control

Project isolation introduced a subtle asynchronous race condition.

Imagine that an upload starts in Project A. The operator switches to Project B and then quickly returns to Project A. A check of the active project ID alone may pass because the ID is A again, even though the original operation belongs to an earlier session context.

SiteLens addresses this with a monotonically increasing project-generation counter:

$$ G_{t+1}=G_t+1 $$

The generation changes when the project context changes through supported lifecycle operations. Asynchronous mutations capture their starting project ID and generation, then validate the context before committing results.

Conceptually, a result can be committed only when:

$$ \text{Commit allowed} \iff (G_{\text{exit}}=G_{\text{entry}}) \land (P_{\text{exit}}=P_{\text{entry}}) $$

If the context is stale, the application rejects the operation with HTTP 409 STALE_PROJECT rather than applying its result to the wrong workspace.

Automated tests cover project switching, delayed operations, and related evidence-isolation scenarios. This is a tested safeguard, not a claim that every possible concurrency or deployment risk has been eliminated.

3.5 A Lightweight TypeScript Application

SiteLens uses modern Node.js and TypeScript, a modular HTTP server, and an SVG-based reactive interface. Its application runtime is designed without declared runtime npm package dependencies.

This reduces dependency overhead, but it does not eliminate external service dependencies: live AI inference still relies on the configured model provider and network connectivity.

4. Challenges We Ran Into

Challenge 1: The “Helpful AI” Compliance Trap

A plausible explanation is not necessarily a correct observation. A photograph may not reveal exact dimensions, hidden reinforcement, material properties, or whether work complies with a particular standard.

We addressed this by separating visual observations from comparison logic and human decisions. Structured data is validated, uncertainty can remain explicit, and model-generated advice cannot independently certify compliance.

Challenge 2: Bounding Boxes in Arabic Layouts

Adding Arabic support introduced a spatial-interface problem. Right-to-left document layout can affect visual positioning if image geometry and text layout share the same directional assumptions.

We isolated the image and coordinate layer in a left-to-right context while allowing surrounding Arabic interface elements to follow right-to-left layout. This preserves the intended relationship between detection coordinates and the displayed image.

Challenge 3: Asynchronous Project Switching

Our tests exposed the A → B → A race condition. A delayed result could appear valid if the application checked only the current project ID.

The generation counter gives each project context a changing version, allowing the application to detect stale work before committing it. We added automated coverage for rapid switching and delayed asynchronous operations.

Challenge 4: Balancing Useful AI With Honest Uncertainty

A construction inspection tool must not turn missing evidence into a confident answer.

We designed SiteLens to preserve an UNDETERMINED state, avoid fabricating spatial coordinates, and require human review before a candidate finding becomes verified. The difference between “the model suggested it,” “the comparison found a discrepancy,” and “the inspector verified it” is part of the product architecture.

5. Accomplishments We're Proud Of

At the latest reported verification checkpoint, SiteLens achieved:

  • 681/681 automated tests passing across 42 test suites.
  • 100% conformance on the built-in evaluation harness for the evaluated trust-boundary and deterministic-comparison cases.
  • Zero TypeScript errors in the reported typecheck.
  • Dedicated project-isolation tests covering clean workspaces, state restoration, and stale asynchronous operations.
  • Four-language support for English, French, Arabic, and Simplified Chinese.
  • Explicit uncertainty and human review, instead of presenting every model output as a verified construction finding.

These results describe the tested software and evaluation scenarios. They do not establish real-world detection accuracy, regulatory compliance, or structural safety. Those require further validation under representative site conditions and review by qualified professionals.

6. What We Learned

AI should investigate, not judge.

In a high-consequence domain, AI should help people focus their attention without assuming the authority of a licensed professional. Trust depends on clear responsibilities and traceable decisions.

Deterministic code is better for deterministic comparisons.

Language models are useful for interpretation and explanation, but arithmetic comparisons should be performed by code whose behavior can be tested directly.

Uncertainty is useful information.

An explicit UNDETERMINED result is better than a confident guess. It tells the inspector that more evidence, another viewpoint, or a physical measurement may be needed.

Reliability includes state management.

Correct model output is not enough if it appears in the wrong project or becomes associated with the wrong evidence. Project isolation and stale-request protection are part of the trust model, not merely interface details.

7. What's Next for SiteLens

Our planned next steps include:

  • 3D reconstruction: Explore Gaussian splatting or NeRF-based reconstruction to provide richer spatial context from multiple photographs.
  • Edge deployment: Investigate running suitable vision components on NVIDIA Jetson-class hardware for sites with limited or intermittent connectivity.
  • Construction-platform integrations: Explore integrations with platforms such as Autodesk Construction Cloud and Procore to help transfer human-verified findings into established workflows.
  • Field validation: Evaluate detection quality, uncertainty handling, latency, and usability across representative construction sites, devices, lighting conditions, and network environments.

These are future development directions, not claims of completed functionality.

8. Closing

SiteLens is built around one principle:

Reality is the reference, but evidence must earn trust.

Vision models can help identify what is visible. Deterministic code can compare observations against an expected state. Reasoning models can explain potential discrepancies. But the final verification decision belongs to the human inspector.

We are building SiteLens to make construction evidence more useful, discrepancies easier to investigate, and AI-assisted inspection more transparent without pretending that a photograph or a model can prove more than it actually can.

Built With

  • ai-agents
  • artificial-intelligence
  • computer-vision
  • construction
  • construction-technology
  • contech
  • docker
  • enterprise-software
  • human-in-the-loop-ai
  • machine-learning
  • multimodal-ai
  • nebius-token-factory
  • node.js
  • nvidia-nemotron
  • postgresql
  • reality-capture
  • rest-api
  • typescript
  • web-application
Share this project:

Updates

Submission history