SiteWork - Nothing gets covered until it's proven.

Inspiration

Every building hides its most important work. Pipes go behind drywall, joists under decking, utilities under backfill. Construction QA handles this with hold points: work stops until an inspector releases it. Miss one and the defect is sealed in, found later as a leak, a sagging deck or a claim.

Rework typically costs 4–10% of total project cost, and systematic QA cuts it sharply. The gap is coverage: inspectors can't stand at every hold point. Some hold points are life-safety; OSHA counted 39 trench-collapse deaths in the US in 2022. Cameras and vision models can now see this work. What's missing is what makes an inspection count: a decision someone is accountable for, against a cited rule, with proof that survives an audit.

What it does

SiteWork makes the hold point machine-enforceable:

Models observe → code measures → policy decides → humans sign when the AI isn't sure → everything is provable.

1. Observe. A webcam or a replayed phone clip watches the work. NVIDIA Nemotron 3.5 Super VL on Nebius Token Factory says roughly where the edges that matter are, as schema-constrained JSON. A deterministic DSP check listens for the click of a push-fit joint seating.
2. Measure. Code snaps each model line onto the straight edge the image actually shows and measures board gaps and angles, trench wall slope and depth, and pipe-to-fitting alignment, scaled by a printed 40 mm ArUco marker. How cleanly a line fits is its confidence. A line with no edge under it escalates.
3. Ground. Each judgment cites a requirement retrieved with Tavily from allowlisted sources for the job's jurisdiction (OSHA and manufacturers in the US, Arbetsmiljöverket, Svenskt Trä and Säker Vatten in Sweden). The excerpt is hashed, and it's usable only after a named person approves its threshold.
4. Decide. SiteWork can only propose a release, as a hashed Agent-Safe Intent. Decionis returns ALLOW, ESCALATE or BLOCK under a versioned policy: no citation, no ALLOW; low confidence, no ALLOW.
5. Prove. Every verdict comes with an Ed25519-signed Decionis dossier bound to the exact evidence: measurements, frame hashes, model IDs, the cited clause and its hash. Anyone can verify it offline, and changing one byte makes verification fail.
6. Act. A phone propped at the work face is the stack light. It turns green only when Decionis allows the release and SiteWork has claimed and finalized the single-use execution grant. A named person's release shows blue, never green.

Three hold points are built: decking (board gaps vs the cited spacing), trench entry (wall slope vs OSHA's protective sloping) and plumbing close-in (alignment plus the seating sound). Switch the job from a US site to a Swedish one and SiteWork re-decides the same evidence against a different authority. Sweden's guidance sets no fixed slope, so the same trench goes to a person.

How we built it

• NVIDIA on Nebius. Nemotron 3.5 Super VL (nvidia/Nemotron-3.5-Super-VL) runs on Nebius Token Factory through its OpenAI-compatible API, with thinking off and a response_format JSON schema. It returns points, not opinions. NVIDIA Cosmos Transfer 2.5 (2B, distilled edge control) ran on a Nebius AI Cloud RTX PRO 6000 in us-central1. It turned three outdoor clips into nine rain, mud and dusk-glare variants, about 50 s per sample.
• Measurement. Python, OpenCV ArUco and NumPy. Model lines are snapped to image edges with gradient profiles and a robust line fit. Confidence = point quality × marker quality × agreement across frames. The seat check uses librosa onset strength and spectral centroid.
• Grounding. Tavily Map, Crawl, Extract and Search build a cached jurisdiction pack per country from allowlisted seeds, storing each source's URL, retrieval time and SHA-256. Tavily Research watches for amendments and recalls, and proposes policy.update through the same gate. Decisions never wait on the web.
• Governance. Decionis AgentSafe through the hosted dci gateway. Intents are canonicalized to reproduce every published Agent-Safe conformance vector byte for byte. Each intent commits to its expected effect before dispatch. SiteWork refuses to claim a grant when its own facts contradict an ALLOW. Dossiers verify with @decionis/verify.
• App. FastAPI and WebSockets with a no-build vanilla JS dashboard, a full-screen phone stack light and a dossier page. 294 Python tests and 13 UI tests.

Results

The completed evaluation is an input-refusal test: whatever SiteWork can't measure, it must never release. It covers three raw outdoor pipe/drainage videos and the Cosmos variants of two of them, each paired across US and SE rules: 18 cases, all labelled ESCALATE before the run. It ran against the real Decionis gateway in Shadow Mode.

Gate p95 latency was 735.5 ms, and 18/18 dossiers verified offline. These are nine unique videos, not 18 independent recordings. None shows a calibration marker, so measurement accuracy isn't claimed; that needs the measured bench captures.

Live checks against the real gate in Enforcement Mode, with synthetic clips:

• In-limit decking and trench were allowed, claimed, finalized, and turned the light green.
• A 59.8° trench wall (true 59.84°) was blocked.
• The same trench under Swedish rules escalated.
• A named human release turned the light blue.

Challenges we ran into

• Model coordinates aren't measurements. On synthetic renders with known geometry, Nemotron VL's raw points gave decking gaps of 10.4/42.3 mm against a true 5.0/15.5 mm, and a trench slope of 87° against 60°. Snapping the points to image edges in code brought trench and plumbing angles within 0.25°. Asked for board edges, the model returned box diagonals; asking for corners fixed it. It also answers on a 0–1000 grid, whatever image size you state.
• No audio models on Token Factory. Every model we probed rejected input_audio (HTTP 422), so the seat check is deterministic DSP, and a borderline reading escalates.
• Execution grants need finalizing. A claimed but unfinalized grant goes INDETERMINATE and locks its target. SiteWork now finalizes every claim inside its 30-second lease.
• Honest scope. Ten raw field videos had no calibration marker or measured truth, so we used them, and their Cosmos variants, to test refusal rather than claim accuracy.

Accomplishments that we're proud of

• A release that is fail-safe end to end. Green requires a Decionis ALLOW, SiteWork's own facts agreeing, and the single-use grant claimed and finalized. A re-decided verdict never releases.
• Every number in Results comes from the evaluation script, and every dossier verifies offline.
• Even changing the rules is governed. A Tavily Research run found Swedish amendments (AFS 2025:6 and 2025:8). Decionis escalated the proposed update, and a named reviewer rejected it because the amendments concern asbestos, not trench criteria.

What we learned

Vision models are excellent at where to look and poor at how far: let the model point and let code measure. Weather variants are as useful for testing refusal as for testing accuracy. And the governance primitive agents need already exists on every construction site: the hold point.

What's next

• A shadow-mode pilot with a contractor's QA team on real hold points.
• Perception on a Jetson Orin Nano at the work face, with decisions still signed by Decionis.
• Measured bench captures for accuracy figures, and Cosmos variants of them.
• The same gate for excavators, pick-and-place cells and drones: any physical agent with an irreversible next step.

Built during the hackathon

Everything in the SiteWork repository was built during the submission period. Decionis and its published packages pre-date the hackathon and are used as a dependency, through the public API only.

Built With

• agent-safe-intent
• aruco
• cosmos-transfer-2.5
• decionis
• ed25519
• fastapi
• httpx
• javascript
• librosa
• nebius-ai-cloud
• nebius-token-factory
• nemotron-3.5-super-vl
• numpy
• nvidia-cosmos
• nvidia-nemotron
• opencv
• pydantic
• python
• tavily
• uv
• websockets