Inspiration
AI can identify objects in a room and suggest creative solutions, but it can also invent physical properties that have never been established. A hook may exist, but that doesn't prove it can support a heavy mirror. A surface may look stable, but that doesn't establish its load capacity.
ZERO was built around one principle: before AI uses an object in a physical plan, it should establish what that object can actually do.
What it does
ZERO transforms a video walkthrough of a real environment into an evidence-backed physical world model and uses that model to solve goals using resources already available.
Its reasoning pipeline follows:
Walkthrough → Keyframes → Visual Perception → Cross-Frame Entity Resolution → Canonical Inventory → Human Confirmation → Object DNA → Guarded Planning → Verification
Object DNA describes an object's grounded capabilities, limitations, unknowns, and cautions. The planner can only rely on capabilities supported by this evidence.
ZERO recognizes four outcomes:
SOLVED: Enough evidence exists to produce a grounded plan.
CONDITIONAL: A possible plan exists, but specific conditions must be verified.
NEEDS_CONFIRMATION: A user decision is required before planning can proceed.
CANNOT_SAFELY_SOLVE: No sufficiently grounded solution satisfies the constraints.
Instead of forcing an answer, ZERO knows when to solve, verify, ask, or refuse.
How we built it
ZERO uses NVIDIA Nemotron 3 Super through Nebius Token Factory for model-backed planning and semantic auditing, alongside visual perception using MiniMax-M3 through Nebius.
The backend is implemented in Python with Pydantic, OpenCV, FastAPI, and deterministic validation layers.
Model-generated outputs are treated as untrusted until validated against typed schemas, object identities, eligible capabilities, and goal constraints.
A human-confirmation boundary prevents uncertain observations from automatically becoming confirmed inventory. The system also preserves evidence fingerprints, supports bounded repair, and records terminal outcomes for reproducible evaluation.
Challenges we ran into
One of the biggest challenges was building a reliable physical world model from visual observations. The same object could appear across multiple video frames, and an AI model might infer capabilities that were never verified.
We addressed this using cross-frame entity resolution, canonical inventories, explicit confirmation states, and Object DNA.
Another challenge was preventing plausible but unsupported plans. ZERO needed to distinguish between situations where it could solve a goal, required physical verification, needed a user decision, or could not safely proceed.
Finally, we needed a way to measure reasoning quality without favoring ZERO over the baseline. We developed a neutral evaluation protocol and preserved the results for reproducibility.
Accomplishments that we're proud of
Four validated walkthrough outcomes:
SOLVED: A $0 product photography setup using existing objects.
CONDITIONAL: A temporary curtain requiring two physical checks before execution.
CANNOT_SAFELY_SOLVE: Refusing to invent a secure mounting method for a full-length mirror.
NEEDS_CONFIRMATION: Asking the user to choose between displaying a guitar or canvas artwork.
Benchmark improvements:
Across 15 paired runs using NVIDIA Nemotron 3 Super, ZERO reduced medium/high-severity findings from 19 to 7, a 63.2% reduction.
ZERO performed better than the generic baseline in 10 runs, tied in 4, and performed worse in 1.
We also preserved unsuccessful evidence: an additional unseen benchmark completed four of five cases, with one failure caused by an output-schema error.
Technical accomplishments:
Evidence-backed Object DNA and capability modeling
Cross-frame object identity resolution
Deterministic validation and semantic auditing
Human confirmation boundaries
Outcome-aware planning and bounded repair
Reproducible benchmark evidence
Publicly deployed interactive evidence demo
What we learned
The most important lesson was that strong language-model reasoning is not enough for reliable physical planning.
An AI can generate convincing suggestions while silently assuming that an object is strong, stable, or suitable for a particular use.
ZERO demonstrates why explicit evidence, deterministic checks, and uncertainty handling matter.
We also learned that refusing an unsupported plan can be just as valuable as generating a successful one.
Our benchmark results are encouraging, but they do not establish universal real-world physical safety.
What's next for Zero - Evidence-Bound Physical Reasoning
Our next priorities are:
Connect the hosted interactive interface to the model-backed ZERO pipeline.
Improve user-guided physical verification and confirmation workflows.
Test against more unseen environments and physical constraints.
Expand walkthrough support for mobile devices and wearable cameras.
Develop a more seamless experience from room observation to executable, evidence-grounded plans.
Our long-term vision is an AI that understands not just what objects are, but what they can reliably do.
Log in or sign up for Devpost to join the conversation.