RealiSIM

Website
Physical Demo
Physical Demo

Inspiration

Robots learn in simulation. But before a robot practises anything, someone has to build its world: model every can and cup, guess sizes, weights and friction, place the arm, wire it all into a simulator. It's slow, it's manual, and afterwards nobody can tell which numbers were real and which were made up.

We flipped it: start from reality, and make the simulation prove where every number came from.

What it does

Reality in, simulation out. Film your object or robot workspace on a phone. RealiSIM:

Autofills it. Claude spots each object and estimates size, mass and friction. Our Apify Actor finds the real product specs online, with sources.
Labels every value. Measured, Supplied, Estimated or Unknown. You approve; unknowns block the pipeline.
Builds the 3D. TRELLIS generates a textured mesh, scaled to your real dimensions, with CoACD collision hulls.
Sets the stage. Your table plus as many robot arms as you have (SO-101, SO-100 and more).
Physics-checks it. Every scene must drop, land and settle in MuJoCo before it can be exported.
Ships it. A pip-installable Gymnasium environment, tested in a clean install, hashed and stored forever.
Mirrors your real robot. Optional live mode streams real arm joints and tracked objects into the sim, read-only.

How we built it

✍️ Notability — From Research to Real-World Robotics

Notability was not a tool we added at the end; it shaped RealiSIM from our earliest research to our final robot deployment.

Before writing code, we imported research papers on the sim-to-real gap in robotics and used Notability's Chat with Your Notes to understand why simulation-trained robots struggle in physical environments. We annotated findings about inaccurate physics, domain randomization, and object reconstruction, using those insights to develop our approach to transparent physical parameter estimation.

Our handwritten sketches evolved into detailed flowcharts connecting AI perception, 3D reconstruction, MuJoCo simulation, reinforcement learning, and zero-shot deployment. We also used Notability to design our AI-agent interactions, multi-robot coordination workflows, Mana website layouts, and final demo storyboard.

Our workflow: research papers → AI-powered research conversations → handwritten sketches → system flowcharts → agent architecture → simulation development → physical deployment.

Notability captured our entire engineering journey, bridging the gap between design and reality, just as RealiSIM bridges simulation and real robotics.

🪐 SpacetimeDB — Real-Time Coordination for Multi-Robot Environments

SpacetimeDB became RealiSIM's dynamic, interactive state-management system, going far beyond an ordinary storage backend.

We used SpacetimeDB to persist projects, simulation versions, physical parameters, and artifacts in 512 KiB chunks. More importantly, we explored its real-time subscriptions and transactional reducers to support synchronized, multi-agent robotic environments. Our multi-arm workcells demonstrate why robotics needs constantly changing shared state: several robotic arms must track object positions, coordinate task assignments, and respond to changes in their surroundings.

We are extending this architecture toward conveyor-belt scenarios, where cargo moves continuously and multiple arms must coordinate pick-and-place operations. SpacetimeDB's reactive state synchronization can keep connected simulations updated as objects move, while reducers manage authorized changes to shared task states. This enables coordination without confusing database updates with direct hardware control.

Our workflow: live object tracking → shared state updates → robot task assignments → multi-arm coordination → synchronized simulation → physical deployment.

SpacetimeDB connects persistent simulation artifacts to the constantly evolving physical world, making interactive, multi-robot environments possible.

🕷️ Apify + OutRouter — Extracting Physical Properties From Video

Apify helped us address a fundamental robotics challenge: videos reveal appearance, but robots need physical properties to interact with objects realistically.

We integrated OutRouter through Apify's Actor ecosystem into our video-to-simulation workflow, using uploaded object footage to extract visual information and infer physical parameters such as dimensions, geometry, and material characteristics. Combined with Claude's object recognition and Apify-powered product research, this allowed us to generate more informed estimates of mass, material, and friction instead of relying entirely on unsupported AI guesses.

Because physical properties cannot always be determined from pixels, we developed a provenance system labeling every value as Measured, Supplied, Estimated, or Unknown. OutRouter-derived estimates and externally sourced specifications help populate these parameters, which users approve before TRELLIS reconstructs the 3D object and MuJoCo configures its physical behavior.

Our workflow: video upload → OutRouter extraction → Claude recognition → Apify specification research → parameter verification → 3D reconstruction → physics simulation.

By combining Apify and OutRouter, we turned video-derived information into structured physical inputs for robotic simulation.

🧠 Anthropic Claude — Turning Visual Information Into Robotics Intelligence

Claude became RealiSIM's reasoning engine, transforming unstructured phone videos into actionable robotics specifications.

A video contains pixels, but a simulator needs object identities, dimensions, materials, masses, and physical relationships. We integrated Claude to recognize objects, interpret workspace contents, and propose initial physical properties. Its outputs guided our Apify research and helped construct structured inputs for 3D reconstruction and simulation.

We deliberately separated AI reasoning from deterministic engineering. Claude handles ambiguous questions like "What is this object?" while specialized tools handle precise calculations and validation. BiRefNet isolates objects, TRELLIS reconstructs meshes, CoACD generates collision geometry, and MuJoCo verifies physical behavior. Estimated values remain explicitly labeled rather than being treated as measurements.

Our workflow: visual perception → object recognition → physical property estimation → source verification → structured scene generation → simulation validation.

Claude made RealiSIM intelligent without sacrificing engineering rigor: AI interprets reality, while deterministic systems ensure that simulations behave consistently.

🚀 Mana — Our Entire Robotics Platform, Frontend and Backend

Mana was not simply where we hosted RealiSIM; every component of our application was built and orchestrated through Mana, from frontend to backend.

We used Mana to develop our entire React interface, including video uploads, object-property approval, simulation configuration, workflow visualization, and artifact downloads. Behind the interface, Mana also runs our backend services, APIs, secrets, scheduled jobs, and pipeline orchestration, connecting Claude, Apify, SpacetimeDB, TRELLIS, CoACD, MuJoCo, and Gymnasium into a unified robotics platform.

We extended Mana beyond basic deployment with three parallel workers, memory admission, persistent checkpoints, and automated upstream updates. Our release gate checks research-code changes every 60 seconds and validates candidates before promotion. When 31 visitors and 14 builders began using RealiSIM, we addressed memory spikes, disk pressure, and stuck queues while preserving active jobs.

Our workflow: Mana frontend → video processing → AI agents → physical parameter estimation → 3D reconstruction → MuJoCo simulation → RL training → robot deployment.

Mana was the foundation of everything we built, transforming fragmented robotics research into a fully integrated, publicly accessible simulation platform.

From simulation to zero-shot deployment

We didn't stop at building the world. We taught robots to act in it.

Trained with reinforcement learning. We used our video-reconstructed 3D objects to create pick-and-place tasks in MuJoCo, training RL policies to grasp, lift and reposition objects entirely in simulation.
Enabled multi-agent interaction. Multiple robotic arms could operate in shared simulation environments, interacting with the same objects and learning coordinated manipulation behaviors.
Bridged sim to real. We deployed simulation-trained policies onto our physical robot for pick-and-place tasks, without collecting real-world demonstrations or training data on the hardware.
Zero-shot transfer. The robot encountered the physical task using only what it had learned in simulation. No hardware-specific policy training was required.

🤖 Factory Droid SDK — Automating Robotics Software Engineering

Our Factory Droid SDK integration concept extends RealiSIM's philosophy: if robots can learn tasks autonomously, why shouldn't the software supporting them improve autonomously too?

RealiSIM depends on rapidly evolving research tools, including the open-source photo2grasp pipeline, which received more than 20 commits during our hackathon. We built an automated release gate that checks upstream changes every 60 seconds, tests updates in isolation, and uses a Mana agent to review compatibility before deployment.

The Factory Droid SDK is our intended programmable engineering-agent layer for repository inspection, code maintenance, compatibility analysis, and automated fixes. This would allow engineering agents to evaluate changes against our stable interfaces while our robotics agents focus on scene construction and manipulation.

Our intended SDK workflow:

Upstream change → Droid analysis → Compatibility review → Proposed code fixes → Isolated testing → Validated release

The goal is to make RealiSIM increasingly self-maintaining, reducing the engineering overhead required to keep an autonomous robotics platform running.

Challenges we ran into

Video can't weigh things. Mass and friction aren't in the pixels, so we made provenance a first-class feature instead of hiding guesses.
A moving target. The research repo shipped 20+ commits mid-hackathon. We built auto-updates that check, review and promote them safely.
Real users, early. Strangers started building before we were done. Memory spikes, a filling disk and stuck queues all had to be fixed live, without killing anyone's job.
Everything has to agree. Units, frames, joint orders, collision vs. visual meshes, file hashes. One mismatch breaks the export.

Accomplishments that we're proud of

A full phone-to-Gymnasium pipeline that works live for people outside our team: 31 visitors and 14 builders since launch on Saturday evening.
Exports verified the way an engineer would: clean install, check_env with zero warnings, 2.6 ms per step, byte-identical re-downloads.
Honest by design: every number shows its source.
Multi-arm workcells and a read-only live hardware mirror at about 20 fps.
Upstream updates that ship themselves.

What we learned

Provenance is a feature. People trust a sim that admits what it doesn't know.
AI for the ambiguous, code for the exact. Claude answers "what is this?"; deterministic code owns geometry, physics and permissions.
Reproducibility means storing the bytes, not just the recipe.
Ship early. Real users find the bugs tests never will.

What's next for RealiSIM

Cloud artifact storage to scale far beyond one sandbox.
Any URDF, any arm.
Pilots with small robotics labs: one workcell, end to end, with a discrepancy report.