Inspiration

Physical setups are often tuned through trial and error: move a guide, release a ball, watch it miss, and try again. But a failed attempt contains useful information about friction, bounce, and motion.

I wanted to build a system that could learn from that information. REALITY.EXE treats the camera as an experimental scientist: observe the setup, predict what will happen, compare prediction with observation, and improve the next plan.

What it does

REALITY.EXE is a debugger for the physical world. Its goal is simple: make a ball reach a target on a tilted tabletop.

The system closes a perception–decision–action loop:

Measure: OpenCV 5 detects board markers, the ball, guide, obstacles, and target.
Plan: A numerical planner simulates candidate guide placements under several plausible physics settings.
Act: The interface gives a placement instruction and verifies the guide’s position before release.
Observe: The camera tracks the ball and measures the outcome.
Learn: After a miss, the system updates its physics estimates and plans another attempt.

Moving an obstacle invalidates the old plan. Blocking the camera pauses the workflow and requests a clear view. The interface shows predicted trajectories, a 3D digital twin, and a decision trace explaining what changed.

How I built it

OpenCV 5 handles ArUco detection, perspective correction, color segmentation, contour measurements, and Kalman tracking.

A vectorized NumPy model evaluates candidate plans quickly. Newton 1.7, running through NVIDIA Warp, supplies the independent simulated world used for evaluation. The agent receives rendered camera frames rather than the simulator’s hidden physics parameters.

FastAPI and WebSockets connect the Python backend to a JavaScript and Three.js interface.

The AWS deployment runs in Docker on Amazon EC2, with S3 for trial evidence and CloudWatch for decision logs. CloudFormation defines the infrastructure. The current public endpoint uses HTTP and the synthetic tabletop; browser webcam access requires HTTPS or running locally on localhost.

Challenges I faced

Full Newton rollouts on CPU were too slow for interactive planning. I separated the roles: a fast reduced model proposes actions, while Newton supplies the benchmark environment.

Another challenge was learning from failures without learning from bad measurements. Rejecting short trajectories initially discarded useful evidence when the ball stuck on the guide. Camera occlusion could also leave an incomplete trajectory. The workflow now handles these cases separately and excludes interrupted tracks from physics updates.

Results and accomplishments

Across 10 synthetic layouts, the adaptive system reached the target:

90% within two attempts
100% within three attempts

The same planner with fixed physics reached 40% within three attempts. Mean trajectory error decreased from 12.0 mm to 3.2 mm to 1.6 mm across the first three trials.

Moved obstacles triggered plan invalidation and replanning in all 10 tested cases, with zero stale-plan actions.

These are synthetic benchmark results. Repeated physical trials have not yet been conducted.

What I learned

Closing the feedback loop mattered more than starting with perfect physics. A simple model that updates from measured motion can outperform the same planner using fixed assumptions.

Uncertainty also needs to change behavior: when observations become unreliable, the system should pause, explain the problem, and request human help.

What's next

The next milestone is repeated physical testing with printed markers and a webcam, followed by automatic color calibration, improved handling of camera perspective, and GPU-accelerated physics experiments.