NYC Robo Garden: Vision-Guided Robot Tasks

An agent uses OpenCV 5 evidence to authorize and verify simulated garden-robot tasks on AWS.

Comment

Inspiration

During the Spooky Civic AI Hackathon, we worked with Judith "Judi" Desire, founder and executive director of Uptown & Boogie Healthy Project. She described the hours spent manually sowing individual seed-tray cells. NYC Robo Garden explores affordable, adaptable robotics for small farms, community gardens, education and workforce development. The current OpenCV demonstration addresses a software prerequisite: visual evidence should change whether a robot acts and whether its work is considered complete.

What it does

An OpenCode agent connects through Open Industries MCP tools to an AWS-hosted simulation of an ASTRA quadruped with an SO-101 arm. It reads the current state, commands a bounded move, captures a rendered camera frame, examines OpenCV evidence, and requests a harvest only with fresh evidence from the correct bed and revision. Each harvest records before/after images, checks reduced color evidence in the pickup region, and verifies a simulated deposit. The robot can pause, resume, and return home. The browser is a read-only view of the same persistent station.

How we built it

The model-issued demonstration used OpenCode with openai/gpt-6-astra and SSH stdio MCP. The meaningful AWS workload runs on EC2 t3.small in us-east-1: the Python farm service, MuJoCo rendering and movement, OpenCV 5.0.0 analysis, and private SQLite state on EBS. The farm adapter imports a pinned version of the existing NYC Robo Garden model and uses its original meshes and joint-limited kinematic segments. New commands require an immutable UUID and the current state revision. Harvests additionally require a fresh OpenCV observation. Images, command receipts and jobs persist across reconnects and service restarts.

OpenCV's role

OpenCV measures configured color candidates inside a fixed pickup region of actual rendered MuJoCo images. Its output is a required gate for the subsequent harvest command and part of the post-action verification. Missing, stale or mismatched evidence cannot authorize a harvest. Crop labels are fixture metadata: we do not claim semantic crop recognition, ripeness estimation or arbitrary target localization.

Results

The saved AWS run completed seven jobs: three moves, three harvests and return-home. All three simulated items were deposited. Pickup-region target pixels changed from 126 to 0 for carrots, 618 to 0 for lettuce, and 436 to 0 for tomatoes. Pause/resume and same-command retry were exercised. Service restart and an independent SSH MCP reconnection preserved the same terminal state and all seven jobs. Separate verification documents 36 passing tests, including absent-target, stale-evidence/revision rejection, authorization and paused-job restart cases. These are fixture results, not population-level reliability or physical farm performance measurements.

Challenges

We needed to connect agent decisions to persistent execution rather than to an animation loop. That required bounded action contracts, fresh evidence checks, exact retry semantics, saved job state and verification after movement. Rendering and vision on a CPU-only cloud host also required explicit dependencies and a reproducible software-rendering configuration.

What we learned / what we are proud of

We demonstrated an actual model-issued sequence of robot task commands with retained visual evidence, durable results and recovery checks. The agent selects the next allowed task segment; the runtime enforces the contract and verifies the result. Keeping observations, command receipts and job results separate made failures and replay behavior observable.

Limitations

The camera is synthetic and the robot is simulated. Movement follows fixed, joint-limited kinematic segments; grasp attachment and release are scripted. The test environment uses known beds, fixed image regions and color thresholds. We have not validated physical contact, outdoor navigation, seed placement, crop recognition, human labor savings or economic return. The existing seed-station and patrol animations are earlier simulation work, not footage of this agent-controlled AWS execution.

What's next

Validate camera-to-arm calibration and one-seed-per-cell placement on the physical workstation; measure misses, doubles, seed damage, operator time and recovery. Compare the complete task with manual work and a dedicated tray seeder. Expand perception and motion only after the bounded task is repeatable.

Prior work / competition contribution

The garden mission, robot assets, grower framing and project homepage originated in the Spooky Civic AI project. This entry builds on that work with the OpenCV camera/evidence service, persistent MCP execution, AWS deployment, the farm control adapter, and the recorded model-issued perception-action-verification run. See the pinned code and original model source in the technical report.

About the video

The demo combines earlier simulation footage, a newly rendered and labelled kinematic replay, and original recorded AWS images and execution results. It is narrated with a generic synthetic voice. It is not a fresh live recording of an agent session or physical robot footage.

Technical report, source, and reproduction

Judge runtime access — pending

The AWS station currently requires authenticated operator access. Public judge runtime access and a live screen-share appointment are not yet confirmed. The attached video, technical report, source and recorded evidence can be inspected independently. The project homepage provides context, not access to the private AWS station.

Built With

Share this project:

Updates

Submission history