DataHarvest

The physical data marketplace bridging the simulation-to-reality gap for robotics. AI labs post bounties, everyday people record spatial data, and serverless AI auto-scores the results.

Comment

Inspiration

Today, 73% of robotics and embodied AI teams cite the lack of real-world training data as their #1 blocker. We realized that web-scraped internet video cannot solve this because it lacks physics. YouTube videos don't have camera trajectories, LiDAR depth maps, or synchronized IMU data—the exact ground-truth metrics that physical robots actually need to learn. We were inspired to solve the "simulation-to-reality gap" by unlocking the most powerful sensor suite on the planet: the modern smartphone in everyone's pocket.

What it does

DataHarvest is a two-sided physical data marketplace.

  1. AI Research Labs post bounties specifying exact physical actions they need data for (e.g., "Pick up a red cup").
  2. Everyday Collectors use our iOS app to record the action. The app doesn't just capture video; it simultaneously records ARKit 6DoF camera poses, LiDAR depth maps, and 100Hz IMU sensor data.
  3. Serverless AI automatically grades the submission. If our AI pipeline verifies the video meets the bounty requirements, the collector gets paid, and the lab gets pristine, robot-ready training data.

How we built it

We built a highly distributed, serverless architecture to handle massive multimodal data:

  • Frontend: Next.js 16 and Tailwind CSS hosted on Vercel.
  • Database & Auth: Supabase PostgreSQL and Edge Functions (Deno).
  • Mobile Capture: Native Swift iOS app utilizing ARKit and CoreMotion.
  • The AI Brain (Backend): We deployed a custom Python pipeline on Modal (Serverless GPUs). When a video is uploaded, Supabase triggers Modal to run 5 parallel AI models: Gemini (for task grading/evaluation), YOLOv26 (object tracking), MediaPipe (21-joint hand pose estimation), and SAM 3.1 (semantic segmentation).
  • Search: We integrated TwelveLabs to allow labs to perform natural language semantic searches across the entire video corpus.

Challenges we ran into

Orchestrating heavy GPU inference across a distributed architecture was incredibly tough. Initially, we hit hardware constraints when trying to deploy our heavy SAM 3.1 and YOLO models to Modal on a free tier. We had to dynamically rewrite our serverless classes and utilize Codex/GPT-5.6 to help us downgrade from restricted H100 GPUs to L4 and T4 GPUs without breaking the pipeline. Additionally, securely syncing environment variables across four different platforms (Vercel, Supabase, Modal, and TwelveLabs) required a flawless deployment strategy.

Accomplishments that we're proud of

We are incredibly proud of successfully linking a physical mobile sensor suite directly into a heavy cloud-GPU pipeline that executes in seconds. We didn't just build a wrapper; we built a completely automated evaluation pipeline that actually scores human movement using advanced AI models, effectively removing the human bottleneck in data validation.

What we learned

We learned that serverless GPU orchestration is immensely powerful but demands strict, unforgiving environment architecture. We also learned how to use LLMs (like Codex and GPT-5.6) not just as code-generators, but as senior DevOps co-pilots that can actively read our project graph and help us debug cloud deployment constraints in real-time.

What's next for DataHarvest

Our next major milestone is expanding our 3D Gaussian Splatting capabilities so that roboticists can take our iPhone LiDAR data and train their models directly inside perfectly reconstructed 3D spaces. We also plan to release a native Android application to unlock millions of new data collectors globally, turning DataHarvest into the definitive data engine for physical AI.## Inspiration

Built With

Share this project:

Updates