Pomme

Pomme is a low-cost autonomous fruit-picking rover with a robotic arm, camera, and LiDAR. It maps changing orchards with SLAM, tracks fruit in 3D, avoids obstacles, and replans each pick in real time.

Comment 1

Check it out -> pomme.live

Inspiration

One of our teammates researches autonomous fruit gripping and soft robotics at a lab, and the same problem kept coming up in his work: fruit-picking is running out of people, due to the large agricultural labour shortage in Canada. During peak seasons, over 3.5 billion in sales are lost due to unfilled jobs. It is one of the hardest, lowest-paid, most seasonal jobs in agriculture, the labour shortage is getting worse every year, and almost nothing being built to fix it is actually autonomous. When there are not enough people to pick and grade fruit at the moment it is ripe, that fruit just gets left on the plant.

That is a large reason why 30 to 40 % of food is lost between harvest and shelf. A lot of it never even makes it off the farm, either because it ripened past its window or got sorted too late to sell. We wanted to attack that at the source, so we built Pomme: a low-cost robot that drives itself into the row, picks fruit, and sorts it by ripeness right where it grows. The pitch is "Battery, not Blood", one machine going after food waste, food prices, and back-breaking stoop labour at the same time, on 5 watts, with no cloud. By using modern technology, we can create solutions to problems that could never be solved before.

What it does

Pomme scouts, picks, and grades on its own:

  • It scans its surroundings with a 360 degree LiDAR and runs SLAM to build a live 3D reconstruction of the space while tracking where it is inside that map, then plans a path and drives itself to look for fruit, no operator in the loop.
  • A camera on the arm locks onto a fruit, and an on-device AI model reads whether it is ripe.
  • The 4-DOF arm reaches out, grips the fruit, and drops it into the right bin for its ripeness.
  • You can also just tell it what to do. Type "pick all the ripe ones" and our FarmHand language model turns that into a safe, checked robot command.
  • A live web dashboard shows telemetry, the LiDAR map, and every pick as it happens.

Every bit of perception and decision-making runs on the robot. Nothing goes to the cloud.

How we built it

Pomme is built from scratch: a custom drive base, a custom 3D-printed arm, and a full power stack (PCA9685 servo driver, buck converters for the separate rails, high-current motor drivers, LiPo battery).

The core of the design is the Arduino Uno Q, used as a true dual-processor system:

  • The Qualcomm Dragonwing MPU (Linux, about 5 watts) runs the vision model on-device and handles the high-level logic: perception, visual servoing, and the pick-and-sort routine.
  • The real-time MCU owns motion and safety. It drives the motors and servos, smooths the arm so it never snaps and browns out the rail, and runs an ultrasonic reflex stop in under 10 milliseconds plus a motion watchdog, all without waiting on Linux. If the vision side ever hangs mid-pick, the arm holds and the drive cuts on its own.

That split, a real-time safety brain next to a Linux AI brain on one board, is the design choice we are proudest of, and it is exactly what the Qualcomm Uno Q track is about.

The vision model is trained and deployed with Edge Impulse and runs entirely on the Uno Q, so ripeness detection happens at the edge with no network in the loop. FarmHand, our natural-language layer, is a small model trained to turn plain English into strict, validated commands, so a garbled instruction gets rejected instead of driving the arm. The 360 degree LiDAR drives a SLAM pipeline on the Uno Q that builds a live 3D reconstruction of the environment and localizes the robot within it in real time. On top of that map the robot plans its own path and navigates autonomously, and the map streams to the dashboard so you can watch it fill in as the robot explores. The dashboard is React and Three.js over Socket.IO, with MongoDB Atlas, Auth0, and Vercel.

Tech stack

Robot

  • Arduino Uno Q as a dual-brain controller: Qualcomm Dragonwing MPU (Linux, on-device AI, visual servoing, pick-and-sort logic) + real-time MCU (motor PWM, servo interpolation, sub-10 ms ultrasonic reflex stop, motion watchdog)
  • Edge Impulse vision model running fully on-device, no cloud inference
  • 360 degree LiDAR with a SLAM pipeline (scan matching + occupancy grid) for live 3D mapping and autonomous navigation
  • Custom 3D-printed 4-DOF arm (PCA9685 servo driver), BTS7960 motor drivers, multi-rail buck power stack, LiPo battery, bearings, various hardware components

AI

  • FarmHand: a small language model fine-tuned with SFT plus reinforcement learning to turn plain English into strict, schema-validated robot commands (Freesolo)
  • On-device inference inside a roughly 5 W envelope

Web and cloud

  • React, Three.js, and Vite dashboard with live telemetry, the SLAM map, and a pick log
  • Node, Express, and Socket.IO real-time hub
  • MongoDB Atlas (native time-series telemetry plus aggregation-pipeline analytics), Auth0 operator login, deployed on Vercel
  • PlayStation controller teleop via the browser Gamepad API

Challenges we ran into

  • Getting the vision model to actually see the fruit. It fought us for hours and would not detect reliably until we fixed the training data and pipeline.
  • Building a whole robot from scratch on a hackathon clock. A custom base, a custom printed arm, and a multi-rail power system all had to physically agree with each other, which meant fighting wiring, power, and mechanical problems before anything moved.
  • Getting the two brains to cooperate. Splitting perception and safety across the Linux MPU and the real-time MCU, and folding LiDAR, vision, and the arm into one loop that drives up and grips, took a lot of tuning to absorb slack and drift.

Accomplishments that we're proud of

  • A working autonomous fruit-picking robot, end to end, from raw hardware to on-device AI, built in a hackathon.
  • The full loop closed: see, drive, reach, grip, sort.
  • All perception and language understanding run on-device on the Qualcomm Uno Q, no cloud, so the robot is self-contained.
  • A genuine Linux-plus-real-time split on one board, with safety enforced independently of the AI side.
  • A completely custom build, printed arm and custom drive base, not a kit.

What we learned

  • How to design mechanically for the real world, so the build survives slack and imperfect assembly.
  • How to bring up LiDAR and turn raw scans into SLAM, a usable map, and autonomous navigation.
  • How to split a robot across a Linux MPU and a real-time MCU, and why safety and timing belong on the MCU.
  • How much of robotics is just iteration. The vision, the reach, and the drive all only worked after several passes.

What's next for Pomme

  • Tighter autonomous driving and pathfinding to scout rows faster.
  • A stronger arm with longer reach and a firmer grip for more fruit types and heavier loads.
  • More crops and finer ripeness grading so it can pick truly selectively.
  • Pushing more of the vision workload onto the Dragonwing GPU for higher frame rates.

Built With

Share this project:

Updates