The problem (Agriculture & Sustainability)
Bees pollinate a huge share of the food we eat, and healthy apiaries depend on rearing new queens. The core step, grafting, means transferring a larva of exactly the right age (~12–24 h — a tiny C-shaped grub in royal jelly) from a worker cell into a queen cup. It is delicate, eye-straining, expert-only work: choose a larva too old, or an egg, and the queen fails. That bottleneck limits how many beekeepers can sustainably grow their colonies.
What we built
Beehive Grafting automates the decision and the transfer:
- OpenCV locates every honeycomb cell in a top-down photo and crops each one.
- A cheap OpenCV filter discards obvious non-targets (reflections, empty, sealed) to focus the AI on real candidates.
- Google Gemini looks at each candidate cell and classifies it —
egg/young_larva/old_larva/empty— with a confidence and an independent graft-suitability score. Gemini makes every biological decision in the system. - The candidates are ranked and the single best larva is selected.
- Its pixel is projected — via camera intrinsics and the robot’s TF tree — onto the honeycomb plane to a 3D pose in the robot’s base frame.
- A ROS 2 + MoveIt arm runs a collision-free Home → Approach → Pick → Lift → Place → Home cycle, placing each larva into the next slot of a 5×5 grafting tray (verified in a Gazebo + RViz simulation).
How we built it
A Python/Flask vision service (OpenCV + Gemini + ranking) is wrapped as a module
and consumed by a ROS 2 Jazzy stack: a Gazebo Harmonic world with a camera over a
honeycomb-textured plane, a vision_node that feeds the live camera frame to
Gemini on demand, a coord_transform module for the pixel→plane→base projection,
and a C++ MoveIt robot_node for the pick-and-place. Components are cleanly
separated (Vision · Coordinate Transform · Motion · Pick · Place), and the motion
node consumes only the published /selected_pose.
Rather than asking Gemini to analyze an entire honeycomb image, OpenCV first isolates individual cells, allowing Gemini to focus exclusively on biological reasoning while reducing latency and cost.
Challenges
- Large multimodal Gemini requests (100+ crops) returned empty/blocked — solved with batching + graceful degradation.
- Distinguishing young vs old larvae vs eggs is exactly where a plain prompt was weak — solved with an in-context RAG reference gallery of ideal young larvae (see below).
- Pixel→world accuracy — solved by sizing the camera so the comb fills the frame and verifying the projection with an RViz marker before any motion.
What we learned
In-context few-shot multimodal examples can replace training a custom model; reliability engineering (batching, strict JSON, fallbacks) is what makes LLM vision usable in a real pipeline.
What’s next
A real gripper and arm, a physical comb, and field trials with beekeepers.
Log in or sign up for Devpost to join the conversation.