Chhaant

A produce-picking cell that inspects each item like a careful shopper with OpenCV 5 on Graviton4, and asks a person exactly where its camera can't be trusted.

Comment

Inspiration

In a 2023 LocalCircles survey of Indian consumers, about 73% said the fresh produce they ordered online was of low quality. Dark-store pickers work at speed, and a careful shopper's habits, like turning the tomato over or buying the firm avocado for the weekend and the soft one for tonight, don't survive that speed. I couldn't find a robot that chooses produce by quality for an individual order; existing systems grade on a conveyor or pick by position. So I built one, and I built it to be honest about what its camera can't see.

What it does

A customer types an order in plain words, such as "2 tomatoes for a salad and an avocado for Tuesday". An agent on Amazon Bedrock, Amazon Nova 2 Lite, reads it into order lines, and the cell, not the model, turns "Tuesday" into a number of days. The arm, simulated in MuJoCo, puts each item on a turntable. OpenCV 5 checks the photo is usable, cuts the fruit out of the background, reads its colour in CIELAB and runs a quality model on the new DNN engine. The cell turns the item until the views agree and enough of the surface is covered, or hands it to a person, and it records which part of the surface it never saw. An avocado is packed only if its shelf-life forecast says it will be ripe and still good on the day it will be eaten; shelf life is validated for avocados only.

Where held-out validation shows the camera can't be trusted (my model missed every spoiled tomato), the cell puts the decision to a person, with the photos and the reason. The rules live in code, not in the prompt: the agent can't pack what a person declined, discard a sound item without a person's yes, or call a short order complete. The cell is also an MCP server on the same site, so any agent can work the crate under the same rules.

How I built it

All image work is OpenCV 5: capture checks (blur, glare, darkness, a cut-off item, no item), segmentation, CIELAB colour, crops, view coverage, and the ONNX quality model on OpenCV's new DNN engine, tested against the PyTorch outputs recorded at export. On top of the vision sit view fusion with rotate-until-sure, a shelf-life forecast with an interval from two photos and the temperature, per-line routing, and sequential crate acceptance. The web app is FastAPI on an EC2 c8g.large (Graviton4) running the Cloud-Optimized OpenCV Library (COOL, OpenCV 5.1 with KleidiCV), behind CloudFront, with every run's evidence in S3 and metrics and alarms in CloudWatch, all in one CloudFormation stack in ap-south-1. Every decision is made on real photographs from public datasets.

Challenges I ran into

The hardest finding was my own model: on held-out data it missed 10 of 10 spoiled tomatoes. Rather than hide that, I made the cell send every tomato to a person and say why. The second challenge was measuring the whole cell, not just the model. I built 32 everyday orders over 4 real crates, scored every run against the datasets' hidden labels, explained every wrong pack from its trace, fixed the cause and measured again. Over six iterations, measured with Claude Haiku 4.5 as the agent, wrong packs went from 22 to 0. Models also miscounted named days, so day counting moved into code. When I gave the same 32 orders to Claude Code over MCP, it found two gaps in my rules that my own agent never hit, and I fixed both.

Accomplishments that I'm proud of

On the 32 orders, Amazon Nova 2 Lite packs zero wrong items while filling 94% of what the crate could fill, in about 12 seconds and $0.024 per order. Claude Code, an agent I didn't write, packed 65 items over MCP with none wrong. Rotating until sure made 38 wrong calls where looking at all 16 angles made 69, using about 4 views instead of 16 and handing 51 of the 312 held-out sessions to a person. Avocado shelf life from two photos and the temperature is off by 1.05 days on average (426 held-out fruit). The capture checks raise 0.8% false alarms on good photos and catch 95 of the 105 spoiled views that a bad photo had turned clean. I also state the limit: when a person answers wrong 10–20% of the time, 11–14% of packs are wrong, so the zero rests on the person.

What I learned

The honest answer is often "ask a person", and the value is in saying exactly where and with what evidence. I also measured instead of assuming: COOL and the PyPI aarch64 wheel run the cell's vision at the same speed with identical readings on the same Graviton4 (both ship KleidiCV), and Graviton4 costs 30–41% less per item than x86.

What's next

A physical arm and turntable in place of the simulation, a live camera, a stronger tomato defect signal so fewer tomatoes need a person, and a trial in a real dark store.

Built With

Share this project:

Updates

Submission history