Inspiration
Queuing to pay is the worst part of shopping. The stores that got rid of the queue did it with ceilings full of sensors, weighted shelves and custom cameras, which no corner shop, bookshop or small supermarket can afford. We wanted the same experience on what a shop already has: the cameras on its ceiling, a tablet at the door, and PayPal.
What it does
Carryout lets you pick up what you want and walk out. PayPal bills you for exactly what you carried.
- Sign up once at the door. Look at the camera, enter your name and email, and save a PayPal account or card. Next time, the door recognises your face and lets you straight in.
- Shop as normal. When you take something off the shelf, it goes on your bill. Put it back and it comes off. In a bookshop, it reads the title on the book in your hands.
- Walk out. As you leave, your bill is checked one line at a time against pictures of you carrying things. You pay once through PayPal and get the invoice by email. Anything it isn't sure about goes to staff instead of onto your bill.
- Run the shop. Staff see sales, customers and every bill, with the pictures behind each line. They can add a line that was held back or refund an item in one click.
- Add products by showing them. Hold a new product up to the camera and turn it. Carryout picks the best pictures and fills in the name, brand and category. Staff only type the price.
How we built it
Carryout uses several AI models, each with one job, and a charge only goes through when they agree.
- SAM 3.1 (Meta) outlines every person, hand and product in each frame. Carryout tells who took what by watching hands and products move together. All products are checked in one batch, in about 139 ms a frame.
- DeepSeek V4.1 Flash, a vision model, takes a second look at each item picked up, reads book titles, and checks the bill before anyone is charged.
- DINOv2 (Meta) compares what a shopper holds with the product's pictures.
- OpenCV's YuNet and SFace recognise faces at the door in about 15 ms.
The check at the door is a short series of small questions rather than one big one: what is in their hands as they leave, is each item still with them, did anything go back on a shelf on the way out, and which item was it. Every answer is one vote. If the model can't tell, the line isn't charged.
PayPal is where the AI's decision becomes a purchase:
- Vault saves the shopper's PayPal account or card when they sign up.
- Orders charges it when they walk out: one payment per visit, with every item listed.
- Invoicing writes the matching invoice and marks it paid, and Resend emails it.
- Refunds give back one item or the whole bill from the dashboard.
- Every request carries an ID made from the visit and the item, so a retry can never charge anyone twice.
Behind it is one FastAPI app on Render, the GPU models on Modal, and face recognition backed by Neon Postgres with pgvector, so finding a shopper stays one quick lookup however many people sign up.
Challenges we ran into
- Every book looks like "book". The camera can outline a book but not name it, so we read the title off the cover in the shopper's hands, from a close-up of the hands. A title has to be read twice before it goes on the bill.
- Which book went back? Two books with the same price and size were hard to tell apart when one was returned. We now ask for the colour first and the title second, which fixed it.
- Names drifting onto neighbours. A phrase meant for a power bank once outlined a tissue pack next to it. Now an item's name needs a second opinion before it can change.
- Keeping it live. Checking every product in one batch halved the time per frame, and the slower vision-model checks run alongside the video, so the camera never waits.
- Fitting in 512 MB. Moving the heavy models to the GPU kept the server steady at about 375 MB.
Accomplishments that we're proud of
- A complete purchase from a phone video: three books picked up, one put back, and USD 154.00 charged through PayPal for the other two, with the invoice emailed. Eight runs out of eight billed the right two books.
- It follows several shoppers at once, each with their own bill.
- It needs no special hardware: one ordinary camera and a laptop.
- Every charge can be explained. Open any bill and you see the pictures and steps behind each line.
What we learned
Two kinds of model, one tracking shapes and one looking at pictures, catch each other's mistakes far better than either does alone. A vision model answers many small, clear questions much better than one big one. And when money is involved, holding something for a person to check beats guessing.
What's next for CarryOut
- A trial in a working supermarket, on its existing cameras.
- A bigger catalogue, added a few products at a time so accuracy stays high.
- A camera at the door and several across the floor.
- More shops and more shoppers on the same system.
Built With
- cv
- fastapi
- render
