Haggle Shack

Thumbnail
Front View 1
Front View 2
Front View 3
Side View 1
Side View 2
Back View
Staining
Building Frame

Inspiration

Some of our members are from near the coast, and it goes without saying that the beaches look disgusting. There's trash everywhere. Plastic bottles. Leftover sports equipment. Cans. Snack Wrappers. Every year, the community would organize a cleanup, but it did little to no good. We wanted to make something to incentivize small, continuous efforts that compound over time. The key is to make something trendy. That's where Haggle Shack comes in. It's interactive. It's funny. All the components needed for virality. That's our edge.

Haggling also turns a chore into a game. Nobody brags about throwing away a bottle, but everybody wants to tell their friends how they talked a greedy crab into paying double.

What it does

Here's how it works. First, grab trash from your local beach. Make sure it's not something natural: our algorithm will pick up on it and tell you to put it back!

Then, put it into the chest one piece at a time. The camera will analyze it and Haggly the Crab will give you an offer. You probably won't like it. Now's your chance to negotiate it. Once the negotiation is complete, you will login to redeem your shellcoins and a servo trapdoor will release your items into our disposal bin!

Keep going and soon you'll be able to redeem your rewards! Here at HackGT, we'll demo one way to go about the reward. Keep a running leaderboard and give away "merch."

Under the hood, a single trade goes like this:

The chest sees the item. A webcam looks into the treasure chest. Motion detection waits until the item has settled, then snaps a photo.
Gemini appraises it. The photo goes to Google Gemini, which identifies the item, its material, whether it's recyclable and how to dispose of it properly, and gives it a ShellCoin value. Litter that's especially dangerous to wildlife (fishing line, six-pack rings, plastic bags, cigarette butts, balloons) is worth more, because getting it off the beach matters most.
The crab lowballs you. The crab, an animated, shifty-eyed SVG character with a villain mustache, opens at about a third of the real value, in a scheming con-artist voice generated by ElevenLabs.
You haggle out loud. Hold the talk button and name your price. Gemini listens to your voice, works out whether you're accepting, countering or walking away, and the crab answers in character within about 3 seconds. It will meet you partway, give a "final offer", or refuse outright if you get greedy.
Deal! The ShellCoins land in your account and the chest's trapdoor opens to drop the item into the bin. Your balance and the leaderboard live in a cloud database, so they follow you across visits.

The kiosk runs as a full-screen app on a tablet or iPad. The chest also has a NeoPixel light show that constantly cycles through colors and flickers.

How we built it

Our build process was fun and jank at every turn.

The first night, we couldn't gather all the necessary hardware, so we got contacts, developed a bill of materials, experimented with different shack designs on Onshape, and tinkered with the software app (Gemini and ElevenLabs integration!)

The next morning was time to put the build together.

With the gracious help of a friend, we were given a ride to Home Depot and Target to get our resources. More on that trip below :) (Check challenges)

After gathering our resources, We put our wood together using tools freely accessible at the Student Competition Center. We also stained our wood; overall, this was roughly a six hour process.

In parallel, our software team was working to make our app.

Software: The app has three parts, all running on one laptop at the stand:

Server (Python, FastAPI): runs the kiosk website, streams the chest camera, keeps live state in sync over WebSockets, stores accounts and ShellCoin balances in MongoDB Atlas, and drives the trapdoor Arduino over USB serial. An admin panel shows the live stand, the leaderboard, and manual controls for balances and the trapdoor.
AI pipeline (Python): OpenCV motion detection triggers a scan. Gemini (through Google Cloud's Vertex AI) appraises the photo and handles every haggling turn in one multimodal call: it transcribes the customer's audio, classifies their intent, and writes the crab's reply. ElevenLabs voices every line.
Kiosk (HTML/JS/CSS): the animated crab lip-syncs to the audio and changes mood (greedy dollar-sign eyes, angry, shocked). The kiosk also has hold-to-talk voice recording, a live camera view, and login. It installs as a full-screen home-screen app (PWA) on an iPad, served over local HTTPS so the tablet's microphone works.

A key design choice: the AI writes the words, but plain code decides the numbers. Gemini proposes a value; fixed rules set the crab's opening offer (35% of the value), the price it accepts on the spot (up to 55%) and the most it will ever pay (85%). No amount of sweet-talking makes the crab overpay, and what it says always matches the screen.

Hardware: an Arduino Uno drives the trapdoor servo and a NeoPixel strip. It takes simple serial commands from the server (R90 opens the door, R180 closes it). The lights animate between moves and switch off while the door moves, so the servo gets all the power.

The last few hours involved integration. To keep spirits high, we built in the dorm lobby and invited the floor to come have fun with us! They cheered us on and brought snacks to keep us going. Huge props to them :)

Challenges we ran into

Technical Side: Primarily, we ran into issues with communicating between each separate subsystem in our project. Between the camera, speaker, multiple APIs, and an entirely separate monitor viewing a self-hosted webpage, a lot had to be coordinated for the system to work properly. Specifically, getting camera streaming to be pushed over a local network proved difficult. Additionally, we were able to minimize our round-trip-latency when querying our AI backend from ~14 seconds down to just under 3, which kept our project's UX feeling smooth and cohesive, with no major gaps in immersion.

How we cut the latency:

One AI call per turn instead of several: transcription, intent and reply all come from a single multimodal Gemini request.
Hedged requests: about 1 in 6 Gemini calls would stall for 10+ seconds. If an answer takes longer than 3.5 seconds, a backup request goes to a second model, and the first answer wins.
No dead air: pre-voiced filler lines ("Hmm, let me ask my accountant…") play the instant the customer stops talking, while Gemini thinks.

Other technical roadblocks:

Free credits vanished mid-hackathon. Our Gemini API key ran out of credits, so we moved to Google Cloud's Vertex AI, which draws on Cloud credits.
Campus Wi-Fi vs. the cloud database. The network's DNS was too slow for MongoDB Atlas's default connection string, so every connection took 13 seconds. We switched to a direct connection string, and it now connects in under a second.
iPads only allow the microphone on secure (HTTPS) sites. We built a small local certificate authority, so the tablet trusts the stand once and survives switching networks.
Safari couldn't play our live camera stream (MJPEG) over HTTPS, so the kiosk now fetches frames one after another (about 25 fps).
The NeoPixels stopped the servo from moving. LED updates and servo timing both depend on the Arduino's timer interrupts, and powering both over USB made it worse. We rebuilt the firmware so the lights pause while the door moves.

Logistics Side: Logistically, this project was pulled off with a combination of great timing and luck, although we ran into a few roadblocks on the way. First, gathering resources wasn't easy. The nearest Home Depot was 4 miles away, and we couldn't go until the morning of Saturday. Once we got there and gathered our 2x4 wood, we ran into another issue. Their cutter didn't work! That was really rough, so we had to crouch down in the car and run the wood over our heads for the drive :). After that, we needed to find tools to get the job done. Luckily, two of our members are members of HyTech electric racing and have 24/7 access to the student competition center. We went there, stained the wood, and borrowed a dolly to make the ~1 mile walk back from the student competition center to our dorm. Our final challenge will be tomorrow - getting our device to the CULC safely!

Accomplishments that we're proud of

Beyond minimizing our round trip latency on our AI backend, we also built a stable physical actuator to sort the trash placed into the chest between recyclables and landfill waste. Our actuator used a servo motor controlled by an Arduino Uno.

We were also able to bring our whole dorm floor together with the project. We did some testing with other people to try out our algorithm, and people couldn't stop playing with it! That was an amazing feeling. We were able to build openly, listening to music and talking to people while finishing up our hack! We made the process incredibly enjoyable, brought other people into the mix, and it truly made this weekend memorable for us.

We're also proud that:

The crab can't be cheated. Because the pricing rules live in code, not in the AI, the crab stays greedy but fair however the conversation goes.
It's a real product loop, not just a demo: accounts with PINs, persistent balances in a cloud database, a leaderboard, and an admin panel for prize redemptions.
It runs on the hardware we had: the kiosk works as a full-screen iPad app, the camera is picked by name so a USB webcam can be swapped in, and the stand recovers by itself when the Wi-Fi, database or Arduino drops out.
The character has personality: a scheming voice, shifty darting eyes, dollar-sign eyes when it's greedy, and claw-rubbing when it thinks it's winning.

What we learned

Dream big and don't be afraid to think big! People around you are willing to help. Without getting a ride and borrowing resources from the different competition centers around, it would be nearly impossible to execute on this project.

On the technical side:

Latency is the user experience. A 3-second pause feels like thinking; a 10-second pause breaks the illusion. Fillers and backup requests mattered as much as a faster model.
Let AI do language, let code do logic. Keeping money decisions out of the model made the system predictable and safe.
Integration takes the most time. Browsers, networks, certificates, serial ports and power budgets caused more trouble than any single component.

What's next for Haggle Shack

Deploying this to beaches and getting real data!

Although we have software prediction of whether a product is recyclable or trash, and our disposal actuator can physically sort between the two, we do not currently have multiple reservoirs to store our recyclables and trash deposited into our system. This is easily solved with more containers underneath the chest, and is our first improvement that we want to implement with more time.

Beyond that:

Weatherproof, solar-powered stands that run off a cellular connection instead of a laptop.
Real rewards: partner with local businesses and beach towns so ShellCoins redeem for discounts, parking or merch.
Impact tracking: every scan is already logged with the item type and material. A public dashboard could show what's washing up where, which is useful data for local cleanup groups and researchers.
Anti-gaming: a scale in the chest, plus limits per account, so nobody earns coins by feeding in the same bottle twice.
Social sharing: a "best haggle of the day" clip or leaderboard to lean into the virality.

Built With

Python · FastAPI · Uvicorn · WebSockets · OpenCV · NumPy · Google Gemini (Vertex AI, google-genai) · ElevenLabs text-to-speech · MongoDB Atlas (PyMongo) · SQLite (offline fallback) · HTML / CSS / JavaScript · SVG animation · Progressive Web App (service worker, web manifest) · Arduino Uno · Servo motor · Adafruit NeoPixel · pyserial · Onshape · wood, stain and a borrowed dolly