Leap motion Gesture Controls.
Making an Affinity!
Voice interview to learn your preferences and create an AI agent that represents your taste
Affinity Fam
“Affinity’s agents discuss the options, explain their recommendations, and help you choose what to buy.”
“Affinity helps your group choose where to eat by considering everyone’s preferences.”

Affinity

Inspiration

Affinity was inspired by Nintendo Mii and the idea that the people you care about could each have a digital character of their own.

Most AI shopping tools personalize recommendations for only one user. Real decisions are often social: choosing a gift, planning a family night, or finding something everyone can enjoy. We wanted to explore what happens when AI represents each person individually, then helps those perspectives work together without replacing the people themselves.

What it does

Affinity is a 3D, AI-powered shopping and group-planning experience built around the people you care about.

Users create customizable Affinity characters for friends and family. Through a voice or typed conversation, Affinity extracts interests, dislikes, personality traits, and important constraints into a locally stored profile.

When someone starts a shopping mission, each invited character becomes a personal agent. The agents independently explain what matters to them, respond to one another's public opinions, reconcile preferences with hard rules and budget limits, and score the resulting recommendations. If satisfaction is too low, the system automatically revises its search. Users can also reject one product and request a targeted replacement without restarting the entire decision.

Affinity searches live Shopify product data and can surface product images and available 3D models. Once the group agrees, the user reviews the recommendation and performs a Leap Motion to approve it. Affinity then rechecks price and availability before creating the Shopify cart and handing the user off to checkout. Nothing is purchased automatically.

Affinity can also plan shared experiences, such as dinner and an activity, using Google Places and send an approved itinerary to participants.

How we built it

We built the interface with Next.js, React, Three.js, and React Three Fiber. The application includes a Mii-inspired plaza, a character creator, animated agents, multiple 3D environments, and an interactive council where users can see the agents deliberate.

For motion control, we used the Leap Motion Controller 2. A native C bridge built on LeapC converts tracking frames into WebSocket data, while a TypeScript gesture engine handles pointing, pinching, dragging, swiping, and handshake approval. Every gesture also has a keyboard or pointer fallback.

The backend uses Python, FastAPI, and LangGraph. LangGraph coordinates parallel personal-agent opinions, public deliberation, search planning, independent scoring, revision loops, human approval, Shopify preflight, and cart creation. OpenAI structured outputs power preference extraction and agent reasoning, while deterministic code enforces budgets and hard constraints.

We integrated Shopify UCP for live catalog search and merchant carts, Google Places for group plans, and Deepgram for speech recognition and character voices. WebCrypto provides an AP2-style signed mandate so the approval is bound to the exact proposal the user reviewed.

Challenges we ran into

One major challenge was keeping a multi-agent system understandable and trustworthy. Each character needed access to its own profile without exposing everyone's private information, while still being able to respond to public opinions and negotiate compromises.

Live commerce created another challenge. Products can become unavailable or change price after the agents recommend them. We added a preflight step that refreshes Shopify data before cart creation. If the price changes, the system asks for approval again; if an item disappears, it searches for a replacement.

Motion control also required translating noisy 3D hand coordinates into reliable interface actions. We added smoothing, gesture state machines, calibration controls, and conventional input fallbacks.

Accomplishments that we're proud of

We are proud that Affinity combines character creation, conversational personalization, visible agent-to-agent deliberation, a real-time 3D interface, physical hand gestures, and live commerce in one complete flow.

A user can create someone, describe what they care about, invite their agent into a council, watch multiple agents discuss and score live products, replace an unsuitable recommendation, approve the final proposal with a signed handshake, and continue to a real Shopify cart.

Most importantly, the agents can recommend and negotiate, but the human remains in control of approval and checkout.

What we learned

We learned that multi-agent systems are most useful when the agents have genuinely different context and responsibilities. Personal agents understand individual people, while the shopping agent understands products and providers.

We also learned that AI reasoning works best alongside deterministic safeguards. Language models can interpret preferences and generate compromises, but code should enforce budgets, hard rules, consent, and commerce boundaries.

Finally, we learned that bringing people closer together with AI means helping people understand one another, not allowing AI to silently make decisions for them.

What's next for Affinity

Next, we want Affinity characters to develop durable, opt-in memory from real interactions. A user could say, "She would never wear that," or, "He has been really into running lately," and Affinity could propose a profile update for confirmation rather than treating every AI inference as permanent truth.

We also want richer character customization, improved motion controls, multi-device councils, and stronger integrations with existing AI assistants.

On the commerce side, we want to extend the signed mandate into tokenized, user-controlled payment authorization, while preserving price rechecks and requiring fresh consent whenever a proposal changes.