LatteArtXR: Pour decisions? Never again.

• 

  About LatteArtXR

We had versioning issues in Meta build upload, here is the link to the apk, timestamp can be provided: https://drive.usercontent.google.com/download?id=10CK9fvkI1_612UJfxcppcVMbyD3wb6XW&export=download&authuser=0

Updated build from SensAI Hack Barcelona (Old version: https://devpost.com/software/b03-latte-art-pour-decisions-never-again )

Although the development started in the competition period, since the old version we made significant changes: -Using PCA for Calibration which we introduced to enhance accuracy -The entire UI has been redesigned to feel more interactive and engaging, actively informed by feedback from the general public to seasoned baristas. -Added guidance system to help endusers educate about features -Moveable coffee station so users can reposition it to a place where usage is convenient -A possibility to select more patterns through hand interaction -New patterns added for learning (recorded by head roaster from Bonafede Coffee Roastery) -A new menu system with 2 modes: training and real pouring -Social media integration: Users can share their real creations on Facebook and on Whatsapp -Gamified Learning Experience: Achievement Drawer and Reward System

List of changes in the github repository: https://github.com/danieloquelis/LatteArtXR/pulls?q=is%3Apr+is%3Aclosed+merged%3A%3E%3D2025-12-02+

Inspiration

Since I drink 2 coffees per day I only had limited chance to learn latte art and with that my first perfect latte art took 4 months of daily 1-2 pours. I want to learn quicker other latte arts and I was not willing to waste milk.

What it does

XR Latte Art leverages mixed reality hand tracking to solve a surprisingly complex motor learning problem: latte art. Traditional learning requires expensive repetition and immediate feedback is impossible—you only see results after the pour is complete. Our solution overlays 3D pour patterns directly into the user's physical space, tracks pitcher movement in real-time using hand tracking, and provides instant biomechanical feedback. Users practice the precise wrist rotations, pour heights, and movement speeds required for specific patterns (like heart, rosetta, tulip, swan, etc.), with scoring based on trajectory matching and timing accuracy. Once you are done with practicing, you can do the real deal with real milk and coffee.

The result: accelerated skill acquisition through deliberate practice, zero material waste, and transferable muscle memory that works with real coffee.

Once you are done, the app will take a personalized picture of your masterpiece which you can share later with other coffee enthusiasts and will give you AI based analysis from the picture taken and the precision data sent to the AI.

How we built it

We began with a focused ideation phase, prioritizing a strict MVP to avoid scope creep and ensure we could deliver a complete, high-quality experience within the hackathon timeframe. Our team split into two parallel workstreams: -Three team members developed the hand-tracking movement recording system along with the replay and evaluation engine. -Two team members focused on the user-facing experience, UI/UX flow, and overall ambience. We leveraged Meta’s Hand Tracking SDK to capture high-fidelity hand joint data. To record a real barista’s movements, we generated a JSON file containing the full frame-by-frame positions and rotations of both hands throughout the demonstration. This JSON serves as the “gold standard” animation sequence: during training sessions, we replay this movement for the user and compare their performance against the recorded data to measure accuracy. For the final evaluation, we integrated OpenAI. Using passthrough camera access, we capture an image of the user’s completed action (endresult of the latte art pouring). This image is then analyzed by an OpenAI model to provide contextual feedback that goes beyond geometric hand-tracking data.

Challenges we ran into

-Being able to find the best technique to follow the movement of the accessories in hand to be used. -Handtracking of the MetaSDK is not accurate enough to describe the precise movement of the barista with the current technology (fallback we used: use controllers with virtual objects overlayed for recording of the training movements) -Some errors with Meta SDK (since we were using the newest version of Meta SDK and Unity there were some incompatibilites and errors)

Accomplishments that we're proud of

-Being able to to iterate from the initial idea through extraordinary individual improvement points -Being able to record, track, measure and playback hand movements with accuracy -Being able to provide pleasant experience for endusers in this short timeframe with ambience -Being able to test the concept in real environment with actual professionals on the field (2 baristas in coffee shops) and apply the feedback to the product. -Being able to do the movement recording for use from professional to ensure that the data is high quality.

What we learned

-How to mix real objects with the MR space and interact with them in a way that we can also interact with the application at the same time. -Good UI designer is key and we were really blissful with ours -You can calculate scores based on fitness for data analysis -Macbook is faster at building -Collaborating with multiple expertise

What's next for Latte Art: Pour decisions? Never again.

Phase 1: Expand the Pattern Library We'll add advanced latte art patterns including swan, phoenix, dragon, and competitive-level designs. Users can unlock progressively harder patterns as their precision scores improve, creating a skill tree from beginner hearts to championship-level pours. Phase 2: Community Pattern Sharing Enable users to record their own signature patterns and share them. A professional barista in Tokyo can upload their award-winning rosetta technique, and enthusiasts worldwide can learn it exactly as performed. Phase 3: The Skill Marketplace - Beyond Coffee Our core technology—hand motion recording with precision scoring—applies to any craft requiring muscle memory. We're building a marketplace where experts teach and learners master: Calligraphy & Hand Lettering - Brush stroke pressure, angle, and flow Knitting & Crochet - Tension control, stitch consistency, pattern following Pottery & Ceramics - Wheel throwing hand positions, clay centering pressure Musical Instruments - Drumstick technique, guitar fingerpicking patterns, piano hand positioning Cake Decorating - Piping bag pressure, icing consistency, pattern precision Woodcarving - Chisel angles, grain direction, pressure control Sushi Rolling - Rice distribution, rolling pressure, knife techniques

Each craft gets the same treatment: experts record their hand movements, the system analyzes the biomechanics, learners practice with AR guidance and precision scoring. Master craftspeople monetize their expertise. Hobbyists learn from the best without geographic barriers. The Vision: Become the definitive platform for hands-on skill transfer—where muscle memory meets mixed reality.

Built With

• chatgpt
• elevenlabs
• metasdk
• unity
• zero-shot-grounding-dino