Fufi | Devpost

Short Description Fufi is a phygital companion that brings together a soft, comforting physical presence ☁️ with a dynamic digital experience to help kids with autism build healthy routines and gradually grow into more independent heroes. 💪✨ Inspirations

Existing tools like pictogram boards and Time Timers support routine-building but remain static and exclude the child's personal traits. Research on AR picture prompts shows autistic students can better learn and maintain multi-step routines (Cihak et al., 2016). We combine these insights into a Mixed-reality experience and star the children users in their own environment.

What It Does Fufi is a phygital AI assistant that helps caregivers guide children through personalized daily routines. Families co-design a custom 3D character and receive a physical figure that comes alive in mixed reality, displaying step-by-step pictograms starring the child or their favorite hero. Using computer vision, voice interaction, and a caregiver app for setup, tracking, rewards, and analytics - making routines engaging, intuitive, and fun.

Prototype Features

Phygital companion “Fufi” that “comes alive” as an animated character via computer vision.
Personalized pictograms starring the child, generated from a photo or prompt.
Guiding paths and arrows show where to go and what to pick up.
Smart step detection updates routines automatically as actions are completed.
Designed for autism & ADHD: clear, predictable visual templates that reduce overwhelm and build independence. *Gamification: reward streaks by unlocking outfits and upgrades.

How We Built It System architecture diagram: https://drive.google.com/file/d/1S3QxKvFV7Z_hniwWe4m6tCwnHYyZUwYs/view?usp=sharing Use case diagram: https://drive.google.com/file/d/17DbJ5bEApJbXNhuAmbC3RpzMVKWYMOp3/view?usp=sharing
Unity & Meta Quest 3: interaction prototyping Custom computer vision dataset & model: https://universe.roboflow.com/jiayao-yu/fufi-u5dae Local Roboflow inference server: faster object tracking (~0.5 second latency) in passthrough camera video streaming Figma design: https://www.figma.com/design/PHA57VDLWC0vQh7zcMpfav/PicturaMR ShapesXR design: https://shapes.app/space/view/a35c3ccd-be6f-4af8-a82b-d11f91d60891/ws8zda95