Inspiration

Ask anyone learning Italian what trips them up, and the same small words come up: articoli and preposizioni! Italian fuses prepositions with articles, and the right form depends on the word that follows and on which way things are moving. Learners memorise tables of these forms for months.

But these rules are about space: on, in, off, next to. Mixed reality is spatial too. So we asked: what if you could learn them by moving real things in your real room, with your own hands?

As native Italian speakers, we also wanted the Italian to be right, and we wanted something people would use every day, not try once. Ten minutes with your morning coffee, in the room you already sit in.

What it does

Lang turns your home into an Italian lesson.

  • Your room gets labels. Furniture from your room scan and everyday objects the headset recognises (a cup, a book, a plant) get floating Italian labels with their articles: la tazza, il libro, lo zaino. If the headset misses something, pick its word from a list and stick it on the object yourself. Stickers stay exactly where you put them between sessions.
  • You weave sentences. Fenn asks to go somewhere. Pinch Fenn and drag a glowing thread to a real object, pick a preposition, and the words fly off Fenn, the ring and the object's label on glowing trails and assemble in the air: Fenn salta sul tavolo.
  • Prepositions one step at a time. Beginners do not meet the whole ring at once. The first four times you play your room, Weave opens with a short lesson: su against da (drop Fenn onto the table, pull her off it), then in, then the lo and l' forms, then two-word prepositions such as accanto a. Fenn acts out an example, two quick questions practise the joined forms (su + la = sulla), and the ring and the fusion chips offer only what you have learned so far. Scene games then practise the same prepositions on new words: join su + il gatto into sul gatto, or pick the Italian for next to the cat.
  • The movement picks the preposition. Fenn the fox asks to go somewhere, and you can carry her there with your hand. Drop her onto the table: Fenn salta **sul* tavolo.* Pull her off it: Fenn salta giù **dal* tavolo.* Weave a thread instead and pick the wrong preposition, and Fenn moves the wrong way.
  • You fuse the words. The preposition flies off the ring and the article lifts off the object's label. The two pieces wait side by side, you choose the fused form, and they slam together into one word, with a snap and a burst of gold when it is right: sul tavolo, sullo scaffale, sull'armadio, sulla sedia. Get it wrong and Fenn wobbles and slips off. You learn from the mistake without a red X.
  • It remembers what you forget. Spaced repetition brings words back right before you'd lose them. The labels fade, and you recall each word at its place in your room, the oldest memory technique there is: tie a word to a place. If the object is behind you, a small arrow and a soft sound show where to turn.
  • Your knowledge becomes visible. Words you know glow steadily on your furniture and words you're forgetting flicker. Your room is a map of what's in your head. When the room section brings words back for review, Fenn sticks a yellow note on each one, Ripassami! (review me!), and it comes off once you recall the word. Labels show only in the room section, so at home and in the other games your room stays as it is. At the end of a room section, its threads shrink onto your wall as a small constellation.
  • Pick a section. Fenn asks Cosa vuoi imparare oggi? and a home menu of big tiles opens, like the Quest launcher: your room, Dov'è?, Italian gestures, hide and seek, topics, Show me, and one tile per scene. Play any section as often as you like; there is no daily limit, and every replay mixes new rounds.
  • Visit a scene. Pick the zoo, the aquarium, vehicles or the street, and the objects of that place appear on your table, one or several at a time: un gatto, tre gatti. On the farm, the bull and the donkey leave the table and walk around your furniture, fish, rays and jellyfish swim in front of your TV (or over your table when the room has no TV), and the other animals breathe quietly where they stand. Outdoor and water scenes tint your room and add a quiet sound: soft blue and surf at the aquarium, wind and birds at the zoo. Fenn first shows the new words one at a time, with their article and voice, then short game rounds practise them and bring back words due from other scenes: name it, find it, how many (pick the number or hold up that many fingers). In find-it rounds the objects spread over the tables and shelves in front of you, so you look across the room for the answer. Then the cat jumps onto the dog, hides under it, floats above it or sits next to or near it, and you fill the gap: Il gatto è ___ cane. Pick from sotto il, sul or sopra il, and accanto al or vicino al (never both of a pair in one round). Every object, model and voice clip is already on the headset, so scenes load without a network and work offline.
  • Rounds that change every time. Scene games mix name it, find it and how many with spot Fenn's mistake, what's missing from the table, sort the words by article, join the preposition and the article, and the Italian for on the cat. Every replay draws new rounds.
  • Fenn feels alive. Wave at Fenn and she greets you by the time of day: Buongiorno!, Buon pomeriggio! or Buonasera! Fenn looks at you, her mouth opens and her head lifts with every syllable she says, and she blinks when you look at her. Pet her back and she rolls over like a happy dog. Answer three in a row on the first try and she raises a paw for a high five.
  • Progress you can see. Every answer earns XP and fills a daily goal ring. Each section ends with its own stats: words learnt, words said well, items won and missed, and the time it took. Seven badges mark 1, 10, 25, 50, 100, 250 and 500 words learnt, and each one gives Fenn, in her reading glasses, a new scarf colour, from coral to gold. The profile shows your level (one per badge), Fenn's scarf colour, words learnt, XP, game items won, your daily streak (with a freeze for busy days) and your badges. Badges are also Meta achievements, so they show on your Meta Horizon profile.
  • Show me a word. Pick the Show me tile and choose a word. Its model appears on your table: Ecco un elefante! The word joins your reviews.
  • Dov'è? Pick the Dov'è? tile. Fenn asks where something in your real room is, Dov'è il libro?, and you answer aloud (Sul tavolo.) or pick sul tavolo, sotto il tavolo or accanto al tavolo.
  • Italian gestures. Pick the Gesti italiani tile. Fenn teaches four friendly Italian hand gestures with their phrases: thumb up for Va bene!, an open palm for Basta!, the ring for Perfetto! and the bunched fingertips for Ma che vuoi?. The headset reads your finger joints, so you make the gesture and Fenn cheers when it is right.
  • Hide and seek. Pick the Nascondino tile. Fenn says Chiudi gli occhi!, then hides on the floor behind or next to a piece of real furniture you have learnt, and purrs from there. Behind a sofa, the Depth API hides her from view, so you look around your room for her. Pick where she is, dietro il divano or accanto al tavolo, and she leaps back: Eccomi! Sono dietro il divano.
  • A postcard from every scene. At the end of a scene, Fenn poses on your table with up to three of the models you just played with, under a caption such as Una giornata allo zoo! The hint shows how to take a photo with Camera in the Meta menu, so you can keep or share the moment.
  • Answer aloud if you like. In Recall, Weave, Dov'è? and Topics you can say the answer instead of pinching its chip: the word with its article, or the preposition or fused form (sul, sulla). Speech is recognised on the headset and never leaves it. Speaking is always optional.
  • Topics. Pick a topic pack from the Topics tile, or say a topic in English, like animals or food. Every pack is curated: there are no generated words.

A section takes a few minutes, and ten minutes covers your room and a scene, entirely with your hands, seated.

How we built it

  • Unity 6 with the Meta XR SDK: passthrough, the Mixed Reality Utility Kit for room understanding, spatial anchors so labels stay put, and the Depth API to place detected objects in the room and to hide Fenn behind real furniture.
  • Hand tracking from the Meta XR Core SDK for every interaction: pinch, hand rays to reach far objects, and thumb microgestures to move through the preposition chips without lifting your hand.
  • On-device speech only. Whisper-tiny recognises Italian and English speech on the headset with the Unity Inference Engine. Lang uses no LLM and no cloud AI, and no voice or text leaves the headset.
  • Passthrough Camera API with an on-device object detection model through Meta AI Building Blocks and the Unity Inference Engine, placed on real objects with the depth frame. Camera frames never leave the headset.
  • A pure C# Italian grammar engine (gender, il / lo / l' / la by first sound, the full articulated-preposition table, simple and compound spatial prepositions, state vs motion verbs) with full unit tests, plus a spatial relation classifier that reads where things really are in your room for Dov'è? and hide and seek, and tells whether you dropped Fenn onto, into or off an object.
  • SM-2 spaced repetition: each review grows the interval, $I_n = I_{n-1} \cdot EF$, where $EF$ is the item's ease factor.
  • Rewards (XP, word badges with Fenn's scarves, streak freezes, game items won) in the same pure C# core, mirrored to Meta Horizon achievements.
  • Word audio (Amazon Polly, every clip approved by a native Italian speaker) and 3D models (most generated with Meshy, the rest scans from Poly Haven and Google Research, and the farm's walking bull and donkey from Quaternius) are prepared ahead of time and ship with the app, so it makes no network calls while you use it. The whole experience works offline.

Challenges we ran into

  • Looking was not enough to choose. Picking menu cards and models with head or eye gaze was unreliable from a chair, and some pinches chose a different item than the lit one. Every choice now uses the ray from your hand, a short line shows where each hand points, and the hands are read every frame so a pinch counts at once.
  • Labels in the wrong place. On device, labels first hung 30 to 40 cm below the furniture, then floated a few centimetres above it. We moved to a floor-level tracking origin, fixed the text pivot so each word sits on its point, and tuned the lift on device.
  • Seated, the models stood almost in your lap. Scene models now stand farther away, where the hand ray reaches them easily, and walking animals keep their distance from you.
  • The word Fenn asked for was not on the table. The zoo once asked for il bambù, borrowed from another scene, among four animals, and groups spread so far round the room that the right one stood behind you. Scenes now borrow only words of their own kind, and answer groups stay in front of you.
  • A language model froze the Quest 3S. Fenn first wrote her lines with a small on-device model (Qwen2.5-0.5B). Its 8-bit weights were unpacked for every token, the headset ran out of memory and Android sent APP_CMD_LOW_MEMORY. We removed it: every line now comes from templates checked by the grammar engine, and the app is faster and steadier for it.
  • Speech recognition was too kind. Whisper-tiny on the headset accepted wrong pronunciations as right, so we hid the pronunciation rounds. Voice now only picks an answer, and the chips are always there.
  • The microphone heard Fenn. Listening right after Fenn's question picked up her own voice, so Lang waits until Fenn stops speaking.
  • Polly read some words wrongly. A few Italian words got the wrong stress or were read as English. A native speaker listened to every clip, and we respelled those words before baking them again.
  • Hundreds of objects, no network. Every model and voice clip is baked ahead of time and shipped with the app. Free scanned models covered only part of the catalogue, so Meshy generated most of the models from text.

Accomplishments that we're proud of

  • Grammar you can feel: the direction of your hand picks the preposition, and you fuse prepositions and articles with your fingers. We have not seen Italian taught this way anywhere else.
  • A complete, satisfying session in ten minutes or less, entirely hands-only, seated.
  • An experience that works fully offline: object detection runs on the headset, and every word, model and voice clip ships with the app.
  • Every voice clip approved by a native Italian speaker.

What we learned

  • Design for the headset, not the editor. Labels, pinches and menus that worked in the editor failed on the Quest 3S, and only daily device tests found it.
  • A small headset needs small models. The language model froze the Quest 3S, and templates checked by the grammar engine gave Fenn better Italian.
  • Seated learners point with their hands. Hand rays were more reliable than eye or head gaze for every choice.
  • Speech recognition on the headset is good at picking an answer and poor at judging pronunciation.

What's next for Lang by Radost IT

  • More languages: Spanish, French and Portuguese coming soon.
  • Rooms beyond the living room: kitchen and office vocabulary packs.
  • Support for Meta VR Glasses with gaze-and-pinch interaction.
  • Launch on the Meta Horizon Store as a one-time purchase with a free trial: no subscription and no in-app purchases, since every AI feature runs on the headset.

Built With

  • amazon-polly
  • c#
  • depth-api
  • gltfast
  • hand-tracking
  • meshy
  • meta-ai-building-blocks
  • meta-platform-sdk
  • meta-xr-sdk
  • microgestures
  • mixed-reality-utility-kit
  • object-detection
  • passthrough
  • passthrough-camera-api
  • python
  • spatial-anchors
  • unity
  • unity-inference-engine
  • whisper
  • xunit
Share this project:

Updates

Submission history