TALE: Training Agents for Learning Experiences

Learn the skill that could save a life. You can't miss the Magic School of Societal Safety, where we help you create enterprise-grade VR training simulations that let you connect with your circle

Comment

Inspiration

When something goes wrong, most people freeze. We wanted to flip that.

Fairy tales already teach through stories in which you are the hero, so we created a VR training simulation platform to choose your story and simulate the steps you need to help you, your close circle, and the people around you in critical situations. Building a custom training world by hand is slow and expensive. Agents can generate one on demand, so your customiziable VR simulation helps you gain the essential life-safety skills.

What it does

TALE: Training Agents for Learning Experiences is a generative, agentic VR training simulation platform. Four inputs (topic, audience, world, social level 1-3) become a playable, gesture-controlled fantasy simulation.

Main ideas

  1. Generative: the simulation is built from your request, not authored by hand.
    • Agents write the story, the set, the palette, the scenery and the props as a SimulationSpec (JSON).
    • The player renders the hall, props, rigged characters and effects from that spec.
    • Approved custom assets are saved to asset_library/ and reused.
  2. Agentic: a team of single-purpose agents with a checker, not one big prompt.
    • Director, ResourceScout, SkillDesigner, Storyteller, WorldBuilder and Validator, each in its own file.
    • The Validator rejects bad specs and sends the failures back for a rebuild (max 3 rounds).
    • A control plane standardizes intent, applies deny-by-default security, plans resources and evaluates every build.
  3. VR training you act out: you do the skill with your hands.
    • First-person 3D with webcam hand tracking: grab, point, open palm, pinch, wave, thumbs-up, plus pull, push, lift, sweep, swipe, trace, circle.
    • You are the heroine, and the story speaks to you in second person.
    • Headset VR (WebXR) is planned. Today it runs on desktop with a webcam.
  4. Skill-first, so the lesson is real: each act teaches one skill as an ordered list of steps (gesture + prop).
    • Interactions are generated in code from those steps, so the story can't skip, reorder or invent one.
    • Fire safety has curated skills (extinguisher PASS, fire blanket rescue, evacuation). Other topics are designed by the LLM.

Features

Feature Details
Social levels Cumulative. Level $N$ plays acts $1 \dots N$ (self, friends/team, community).
Ask Push-to-talk (hold button or V). Answers are at most 3 plain second-person sentences, shown as text.
Find my people End screen suggests real groups, clubs and nearby places (e.g. fire stations). Links out only: nothing is scraped, posted or joined.
Create wizard 11 plain-language questions at /create/ build a new training with no code.

How we built it

request -> Director -> ResourceScout -> SkillDesigner -> [Storyteller -> WorldBuilder -> Validator] x<=3 -> SimulationSpec (JSON) -> web player
  • Contract: SimulationRequest in, SimulationSpec out. Nothing else reaches the player.
  • Control plane (aep/control/):
    • standardize: free text to a closed-vocabulary AgentSpec
    • security: model recommends rules from a fixed list; PolicyEngine enforces them, deny by default, with an audit log
    • compute: derives model tiers and rebuild budget
    • evaluation: criteria plus deterministic checks on every build
  • Player: three.js, MediaPipe hand tracking. The wizard is React. The backend is FastAPI.
  • Voice: audio is posted to our server and relayed to Muse Voice Transcribe, so the API key never reaches the browser. Muse Spark writes the answer.
  • Social discovery: SerpAPI finds results. The LLM can only rank them by index, and every URL comes from search results. Facebook Login is optional and used only for consented profile signals.
  • Offline mode: with no API key, curated fire-safety templates still build a full simulation.

Challenges we ran into

  • Story drift: LLM stories skipped real safety steps. Fix: skills are the source of truth, and interactions are built in code.
  • Meta removed the Groups API (v19): group discovery can't come from Facebook. Fix: use SerpAPI and keep Facebook for profile signals only.
  • Model-granted permissions: limited to a closed rule list, enforced in a separate policy layer.
  • Gesture accuracy: pinch vs. grab on a laptop webcam. Thresholds still need tuning on real hands.
  • Visual quality: wrote docs/VISUAL_STANDARD.md. Agents design the set, palette and scenery, and the player renders them.

Accomplishments that we're proud of

  • Plain-language idea to playable training with no code.
  • The lesson can't be skipped: the schema and Validator reject any spec that does.
  • Deny-by-default security with an audit log, and no model-invented URLs.
  • Voice input with no API key in the browser.
  • Reusable platform: a new simulation family needs new agents, prompts, props and gestures, plus a new spec model.

What we learned

  • None of us had game developer/VR experience, so we grew familiar with 3D asset generation and virtual worlds.
  • Constraining what the model decides (skills, rules, URLs) beat prompting it to behave.
  • A bounded validate-and-rebuild loop catches most contract failures.
  • Third-party APIs change, so keep fallbacks (offline templates, SerpAPI).
  • Safety content needs expert review. Skills are not certified, and the end screen carries a disclaimer.

What's next for mssc

  • [ ] Expert review of curated skills (treat LLM-designed ones like CPR as drafts)
  • [ ] Test the player on real webcams and tune gesture thresholds
  • [ ] True headset VR (WebXR)
  • [ ] Read declared knowledge sources, and add per-user API auth
  • [ ] Link-check curated resource URLs
  • [ ] Run the Anthropic model path live
  • [ ] More topics and worlds

Built With

Share this project:

Updates

Submission history