Project M.E.G.A.N

An MR AI mentor that lives in your real room, walks to you, visualizes concepts in 3D, and proactively supports students through academics, finances, and wellbeing.

Comment

Project M.E.G.A.N

Multimodal Empathetic Guidance & Academic Navigation

Inspiration

The idea for Project M.E.G.A.N was born from a simple observation: student life is fragmented.

A student at Algoma University juggles course deadlines, tuition payments, mental health, part-time work, and a packed schedule — all managed through disconnected portals, emails, and spreadsheets. No single system understands the full picture of a student's life.

We looked at the reality on the ground: Algoma University is facing a 60% decline in international enrolment, a $6.6 million tuition shortfall, and a student population that dropped from over 9,000 to approximately 4,500 in a single year. Students arriving in a new country, navigating an unfamiliar academic system, managing tight budgets, and living alone in a small Northern Ontario city are among the most at-risk populations in Canadian post-secondary education.

We asked ourselves: what if a student had a mentor who lived in their room, knew their schedule, understood their finances, could explain any concept visually — and reached out before a crisis happened rather than after?

That question became Project M.E.G.A.N.

The name draws from two real Algoma University staff members — Megan Jones-McLeod (Access Services, Wishart Library) and Megan Parlowe (Communications) — both of whom embody the university's spirit of proactive, cross-cultural student support. Megan the AI inherits that same mission: to be present, empathetic, and frictionless.

What It Does

Project M.E.G.A.N is a Mixed Reality AI mentor running on the Meta Quest headset. She is a full-body animated character who physically exists in your room — she patrols your space, walks toward you when you speak, and responds with voice, gesture, and floating 3D visualizations.

Core Capabilities

1. Conversational AI with Institutional Knowledge Megan knows Algoma University inside and out — every program, tuition fee, graduation requirement, OSAP deadline, admissions pathway, and campus support service. Ask her anything:

"What's the graduation average for the BSW program?" "How much does international tuition cost this semester?" "What is corridor funding and why does it matter?"

2. 3D Concept Visualization — Any Topic Say "explain photosynthesis" or "map out the causes of World War 2" and a floating 3D node graph appears in your room. Megan walks through each node sequentially, pulsing it as she speaks. Touch any node with your fingertip for deeper detail. Works for any subject — science, history, CS, economics, philosophy.

3. Real Schedule Analysis Megan fetches your actual Google Calendar via a public iCal feed. She renders your week as height-scaled 3D pillars — taller means busier. Overloaded days pulse red. Touch any pillar and she describes that day's events and flags burnout risk.

$$ h_{\text{pillar}} = \frac{\text{loadH}}{\text{maxHoursRef}} \times H_{\text{max}} $$

4. Financial Visibility Megan visualizes your monthly budget as stacked 3D income vs. expense pillars, flags deficit months, and gives specific money-saving advice relevant to international students.

5. Proactive Intervention When Megan detects signals of burnout, financial stress, or academic crisis in your conversation, she doesn't wait for you to ask for help. She physically walks to you and initiates the conversation.

How We Built It

System Architecture

The full pipeline from voice to action is:

$$ \text{Pinch} \rightarrow \text{STT (Whisper)} \rightarrow \text{GPT-4o} \rightarrow \text{Action Dispatch} \rightarrow \text{TTS} + \text{3D Visual} $$

Every GPT-4o response is parsed for a trailing JSON action block:

{"action": "VisualizeData", "data_type": "photosynthesis"}

This drives all Unity behavior. No hardcoded triggers, no keyword matching — the AI decides when and how to act based on context.

Spatial Awareness — Meta MRUK

Megan uses the Meta MR Utility Kit to understand the physical room:

  • room.GetFloorAnchor() — snaps her Y position to the real scanned floor
  • room.GetRoomOutline() — gets the actual room polygon for patrol waypoints
  • NavMesh.SamplePosition() — validates every waypoint is walkable
  • agent.Warp() — teleports her back if she ever falls off the NavMesh

Voice Interaction — Pinch to Speak

We use Meta's SpeechToTextAgent building block with OVRHand.GetFingerIsPinching(). Hold the pinch to record, release to send. The STT block handles VAD internally. Manual Stop Only mode gives full control over recording duration.

Dynamic Topic Graphs

When a topic graph is requested, GPT-4o generates a structured JSON response:

{
  "topic": "Photosynthesis",
  "nodes": [
    {"id": "root", "label": "Photosynthesis", "x": 0.0, "y": 0.3,
     "isRoot": true, "detail": "The process plants use to convert light to energy."},
    {"id": "n1", "label": "Chlorophyll", "x": -0.5, "y": 0.0,
     "isRoot": false, "detail": "The pigment that absorbs sunlight."}
  ],
  "edges": [{"from": "root", "to": "n1", "relationship": "requires"}],
  "speakOrder": ["root", "n1", "n2"],
  "explanation": "Photosynthesis converts light energy into glucose..."
}

Nodes are spawned as Unity PrimitiveType.Sphere objects with NodePokeCallback components that detect index fingertip proximity using OVRSkeleton.BoneId.Hand_IndexTip. Each node starts dim and is revealed with a pulsing glow as Megan speaks about it.

Institutional Knowledge Injection

The entire Algoma University academic calendar — programs, fees, regulations, history, financial situation, strategic projects — is compressed into the GPT-4o system prompt. This makes Megan dramatically more useful than a generic AI assistant for Algoma students.

Challenges We Ran Into

NavMesh Not Ready at Scene Start

StartPatrol() fired before MRUK finished scanning the room, causing:

"GetRemainingDistance" can only be called on an active agent placed on a NavMesh.

Solution: agent.isOnNavMesh guards on every NavMesh call + EnsureOnNavMesh() which calls agent.Warp(hit.position) every frame until she lands successfully.

iCal Timezone Parsing

Google Calendar iCal timestamps use UTC (Z suffix). Converting to local time shifted events outside the 7-day window, showing all days as "Free":

return isUtc ? dt.ToLocalTime() : dt;

Meta SDK API Inconsistencies

Multiple documented APIs turned out to be private or version-specific:

  • SpeechToTextAgent.manualStopOnly — private field, not settable externally
  • HandGrabInteractable.HasSelectingInteractor() — required an argument in our version
  • HandGrabInteractable.SelectingInteractorCount — didn't exist at all

We built NodePokeCallback and PillarPokeCallback — lightweight MonoBehaviour components that poll fingertip proximity each frame. More robust than SDK events.

GPT JSON Reliability

GPT-4o occasionally wraps JSON in markdown fences despite instructions not to. Defensive stripping:

if (json.StartsWith("```"))
{
    int first = json.IndexOf('{');
    int last  = json.LastIndexOf('}');
    json = json.Substring(first, last - first + 1);
}

TTS Speech Sequencing

When walking through a concept graph node by node, each GPT call would fire before the previous TTS finished, causing overlapping speech. The SpeakAndWait coroutine solved this:

yield return new WaitUntil(() => !IsTTSSpeaking());
// → GPT call → SpeakText() → WaitUntil done → next node

Graph Layout — Star vs. Tree

Early concept graphs connected everything directly to root, producing a star pattern instead of a meaningful hierarchy. Fixed by enforcing tree layout rules in the prompt:

$$ y_{\text{root}} = 0.3, \quad y_{\text{level 1}} = 0.0, \quad y_{\text{level 2}} = -0.35 $$

With explicit instructions: "NEVER connect a leaf node directly to root."

Accomplishments That We're Proud Of

  • Any-topic concept graphs in 3D — you can say "explain black holes" or "map out the French Revolution" and a meaningful, hierarchical node graph appears in your room within seconds. No hardcoding, entirely GPT-generated.

  • Real calendar integration with zero authentication — Megan reads your actual Google Calendar events from a public iCal URL and renders them as physical objects in your space.

  • Proactive behavior without scripting — Megan decides when to walk to you, when to spawn a visualization, and when to flag a crisis entirely based on GPT context. There are no if/else rules for this.

  • Full Algoma University knowledge — Megan can answer specific questions about the BSW graduation average (70%), the UHIP cost ($792/year), the Makwa Waakaa'igan project, corridor funding advocacy, and the Shingwauk legacy — because all of it lives in her system prompt.

  • Fingertip-driven interaction — touching floating 3D nodes and schedule pillars with your physical index finger in Mixed Reality feels genuinely magical. No controllers, no buttons — just reach out and touch knowledge.

  • A complete MR AI character — Megan patrols, navigates, faces you, talks, visualizes, intervenes, and remembers your conversation across turns. She feels like a presence, not a chatbot.

What We Learned

  • MRUK floor anchors are not always axis-aligned — real scanned rooms have slightly tilted normals. Never assume world Y = 0 for floor height.

  • JSON in reply body outperforms function calling for Unity — parsing a trailing {} block with JsonUtility.FromJson is simpler, faster, and more reliable than OpenAI's function calling API when working with coroutines.

  • Spatial audio and timing define MR presence — the 0.6s buffer before checking IsTTSSpeaking() was the difference between Megan feeling alive and feeling laggy.

  • System prompt size is worth it — a 3,000-token institutional knowledge block transforms a generic assistant into a genuinely useful university advisor. The specificity of answers improves dramatically.

  • Hand tracking requires physical design thinking — the node sphere size, spacing, and poke collider radius all needed to be tuned for real hand movement in physical space. What looks good on screen may be unreachable in MR.

What's Next for Project M.E.G.A.N

Short term:

  • LLM-driven registrar actions — Megan drafts emails to advisors, fills OSAP forms, and books Learning Centre appointments on the student's behalf.
  • Emotion detection — using voice tone analysis to detect stress and trigger ProactiveIntervention without the student having to mention it.
  • Multi-student mode — Megan facilitates study groups, assigns roles, tracks each student's contribution to a shared concept map.

Medium term:

  • Persistent memory — Megan remembers previous conversations, tracks grade trends over time, and surfaces patterns the student hasn't noticed.
  • Expand to all Ontario universities — the institutional knowledge system is modular. Any university's academic calendar can be injected as a new knowledge block.
  • Indigenous language support — Anishinaabemowin (Ojibwe) voice interaction, aligned with Algoma's Special Mission and language revitalization goals.

Long term:

  • Makwa Waakaa'igan integration — when Algoma's $45M global centre opens in 2026, Megan becomes the spatial interface for the Shingwauk Residential Schools Centre archives — students explore history by walking through it in MR.
  • Predictive academic intervention — trained on anonymized retention data, Megan identifies at-risk students 6 weeks before they withdraw, not after.
  • The Frictionless Campus — a fully AI-augmented university experience where every administrative interaction, every tutoring session, and every wellness check-in happens through a single spatial AI that knows you, your program, and your institution.

Megan was built for Algoma University. But the problem she solves — fragmented, reactive, impersonal student support — exists at every university in the world.

Built With

Share this project:

Updates