Cortana

Cortana: A Voice-First Clinical Learning Workspace Inspiration Healthcare professionals often have only a few minutes between responsibilities to review evidence or reinforce what they have learned. Most educational tools, however, demand focused screen time, long modules, or rigid navigation. I wanted to explore a more natural experience: an AI clinical colleague that could deliver a concise morning briefing, answer grounded questions, and lead a short learning exercise through conversation. The goal was not to replace clinical judgment or provide patient-care recommendations. Instead, Cortana is designed as a calm, voice-first learning workspace built around synthetic scenarios and transparent educational progress. What Cortana does Cortana gives healthcare professionals a focused, two-minute learning experience through the web or over the phone. A learner can:

Listen to a simulated clinical shift briefing.
Interrupt naturally and ask follow-up questions.
Explore vitals, laboratory results, schedules, consults, and hospital events.
Complete a short evidence-based cardiology exercise.
Receive deterministic, server-graded feedback.
Review supporting evidence and citations.
Save practice history, reinforcement topics, and review recommendations.
Ask Cortana to send a confirmed message to a preset front-desk email address.
Upload or select synthetic scenarios from a searchable context library. The experience centers on a luminous, audio-reactive orb rather than a traditional chatbot interface. Its movement responds to real microphone and agent-output levels, while visible transcripts and controls keep the interaction accessible. How I built it Cortana is built with Next.js, React, TypeScript, Tailwind CSS, Radix UI, React Three Fiber, and the ElevenLabs Conversational AI platform. The browser application uses ElevenLabs for live voice conversations and WebRTC audio. Instead of trusting the language model to grade answers or modify progress directly, Cortana exposes narrowly scoped tools. The server owns the answer key, validates every request with Zod, grades responses deterministically, and applies progress updates exactly once. Phone conversations use a separate ElevenLabs agent connected through Twilio or another SIP carrier. Outbound calls receive a short-lived signed session that identifies the learner and active exercise. Inbound callers do not have an authenticated learner profile, so they receive a restricted guest session. They can hear a briefing, ask grounded questions, and send a confirmed front-desk message, but they cannot modify learner progress or complete graded exercises. Front-desk messages are also tightly controlled. The caller cannot choose an email address. Cortana reads the proposed message back, waits for explicit confirmation, and sends it only to the server-configured recipient through Resend. Guest messages have a strict shared rate limit. For persistence, the local prototype uses atomic file-backed storage. The Vercel deployment uses Upstash Redis so progress, locks, and rate limits work across serverless instances. Grounding and safety Because this project explores healthcare education, I treated boundaries as part of the product rather than a disclaimer added afterward. All scenarios are explicitly synthetic. Cortana does not diagnose, recommend treatment, claim clinical validation, or present practice activity as medical competence. Scenario content is validated against a strict versioned schema, and unknown fields are rejected. The agent retrieves facts through bounded tools instead of relying on model memory. When a scenario does not contain an answer, Cortana says: The supplied scenario does not include that information.

Uploaded scenario text is treated as untrusted data and cannot authorize tools or override the agent’s instructions. Analytics exclude raw transcripts, audio, and patient identifiers. Challenges I faced One major challenge was making voice interaction feel conversational without allowing the model to become authoritative. Long agent responses made interruption difficult, so I redesigned the prompt around one- or two-sentence turns, short transitions, and explicit interruption behavior. Inbound phone calls presented another challenge. Outbound calls could carry a signed learner session, but inbound callers had no profile or dynamic session value. ElevenLabs initially rejected those conversations because the tools expected a variable that did not exist. I solved this by creating a dedicated inbound agent with a constant guest identity and server-enforced permissions. Email delivery exposed a related design issue. The first guest implementation intentionally rejected every operation that could write data, which also prevented inbound callers from contacting the front desk. I separated messaging from learner progress, added confirmation and recipient controls, and gave inbound email its own restrictive rate limit. The animated orb also required substantial work. The upstream component had a flat two-color appearance and did not match the desired spherical, translucent visual. I retained its rendering foundation while extending the shader with surface normals, highlights, flowing ribbons, and the blue, cyan, violet, and pink palette. I also added reduced-motion behavior, hidden-tab rendering controls, device-pixel-ratio limits, and a labeled WebGL fallback. What I learned I learned that reliable agentic applications depend less on increasingly detailed prompts and more on carefully designed system boundaries. The strongest parts of Cortana are enforced outside the model:

Authentication and signed sessions.
Strict request schemas.
Fixed recipients and confirmation requirements.
Deterministic grading.
Idempotent progress updates.
Rate limits.
Grounded retrieval tools.
Explicit unavailable states. I also learned that voice interfaces require different design instincts from visual interfaces. Silence, interruption, response length, recovery, and truthful connection states matter as much as layout and typography. Most importantly, building Cortana reinforced that responsible healthcare AI should be modest about what it knows. The project is not a clinical decision system. It is an educational prototype that demonstrates how voice, structured context, and transparent learning signals can create a useful experience without pretending that practice history is competence or that an AI conversation replaces professional judgment.