Misi - An emotion-aware posture monitoring safety assistant

While millions of us spend hours hunched over screens leading to chronic pain and fatigue. Using the Gemini Live API misi provides multimodal biofeedback. It’s a companion for the modern day workspace

Comment

Misi: Your Emotion-Aware Postural Assistant / Guardian

AI is not just about increase productivity and profits for corporations it is power that is made available to everyone and can be used for good.

Inspiration

In an era where digital screens dominate our lives, the "tech neck" epidemic has become a silent crisis. Millions of people spend hours hunched over laptops, leading to chronic pain and long-term spinal issues. The inspiration for Misi came from a simple observation: we have fitness trackers for our steps and heart rate, but we lack a proactive guardian for our most fundamental physical state—our posture.

We wanted to build more than just a timer; we wanted an agentic companion. An AI that doesn't just watch, but understands the nuances of human movement and emotional fatigue.

What it does

Misi is a real-time, AI-powered safety and postural assistant. Using the camera and microphone, she monitors your ergonomic alignment and emotional state to provide proactive biofeedback.

Key Features:

  • Live Postural Analysis: Detects forward head posture, rounded shoulders, and spinal compression.
  • Emotion-Aware Feedback: Analyzes facial expressions and vocal tone to adjust her persona—being supportive when you're stressed and firm when you're slouching.
  • Dynamic Health Scoring: Quantifies your postural health using a mathematical index $H$: $$S(t) = \max\left(0, 100 - \sum_{i=1}^{n} v_i \cdot w_i\right)$$ where $v_i$ is alert severity and $w_i$ is the weight of the postural red flag.
  • Workspace Integration: Connects to Google Calendar to schedule "Recovery Breaks" when fatigue is detected.
  • Automated Reporting: Synthesizes session data into professional PDF reports using custom layout logic: $$Y_{pos} = Y_{start} + \sum_{j=1}^{m} (L_j \cdot H_{line} + \delta)$$

How we built it

Misi is a high-performance, full-stack application designed with a "Technical Dashboard" aesthetic.

The Tech Stack:

  • Intelligence: Powered by the Gemini 2.5 Flash Native Audio model for simultaneous video and audio processing.
  • Real-time Engine: Built on a Bidirectional WebSocket using the Gemini Live API for sub-second latency.
  • Frontend: Developed with React 19 and Tailwind CSS, following a "Mission Control" design recipe.
  • Backend: Express.js handles OAuth2 flows for Google Workspace and proxies calendar requests.
  • Cloud Infrastructure: Firebase Firestore for real-time session logs and Firebase Storage host the generated PDF reports.

Challenges we ran into

Building a real-time multimodal assistant presented several non-trivial hurdles:

  • The Reconnection Loop: Initially, Misi would re-introduce herself every time the connection flickered. We solved this by implementing Session Resumption handles and state-aware greeting logic.
  • Audio Interruption Sensitivity: The Gemini Live API's default activity detection was too sensitive, causing Misi to interrupt herself. We had to fine-tune the automaticActivityDetection parameters to balance responsiveness with stability.
  • Mathematical Recovery Logic: Designing a fair recovery function $R(t)$ that rewards sustained good posture without allowing "gaming" of the system: $$S_{t+1} = \min(100, S_t + \int_{t}^{t+1} \rho(p) \, dt)$$

Accomplishments that we're proud of

  • Zero-Latency Biofeedback: Achieving a sub-200ms response time for postural corrections.
  • Consent-First Architecture: Hardcoding the system to remain "blind" until the user provides explicit verbal consent.
  • Seamless Workspace Integration: Successfully automating calendar scheduling based on AI-detected physical fatigue.

What we learned

The biggest takeaway was the power of Multimodal Context. By combining visual data (posture) with emotional data (facial expressions) and environmental data (calendar), the AI moves from being a "tool" to being a "partner." We also learned that in health-tech, privacy and user control are the most important features you can build.

What's next for Misi - An emotion-aware safety assistant

  • Multi-User Collaboration: Allowing teams to monitor collective "ergonomic health" in shared workspaces.
  • Advanced Exercise Library: Integrating real-time guided stretching based on the specific muscle groups Misi detects are strained.
  • Hardware Integration: Connecting with smart desks and ergonomic chairs to physically adjust the user's environment in real-time. -** Google Integration** : Integrating google workspace so the assistant can do some tasks work on your behalf
Share this project:

Updates