Valence

Inspiration

Emotion AI mostly reads faces, and faces fail in predictable ways: a forced, polite smile reads as happy while the frustration underneath is missed, and a focused frown reads as anger. Most face-first pipelines also store frames and give labels with no reasons. I wanted the opposite: read the context around a person over time, keep none of the raw media, and explain every call.

What it does

Running on one laptop, Valence labels a person’s likely state (Focused, Frustrated / Masked, Exhausted or Collaborative / Happy) every 5 seconds from 15 minutes of context: room noise and light relative to the room’s own baseline, time since a break, posture, clenched fists, laughter, and the face as just one input. A face-only model runs alongside, and when the face says happy but the context clearly disagrees, Valence flags a MASK OVERRIDE. Every call comes with a plain-language explanation whose numbers are checked, and raw video and audio are zero-filled in memory right after feature extraction and never stored.

How we built it

YOLO pose with ByteTrack, MediaPipe face blendshapes and gestures, YAMNet and a Vosk 8-phrase grammar handle perception on the laptop. Features feed a 15-minute rolling window and a LightGBM context model whose TreeSHAP contributions, grouped into four layers, drive the explanations, written by a local Qwen3-1.7B model on MLX with a LoRA and a validator. A FastAPI + WebSocket backend serves a React console, and a purge gate, static scan and SHA-256 hash-chained audit log handle governance.

Challenges we ran into

The mock dataset we started from could not test our idea (context alone scored 100%, with no time order), so we built our own simulator. A small LLM invented numbers in 17.1% of raw attempts until we trained a LoRA and put a validator in front of it. Everything (YOLO, MediaPipe, YAMNet, Vosk and the LLM) shares one laptop GPU, and honest purging in Python meant zero-filling the memory itself, since del only removes a name.

Accomplishments that we're proud of

On 45 held-out simulated sessions, the context model catches masked frustration 68.0% of the time vs 11.2% for the face alone, and macro-F1 is 0.797 vs 0.422 (simulated data, which partly encodes our own assumptions). Our live demo on one of us shows the whole pipeline working end to end on a laptop, experimentally, with a governance story we can demonstrate: 23 governance tests, a zero-violation persistence scan, and an audit log that detects a one-character edit.

What we learned

Context over time carries information a single face snapshot cannot. Mock data that makes your idea look perfect is a warning sign, and a few minutes of real capture from one person found bugs the simulator never would have. Governance claims need precise wording: “zero-filled” not “erased”, “tamper-evident” not “tamper-proof”.

What's next for Valence

Valence is an experimental research prototype that needs further development and expansion: consented, labeled real-world data from more people, validation across demographics, confirming the live results more broadly, and closing the remaining governance gaps before any real-world claim.

Built With

• bytetrack
• fastapi
• lightgbm
• lora
• lovable
• mediapipe
• mlx
• numpy
• opencv
• pandas
• pydantic
• python
• pytorch
• qwen
• react
• recharts
• scikit-learn
• tailwindcss
• typescript
• ultralytics
• vite
• vosk
• websockets
• yamnet
• yolo