Inspiration

In a special-education classroom, one teacher often supports several students who each need something different. A student who is stuck rarely says so. They may not be able to, and by the time anyone notices, the moment has passed. We wanted to give teachers a second set of eyes that points to who might need help right now, so the teacher can spend their attention on the child rather than on scanning the room.

We also approached this like forward deployed engineers (FDEs). We didn't start with a model and look for a use. We started by talking with the founder of BridgeAI, an SEN education business, regarding a real, education problem we could solve with an actual workflow. That is why our product is build with real SEN classroom situations in mind.

What it does

Cues watches simple classroom tasks through a local camera: raising a hand, a thumbs up, or a gesture the teacher recorded themselves. A rules engine decides when an event matters. A deep AI agent then suggests a support from an approved intervention menu. The teacher assesses every change before anything happens to a student's task or instructions.

  • Gesture detection for built-in gestures and teacher-recorded ones
  • Facial-expression context as supporting evidence only. It never triggers an alert on its own
  • Human in the loop: the agent suggests and the teacher decides
  • Privacy by design: camera images and video stay on the laptop. Only structured events leave it

How we built it

  • Vision service (Python, FastAPI): a local pipeline using MediaPipe for hand and gesture tracking, and HSEmotion for facial-expression probabilities. Frames are never stored or sent anywhere.
  • Rules engine: turns raw events into meaningful ones, so the teacher isn't flooded with noise.
  • Deep agent: receives only structured events, never frames. It picks from an approved intervention menu and never makes claims about a child's internal state.
  • Dashboard (Next.js + Supabase): shows each student's status and a review queue where the teacher approves or dismisses suggestions.
  • Messaging (Photon): notifications carry the minimum necessary information. No images and no expression labels.
  • Feature flags let us switch off any feature that misbehaved during the demo.

The business case

Special educational needs (SEN) support is stretched thin. Schools need more individual attention than staffing allows, and most classroom tech is built for the average student. Cues is aimed at schools and districts that want to extend what a teacher can notice without replacing their judgment. The privacy-first, teacher-approved design matters commercially as well as ethically, because schools won't adopt a tool that sends student video to the cloud or acts without a teacher.

Challenges we ran into

  • Staying honest about expression models. A model gives a probability about a face, not a feeling. We deliberately treat it as supporting context and never as a diagnosis or a trigger.
  • Keeping the privacy promise while still using AI. We had to design the agent to work from structured events alone.
  • Real-time vision on a laptop. Getting stable hand and gesture tracking without a GPU meant a lot of tuning.
  • Noise. Early versions flagged too much. Tuning thresholds, such as how long an open palm must be held, took real iteration.
  • Three people, one codebase. Vision, web, and agent were built in parallel against a shared event contract.

What we learned

  • Human-in-the-loop design is a feature, not a limitation. It is what makes the tool trustworthy.
  • Shipping behind feature flags let us merge unfinished work safely and turn things off live.
  • Building for a specific user, the way an FDE would, produces better product decisions than building around a model.

Built With

+ 20 more
Share this project:

Updates

Submission history