Dashboard
CAMMY Session Limit
SORA Architecture

Inspiration

Sora started with something we noticed during our own work and study sessions: it is easy to spend hours at a computer without paying attention to how we are sitting or when we last took a break.

We wanted a desktop companion that could notice relevant changes and give a small reminder. It should fit into the workday and leave the user in control.

What it does

Sora is a Windows desktop prototype that combines webcam observations with adjustable reminders. CAMY, our animated character, delivers short messages about posture, screen distance, water breaks, movement, eye breaks, and session duration.

Users begin by calibrating their own reference position. Sora then looks for sustained changes, such as a lower head position or moving closer to the camera. Experimental eye and mouth movement cues can also trigger a suggestion to take a break.

The posture workflow includes a follow-up. If the observed condition continues after a reminder, Sora can offer a short break. Users can dismiss, acknowledge, or snooze reminders and open Agent activity to see the decisions behind them.

Sora also attempts to recognize drinking by following a cup or bottle toward the mouth and back away. A recorded event resets the water reminder. This feature is still experimental, so users can confirm a drink manually or undo an incorrect record. It does not measure hydration or the amount consumed.

How we built it

We built Sora in Python. OpenCV 5 handles camera processing, person and pose inference, object detection, and tracking. MediaPipe face landmarks provide additional visual cues.

The interface uses HTML, CSS, and JavaScript, based on our team's Figma design. Tkinter and Pillow display the transparent desktop notifications using our original CAMY animations.

A session controller combines visual observations, timers, and user responses. A rule-based agent decides when to show a reminder, follow up, or close a goal after observed recovery. It does not use an LLM.

The current demonstration runs the vision pipeline and reminder policy locally. Camera frames are not saved or uploaded. Session summaries and decision traces are stored on the computer so users can review what happened.

We have prepared an AWS Lambda component for the reminder policy, but account activation has prevented deployment. It is designed to receive a small state summary, return an action, and let the desktop validate that action before applying it. A local fallback is included in the integration code. This cloud path has not been verified on AWS and is not active in the demonstration. The architecture diagram marks it as planned.

Challenges we ran into

Drinking was harder to recognize than we expected. A bottle can cover the mouth, look different when tilted, or be confused with another object on the desk. We added tracking and checks across the approach, contact, and withdrawal stages instead of treating one detection as a drink.

Those changes still need more real-camera testing. Manual confirmation and undo remain part of the interface because the detector can miss or incorrectly count an event.

We also had to distinguish between a dismissed reminder and an improved condition. Closing a notification should not count as recovery, and losing the tracking signal should not be treated as a return to the reference position.

Account activation has also blocked our planned AWS deployment. We are documenting that limitation separately from the local features shown in the video.

Accomplishments that we're proud of

We brought the camera pipeline, interface, animated reminders, and session history together in a working desktop prototype.

Our recording shows a posture reminder changing into a break suggestion when the condition continues. That follow-up is central to Sora: what it observes affects what it does next.

We also added a camera preview and an activity trace to make the system easier to inspect. These help us understand missed detections and reminder decisions while testing.

What we learned

Recognizing an object is only part of recognizing an action. Seeing a bottle does not mean someone drank from it, and losing sight of a person does not mean their posture improved.

We learned to treat uncertain observations carefully and keep correction controls available. We also found that the timing of a reminder matters as much as the message itself.

What's next for Sora AI Work Companion

Our immediate priorities are completing and verifying the AWS integration and testing the latest drinking changes with real camera input.

We also want to test Sora with more people, webcams, containers, and everyday desk setups. We will measure missed events and unnecessary reminders, and work on keeping the application responsive on less powerful computers.

Longer sessions will help us understand which reminders people find useful and how their timing should adapt to different routines.