Agency

The home page
Ordering food through Agency
Setting up a task with Agency

Inspiration

My grandma has ALS. She’s still completely herself, but her body can no longer keep up. Even ordering dinner or messaging family means relying on someone else. Her eyes are the one thing she can still reliably control. We asked: what if her eyes were enough? AI agents can already navigate websites and complete tasks, but they assume users can use a mouse, keyboard, or voice. We built AGENCY so my grandma could control an AI agent using only her eyes and an ordinary laptop webcam.

What it does

AGENCY turns a standard laptop into an eye-controlled AI assistant without expensive specialized hardware. A five-second calibration replaces the usual 9–16 calibration points: users follow a moving light, and the webcam estimates where they look. Holding their gaze selects buttons, with selections persisting through blinks and quick glances away. Three systems make AGENCY more than an eye-controlled interface. Synapse-Mesh coordinates specialized AI agents, selecting the right experts for each task—whether medical, legal, technical, or everyday—and bringing their reasoning together to produce a more useful plan. Instead of relying on one general-purpose model for everything, AGENCY can distribute work across specialists while keeping the user in control of the final decision. Our Cisco network sentinel makes that intelligence adaptive. It monitors connection quality and adjusts how AGENCY operates: on a strong connection, it can involve more experts and deeper reasoning; when bandwidth is limited, it reduces the workload and prioritizes faster responses. If the connection fails, AGENCY can fall back to built-in plans rather than treating the network as a single point of failure. Finally, Lifeline is designed for the moments when a user needs help most. A user can trigger it with three long blinks, starting a countdown before AGENCY sends a message to a trusted contact and, when configured, initiates a call. Crucially, Lifeline is designed to work independently of the AI agent, so a failure in the main system doesn't automatically disable the user's way to ask for help. Together, these systems address three different problems: coordinating intelligence, adapting to unreliable connectivity, and preserving access to help when the AI fails. Camera video stays on the laptop; only task text goes to the cloud.

How we built it

We combined MediaPipe-based eye tracking, our own gaze-calibration system, RealEye's eye-image model, Gemini-powered planning, and browser automation. We also built independent safety controls and offline fallback plans so essential functions don't depend entirely on the AI or network.

Challenges we ran into

The hardest part was making eye tracking accurate with an ordinary webcam. Noisy measurements caused cursor jitter, eye movements lagged behind calibration targets, and head movements affected accuracy. We repeatedly rebuilt calibration, replacing fixed points with a moving light and accounting for individual response delays. We also integrated RealEye's model when it improved accuracy for a particular user.

Accomplishments that we're proud of

We built an end-to-end prototype connecting eye movements to real computer tasks without specialized hardware. Users can select goals, approve consequential actions, and verify results while retaining independent Stop and Help controls. We also wrote a research paper during the hackathon, AGENCY OS: Gaze-Operated Agentic Computing with Fail-Safe Human Control. It formalizes two ideas: stop amplification, a failure mode involving safety controls triggered from multiple places, and approval binding, which ties user consent to the exact action approved. We went beyond building a demo to document the principles behind safer, human-controlled AI.

What we learned

The hardest question in agentic AI isn't just what an agent can do, but who gets the final say. AGENCY keeps consequential decisions with the user and verifies outcomes independently of the AI's claims. We also learned that accessibility must shape every interaction: when someone cannot easily undo a mistake, accuracy, confirmation, and honest uncertainty become essential.

What's next for AGENCY

First, we want to put AGENCY in front of my grandma and evaluate whether it genuinely makes everyday tasks easier. Then we plan to work with ALS clinics, rehabilitation hospitals, and occupational therapists to test it with people who have limited movement. Longer term, we want to help people contact nurses, communicate with family, and control their digital environments through gaze alone. Because AGENCY runs on existing hardware, it could make eye-controlled computing accessible without expensive dedicated devices.