Telly

Health metrics.
Object detection and tracking. And location finding.

Inspiration

Most of us have someone we love who could use a little more help as they get older. Sometimes that means remembering a routine or finding medication. Sometimes it means navigating a home that used to feel completely familiar.

We wanted to build something that could help in those moments without requiring someone to learn a complicated interface. We also kept coming back to a personal question: would this work for our grandparents, especially if they do not speak English?

That question became Telly, a visual voice assistant designed to help older adults feel more confident and independent in their daily lives.

What it does

Telly combines a camera-based view of the world with conversational assistance. A person can ask for help in their own language, show Telly their surroundings, and receive spoken guidance through an everyday task.

Our demonstration follows someone looking for medication at home. They speak to Telly in Tamil while moving through the apartment, and the assistant responds with short directions that help them continue the search.

We designed the experience around natural conversation. The person should be able to ask a follow-up question, point the camera at something, and continue without navigating menus or typing instructions.

The broader product brings this assistance together with reminders and family-facing information, so support can fit into the person's existing routine.

How we built it

We organized Telly around a visual interface, a conversational voice experience, and a web application for managing the information that supports that experience.

The camera provides visual context, while the voice interaction lets the person explain what they need and hear the next step. Our design separates the person's request, the assistant's interpretation, and its response so that the interaction remains understandable.

We focused our demonstration on a complete household task rather than a collection of disconnected screens. This helped us think through how the experience should behave as someone moves between rooms, asks another question, or brings an object closer to the camera.

Challenges we ran into

One of the hardest design problems was deciding how much the assistant should say. Long explanations can become distracting when someone is already trying to complete a task. We worked toward short, relevant instructions that leave room for the person to respond.

Multilingual interaction also requires more than translating English sentences. The language needs to sound natural, respectful, and familiar to the person using it.

Another important challenge was defining the assistant's limits. Helping someone locate a medication bottle is different from confirming that they should take it. We want Telly to communicate uncertainty clearly and keep medical decisions with the person and their care team.

Accomplishments we're proud of

We are proud of shaping the product around a specific human experience: someone asking for help at home in the language they are most comfortable speaking.

The Tamil walkthrough captures what we want Telly to become. The technology should feel approachable enough that a person can focus on their day rather than on how to operate the product.

What we learned

We learned that accessibility depends on the entire interaction. The length of a response, the language it uses, and the moment it arrives can matter as much as the information it contains.

We also learned the value of following one task from beginning to end. It exposes the practical details that are easy to miss when designing individual features in isolation.

What's next for Telly

Our next steps are to test the experience with older adults and caregivers, improve visual reliability, and make multilingual conversations more responsive and natural.

We also want to develop personalized reminders, consent-based family updates, and clearer summaries of health and lab information. Wearable and smart-glasses integrations could make the experience more convenient as those connections are developed and validated.

Our goal is to help people handle more of their everyday lives with confidence while staying connected to the people who care about them.