Tink - Tech Interprets Needs & Knowledge

Main loading
ASL Proof of Concept
Gestures and Sounds Detector

Inspiration

One of our teammates' closest friends, Surya Kasibhatla, is one of the most remarkable people I know. He's a computer science student at the University of Texas at Dallas, an Eugene McDermott Scholar, a Bollywood actor, and a software entrepreneur. He also lives with cerebral palsy, which presents unique challenges with mobility and speech. Despite these challenges, Surya continues to accomplish extraordinary things through his intelligence, determination, and ambition. Knowing him has made me realize that someone's ability to communicate or move should never determine the limits of their potential.

This inspired a question that became the foundation of Tink: Why should the way someone moves or communicates determine how easily the world can understand them? Communication is something most of us take for granted, yet millions of people face barriers to expressing even their simplest thoughts. Traditional assistive tools often rely on typing, speech, or predefined commands, overlooking the countless ways humans naturally express themselves through gestures, movements, and visual cues. We wanted to challenge that limitation by creating technology that adapts to the individual rather than forcing the individual to adapt to technology.

That's why we created Tink, a multimodal, AI-powered assistive communication platform that combines real-time computer vision, gesture recognition, and probabilistic modeling to bridge the gap between human intent and understanding. Inspired by Surya's story, we wanted to build something that recognizes the intelligence and individuality behind every form of expression. Tink began with one extraordinary friend in mind, but our mission extends to anyone who has ever struggled to make themselves understood. We believe everyone deserves not only a voice but also the opportunity to reach their fullest potential.

What it does

Tink interprets a user's intent by combining multiple communication signals in real time. The platform processes visual input from the camera to recognize gestures and body movements while simultaneously analyzing audio for speech and nonverbal vocalizations. Instead of treating these inputs as separate predictions, Tink combines them with conversational context to determine the most probable interpretation of what the user is trying to communicate.

At the center of this process is a probabilistic inference system. Visual observations provide an initial distribution of possible meanings, and additional evidence from gestures, audio, and context updates the probability of each interpretation. This allows Tink to handle ambiguity rather than depending on a single classification. If one signal is unclear or unavailable, the system can still form a prediction from the remaining inputs; as more information becomes available, that prediction can become more confident.

Tink also includes an ASL mode that uses real-time computer vision to recognize signed letters from a camera feed and convert them into a constructed message. Alongside this, users can access predefined phrases for immediate communication when a faster response is needed.

By combining computer vision, audio analysis, contextual reasoning, and probabilistic inference, Tink turns different forms of human expression into a unified interpretation of intent that can be communicated to others.

How we built it

At the start of our project, we were exploring how to get inputs into the tool. Knowing that disability can take various forms, we prioritized making the website as accessible as possible, so we wanted as little work as possible to be required of the user.

With that in mind, we decided to create a system that could interpret multiple forms of communication rather than requiring users to interact through one specific method. We focused on two primary pathways: ASL translation and general non-verbal communication through video and audio. This would allow users who communicate through both sign language and nonverbal language to use gestures, body movements, and nonverbal vocalizations.

For the audio component, we discovered voc2veec, a research model specifically designed for nonverbal human vocalizations. Unlike traditional speech-recognition systems that exclusively focus on converting spoken words into text, voc2veec provides representations of sounds such as laughter, crying, and other vocal expressions. We chose this approach because we wanted our application to accommodate individuals who may not communicate through conventional speech.

After identifying these research models, our next decision was to determine how their outputs could work together. We designed an architecture centered on Bayesian inference, which allows us to maintain a probability distribution over possible interpretations and update that distribution as new evidence becomes available. Visual observations serve as our primary source of information, while additional gestures, nonverbal audio, and conversational context can help refine the predictions.

Challenges we ran into

One of the primary challenges we ran into was implementing an ASL feature that enables real-time translation. Due to the limitations of camera-reading technology and the monocular cameras on most devices, depth detection is increasingly difficult, and the existing APIs for live translation are either ineffective or expensive. Because of that, we had to resort to letter-by-letter translation, but additional future features could include getting the live ASL translation working.

Furthermore, in the main multimodal transcription, our original idea was to use a classical machine learning model to obtain a classification from a dataset. However, the classification often yielded singular outputs, meaning it was not generating a unique response to the specific situation being described. As a result, we ended up using Google Gemini LLM to create a unique output that matches the conversation's context rather than a static response, which we believed would be much more in the spirit of the project.

Accomplishments that we're proud of

One of the proudest accomplishments of this program is a functioning multimodal system that takes input from different sources and produces structured output that uses all of that information. We were initially pretty confused about how to make all of these different systems work together; however, researching Bayesian inference taught us that we can create mutually reinforcing, self-correcting systems, as the model would still work even without all the information. However, providing more information would improve its accuracy, making the model more accessible to people with speech disabilities.

What we learned

One thing we learned in the creation of this project is that many of the assumptions we take for granted in our communication with each other. Communication is a nuanced aspect of life, with verbal and nonverbal cues that shape how people perceive one another. Not only were we coding these aspects, but we were having conversations with each other about how these differences can really impact who we are and what we take for granted. Coming out of this Hackathon, our entire team has a newfound appreciation for the simple act of communication.

What's next for Tink

Our long-term vision is to evolve Tink from an assistive communication platform into an intelligent interface that connects human intent with digital action. As AI agents become capable of completing increasingly complex tasks on computers, we want to make those capabilities accessible to people regardless of how they physically interact with technology. Someone with limited hand mobility could use gestures, facial movements, speech, or other personalized inputs to communicate what they want to accomplish.

Tink could understand that intent and work with an AI agent to navigate a computer, write an email, create a document, develop software, or complete everyday digital tasks. We are also considering integrating Tink with digital hardware. For example, one of the earlier issues we had with ASL was the lack of binocular vision. By adding smart glasses that can actually see the depth at which ASL is performed, we would be able to better scan for live ASL, which could also be transmitted as audio.

As Tink grows, we want it to learn each user's unique communication patterns and become increasingly personalized to them. By combining multimodal understanding with AI agents, Tink could eventually turn a person's intent directly into action while keeping them in control throughout the process.