tl;dr
i'm an AI intern at a firearms and defense manufacturing company. i watched a deaf coworker fight to be independent at work. she never lacked ability. phone calls, meetings, and training were all built around sound.
so in 36 hours we built one platform for her whole day, not one app for one problem.
call
sign into your camera, and a real phone rings somewhere. the other person hears a natural voice. they install nothing, and there's no interpreter in the middle. their replies come back as captions. it works like autocomplete for your voice: sign 2–3 words, and it suggests the full sentence based on what the caller just asked. as far as we know, nothing else does all of that at once.
the AI can't put words in your mouth. every number, time, price, and name has to come from you. a signed no is always a no.
meet
meetings caption voices. we caption hands. your signs show up right inside your camera tile and overlay in meet or zoom. it's just a chrome extension.
learn
this is the crazy one. film a job once on your phone, and in under 40 seconds it becomes a step-by-step lesson you follow with your eyes: every step timestamped, looped, located in the frame, and open to your questions. industrial process analysis from the factory floor, turned into accessible training. i'm going to go back and show this to her.
play
your gestures make music you can see. jam with friends and visualize nodes, chords and pitch.
Inspiration
- a lot of the work at my internship gets taught by someone standing next to you, explaining as they go.
- my coworker is good at her job. what slowed her down was everything around the job.
- vendor calls meant finding someone to make them for her. in meetings she'd catch part of it and get the rest later. training videos said the important parts instead of showing them.
- each app comes from one of those situations. and it's not just her: about 1 in 20 deaf or hard-of-hearing people can get things done by phone on their own, and 70% ask family to call for them.
What it does
- call: sign, approve the sentence, and the other person hears it as a natural voice on a normal phone call. replies come back as captions.
- meet: your signed or typed sentence shows up as a caption in your own camera tile.
- learn: a phone video of a hands-on task becomes a step-by-step lesson in under 40 seconds.
- play: hand movements make music, shown as shapes and color.
How we built it
the call pipeline
- mediapipe: tracks hand and body landmarks in the browser. only x/y points leave the device, never video.
- signbart → onnx: reads a sequence of keypoint frames, not a single hand shape. if it isn't sure, it shows the top two guesses and you pick.
- gemini: takes your signs plus what the caller just said and writes one natural sentence for you to approve.( honestly .its predictive autocomplete for speech)
- guardrails: every number, time, price, and name has to come from you, your notes, or the caller. money and bookings need an extra tap.( can also confirm through keypad)
- grok voice: outputs μ-law at 8 kHz, the phone line's own format, so there's no conversion. it stops when interrupted. approved sentences go out with force_message, word for word.
- twilio + fastapi on vultr: twilio carries the call, and one service coordinates everything. elevenlabs is the fallback voice.. though we never needed it
learn
- gemini processes video and audio together and returns structured steps with timestamps and positions in the frame. we build the lesson from that. I compared several frontier and open sourced models and none came close to its capability
also: react, typescript, python, opencv, ffmpeg, docker.
Why Gemini and Grok
grok
- i assumed we'd use elevenlabs. it sounds good, and i had it hooked up through twilio first.
- the problem was the phone line. calls run on μ-law at 8 kHz, so most voices get generated in high quality and then converted down. on a live call, you feel that extra step.
- grok outputs phone audio directly. no conversion.
- when the caller interrupted, the old setup kept talking. grok stops, like a person would. that alone made calls feel a lot less robotic.
- if someone approves "thursday at 3:30, $40," it can't come out as "around 3:30-ish." force_message makes grok say it exactly.
- elevenlabs is now the fallback. i haven't needed it.
gemini
- for learn i tried the big hosted models and the open-weight vision models i could run or find demos for.
- most treat video as frames plus a transcript. fine for describing a scene, not for a procedure, where what matters is how things change from one moment to the next.
- gemini kept context across frames. it knew the part in hand was the same one from a few seconds ago, now flipped. it took in scene, speech, and hand actions together.
- and it returned structured steps: start and end times, position in the frame, tool used. that's what we could actually build on. -btw all of this setup costs just 40cents / 3 mins... not a bad deal at all.
the shop floor video
- i had an old video on my phone from the shop floor. no script, bad audio, someone just doing the job.
- i ran it through out of curiosity, expecting a rough summary.
- it came back as numbered steps, each tied to a time range and a spot at the workstation. -flat rgb mobile video was turned that into an adjustable 2d map of the station with the exact spacial reconstruction, with step numbers placed where the work happened, each linked to its clip.
- people spend days making training material like that by hand. that's when i got genuinely excited about learn.
Challenges we ran into
- signs run together. natural signing has no clean gaps, just like speech. continuous recognition is still an open research problem, so we recognize one sign per pause for now.
- frame rate. the webcam ran at a few fps early on and chopped signs into pieces. we moved to lighter tracking models and reworked how signs get split.
- latency. recognition- gemini - voice - phone network, and every step adds delay. we cut wherever we could and added short fillers so the caller isn't left in silence.
- syncing learn to the video. getting steps out of gemini was quick. lining every annotation up with the right frame was not.
Accomplishments that we're proud of
- four tools from four real situations, all in one app.
- the guardrails are in the code. the AI can't confirm a time, price, or booking you didn't give it.
- learn takes the way manufacturing breaks down and documents work steps and uses it to train people who can't rely on audio.
- getting recognition, gemini, grok, and twilio working together on a live call.
What we learned
- the right model for each job mattered more than the newest model.
- for video, handling frames, audio, and timing together beats stitching separate outputs afterwards.
- being able to interrupt the voice made calls feel more natural than making it sound better.
- if AI speaks for someone, they have to approve what it says. accuracy alone isn't enough.
What's next for FingerTips
- test it with deaf users, starting with the person who inspired it.
- continuous sign recognition, so people can sign naturally.
- more signs, including workplace and technical vocabulary.
- real-time learn: point your phone at your own workstation and get step-by-step guidance while you work.
- play, together: live jam sessions with friends.
- a pilot in a real manufacturing training program.
Built With
- chrome
- docker
- google-gemini-api
- mediapipe
- postgresql
- twilio
- vultr
- websockets