When I was small I got lost in Hong Kong. I didn't speak the language. On later trips to China and other countries, I didn't know the language either, and I couldn't use mobile data. Each time I wished for the same thing: a translator that needs no connection, so I could understand people and talk with them. Knowless is my attempt at it: once its languages are downloaded, the core of it works without a connection.
My first version was a radar that listened all the time. I imagined it could tell where a speaker was from the sound alone. Not on my phone: my iPhone 15 Pro Max gives me no spatial audio from a recording, so there's no direction to read. Then I tried eyes instead of ears, using both cameras at once to find the face that was speaking. The phone ran very hot, the battery drained fast and it kept freezing, so I dropped that too.
Another idea was noise reduction for loud places. It sounded reasonable on paper. Then I listened to a noisy recording with my own ears and couldn't make out what I was saying. No amount of denoising would fix that. That's what I mean by a human in the loop: I build with AI, and I have to keep checking that what I'm asking for makes sense in the real situation.
What was left is one screen and one button. Press, talk, translate. The first prototype also kept a conversation history. I cut it to ship in time for Shipaton, and still nearly missed the deadline.
A very small feature turned out to be extremely complex: how the words appear. Speech becomes a guess, shown at once so it feels live. Then the guess is corrected. After a long sentence it's corrected again, using the whole sentence as context. Each of those states has to look right, and so does the waiting in between. I knew from the start I wanted it to feel intuitive. I didn't expect how long that would take: I went back and forth between the app and a design mock for more than two days.
Next I want live voice translation between two people talking to each other. That isn't built yet.
Log in or sign up for Devpost to join the conversation.