Twin | Devpost

Inspiration

Cluely offers live AI meeting insights. While Cluely has disrupted the interview market, for daily meetings it can only act as an assistant and does little to save time.

What it does

Whereas Cluely is designed to assist you during important meetings, our bot is designed to automate repetitive meetings like daily standups, giving software engineers more time to write code.

Our bot monitors your team chat, detects when a standup is starting and automatically joins the voice call on your behalf. During these meetings it listens to the discussion and responds in your cloned voice as if you were there. It transcribes the meeting, understands when you're being addressed, generates concise standup updates and speaks them naturally using your voice. All with minimal latency (roughly 2 seconds) so the conversation feels authentic.

How we built it

As we are broke we tried to design this bot to be almost free which informed the technology we used:

The backend is built with Node.js, TypeScript and Express.
A locally hosted instance of Whisper is ran one of our PCs and has an API for transcribing audio files
The transcript is passed through a light weight wake-word detector that parses only relevant sentence to gemini, drastically reducing calls to the API
Our gemini key gives access to a free 3.0 flash model of gemini that is prompted to produce concise responses and is given context of the individual it is emulating
Gemini's text is passed into eleven labs api and creates a voice cloned text to speech audio file that is outputted by the bot's virtual microphone

Challenges we ran into

Long Latency

To respond naturally in real time the bot must respond in at most 2-3 seconds. To achieve this we had to make optimisations throughout our architecture
whisper by default runs on CPU so we had to configure it and use pytorch to allow it to run on our gpu, shaving off nearly 2 seconds in response time

Installing and Configuring Linux

The PC hosting our whisper was running windows so we decided to dual boot debian so we could use shell
None of us had installed linux before so that was an interesting experience

Accomplishments that we're proud of

Completing our first every hackathon.

What we learned

In our class 1531 we learned node.js, typescript and express and were able to make basic webroutes. In this project we were able to get experience actually deploying this code to the web and creating APIs.

What's next for Sit Down Meeting Bot

Sumarise meeting notes and responses
Automatically pull recent git logs and pass it into the bots context to accurately talk about recent contributions