Téléphonique (Hotline Mistral)

Cover Image

Inspiration

Mistral and other state of the art LLMs represent, in many ways, the sum of human knowledge to date. For many, they are increasingly becoming an indispensable tool to elevate the skills of any layperson to those of an expert; for some, they increasingly act even as a friend or companion. But for many, accessibility still remains an issue: many regions of the world, and even parts of the US, lack reliable internet access; and many individuals, particularly the elderly, do not have a smart phone or comfort using apps. Rather than limiting access to some of mankind’s greatest achievements, we should be promoting it—to everyone and everywhere.

What it does

Hotline Mistral is a simple idea with a great deal of reach: a phone number anyone can call, or even text, to speak directly with everyone’s favorite LLM. Ask questions about any topic and get immediate answers or feel free to have a simple conversation, anywhere with cell service or a landline. Hotline Mistral knows you individually by your phone number, remembers your ongoing interactions, and will email you notes of your discussions any time you ask. With integrations to Amazon Alexa and OpenInterpreter, Hotline Mistral can even allow you to control your home computer and smart home devices just by making a call!

How we built it

Hotline Mistral uses a Twilio to provide and host the phone number, Whisper to transcribe the incoming audio into text, Mistral (via API) to process requests, and Eleven Labs to synthesize audio for the responses. Integrations with Amazon Alexa and OpenInterpreter extend the functionality to enable Hotline Mistral to control your home computer and smart home devices.

Challenges we ran into

Audio latency: Our initial implementation had a 7 second wait time between speaking to the model and hearing a response - in the context of a phone call, that feels like an eternity. We reduced this to a more palatable 3 seconds by streaming audio and text directly through websockets between each endpoint.
Whisper hallucinations: It turns out, for some reason, that Whisper interprets silence as random words spoken in Korean. This meant Mistral was constantly being fed unintended inputs any time the user wasn't speaking. We fixed this with a series of creative prompts both to Whisper and Mistral.

Accomplishments that we're proud of

Achieving latencies that feel more in-line with natural conversation. (update: as of 9am today, the Mistral API is now introducing a 20-30 second lag!!)
Enabling control of smart home devices - just feels like magic through a phone call.
Personalization to each user - having the AI recognize you by phone number, greet you, and remember your relevant email and integrated accounts also feels like magic.

What we learned

Sometimes, modern problems don’t require modern solutions. Also, silence is Korean.

What's next for Téléphonique (Hotline Mistral)

Web Portal: Log in via your phone number to access logs from each of your calls and integrate your relevant accounts.
More Integrations: The current implementations of Amazon Alexa and OpenInterpreter are messy and personalized to the development team. With a bit of additional work, we can generalize these for everyone, and add access to Zapier, IFTTT, and any number of addition systems.
Support for more users: Currently we are intentionally limited to one user at a time, as this is a hackathon and we don't want too much traffic while building—but the goal is to support any number concurrently.
Silent speech to text: We spent a lot of time at this hackathon working on enabling near silent mumble speech to be reliably detectable, and could get it working with just a bit more time. This is important both for public privacy and for accessibility for those with difficulty speaking loudly.

Callable phone number will be released after demo time as it's currently limited to 1 user at a time - but we're happy to demo it live!