PhoneticAI

Landing
Usage & Performance Dashboard
Agent Offerings
Contact Details
Agent Assignment (to serve contact)
Calendar & Events
Contacts

Inspiration

The idea for PhoneticAI stemmed from a need for better solutions than voicemail for people with busy schedules and inabilities to take calls. The use case then also extends to various enterprises most relevantly CRM platforms, that want to be able to manage and serve their customers at scale without having to worry about the logistics of writing notes, capturing essential information, and scheduling actionable events.

What it does

PhoneticAI is a robust solution for real-time voice agents over phone calls, agent assignment tailored for contact lists, and deep phone call analysis. When a call comes in, our platform routes the call to be managed the assigned AI agent that best meets the characteristics and capabilities to serve that customer or contact. Through a completely normal human like conversation, the system transcribes the entire call, gathers relevant sentiment and key events data, as well as adds important time sensitive events to a calendar. Users can change agent assignments as they please, go back and listen to audio recordings of all calls, get high level summary statistics and visual dashboards about how the AI agents are doing. Designed to work seamlessly so that no message is ever lost.

How we built it

PhoneticAI makes use Twilio to register phone numbers that we can then accept and manage calls through via code. We used RetellAI with a combination of complex prompts to build AI agents. We use webhooks to listen into call start and end events to perform real time processing of calls as they happen. We make use of OpenAI's API to gather sentiments and key events from transcripts of calls that we receive, and process and store relevant information into our MySQL database, which stores all our data including contacts, agents, assignments, calls etc. Our backend is built using Python & FastAPI for our client facing endpoints that our utilized by a React frontend to visualize and display all the information clearly and aesthetically.

Challenges we ran into

Managing real-time processing from our AI agents and actually setting up routing such that phone calls can be taken by AI agents was the most challenging part. Additionally, we had to carefully tailour or prompts while building our agents to ensure we had a diverse set of characteristics that served different purposes. Lastly, ensuring that the UI is not just pleasing to look at, but captures all the information a user could and will look for was also very important to us, and we spent a great deal of time brainstorming and iterating on our front end.

Accomplishments that we're proud of

We’re proud of creating a tool that is not only functional but also has the potential to make a meaningful impact on accessibility. Our team successfully reduced transcription errors by 30% compared to standard APIs, integrated real-time translation into 10+ languages, and built a visually appealing and intuitive interface. The project also brought out the best in our team’s collaboration, with each member contributing significantly to the final product.

What we learned

Throughout this project, we learned about the intricacies of phonetics and how they play a crucial role in understanding spoken language. We gained hands-on experience with speech-to-text APIs and machine learning models for audio analysis. Additionally, we learned the importance of efficient backend architecture to handle real-time data processing and how to design interfaces that simplify complex processes for end users.

What's next for PhoneticAI

The future of PhoneticAI lies in expanding its capabilities. We aim to:

Improve support for more languages and regional dialects.
Integrate with popular Calendar providers for better visibility on events
Use past transcripts and call data to build Automated Agent Assignment algorithms
Load balance agent usage to ensure minimal downtime and maximum usage of preffered agent while taking a call
Build your own agent (define agent characteristics and prompt)
Spam Call Handling (Classification)