Synergy

Synergy
Data Flow Diagram

Inspiration

The inspiration for Synergy came from a realization that a large number of customer inquiries are centered around similar questions or issues. These repetitive tasks take up valuable time and resources from customer service agents, who could otherwise be solving more complex issues. By automating the process for these commonly-asked questions, we aim to enhance the productivity of customer service teams, improve customer experience by minimizing wait times, and provide accurate, instant information round-the-clock.

Current tech enables this to happen, but a big issue is latency. Speech to text, text-to-text, and then text-to-speech done sequentially would take 10s of seconds to process for a single query, which is way too long for a conversation over the phone. In contrast, our latency from end-of-speech to first-word-heard is on average 2-3 seconds. To do this, we built extensive infrastructure (mainly using websockets and HTTP StreamingResponse) that

streams speech to text transcription in realtime using Google STT
streams GPT tokens as they're generated to elevenlabs to produce audio
streams the audio from elevenlabs back to the node.js server as it's generated.
streams the audio from node.js to frontend and builds an audio buffer queue to play sequentially for the user.

This workflow ensures that before the entire output is even generated, the user receives the beginning of the audio response, which simulated a human conversation much better than current chatbots, who's latency makes them terribly inhuman.

We measured both of these approaches and found that our infrastructure provides a >10x speedup in latency (38 seconds on average for non-streaming naive script vs 3.5 seconds on average for Synergy).

What it does

Synergy acts as a virtual customer service agent capable of understanding spoken customer inquiries and answering them in real-time by tapping into company-specific documentation. In our MVP, we focused on H&M's documentation to provide instant and accurate information about Delivery, Return Policy, and Membership. The bot uses voice recognition and speech-to-text technology to understand the customer's query, searches the database to retrieve relevant information, and uses text-to-speech technology to respond, thereby providing a seamless conversational experience.

How we built it

We used a stack consisting of GCP's Speech-to-Text API, Palm/VertexAI/GenerativeLanguage/MakerSuite for natural language understanding, and Text-to-Speech API for vocalizing responses. All of this runs on Compute Engine to handle high workloads efficiently.

We set up a MongoDB Atlas database with three different collections corresponding to Delivery, Return Policy, and Membership. The bot fetches and compiles this data to form a coherent response.

For the conversational aspect and real-time processing, we employed GPT-3.5 Turbo along with ElevenLabs, NodeJs, Express, and Websockets to assemble the complete product.

Challenges we ran into

Speech Recognition: Achieving accurate speech recognition in noisy environments or for various accents was challenging.
Database Optimization: We needed to ensure our MongoDB collections were optimized for quick and precise data retrieval.
Real-time Processing: Creating a seamless conversational experience required meticulous integration of all the different technologies.
Speech Buffering: We encountered a very frustrating problem pertaining to the streaming process; our buffering resulted in some responses being overlapped as tokenization was pushing information too fast in, but not fast enough out.

Accomplishments that we're proud of

Seamless Integration: We successfully integrated multiple services and technologies into a cohesive product.
Real-time Responsiveness: Synergy can answer customer queries in real-time, providing a much-needed boost to customer service efficiency.
Scalability: Our architecture is designed for easy scalability, allowing us to include more features and serve more businesses.
Documentation Accuracy: The system has a high rate of success in pulling the correct information from the database to answer questions accurately.

What we learned

The importance of optimizing database queries for real-time applications.
How to efficiently integrate multiple APIs into a cohesive product.
The nuances of creating a conversational AI experience that is both natural and informative.
Scalability considerations in real-time applications.

What's next for Synergy

We will primarily focus our efforts on using vector databases to even more accurately help our customers use our application. This will ensure even better integration and will require less compute power than retraining models on every companies' documentation pertaining to every aspect of their public facing statements.