-
-
High Level Design for Syntax Realty - AI Lead Conversion POC
Inspiration
I believe that mortgages move the economy forward and that it all starts with the home-owner and the realtor. I knew how AI was changing the face of the mortgage industry from my own professional experience and I wanted to build something that could change the way homes are sold.
What it does
This tool runs in a browser and a realtor can use either the text or audio agents. The realtor chats with the agent and is asked for a location to find home listings online that are For-Sale-By-Owner. Once the realtor provides a location, the multi-agent system searches the web to find these listings. The realtor is asked if they would like to send a voice-ai call to the home-owner or if they want to send a text. After confirmation, a text or call gets sent out. The goal is to persuede the home-owner to meet with the real estate agent for a consultation.
How we built it
This is a multi-agent system that uses agents as tools. There is a supervisor that communicates with the user and leverages a sequential agent for lead generation, a loop agent for text message content generation, or a tool call that initiates a voice-ai call. The multi-agent system was built in Google ADK, and the voice-ai call was built in pipecat, both leveraging gemini models.
The Text agent is built differently than the voice agent. The text agent is a traditional Google ADK multi-agent system that uses AgentTool's to communicate with the sequential agent or loop agent.
The Voice agent actually communicates with sub-agents using an additional FastAPI server that does run on cloud run. There are 2 endpoints that the supervisor can leverage to use either the sequential agent or loop agent.
Both of these agents send an internal HTTP request to a Compute VM instance that places the voice AI call when the initial_phone_call tool is called by the supervisor.
All secrets are stored in Google Secret Manager.
Challenges we ran into
- The reason the voice supervisor communicates with subagents via HTTP is because the native-audio model doesn't support AgentTool. I could have used a gemini-2.0-flash-live-001 model, but the native-audio user experience is preferred.
- The reason the voice-ai phone call doesn't run in cloud run is due to networking restrictions of cloud run which prevented the Daily.co WebRTC room from intercepting user audio #WorksOnMyMachine. After quite a bit of trial and error, I made the decision to move just the voice-ai phone call piece over to Compute VM while keeping the majority of the work on Cloud Run.
- I would've also chosen to use Google's GCES platform for a voice calls, but could not acquire an individual license in this short period of time.
- Because it takes 2-7 days to register a phone number for text in Twilio, the text messages don't actually send. (see Twilio error 30034 - US A2P 10DLC)
Accomplishments that we're proud of
Honestly, I'm proud of this entire experience. The biggest challenge I had was overcoming the network restrictions where I was not getting any audio on my phone calls. I tried every-thing I could to get the voice-call portion to work in cloud run, then I took a step back, re-architected the system, and moved that piece over to Cloud Run.
What we learned
I learned how Cloud Run can access Secret Manager so I didn't have to deploy a .env file with my secrets. I also learned about native-audio and Cloud Run limitations, and how to log input/output between agent inferences.
What's next for Realtor Assistant
I would like to give this application a new UI. My vision is to add a Log Mel Spectrogram to all voice assistances, add live transcriptions, and improve conversational system prompts. I also feel like there are other use cases for this application in the Realty industry.
Log in or sign up for Devpost to join the conversation.