Spoken.ai

Spoken.ai starts with a simple promise: type your request and let Sam handle the phone call.
Users upload a photo, describe the repair, and set a maximum budget without speaking.
Two quick questions help Sam understand the repair before calling providers.
The app turns the photo and answers into a clear request that Sam can explain to providers.
Sam calls local providers, confirms the work, and negotiates within the user’s budget.
Users compare provider offers by price, scope, and availability.
After choosing an offer, users select a time and send a booking request.

Inspiration

A simple repair can become difficult when getting a quote requires calling multiple providers, explaining the same problem repeatedly, and negotiating over the phone.

For speech-impaired people, people facing language barriers, and anyone who finds phone calls difficult, this can become a barrier to accessing local services independently. They may have to depend on someone else to make calls, pay more without comparing prices, or give up altogether.

We wanted to create a way for people to communicate their needs silently while still accessing the phone-based services that many local businesses require.

What it does

Spoken.ai helps users request local services without making a phone call.

Users upload a photo, describe the problem using text, set a budget, and answer a few simple questions. Sam, our AI voice agent, then calls nearby providers on the user’s behalf.

Sam explains the problem, confirms that the provider can help, asks about pricing and availability, and negotiates within the user’s budget. The user receives the results as text and can compare valid offers before requesting a booking.

How we built it

We built the mobile application using React Native and Expo. The backend uses Node.js and Express to manage intake, clarification questions, provider calls, results, and booking requests.

Gemini analyzes the uploaded photo and written description, identifies the service category, generates clarifying questions, and extracts structured offers from call transcripts.

ElevenLabs powers Sam’s conversational voice agent and transcription. Twilio connects Sam with provider phone lines. After each call, the system extracts details such as price, scope, availability, and conditions so offers can be compared consistently.

Resend is used to send booking requests and calendar information to the selected provider.

Challenges we ran into

The biggest challenge was making phone conversations reliable enough to produce useful and comparable offers. Providers may describe prices differently, include labor or materials, or provide conditional estimates.

We addressed this by giving Sam a structured conversation plan and extracting important fields from the transcript. Prices are normalized before comparison so the system compares structured offers instead of simply comparing raw text.

Another challenge was coordinating image analysis, clarification questions, outbound calls, transcript processing, price comparison, and booking within a simple user experience.

Accomplishments that we’re proud of

We built a working end-to-end flow from a photo and text description to an AI-powered provider call and structured price comparison.

We are also proud that the entire user experience can be completed without speaking. Users can communicate through photos, text, and simple selections while Sam handles the spoken conversation.

We added safety checks for unsafe, unrelated, or manipulative descriptions before the content reaches the voice agent. Sam also identifies itself as an AI assistant and does not disclose or infer a user’s disability.

What we learned

We learned that building a voice agent is more than generating speech. The agent needs a clear conversation plan, strict boundaries, reliable transcript extraction, and safeguards for unexpected responses.

We also learned that accessibility must be part of the complete product flow. It should influence how users provide information, how results are displayed, and how much control the user retains.

What’s next for Spoken.ai

We started with local repair and service requests, but the same idea can support many situations where people are required to use phone calls.

Future applications could include clinics, landlords, government offices, insurance providers, and other local services.

Our long-term goal is to make phone-based services accessible to everyone, regardless of how they speak, what language they use, or whether they feel comfortable making calls.