Accenture pls i need the bag

We wait in your GP's 8am phone queue for you, navigate the menu, and an AI voice agent books your appointment, then texts you the result.

Comment

Inspiration

Anyone in the UK knows the 8am scramble: you ring your GP the second lines open, sit through "you are number 14 in the queue", and by the time someone picks up, every appointment is gone. It's worst for the people who need it most: people with jobs they can't step away from, carers, people with anxiety about phone calls, and people who find long phone menus hard. We wanted to take the waiting out of the patient's hands entirely.

What it does

A patient fills in one form: name, date of birth, address, reason for the appointment, when they're free, and their surgery's number. Then we take over:

  1. We phone the surgery and wait in the queue for them.
  2. We navigate the phone menu, recognising things like "press 2 for appointments" and pressing the right key.
  3. We sit on hold and recognise recorded queue messages, hold music and "the surgery is closed".
  4. When a real receptionist answers, an AI voice agent introduces itself as an AI assistant calling on the patient's behalf and books the appointment using the patient's details. It only shares those details when asked.
  5. The patient gets an SMS with the booking: date, time, clinician.
  6. If the receptionist insists on speaking to the patient, we ring the patient and join them into the call.

A live status page shows every step as it happens. The intake form also catches emergency wording and points people to 999 or 111 instead.

How we built it

  • FastAPI backend serving the intake form, a live status API and the Twilio webhooks.
  • Twilio places the outbound call to the surgery, transcribes each phrase it hears, sends SMS and handles the conference call when we bring the patient in.
  • Google Gemini classifies what we're hearing on the line (hold, ivr_menu, human or closed) using structured JSON output. Fast keyword rules handle the obvious recordings first, and we fall back to them if Gemini takes longer than 2.5s.
  • ElevenLabs powers the agent: a text-only conversational agent with client tools (record_booking, connect_patient, report_outcome, hang_up) that drive the call, plus ElevenLabs text-to-speech in a British voice. The audio streams to the call while it's still being generated, so the receptionist isn't left waiting in silence.
  • pytest suite, and an offline script that compares Gemini against the keyword rules on real queue phrases.

Challenges we ran into

We built the whole thing on a Twilio trial account, which fought us at every step:

  • No live audio streaming, so we had to work phrase by phrase with Twilio's speech transcription.
  • The call is cut off after 10 webhook callbacks, which gives us roughly 9 exchanges for the whole call, hold time included. We stacked several listen steps into each response so silence and hold music cost nothing, made the agent's opening line a template so taking over is free, and told the agent how many turns it had left so it wraps up in time.
  • Twilio waits only 5 seconds for each response, so the classifier, the agent's reply and speech synthesis all have to fit in that window. That's why we stream the audio and cap each turn.
  • Telling a live receptionist ("Good morning, Riverside Surgery") apart from a recording that sounds just like one. Recordings repeat and people don't, so we remember phrases we've already heard.

Accomplishments that we're proud of

  • It makes real phone calls: it goes from a web form to waiting on hold, a booked appointment and an SMS.
  • It works within brutal trial-account limits: a whole booking in about 9 conversational turns.
  • It hands the call to the patient when a human really is needed, instead of the AI bluffing.
  • We thought about safety and privacy from the start: the agent says it's an AI, there's emergency triage on the intake form, and session data is deleted an hour after the call.

What we learned

  • Real-time voice is mostly a latency problem. Every millisecond between "the receptionist stopped talking" and "the agent starts talking" matters.
  • Cheap rules plus an LLM beat either one alone: rules are instant and the LLM handles the long tail.
  • Health data is special-category data under UK GDPR, which changes how you design storage from day one.

What's next for Accenture pls i need the bag

  • A paid Twilio account for live audio streaming and unlimited turns.
  • Checking Twilio's request signatures, plus encrypted, short-lived storage of patient data.
  • Pilots with real GP practices and clinical input on the emergency triage list.
  • Booking through the NHS App or online systems where a surgery offers them, and using the phone only as the fallback.

Built With

Share this project:

Updates

Submission history