Inspiration
Booking an appointment is a chore. You find clinics that take your insurance, call each one, wait on hold, read out your member ID, hear that the first opening is in three weeks, and call the next one. Then you figure out how to get there.
For Mary, who is 78, that chore is why she keeps skipping her dental cleaning. Her knee makes walking hard, she doesn't drive anymore, and her daughter Sarah lives three hours away. So Sarah ends up booking it from work on her lunch break with a photo of Mary's insurance card.
We started with seniors because the friction hits them hardest, but it's not only them. Anyone who just moved to a new city, anyone shopping on price without insurance, anyone with a language barrier, anyone who just hates phone calls has the same problem. We wanted something that makes the calls for you, asks you one question, and keeps your private details on your own device.
What it does
Mary taps the microphone on the dashboard and says: "I need a dental cleaning this week, under $150. I have Delta Dental. My knee makes walking hard so I'll need a ride."
Then GranniAI does the rest.
- A Nemotron Nano model on the laptop turns that sentence into the few facts a clinic needs: care type, budget, insurance carrier, free time, and whether she needs a ride. Her knee, her date of birth, her member ID, her address, and Sarah's number all stay on the device. The chips on screen say "Processed on device."
- Three AI call agents phone three nearby clinics at the same time. Each one says it's an AI assistant calling for a patient, then asks about openings, price, and whether they take the carrier. Transcripts show up live on the dashboard.
- Code throws out anything over budget, outside her free time, or not covered. Nemotron Super ranks what's left and explains the pick in plain words: fits her free time Thursday 9:30, cheapest at $95 vs $120, closest at 1.2 miles, and the trade-off in one sentence.
- Mary taps Approve, or replies YES by text. That's the only question she gets asked.
- GranniAI books it, puts it on her calendar with a reminder, and checks for an accessible ride. If there isn't one, it texts Sarah just what she needs: the clinic, the time, the address, and that Mom needs a ride. Sarah replies "I can drive her" and everyone gets the confirmation.
One sentence from Mary. One tap. One text from Sarah.
How we built it
We use three Nemotron models, each one sized for its job.
- Nemotron 3 Nano 4B, 4-bit, running locally through Ollama. It redacts the spoken request into structured JSON and writes Sarah's scoped text message from Mary's full private record. It never touches the network.
- Nemotron 3.5 Lightning 30B on the NVIDIA API. It's the live call agent, plugged into ElevenLabs as a custom LLM through a small proxy we wrote. It streams replies into three phone calls at once at about 340 ms to first token.
- Nemotron 3.5 Super 120B on the NVIDIA API. It ranks the offers that survive the hard rules and writes the reason tags and the trade-off sentence.
The reason intake runs locally is the whole point of the project. Mary's raw request is medical information. A cloud redactor has already received the sensitive text by the time it redacts it. The only way to guarantee the raw words never leave the device is to redact on the device, and a 4-bit Nano makes that work on a normal laptop in a couple of seconds. We measured what actually goes to the cloud per request: about 240 bytes of redacted JSON. Lightning, Super, Google Places, ElevenLabs, and Twilio never see anything Nano dropped.
That's how we think about HIPAA's minimum-necessary rule in an agent system. Not a privacy policy that promises restraint, but a pipeline where the sensitive data can't physically reach the parts that don't need it. Open weights mean anyone can audit the redaction. And because Nano works with Wi-Fi off, our demo literally turns Wi-Fi off, speaks the request, shows the chips, and turns Wi-Fi back on before the calls start.
Everything else is deterministic code. A FastAPI orchestrator runs the steps in a fixed order: intake, calendar, places, calling, ranking, approval, booking, transport, contact. The models rank and explain. They never decide what happens next. State lives in SQLite with idempotency keys on every webhook and side effect, so a retry or a double tap never books twice. Every step emits an event that streams to the Next.js dashboard over Server-Sent Events, scoped per viewer so Sarah only ever gets what she's allowed to see.
Stack: Nemotron Nano, Lightning and Super, Ollama, NVIDIA NIM API, ElevenLabs Agents, Twilio, Google Places, Routes and Calendar, FastAPI, SQLite, Next.js.
Challenges we ran into
- Proving the raw request never leaves the device, not just saying it. We built a 20-utterance redaction test set with expected kept and dropped fields, a code-level backstop that catches identifier-shaped values if the model misses one, and a metrics log that records bytes sent to the cloud.
- Streaming a custom LLM into live phone calls. ElevenLabs sends the whole conversation to our proxy every turn. The call ID and secret have to travel in the extra body and never be trusted from the model. Lightning has to run with thinking off, or it spends its whole token budget reasoning and says nothing to the receptionist.
- Never treating what the agent said as proof. A receptionist saying "you're all set" is not a booking. Status comes from receipts. A confirmed booking creates the calendar event once, and transcript text never changes anything.
- Scoping Sarah's view. She shares the dashboard but must never see the raw request. Every read is viewer-scoped, and the contact view scrubs identifier-shaped values out of clinic lines as a second layer.
- Twilio trial limits. Outbound texts on a trial only send preset templates, and 10DLC registration takes days. So outbound texts are mocked and labelled, while the inbound YES is real.
Accomplishments that we're proud of
- A full loop: spoken request, on-device redaction, three parallel calls with live transcripts, an explained decision, one approval, a booking, a calendar event, a ride check, a text to family, a reply, and an honest final status.
- Three real phone calls through ElevenLabs over Twilio with Nemotron Lightning answering, returning a usable price and time slot.
- About 240 bytes per request leave the laptop, and we can show exactly which bytes.
- Two roles on one dashboard, owner and contact, with the contact seeing only their scope.
- Every outside service has a mock mode that emits the same events, labelled "simulated" on screen anywhere a judge could be misled.
What we learned
- Small local models earn their place when the job is redaction. Nano doesn't need to be clever. It needs to be local, fast enough, and auditable.
- Voice and ranking want bigger models, but they should only ever see the redacted request. That split isn't a compromise. It's the design.
- Rules in code, judgment in the model. Budget, calendar, and insurance rules are enforced in code. The model ranks and explains inside them. That's what keeps the demo predictable.
- One approval is enough. Every time we wanted to ask Mary another question, we asked whether a good human assistant would. Usually they'd just handle it.
What's next for GranniAI
- Let the user steer mid-run. Things like "actually, mornings are better" or "skip that clinic" without restarting. The pipeline already treats missing fields as part of intake, so this is a small extension.
- An insurance card and profile screen. Photograph the card once, keep carrier, member ID, and address in a profile. It fits the privacy model as-is: the profile stays on the device, Nano reads it locally, and only the carrier name ever leaves.
- Wire live voice into every run now that the individual calls are proven, and measure time-to-booked against calling three clinics yourself.
- Beyond dental: eye exams, flu shots, the vet, a haircut. Spanish-language calls. Letting the user phone GranniAI instead of opening a dashboard.
GranniAI never calls a real clinic without consent. During this hackathon every clinic phone was a teammate's, every patient was synthetic, and the agent always said it was an AI.
Built With
- elevenlabs
- fastapi
- google-calendar
- google-places
- nemotron
- next.js
- nvidia
- nvidia-nim
- ollama
- python
- react
- server-sent-events
- sqlite
- twilio
- typescript