yoyaku agent

A Voice Receptionist for Small Shops

Comment

Inspiration

In Japan, many massage, acupuncture and shiatsu practitioners are blind or have low vision. Yet almost every booking tool is built around screens, calendars and forms. Many of these practitioners still run their business on paper, phone calls and help from sighted people.

Earlier this year I built a small practice prototype on Replit. It taught me the domain, but it never went beyond a prototype. For this hackathon I rebuilt it from scratch around one question: what if the owner never had to look at a screen at all?

What it does

GoYoyaku-chan Neo is a voice-first booking assistant. The main screen is one large microphone button. The owner says what they want, hears a one-sentence confirmation, says "yes", and it's done.

  • Bookings: add, move, cancel, repeat weekly, "when am I free tomorrow?"
  • Money: hear sales and unpaid amounts. Take a photo of the cash a customer hands over, and the app counts it and works out the change.
  • Customers: notes like "Tanaka has a bad back" are read back before the next visit.
  • Staff: clock-ins, payroll, and booking a practitioner by name.
  • Shop: multi-visit tickets, coupons, expenses, follow-up emails.
  • Advice: "What should a multi-visit ticket cost?" is answered from the shop's own numbers.

When the owner opens the app, it speaks what changed since last time and who the next customer is. Customers book on an ordinary web page and can pay at the shop or by card in advance.

How we built it

  • Oct 21: a Python CLI prototype: a Nemotron tool-calling agent on Nebius Token Factory plus a pure calculation layer with tests. We compared Nemotron Lightning and Super here.
  • Oct 22 onward: rebuilt as a web app (React, Express, PostgreSQL) hosted on Replit, then voice-only onboarding, staff features, money features, card payments and push notifications.

All AI is NVIDIA Nemotron via Nebius Token Factory, called with the standard OpenAI SDK:

  • Nemotron 3 Super chooses among dozens of tools on every spoken request, writes the spoken confirmation, and answers business questions.
  • Nemotron 3.5 Super VL counts banknotes and coins from a photo and returns JSON. The server recalculates the total and flags the reading if the sums disagree.

Speech uses the browser's own speech recognition and speech synthesis, so the app works on an ordinary phone or computer with nothing to install.

Safety by design: the AI proposes, code decides

  • Every change is confirmed out loud first. The server holds the action until the owner says "yes".
  • Cancelling, recording money and changing settings also need a spoken passphrase. The passphrase is never sent to the AI.
  • Unknown tool names are refused. Card payments run in Stripe test mode only, and card numbers never pass through the app.

Challenges we ran into

  • A confident false "done". Nemotron Lightning was our first choice for speed, but in 7 of 8 test questions it answered without calling any tool and made up bookings. In the web app it said "I've booked it" when nothing was booked. A blind user can't glance at a screen to catch that, so we switched to Super, which called the tools every time in 3 to 5 seconds.
  • Silent speech. Speech often failed silently in Chrome and on iPhone. We added fixes for Chrome quirks and iOS audio unlocking, and a /voice-check page that explains why speech is silent.
  • Two voices at once. The app's voice talked over screen readers like VoiceOver. The app now asks on first launch whether a screen reader is in use, and can turn its own voice off.
  • Onboarding without sight. Sign-up, login and voice settings had to work entirely by voice, including choosing a voice by listening to it.

What we learned

  • For agents that act on someone's behalf, reliable tool use matters more than speed, and the final safety rules belong in code, not in the prompt.
  • Accessibility is not a layer added at the end. Focus movement, live announcements, keyboard control and non-color signals shaped every screen from the start.

What's next

  • Smart glasses: hands-free use during treatments, and counting cash through the glasses' camera instead of holding up a phone
  • Testing with blind practitioners in real shops
  • Confirming push notifications on real Android and iPhone devices
  • Keeping more of the voice pipeline on Nemotron as multilingual speech

Built With

Share this project:

Updates

Submission history