Betty AI Truck Driver Welfare

This Live AI agent aims to provide welfare checks based on fatigue triggers, erratic driving and reminders to take breaks.

Comment

Betty — AI Voice Companion for Truck Drivers

Inspiration

Watching truck crashes from fatigue and distraction events from my consultancy with an Australian large logistics company gave me the inspiration to ask: can AI tech help? Right now there are fatigue and driver monitoring systems but they rely on a fleet manager or supervisor to act on these alerts. By using AI to assist we can have this running 24/7 across thousands of truck drivers which can then reduce incidents on the road saving lives and billions in costs per year.

What it does

Betty is an AI voice companion that proactively calls truck drivers when safety systems detect danger. Instead of a buzzer or an alert on a screen that someone has to act on, Betty picks up the phone and has a real conversation that handles natural interruptions gracefully.

  • Fatigue camera detects droopy eyes or yawning — Betty calls and checks in naturally
  • Erratic driving like lane drift or harsh braking — Betty calls to see if the driver is okay
  • Approaching mandatory break limits — Betty calls and suggests a nearby rest stop, sends a visual card to the driver's screen
  • Betty remembers previous calls within the shift — if she suggested a rest stop 30 minutes ago, she'll follow up on it
  • If the driver won't pull over despite being clearly fatigued, Betty escalates to the fleet manager with an incident report
  • For the demo, two AI sessions talk to each other (Betty + a simulated driver) so you can see the full experience without needing a mic

How we built it

  • Gemini Live API for real-time two-way voice conversations
  • Google GenAI SDK for session management and function calling mid-conversation
  • Google Search grounding to get real rest stop info for recommendations
  • Imagen 4 to generate scenic backgrounds for the rest stop cards
  • Google VEO to generate the simulated fatigue camera footage
  • Cloud Run for deployment — scales to zero when not in use
  • FastAPI + WebSockets for the real-time dashboard and driver phone UI
  • AES-256-GCM encrypted memory per driver with 14-hour auto-expiry
  • Two Gemini Live sessions bridged together for the simulation — audio resampled and streamed between them in real-time

Challenges we ran into

  • Getting two Gemini sessions to talk to each other smoothly — audio resampling between 16kHz and 24kHz, managing turn-taking so they don't talk over each other
  • Making it sound real — mixing cabin noise into the audio so Betty hears road noise like a real phone call
  • Browser audio policies kept blocking our ringtone and call sounds — had to preload everything as blobs and create fresh audio elements on each click
  • Prompt engineering for voice is nothing like text — Betty needed to sound like a caring person, not a chatbot reading a script. Short responses, natural phrasing, Australian personality without being over the top
  • Generating visual cards mid-conversation caused noticeable pauses — fixed by running card generation in the background so Betty keeps talking while the card renders

Accomplishments that we're proud of

  • Betty actually sounds like a real person checking in on you, not a system reading alerts
  • The simulation mode means anyone can experience a full conversation without a microphone — just click and watch
  • Cross-call memory makes a huge difference — Betty referencing something from an earlier call feels surprisingly human
  • Driver data is encrypted per-driver with derived keys so even if the database is compromised, you can't read another driver's data
  • End-to-end from fatigue event to voice call to rest stop card in under 10 seconds
  • 119 unit tests running with no API key needed

What we learned

  • Voice AI is completely different to text AI — tone, pacing and brevity matter way more than how much info you pack in
  • Proactive AI (the system calls you) hits different to reactive AI (you ask it a question). Drivers respond more naturally when Betty initiates
  • Gemini's native audio model picks up emotional nuance — it can hear when someone sounds tired vs annoyed vs anxious, which text-to-speech pipelines miss entirely
  • Combining Gemini Live API + function calling + grounded search gives you a surprisingly capable real-time agent that can look up data and take actions mid-conversation

What's next for Betty AI Truck Driver Welfare

  • Connect to a telephone service (eg. Twilio) so that real world conversations can take place
  • Connect the dashboard and browser to all access to the microphone so that we can interact with Betty via the web
  • Connect to real telematics and fatigue camera APIs (Seeing Machines, Lytx) instead of mock data
  • Multi-language support — Australia's trucking workforce speaks Mandarin, Hindi, Vietnamese, Arabic and more
  • Shift-over-shift wellness trending — spot patterns like a driver who's always fatigued on night runs and flag it to management
  • Let drivers call Betty themselves when they're feeling lonely or stressed on long hauls — a welfare companion, not just a safety tool
  • Integrate with fatigue management rules for jurisdiction-specific break calculations
  • Take it to other industries — mining, aviation, maritime — anywhere human fatigue monitoring and real-time AI intervention can save lives

Built With

  • audio
  • cryptography-(aes-256-gcm)
  • cryptography-(aes-256-gcm-+-hkdf)-other:-websockets
  • css
  • css-frameworks:-fastapi
  • fastapi
  • gemini-flash-api
  • gemini-live-api
  • google-artifact-registry
  • google-artifact-registry-databases:-sqlite-apis:-gemini-live-api
  • google-cloud-build
  • google-cloud-run
  • google-genai-sdk
  • google-search-grounding
  • google-veo
  • google-veo-libraries:-google-genai-sdk
  • html
  • imagen-4
  • javascript
  • numpy
  • opencv
  • pillow
  • python
  • sqlite
  • uvicorn
  • uvicorn-platforms:-google-cloud-run-cloud-services:-google-cloud-build
  • web
  • web-audio-api
  • websockets
Share this project:

Updates

Submission history