Colonoscopy Prep Coach on Raspberry Pi

A portable healthcare advice provider powered by OpenAI OSS with speech-to-text and text-to-speech for patients who do not have readily available access to the Internet.

Comment

Inspiration

As a practicing gastroenterologist, I see many patients who struggle to remember or understand bowel preparation instructions before their colonoscopy. Missing or misunderstanding prep can result in poor visualization, repeated procedures, and unnecessary patient anxiety.

At the same time, I’ve been inspired by the rise of open-source OSS language models and wanted to explore how far we can push their accessibility — even onto low-cost edge devices like a Raspberry Pi.

This project grew out of the intersection of:

  1. Clinical need: Helping patients get clear, personalized, repeatable prep instructions.
  2. Open AI models: Experimenting with GPT-OSS, Whisper, and Piper as fully local tools.
  3. Accessibility & inclusivity: Making AI assistants work offline and on affordable hardware.

What it does

Advise patients on preparing for a colonoscopy

How we built it

  1. Hardware: Raspberry Pi 5 (16 GB) 7” Roadom touchscreen USB microphone/speaker setup
  2. Software: Whisper.cpp for speech-to-text (STT) GPT-OSS 20B (quantized by Unsloth) with Ollama for reasoning Piper TTS for text-to-speech A shell loop for live STT → LLM → TTS A Streamlit UI for kiosk-style demo on the touchscreen
  3. Process: Set up RPi 5 with PiOS and necessary build tools. Installed and built Whisper.cpp, Ollama, and Piper. Pulled GPT-OSS quantized GGUF from Hugging Face (Unsloth). Designed a bash loop (pi_coach.sh) to capture audio, transcribe, query the LLM, sanitize outputs, and speak them back. Created a Streamlit UI (pi_coach_ui.py) for a simple touchscreen-based kiosk.

Challenges we ran into

  1. Model compatibility: Different OSS models (Ollama, Whisper.cpp, Piper) update frequently, and CLI flags or file paths often broke older instructions.
  2. Hardware limits: GPT-OSS 20B is heavy; balancing quantization and responsiveness on a Raspberry Pi required patience.
  3. Voice cleanup: The LLM often produced meta-reasoning text; we had to engineer sanitization filters and tiny guardrail prompts to ensure only concise patient-friendly responses were spoken.
  4. Debugging in unknown territory: Unfamiliarity with CMake errors, deprecated Hugging Face CLI, and ALSA device mapping slowed progress — but each problem was eventually solved through iteration.

Accomplishments that we're proud of

  1. A working prototype of a fully offline, Pi-based voice assistant for colonoscopy prep.
  2. Patients (in the future) could simply ask, “Can I drink coffee today?” and receive a short, empathetic, spoken answer like: “No, avoid coffee before your colonoscopy; stick to clear liquids.”
  3. The project demonstrates that accessible AI healthcare tools can be built using open-source models and inexpensive hardware.

What we learned

  1. Quantization & model optimization: Learned how GGUF and MXFP4 quantization enable very large models (20B parameters) to run on consumer-grade hardware.
  2. Speech processing pipeline: Explored how Whisper.cpp and Piper TTS can be chained with GPT-OSS to create a “talk → think → talk” loop.
  3. Edge AI feasibility: Discovered that even on a Raspberry Pi 5, it’s possible to run an end-to-end voice assistant locally with no cloud connection.
  4. It was surprising — and motivating — to realize that patients could eventually use this without relying on the internet.

What's next for Colonoscopy Prep Coach on Raspberry Pi

  1. Train or fine-tune smaller GPT-OSS models for domain-specific medical prep coaching, improving both speed and accuracy.
  2. Expand beyond colonoscopy prep into general procedure readiness coaching (endoscopy, surgery, imaging).
  3. Explore Arduino or simple robotics add-ons (e.g., LED indicators or printed handouts) for multimodal patient guidance.
  4. NVIDIA Jetsons?

Built With

Share this project:

Updates