ClinicalTalkAI

Python agent runs real-time NLP via Amazon Bedrock and streams medical insights through voice

Comment

Inspiration

Clinical talk AI was inspired by real world operational friction faced clinical trial investigators and doctors in managing and searching complex patient data. In their high stress environment workplace they are forced to look away from active care context to navigate brittle database query builders or write manual code filters. The idea stemmed from wanting to build a autonomous voice driven data science workspace where investigators can just use natural language question to receive instant visual and vocal answers.

What it does

ClinicalTalkAI is a autonomous voice driven clinical data partner built for clinical trial investigators.It listens to verbal user instructions, automatically synthesizes complex analytical parameters using Amazon Bedrock (Claude Haiku 4.5), executes secure data queries via FastMCP tools over S3-hosted clinical datasets, updates active Streamlit data grid pipelines natively, and responds back vocally using Amazon Polly. It features both a browser-based speech-to-text simulation interface and a full AWS Lambda Alexa Skill backend for smart speaker integration.

How we built it

  • Frontend & Voice Interface: Developed using Streamlit (streamlit==1.38.0) paired with streamlit-mic-recorder for secure browser audio capture and Amazon Polly (Neural Voice: Joanna) for real-time text-to-speech telemetry.
  • Agentic Orchestration: Powered by Amazon Bedrock using the global cross-region inference profile (global.anthropic.claude-haiku-4-5-20251001-v1:0) via the Converse API for multi-turn tool calling and reasoning in backend/agent.py.
  • MCP Backend Engine: Utilizes FastMCP (backend/mcp_server.py) to expose structured data query tools (query_patients_by_disease, get_enrollment_summary, summarize_adverse_events) for autonomous LLM discovery and execution.
  • Zero-Disk Data Privacy: Streams clinical datasets securely from Amazon S3 directly into volatile in-memory Pandas byte buffers, maintaining strict data privacy compliance without local disk caching.
  • Smart Speaker Integration: Implements an AWS Lambda handler (backend/alexa_lambda.py) built with the Alexa Skills Kit SDK (ask-sdk) for voice-driven smart speaker execution.

Challenges we ran into

We have a dedicated friction_log.txt submitted in github listing all challenges faced in the project summarising shortly here

  • Infrastructure, Native Build, & Financial Governance We ran into some pretty frustrating setup headaches early on, especially trying to get cryptography packages to compile on macOS without proper OpenSSL headers. We ended up standardizing our container environment to bypass those native build traps. On top of that, we had to keep a close eye on our cloud budget; since open-ended LLM loops and text-to-speech generation can burn through promotional credits fast, we implemented strict token caps and character truncation filters to keep operational costs down to just a fraction of a cent per loop.

  • API Schema Validation & Frontend Audio Sandboxing Getting the browser's microphone inputs to work smoothly inside Streamlit proved tricky because of secure iframe sandboxing, which forced us to switch over to a dedicated binary recording wrapper. We also hit a major roadblock with Bedrock's Converse API when stripping out tool configurations on final turns kept triggering schema validation exceptions. We eventually solved it by ensuring our tool configs stayed active across historical turns and tweaking our system prompt conditioning to prevent any looping.

Accomplishments that we're proud of

  • Zero-Persistence Data Privacy Architecture: Designing a secure, memory-only S3 data streaming pipeline that eliminates local disk persistence to align with strict healthcare data privacy standards.

  • Bedrock Converse API Mastery: Successfully reverse-engineering and resolving multi-turn chat state requirements, ensuring zero validation dropouts during autonomous tool handshakes.

  • Extreme FinOps Efficiency: Capturing the 10% cross-region global inference discount and enforcing strict token/character caps to operate at a negligible cost of ~$0.003 per loop.

What we learned

  • Generative AI agent architectures require strict adherence to API schema rules, especially regarding historical tool configuration persistence across multi-turn chat loops.
  • Decoupling tool definitions via FastMCP cleanly bridges reactive UI dashboards and autonomous LLM reasoning.
  • Production-ready cloud applications must proactively integrate FinOps guardrails and zero-persistence data governance from day one.

What's next for ClinicalTalkAI

  • Expanding multi-surface voice execution capabilities and deep Alexa skill integrations for clinical trial site environments.
  • Enhancing real-time data pipeline streaming and enterprise IAM role auditing for broader healthcare compliance frameworks.

Built With

Share this project:

Updates

Submission history