Inspiration
Most medical chatbots are glorified search engines — they match your symptoms to documents and spit out generic advice, forgetting everything about you the moment the chat ends. We wanted something closer to a real doctor: someone who remembers your history, reasons through complex symptoms step by step, and gets to know your health profile over months and years of conversation. BayMD was born from the conviction that an AI doctor needs three things a traditional RAG pipeline cannot provide — autonomous reasoning, persistent memory, and the humility to know when it needs more information.
What it does
BayMD is an AI personal doctor powered by a ReAct Agent that autonomously decides when to search medical knowledge bases, when to call external MCP tools, and when it has enough context to give a final answer — reasoning through up to 10 iterations per query across 7 clinical scenarios including symptom triage, drug inquiry, TCM syndrome differentiation, lab report interpretation, and hospital recommendation. Behind the Agent is a three-tier progressive memory system inspired by EverOS: a sliding window preserves recent conversation turns verbatim, an LLM-powered compressor generates long-term summaries when the window overflows, and a semantic layer asynchronously extracts structured AtomicFacts and Episodes after every conversation, deduplicates them via SHA-256, stores them in pgvector with HNSW indexing, and retrieves the most relevant memories to inject into the system prompt on the next visit — so the system builds a richer health profile the more you use it. When a user accumulates 30+ Facts of the same type, the system automatically generates a health profile. When a user downvotes an answer, the linked Facts lose half their confidence score; three consecutive downvotes on the same Fact type trigger a full purge. Every answer is scored asynchronously by an LLM-as-Judge across four dimensions — accuracy, completeness, faithfulness, conciseness — and every tool call is wrapped in exponential-backoff retry with jitter and circuit-breaker-protected model routing. The React frontend streams the Agent's thinking process in real time so you can watch it decide which tool to call and why.
How we built it
The backend uses Java 17 on Spring Boot 3.5.7 with PostgreSQL + pgvector (HNSW), Milvus, Redis, and RocketMQ. The ReAct loop is a hand-rolled execution engine that injects tool definitions into the system prompt, parses XML tags from LLM responses with regex, executes the requested tool through a unified AgentTool interface (treating KB search and MCP tools as equal citizens), feeds the observation back into the message list, and repeats until the LLM produces a final answer or hits the 10-iteration cap — at which point it forces a summarization from gathered evidence rather than failing silently. The memory system runs on the Strategy pattern: users can switch between sliding_window, summary_compression, and semantic modes via a single config key. The semantic strategy chains together a FactExtractionService and EpisodeExtractionService (both LLM-powered) with a MemoryRetrievalService that runs pgvector cosine-similarity search over stored embeddings and a MemoryInjector that formats the results into prompt fragments. The multi-channel retrieval engine fans out vector-global and intent-directed searches in parallel, fuses results with RRF reciprocal rank fusion (k=60), reranks, and truncates to a token budget. The model routing layer implements a three-state circuit breaker (CLOSED → OPEN → HALF_OPEN) with CAS atomic transitions and a single-probe semaphore to prevent thundering-herd behavior on recovery. The frontend is built with React + Vite + TypeScript + Tailwind CSS, consuming SSE streams with separate channels for thinking deltas and content deltas so the agent's internal reasoning unfolds visibly without blocking the answer stream.
Challenges we ran into
The hardest problem was ReAct loop divergence — LLMs would sometimes call tools repeatedly without converging, burning through the iteration budget with no answer. We solved this with a hard 10-iteration cap plus a forced summarization fallback that injects "you've reached the limit, give your best answer now" into the prompt, ensuring the user always gets a response. Parsing tool calls reliably from LLM output was another challenge; different models format their blocks inconsistently, so we built a custom regex parser that tolerates whitespace and formatting variations rather than depending on any model-specific function-calling API. Memory quality was the third frontier — raw LLM-extracted Facts are noisy, contradictory, and sometimes flat-out wrong. We introduced a confidence scoring system where user upvotes boost Fact confidence by 1.2× and downvotes halve it, plus automatic merging when 10+ Facts of the same type accumulate and automatic purging when 3+ Facts of a type fall below 0.3 confidence. On the infrastructure side, external LLM APIs fail in bursts — without a circuit breaker, every request would hang for seconds. The HALF_OPEN state with a single probe request was critical: it lets the system test recovery without risking a flood of requests to a still-failing model. Finally, streaming the Agent's internal monologue to the frontend without jank required careful SSE channel design — thinking content and final answer content travel on separate logical channels so the UI can render them independently.
Accomplishments that we're proud of
We shipped an AI doctor that genuinely improves with use — the semantic memory pipeline extracts, deduplicates, and retrieves personal health facts, and the feedback loop directly shapes memory quality through confidence scoring and automatic cleanup, which most RAG demos do not even attempt. The ReAct Agent treats knowledge base search and external MCP tools as first-class equals, making autonomous decisions about which tool to call and when to stop, rather than following a hardcoded retrieval-then-answer script. The entire system is production-hardened — circuit breakers, exponential backoff with jitter, Redis checkpointing for Agent loop recovery, and asynchronous four-dimensional quality evaluation — not just a hackathon prototype. The feedback-to-memory linkage (downvote → Fact confidence halved → repeated downvotes → auto-purge) closes the loop between user satisfaction and knowledge quality in a way that makes the system learn from its mistakes. And the frontend makes the Agent's thinking visible — users see each tool call, each observation, and the final synthesis unfold in real time, which builds trust in a domain where trust is everything.
What we learned
ReAct Agent autonomy is powerful but needs guardrails — without iteration caps and forced summarization, loops diverge, and in a medical context a silent failure is unacceptable. The XML tag approach turned out to be simpler and more portable than SDK-level function-calling APIs, especially when targeting multiple model providers — it relies only on text parsing, with no vendor lock-in. Circuit breakers are not optional in production LLM systems; model APIs fail in correlated bursts, and the HALF_OPEN single-probe pattern is the difference between graceful degradation and total outage. Memory quality is fundamentally an OODA loop — Extract, validate through user feedback, refine — and without the feedback linkage (likes/downvotes → confidence → merge/purge), extracted Facts accumulate noise until the memory system becomes a liability rather than an asset. Streaming the Agent's thinking process in the UI turned out to be as important for trust as the answer itself — users tolerate latency when they can see the system working through their problem step by step.
What's next for BayMD
Multi-modal medical image analysis is the obvious next step — letting users upload X-rays, CT scans, or lab report photos for the Agent to interpret alongside text symptoms. A voice interface would make the experience far more natural for patients who struggle with typing. We want to explore federated health profiles that let a user's memory travel with them across clinics while preserving privacy through on-device encryption. The semantic memory pipeline is ready for proactive health alerts — when the system notices a pattern in accumulated Facts (e.g., recurring symptoms at specific intervals), it could nudge the user to schedule a checkup. Finally, we plan to expand beyond Chinese-language medical scenarios to add multi-language support, and integrate with wearable data streams (heart rate, sleep, activity) to enrich the health profile with continuous physiological signals rather than just episodic conversation data.
Log in or sign up for Devpost to join the conversation.