Butler AI - Living Document Assistant

About the Project

Inspiration

As a computing university student, I've lost count of how many times I've scrolled through endless chat histories trying to find a single decision I made three weeks ago. "Did we choose React or Vue?" "What was the budget again?" "Why did we reject that approach?"

Traditional AI assistants have a massive problem: context drift. You have a brilliant brainstorming session, the AI helps you explore ideas, but by message 50, nobody remembers what was decided at message 12. The conversation becomes a messy blur. When working on technical projects with multiple components - APIs, databases, frontend frameworks, deployment strategies - keeping track of every decision is nearly impossible.

This confusion has cost me grades. It has cost me deadlines. And frankly, it's cost me sanity.

That's why I built Butler AI - a conversation assistant that doesn't forget. Every decision, every option considered, every plan made gets automatically documented in real-time. And if you want to explore a "what if" scenario? Just branch the conversation like Git branches, without losing your original path.The best part? Anyone can use it. From a grandparent planning a family reunion to a developer architecting a microservices system. The interface adapts to you, not the other way around.

What it doesButler AI is an autonomous workspace that turns spoken or typed development conversations into structured, living documentation in real time. While you chat, a background worker called Janitor AI continually watches the stream, cuts out the fluff like filler words and casual greetings, and updates a dynamic Markdown document on your screen every 3 seconds. It automatically shapes itself around your topic—swapping layout sections between things like system architectures and roadmaps for coding, or budgets and itineraries for general tasks.
At the same time, it builds a visual evolution timeline on your sidebar where every user message is a clickable node. Each node captures a complete snapshot of your messages, document state, and active files. You can click any past point to completely restore the workspace to that exact moment or branch off into parallel paths with completely unique branch colors. On top of that, the second it catches a request for code, a dedicated panel called CodeSpace automatically slides open with a multi-file explorer and live HTML/CSS preview. How we built itThe frontend uses a clean three-panel layout built with React and Tailwind CSS. The left sidebar handles the multi-session chat list and maps out the branching timeline nodes. The center panel is the main chat stream with file attachments and voice controls, while the right panel houses the Living Document canvas and the slide-out CodeSpace panel. The bottom is anchored by a collapsible Activity Log that tracks background agent actions in real time.
To stop the AI from forgetting core goals or drifting off-topic, we avoided dumping whole data dumps into the LLM context. Instead, we designed a defensive memory window that strictly feeds the AI the last 6 messages alongside the current distilled state vector of the Living Document. We also wrote a background loop running a 3-second cron-style check. This lets the automated Janitor AI worker analyze current dialogue turns, notice when an active problem has been resolved, and update the document data structure autonomously.

Challenges

we ran into Challenge 1: Voice Echo & Loopback - In Secretary Mode, the AI needs to listen continuously and update the document, but only respond when addressed by name. Early on, when the AI spoke or triggered a notification, the microphone would pick it up and transcribe it again, causing an infinite loop. The solution was isolating the microphone input stream to mute local audio detection while the system is actively outputting speech or playing audio confirmation.
Challenge 2: The Stale Closure - The onresult handler in Web Speech API captured old state variables, causing push-to-talk to work only once. Moving to useRef for real-time state fixed it. Challenge 3: Recursive Rendering - The branching timeline initially caused infinite recursion. Adding a visitedBranches Set to track already-rendered branches solved the "Maximum call stack exceeded" error. Challenge 4: Document Updating Race Condition - Timeline nodes weren't capturing the latest document state because the 3-second auto-update loop and rapid user inputs would trip over each other, causing asynchronous LLM calls to save out of order. Adding an 800ms debounce delay right after a user stops typing/speaking ensured the AI finished generating its thoughts before the state was deep-cloned via JSON serialization and committed safely as a node.
Challenge 5: Word Export Formatting - Word renders HTML inconsistently. PDF works perfectly, so I kept Word as "best effort" and recommend PDF for professional documents.
Challenge 6: The Export Dropdown - The menu would disappear when moving the mouse down. The fix? Remove the gap between button and menu, and keep the menu open when hovering over it, not just the button. Challenge 7: Intent Parsing for CodeSpace - We struggled to make the application accurately differentiate between a user talking about a language and a user actually asking for functional files. If someone casually typed the word "Python," the CodeSpace panel would slide open inappropriately. We solved this by creating strict pattern-matching logic and intent-detection keyword filters so CodeSpace only triggers for true functional programming layout requests.

Accomplishments

I am incredibly proud of making a Git-style branching engine feel entirely natural and hidden from the user. Managing the complex state behind the scenes—deep cloning your entire conversation history, specific file trees, and custom document categories down to a single clickable node—was tough, but it runs perfectly smoothly. Seeing the background Janitor AI agent automatically shift a problem from "Active" to "Resolved" in the sidebar document exactly 3 seconds after you say "oh wait, that fixed it" feels like absolute magic.

What I Learned

Building Butler AI taught me more than any classroom lecture:Voice is hard. The Web Speech API is powerful but finicky. Echo cancellation, wake word detection, and handling "um" and "uh" without breaking the document required constant iteration.State management is everything. With branching timelines, every message needs to be a complete snapshot. I learned to think in terms of "nodes" and "branches" rather than linear history.AI prompts are an art. The background worker—my document-updating agent—needed dozens of prompt iterations to learn what to keep (decisions, budgets, deadlines) and what to ignore ("hey", "thanks", "you know").

Real-time is addictive. Streaming responses and live document updates create a magical experience. Once users see text appear word-by-word while the document updates simultaneously, they never want to go back to "click and wait." Markdown is underrated. Using Markdown as the document format made exporting to PDF, Word, and plain text trivial. It also gave users familiar formatting without a complex rich text editor.

What's next for Butler AI - Living Document Assistant

To ship a flawless core application, we left a lot of advanced features strictly out of scope for this release. Next up, we want to build user accounts and cloud synchronization so your workspace state carries over to any machine. We also plan on introducing real-time multi-user collaboration so multiple team members can branch and edit the same conversation stream together. Finally, we want to integrate native version control connections like GitHub, letting the Janitor AI commit your auto-generated documentation straight into a real project repository.