Legacy Avatar: The Tacit Knowledge Bridge

Don't just archive their files, archive their wisdom. Legacy Avatar uses Gemini Live to capture expert reasoning into interactive AI mentors, ensuring successors inherit experience, not just documents

Comment

Inspiration

The idea for Legacy Avatar was born from a frightening statistic: in the software industry, the average employee tenure has dropped to just 2.1 years. High turnover means systems are constantly being managed by people who didn't build them.

We realized that tech companies are excellent at archiving Explicit Knowledge ()—user manuals, process docs, and "How-To" guides. But they are terrible at capturing Tacit Knowledge ()—the "gut instinct," the unwritten rules, and the "weird tricks" that keep things running.

We saw a Senior Architect leave our previous company. He left behind perfect documentation. But when the website went down a month later during a big sale, we realized the real knowledge—why we have to restart the main server specifically on Tuesday nights, or why the billing system crashes if you upload a file with a specific name—had walked out the door with him. We built Legacy Avatar to ensure that this hidden wisdom is inherited, not just archived.

What it does

Legacy Avatar: The Tacit Knowledge Bridge is an autonomous AI system that acts as a continuous "Digital Biographer" for your key tech experts. It doesn't just read code; it captures intent.

  1. The Biographer (Active Capture): Using Gemini Live, the system joins team meetings and problem-solving sessions. It acts like a curious new employee, noticing when an expert does something unusual and interrupting to ask, "Wait, why did you just ignore that error warning? The manual says to stop, but you clicked 'Continue' anyway."
  2. The Archivist (Synthesis): It processes these interactions to extract Thought Signatures—Gemini 3’s native representation of a reasoning pattern. It maps these into a "Wisdom Graph" that links common problems to these "secret" solutions.
  3. The Avatar (Interactive Mentor): Years later, a new hire can talk to the "Avatar" of that expert. The Avatar uses the stored Thought Signatures to answer new, unseen problems using the logic style of the original creator.

How we built it

We architected the solution as a three-stage pipeline leveraging the specific multimodal capabilities of Gemini 3.

1. The Voice Interface (Gemini Live)

We used the Gemini Live API via WebSockets for instant, two-way communication. The critical component here was the interruption_handling. We tuned the model's system instructions to be investigative rather than passive, specifically trained to ask "Why?" whenever a human does something manual.

2. The Wisdom Extraction Engine

This is the core innovation. We defined a transformation function to convert raw audio transcripts () into a portable knowledge artifact.

We utilize Gemini 3 Pro with thinking_level="HIGH" to analyze the transcript and separate the signal from the noise. We then generate a Thought Signature (), which acts as a saved snapshot of the expert's "state of mind" during that specific session.

3. The Retrieval System

We used Google Cloud Vertex AI Vector Search to store the context. However, unlike standard search tools, we don't just find text. We inject the serialized thought_signature to "prime" the model, effectively "loading" the personality and logic of the expert before it answers the user.

Challenges we ran into

  • The "Yes Man" Problem: Early versions of the AI were too polite. If the expert gave a vague answer like "It's just a glitch," the AI accepted it. We had to program the "Biographer" persona to be a "Socratic Questioner," challenging the expert to explain the root cause (e.g., "Is it a glitch in the software or a connection problem?").
  • State Management: Handling Thought Signatures was tricky. We had to figure out when to apply them. Applying a "Sales Expert" signature to a "Technical Bug" problem resulted in confused answers. We solved this by categorizing signatures with a smart classifier.
  • Latency vs. Depth: We wanted the deep reasoning of Gemini 3 Pro but the speed of Gemini Live. We compromised by using Live for the interview (Stage 1) and Pro for the deep analysis later (Stage 2).

Accomplishments that we're proud of

  • The "Intuition" Demo: We successfully demonstrated the system solving a problem it had never seen before. We taught the Avatar a specific trick about fixing a frozen login screen. We then asked the Avatar about a different screen that was freezing. It correctly applied the expert's "trick" to solve it, proving we captured the logic, not just the memory.
  • Seamless Voice Interruption: Achieving a natural "back-and-forth" conversation where the AI could interrupt the user to ask for clarification without breaking their flow.
  • Full Multimodal Storage: The system can pull up the exact timestamped audio clip of the original expert explaining the fix, adding a layer of trust to the answer.

What we learned

  • Wisdom Information: We learned that "Tacit Knowledge" is almost always conditional. Standard search tools fail here because they find facts, not these specific conditions.
  • The Power of Thinking Levels: Gemini 3's ability to "think" before speaking was crucial. It allowed the model to figure out why a user skipped a step even if they didn't say it out loud, just by analyzing the situation.

What's next for Legacy Avatar: The Tacit Knowledge Bridge

  • Coding Tool Integration (Project "Codebase Whisperer"): We plan to integrate with coding tools so the "Avatar" can see the code the new employee is writing, popping up with warnings like "Warning: Bob advised against changing this line because it breaks the login page."
  • Passive Shadowing: Moving beyond "Interviews" to a background process that silently watches work sessions, asking questions only when it detects someone doing something new or risky.

Built With

Share this project:

Updates