Saul Goodman

• GIF

• 

  Architecture

Inspiration

Tax is one of the few things that affects every single person on the planet — and yet it remains one of the most anxiety-inducing, confusing, and inaccessible experiences in modern life.

64% of people report genuine stress during tax season. Only 25% feel confident filing on their own. In India, 8.56 crore ITRs were filed in FY 2024-25 — most without professional help, many with the wrong form, many attracting notices they didn't deserve. In the US, 18 million e-filed returns are rejected every year. $311 billion is overpaid annually — not from fraud, but from confusion.

And for the 32 million Indians living abroad? It's worse. Two tax systems, two sets of deadlines, DTAA treaties nobody told them about, NRE accounts that silently become taxable, and an RNOR window worth crores that almost nobody knows exists.

The tools available today are either cold portals, expensive CAs, or chatbots that answer one question and forget you exist.

We wanted to build the friend everyone deserves — the CA best friend who explains DTAA over a conversation, celebrates when you find a deduction, drops a meme when you need to laugh, and never forgets your situation. We called him Saul Goodman. Better call Saul.

---

What it does

Saul Goodman is a voice-first, multimodal AI tax optimization agent that works across India and US tax jurisdictions simultaneously.

Conversational tax intelligence — Users talk to Saul naturally via Gemini Live. Saul gathers context through a warm, human conversation — no forms, no dropdowns. It identifies their jurisdiction, income sources, filing status, and tax concerns across turns, then routes queries to specialist agents that scrape authentic public tax sources in real time.

Multi-agent research architecture — Four specialist agents built on Google ADK handle research in parallel: CAClub India and TaxMii for Indian tax law, TurboTax blog and TaxProfBlog for US tax intelligence. Each agent scrapes expert articles, forums, circulars, and news. Gemini 3 Pro is the central root agent that aggregates all four responses and synthesizes a final cited answer via A2A protocol.

Document vision — Users point their phone camera at any tax document ITR-1, W-2, Form 16, 1099. The camera feed streams via WebRTC MediaStream. Client-side edge vision tracking locks onto the document and snaps a high-resolution still frame. That frame goes to Google Document AI for structured extraction, with Gemini Vision as a fallback for unstructured or handwritten documents. The extracted data flows into Cloud SQL, gets injected into a prompt via the Context Injector, and feeds Nano Banana 2 — a Vertex AI model — which powers the Recomputing HUD: live sliders, what-if tax scenarios, and real-time SQL-backed math.

Personality and engagement via KLIPY — Every emotional beat in the conversation triggers contextually relevant GIFs, memes, and stickers via the KLIPY API. A big portfolio reveal gets a "we are not the same" meme. A found deduction gets a celebration GIF. A deadline warning gets a panic sticker. Tax stops feeling like a chore and starts feeling like a conversation with someone who actually cares.

Long-term memory — Saul never forgets. Vertex AI Memory Bank stores long-term user context across sessions. Every user interaction maps relationships between users, income sources, jurisdictions, tax forms discussed, and unresolved queries — enabling genuinely personalized continuity across every return, every year.

---

How we built it

Voice layer — Gemini Live SDK handles real-time bidirectional audio with interrupt-awareness. The system prompt encodes Saul's full personality framework: CA Best Friend Energy, four interaction modes (Hype, Nudge, Simplify, Milestone), and strict conversational rules — one question per turn, earn the next question, never lead with jargon.

Agent architecture — Google ADK orchestrates four specialist sub-agents. Each agent is scoped to its source domain and returns structured results with source URLs and relevance scores. Agents communicate via the A2A protocol. The root Gemini 3 agent aggregates all four responses, surfaces conflicts rather than silently resolving them, and produces a final answer with inline citations.

Geo routing — A lightweight Geo Router sits between the chat interface and the memory layer. It detects the user's jurisdiction from conversational context and routes accordingly — India agents, US agents, or both for cross-border scenarios.

Memory stack — Vertex AI Memory Bank handles cross-session long-term recall. A Knowledge Graph in Obsidian stores typed relationships between the user's tax entities: income sources, jurisdictions, forms discussed, unresolved queries, and proactive nudges that haven't been delivered yet. This enables the system to surface insights like "this user asked about DTAA twice and never filed Form 67" as a proactive conversation opener.

Document vision pipeline — WebRTC MediaStream streams the phone camera feed to the browser. Client-side edge vision tracking locks onto the document and captures a high-resolution still frame. The frame goes to Google Document AI for extraction. Gemini Vision handles fallback cases. Extracted structured data lands in Cloud SQL. The Prompt + Context Injector combines it with the user's session context and feeds it to Nano Banana 2, a Vertex AI model that powers the Recomputing HUD — live tax sliders, what-if scenarios, and SQL-backed math rendered in real time.

KLIPY integration — Every Gemini response includes a structured <klipy> JSON block with content_type, query, and intensity. The frontend parses this tag, strips it from the voice text, and triggers the appropriate KLIPY API call (GIF, meme, sticker, or clip) to render alongside the audio response.

Infrastructure — Fully deployed on GCE VM. Vertex AI for memory and Nano Banana 2 model serving and Cloud SQL for document extraction output

---

Challenges we ran into

Query precision vs. recall tradeoff — Generic queries to tax sources return hundreds of irrelevant articles. Too-specific queries miss relevant precedents. Building the two-layer query system — scenario templates plus Gemini refinement — took significant iteration to produce queries that returned authentic, citable content without hallucinated sources.

Keeping the voice natural under a complex system prompt — Early versions of Saul sounded like a compliance officer. Getting the personality to hold consistently across emotionally varied situations — celebrating a found deduction vs. delivering bad news about a tax liability — required extensive prompt engineering and explicit anti-patterns ("never say Certainly!, never ask two questions at once").

ORB tracker jitter(Not used) — The initial JavaScript-based frame analysis was too jittery for a credible AR experience. Offloading to the Python backend with proper ORB feature matching on a pre-cached template introduced the necessary precision, but added a latency challenge. Solved by running tracking and voice pipelines asynchronously — the SVG overlay locks at 15fps independently of the Gemini voice response cycle.

Client-side document tracking stability — Running edge vision tracking entirely in the browser meant working within tight performance constraints. Getting the frame lock stable enough to produce a clean still image for Document AI required careful frame selection logic — only snapping when the document edges were stable for at least 3 consecutive frames.

Cross-jurisdiction reasoning conflicts — India and US tax law occasionally point in opposite directions for the same user action. The root aggregator needed explicit conflict resolution logic: surface the conflict to the user rather than choosing one silently, with the specific treaty article or section that governs the India-US interaction.

KLIPY timing with voice — GIFs firing too early or too late break the emotional rhythm. Getting the <klipy> block parsed before the Gemini voice stream completed required streaming the JSON tag first in every response and processing it client-side

---

Accomplishments that we're proud of

The RNOR insight delivery — Watching a user react to learning about the 2-3 year RNOR window in real time — "You mean all that US income is tax-free in India for two years?!" — is the moment the system was built for. Saul delivering that reveal naturally, with the right pacing and the right meme, is something no existing tax product does.

The Recomputing HUD — The live what-if slider experience — where changing one input instantly recomputes your tax liability across jurisdictions via Nano Banana 2 — makes abstract tax math tangible and interactive in a way no tax tool has attempted before.

The personality holding at scale — Saul maintaining CA Best Friend Energy across a 10-turn conversation covering three different tax concepts, two jurisdictions, and a full range of emotional beats — from anxiety to relief to celebration — without breaking tone once.

---

What we learned

Personality is not a feature — it's the product. The KLIPY integration and conversational tone design drove more engagement than any technical capability. People will use a tool they enjoy using. Tax tools have ignored this entirely.

Long-term memory changes the relationship. The moment Saul remembered a user's income sources from a previous session and opened with a proactive nudge — rather than starting from scratch — the interaction felt genuinely different from every other AI tool. Memory is what turns a chatbot into a companion.

The document pipeline taught us that client-side vision tracking, when paired with a reliable cloud extraction layer like Google Document AI, is a practical and deployable approach for document-heavy domains — no custom model training required.

---

What's next for Saul Goodman

More countries — The agent architecture is designed to scale horizontally. Adding a country means adding two agents (forum source + official source), injecting the relevant tax documents via Page Index, and mapping new DTAA treaties. UK, Canada, Australia, and UAE are the immediate next targets — covering the largest NRI corridors globally.

CA booking integration — When Saul reaches the boundary of what an AI should handle alone — complex litigation, trust structures, estate planning — it should seamlessly connect the user with a verified CA. The Knowledge Graph context transfers to the CA, so no re-explanation required. The CA gets a pre-built client brief from Saul's memory.

Proactive tax calendar — Saul knows your jurisdiction, income sources, and filing history. It should push reminders before every relevant deadline — advance tax dates, Form 67 filing window, TRC renewal, FBAR deadlines — as a voice notification, not a push alert.

Legal precedent layer — Integrating Supreme Court orders and ITAT tribunal rulings into the root agent's citation synthesis would give Saul legal precedent-level authority for contested tax positions — something no consumer tax tool has ever offered.

---

Built With

• a2a
• adk
• agent
• documentai
• geminilive
• google-cloud
• klipy
• knowledge-graph
• nano-banana-2
• sdk
• vertex-ai
• webrtc