DuckBuddy

Inspiration

Many websites are difficult to understand and navigate. That matters most when a confusing form or instruction can affect someone’s benefits, finances, health, or access to services. DuckBuddy aims to make the web easier to use while keeping people in control.

What it does

DuckBuddy is a Chrome extension with a friendly mascot companion. Users describe what they want to do, and DuckBuddy offers clear, step-by-step guidance, explains confusing fields, and highlights where to look. It also includes readability tools, optional voice interaction, screenshot-based help, and support for interactive PDFs. Users perform all website actions themselves.

How we built it

Chrome Extension Manifest V3 and Chrome APIs give DuckBuddy access to the current tab when the user activates it. The extension reads limited page structure, shows the companion, and applies reversible visual changes.
TypeScript and JavaScript power the extension and backend. Zod validates page context and model responses so guidance can only point to known page elements.
Node.js runs the local backend and keeps provider credentials out of the browser extension.
Gemini interprets the user’s goal and reviewed page context to generate structured, plain-language guidance. A Groq fallback path is built for Gemini provider failures and requires its own valid API key.
Mastra organizes the guidance workflow, pauses while the user takes action, and resumes the task for the next step.
Bright Data helps DuckBuddy find supporting public instructions when a user asks about requirements or a field lacks useful on-page help. It searches public sources, then the backend reads a bounded number of approved pages and extracts relevant passages. DuckBuddy can show source links, excerpts, and retrieval times, helping the companion ground explanations in traceable information instead of relying only on the model.
ElevenLabs provides optional speech transcription and spoken guidance, so users can interact by voice and hear responses aloud.
Tiger Data is an optional PostgreSQL-backed store for saved task summaries, reusable guidance, cited public excerpts, and consented, coarse help events. Full task checkpoints remain local.
PDF.js lets DuckBuddy display and interact with supported PDFs in the browser. PDF content and entered values stay local; the helper sends selected labels and questions for guidance.
Playwright and Node’s test tools help check browser interactions, backend validation, privacy boundaries, and task flows. ## Challenges we ran into
Keeping guidance accurate across varied website layouts and changing pages.
Protecting privacy while giving the companion enough context to be useful.
Grounding explanations in relevant sources and showing where that information came from.
Keeping visual changes reversible and ensuring the companion does not cover important content.
Handling provider errors without interrupting use of the original website. ## Accomplishments that we're proud of
Built a page-aware companion that guides users through websites one step at a time.
Connected model guidance to validation, task continuity, and optional public-source retrieval.
Added reversible readability tools, voice interaction, reviewed screenshot help, and interactive PDF support.
Kept navigation, form entry, and submission under the user’s control. ## What we learned Useful accessibility support needs to adapt to different websites, protect user privacy, and make its guidance understandable and traceable. External sources can strengthen an explanation, but the person should always be able to review the guidance and decide what to do. ## What's next for DuckBuddy
Test with a broader range of websites and users with varied accessibility needs.
Improve source retrieval and make provider fallback behavior more reliable.
Expand document support while keeping private values local.
Complete optional voice and data-storage setup.
Prepare the backend for secure multi-user use.