SayDo

Record once by voice. Your app works with every AI agent. Amazon Q-style agents for any product, built by talking, powered by NVIDIA Nemotron on Nebius Token Factory.

Comment

Inspiration

Using Amazon Q inside AWS, I watched an assistant actually do things in the console instead of just explaining them. Every product wants that, but building an agent that safely takes actions inside an app takes an engineering team: wiring APIs, writing tool definitions, handling permissions. Small teams like ours (DEHIX, a Web3 talent platform) can't justify that. So we asked: what if a product owner could build their app's agent the way they'd train a new teammate, by doing each task once and explaining it out loud?

What it does

SayDo makes any web app agent-ready by voice.

  1. Discover (Tavily): SayDo reads your product docs with Tavily and suggests which skills your agent needs, plus the business rules hidden in the docs, with a coverage meter.
  2. Teach by doing and talking: Add one script tag, turn on Record, and do a task in your app while narrating it (Wispr Flow or the mic). SayDo captures the app's own API calls (never cookies or tokens) together with your narration.
  3. Compile (Nemotron): A deterministic value tracer links calls (e.g. the project id from "create project" feeds "invite member"), and Nemotron turns recording + narration into typed skills: inputs, constants, outputs, dependencies.
  4. Rules by voice: "Budget can never exceed 10 lakh." "Never invite admins through the assistant." Spoken rules become structured policies enforced in code, not just in the prompt.
  5. The agent acts: "Create Website Revamp with a 2 lakh budget and invite Arpit as editor." The agent plans, chains skills, shows a confirmation card before every write, executes in the user's own browser session, and verifies the result by reading it back.
  6. Grounded answers: Questions without a skill are answered from live docs through Tavily, with sources.
  7. Any agent: The same skill library is exported as an MCP server, so any MCP client can operate the product.

How we built it

  • NVIDIA Nemotron on Nebius Token Factory (OpenAI-compatible API) with model routing: a fast model compiles skills and parses spoken rules; Nemotron Super runs the tool-calling loop; Nemotron Ultra plans multi-step requests and powers evals. Native tool calling with a JSON-mode fallback behind one interface.
  • Tavily: extract + search power Discover (docs to skill and rule suggestions), ask_docs (grounded answers with sources) and Drift Watch (changelog to skills at risk).
  • Stack: pnpm + Turborepo monorepo; Next.js 14, Tailwind, shadcn/ui (Studio, landing, DEHIX-lite demo app); Express + TypeScript engine; embeddable SDK (Vite, Shadow DOM); Supabase Postgres; zod; MCP TypeScript SDK.
  • Safety: no headers or cookies stored, sensitive fields redacted, confirmation on every write, policies checked on server and client, tool results and web content treated as untrusted data, full run audit log.
  • Evals: 20 natural-language tasks (multi-step chains, policy blocks, docs questions) scored per Nemotron model. Results: [TODO]

Challenges we ran into

  • Linking calls reliably: instead of letting the model guess how requests connect, we trace values deterministically and let Nemotron only name and describe the links.
  • Dependable tool calls: a per-model health check plus a JSON-mode fallback. [TODO: specifics]
  • Safety: moving rules out of the prompt into a policy engine the model can't bypass.

Accomplishments that we're proud of

  • A product owner goes from zero to a working, rule-abiding in-app agent by talking, with no integration code.
  • Multi-step tasks with confirmation and read-back verification.
  • One skill library powering both the in-app agent and MCP.

What we learned

  • Narration is the missing signal: network traffic shows what happened, and the owner's words explain which values are inputs and which rules apply.
  • Open models on Token Factory are capable enough for real agent loops when each task is routed to the right model size.

What's next for SayDo

  • WebMCP export so browser agents call declared tools instead of guessing clicks
  • Self-healing skills: replay checks and automatic re-recording when an app changes
  • A marketplace of verified skill packs (Tally, Zoho, Shopify, GST and other Indian business portals)
  • Hindi and Hinglish voice teaching

Built With

Share this project:

Updates

Submission history