Ideate

About the project

Inspiration

The best way I ever worked through anything hard, whether an algorithm or a budget, was sitting next to someone with one sheet of paper. They'd sketch it, I'd ask "wait, why does that work?", they'd hand me the pencil, then we'd open a laptop and check the numbers. Doing it alone loses all of that. You've got a drawing app in one tab, a spreadsheet in another, a code editor, notes somewhere else, and a chatbot that can't see any of them.

Ideate puts the paper, the laptop, the spreadsheet, and the friend on the same desk. The friend is called Buddy Buddy.

What it does

Ideate is a 3D desk for thinking through a problem with Buddy Buddy, an AI partner that can see everything you're working on. Click the whiteboard, computer, journal, or spreadsheet and it opens as a full-size 2D tool. Buddy Buddy reads all four, explains what you select, and proposes edits to any of them.

For money: hit "Connect mock bank" and Ideate pulls accounts and a month of transactions from Capital One's Nessie sandbox into a spreadsheet ledger with totals and a category breakdown. Then ask Buddy Buddy where your money goes, have it sketch your cash flow on the whiteboard, or turn expenses you jotted in the journal into a budget with live formulas. Every edit is reviewable and undoable.

The same loop works for anything else you'd sit down at a desk for: sketch an algorithm and have it turned into runnable Python, work through a math derivation with rendered LaTeX, or just take notes while Buddy Buddy fills in what you're missing.

You can also talk to it. Voice mode uses ElevenLabs for live transcription and speech, and Buddy Buddy draws or types while it talks. Interrupt it mid-sentence and the pen stops where it is; the partial work stays as an undoable checkpoint.

Python runs sandboxed in the browser with a line-by-line debugger. Everything saves locally; no account needed.

How we built it

Next.js 16, React 19, and TypeScript. The desk is Three.js via React Three Fiber; Excalidraw for the whiteboard, CodeMirror for code and notes, a custom spreadsheet with its own formula engine (SUM, AVERAGE, MIN, MAX, COUNT, ROUND, cell references, ranges), Dockview for split/tab/float layouts, Zustand for state, IndexedDB for persistence.

Buddy Buddy is Gemini (gemini-3.8-flash) running server-side. It gets an immutable snapshot of the workspace plus your selection and an optional board image, streams text back, and calls tools to edit code, draw on the board, edit notes, edit cells, or import bank data. Each call is validated with Zod and becomes a proposal; only the proposal executor can mutate documents.

Nessie is called only from the server. The first connect seeds a demo customer with three accounts and about 45 transactions; the sheet is then built deterministically from the normalized data, so Buddy Buddy never sees raw Nessie JSON. If the key is missing or Nessie is down, a bundled snapshot loads instead and the sheet says so.

Python has its own origin with a strict CSP and a fresh Worker per run. The debugger uses WebAssembly JSPI to pause before each line.

Tested with 133 Vitest tests and 19 Playwright tests against real Pyodide and real Excalidraw.

Challenges we ran into

Nessie behaves differently from its docs. Purchase and deposit amounts get truncated to whole dollars while bills keep cents. POST /transfers rejects the documented payee_id and medium fields, so we post one leg per account and infer direction from the description. Some empty collections return 404 instead of an empty list. Customers can't be deleted, so a half-seeded customer has to be repaired by deleting its accounts and re-seeding. And it went down during the event, which is why the offline fallback exists.

Our formula engine has no SUMIF, so the by-category block uses explicit cell references generated at import time. Every formula has to evaluate clean or the import is rejected.

Making AI edits safe was most of the work overall. Buddy Buddy proposes a change, you keep typing, and the change lands on a document that no longer exists. Revision checks, operation-ID deduplication, and a conflicting-undo flow that shows you what it's about to replace all came out of that.

Gemini's tool loop silently breaks if you drop thought signatures on continuation. We also hit 503s and rate limits and had to decide what to retry.

Voice was the hardest part. A chat response is one blob of text; a spoken explanation with drawing is two output streams that have to run at the same time and stop at the same time. Our first attempt had Buddy Buddy finish talking, then draw, which felt like watching someone dictate and then hand you a fax. So we broke every response into teaching steps: a teach_step tool call carries one short sentence of speech and at most one workspace edit. The model generates steps sequentially, but playback overlaps, so step two's audio can be generating while step one is still being drawn. A playback coordinator owns each step: it waits for the browser's actual audio-start event before it begins revealing strokes or editor lines, tracks both outputs to completion, and holds a single AbortSignal. Interrupting fires that signal, which halts speech and pen together, freezes the visible partial edit, and commits it as one undoable checkpoint after re-validating against the current document. Late audio chunks, transcripts, or previews from the cancelled step are dropped by turn ID so nothing can revive it. The reveal itself is a provisional preview: excluded from saves, exports, Python runs, and the screenshots Buddy Buddy sees, until the step completes and becomes one normal revision. Sentence-level pacing works; word-level sync still needs tuning on a real device.

Accomplishments that we're proud of

Clicking one button and watching a real Nessie customer turn into a ledger with working totals, then asking Buddy Buddy "where does my money go?" and getting an answer grounded in the actual cells.
The voice interruption model. Stopping Buddy Buddy mid-stroke and having the half-finished drawing stay put, undoable, felt right the first time it worked.
No unsandboxed exec anywhere. The runner never sees workspace data, bank data, or secrets.
A table-flip animation on "Clear all workspace data." Reduced-motion users get a static confirmation.

What we learned

The hard part of an AI collaborator isn't the model. It's the contract around it: what Buddy Buddy can see, what it can change, how you take the pencil back, and what happens when its answer arrives late. Once that contract was solid, adding the spreadsheet, the bank import, and voice was mostly one Zod schema and one executor each.

What's next for Ideate

Live collaboration, so two people (or a student and a tutor) can share one desk and one Buddy Buddy.
Additional programming languages beyond Python in the computer.
Mobile support past the current simple view, including touch drawing on the whiteboard.
Real bank linking in place of the mock customer, with multiple accounts and more than one month of history.
SUMIF and multiple sheets, so budgets can grow past a demo.
Recurring bills, subscription detection, and a cash-flow forecast Buddy Buddy can draw on the board.
Public deployment with separate app and runner origins and per-user quotas.