Butterfly Sciences

The AI research workbench where every claim is cited, every result is runnable, and a council of models checks the answer.

Comment

Butterfly Sciences

The AI research workbench where every claim is cited, every result is runnable, and a council of models checks the answer.

Inspiration

Every researcher I know uses AI chat tools the same way: ask, get a confident answer, then spend twice as long checking whether the citations are real. LLMs are brilliant research assistants and terrible research instruments: they hallucinate references, they answer alone with no second opinion, and their output dies in the chat window instead of becoming a paper, a dataset, or an experiment.

Three ideas collided into Butterfly Sciences. First, Andrej Karpathy's llm-council experiment, where multiple models anonymously peer-review each other's answers, felt like exactly the kind of adversarial rigor scientific questions deserve. Second, grounding: what if a model could only cite from a numbered list of real papers fetched from the actual literature? Third, deliverables: research isn't done until it's a cited report you can compile in LaTeX or a plot you produced from runnable code. I wanted one workbench where research takes wing: literature in, verified and reproducible science out.

What it does

Butterfly Sciences is a full research workbench that runs on your own API keys (OpenAI, Anthropic, OpenRouter, NVIDIA NIM):

  • Research & Deep modes: search Semantic Scholar, OpenAlex, CrossRef, and arXiv; every claim in the answer carries a [n] marker resolving to a real paper. Deep mode plans sub-questions, searches in rounds, gap-checks, and writes a long-form cited report.
  • LLM Council: a question goes to 5+ different models over the same shared source list. Each answers independently, then anonymously ranks all answers (labels shuffled per reviewer to kill position bias), and a chairman synthesizes the final cited answer. Every sub-process (grounding, per-member opinions, peer review, synthesis) streams live in an inspectable stepper, with a leaderboard. The aggregate rank for answer across evaluators is simply where lower is better, and the raw evaluations stay inspectable so you can audit the parse.
  • Agent mode: a tool-driven harness loop. The model picks its own literature/web/library searches and runs Python until the evidence suffices; the loop owns budgets and the audit trail.
  • Reviewer agent: after any cited answer, a second pass checks each citation against its source and badges it supported / partial / unsupported. A consensus meter classifies each source's stance (yes / mixed / no) on the claim.
  • Papers mode: local RAG over your own PDFs, parsed in-browser, embedded with local transformer models, and answered strictly from your library with page-level citations.
  • Document artifacts: drafts open in a split-pane preview (Preview/Code tabs, live while streaming) and export to Word, single-file LaTeX with embedded BibTeX, Excel, and Markdown, with CSL citation styles.
  • Code Lab: implement a paper's method as Python and run it in-browser via Pyodide (numpy, scipy, pandas, matplotlib, scikit-learn), with a provenance record for reproducibility.
  • The long tail of real research work: evidence extraction tables, PRISMA-style systematic review screening, Zotero import, a citation graph explorer, notes, and a 3D studio doing in-browser monocular depth estimation.

Everything persists locally in IndexedDB. No accounts, no server-side storage: your keys and your library never leave your machine.

How I built it

  • Stack: Next.js 15 (App Router) + React 19 + TypeScript + Tailwind v4, zustand for state, IndexedDB (idb) for persistence. Deployed on Vercel.
  • One streaming protocol, four providers: a single /api/chat route normalizes OpenAI, Anthropic, OpenRouter, and NVIDIA NIM (including their two different tool-calling dialects) into one SSE event stream the whole app consumes.
  • Grounding pipeline: /api/scholar fans out to four scholarly APIs, dedupes by DOI/title, and builds the numbered source block every grounded mode shares. Citations are resolved from [n] markers back to real paper metadata; the model is never allowed to invent a reference list.
  • The council is a four-stage pipeline (ground, then opinions in parallel, then anonymous shuffled-label peer review, then chairman synthesis) that emits structured state on every sub-process transition, so the UI renders a live audit trail rather than a spinner.
  • Browser-native heavy lifting: Pyodide for Python execution, transformers.js for local embeddings and Depth Anything V2 depth estimation, pdf.js for PDF parsing, all loaded from CDN on demand so the bundle stays light and no compute server is needed.
  • Exports: a markdown block parser feeds docx, exceljs, and a LaTeX generator that rewrites [n] chains into \cite{a,b} and embeds the bibliography via filecontents* so a single .tex file compiles anywhere.
  • Quality loop: dependency-free node --test unit tests lock in the accuracy-critical logic (citation rewriting, ranking parsing, RAG caps, stream normalization), and the UI uses Floating UI so every menu flips, shifts, and resizes to any viewport.

Challenges I ran into

  • Four providers, four streaming dialects. OpenAI-style deltas vs. Anthropic content blocks vs. two tool-call encodings: normalizing them into one SSE protocol without dropping tool calls or stop reasons took careful contract tests.
  • Position bias in AI peer review. Models systematically favor "Response A." I shuffle the answer labels independently per evaluator and persist the label order, so rankings are de-biased yet still auditable after reload.
  • Hallucinated citations. Grounding constrains generation, but models still occasionally cite [7] for a claim source 7 doesn't support, hence the reviewer agent that re-reads each cited source and flags unsupported claims instead of silently trusting them.
  • Streaming markdown is hostile to previews. A document mid-stream has unclosed code fences (everything renders as code), and models love wrapping whole documents in `markdown fences. I built a normalizer that unwraps packaging fences and balances unclosed ones live.
  • No backend allowed (by choice). Running Python, embeddings, and depth estimation entirely in the browser meant wrestling Pyodide memory limits, CDN loading, and keeping the main thread responsive.
  • A thousand small UX battles: viewport-clipped dropdowns (solved with Floating UI's flip/shift/size middleware), live progress for multi-minute council runs, and making a five-model roster editable without ever typing a model ID.

Accomplishments that I'm proud of

  • A working multi-model deliberation pipeline with anonymous, de-biased peer review and a full audit trail. Not a demo, but something I now use for real literature questions.
  • Citation integrity as a system property: grounded generation + reviewer verification + consensus measurement, so the tool argues from the literature instead of about it.
  • True single-file LaTeX export that compiles on Overleaf with its bibliography embedded, the "last mile" most AI tools ignore.
  • An entire scientific computing stack (Python, embeddings, PDF parsing, 3D depth estimation) running client-side with zero server compute and zero data leaving the browser.
  • A polished, professional UI: live sub-process steppers, Claude-style document artifacts with split-pane preview, collapsible workspace panels, and viewport-aware menus.

What I learned

  • Structure beats prompting. The biggest quality gains came from pipeline design (shared grounding, shuffled anonymous review, a verifying second pass), not from cleverer prompts.
  • Trust is a UI problem too. Showing every sub-process (who's drafting, who's reviewing, what the raw evaluations said) changes how much you believe the final answer.
  • The browser is a legitimate compute platform. Pyodide and transformers.js are production-ready if you respect their loading and memory constraints.
  • Provider abstraction pays off immediately. The council literally could not exist without a uniform streaming contract across vendors.
  • And practically: position-bias mitigation, CSL citation machinery, PRISMA workflows, and more about LaTeX's filecontents* than I ever planned to know.

What's next for Butterfly Sciences

  • Cross-examination rounds in the council: members challenge each other's citations before the chairman rules, not just rank answers.
  • Multi-document synthesis: evidence tables that update live as your library grows, and contradiction detection across papers.
  • Collaboration: shareable workspaces and published, citable reports with verifiable source trails.
  • Local models: Ollama/llama.cpp seats on the council so a fully offline council is possible.
  • Deeper reproducibility: notebook-style Code Lab sessions with pinned environments, and one-click "reproduce this figure" from any cited paper with available code.

Built With

Share this project:

Updates