Scientific Reasoning OS

A Gemini-powered agent that critiques research questions on clarity, feasibility and falsifiability, then hands the decision back to the researcher. It doesn't invent literature or decide for you.

Comment

Inspiration

Scientific research begins with asking the right question. Yet developing a defensible research question remains slow, manual and unevenly available. Researchers depend on scarce supervision and informal feedback through many undocumented revisions. Early-career researchers, especially outside elite institutions, often have no structured critique at the exact moment it matters most. We run AI-for-research workshops for Latin American researchers, and the same bottleneck appears in every cohort: people arrive with a topic, not a question. Scientific Reasoning OS was built to fix that stage — and to fix it without taking the science away from the scientist.

What it does

Scientific Reasoning OS (SROS) is an agentic research environment built on Gemini 2.5 Flash (Vertex AI). Its Scientific Question Agent (A-02) critiques a researcher's question on four formulation-stage criteria — clarity, relevance, feasibility, falsifiability — classifies the inquiry type, exposes implicit assumptions, flags missing information, and proposes concrete revisions. Every version of the question is preserved with full lineage and provenance notes. Methodological demands (instruments, thresholds, measurement) are deliberately routed into non-blocking "design-stage notes" for the later study-design stage — because a research question is not a methods section. The agent never declares novelty, never invents literature, and never validates. Only the researcher can validate and lock a question, and that decision is recorded. The product's masthead states the rule: It doesn't invent literature. It doesn't decide for you. The commercial pipeline is autonomous end to end. A customer completes a Stripe checkout; a signature-verified, idempotent webhook emits a payment_confirmed event, creates the customer record, provisions a personal research project, mints an access token and redirects the customer into their workspace — 13 seconds from payment to working environment, zero human involvement (Cloud Run logs, 7 Aug 2026), and as fast as ~60 seconds from payment link to first agent critique for invited researchers. Inside the product, Gemini performs the substantive knowledge work: each critique costs ~US$0.003 and returns in 10–20 seconds, recorded per run in an auditable ledger (model, prompt version, tokens, latency, cost). Every state change is an append-only event with correlation and causation chains — it is possible to show exactly what the AI did, when, and at what cost. Total AI operating cost across the entire beta: US$0.15 for 50 agent runs. Cloud infrastructure runs under one US dollar per month.

How we built it

  • FastAPI on Google Cloud Run (us-central1) — API + a Spanish-first, English-ready server-rendered UI
  • Gemini 2.5 Flash via Vertex AI with structured JSON output validated by Pydantic before anything is persisted; one repair retry, transient-error backoff, work-preserving degradation
  • Firestore: versioned research questions, append-only events, per-run agent ledger
  • Stripe Checkout + signature-verified webhook for autonomous onboarding
  • Versioned prompts in-repo (v1→v3); 20 automated tests; 21 recorded architecture decisions; 8 milestone tags
  • Built by a solo founder in eight working days (3–11 August 2026): first vertical slice → revision lifecycle → deployed API → autonomous payments → multi-step outcomes → real pilot cohort → traceable commercial evidence.

Challenges we ran into

  • The agent over-reached. Early prompts conflated question formulation with study design, blocking valid questions by demanding instruments and thresholds. We introduced a stage-aware severity rule and verified it as a controlled A/B on identical question text — both runs preserved in the ledger.
  • The agent caught its own founder. It flagged that a revision had silently broadened the founder's research question while claiming to narrow it. The human then wrote a better version.
  • Production reality. A Google Frontend reservation on /healthz, a Stripe SDK object-vs-dict mismatch, and a Vertex throttling event (429) each surfaced live and were fixed within the hour — the last one with retry-with-backoff and a page that keeps the researcher's text instead of losing it.
  • A webhook defect provisioned a workspace for a workshop purchase that wasn't an SROS sale. We segmented that record out of every SROS metric, documented it (decision 19), and scheduled the fix — rather than counting it.

Accomplishments that we're proud of

  • A working, paying, audited product in eight days — from an empty cloud project to a live commercial funnel with real researchers inside it, every step tagged in git.
  • Evidence-by-design. Every claim in this submission traces to an artifact a judge can verify independently: git history, Stripe exports, GCP billing, an append-only event ledger, per-run cost accounting.
  • A feedback-to-production loop measured in hours. A pilot's 12-point UX audit at 10:42am was live in production the same afternoon.
  • The agent held its rules under audit. A pilot suspected the agent had "suggested" his study population; the ledger showed it had asked him to declare it, with exemplars confined to revision prompts — the behavioural rules held on real questions in a domain (neuroimaging) we never tuned for.
  • We reported the uncomfortable finding (below) instead of hiding it.

What we learned

Four external PhD-level researchers used the system through the live funnel; three of them provided substantive feedback (the demo video, recorded when three had reported back, reflects those three). One's UX audit shipped to production the same afternoon. An expert who called the elicitation "exhausting" nonetheless took her question through three versions to ready-for-validation in eleven minutes. The most important finding arrived in two acts, and we report both. For the first five days, of seven externally submitted questions, zero were validated — researchers engaged deeply, then stopped at the gate the agent opened for them. We shipped an in-product explanation of what validation means and told one pilot the button was his. Within hours he validated his strongest question — and deliberately left his three weaker ones unvalidated, then doubled his usage. Our reading: the gate was under-explained (fixed), and validation is a genuine scientific commitment researchers don't make lightly. A research-support agent whose users validate selectively is a more credible artifact than one they rush through. Consent: two testers granted written permission to be named; the third gave an ambiguous reply, which we treated as a no and did not re-ask. Behavioural data is reported in aggregate.

Business viability

SROS launched commercially in beta on 6 August 2026 (MX$490 one-time via Stripe, live mode; a founders code for invited pilots). It connects to an existing revenue-generating education business — the founder's AI-for-literature-review workshops: 10 paying customers, MX$15,300 banked in July 2026, rated 10/10 — which double as the acquisition channel and the upsell ladder: SROS produces the defensible question the September advanced workshop assumes.

What's next for Scientific Reasoning OS

  • Near term: per-version critique history, a researcher dashboard, an Advanced Researcher mode (import literature first, critique-first), and the webhook product filter.
  • Then the next stages of the reasoning ladder — evidence mapping, study design, argument construction — always with the scientist sovereign at every decision that is theirs. The evidence-mapping stage carries a problem most research tools ignore: outside institutional subscriptions, the majority of researchers can only feed papers into an AI tool by obtaining them from unauthorized sources. We intend to build literature intake that is legal by construction — open-access-first sourcing and publisher agreements (Elsevier, Springer, Wiley and peers) so that a researcher, or an institution buying for its researchers, never has to worry about where the PDF came from. That turns a compliance risk into a reason institutions can buy.

Built With

Share this project:

Updates