Benchwork

Inspiration

Early stage drug target research means manually cross referencing six or seven different biology databases per question. Open Targets for disease associations, ChEMBL for compound activity, STRING for protein interactions, PubMed for literature, AlphaFold for structure. Then stitching the answer together by hand. We kept coming back to the same frustration. A general chatbot can answer these questions instantly, but it answers from memory, and in biology a plausible sounding but wrong gene name doesn't just waste time, it can send a whole research direction down the wrong path for months.

So the question we wanted to answer was, what if every single fact a system gives you came with a receipt. Not trust me, but here's the exact database record this claim came from, click it yourself. And further, what if the system was honest about the difference between what's actually been measured (evidenced), what can be reasonably inferred by chaining measured facts together (inferred), and what's just not known yet, but here's the specific experiment that would tell you (a gap). That last part, Tier 3, is the piece no chatbot does, and it became the whole thesis of the project.

What it does

You ask a biology question in plain English. Here's what Benchwork does with it.

Plans the query, extracts intent, resolves messy terms to real database IDs
Fans out to live APIs in parallel (Open Targets, ChEMBL, STRING, PubMed, AlphaFold), with retries and graceful degradation if one source is down
Verifies every extracted claim against the raw JSON it came from, anything that doesn't trace back gets dropped, not hedged
Builds a typed knowledge graph (genes, proteins, diseases, compounds, pathways) instead of a pile of disconnected query results
Tiers every claim as T1 (directly retrieved), T2 (an inferred path stitched from T1 edges), or T3 (a gap, flagged honestly, with a concrete proposed experiment instead of a shrug)
Renders a sourced brief with a clickable evidence network, bioactivity chart, and 3D structure viewer, every point on every chart traces back to a raw record
In high mode, runs actual path search algorithms over the graph to find and rank mechanistic routes between two proteins, not just retrieve facts about each independently

There is also a grounded vs ungrounded toggle. Same question, same model, tools switched off. Watching the ungrounded column invent a confident sounding but citation free answer next to the grounded one is the clearest demo of why this matters.

How we built it

The backbone is a FastAPI plus asyncio backend. A planner LLM call turns the natural language question into a structured intent and a dependency graph of tool calls, which a DAG orchestrator executes concurrently. Independent branches (ChEMBL, STRING, PubMed, AlphaFold) all fan out from the Open Targets result and run in parallel rather than one after another. Every adapter is cache wrapped through SQLite, so repeated runs are deterministic and free.

The graph itself is an in memory networkx MultiDiGraph with a frozen contract. Every edge carries a source, a source url, a confidence score, and the exact raw slice of JSON it was extracted from. Nothing gets into the graph without that provenance attached.

Verification runs before tiering. For each edge, we check the claim against its own raw source record. The citation precision metric is just

$$ \text{citation precision} = \frac{\text{supported edges}}{\text{total edges}} $$

and we scan the narrative text itself for gene symbol shaped tokens that are not backed by a verified node, dropping the whole narrative rather than let one unsupported mention slip through.

High mode runs bounded k shortest paths search over the induced subgraph between two requested entities, scoring each path by

$$ \text{score}(p) = \frac{1}{n}\sum_{i=1}^{n} c_i \cdot w_i - 0.05(n-1) $$

where c_i is each edge's confidence and w_i is a source tier weight (STRING and ChEMBL trusted slightly higher than a text mined interaction), with a small penalty per extra hop so shorter, better evidenced paths win. An LLM then narrates the winning paths, but strictly from the path's own steps, never adding outside biology knowledge.

The frontend is React, TypeScript, and Vite, with Cytoscape.js for the evidence network, Vega Lite for the bioactivity and ablation charts, and 3Dmol.js for the structure viewer, all fed via a live SSE trace so you watch each database call happen in real time instead of staring at a spinner.

Challenges we ran into

The verification layer looked simple on paper and wasn't. Real API responses are messy in ways that broke our own safety checks in subtle ways. The hallucination scanner false flagged the model correctly citing AlphaFold DB as an unverified entity, because it only matched exact node labels split on whitespace, not fragments like DB. Hyphenated disease terms like ALS associated tripped the same regex for a similar reason. Small bugs, but exactly the kind that quietly break the grounding guarantee the whole project is built around.

The bigger one, our druggability filter rebuilt the graph around only the surviving druggable gene IDs at zero radius, which silently deleted every ChEMBL, AlphaFold, PubMed, and STRING edge we had already fetched and verified, any time a question so much as mentioned druggable. And separately, STRING returns protein interactions under its own internal ID scheme, completely disconnected from the Ensembl gene IDs the rest of the graph uses for the same biological entities, so even with the filter fixed, path search could never actually route through real protein interaction evidence, because there was never an edge connecting the two representations of the same gene. Both took real end to end testing against live data to surface, neither showed up in unit tests written against mocked data.

Accomplishments that we're proud of

Getting citation precision to sit at one hundred percent on real live queries, not a mocked demo, actual calls to Open Targets, ChEMBL, STRING, PubMed, and AlphaFold with every single edge traced back to its raw source. Watching the ungrounded column confidently produce an answer with zero citations sitting right next to the grounded one made the whole pitch click in a way no slide could.

We are also proud that high mode actually works the way we hoped it would from day one. Real ranked mechanistic paths between two proteins, pulled from real protein interaction data, plus an honest Tier 3 gap with a concrete proposed experiment attached, rather than a hand wavy disclaimer. And once we moved reasoning calls over to a faster hosted model, the whole pipeline runs end to end in under four seconds.

What we learned

The unit tests we wrote against mocked API responses gave us false confidence. Every serious bug we found, the druggability filter quietly deleting verified evidence, the STRING protein nodes being disconnected from the rest of the graph, only showed up once we ran real queries against real live data. Mocked tests are good for catching regressions, they are not a substitute for actually exercising the system end to end.

We also learned that model choice is not a minor implementation detail, it is closer to an architectural decision. The same prompts and the same code produced reliable structured output and four second responses on one model, and slow, occasionally malformed output on another. A grounding system is only as trustworthy as the weakest link in that chain, including the model doing the reasoning.

What's next for Benchwork

Extending high mode beyond the one flagship protein pair to general purpose path search across any two entities a researcher names. Adding the remaining adapters from the original spec, UniProt for protein annotation, Reactome and KEGG for pathways, and ClinicalTrials for trial phase and status. Building out cross source conflict detection, so that when two databases disagree on a claim, Benchwork surfaces the disagreement instead of silently picking one. And turning the knowledge graph into something session persistent, a workbench that grows and stays queryable across an entire research session rather than resetting with every question.