Inspiration
We picked the Signal-to-Dose track because it puts several real constraints in one problem. Pharma teams can't send launch intelligence to a cloud API, so everything had to run on a model we host, capped at 3B parameters. Every claim had to be backed by an exact quote, so a fluent but wrong answer scores nothing. And the task mixes classification, sentiment and quote extraction on messy sources: abstracts, trial records, FDA reports and news. We wanted to see how far small, private models could get against frontier models on a task where being right and being cited both count.
What it does
For each public item (abstract, trial record, FDA report, press release, news) it predicts the signal type, whether a launch team should act, and sentiment on five aspects, each with a speaker and an exact quote:
Input: 1 Source Document
Output:
- Signal Type: The category of the signal.
- Actionability: A Yes/No recommendation for the launch team.
- 5 Aspect Evaluations, each containing:
- Sentiment Score (on a scale of -2 to +2)
- Speaker Name
- Exact Quote backing up the claim
A demo page turns these labels into insight cards and sentiment trends.
How we built it
- Data. About 5,700 items: the package's unlabeled items plus PubMed, web and news we collected with Tavily and Exa.
- Teacher. Claude labeled them with a prompt tuned on the dev set (composite 0.92).
- Students. Three small BiomedBERT models (about 110M parameters each) learned from those labels: an item model for the labels, an extractive quote model that always copies text exactly, and a quote judge that sets polarity from the chosen quote so label and quote agree.
- Rules. Trial records and FDA adverse-event reports are labeled by rules straight from the guide.
Everything runs offline in one Docker command with networking disabled.
Results
The score is the mean of seven metrics:
$$ \text{composite} = \tfrac{1}{7}\left(F_1^{\text{signal}} + F_1^{\text{act}} + F_1^{\text{aspect}} + \kappa_{\text{pol}} + \text{Acc}_{\text{spk}} + F_1^{\text{span}} + \text{Cit}\right) $$
| Approach | Dev composite |
|---|---|
| Zero-shot Qwen3.5-2B | 0.758 |
| Our pipeline (about 330M parameters total) | 0.865 |
| Claude teacher (not offline) | 0.923 |
It labels 1,733 items in 93 s on a laptop GPU, at no cost, with 0 network connections.
Challenges
- A 3B model alone was weak: free-form answers broke the format, and forced JSON fixed the format but not the judgement.
- With only 60 gold items, one item moves the score by about 0.02, so many ideas (a larger BERT, Opus labels, direct weight editing) looked good but stayed inside the noise. We built a larger check set to tell them apart.
- Label and quote could disagree (negative safety, "well tolerated"). The quote judge fixed that.
What we learned
Small specialised models trained on a strong teacher beat a general 2B model on a narrow task, and they run anywhere. Measuring honestly, with a held-out set chosen before looking, mattered more than any single trick.
Built With
- biomedbert
- llama.cpp
- python
- qwen
