Inspiration

A health story quotes "an independent expert," and the reader has no way to know who pays them. A 2017 CMAJ study (Wang, Grey and Bolland) found that about a third of commenters quoted in health news had financial conflicts. Two-thirds of those went unreported. The information is usually public, in the expert's own papers' conflict-of-interest statements, but nobody checks it while reading. We wanted a tool that does the check in seconds.

What it does

Paste a health news article or its URL. Conflict Check pulls out the quoted experts, confirms who each one is, and searches papers and the web for funding, consulting and board ties the article didn't mention. Every tie comes with the quote and source that support it.

How we built it

A pure-Python pipeline with no third-party packages and a single-page web UI:

  1. Load the article with Tavily extract.
  2. Extract quoted experts, their roles and affiliations, using rule-based patterns.
  3. Confirm identity through the ORCID public API. We need exactly one match for name and affiliation.
  4. Gather evidence with Exa research-paper search (conflict statements) and Tavily search.
  5. Extract ties. A tie must be a clause naming the person, an organization and a tie keyword.
  6. Classify each tie as disclosed in the article, related to this story, or other.
  7. Confirm with a CMS Open Payments badge.

Every API call goes through one session layer that caches to disk, supports replay mode for stage demos, and streams each call to the UI as it happens.

Each connection gets a reliability score from 0 to 100, computed from seven weighted factors:

The factors are identity (20%), attribution (20%), source (20%), corroboration (15%), wording (10%), recency (10%) and organization (5%). Scores of 80 and above are Strong, 60 to 79 are Moderate, and anything lower is Weak. The Why 92? control shows every factor behind a score. Ties are ordered by a fixed set of rules, so the same input always gives the same output.

Challenges

  • Wrong-person risk. Two experts can share a name. If ORCID can't return exactly one match, we run no search until a person picks the right record, and we re-check that pick against the surname.
  • Not accusing anyone falsely. No model writes a tie. A tie appears only if a fetched page contains a clause naming both the person and the organization, and verify_quote confirms the quote is on that page. Open Payments never creates a tie, because it matches on name and city rather than employer. It only confirms one. We say "connections the article didn't mention," never "hidden conflicts."
  • Fragile rule-based extraction. Early versions read places, job titles and link text as organizations. We added tests with fictional fixtures and tightened the patterns.
  • Many steps that can fail. Our own risk review rated reliability 2/5 because each expert passes through many steps. So every result shows how much to trust it and why, and "No confirmed connections" and "Identity not confirmed" count as valid results.
  • Demo reliability. A replay cache plus a fully fictional demo mode means the demo runs without the network.

What we learned

  • Declining to answer builds more trust than guessing.
  • For a claim about a real person, quoting a verifiable source is safer than generating text.
  • A design system built from a small set of tokens kept a fast-moving UI consistent.

What's next

Build a hand-checked test set of about 20 real health articles and report precision and decline rate. Add an optional AI fallback for experts the rules miss, with its output still going through the review step.

Built With

Share this project:

Updates

Submission history