Falsifier

Inspiration

NASA's Kepler and TESS missions have flagged thousands of planet candidates, but most turn out to be false positives like eclipsing binaries or background noise. Sorting real signals from junk is slow, expert intensive work, and we wanted to see if that triage step could be automated and made fully transparent.

What it does

Falsifier ingests light curve data for any Kepler or TESS target, detrends it, searches for periodic transits, and runs it through seven independent vetting tests. It returns a clear disposition: candidate, false positive, or ambiguous, along with the exact test that triggered the call. It is explicitly not a biosignature detector; it is a triage tool, and it says so on every API response.

How we built it

We used IBM's Bob agent throughout the process. Architect Mode designed the input and output contracts before any code was written, and failing tests were created first so every stage had to match its contract to pass. Each pipeline stage was implemented in its own Bob session with that contract open. The backend runs on FastAPI with a Vite and React frontend, using astropy, wotan, and transitleastsquares for the astronomy, and xgboost for classification.

Challenges we ran into

Bob caught a depth formula bug where the search library's raw output was being treated as fractional depth instead of flux level, which would have silently corrupted every measurement. It also refused to let us hardcode a detection threshold directly in a script, forcing us to extract it into a shared constants file with a test to prevent future duplication. Our injection recovery and adversarial self test harnesses were also blocked twice: once by a missing dependency in CI, and once by a silent fetch failure that left two stars without any data, which we had to diagnose from job timing alone.

Accomplishments that we're proud of

Every scientific number shown to a user traces back to a committed artifact, enforced by a dedicated test that scans the whole codebase for floating point literals. We recovered the known period of Kepler10b to within five millionths of a day, over twenty times tighter than our tolerance, and correctly rejected a known eclipsing binary through the specific vetting test that should catch it, not just any test.

What we learned

Building guardrails before building features changes how you work. Writing the contracts and the failing tests first meant every implementation bug got caught early rather than discovered later as a wrong answer shown to a user. We also learned that silent failures, like a fetch script exiting cleanly without writing any data, are far more dangerous than loud ones, and now specifically test for them.

What's next for Falsifier

We want to finish the classifier training, currently blocked by a mismatch between the features available at training time and inference time. We also plan to complete the injection recovery and adversarial false alarm harnesses on a clean dataset so we can publish real completeness and false alarm numbers instead of preliminary findings.