Aletheia

Inspiration

I have always wanted to combine my love for technology with my desire to help people. That inspired me to build Aletheia, a developer tool that helps detect, explain, and repair supported data-leakage patterns in cancer gene-expression machine-learning pipelines before their results are trusted.

This problem is especially important in gene-expression projects, where datasets may contain only a small number of patients but thousands of features. A model can report extremely high accuracy even when samples from the same patient appear in both training and testing, or when preprocessing is fitted before the data is split. These mistakes can make an evaluation appear much stronger than it really is.

What it does

Aletheia currently audits three supported leakage patterns:

Patient overlap between training and testing
Scaling fitted before the data split
Feature selection fitted before the data split

For supported patterns, Aletheia can:

Inspect a Python ML project using static analysis
Report whether each audit passed, failed, or could not be evaluated safely
Explain the evidence behind the result
Generate a repaired copy without modifying the original source
Display the source-code diff
Audit the repaired copy again
Mark the repair as verified only when the target issue is removed without causing the other audits to regress

Aletheia does not execute the submitted training code or the repaired candidate.

It also includes an optional GPT-5.6-assisted project-understanding layer. GPT-5.6 can propose project metadata, but its proposal is treated as untrusted. The user must review and confirm it, and the deterministic audit engine makes the final pass-or-fail decision.

How we built it

Aletheia is written in Python and uses the Python Abstract Syntax Tree, or AST, to inspect supported scikit-learn pipeline structures.

The project contains:

Deterministic audit engines
Narrowly scoped repair engines
Before-and-after static verification
Secure manual project manifests
Optional GPT-5.6-assisted manifest proposals
Typed application orchestration
A Streamlit user interface
Synthetic benchmark projects
A real-data validation case study
More than 900 automated tests

I used Codex throughout development to help implement features, write regression tests, investigate defects, review security boundaries, and challenge the analyzer with adversarial examples.

The workflow follows this structure:

Project → Manifest → Audit → Repair a copy → Verify again

The deterministic engine remains responsible for the final result, while GPT-5.6 provides optional assistance in understanding the project structure.

Challenges we ran into

The biggest challenge was avoiding false confidence.

Python is a highly dynamic language, so a static analyzer cannot safely understand every possible assignment, helper function, wrapper, mutation, or control-flow structure. Some early versions classified unsupported patterns as safe.

Instead of trying to understand all Python programs, we made Aletheia conservative. It uses three audit statuses:

passed
failed
not_evaluated

When the analyzer cannot positively establish that a structure is safe, it returns not_evaluated instead of guessing.

Another challenge was safely repairing source code. A repair must:

Be written to a separate output file
Leave the original source unchanged
Produce valid Python syntax
Match only a narrowly supported source pattern
Be audited again after transformation
Never be marked as verified simply because a transformation was applied

We also had to keep sensitive information out of public results. Raw patient identifiers, local absolute paths, API keys, and provider internals are not included in normal serialized output.

Accomplishments that we're proud of

We are proud that Aletheia became more than a simple leakage detector. It provides an evidence-backed workflow from detection to repair and verification.

We validated it using the public NCBI GEO GSE15852 gene-expression dataset, containing:

86 samples
43 matched tumour-normal patient pairs
22,283 expression features

The original row-level split placed samples from 18 patient groups on both sides of the evaluation boundary.

Before repair:

Patient-overlap audit: Failed
Overlapping patient groups: 18
Scaling audit: Passed
Feature-selection audit: Passed

Aletheia generated a patient-aware grouped-split repair in a separate file.

After verification:

Patient-overlap audit: Passed
Overlapping patient groups: 0
Scaling audit: Passed
Feature-selection audit: Passed
Static verification: True

The original source and dataset remained unchanged, and the training code was not executed.

We are also proud of building a large regression and adversarial test suite. Many tests were created from real false-success cases discovered during review, helping the analyzer fail closed instead of returning an unjustified success.

What we learned

Building Aletheia taught us that reliable machine learning requires more than training a model and reporting an accuracy score.

We learned about:

Data leakage in small-sample, high-dimensional datasets
Patient-aware train/test splitting
Python AST analysis
Conservative static-analysis design
Evidence-backed audit reporting
Safe source-code transformation
Regression and adversarial testing
Privacy-aware result serialization
Designing an LLM-assisted workflow where deterministic code remains authoritative

One of the most important lessons was that uncertainty should be reported honestly. Returning not_evaluated is better than presenting unsupported code as safe.

What's next for Aletheia

Aletheia is currently a scoped hackathon prototype rather than a universal Python analyzer or clinical-validation system.

Future work could include:

Supporting additional preprocessing and feature-selection tools
Improving analysis across more helper functions and control-flow structures
Adding more leakage audits, such as duplicate-sample and batch-confounding checks
Producing downloadable audit reports
Improving evidence visualization inside the interface
Validating the tool on additional public gene-expression projects
Expanding the repair engine while preserving conservative verification
Adding carefully controlled live testing of the optional GPT-5.6 integration

The long-term goal is to help developers question impressive model results before trusting them: