ALETHEIA

Autonomous Loop for Experimental Theory and HEP Inference Across-data. We propose a self-calibrating physics foundation model: an agent trains it, Phoenix watches it drift and prompts improvement.

Comment 1

ALETHIA: an agent that maintains explores physics through self experimentation with foundation models

Inspiration

It's not rocket science... well it's not always rocket science... let's begin again. Physics is not that complicated. A scientist proposes a model, nature provides data, and the experiment adjusts the model to fit. ALETHEIA comes from the greek word for "truth". Literally meaning "unconcealedness" or "disclosure". For us this means encoding the scientific quest for truth directly in an agentic way. We don't rely on dusty theorems or long forgotten proofs, the model looks at the data and learns the underlying truth. Bit by bit we allow the model to see more data and expand its horizons. Always checking its vectors as it learns a complete representation of the world it has been given.

We wanted to see whether an agent could do the thing a careful scientist does: notice that its own predictions have stopped holding up, and go re-measure exactly where the uncertainty is highest; crucially without being told to.

Our substrate is SMEFT Drell–Yan. The observable is the ratio of the new-physics cross-section to the Standard Model one,

$$ \mu(c, m) \;=\; \frac{\sigma_{\text{BSM}}(c, m)}{\sigma_{\text{SM}}(m)}, $$

which, for dimension-six operators O_i with Wilson coefficients c_i entering as: $$\mathcal{L} = \mathcal{L}_{\text{SM}} + \sum_i \frac{c_i}{\Lambda^2}O_i$$ is quadratic in the couplings,

$$ \mu(c, m) \;=\; 1 \;+\; \sum_i c_i\, a_i(m) \;+\; \sum_{i \le j} c_i c_j\, b_{ij}(m). $$

The agent never sees c. It only sees μ at a handful of dilepton masses m, and it has to stay calibrated as the underlying physics shifts.

What it does

ALETHIA is a Gemini-orchestrated agent that drives a physics foundation model (the "Intention" surrogate) through a single conversation:

  1. Predicts $\mu(m)$ with a calibrated uncertainty band.
  2. Checks itself for drift with three physics-aware detectors.
  3. Recovers when drift fires — it chooses the most informative new measurements, queries an oracle, folds them into context, and recalibrates.
  4. Reads its own traces through the Arize Phoenix MCP server to report how a prediction or recovery actually went.

In our live deployment, one turn took the model from a target-set entropy of $H = 126.08$ down to $H = 37.90$ after five acquired measurements. The band tightened because the agent decided, on its own, that it needed data.

How we built it

The agent. Google's Agent Development Kit (ADK), with Gemini 2.5 Flash orchestrating six deterministic physics tools: fm_predict, check_drift, recover_from_drift, epig_select, oracle_query, fm_update over a shared state that owns the trained foundation model, the conformal calibrator, and the running context ${(m_j, \mu_j)}$.

The foundation model. The Intention surrogate predicts $\mu(m)$ in closed form with a per-point leverage that doubles as an uncertainty signal. We chose it after an architecture bake-off: on the same SMEFT interface, Intention reached $R^2 = 0.9999$, well past a DeepSets baseline. The right inductive bias mattered far more than capacity.

Acquisition. When the agent needs data, it selects masses by expected predictive information gain,

$$ m^\star \;=\; \arg\max_{m}\ \mathrm{EPIG}(m) \;=\; \arg\max_{m}\ \Big[\, H(\text{target}) - \mathbb{E}\big[H(\text{target}\mid \mu(m))\big] \,\Big], $$

which is available in closed form for this model. No inner-loop retraining to score a candidate.

Calibration and drift. The band is conformal: a held-out calibration set fixes a quantile so that the interval $[\hat\mu \pm q_{1-\alpha}\,\hat\sigma]$ covers at the nominal rate. Drift is then three orthogonal tests on the live stream:

  • Accuracy — a DAS-CUSUM on standardized residuals $z_t$, firing when $S_t = \max(0,\, S_{t-1} + z_t - k) > h$.
  • Calibration — a Benjamini–Hochberg-corrected per-region binomial coverage test, so a single bad region can't be masked by good ones, and many regions can't raise false alarms.
  • Coverage / identifiability — the condition number $\kappa(A) = \sigma_{\max}/\sigma_{\min}$ of the design matrix, catching the case where the context no longer spans the directions the prediction needs.

The three combine into a 3-bit flag and a single action; in the demo, combined_flag = 001 with target_signal = cov produced action = local_retrain. That decision gates the recovery branch.

Observability — and the partner integration. Every tool emits an OpenInference span to Arize Phoenix with aletheia.*-namespaced physics attributes. A full turn is a 15-span waterfall: the Gemini reasoning steps (call_llm), the tool executions, and beneath them the physics spans (tool.surrogate.predict, chain.drift.evaluate and its three detectors, tool.drift.aggregate). The drift detectors are, in effect, online evaluations running on the model's own output.

We then took the Arize integration one step past dashboards: the deployed agent carries an ADK McpToolset over @arizeai/phoenix-mcp, so it can query the very traces it just produced. Asked "what do the traces say?", it called Phoenix, counted the spans, and reported the logged local_retrain action from the trace data, not its own memory. Observability became an input to the agent, not just a readout for us.

Hosting. The whole thing runs on Google Cloud Run. main.py serves the ADK web UI and JSON API via get_fast_api_app; a Dockerfile (Python for the agent, Node so the agent can launch the Phoenix MCP server over stdio) builds remotely through Cloud Build; deploy.sh is one command. The trained checkpoint ships inside the image, so the service is self-contained.

What we learned

  • The strongest place for a partner MCP server is inside the agent, not beside it. Wiring Phoenix into the developer CLI is useful; giving the agent its own observability as a tool is a different capability, allowing it to close an observe→reason→act loop on live telemetry.
  • Drift detection is most convincing as a gate, not a chart. The moment the 3-bit flag decides whether recover_from_drift runs, the trace stops being a log and becomes a control signal.
  • Inductive bias beat scale. The $R^2 = 0.9999$ vs. a struggling DeepSets baseline taught us more than any hyperparameter sweep.
  • Closed-form acquisition changes the agent's tempo. Because EPIG scores candidates without retraining, the agent can decide where to spend an expensive oracle call in milliseconds, mid-conversation.

Challenges

  • Closing the MCP loop without breaking cold starts. Naively constructing the Phoenix toolset at import could hang the container if the MCP server or Phoenix was unreachable. We made it guarded and lazy: the agent always boots, and the MCP transport only spins up on the first call that needs it.
  • Shipping the model into the image cleanly. A .gcloudignore can't reliably re-include a file beneath an excluded directory, so the 24 KB checkpoint kept getting dropped from the build context. We packaged it into the agent itself and resolve it by env > packaged > training-run precedence. The image is now self-contained with no special-casing.
  • Keeping the container slim. The full stack includes a MadGraph adapter and a vendored LHAPDF build; neither belongs in a request-serving image. The deployed agent uses the analytic SMEFT oracle only, which is pure NumPy and PDF-independent at leading order for the ratio $\mu$.
  • Making the agent reason about its own traces reliably. The first attempts had the model summarizing from memory instead of querying Phoenix. Tightening the instruction so it reaches for the MCP tools and naming the active project got it reading the real span data.

What's next

Hosted Phoenix experiments comparing acquisition strategies on live traffic, and the agent proposing A/B promotion of a recovered model version under its BH-corrected promotion rule: improvement on the drifted region and no regression elsewhere, at FDR $0.05$.

Built With

Share this project:

Updates