Pramit

Pramit turns uncertain R&D into predictable progress by making experiments reproducible, comparisons valid, decisions traceable, and claims defensible

Comment

Inspiration

Pramit grew out of problems I had repeatedly seen while working with research and machine-learning teams.

One of the most frustrating was also one of the simplest:

A model produced a significantly better result, but nobody could reliably reconstruct exactly how that model had been produced.

The improved metric was visible. The experiment had clearly done something right. But parts of the configuration, data identity, preprocessing, evaluator settings, code state, environment, or experimental context were missing or scattered across different systems.

The team had effectively discovered something valuable that it could no longer fully reproduce.

And that was only one version of a broader problem.

As R&D organizations grow, research knowledge becomes fragmented across notebooks, repositories, experiment trackers, dashboards, tickets, documents, chat threads, and people's memories. That creates recurring failures:

  • A result appears better, but the evaluator or test setup changed, so the comparison is not actually valid.
  • Several variables change at once, yet one of them gets credited for the improvement without defensible causal evidence.
  • A researcher remembers why an experiment was run, but the hypothesis, controls, assumptions, and expected evidence were never captured.
  • A promising experiment from months earlier cannot be resumed because the result survived but its context did not.
  • Teams optimize metrics while gradually drifting away from the original research or product objective.
  • Leadership hears that a project is progressing but cannot easily answer: What improved? Why do we believe it improved? What remains uncertain?
  • Claims make their way into decisions and releases even though the supporting evidence is incomplete, outdated, or non-comparable.
  • When people leave or projects change hands, part of the organization's scientific memory leaves with them.

Small research teams can compensate for this with discipline and individual memory.

At scale, that stops working.

The question behind Pramit became:

Can scientific rigor itself become infrastructure?

Instead of relying on researchers to manually preserve every configuration, comparison, confound, decision, and assumption, we wanted a system that continuously maintains the evidence chain behind R&D.

The goal is simple:

Make research progress more predictable without making research less exploratory.

What it does

Pramit is a local-first R&D evidence control plane that makes experiments reproducible, comparisons valid, decisions traceable, and claims defensible.

It connects the parts of the research lifecycle that normally live in different systems:

intent → hypothesis → experiment → provenance → evidence → comparison → claim → decision → release

Pramit does not try to replace notebooks, experiment trackers, source control, dashboards, or project-management tools.

Instead, it acts as the evidence layer across them.

For every meaningful research result, Pramit can preserve and reason over questions such as:

  • What exactly was run?
  • Which configuration produced this result?
  • Which code revision was used?
  • Which dataset, split, evaluator, and protocol produced the metric?
  • What changed relative to the previous experiment?
  • Are these two results actually comparable?
  • Were multiple variables changed at the same time?
  • Is causal attribution justified?
  • What hypothesis was being tested?
  • Which evidence supports a claim?
  • Does contradictory evidence exist?
  • Is the work still aligned with the stated objective?
  • Is there enough defensible evidence to make the next decision or release?

Experiment provenance

Pramit creates a Run Passport around experiments so that a result is not separated from the conditions that produced it.

That can include:

  • model or system configuration,
  • code revision,
  • dataset and split identity,
  • evaluator,
  • evaluation protocol,
  • environment,
  • seed,
  • parent experiment,
  • relevant artifacts,
  • hypothesis,
  • intervention and controlled variables.

This means a valuable result should not become unusable simply because its provenance disappeared.

Comparability and forensics

Pramit does not assume that two numbers with the same metric name can be compared.

Its deterministic evidence engines inspect whether material conditions changed between experiments.

A comparison may therefore be classified as:

  • comparable,
  • non-comparable, or
  • confounded.

For example, an apparent improvement can be flagged if the evaluator, dataset, protocol, or several experimental variables changed at the same time.

This stops misleading conclusions from propagating into reports, decisions, and releases.

Claims backed by evidence

Pramit explicitly separates:

what happened

from

what we believe happened.

Claims are connected to the evidence that supports or contradicts them.

That allows Pramit to identify cases where the available evidence does not yet justify the conclusion being made.

Objective alignment

Research activity is not the same thing as research progress.

Pramit connects hypotheses, experiments, evidence, claims, and decisions back to the objective they are supposed to advance.

This gives different stakeholders views over the same underlying evidence:

A researcher can ask:

What should I investigate next?

A research lead can ask:

Which hypothesis remains unresolved?

Leadership can ask:

What evidence demonstrates that this objective is actually progressing?

Governed AI review

When deterministic evidence detects a condition that deserves investigation, Pramit can autonomously launch an AI review.

The agent receives a sanitized, governed evidence packet rather than unrestricted access to the workspace.

It can:

  • inspect relevant evidence,
  • identify likely confounds,
  • explain inconsistencies,
  • connect evidence to objectives,
  • and propose the next defensible action.

It cannot:

  • rewrite experimental facts,
  • silently alter evidence,
  • declare causal truth,
  • override comparability,
  • override claims,
  • or bypass release gates.

The agent reasons over evidence.

Pramit remains authoritative about the evidence itself.

How we built it

A core architectural decision was to separate deterministic evidence authority from probabilistic AI reasoning.

Local deterministic evidence plane

Pramit's authoritative evidence system runs locally.

It contains deterministic engines for areas including:

  • Run Passports and provenance,
  • experiment preflight validation,
  • experiment diffs,
  • comparability,
  • forensic and confound analysis,
  • objective coverage,
  • claim auditing,
  • delivery and release readiness.

Evidence is stored in an append-oriented local ledger backed by SQLite/WAL with integrity mechanisms that preserve the history of what the system knew and when.

Pramit therefore remains fully useful even without a cloud connection or an LLM.

Privacy boundary

Research environments frequently contain proprietary data, configurations, source code, customer information, and artifacts that should not be sent to an external model.

Before any cloud review occurs, Pramit builds a sanitized evidence packet.

The boundary performs checks such as:

  • schema validation,
  • restricted-field filtering,
  • artifact-byte rejection,
  • contract enforcement,
  • and packet identity generation.

The cloud agent receives only the evidence required to perform the specific review.

Asynchronous Google Cloud review plane

The governed review workflow uses:

  • Cloud Run for authenticated ingestion and agent execution,
  • Pub/Sub for durable asynchronous delivery,
  • Firestore for job and proposal state,
  • Google ADK for agent orchestration,
  • Gemini through Vertex AI for evidence reasoning.

The workflow is deliberately asynchronous rather than a synchronous model API call.

Pramit can detect a review condition, dispatch work, continue operating locally, and receive a governed proposal once the review completes.

Idempotency and state

Each review receives an identity based on:

workspace_id + job_type + target_id + target_revision

The Firestore job is created atomically before Pub/Sub publication.

This means duplicate Pub/Sub delivery cannot create multiple proposals for the same evidence state.

Jobs move through explicit states such as:

queued → running → succeeded / failed

Transient infrastructure or model failures can be retried through Pub/Sub.

Permanent packet or contract failures are persisted as failed jobs rather than retried indefinitely.

Bounded agent tools

The ADK agent receives narrowly scoped tools over the evidence packet.

It does not receive:

  • arbitrary filesystem access,
  • arbitrary shell execution,
  • unrestricted workspace access,
  • or tools capable of mutating authoritative evidence.

The output is a structured, evidence-cited proposal.

Before becoming available for acceptance, that proposal is validated against its schema and governance rules.

Human-controlled authority

The investigation can run autonomously from detection through proposal generation.

Human involvement is required only when a proposal should become part of the authoritative research record.

An accepted proposal is written back as an immutable annotation rather than rewriting historical facts.

This allows us to automate research investigation without handing a probabilistic model authority over scientific truth.

Challenges we ran into

Finding the right abstraction

The problems that inspired Pramit came largely from ML research, but we did not want to build another ML experiment tracker.

The same underlying failure appears across many forms of R&D.

Changing an evaluator in ML is conceptually similar to:

  • changing an assay in biology,
  • changing an instrument in physics,
  • changing inclusion criteria in a clinical study,
  • changing survey methodology in social science,
  • or changing a benchmark in systems research.

The useful abstraction was therefore not "model run."

It was:

hypothesis + conditions + intervention + observation + evidence + claim

Designing around that model allows Pramit to remain useful beyond any single research domain.

Deciding what AI should not do

One of the hardest design choices was resisting the temptation to make the LLM responsible for everything.

For example, we could simply ask:

Are these two experiments comparable?

But if comparability can be determined from structured facts, delegating it to a probabilistic model weakens the system.

We therefore learned to make questions deterministic whenever the evidence allows it.

AI is used where reasoning adds value:

  • synthesizing multiple pieces of evidence,
  • explaining anomalies,
  • identifying possible confounds,
  • proposing investigations,
  • and connecting findings to larger objectives.

Preserving privacy while retaining useful context

Sending an entire research workspace to an AI model would make the system easy to build but unsafe to deploy.

Sending too little context makes the review useless.

Finding that boundary required us to design the evidence packet itself as a first-class contract.

The result is a model where agents receive derived evidence rather than unrestricted data access.

Handling distributed-system failure correctly

Once reviews became asynchronous, we had to deal with problems that are easy to ignore in a prototype:

  • duplicate messages,
  • partial failures,
  • stale revisions,
  • retries,
  • malformed packets,
  • interrupted model calls,
  • and repeated requests.

Explicit job identity and state became as important as the agent prompt.

Avoiding another tool researchers have to maintain

Researchers already have too many tools.

Requiring them to migrate everything into Pramit would create more friction and probably make the evidence problem worse.

So we designed Pramit as a control plane over existing workflows, not as a replacement for every notebook, tracker, repository, or dashboard.

Accomplishments that we're proud of

We turned research rigor into executable infrastructure

The part we are most proud of is that Pramit does not merely display experiment history.

It can actively determine when:

  • evidence is incomplete,
  • provenance is missing,
  • comparisons are invalid,
  • results are confounded,
  • claims are insufficiently supported,
  • objectives lack evidence,
  • or a release does not yet have a defensible evidence trail.

That turns rigor from documentation into something the system can continuously enforce.

We kept deterministic evidence authoritative

Pramit uses AI without making AI the arbiter of scientific truth.

The system can autonomously detect, investigate, and recommend actions while maintaining a hard authority boundary between:

facts produced by evidence

and

interpretations proposed by an agent.

We made the cloud workflow production-oriented

Instead of treating Gemini as a synchronous API endpoint, we built a durable background workflow with:

  • authenticated ingestion,
  • deterministic job identity,
  • Firestore state,
  • Pub/Sub retries,
  • asynchronous ADK execution,
  • bounded tools,
  • proposal validation,
  • and explicit failure semantics.

We made the architecture local-first

Pramit's essential evidence computation remains local.

Cloud reasoning is optional.

If the cloud review endpoint is not configured, deterministic recomputation and evidence governance continue to work.

This was important because research evidence often lives in environments where privacy and control matter.

We connected the entire evidence lifecycle

Rather than solving only experiment reproducibility, Pramit connects:

experiments → provenance → comparisons → forensic reasoning → claims → objectives → decisions → delivery

That broader chain is what allows it to address the organizational failures that originally motivated the project.

What we learned

The biggest lesson from building Pramit is that reproducibility alone is not enough.

A perfectly reproducible experiment can still be compared incorrectly.

A valid comparison can still lead to an unjustified causal claim.

A defensible claim can still optimize the wrong objective.

And high-quality research can still be lost organizationally if nobody can reconstruct why a decision was made six months later.

The real unit of trustworthy research is therefore not the experiment.

It is the complete chain:

$$ \text{Intent} \rightarrow \text{Hypothesis} \rightarrow \text{Experiment} \rightarrow \text{Provenance} \rightarrow \text{Evidence} \rightarrow \text{Claim} \rightarrow \text{Decision} $$

When parts of that chain live only in human memory, R&D becomes increasingly unpredictable as teams grow.

We also learned that AI becomes more useful when its authority is intentionally constrained.

Giving an agent unrestricted access and asking it to decide "what is true" may look more autonomous, but it is not necessarily better automation.

A stronger pattern is:

deterministic systems establish the evidence; agents investigate what the evidence implies.

That separation became one of the central architectural principles of Pramit.

What's next for Pramit

Our goal is to make Pramit a general evidence infrastructure layer for research and engineering organizations.

The next steps include expanding integrations so Pramit can observe evidence from existing systems without forcing teams to change how they work.

Potential integration surfaces include:

  • experiment trackers,
  • source-control systems,
  • notebooks,
  • data and model registries,
  • issue trackers,
  • CI/CD systems,
  • observability platforms,
  • laboratory systems,
  • and research documentation.

We also want to deepen Pramit's ability to reason about research portfolios, not only individual experiments.

That includes questions such as:

  • Which objectives are receiving activity but producing little evidence?
  • Which findings have been independently reproduced?
  • Which claims depend on fragile assumptions?
  • Where are multiple teams unknowingly investigating the same question?
  • Which unresolved confounds are blocking decisions?
  • Which prior experiments should inform the next research direction?
  • How much of a release is supported by reproducible evidence rather than institutional memory?

Over time, Pramit should become a form of organizational scientific memory: a durable record not only of what teams tried, but why they tried it, what the evidence established, what remained uncertain, and how those findings affected subsequent decisions.

The broader ambition is to make R&D more predictable without making it more bureaucratic.

Researchers should remain free to explore.

Pramit should make sure that when something important is discovered, the organization can reproduce it, understand it, trust it, and build on it.

Built With

Share this project:

Updates