Camarade

Inspiration

Here's a question that nobody in the "prompt engineering" space actually wants to answer: if a coding agent gets fed a giant unstructured wall of repo instructions, notes, and half-updated conventions, how do you know which sentence in that wall is helping and which one is quietly sabotaging it? I used to think context length was the enemy, that if you just trimmed the prompt down, the agent would perform better. That's not really true though, and it took building this thing to figure out why.

Real repos are messy in a way that's almost funny once you notice it. Duplicated rules. Contradictory rules. A rule written for an architecture that got ripped out two years ago and nobody deleted the doc, so the agent is out here confidently following instructions for a system that no longer exists. So when an agent fails, is it actually reasoning badly, or is it just obeying a broken spec? That's basically the whole thesis:

Can we scientifically determine which repository instructions help an AI coding agent, which hurt it, and why?

Formally, every instruction $i$ interacting with a task $t$ contributes some signed effect on outcome quality, and most tools treat context selection as a compression problem, minimize token count and call it a day. Camarade treats it as a controlled experiment instead, where "shorter" is not the objective function. "Causally correct" is.

What it does

Camarade is an agent-independent repository intelligence, context-compilation, and experimental-evaluation layer. Works with Codex, Claude, Cursor, anything MCP-compatible, doesn't care which model is doing the coding. The pipeline:

1. Inspect the repo and its instruction sources
2. Build a structured repository-intelligence model, an evidence graph, not a flattened summary
3. Compile task-specific context
4. Run the same task under two controlled conditions
5. Score both with a deterministic evaluation contract
6. Verify the experiment is actually fair
7. Explain which instructions helped, which hurt, and why
8. Surface it through CLI, MCP server, stored artifacts, dashboard

Every fact Camarade extracts, conventions, contradictions, historical behavior, scope boundaries, gets tagged with provenance. Why does that matter? Because a fact without a source is just a vibe, and vibes don't hold up when you're trying to explain to an agent why it should stop doing something. Camarade distinguishes directly observed repo evidence from explicit instructions from historical conventions from inferred relationships from low-confidence assumptions. Five different tiers of "how sure are we about this," basically.

The actual math

This is where it stops being a vibes-based tool. For instruction (or fact) $i$ and task $t$, define a relevance score:

$$ R(i,t) = w_s S(i,t) + w_d D(i,t) + w_r R_f(i,t) + w_c C(i,t) - w_n N(i,t) $$

where $S$ is semantic relevance, $D$ is dependency proximity, $R_f$ is reference strength, $C$ is confidence, $N$ is noise or contradiction or plain obsolescence, and $w_s, w_d, w_r, w_c, w_n \geq 0$ are tuned weights. Notice the sign on $N$. That's not an accident. An instruction can score high on everything else and still get buried if it's stale, the same way a great-sounding stat can still be misleading if the sample behind it is garbage. This isn't "shorter context wins." It's a signed, weighted optimization over a graph, closer to feature selection in a regression than to summarization.

An experiment itself is the tuple

$$ E = (T, R, M, C_b, C_c) $$

task, repo state, model config, baseline context, compiled context. Every variable except the context condition stays fixed. Same commit, same adapter, same validation commands. If you don't hold everything else constant, you haven't run an experiment, you've run two conversations and picked the one you liked better. That's not science, that's a coin flip with extra steps.

Each run gets scored with a fixed weighted contract:

$$ Q = 0.40C + 0.25K + 0.20S + 0.10E + 0.05F $$

correctness, completeness, scope discipline, efficiency, format compliance, in that weighted order, and comparison uses an inclusive tie tolerance:

$$ |Q_b - Q_c| \leq 1.0 $$

If evidence is incomplete or contaminated, Camarade doesn't manufacture a winner, it just says "inconclusive." I used to think a tool that refuses to answer was a broken tool. Turns out for an evaluator, the refusal is the entire feature.

Here's the sneaky failure mode nobody talks about, though: an evaluator that sees both outputs before locking its scoring rubric can unconsciously move the goalposts toward whichever one it already likes. So the evaluation definition gets sealed, cryptographically bound to the rubric, before $Q_b$ or $Q_c$ ever gets computed. Content hashes instead of path-based ones too, since paths shift across isolated worktrees but content doesn't lie.

How we built it

Contract-driven TypeScript, every stage independently certifiable, schemas defined before a single line of logic touched them. Repository intelligence, compiled context, experiment requests, execution evidence, evaluation definitions and seals, run outcomes, explanation evidence, all schema-first. Forced every subsystem to talk through validated data instead of implicit assumptions, and honestly that lesson generalizes way past this one project.

The repository-intelligence pipeline runs file inventory, instruction parsing, scope detection, reference resolution, duplicate and contradiction detection, convention extraction, Git-history analysis, confidence scoring, evidence-graph construction. Provenance survives every stage. The context compiler is deterministic given the same $(t, R, M)$ input, follows dependency chains, strips irrelevant global context, keeps local overrides, flags contradictions instead of silently picking one.

Two adapter types mattered a lot here: a deterministic fake agent for certification, no nondeterminism, no API cost, no availability risk, and a live Codex path gated behind explicit authorization. Keeping those separate was maybe the single most important architectural call in the whole build. Conflate "protocol test" with "real empirical claim" and you either make your tests flaky or your experiments fake, and neither is acceptable when the entire pitch is trustworthiness.

The explanation engine compares baseline-vs-compiled instruction sets, contradictions, diffs, test failures, dimension-level $Q$ deltas, then assigns a confidence tier to each causal claim: directly implicated, strongly associated, plausibly associated, unsupported. It suppresses a claim outright when the evidence is thin. This is the part I'm most proud of, because it would've been so easy to let the system say "this instruction caused the regression" every single time, and that's just persuasive nonsense wearing a lab coat.

109 test files. 1,299 passing tests. Type-checking, production builds, MCP protocol verification, tamper-detection tests, the whole certification suite. Camarade is itself an evaluator, so an unreliable evaluator poisons every downstream conclusion it produces. That's not a nice-to-have. That's load-bearing.

Challenges we ran into

Defining "fair" was the hardest part, harder than I expected going in. Model outputs are nondeterministic, so running an agent twice and comparing isn't an experiment, it's noise dressed up as signal. Fixed task, fixed commit, fixed config, isolated worktrees, sealed criteria, explicit invalidation conditions, that's what actually earns the word "fair," and it took a few failed passes to land on it.

Correlation versus causation showed up constantly. Say Camarade drops five instructions and $Q_c > Q_b$. Which of the five actually mattered? You can't assume all five contributed equally, that's the same mistake as reading a regression coefficient without checking for multicollinearity first. So explanations got bucketed by evidence strength instead of asserted flatly, which fixed most of it but not all of it, some ambiguous cases just get flagged unsupported and left alone.

Explainable compression was its own headache. Flatten a pile of instructions into one summary paragraph and you lose the ability to trace any single statement back to where it came from. Provenance, scope, and confidence had to survive every compilation stage or the "explanation" step is just guessing with better vocabulary.

Fixture data on the dashboard could visually pass as real benchmark evidence too, which is a genuinely dangerous failure mode for a tool whose entire pitch is trustworthy measurement. Simulated, failed, invalid, and live results are now labeled explicitly, no dressing up a fixture as proof. And early certification code risked validating expected constants instead of exercising the real protocol, hard-coded success paths basically, which got replaced with full MCP request/response execution and real artifact inspection.

Accomplishments that we're proud of

$$ \text{Repository} \rightarrow \text{Intelligence} \rightarrow \text{Compiled Context} \rightarrow \text{Controlled Execution} \rightarrow \text{Sealed Evaluation} \rightarrow \text{Explanation} $$

Every arrow carries explicit contracts, evidence, and failure states. But if I had to pick one thing to be proud of, it's not the pipeline itself, it's the restraint built into it. Camarade knows when it doesn't have enough evidence, and it says so instead of forcing a winner. That kind of discipline is rarer than it should be in this space, where most tools would rather give you a confident wrong answer than an honest shrug.

What we learned

Context quality is not context length. A smaller prompt can underperform if it strips a critical local convention, a larger prompt can underperform if it's just full of contradictions padding it out. The naive objective,

$$ \min |\text{context}|, $$

is the wrong one to optimize for. The real objective looks closer to

$$ \max \frac{\text{task-relevant, correct, actionable information}}{\text{noise} + \text{contradiction} + \text{ambiguity}}. $$

Repository instructions behave like executable dependencies, not documentation, which is a distinction I didn't take seriously enough at first. An outdated instruction is functionally an outdated software dependency, it silently alters behavior until something breaks and someone finally bothers to read the file. Which probably means instructions need versioning, ownership, and deprecation eventually, the same lifecycle we already give code and somehow never gave the docs telling the code what to do.

Evaluation infrastructure needs stronger guarantees than the system it's evaluating. A defect in the evaluator is a defect in every conclusion downstream of it, full stop. And an invalid result is still a useful result, an "inconclusive" tells you something real about your measurement process, it's not a null result you bury and move past.

What's next for Camarade

Live multi-repo experiments across languages and agents is the obvious next step. Statistical aggregation across $n$ paired runs would help too:

$$ \Delta Q_i = Q_{c,i} - Q_{b,i}, \qquad \bar{\Delta Q} = \frac{1}{n} \sum_{i=1}^{n} \Delta Q_i $$

variance, confidence intervals, effect sizes, all computed alongside instead of eyeballed. Instruction ablation studies over a full set $I = {i_1, i_2, \dots, i_k}$, selectively removing subsets to estimate the marginal contribution of each one, basically an influence function for repo docs. Longitudinal drift detection matters too, since a rule that's correct today can quietly turn harmful the moment the architecture underneath it shifts.

And eventually, a CI check: does this pull request invalidate any of my AI instructions? Code quality is measurable right now and context quality mostly isn't, and closing that gap is really the whole point of building this in the first place.

Built With

• agentorchestration
• aiagents
• automatedevaluation
• cli
• codeanalysis
• codex
• contextcompression
• contextengineering
• dashboard
• developertools
• experimentation
• explainableai
• git
• github
• gpt-5.6
• jsonschema
• mcp
• node.js
• openai
• promptengineering
• repositoryintelligence
• restapi
• softwaretesting
• typescript