Journeyman

Journeyman is a learning agent that improves at third-party tools without prompt rewrites. It learns from failures, versions its playbook, re-runs safely,and promotes only after hold-out validation.

Comment

Project Story

About the project

Journeyman is an apprentice agent for Automated Agent Engineering. It is not a chatbot with a toolbelt. It is given one third-party app — GitHub, through the official readonly MCP — a task family (repo triage: labels, duplicates, owners, summaries, fix-PRs), and a frozen eval. Then it has to get better on that app’s data: traces, reflection, a growing playbook, a sealed hold-out, and a scoreboard for accuracy, cost, and speed.

The product runtime is Python (journeyman/). The UI is an artifact reader. Partners are pinned: TensorMux (glm-4-7-flash) as the cheap everyday brain, OpenAI gpt-5-nano only after a logged quality-gate miss, text-embedding-3-small for playbook RAG, and Neatlogs as the trace system of record.

What inspired us

Most “agent demos” look smart in a single session and amnesiac the next. They crawl a repo, burn tokens, invent labels, and never write down what the repo itself taught them. We wanted the opposite: an apprentice.

An apprentice does the same kind of work over and over (triage), keeps a notebook (journal + playbook), and only graduates when they can still pass a test they were not allowed to study. That last part is the whole product. If hold-out traces leak into training, you do not have learning — you have contamination.

The Automated Agent Engineering brief matched that instinct: third-party access, a general task, a loop that improves skill from that app’s data, and numbers a judge can read in three seconds (v0 vs learned).

We named it Journeyman because v0 is a cold explore with an empty playbook. After Reflect, it is supposed to look like someone who has actually read CONTRIBUTING.md and CODEOWNERS.

How we built it

One pipeline. Two modes. The CLI is:

python -m journeyman.demo.run --challenge frozen --mode offline
python -m journeyman.demo.run --challenge frozen --mode live

--mode branches only at Chat / GitHub MCP / Neatlogs IO. Everything above that — Actor, router, gates, policy, SuperviseCycle, Reflect, DEV eval, sealed hold-out, promote, RedTeam — is shared. Offline doubles and live HTTP run the same graph, so we could prove the loop before the partners were green.

The loop

  1. Run1 / DEV only. Actor answers the frozen DEV tasks with a weak (empty) playbook. Hold-out is not scored. Hold-out spans never enter ingest.
  2. SuperviseCycle on DEV failures: diagnose → probes → candidate prompt. LiveScorer is aux only. It is not the promote signal.
  3. Reflect merges a diffable playbook from those DEV-fail spans plus RAG over this repo (CODEOWNERS, CONTRIBUTING.md, labeled corpus issues). A journal lesson is appended.
  4. Eval DEV (the promote signal): Actor re-runs the same frozen JSON with the candidate playbook.
  5. Sealed hold-out, only if DEV improved: prior active vs candidate. Promote iff hold-out does not collapse.
  6. RedTeam on LIVE after promote. Then flush one nested Neatlogs tree, persist playbook / scripts / version pointer, write runs/last.json. --rollback restores the prior pointer.

Promote is a predicate, not a vibe:

$$ \mathrm{promote} \iff \mathrm{pass}{\mathrm{DEV}}^{(N)} > \mathrm{pass}{\mathrm{DEV}}^{(1)} \;\wedge\; \mathrm{pass}{\mathrm{hold}}^{(N)} \ge \mathrm{pass}{\mathrm{hold}}^{(\mathrm{prior})} $$

Accuracy on a split is just the mean of exact-match indicators:

$$ \mathrm{pass} = \frac{1}{n}\sum_{i=1}^{n} \mathbf{1}{\hat{y}_i = y_i} $$

Labels are exact-set; duplicates and owners are exact fields; summarize requires required keys. Every run also records cost (tools + tokens / \$) and speed (wall-clock ms).

Partners

Role Pin
App Official GitHub MCP readonly
Cheap chat TensorMux glm-4-7-flash
Escalate gpt-5-nano after a logged gate miss
Embed text-embedding-3-small via CostRouter.embed_route (never a child of chat)
Traces Neatlogs POST https://ingest.neatlogs.com/v1/trace with Authorization: Bearer

The world is a seeded fixture repo, arjun7n9s/journeyman-fixture, with a real taxonomy, CODEOWNERS, duplicates, and PRs. The playbook has to learn their rules, not a hardcoded labeler.

The desk

The Vite UI does not run the loop. It reads artifacts (last-report.json / a demo snapshot) and shows Overview, Learning, Trace, Promote: v0 vs learned, hop timeline, playbook memory, promote gate. Python remains source of truth.

What we learned

  • Learning is a protocol, not a prompt. “Be better next time” does nothing. Frozen JSON, sealed hold-out, a version pointer, and rollback do.
  • The promote signal has to be the Actor itself. Overlay scorers and live judges are useful telemetry. If they decide promotion, you will ship a playbook that cannot actually answer the tasks.
  • Cheap-first is a product. TensorMux does the everyday hops. OpenAI escalate is gated and logged. Embeddings are a separate route so RAG does not hide inside chat.
  • Traces are useless unless they are a tree. Buffering spans and flushing one Neatlogs WORKFLOW with children (LLM, MCP_TOOL, GUARDRAIL, EVALUATOR) is what makes the cockpit match the run.
  • UI that dumps JSON is not an agent product. Judges need a scoreboard and a hop timeline. The loop still has to live in Python, or you will rebuild CostRouter in TypeScript by accident.
  • Readonly is a feature. Policy-gated MCP (no writes, no secrets, no workflows) let us hit a real GitHub app without turning the demo into an incident.

On a live frozen run, empty playbook failed; after Reflect + MCP tools, DEV rose and hold-out did not collapse — then RedTeam on the LIVE pointer passed. That is the claim: v0 cold explore → grounded apprentice, with a rollback if the gate fails.

Challenges we faced

GitHub MCP is SSE, not “POST JSON and parse.” Live tools returned nothing until we did the real handshake: initialize → Mcp-Session-Id → notifications/initialized → tools/call, then parse event: message / data: {…} and unwrap result.content[].text. Until that was fixed, “live” was a costume: zero tool calls, guesses, no learning.

Neatlogs has an ingest host and a dashboard host. The sink must POST https://ingest.neatlogs.com/v1/trace with Bearer write keys. Pointing at app.neatlogs.com or sending x-api-key looks configured and still drops the tree. We remap a mistaken dashboard base URL and flush once per run.

Hold-out leakage is the easy bug. It is tempting to score hold-out “just to see.” We had to make ingest skip hold-out spans, keep Challenge from appending to the sealed set, and only open hold-out at promote. Otherwise the playbook memorizes the test.

Offline vs live without two codepaths. Doubles that do not share the pipeline lie. Branching only at IO meant pytest on frozen offline (empty playbook fails, learned playbook hits 1.0 on doubles) actually predicted live behavior once MCP spoke SSE.

The desk vs the loop. Shipping a Vite app to Vercel 404’d because the repo has no root index.html and Vite emits dist/web. Fixing deploy without turning TypeScript into a second agent was a constraint we kept on purpose.

Partner lock under time. One app, one family, three metrics. Skipping extra vendors (voice, payments) was how the loop actually shipped.

What we would do next

Keep Python as SoT. Host the thin API (python -m journeyman.server) so the public desk can run a challenge for real, not only read a snapshot. Grow scripts for the cheap path once a rule has coverage. Never unseal hold-out to chase a screenshot.

Built With

Share this project:

Updates

Submission history