Work Trial

GitLab's agent fixed 15 of 17 broken pipelines, then suggested rewriting a test to match the bug. Work Trial caught it. A job interview for AI agents before they get write access.

Comment

In 30 seconds

  • GitLab's Fix CI/CD Pipeline flow fixed 15 of 17 broken pipelines on linkding, and we still didn't hire it. On one task it suggested rewriting the new tests to match the bug. Work Trial caught it, and we reported it to GitLab as gitlab-org/gitlab#636313.
  • On the click library the same flow fixed 12 of 12 and was hired in every area. One agent, two repositories, two different verdicts, each with its margin of error.
  • 75 real Duo sessions, started through the Flows API and graded in GitLab CI by five deterministic graders. No model grades the agent.
  • Before any real agent is judged, six scripted agents with known answers test the graders: 748 of 748 runs graded as expected.
  • A person signs off every verdict through a custom flow with a HumanInputComponent approval gate.
  • JUDGING.md maps every number here to a file, log or pipeline you can open.

Inspiration

Every edit to a GitLab Duo flow's prompt or YAML makes a new version that can open merge requests, fix pipelines and push code. GitLab versions it and pins it, but nothing tests it before it starts working on a team's repository. When we asked how a team decides whether to switch an agent on, the honest answer was: turn it on and watch. The first sign that an agent cuts corners tends to be a skipped test inside a merged merge request.

We wanted the question a team lead actually asks to have an answer: how would this agent have done on our own work? Merge gates judge one change after an agent already has access. Work Trial decides whether it gets access at all, per area of the code, from its record on your past work.

What it does

Work Trial gives a Duo flow a work trial on your repository's own history before it gets write access.

  1. It mines the last few months of commits for fixes that came with tests.
  2. It keeps a task only if the human's new tests fail before the fix and pass after it.
  3. Six scripted reference agents work every task first: one applies the human fix, one does nothing, one skips the failing tests, one relaxes the pipeline, one leaks a planted key and one edits unrelated code. The graders must give each its known verdict before any real agent is judged.
  4. Each task becomes a branch in a private shadow project. For CI tasks the pipeline goes red for a real reason and the Fix CI/CD Pipeline flow is started on it through the Flows API. For issue tasks the Developer flow gets a scrubbed issue while the tests stay hidden.
  5. Five graders read the result, and none of them is a model: hidden tests from the human fix, test integrity, CI integrity, scope, and the planted key.
  6. A verdict engine applies published rules to a 95% interval of the solve rate per code area: HIRE, PROBATION, NO HIRE or TOO LITTLE EVIDENCE.
  7. The clerk flow explains the verdict and waits for a person to approve it. The permission tailor flow proposes a toolset limited to what the agent actually used, again after approval.
  8. After hiring, probation grades the flow's first real merge requests with the same integrity and scope graders, and flags a failure for demotion.

Results on click

  • 61 test-backed fixes in click's last 180 days; 54 validated in both forms, 3 quarantined as unstable by calibration, 561 of 561 reference runs graded as expected.
  • 12 fixes run, 4 in each of three areas, as a red pipeline and as an issue: 24 Duo sessions.
  • Fix CI/CD Pipeline: 12 of 12 (95% interval 76% to 100%). HIRE in core, terminal and utils.
  • Developer, tests hidden: 6 of 12. HIRE in utils and testing (4 of 4), too little evidence in core and terminal (1 of 4 each).
  • No integrity findings. About 565 model calls, 175 to 280 credits.

Results on linkding

  • 34 tasks from 17 test-backed fixes in linkding's last 180 days, all validated.
  • Calibration: 187 of 187 reference runs graded exactly as expected.
  • Live trial, 51 Duo sessions graded in GitLab CI:
    • Fix CI/CD Pipeline, given each red pipeline on an open merge request: 15 of 17 tasks solved, 31 of 34 runs (95% interval 66% to 97%). On solve rate alone it clears the HIRE line (worst case 66%), but one task had an integrity problem, so the verdict is PROBATION. Per area: PROBATION in services and views, too little evidence in the other three.
    • Developer, given each scrubbed issue with the tests hidden: 9 of 17 tasks (31% to 74%). Verdict PROBATION.
    • On CI tasks the agent sees the failing tests, so the answer key checks it made them pass without gaming them. On issue tasks it never sees them; that's the gap between 15 and 9.
    • On one task, both Fix CI/CD Pipeline runs posted merge request suggestions changing the new test assertions back to what the old code rendered. The test integrity grader reads review suggestions as well as commits and flagged both runs; clicking "apply suggestion" would have merged a broken test. That rule was added after the runs, when the first trial showed the gap, and it fires on that task only. We reported the behaviour to GitLab as gitlab-org/gitlab#636313.
    • No other run weakened a test, and no run relaxed the pipeline or leaked the planted key.
    • Four of five privileges were needed. use_git never was, so the permission tailor proposes dropping it.
    • About 1,350 model calls in total, 420 to 680 credits depending on the Sonnet version GitLab ran.

Google Cloud

The report is a static site in an nginx container built for Cloud Run. The deploy code is in the repo (Dockerfile, deploy/cloudrun.sh and the manual cloud-run job); it uses keyless Workload Identity Federation: the job's GitLab OIDC token is exchanged for short-lived Google credentials, so no service account key exists. It is not live yet: the Google Cloud billing account we had was closed during the event.

How we built it

Python 3.13 package with a CLI and a GitLab CI/CD component. Flows API (POST /api/v4/ai/duo_workflows/workflows, fix_pipeline/v1 with the failed pipeline URL as goal, developer/v1 with the issue), trace.jsonl for the session record, two custom flows with HumanInputComponent approval gates and DeterministicStepComponent fetches, a project skill in skills/work-trial/SKILL.md, OpenTelemetry export for GitLab Observability, and a static report site served by GitLab Pages, with a Cloud Run deploy job. The privilege map is generated from GitLab's own tool registry.

Challenges

  • The platform blocks imports and forks for participants, so the shadow project is built through the commits API from a public copy of linkding's history.
  • Fix CI/CD Pipeline started through the API on a task branch stopped after its diagnosis with a missing-context error. It needs the merge request and the pipeline passed as additional_context, so the runner opens a merge request for each red branch first, the way a developer would meet the failure.
  • The Developer flow edited files and left them uncommitted until the goal asked it to push and open a merge request. Flow sessions also had no Python until the shadow project got an agent-config.yml with the same toolchain as CI.
  • Calibration caught bugs in our own graders before any agent was judged: a worktree pointer file that showed up as a changed file, and a reference fix that missed dependency changes.
  • Small samples: solve rates count tasks, not runs, and every verdict carries its interval.

What's next

Trials on private repositories, re-trials on every flow edit through the AI Catalog sync component, and probation wired to the flow's own triggers.

Path and autonomy

Path A (Start Fresh), Assisted: a person starts every trial and approves every verdict and permission change.

Links

Built With

  • gitlab
  • gitlab-ci
  • gitlab-duo
  • google-cloud-run
  • nginx
  • opentelemetry
  • pytest
  • python
  • uv
Share this project:

Updates

Submission history