authz.lock

A lockfile for who can access what in your API: a CI check that catches authorization leaks (IDOR) in AI-written FastAPI pull requests, proves them with a generated test, and explains the root cause.

Comment

Inspiration

I kept seeing the same bug in code written by coding agents. A refactor moves a database lookup into a helper and the ownership check doesn't come along. Every test still passes. The endpoint returns 200, just to the wrong user. This is IDOR, which OWASP calls Broken Object Level Authorization, and it's number one on the OWASP API Top 10.

Reviewers miss it because the diff looks like cleanup. Unit tests miss it because nobody writes "user B can't read user A's item" for every endpoint. Dependencies have lockfiles that make any change visible in review. I wanted the same thing for authorization: write down who can do what, and make a pull request explain itself when that changes.

What it does

authz.lock is a CI check for FastAPI apps.

  1. It logs in as every kind of identity: two normal users, a superuser, an inactive user, a deleted user, anonymous, and forged, expired and alg:none tokens.
  2. It calls every endpoint against every other user's data, on a fresh copy of a seeded database for each probe. Each user's rows carry a random canary string, so a leak is detected by finding someone else's canary in a response, not by guessing.
  3. It saves the outcome of every cell (identity x operation x target x request variant) as a lockfile. On the main demo app that is 603 probes.
  4. On a pull request it builds the same lockfile for the PR branch and diffs the two. Six fixed invariants decide PASS, FAIL or NEUTRAL (read isolation, write isolation, privilege fields, invalid identities never allowed, and so on). No language model decides the verdict.
  5. When a PR fails, an agent on NVIDIA Nemotron Super writes a pytest regression test and keeps repairing it until proof.py accepts it: the test must pass on the base branch and fail with an AssertionError on the PR. Those runs happen in Nebius Sandboxes. Nemotron Ultra then writes a short root cause and fix, with OWASP/CWE citations found by Tavily.
  6. The result is a PR comment and an HTML report showing the identity x operation matrix with the changed cells in red.

The part that makes this work on an app it hasn't seen is the onboarding agent. To probe an app you need users and data in it, and a manifest saying who owns which rows. Writing that seeder by hand for every app is what stops tools like this from being adopted. A Nemotron Super agent reads the app's OpenAPI and source, writes manifest.json and seeder.py, runs them through the harness, and repairs them until the harness gates pass. On the second app, eladgunders/fastapi-todos, it did this with 0 human edits to its output.

How we built it

  • Deterministic core (authzlock.py, probe_worker.py, probe_lib.py): Python, Starlette's TestClient, Postgres
    1. Each probe runs against a fresh clone of the seeded database. A test enforces that the core imports nothing from the agent layer.
  • Agent layer (agent/): the OpenAI Python SDK pointed at https://api.tokenfactory.nebius.com/v1/, a tool loop with forced tool_choice, file writes jailed to .authzlock/artifacts/, a fixed set of command templates instead of a shell, a JSONL cost log on every call and a hard dollar cap per run (default $1).

  • Models on Nebius Token Factory:

    • nvidia/nemotron-3-super-120b-a12b runs both agent loops (onboarding, regression tests). 488 calls in total, $1.79.
    • nvidia/Nemotron-3-Ultra-550b-a55b makes one root-cause call per failing PR. 24 calls, $0.05.

  • Nebius Sandboxes (sandbox_run.py, contree-sdk 0.3.6): one snapshot per app and base commit with Python, Postgres and the app's pinned dependencies. Every regression proof runs in a disposable VM forked from it, with networking off and no environment variables. 4 proofs take 12.2 s in parallel against 44.7 s run one after another on my laptop.

  • Tavily: one search per FAIL finding, restricted to owasp.org, cwe.mitre.org and the OWASP cheat sheets. Results are treated as quoted data and only go into report prose.

  • GitHub Actions, split three ways so API keys never share a process with PR code: authz-probe runs on pull_request with read-only permissions and no secrets; authz-comment runs on workflow_run and only renders the artifact; authz-agent holds the Nebius key and never runs on PR events. Every action is pinned by SHA.

  • GitHub Pages: replay.py recomputes all 51 recorded verdicts and proofs from committed files with the standard library only, and publishes the HTML reports.

Everything here was built during the submission period. The repo starts on 19 Oct 2026 with the deterministic core prototype (written 25 Sep, also inside the period). The agent layer, Sandboxes runner, Tavily step, GitHub Actions, benchmark, second app and reports were all added after that.

Challenges we ran into

  • The agent kept breaking working output. On the second app, Nemotron Super often rewrote a working manifest one last time before calling done, and the rewrite was broken. It took 5 onboarding runs. Each failure became an engine fix with a test, never a hand edit of the agent's files: validate on every write, run the seeder in its own process, and keep the last artifacts that passed instead of the last ones written.
  • Async seeders and closed event loops. The second app uses async SQLAlchemy. The agent's seeder left the app's engine bound to a closed event loop, so every login returned 500. Seeding now runs in a separate process.
  • Postgres inside Sandboxes. The guest kernel has no SysV shared memory, so stock initdb fails. A 20-line LD_PRELOAD shim plus shared_memory_type=mmap fixed it (sandbox/shmshim.c).

  • My first benchmark missed 2 of 11 bugs. Both passed another user's id through an optional ?user_id= query parameter or a categories_ids list, and the probe never sent optional params. I wrote 4 new breaking PRs and 1 harmless one and committed them before writing the fix, so they would work as a held-out check rather than a tuning set. The fix took the original set from 9/11 to 11/11 and the held-out set from 1/4 to 4/4, with 0/4 false positives.

  • Keeping keys away from PR code. The probe imports PR code in the same process. The fix was structural: the job that runs PR code has no secrets at all, and the job that has the Nebius key never runs on PR events.

Accomplishments that we're proud of

  • The verdict is reproducible without trusting me. Anyone can run python replay.py site with no key, database or model and get the same 51 verdicts.
  • Benchmark: 11/11 original breaking PRs, 4/4 held-out breaking PRs, 0/4 false positives on harmless PRs across two real open-source apps.

  • A second app with zero human edits to the agent's manifest and seeder: fastapi-users JWT, async SQLAlchemy, integer ids, a link table and shared rows with no owner. 693 probes on 17 of its 20 operations.

  • Every regression test is checked, not trusted. In the held-out run, proof.py rejected two first attempts (one failed to collect, one failed on base too) and the agent fixed both on a second run. 6/6 ended up proven.

  • Cheap: a passing PR costs $0 in model calls. Proving a failing PR costs about $0.073, including failed attempts. Probing takes 64 s per PR on app1 and 103 s on app2.

  • The whole project cost under $2 of Token Factory credit so far.

What we learned

  • Let the model write things that can be checked, and let plain code do the checking. Every time I gave Nemotron a gate it could fail against (the harness gates, proof.py), its output got better through repair. Every time I trusted its last message, it broke something.
  • A held-out set is worth the extra hour. Writing those 5 PRs before the fix is the only reason I can say the fix generalises, even on a small sample.
  • Sandboxes work best when you stop treating them as a machine you log into. Build one snapshot, fork a fresh VM for every proof, and throw it away.
  • Bigger isn't always needed. Super handled the tool loops reliably (14/14 tool calls well formed on the first onboarding run), so I kept Ultra for the one call per PR where a careful explanation matters.

What's next for authz.lock

  • Run the probe itself in Nebius Sandboxes, sharded across parallel VMs. Today only the proofs run there; the probe runs in the CI runner.
  • Recognise foreign references by type and schema, not only by <x>_id field names, so fields like owner or assignee get probed too.
  • Close the loop: have Nemotron Ultra propose a patch, apply it, re-lock, and show FAIL turn into PASS with the regression test now green.
  • Support Django REST Framework and Flask, plus multi-tenant rules (org membership), which are out of scope today.
  • Package it as a reusable GitHub Action that any FastAPI repo can add in one file.

Built With

nebius-token-factory, nvidia-nemotron, nemotron-3-super, nemotron-3-ultra, nebius-sandboxes, tavily, python, fastapi, postgresql, sqlalchemy, pytest, openai-python-sdk, github-actions, github-pages, docker

Testing instructions for judges

No account, key or login is needed for any of this.

  1. Reports (2 minutes): open https://mathuryashash.github.io/authzlock/ . The index lists every recorded PR with its expected and actual verdict. Open app1-pr-idor-refactor to see the identity x operation matrix with the red cells. Click the FAIL cell in row GET /items/{id}, column user_a: it shows ALLOW 200 with leaked canaries from user_b.
  2. Live demo PRs:
    • https://github.com/mathuryashash/authzlock/pull/1 is an IDOR refactor. The github-actions comment says ❌ FAIL, I1 read-isolation on GET /items/{id}. The probe check log shows the real run.
    • https://github.com/mathuryashash/authzlock/pull/2 is a harmless pagination change and gets ✅ PASS.
    • The regression test, Sandboxes proof and Nemotron Ultra root cause for PR #1's change are in results/agent_regress2/pr-idor-refactor.comment.md. They come from the agent workflow, which has the Nebius key and so never runs on PR events.
  3. Recompute every verdict yourself (any OS, Python 3.10 or newer, standard library only, a few seconds): git clone https://github.com/mathuryashash/authzlock && cd authzlock && python replay.py site It ends with 51/51 verdicts reproduced; 46/51 equal expected. The 5 that differ from "expected" are the probe v0 benchmark misses, kept on purpose so the before/after comparison stays honest. The command exits 1 if any recorded verdict fails to reproduce.
  4. Full live run (Docker Postgres; the agent steps need a Token Factory key): see Quickstart in the README.

The demo URL and PRs will stay public and unchanged until judging ends on 15 Dec 2026.

Feedback on Nebius Token Factory, Sandboxes and NVIDIA Nemotron (required)

I used Token Factory for every model call (488 to Nemotron 3 Super, 24 to Nemotron 3 Ultra), Nebius Sandboxes for every regression proof, and kept a running log while building (FEEDBACK.md in the repo). This is the short version, ordered by how much time each item cost me.

Token Factory inference

What worked. The OpenAI SDK with base_url worked unchanged. Forced tool_choice was honoured on every turn: on the first onboarding run, 14 of 14 calls returned a tool call and none had malformed JSON arguments. Latency was 2 to 5 s per turn at 10k to 16k prompt tokens. That first run cost $0.051. Tool calling on Super was reliable enough that I never needed my planned fallback to Ultra for the agent loop. Time to hello world once I had a key was minutes.

What needs work.

  1. Billing worked first time. My Indian card was accepted at Token Factory billing with no extra steps, even though other Indian participants reported declines on the forum. Saying up front which card types work (and offering a prepaid top-up) would save others the worry.
  2. Model ID casing is inconsistent. nvidia/nemotron-3-super-120b-a12b is all lowercase while nvidia/Nemotron-3-Ultra-550b-a55b is mixed case, and third-party catalogs list both spellings. A wrong case is an easy mistake to make in config. Either accept IDs case-insensitively or normalise them to one style.
  3. Prices aren't machine-readable from the API. I could only find per-token prices for Super and Ultra by reading pages, and some of them were third-party aggregators. My per-run dollar cap depends on numbers I copied by hand. Adding price fields to /v1/models would make cost caps reliable.
  4. No cached-token reporting. Agent loops resend a growing history, so prompt caching matters a lot for cost, but the usage object doesn't report cached prompt tokens.
  5. Ultra returns its reasoning inline. I strip everything before </think>. A documented flag to turn reasoning output on or off would save output tokens on calls where I only want the answer.

Nebius Sandboxes

What worked. The immutable image model is exactly right for proving a test on two branches: build one snapshot with disposable=False, then fork a disposable VM for every proof. Uploading a git archive tar through files= made "only run the committed tree" trivial. 4 of 4 proofs matched my local results exactly, and the set took 12.2 s in parallel against 44.7 s locally. Python 3.14 images worked, and the SDK ran fine on 3.13 despite the docs pinning 3.12. Time to hello world after activation was about 15 minutes. I'd build with it again.

What needs work.

  1. 403 until the beta activation email arrived. My key and project worked for inference, but every Sandboxes call returned 403 until activation. The 403 body didn't say "Sandboxes beta not enabled for this project". A specific error message, and a published turnaround time for access (or automatic access for hackathon participants), would have saved a support round trip and let me commit to Sandboxes earlier.

  2. Transient 500s. I saw them on POST /instances and on GET /operations/{id} while polling, about 1 in 5 spawns at worst. The SDK retries 429s and timeouts but not 5xx. Worse, a 500 while polling makes the SDK cancel an operation that was running fine (the record says "Cancelled by user"). I added my own bounded backoff. Polling errors should be retried inside the SDK and should never cancel the operation.

  3. The immutable-run model surprised me. I expected a box I could exec into several times. It took a re-read to learn that every run() is fresh compute and nothing persists unless disposable=False returns a new image, so a background service like Postgres has to start in the same command as the test. The docs explain it once you find the page; one line in the quickstart would save the detour.

  4. Network egress is open by default. networking.enabled=false is documented on the security page, but contree-sdk 0.3.6 doesn't expose it as a run() parameter, so I inject it into the request body myself. For an environment meant for running agent-written code, I'd suggest off by default, or at least an SDK flag. I verified the setting in the server-side operation records.

  5. No SysV shared memory in the guest. shmget returns ENOSYS, so stock Postgres initdb fails. My workaround is a 20-line LD_PRELOAD shim plus shared_memory_type=mmap. Also, python:*-slim images have no UTF-8 locale, so initdb picks SQL_ASCII unless you pass -E UTF8 --locale=C.UTF-8. A short "databases in Sandboxes" recipe in the docs would help a lot of people testing web apps.

  6. Image import naming. images.use('python:3.9-slim') returns 404 until the image is imported. images.pull resolves tags only and import_from is the call that works. The names are easy to confuse.

  7. One cold proof took 196 s against about 17 s for its siblings, with no error. I couldn't reproduce it in 3 later runs, so this is a data point rather than a bug report.

NVIDIA Nemotron

Nemotron 3 Super wrote working seeders, manifests and pytest regression tests for two different FastAPI codebases. On the IDOR PR it read the app's own test helpers and wrote an idiomatic test using the canary. Nemotron 3 Ultra named the exact dropped lines in the diff and cited CWE-639 and OWASP API1:2023 correctly.

Where it struggled, and what would help:

  • It often rewrote a working file one more time before calling done, and broke it. I now keep the last artifacts that passed. A model that stops when the tool reports success would cut about 30% of tokens on my loops.
  • It guessed file paths (tests/conftest.py instead of todos/tests/conftest.py) and dropped the /api/v1 prefix until both were in the first message. That's partly my tool design; I should add a list_dir tool.
  • The costly failures were runs that spent all 24 turns rewriting a test pytest couldn't collect. Reading the 422 response body before rewriting would have saved most of them.

Would I build with Token Factory, Sandboxes and Nemotron again? Yes. Total model spend for the whole project was under $2, and forkable snapshots are the right building block for "prove this test on two branches".

Built With

  • docker
  • fastapi
  • github-actions
  • nebius-sandboxes
  • nebius-token-factory
  • nvidia-nemotron
  • pytest
  • python
  • tavily
Share this project:

Updates

Submission history