Autonomous PM Agent

Inspiration

I'm an engineer. Most mornings I open a ticket that says something like "fix the news page" and nothing else. No context, no acceptance criteria, requirements that changed since a call nobody minuted. Reconstructing what the ticket actually means costs more energy than building it. As a lead I have the opposite problem: I can't see where the team really is without pinging everyone.

The annoying part is that all of this information existed. Someone said it out loud in a meeting, and then it evaporated before it reached the ticket. So I built an agent that catches it at the source.

I made one design bet up front: no approval prompts. Anthropic's engineers measured that people approve about 93% of agent confirmation prompts, which means oversight decays into a rubber stamp. This agent acts first, proves everything with citations, and makes undo one click.

What it does

A product call ends. Fathom fires a webhook, and pm-agent, which runs around the clock on Cloud Run, does the PM's next two hours in a few minutes:

• It reads the call. Every action item needs a verbatim quote behind it, or it gets dropped.
• It checks Linear, the docs and the actual codebase before filing anything. A re-raised issue becomes a note on the existing ticket instead of a duplicate.
• It files tickets I'd actually want to receive: why this matters, what was said (with who said it and when), acceptance criteria as checkboxes, an Investigation section pointing at the file and line where the bug probably lives, with an honest confidence label, and a frontend/backend label it works out from the call.
• It schedules its own follow-through: a dependency-ordered chain of checks (underway by Tuesday, then a PR exists, then it landed), each with a date and a declared consequence. Bad news waits for its deadline. Good news doesn't: when an engineer drags a ticket to In Progress, the pending check resolves in seconds off the webhook.
• You can just talk to it in Slack. It reacts 👀 instantly, drops "On it…", and that same message turns into the answer. Tell it "from now on, assign frontend bugs to Priya" and it stores the rule with your message as the source, then cites that exact message the next time it applies the rule.
• Every action carries a one-click revert. That's the deal behind never asking permission.
• Everything it does is visible on a live timeline: time across, work down (heard, understood, did, watching, learned), with a replay scrubber and a plain-sentence story panel on every node.

It also posts a morning standup and writes sprint reports where every single claim carries a citation. That's a gate, not a habit.

How we built it

Five ADK LlmAgents on Gemini 3.5 through Vertex AI (extractor, reconciler, planner, reporter, reviewer), with schema-enforced output and per-agent tool allowlists. Gemma rides along as a second Google model, triaging transcript segments and classifying Slack intents through the GenAI SDK.

The orchestrator is not a framework. It's a Firestore task graph: leases via transactions, depends_on edges, retry with backoff, and plans that materialise atomically, so a plan is never half-scheduled. Cloud Scheduler ticks it once a minute, webhooks short-circuit it the moment reality moves, and Cloud Run scales it down between ticks (one warm instance stays up through the judging window).

Between every model output and every external effect sit deterministic gates: quotes must exist in the transcript, identifiers are re-fetched before they may be cited, owners must be on the roster, a priority only escalates if someone actually said escalation words, a date is only set if someone actually spoke it, and daily caps plus quiet hours bound the blast radius. GATES.md is generated from the code, so the documentation cannot drift from the enforcement. Writes are intent-before-effect, with idempotency keys and stored revert payloads. The timeline is self-contained vanilla JS with no CDN; it renders with the network down.

Challenges we ran into

Honestly, the biggest one was quotas, and it hit on deadline week. The free Gemini tier turned out to allow 20 requests a day for the model I needed. I made a fresh API key and ran into AI Studio's prepayment wall instead. I ended up moving all inference to Vertex AI so it just bills the project, and weirdly I'm glad it happened, because the retry and backoff code I had written "just in case" finally got tested for real.

Then the planner started breaking its own JSON on Vertex. Around 14,000 characters of plan output, malformed, six runs in a row. I was sure the thinking budget was the culprit, turned it off, and it broke again. So I stopped hunting the culprit and changed the rule instead: a plan that doesn't parse now falls back to a boring deterministic follow-up chain, labelled as such. The checks get scheduled either way, which is what actually matters.

ADK also has an internal tool called set_model_response that carries structured output. My per-agent tool guard was blocking it, silently, so agents returned nothing at all while every unit test stayed green. Only a live run caught it.

The one I didn't expect to be hard: making the agent sound like a person in Slack. The first posts were painfully robotic, so tone became actual code, with tests. There's now a function that knows "set up" is a phrasal verb, written the day the agent proudly announced "the up a regression test".

Accomplishments that we're proud of

The thing I'm proudest of is that the agent manages its own future. When it files a ticket after a call, it also decides what it should check later: is the work underway by Tuesday, is there a pull request by Thursday, did it land by Friday. Those become real tasks in its own queue, with dates, dependencies and a consequence attached to each one, and a dumb once-a-minute heartbeat executes whatever comes due. Nobody tells it to follow up.

That's close to how a person carries work in the back of their head. You don't consciously re-decide every morning to check on the thing you're waiting for. The intention is just there, and it surfaces on the right day. Here the intention is a document I can open and read, and when reality moves early, say an engineer drags the ticket to In Progress, the webhook resolves that worry four days ahead of schedule, in seconds.

The numbers held up too. Across six eval runs: 0 fabricated identifiers, 100% report citation coverage, 0 invalid plans materialised. The runs I'm proudest of are the worst ones, the quota-starved ones where judgment scores dropped, because every guarantee held anyway. It degrades to silence, never to lies.

And it's all live-proven off real recorded calls against my real product: it caught a re-raised issue and updated the existing ticket instead of duplicating it, its investigation sections point at real files and lines, and a call becomes filed, cited tickets in minutes, not at the end of somebody's day.

What we learned

I went in thinking the hard part would be getting good judgment out of the model. It wasn't. Gemini's calls were reasonable from day one. The hard part was earning the right to act on them, and that turned out to be ordinary engineering: a gate that re-fetches every identifier before it can be cited, a rule that a priority only escalates if somebody said "urgent" out loud, a revert payload stored before every write. I spent maybe a tenth of this project on prompts and the rest on the machinery around them.

One split saved me more than once: the model gets judgment, never authority. The planner proposes a schedule, and code decides what actually materialises. So when the planner spent a day producing broken JSON, the cost was a boring fallback plan instead of a broken product. Anything a model writes needs a deterministic floor under it, because it will eventually write nonsense, and it will pick the worst possible day to do it.

I also stopped believing in approval prompts. People approve about 93 percent of them, so they protect nobody. They just slow the agent down and train the human to click yes. What actually made me comfortable watching this thing act on its own was undo: one click, on every action. I check the revert count on the dashboard the way you check a smoke detector. Glad it's zero, glad it's there.

What's next for Autonomous PM Agent

More trackers (Jira, GitHub Issues) behind the same connector seam. Deeper PR awareness: it already checks that pull requests exist, and next it should notice ones going stale. A brain that spans projects, so a rule taught once holds everywhere. Lessons from the daily review feeding prompts automatically instead of just advisorily. A prompt-injection screen ahead of extraction, because transcripts are untrusted input. And human review for the rare class of action where a revert isn't enough.

Built With

• cloud-run
• cloud-scheduler
• fastapi
• firestore
• gemini
• google-adk
• javascript
• linear
• pydantic
• python
• secret-manager
• slack-api
• vertex-ai