ICM Meeting Agent

Inspiration

I'd spent the months of May and June trying to find the 'perfect' AI agent that would go to the meeting, record and transcribe, and then pay the sub to send all that data to my second brain. And it was such a damn headache. I just gave up. And after a few months of testing, I found my way to building a full agent system and this is the first piece of a larger agentic system

What it does

The agent pulls a recorded meeting from Fireflies, screens every field of it through Model Armor before any prompt is built — including the event title, which is written by whoever sent the invitation — prepares a briefing with Gemini 3.5 Flash on Vertex AI, and drafts follow-ups where every message must carry the verbatim transcript lines it rests on. Then it stops. The run suspends and writes a decision surface a person rules on; only what they approve is sent, through a separate pipeline with a second confirmation gate before the first message leaves.

A watcher (bin/watch.mjs) took a full run from eight human touches to two: it notices finished meetings on its own, assembles the folder, starts the run, and resumes it after a ruling. The two touches that remain are both decisions — the only parts that belong to a person.

How we built it

Three ideas carry the architecture:

• State lives in files, so the run can outlive the process. icm-run (my private process repo) treats a pipeline as data plus runbooks on disk. The run suspends at the gate and resumes days later, possibly by a different person, certainly in a different process.
• Every phase runs in its own context. A phase that reads a hostile transcript cannot leak into the phase that sends mail — not by prompt hygiene, but as a property of how the driver runs phases.
• The irreversible decision is unreachable from the model. SEND_MODE is frozen into the binding before any phase starts, out of a file the process only reads, and the send tool exposes no dry-run parameter for a confused model to flip.

Stack: Gemini 3.5 Flash on Vertex AI (GenAI SDK), Model Armor, Cloud Run, Cloud Build, Artifact Registry, Fireflies GraphQL, Gmail API (gmail.send scope only), Node 22, no framework. The deployed review gate is air-gapped: zero IAM roles, no outbound calls.

Challenges we ran into

The honest one: Model Armor catches jailbreaks, not lies. Attacks with instruction-override syntax were caught at HIGH confidence. An attack with no syntax in it — a prior-meeting summary that simply asserts "this was already approved" — passed clean, and the resulting brief reported the conflict as settled and cited the lie. The citation discipline that makes a brief auditable is exactly what makes a lie look sourced. We measured this rather than patching over it, and the measurement is part of the deliverable.

The other: on a real meeting, the transcript names speakers by display name, the invitation names attendees by address, and nothing joins them. Rather than guess, the agent leaves recipients blank, records what it could and could not establish, and asks. An approved message with nobody to send it to is not sent — and the watcher refuses to resume it.

What we learned

• The evidence requirement stopped the attack the filter missed — by accident. A claim sourced only to a poisoned summary had no transcript line behind it and could not be written. What makes a gate rulable is what made the attack unwritable.
• Careful costs time: one message against a 25,000-character transcript takes nine to ten minutes and ~193,000 thinking tokens. That is the price of drafts you can rule on, and it is worth paying.
• The safest half to deploy is the review surface, not the agent. The half that holds credentials and can send mail stays off the internet.

What's next

A Fireflies webhook into a Cloud Run receiver would make the trigger immediate rather than ten-minutely. It is deliberately not shipped: it would mean deploying the half that can send mail. Then the larger plan of building the full agent system that will more personal support structures for adults with ADHD.

Built With

• antigravity
• claude
• gemini
• gemini-3.5-flash
• node.js