Fleet

• 

Inspiration

At a twenty-person company, the person with the most context often becomes the project manager without the title. Every bug report, document request, billing issue, and customer follow-up reaches that person because they are the only one who knows who should handle it.

Adding AI agents creates more workers, but it does not solve the routing problem. Most agent systems also rely on the model saying it is confident, even though that confidence may not reflect how well the agent has performed on similar work.

We built Fleet around a different question:

What if people and agents shared one assignment pool, and routing was based on measured outcomes instead of self- reported confidence alone?

## What it does

Fleet is a Slack-first dispatcher for a workforce made up of people and specialized agents.

An employee can submit work through a Slack message shortcut, /fleet, or an app mention. Fleet separates the request into discrete tasks, classifies each task into a fixed taxonomy, assigns a risk tier, and decides whether:

• An agent may complete it unattended.
• An agent may prepare work for human review.
• The task must remain with a person.

Fleet currently uses three risk tiers:

• Tier 0: low-risk work may run unattended.
• Tier 1: an agent may draft or propose, but a person must approve it.
• Tier 2: software never attempts the task.

The dispatcher is deterministic code rather than another agent. Its central decision is:

[ \text{effective confidence}

\text{model confidence} \times \frac{\text{approvals}+2} {\text{approvals}+\text{send-backs}+3} ]

This combines Gemini's structured assessment with the selected worker's observed record on that exact task class.

Human review closes the loop. Approving or sending work back changes only that agent-task pairing. A send-back also stores the reviewer's correction for future attempts. Slack then displays a Learning Receipt showing:

• Track record before and after
• Effective confidence before and after
• The applicable risk threshold
• Whether the pairing remains eligible for agent routing
• Whether the reviewer's guidance was saved

Fleet's web application acts as the control plane:

• Work contains decisions that require a person.
• Activity explains what arrived and how it was routed.
• Automations shows the mixed human/agent workforce and its track record.
• Connections distinguishes deployed integrations from verified runtime activity.

## How we built it

Fleet uses Gemini 3.7 Flash through Google's GenAI SDK on Vertex AI for structured task separation, classification, and schema-constrained argument extraction. Gemma 4 26B independently evaluates each classification as a second reader.

The production architecture runs on four Cloud Run services:

• fleet hosts intake, general agents, APIs, and the web control plane.
• fleet-operator is a private service containing the only agent allowed to modify the back office.
• fleet-connectors verifies Slack requests and owns no model or business-system credentials.
• acme is a separate synthetic back-office system with its own database and per-action permissions.

Five Pub/Sub topics separate intake, human actions, agent task lanes, Slack notifications, and dead-letter handling. A filtered subscription ensures privileged operator tasks can only reach the isolated operator service.

Firestore stores requests, tasks, routing decisions, audit evidence, and outcome calibration. Slack connector replay claims and delivery receipts live in a separate named Firestore database, preventing the public connector from accessing Fleet's operational state.

Additional controls include:

• Model Armor for inbound prompt-injection screening and approved outbound sensitive-data checks
• Secret Manager and service-specific IAM for capability-scoped credentials
• OpenTelemetry and Cloud Trace for one trace across intake, routing, execution, and review
• React and Vite for the live control plane
• FastAPI for service boundaries and authenticated mutations

The local development path uses the Firestore emulator and recorded model responses, so the complete workflow can be reproduced without a Google Cloud project or API key.

## Challenges we ran into

### Model confidence was not a probability

Requests with very different levels of difficulty repeatedly received similarly high confidence scores. We learned that self-reported confidence was useful as an input, but unsafe as the final decision.

That led to Fleet's outcome-calibrated routing equation.

### Making separation of concerns real

It was easy to draw multiple agents in an architecture diagram. It was harder to ensure they actually had different identities, credentials, delivery paths, and failure boundaries.

We separated the Slack edge, text-reading triage, general agents, privileged operator, and external back office into enforceable Cloud Run, IAM, Secret Manager, Pub/Sub, and database boundaries.

### Reliable Slack interaction

Slack expects interactive requests to be acknowledged quickly, while model calls and agent execution may take much longer. We had to acknowledge first, enqueue work through Pub/Sub, and update durable cards asynchronously.

We also had to handle replayed events, concurrent button presses, stale task versions, ambiguous provider responses, connector restarts, and failed deliveries without committing the same human action twice.

### Learning from the correct worker

After a task was reassigned, review initially risked changing the record of the wrong worker. Tasks now persist performed_by, and only a verdict on actual agent output changes calibration. Delegating an attempt does not improve the agent's score by itself.

### Turning the frontend into a product

The original UI explained a scenario but did not feel like something employees would use. We repositioned Slack as the employee-facing operating surface and rebuilt the web application as a control plane for decisions, evidence, workforce policy, and integration health.

## Accomplishments that we're proud of

• A real Slack workflow from intake to agent execution, review, send-back, learning receipt, and human reassignment

• Outcome-based routing that visibly changes after a real human verdict reading agent

• Deterministic, inspectable routing decisions with explicit arithmetic and reasons

• Failure handling that returns unresolved work to a person instead of silently dropping it or retrying forever

• One trace ID connecting activity cards, audit evidence, and Cloud Trace across services Most importantly, routing a task to a person is treated as a correct outcome—not as an agent failure.

without introducing a separate feedback workflow.

We also learned that strict arithmetic can create useful safety guarantees. With Fleet's prior and thresholds:

• A new agent-task pairing cannot immediately automate unfamiliar Tier 1 work.
• A classification contradicted by the second reader cannot clear an automation threshold.
• A failed attempt can lower eligibility without waiting for an administrator to update a prompt.
• A below-threshold agent can recover only through supervised attempts and satisfactory human verdicts.

The most useful agent memory was not a conversation transcript. It was two counters, several concrete corrections, and the context required to apply them to the next decision.

## What's next for Fleet

• Multi-workspace Slack OAuth with encrypted installation records
• Native Linear, Asana, and Gmail adapters so requests and outcomes stay connected to existing work systems
• Organization-specific taxonomies, risk policies, and automation thresholds
• Dependency-aware workflows where one task can safely unlock another
• Longer-term agent evaluations across task classes, teams, and policy revisions should—and should not—be trusted to do.

Built With

• gcp
• genaisdk
• python
• react