FrontDesk

Agentic operations workspace for customer support.

Comment

Front Desk

An agentic workspace for customer operations.

Inspiration

Customers want problems solved quickly and one-on-one. One-on-one support is expensive for a small team and impossible at scale.

Each customer has different needs, and those needs change as the business changes. Support systems treat every request as a new case, pass it between people, and answer with a promise to get back to you in a couple of weeks.

Most of the work behind a request does not need the customer waiting on it. Front Desk gives each customer a dedicated agent that keeps their history, does the work in the systems where the answer lives, and runs asynchronously.

What it does

Front Desk gives every customer a dedicated agent and a permanent place holding everything about them: their profile, their documents, the emails they have sent, and the work open for them right now. The business works out of that place, and so does the agent. The agent writes the profile as the relationship changes, and the operator opens the same file and corrects it.

The operator can chat with the agent about a customer, or open a live voice session and talk to it.

The agent can:

  • read incoming customer email and decide what each message requires;
  • open a goal for a request and break it into assignments;
  • read and change records in connected systems, including orders, support cases, Jira issues, Slack channels, and Google Workspace files;
  • call the customer on Google Meet and hold a live conversation;
  • run application work during that call while the conversation continues;
  • pause a request that is waiting on the customer and resume it when they reply;
  • run a goal on a repeating schedule, so standing work continues without anyone starting it; and
  • hand a case back to a person when identity or intent cannot be decided safely.

Plugins connect the business's systems. Skills hold the procedures the agent follows in them. Both are installed into the workspace rather than built into the product.

How we built it

Front Desk runs three paths with different timing requirements. A conversation must stay responsive. Work can run for minutes and must survive a restart. Deciding what an incoming message means happens before either.

                            FRONT DESK

   Customer email                              Customer on Google Meet
        |                                              |
   IMAP IDLE                                     Chrome extension
        |                                              |
        v                                              v
   Intake agent  --> goal + assignments -->  Gemini Live conversation
        |                    |                         |
        |                    v                    coordinator
        |            ADK worker on Agent Engine        |
        |                    |                         |
        |               tool call over the             |
        |            authenticated Cloud Run gateway   |
        |                    |                         |
        +--------------------+-------------------------+
                             v
        Cloud SQL  |  Aqualabs store (MCP)  |  Workspace  |  Jira  |  Slack

Intake decides what a message means before any work exists

Most incoming mail is not new work. A customer acknowledges a reply, supplies a missing order number, changes a request, or asks about work already running. Creating a task per email produces duplicate goals, duplicate tickets, and duplicate actions on real orders.

Intake is a separate agent. It matches the sender to a customer, reads that customer's profile, conversation history, documents, and open goals, then returns one of four outcomes: record the message in history, resume an existing goal, create a new goal, or ask a human.

Mail arrives over an IMAP IDLE connection, so the backend is notified rather than polling. Each message is stored before a model sees it, and the routing decision and any failure are stored beside it. The implementation is in backend/app/email_agent.py and backend/tools/email_agent_tools.py.

Customer memory is stored as application data

Every customer has a durable record containing a profile, documents, conversations, and goals. The profile holds confirmed facts, current problems, preferences, and outstanding commitments. Intake adds what a new message contributes and leaves the rest in place.

An order worker reads the same profile the voice agent receives before a call, and a staff member can open and correct it. None of them needs an earlier model conversation in context to know the customer's address or what was promised last month. The implementation is in backend/tools/client_context.py.

Intake decides what a message means before any work exists

Before changing anything, the planner reads the existing task ledger and returns typed operations: create, reuse, retry, steer, or cancel. Work is split where there is a dependency or a wait. Finding an order and applying an authorized change belong in one assignment. Asking the customer for a missing order number is a wait, so the question is stored and the assignment resumes when the reply arrives, and no model session stays open for three days.

Business procedures are written as skills. The Aqualabs order skill requires an exact customer and exactly one actionable order before any mutation. An amount can filter results but cannot identify an order, and two plausible candidates produce a clarifying question. See backend/agents/goal_planner.py and backend/app/skills.py.

Workers load capabilities during the run

Front Desk uses one worker design on gemini-3.6-flash. Each assignment gets an isolated session containing its instruction, verified outputs from work it depends on, the selected skills, and a compact directory of available tool namespaces. The worker requests a namespace during the run and receives those tools on the next model step.

A task that only touches the store never connects to Meet, Jira, or Slack, and a failure in an unrelated integration cannot block it. A worker handling a delivery complaint starts with store access and loads Meet and mail when it reaches the call. The registry is in backend/tools/goal_tool_registry.py.

Intake decides what a message means before any work exists

Front Desk calls the customer when a case involves a confirmed fault, a repeated complaint, or a resolution the customer has not accepted. It creates an instant Meet space, emails the customer that link, and joins the meeting.

A browser worker handles the meeting mechanics: opening the correct space, watching participant state, arranging media, and leaving at the end. It drives the operator's own Chrome through an extension. gemini-3.1-flash-live-preview handles the conversation. Two macOS virtual audio devices connect the model to the call, one carrying received audio to the agent and one carrying generated speech into Meet's microphone input. If either device is missing, the call fails explicitly and never falls back to the operator's physical microphone.

The call supports barge-in. When the customer speaks while the agent is talking, the agent stops and listens.

  Customer speaks -----> Gemini Live -----> confirms the exact action
                              |
                              | bounded task, returns once accepted
                              v
                        coordinator ------> Aqualabs store
                              |                   |
     conversation continues   |             work runs
                              |                   |
                              +<--- result -------+
                              |
                              v
                    agent explains the outcome

Gemini Live uses synchronous function calling, so a voice tool that waits for an order workflow to finish stops the conversation for the length of that workflow. The dispatch returns as soon as the coordinator accepts the work. The operation runs on its own and reports back into the same call, so the customer can keep talking and can change the request before it completes.

Work delegated from a meeting reaches the Aqualabs store and nothing else. It cannot create another meeting or send unrelated messages. A worker's question returns to the same call, and the customer's answer resumes that task. See backend/meetings/coordinator_planner.py.

Credentials and tool implementations stay behind an authenticated gateway

The ADK reasoning loop runs in Vertex AI Agent Engine. For each run, a Cloud Run gateway gives Agent Engine the agent definition: instructions, model configuration, and tool declarations. Tool calls return through that gateway, which resolves each one against the current account and its capabilities, executes it, and returns the result with updated state.

Customer data, provider credentials, and every tool implementation stay inside the authenticated service. Intake and goal workers use different instructions and different tools in the same managed runtime, and the business logic exists in one deployment. See agent_runtime/agent.py and backend/app/agent_tool_gateway.py.

Business requirements are enforced as tool preconditions

The completion tool rejects a call-required goal that has no meeting record. The mail tools reject a written resolution on that kind of case before the call happens, and always allow the meeting invitation through. Each rejection names the operation required next, so the agent continues.

Shared state lets instances coordinate

An email notification, a browser connection, and an agent tool call do not necessarily reach the same process. Shared PostgreSQL records and notifications route account events and browser relays. Runtime leases stop two instances claiming the same protected work.

A worker may have sent an email or changed an order before its process stopped, so durable state alone does not make replay safe. Interrupted goals are paused at startup with their assignments and progress intact, and resumption begins from the recorded state.

Challenges we ran into

Enforcing a required step the model could skip

The resolution skill instructed the agent to call the customer when a case needed explaining. It usually did. Occasionally it investigated the order, wrote an accurate summary, and closed the case, which was the outcome the customer had already rejected by writing in a second time. A rule that exists only in an instruction can be skipped on any given run, and the failure appears only afterwards. The rule moved into the completion and mail tools as a precondition that names the next required operation.

Connection ownership competing with query traffic

Front Desk uses PostgreSQL advisory locks so two instances cannot run the same work twice. An advisory lock is session-scoped, so holding one holds a connection for the length of the job rather than the length of a query. For a single inbound email, the mailbox listener, the message lock, the agent session lock, the dispatched goal, and its session lock held five connections before any query ran. The pool was sized for short queries, and tool calls began timing out.

Leases now share a single connection on a separate engine, because distinct lock keys can be held on one session. Lease cost is constant per instance instead of growing with concurrent work, and the query pool was resized against the database the service runs on.

Repeat emails about one problem

A customer who has written twice will write a third time. Treating each message as new work produced parallel goals for one problem. Separating messages from goals, and having intake read open work before deciding, resumes the correct case.

Reaching the operator's browser from a cloud worker

Meet cannot be driven from Agent Engine, because the meeting needs a real Chrome session and real audio devices. A relay routes browser control to the operator's machine while the reasoning stays in the managed runtime. The meeting worker owns the tab, so no other task can use it.

Matching the deployed runtime to the tested one

The mailbox listener uses an IMAP call available only in a newer Python than the container image pinned. Every idle wait failed, the mailbox was marked failed, and live delivery fell back to retry polling while the tests passed locally. The image now matches the interpreter the tests run on, and the release checklist covers it.

Accomplishments that we're proud of

The system currently supports:

  • an IMAP IDLE mailbox listener that routes each message to one of four outcomes;
  • durable customer identity with profile, documents, conversations, and goals;
  • a planner that reuses, retries, steers, and cancels work as well as creating it;
  • assignments that pause on an external wait and resume when the answer arrives;
  • ADK workers on gemini-3.6-flash running in Vertex AI Agent Engine;
  • tool namespaces loaded during a run rather than declared upfront;
  • live customer calls on Google Meet with barge-in and streamed transcripts;
  • application work dispatched mid-call while the conversation continues;
  • business requirements enforced as tool preconditions;
  • credentials and tool implementations held behind an authenticated Cloud Run gateway;
  • runtime leases and shared notifications so multiple instances coordinate; and
  • interrupted goals paused with their progress intact for resumption.

The deployed worker is the same ADK application the local runtime uses. Agent Engine manages reasoning and sessions, Cloud Run holds the tools and credentials, and Cloud SQL holds the state that outlives a conversation.

What we learned

Most of the engineering was in the system around the model. Durable records let a customer's history outlive a session. A task runtime that can pause for days removes the need to hold anything open. Preconditions in tools make a required step impossible to skip.

Separating messages from outcomes is the decision the rest depends on. It prevents duplicate work, it lets a follow-up resume the correct case, and it makes the customer relationship readable to parts of the system that never saw the original email.

Requirements belong in code and judgment belongs in instructions. When a requirement is left in an instruction, the agent produces reasonable output that does not meet it, and nothing reports an error.

Connections held for ownership and connections used for queries have different lifetimes and should not share a pool.

What's next for Front Desk

The Aqualabs connector uses deployment-level credentials, and the call bridge depends on macOS virtual audio devices. Both need to be per-tenant and installable before another business can run this, along with managed connectors, team permissions, and retention controls.

Evaluation needs to move from component tests to repeatable end-to-end cases: a real message arriving, a real record changing, a real call completing. The failures that mattered here passed the test suite and appeared only in the deployed system.

The measure is customer requests completed without a person moving the work between systems.

Links

Built With

Share this project:

Updates

Submission history