CommSync

AI agents that run your business inbox - SMS and email.

Comment 2

Inspiration

We are a digital agency in Atlanta. We build websites, run SEO and ads, ship apps, and handle IT for other businesses. That also means managing a lot of phone lines and mailboxes, both ours and the ones we operate for clients.

That is the origin of CommSync. Not a market thesis. Twenty inboxes.

Robin Dunbar put a rough number on how many stable relationships a person can keep track of, around 150. Business owners can blow past that before lunch on a Tuesday. The line that kicked off the project is still the clearest description of the problem:

"The number of people I'm supposed to remember doubles every year. The number of inboxes I have to check hasn't."

The bottleneck is fragmentation. A reply lives in one inbox while the relationship lives across seven. Priya texts about an invoice on Tuesday and emails about it on Thursday. Most software treats those as separate conversations with separate unread badges.

CommSync treats them as one relationship.

A unified inbox solves part of that problem, but merging channels does not reduce message volume. Businesses still need to answer the 7pm text asking whether they are open tomorrow. They also need the system to know when not to answer, like when someone sends STOP, a message comes from a no-reply@ sender, or a teammate has already taken over the conversation.

That gets serious fast when a model can send a message to a customer under your business's name. A bad answer is not a demo failure. A real person receives it.

So CommSync puts the safety and control system around the model, not inside a prompt and not on a settings page nobody checks.

What it does

CommSync is a communications platform for businesses that rely on SMS and email.

Phone lines across three carriers sit behind one provider interface. IMAP and SMTP mailboxes, MMS in both directions, call logs, voicemail, and attachments all land in one organization-wide message store. Threads belong to the workspace. Read state, labels, snoozes, and mutes stay personal to each member.

The core rule is simple:

A conversation belongs to a contact, not a channel.

A contact is a person with multiple handles, including phone numbers, email addresses, and aliases. One thread can contain both SMS and email from the same person.

Identity merges are manual. A wrong merge can silently combine two customers, so CommSync does not let a model make that decision.

Agents

Agents are organization-configured responders that handle inbound messages on approved lines.

Each turn runs a native Gemini function-calling loop with access only to the tools that agent is allowed to use. Those tools include message search, thread reads, contact history, attachment reading, image inspection, calendar lookup, organization knowledge stored with pgvector, and external MCP servers connected to the workspace.

Each agent has its own service account. It does not consume a seat or appear as a normal workspace member, but normal channel permissions, message delivery rules, and audit logging still apply.

Before Gemini receives anything, fifteen checks run in a fixed order. Every check fails closed.

The system verifies the channel whitelist, requires a one-to-one thread, rejects empty threads, confirms the newest message is inbound, rejects triggers older than one hour, checks for human takeover, handles SMS STOP, checks existing opt-outs, inspects deterministic machine-mail headers, checks cached machine-sender classification, enforces quiet hours in the agent's timezone, caps consecutive agent turns, caps daily turns per contact, checks organization entitlement, and verifies credit balance.

STOP enters the permanent opt-out list before any model runs.

If the turn passes the gate, the agent gets at most ten function-calling steps. It must finish with exactly one terminal action: send a reply, escalate to a human, or decline with a recorded reason.

Prompt-injection defense

Every piece of third-party text is treated as untrusted input. That includes the trigger message, thread history, and stored contact memory.

Two screeners inspect the material before Gemini receives it.

The first is a deterministic pattern scanner that answers in under a millisecond.

The second is Gemma, an independent model family that reviews the same input.

A HOSTILE result ends the turn before the main model runs, so no reply is generated and no main-model credit is spent.

A SUSPECT result removes external write tools and sends any drafted response to a human for approval.

Tool results are screened again during the turn. External writes are rejected when their arguments were copied from untrusted contact text. Drafted replies also pass an outbound scan for leaked system instructions and credential-shaped text.

Any turn that performs an external write requires human review.

The audit record keeps the security decisions alongside the model run.

Turn limits and approvals

The consecutive-turn count comes from message history instead of a stored counter.

CommSync walks the last twenty outbound messages and stops counting when it reaches a human reply. That makes the message record the source of truth.

Approval sends use an atomic claim so two admins cannot approve and send the same response at the same time.

The system also runs the compliance gate again at approval time. If someone sends STOP after a draft enters the approval queue, approving that draft still cannot send it.

Agent memory

After a successful reply, the agent updates private memory for that contact.

Memory contains a rolling 150-word summary plus durable preferences, commitments, and open items. The memory update is a separate audited model run with its own cost record.

Read and write operations use the same lookup function.

When two identities merge under one contact, CommSync re-keys the memory row so the agent keeps one memory for that person.

Ask

Ask is agentic chat over the workspace's own communications.

It has nineteen tools. Retrieval combines keyword search and pgvector search with reciprocal rank fusion. Both retrieval paths run in parallel.

Ask can read attachments, inspect images, render charts and tables, calculate free and busy time from .ics invitations, search previous Ask conversations, and draft replies.

It cannot send those replies.

Ask also handles in-app help.

Documentation retrieval does not use a crawler or embedding index. The documentation is MDX compiled into the web application. The server exposes it through a static corpus route and caches responses with stale-while-revalidate.

Search uses field-weighted keyword scoring with a breadth multiplier. The documentation set is small enough that exact scoring is practical.

Background intelligence

Inbound email first passes through deterministic header checks.

Signals such as Auto-Submitted, Precedence, Feedback-ID, and ESP campaign headers can classify machine-generated mail before a model runs.

This matters because marketing messages often imitate personal email. A message like "Hey, I'm one of the founders, what made you sign up?" can still be part of an automated campaign.

Messages that survive the header checks go to Gemini for classification.

The verdict is cached per sender and versioned by taxonomy. When the classification taxonomy changes, the sender is classified again when the next message arrives.

Daily brief

The daily brief runs in two passes.

The first pass reads metadata for the day's messages and filters aggressively for recall. Its instruction is explicit: when uncertain between dropping and keeping an item, keep it.

The second pass loads full message windows only for the survivors and ranks them.

That keeps the brief cheap on high-volume accounts.

Ranking uses the cost of missing an item rather than the emotional tone of the message.

Agent statistics in the brief are calculated directly from turn records in code. The model never generates those numbers.

Action items

AI-extracted action items carry quote anchors.

Each item stores the source message that established the request and records whether the model actually cited that message. If it did not, CommSync can associate the newest inbound message for context, but it does not present that fallback as a verified quotation.

A to-do closes only when there is evidence that the workspace user acted.

Another message from the customer does not count as resolution. A follow-up usually means the opposite.

MCP and webhooks

CommSync is both an MCP client and an MCP server.

Its authenticated MCP endpoint exposes 76 tools, which lets external agents interact with the inbox through channel-scoped credentials.

Ten signed outbound webhook types cover the reverse direction.

External MCP agents cannot create CommSync Agents, change their configuration, or approve their turns.

Auditing and cost

Every model call writes an audit row and validates its output against a schema.

AI usage draws from an organization credit ledger denominated in dollars instead of abstract token counts.

A live agent reply costs about $0.0012.

That cost is visible in the product for each call.

The evidence layer

We assume at least one reviewer will inspect this project with an agent, so the repository includes machine-checkable evidence for the claims we make.

bun run eval:agent runs a 54-case prompt-injection corpus across four arrival paths, the fail-closed gate suite, and fourteen live agent turns against the same prompt module and Vertex path used by the application.

The command writes docs/agent-evals.md.

Published results come directly from that report.

The deterministic pattern scanner catches every attack in its known table in under a millisecond, but catches 0 of 5 held-out attacks.

Gemma catches 36 of 36 attacks from the table, 5 of 5 held-out attacks, and produces 0 false positives across 18 ordinary customer messages.

That held-out result is why the second model is worth roughly 1.9 seconds of screening latency.

The live behavior suite passes 13 of 14 cases. It records a 0% unsupported-claim rate, passes 4 of 4 injection-handling tests, and costs about $0.0011 per turn.

One case fails.

When asked for an invoice total, the agent escalates instead of reading the PDF because its instructions require billing questions to go to a human.

The failure stays in the report. An evaluation with nothing but green checks is advertising, not measurement.

bun run demo starts the system from a clean clone, launches Docker, runs migrations, seeds a fictional workspace, configures an agent on the office line, starts both applications, and executes one scripted inbound turn end to end.

CI runs the test suites and a secret scan on every push.

The judged deployment is pinned to release v1.0.0-hackathon.

How we built it

Google Cloud runs the deployed system.

There are two Cloud Run services.

The frontend runs on Next.js.

The backend is one Express process hosting 388 API endpoints, the Socket.io server, and 33 workers across 32 BullMQ queues. Those queues handle sends, IMAP IDLE, carrier reconciliation, classification, extraction, embedding, digests, agent turns, sweeps, and worker supervision.

The backend intentionally runs as a stateful singleton. Cloud Run has both minimum and maximum instances set to one, CPU stays allocated, and session affinity is enabled.

Cloud SQL runs PostgreSQL 16 with pgvector. It holds a 79-model schema and the semantic index.

Memorystore runs the queues.

Cloud Storage stores attachment bytes and serves them through v4 signed URLs.

Secret Manager holds credentials.

Cloud Build produces the deployment images, which avoids cross-building amd64 containers on an ARM development machine.

The codebase is about 289,000 lines of TypeScript with 2,185 tests.

Gemini and Vertex AI

The server reaches Gemini through Vertex AI using its service account.

There is no Gemini API key in the main application stack.

Gemma screening uses a separate Gemini API key restricted to that job. The screener has a kill switch and a fail-open path so a screening outage does not silently stop agent replies.

The AI layer uses two transports.

Agent execution and background AI features call the Google GenAI SDK, @google/genai, directly.

Structured output uses responseJsonSchema constrained decoding. Model tiers use separate thinking levels. The agent loop is implemented directly and preserves Gemini thought signatures between function-calling steps.

Ask and the composer writing tools use a streaming transport over the same Gemini models because those interfaces stream model output into the chat UI.

Embeddings

Semantic retrieval uses gemini-embedding-001.

Vectors are truncated to 1024 dimensions and normalized to unit length before storage in vector(1024).

There is no fallback embedding model.

Mixing vectors from different embedding models would make similarity comparisons unreliable, so an embedding failure is treated as a failure instead of silently substituting another model.

Documents and attachments

Gemini receives supported PDFs directly as document parts.

That preserves page structure and layout, which matters for files such as invoices and contracts.

Formats Gemini does not accept directly go through dedicated extractors.

That includes Word documents, spreadsheets, slide decks, RTF files, calendar invitations, and .eml messages.

Challenges we ran into

Vertex and Gemini expose different model catalogs

The model list available through Vertex AI did not match the broader Gemini API catalog.

We tested the live endpoint instead of relying on the catalog page. Several model IDs returned an HTML-shaped 404 through Vertex even though they appeared in broader documentation.

The deployed model tiers now point only to IDs that the Vertex endpoint accepts.

The tier abstraction keeps model selection in one place, so changing a model does not require touching every AI feature.

We also hit a billing trap on the Gemini API side. A valid API key can authenticate successfully while still having no prepaid Gemini API credit.

Authentication and billing failure look like two separate problems because they are.

A provider-spec mismatch appeared only at runtime

One streaming integration compiled and passed tests but failed on the first real stream with unsupported model version.

The installed transport packages implemented different revisions of the provider specification.

Pinning both packages to compatible versions fixed the stream.

Static type checking could not catch it because both packages were internally valid. They simply disagreed at the protocol boundary.

A native image module failed inside the container

Bun's isolated package store and Next.js compiled external imports could not resolve the platform binary required by the image pipeline.

The failure affected both image building and runtime execution.

Lazy loading the module and installing explicit platform dependencies fixed it.

Google Cloud defaults needed explicit configuration

Several defaults were wrong for this workload.

Cloud SQL requires the enterprise edition before custom machine tiers are available.

Memorystore needs noeviction for BullMQ. Evicting queue data can corrupt queue state.

Database migrations through the SQL proxy can exceed the client's default five-second timeout.

Organization policy changes also need care. Resetting a policy restores inheritance. It does not create an explicit allow rule.

Those details are small individually. Missing one can stop the whole deployment.

Accomplishments we're proud of

A real agent answered a real SMS on a real phone number.

We attached a live carrier number to the deployed service, sent it a text, and received an on-instruction response in about 1.7 seconds through the native Gemini tool loop.

The system recorded the turn, linked the model run, and billed about $0.0012.

No human touched the reply.

The safety controls matter more to us than the demo.

Fifteen checks run before the main model receives a token. STOP is handled before AI runs. Quiet hours skip replies instead of delaying them until morning. A human takeover path remains available to workspace members so a person can always take control of a conversation.

The injection defense also catches attacks that are not in its pattern table.

We published the red-team corpus and use Gemma as a second opinion because deterministic matching alone scored 0 of 5 on held-out attacks.

Two injection attempts sent to the deployed phone line ended as escalations with $0 of main-model spend.

One used phrasing absent from both the corpus and the deterministic pattern list. Gemma flagged it in 1.3 seconds.

We also measured the attachment pipeline instead of guessing.

Across 9,327 real attachments, 72% of images produced no useful text because they were logos, signature graphics, or tracking pixels. A small pre-filter removes about two thirds of unnecessary OCR work.

Of the PDFs measured, 91% already contained a text layer. Expensive extraction is the exception.

The deployed stack also handled a 30-day mailbox ingest containing roughly 296 messages, 120 threads, and 52 attachments. The messages passed through classification, extraction, and embedding while the full test suite remained green.

What we learned

Probe the platform

Two of our most annoying debugging sessions came from trusting documentation more than the endpoint we were actually deploying against.

A small probe script would have exposed both problems immediately.

For infrastructure and model availability, the live response wins.

Constrained decoding changes the code around the model

Schema-constrained output moves validation to a clear boundary.

Once the response shape is guaranteed, the rest of the application can work with ordinary typed data instead of defensive parsing.

Long generated fields still need careful handling. We clamp them with a transform instead of a hard schema maximum. Rejecting an otherwise valid response because one field is too long can turn a useful result into no result at all.

The model is a small part of an agent

The function-calling loop is not the hard part.

The gate, permissions, credit ledger, message delivery rules, memory keys, audit trail, approval system, and human takeover path determine whether an agent can safely talk to customers.

Prompt quality matters. Control over what the model can do matters more.

Different defenses should fail differently

Running the same check three times does not give you three layers of protection.

The deterministic scanner and Gemma solve different failure modes.

The scanner is fast, predictable, and cannot be persuaded by the message it is inspecting.

Gemma can recognize attacks whose wording never appeared in a pattern table.

The evaluation work also caught concrete implementation bugs, including reading the wrong model field as a verdict, confusing sensitive information with hostile input, and blocking legitimate external writes that happened to quote customer text.

Claims need receipts

Published security and behavior numbers come from executable evaluations.

Costs come from recorded model runs.

Agent statistics come from turn records.

Failures stay visible in the report.

If a number matters, we want a reviewer to be able to trace it back to the thing that produced it.

What's next

The next step is agent-to-agent handoff across organizations.

We also want deeper MCP catalogs so a workspace can give its agent access to the tools it already uses to run the business.

Voice is the other obvious direction.

CommSync already handles call logs, voicemail on top of Twilio, and transcription. The next piece is an agent that can pick up the phone.

Built With

Share this project:

Updates

Submission history