Dory Coworker Assistance

Dory brings AI coworkers into the workplace, powered by NVIDIA Nemotron and Nebius to help teams collaborate, preserve knowledge, and turn conversations into action.

Comment

Dory — Your AI Coworker

Inspiration

Communication is at the heart of every organization. Every day, teams exchange ideas, make decisions, share knowledge, and coordinate work through meetings, messages, and conversations.

Yet much of that valuable information is easily lost. Important decisions get buried in chat histories, ideas fade with time, and tasks are forgotten before anyone follows up. As organizations grow, keeping everyone informed and aligned becomes increasingly difficult.

While countless productivity tools exist, many introduce steep learning curves, fragmented workflows, and additional administrative work. Instead of making collaboration easier, they often require people to spend more time managing tools than doing their actual work.

This led us to a simple question: What if AI could become an actual member of the team?

Imagine an AI coworker who joins your meetings, participates in conversations, understands your team's context, and remembers what matters. Someone you can naturally ask what was discussed, why a decision was made, or what needs to happen next — just like asking a knowledgeable colleague.

Rather than expecting people to adapt to another tool, we envisioned an AI that adapts to the way people already work.

That vision became Dory, an AI coworker designed to listen, remember, and take action alongside your team.

Starting with the communication platforms and workflows familiar to small businesses, we're building Dory toward a much larger goal: making AI-powered collaboration accessible to teams and organizations everywhere.

We believe the future of work isn't just about giving people better tools. It's about giving them a new kind of teammate.

What it does

Dory works directly within the platforms teams already use, including Discord, Slack, LINE, Telegram, WhatsApp, and a web command center. Instead of requiring people to learn special commands or maintain another system, Dory lets them interact naturally, just as they would with a colleague.

Meetings and Conversations

"Wait, what did we just agree on?"

Dory can join meetings, follow discussions, and capture key decisions and action items. If someone misses a point or joins late, they can ask Dory to recap what happened.

Team Memory and Knowledge

"Didn't we talk about this before?"

Important information often disappears into thousands of messages. Dory keeps track of accessible team discussions and makes that knowledge searchable, helping people find past decisions, relevant context, and information they might otherwise forget.

Tasks and Coordination

"Who's supposed to be doing that?"

Tasks mentioned during conversations can easily be overlooked. Dory turns them into trackable assignments, lets team members confirm responsibilities, and helps everyone stay updated through follow-ups and daily briefings.

Documents and Administration

"Do I really have to do all this manually?"

Dory helps process receipts, record expenses, understand documents, analyze spreadsheets, and generate reports. Instead of moving information between applications manually, teams can simply share what they need help with.

Voice and Everyday Assistance

"Can someone help me with this?"

Dory also supports voice conversations with access to real tools. Users can ask her to retrieve information, check the weather, explore travel options, or set reminders without switching between applications.

All these capabilities connect through the same AI agent and shared company context. Whether someone interacts with Dory through chat, a meeting, or the web, they are working with the same coworker.

How we built it

We designed Dory around one central principle: the AI should remain consistent across platforms while being able to understand context, retrieve knowledge, and perform meaningful actions.

Unified Agent Architecture

At the center of Dory is a Google ADK LlmAgent, powered by GLM-5.3 through Nebius Token Factory.

Rather than building independent bots for every communication platform, we developed a shared agent architecture with approximately 140 tools covering tasks, documents, financial records, company knowledge, meetings, and external services.

This allows Dory to use the same capabilities and business context regardless of where a conversation begins.

We also enforce tenant isolation and user permissions directly in application code. Company identifiers and caller identities are bound to tool execution rather than exposed as model-controlled parameters, preventing the agent from selecting another company's data.

Model Routing and NVIDIA Integration

Different tasks require different levels of reasoning and processing capabilities. Running every request through a single large model would introduce unnecessary latency and cost.

We therefore use specialized models for different workloads:

Workload Model
Agent reasoning and tool execution GLM-5.3
Summarization, translation, and lightweight processing GLM-5.3-Flash
Receipt and document-image understanding NVIDIA Nemotron 3.5 Super VL
Semantic memory embeddings Qwen3-Embedding-8B

Our text reasoning workloads run through Nebius Token Factory, while NVIDIA Nemotron 3.5 Super VL handles visual information, including photographed receipts and scanned documents.

This model-routing approach allows us to reserve more capable reasoning for complex operations while using cost-efficient models for routine processing.

Hybrid Memory and Knowledge Retrieval

Remembering an earlier conversation is not always as simple as searching for a keyword. Users may remember an exact phrase, or they may only remember the meaning of what was discussed.

To support both situations, Dory combines traditional keyword retrieval with semantic vector search.

The results are merged using Reciprocal Rank Fusion (RRF), allowing Dory to retrieve relevant discussions even when the user's question differs from the original wording.

Embeddings are generated using Qwen3-Embedding-8B, with a company-scoped 64 MB LRU cache that avoids the need for a separate vector database in our current architecture.

Memory retrieval also respects channel permissions. Information from restricted conversations is only used when the audience receiving the answer is authorized to access it.

Tool-Enabled Voice Architecture

We developed a voice system that connects directly to Dory's existing agent and tools, allowing users to perform actions through natural conversations.

The voice pipeline runs on a preemptible NVIDIA H200 GPU on Nebius AI Cloud and consists of three components:

  1. Parakeet-TDT converts incoming speech into text.
  2. Dory's AI agent interprets the request, retrieves context, and executes tools.
  3. Kokoro-82M generates spoken responses sentence by sentence.

Unlike a standalone voice chatbot, this architecture gives voice conversations access to the same capabilities available through text.

We also measure latency across each processing stage to identify bottlenecks and evaluate response performance.

To control infrastructure costs, the voice GPU starts on demand and automatically stops after 30 minutes of inactivity. Persistent storage preserves the environment between sessions, with subsequent cold starts taking approximately two minutes.

Application Infrastructure

Dory's backend is built with Python 3.12 and FastAPI, deployed on Google Cloud Run.

A Cloud Run worker pool hosts the Discord bot, while Firestore provides shared data storage and Pub/Sub coordinates background operations such as meeting finalization.

The web command center uses HTML, CSS, and JavaScript with a shared design system.

We use uv, ruff, and pytest for dependency management, code quality, and automated testing. Browser-level testing is handled through Playwright.

Together, these components support a multi-platform AI system with shared memory, tool execution, company isolation, and cost-aware infrastructure.

Challenges we ran into

Building an AI coworker required more than connecting a language model to a collection of tools. We encountered several challenges involving agent reliability, model behavior, language processing, infrastructure costs, and data security.

Getting AI to Act Instead of Just Responding

One of our most unexpected problems was discovering that the agent could confidently say "I'll set a reminder" without actually calling the required tool.

With approximately 140 tools available, reliable tool selection became increasingly challenging.

We developed a testing harness that replayed realistic requests against a copy of our application data. These experiments showed that enabling reasoning and providing explicit tool-use instructions significantly improved execution reliability.

We also encountered false action confirmations, including a case where the model claimed a booking had been completed when no booking occurred.

To address this, Dory checks action claims against actual tool executions, while critical confirmations are handled through deterministic application logic.

Managing Different Model Behaviors

We initially expected inference settings to behave consistently across models, but each model family introduced different constraints.

GLM-5.x reasoning configurations caused unexpected token consumption and truncated responses. Meanwhile, enabling reasoning on our vision model sometimes resulted in empty structured outputs during receipt processing.

We addressed these issues by testing and configuring each model independently rather than applying the same inference parameters to every workload.

Processing Thai Language Correctly

Thai presents challenges that are less common in English-language applications, particularly in document extraction and text rendering.

Early versions of our generated PDFs and image cards broke Thai sentences in unnatural places. Some government PDFs also contained unreliable text layers that omitted tone marks.

We introduced HarfBuzz/RAQM text shaping and syllable-aware line breaking, while using image-based processing for documents with unreliable extracted text.

These improvements help Dory support multilingual business workflows more consistently.

Keeping GPU Infrastructure Affordable

Our voice infrastructure introduced an unexpected billing problem: shutting down the GPU instance from within Linux caused the VM to restart automatically instead of remaining stopped.

We resolved this by controlling instance shutdown through the Nebius Cloud API.

We also encountered GPU compatibility issues with our voice model's PyTorch dependencies, which led us to move from a lower-cost Blackwell configuration to a preemptible NVIDIA H200.

These experiences shaped our on-demand GPU architecture, allowing us to reduce unnecessary runtime costs.

Protecting Knowledge in Shared Conversations

A particularly important challenge involved information access in shared communication channels.

A user asking a question might have permission to view certain information, but other people reading the generated response might not.

We developed audience-aware permission checks that consider who can see the response, not just who requested it.

When permissions cannot be verified, Dory restricts memory retrieval to the current channel rather than risking exposure of private information.

Supporting Bilingual Teams

Teams often switch between languages during conversations, particularly in multilingual workplaces.

Initially, Dory followed the language of each incoming message, causing responses to alternate unpredictably between Thai and English.

We introduced a company-wide default language with optional personal preferences, making interactions more consistent while retaining flexibility.

These challenges reinforced an important principle: an AI coworker must do more than produce intelligent responses. It must act reliably, respect boundaries, and behave consistently in real working environments.

Accomplishments that we're proud of

What began as an idea for an AI teammate has developed into a working system that connects communication, shared knowledge, task execution, and business workflows.

Our most significant achievements include:

1. One Agent Across Six Platforms

We connected Discord, Slack, LINE, Telegram, WhatsApp, and the web through a unified agent architecture.

Instead of maintaining separate AI systems, these interfaces share the same core agent, memory, tools, and business context.

2. Tool-Enabled Voice on Nebius Infrastructure

We developed a voice assistant running on a preemptible NVIDIA H200 GPU through Nebius AI Cloud.

Dory can use tools during spoken conversations, while the GPU infrastructure starts on demand and stops automatically when idle.

3. Privacy-Aware Organizational Memory

We combined semantic and keyword retrieval with company isolation and channel-level access controls.

This enables Dory to retrieve useful knowledge from previous discussions while respecting organizational information boundaries.

4. Automated Testing and Engineering Reliability

Our codebase contains approximately 113,000 lines of Python and more than 7,000 offline automated tests, with external services mocked for repeatable testing.

We also integrated Playwright browser tests and WCAG 2.2 AA accessibility checks into our development workflow.

5. Cost-Conscious AI Infrastructure

We implemented model routing, GPU scale-to-zero, and budget controls for paid AI operations.

These mechanisms help balance the capabilities of a complex AI system with the operational costs required to run it.

What makes us particularly proud is not any individual feature, but the integration of these components into one coherent system.

Dory can participate in conversations, preserve context, retrieve information, and take action through a shared architecture designed for real organizational workflows.

What we learned

Developing Dory changed how we think about building AI agents. We learned that model intelligence alone is not enough to create a system people can depend on.

Evaluate Actions, Not Just Responses

A convincing answer does not guarantee that an action was completed.

Testing actual tool execution against realistic scenarios proved more valuable than evaluating response quality alone. Reliable agents need behavioral evaluations that verify what happened, not just what the model said.

Keep Critical Rules in Code

Language models are effective at interpreting requests and deciding which tools to use, but they should not independently control permissions, financial limits, or sensitive confirmations.

We learned to separate model-driven decisions from deterministic application safeguards.

Match Models to Their Workloads

Different operations require different capabilities.

Routing suitable tasks to smaller, less expensive models allowed us to reduce processing costs while reserving more powerful reasoning for complex requests.

Infrastructure Efficiency Matters

Running our own GPU infrastructure taught us that cost optimization goes beyond model inference.

Startup behavior, idle time, persistent storage, and correct instance shutdown can significantly affect the economics of an AI service.

User Experience Defines Practical Value

Users experience an AI system through its responsiveness, consistency, and ability to help them accomplish tasks.

We learned that clear acknowledgments, predictable behavior, and natural interactions can be just as important as the underlying model's capabilities.

Ultimately, building Dory showed us that successful AI agents require a balance of intelligence, reliability, security, cost efficiency, and usability.

What's next for Dory

Our next goal is to move Dory from a working prototype toward a dependable AI coworker that organizations can use in their everyday operations.

Production Readiness

We plan to strengthen authentication, application security, session management, and infrastructure scalability.

We will also expand automated evaluations and introduce continuous testing requirements for changes to models, prompts, and agent behavior.

Consistent Cross-Platform Capabilities

We aim to develop a unified workspace interface that provides consistent functionality across Discord, Slack, LINE, Telegram, WhatsApp, and the web.

This will allow organizations to interact with Dory through their preferred communication platforms without losing access to essential capabilities.

More Reliable Real-World Actions

We plan to expand integrations with external business services and introduce approval workflows for sensitive operations.

For example, travel-related capabilities will move beyond sandbox inventory toward real providers, with appropriate confirmation and authorization mechanisms.

Real-World Pilots and Evaluation

Starting with small businesses in Thailand, we plan to evaluate Dory in real working environments.

These pilots will help us measure administrative time saved, identify usability challenges, improve reliability, and strengthen privacy and data-handling practices.

The lessons from these deployments will guide our expansion to organizations in other markets.

Our Long-Term Vision

Our ambition goes beyond building another productivity application.

We envision a future where AI coworkers become a natural part of how people collaborate — participating in discussions, preserving organizational knowledge, coordinating responsibilities, and helping teams move from decisions to action.

We want organizations of different sizes, industries, and locations to benefit from capable AI assistance without needing specialized technical expertise or complicated new workflows.

Dory represents our vision of a more collaborative future of work — one where AI doesn't simply provide another tool, but becomes a dependable teammate working alongside people.

Built With

  • discord.py
  • duckdb
  • fastapi
  • firestore
  • glm-4-voice
  • glm-5.3
  • google-adk
  • google-cloud
  • google-cloud-pub-sub
  • google-cloud-run
  • javascript
  • kokoro-tts
  • line-messaging-api
  • litellm
  • nebius
  • nebius-token-factory
  • nvidia-h200
  • nvidia-nemotron
  • nvidia-parakeet
  • python
  • qwen3-embedding
  • slack
  • telegram-bot-api
  • websockets
  • whatsapp-cloud-api
Share this project:

Updates

Submission history