Private AI Workmate

• 

  main chat ui

• 

  memory with context

• 

  Documents/RAG page

• 

  GitHub Developer Agent result

• 

  Memory or Security Center

Inspiration

Personal AI assistants are becoming more capable, but many assistants still treat memory, documents, and external tools as separate features. We wanted to build a workmate that can remember useful information, work with private documents, interact with selected developer tools, and still keep clear boundaries around what it is allowed to access.

This led us to build Private AI Workmate for the Personal AI track: an AI assistant designed around persistent memory, private document knowledge, controlled tool execution, and developer workflows.

What it does

Private AI Workmate combines several capabilities into one assistant:

• Persistent memory — stores useful long-term information and retrieves relevant memories when answering future questions.
• Private document RAG — users can upload documents and ask questions grounded in their contents.
• Controlled tool execution — the agent can use approved tools such as calculators, time lookup, document search/read, and other registered tools.
• GitHub Developer Agent — provides read-only repository intelligence, including repository information, file listings, code search, file reading, issue information, and pull-request information.
• Multi-step reasoning — complex requests can involve multiple approved tool calls before the assistant produces a final answer.
• Security boundaries — tool permissions, argument validation, prompt-injection detection, untrusted-content boundaries, and audit logging help keep tool execution controlled.
• Web application — a responsive interface provides chat, memory, documents, GitHub analysis, security information, and connection status in one workspace.

The system is designed so that the AI does not receive unrestricted access to the machine. Tools are explicitly registered, validated, permission-checked, and executed through a controlled execution layer.

How we built it

The backend is built with Python and FastAPI, while the frontend uses React and Vite.

For AI inference, the project uses NVIDIA Nemotron through Nebius Token Factory. Nebius Token Factory provides the model API infrastructure used by the application, allowing the assistant to use Nemotron without managing model-serving infrastructure ourselves.

For semantic memory and document retrieval, the application uses vector search with Qdrant. Structured application data is persisted separately, while uploaded document content is processed into chunks and indexed for retrieval.

The application also integrates GitHub through a read-only developer tool layer. Instead of giving the model unrestricted access to GitHub, each operation is exposed as a specific registered tool with a defined schema and permission policy.

The frontend and backend were also containerized with Docker and deployed using managed cloud services. The production architecture separates the web frontend, API backend, database, vector database, and AI inference service.

Security and privacy approach

Privacy was treated as an architectural requirement rather than only a UI feature.

The application uses explicit tool permissions and currently operates with a read-only tool boundary for external developer resources. Tool arguments are validated before execution, unknown tools are denied, and sensitive/private tool results are not unnecessarily exposed through audit logs.

User input and retrieved external content are also checked for prompt-injection patterns. Retrieved documents and tool results are treated as untrusted data rather than instructions to the agent.

The system does not provide arbitrary shell or Python execution to the model.

Challenges

One of the main challenges was making the different parts of the system work consistently across local development and Docker deployment.

During integration testing, document retrieval worked through the Documents interface but initially failed when invoked through the chat agent. We traced this to authentication context not being propagated into the synchronous agent/tool execution path. The issue was fixed by explicitly propagating the server-resolved user context and safely resetting it after execution.

Another challenge was designing the tool layer so that the agent could perform useful multi-step tasks without becoming an unrestricted autonomous executor. We solved this with registered tool schemas, permission policies, argument validation, iteration limits, prompt-injection checks, and audit logging.

What we learned

Building this project taught us that a useful personal AI system is not only about choosing a powerful language model. Memory, retrieval, tools, permissions, security boundaries, persistence, and a reliable user interface all need to work together.

Using NVIDIA Nemotron through Nebius Token Factory also allowed us to focus on building the application and agent architecture instead of maintaining model-serving infrastructure.

Future improvements

Future versions could add more user-controlled skills, richer workflow automation, additional integrations, stronger identity and access management, and more advanced model routing for different task types.

Private AI Workmate is intended to demonstrate how an AI assistant can become a practical work companion while keeping its capabilities explicit, controlled, and auditable.

Built With

• agents
• ai
• api
• database
• docker
• factory
• fastapi
• github
• javascript
• llm
• nebius
• nemotron
• nvidia
• prompt
• python
• qdrant
• rag
• react
• rest
• token
• vector
• vite