Inspiration

Context Is All You Need

LLMs are kind of dumb. They excel at predicting the next token, but if you ask them about anything outside of their training set, you're SOL. Recently, this problem has been tackled via function calling, but setting these tools up has proved to be quite the ordeal. Thus, Patrick started with a simple goal: build an agentic, modular, and context-aware office assistant with access to all of your tools and files!

What It Does

Patrick is a desktop application that can read and respond to emails for you, search your filesystem, (you can restrict its access to specific directories!), and create files like text, excel, and powerpoint. Patrick leverages the Model Context Protocol (link) to enable real-time, secure communication between large language models (LLMs) and popular productivity tools. It helps users automate tasks, generate content, analyze data, and streamline workflows—all while ensuring efficient, context-aware interactions.

How We Built It

We built Patrick by integrating LLM APIs with applications such as Excel, PowerPoint, and Email through MCP. Our architecture consists of:

MCP Client: Facilitates communication between the user and the backend.
MCP Servers: Lightweight, modular servers that provide the LLM access to various resources through a function-calling interface. Specifically, we built MCP servers for Gmail, Excel, and Powerpoint.
LLM API: Our implementation uses Gemini 2.0 Flash, but MCP supports any model with function calling capabilities.

Challenges We Ran Into

One major challenge was the lack of a mature ecosystem for MCP. Being a relatively new standard (released in November 2024), MCP did not have extensive resources or best practices available (competing methodologies for building MCP servers – FastMCP vs MCP SDK), which sometimes led to compatibility issues and unexpected errors during development. We were able to overcome some of these through lots of hardwork and ingenuity! :D

Accomplishments We’re Proud Of

Multiple working MCP Servers with some pretty cool functionalities:

Gmail MCP with support for reading and sending emails, including attachments.
Excel MCP with support for creating, reading and writing to excel files.
Powerpoint MCP with support for creating and modifying Powerpoint files.

What We Learned

We learned alot about how AI models reason about tasks. Since MCP provides only a function calling interface for the model, it's up to the model to pick which functions to call, and when. We tried running MCP servers with Qwen2.5:7b and Llama 3.2, but the best results by far came when using Google's Gemini 2.0 Flash!

What’s Next for Patrick

Adding a bunch of more MCP servers, tightening up on application security, and shipping everything in a single executable.

Built With

ElectronJS: For a modern, cross-platform desktop experience.
Model Context Protocol: The backbone that powers secure and efficient LLM communication.
Model: Our engine for natural language understanding and generation - Google Gemini 2.0 Flash.
Libraries and Frameworks: Node, React, Python