Felix

Inspiration

We wanted to build something that felt like magic, the kind of thing where you point your phone at the world and it just does stuff for you. So many great ideas (notes from a whiteboard, a random thought you have, a flyer for an event) just die in your camera roll because turning them into something useful takes too many steps. We wanted Felix to close that gap completely. Capture it, say what you want, and it happens.

What it does

Felix lets you capture an image and a voice command from any device. That input goes to Gemini, which figures out what you actually mean, then a multi agent system on Google Cloud Platform plans and executes the task. Through MCP servers, our agents connect directly to real Google services like Docs, Sheets, Calendar, and Drive, so the output isn't just a suggestion, it's a finished document, a scheduled event, or an updated sheet. Everything updates live across devices through WebSockets, so your web dashboard reflects what just happened on your phone in real time.

How we built it

We split Felix into three layers. The frontend (iOS Swift app and a web dashboard) only handles capture and display, no AI logic lives there. All input gets sent to a backend running on Google Cloud Run, which is the real brain of the system. On the backend, Gemini takes the image and voice input and converts it into structured intent. From there, our multi agent system takes over. We have specialized agents for interpretation, planning, execution, and verification, and we used reinforcement learning so the agents get better at picking the right tools and workflows over time instead of relying on hardcoded rules. The MCP layer was huge for us. Instead of writing fragile API glue code for every Google service, our agents use MCP servers to talk directly to Docs, Sheets, Calendar, and Drive. That made the system way more flexible and easy to extend. Finally, an event bus broadcasts results over WebSocket, so any connected client sees the output the moment it's ready.

Challenges we ran into

Getting the multi agent system to coordinate cleanly was probably our biggest challenge. Making sure the interpretation agent, planning agent, and execution agents all stayed in sync without stepping on each other took a lot of iteration. We also had to think carefully about how reinforcement learning would actually improve agent decisions in a hackathon timeframe, since RL usually needs a lot of data to show real improvement. Wiring up MCP to multiple Google services and keeping everything authenticated correctly was another big one. And on the frontend side, getting real time updates to feel instant across iOS and web at the same time took some debugging with our WebSocket setup.

What we learned

We learned a ton about how MCP servers can replace a huge amount of manual API integration work, and how powerful it is to separate "thinking" (the AI layer) from "doing" (the frontend). We also got hands on experience designing a multi agent architecture where each agent has a clear role, which made debugging so much easier than one giant monolithic AI call. On top of that, we learned a lot about real time systems and how to keep multiple devices in sync.

What's next for Felix

We want to expand Felix beyond Google Workspace into more tools and platforms, build out a proper desktop execution layer, and keep training our agents with reinforcement learning on real usage data so the system keeps getting smarter the more people use it. Long term, we think Felix could become a universal AI backend that works with any frontend and any service, turning the real world into structured digital action automatically.

Built With

• calendar
• drive
• framework
• google-workspace-apis-(docs
• sheets
• speech
• swift-/-ios-(avfoundation