Bulter AI Personal Assistant

One app. Everything handled.

Comment

Inspiration

We wanted an AI that goes beyond one-off chats. Butler AI was born from the need for an assistant that actually remembers you—your preferences, facts, and past interactions—while being able to act on your behalf across all your favorite apps.

What it does

Butler is a proactive personal assistant that manages your life and work. It uses natural language to execute tasks (GitHub issues, Slack messages, calendar invites) and features an intelligent "User Memory" that learns from every interaction to provide hyper-personalized support.

How we built it

  • Unified Protocol: Used Serverpod for a seamless, type-safe Dart connection between backend and mobile.
  • RAG-based Memory: Implemented semantic search using Pinecone and Gemini embeddings to store and retrieve personal facts.
  • Modular Integrations: Built a flexible tool-calling architecture enabling Gemini to orchestrate 25+ distinct functions across 10+ external services.
  • Multi-platform: Leveraged Flutter for a premium UI that works across web and mobile.

Challenges we ran into

  • Tool Selection: Orchestrating a large number of tools while ensuring the model doesn't hallucinate API calls or get overwhelmed by context.
  • Vector Support: Enabling fast, scalable semantic search and contextual recall via Pinecone integration.

Accomplishments that we're proud of

  • Context-Awareness: The assistant can recall details from weeks ago to inform its current actions.
  • Full-Stack Dart: Building a high-performance, maintainable codebase where every layer speaks the same language.

What we learned

The immense potential of persistent memory (User Memory) in transforming LLMs from static tools into dynamic partners. Strategies for efficient tool-calling and prompt engineering to handle complex, multi-service workflows.

What's next for Butler AI Personal Assistant

  • Proactive Agency: Enabling the assistant to take initiative (e.g., "I noticed your meeting finished, should I follow up on the tasks we discussed?").
  • Voice Immersion: Full voice-to-voice interaction using ElevenLabs.

Built With

Share this project:

Updates