Butler Glue

From words to action: an AI butler that captures intent from real conversations and executes it — eliminating the gap between saying and doing.

Comment

Inspiration

Modern life is full of micro-commitments — things we fully intend to act on but never formally capture.

A screenshot of a message we meant to reply to. A photo of a document we planned to follow up on. A casual “I’ll call you in an hour” typed in a chat.

These fragments get dumped into photo galleries, chats, and cloud storage, where they slowly disappear into clutter. Not because they weren’t important — but because there was no frictionless way to convert intent into action.

Existing assistants require explicit commands and structured input. Real life doesn’t work that way.

We built Butler Glue around a simple belief:

If a human butler would recognize something as a commitment, software should too.

What it does

Butler Glue is an intelligent personal commitment tracker that transforms raw intent into scheduled action — automatically.

Users don’t fill out forms or issue commands. Instead, they:

Type natural language

Take a photo

Share a screenshot or message from another app

Butler Glue:

Extracts meaning from text and images

Detects promises, tasks, and deadlines

Converts them into structured, scheduled jobs

Manages reminders until the commitment is fulfilled

All actions are reviewed and confirmed — nothing happens silently.

To make the experience human and memorable, users interact with a butler persona (Alfred, Jarvis, or Peeves), each with distinct tone and visual identity.

How we built it

Butler Glue is a mobile-first, intent-native system built for speed, autonomy, and trust.

Client (Flutter Mobile)

Framework: Flutter (stable)

State Management: flutter_riverpod for reactive job tracking

Navigation: go_router with a global navigator key for background share intents

Local Storage: Drift + SQLite (scaffolded for offline persistence)

Notifications:

flutter_local_notifications for precise local scheduling

firebase_messaging for background and push handling

AI & ML

OCR: Google ML Kit (on-device) for fast, private text extraction from images

Intent Parsing: OpenAI GPT-4o-mini parses raw text into structured JSON:

Action

Deadline (ISO 8601)

Summary

Commitment Type (Promise / Task)

Confidence score

Avatar Generation: OpenAI DALL·E 2 generates persona-based 2D butler portraits

Backend & Cloud

Serverpod: Scaffolded as the long-term backend “glue”

Future migration of AI parsing and persistence

Multi-device sync and secure API key handling

Firebase: Messaging and background delivery

Challenges we ran into

Intent ambiguity: Not every screenshot implies a task — false positives had to be minimized

Trust boundaries: Automation had to feel helpful, not intrusive

Latency: AI parsing needed to feel instant on mobile

Platform constraints: Deep OS integrations vary, especially around background intents

These challenges pushed us toward a review-first, confidence-scored workflow instead of blind automation.

Accomplishments that we’re proud of

Built an end-to-end intent → job → reminder pipeline

Achieved sub-second OCR + parsing flows on mobile

Designed a system that works with unstructured, real-world inputs

Created a personality-driven UX that enhances clarity rather than novelty

Delivered a complete, extensible architecture without scope creep

Most importantly: We turned forgotten screenshots into follow-through.

What we learned

Screenshots are deferred decisions, not memories

The best automation is selective and explainable

AI assistants shouldn’t wait for commands — they should notice intent

Personality can improve usability when tied to function, not gimmicks

What’s next for Butler Glue

Move AI parsing fully server-side via Serverpod

Enable cross-device commitment syncing

Add long-horizon routines and recurring intent detection

Expand ingestion to email and messaging platforms

Introduce adjustable autonomy levels per user

Our long-term vision is simple:

Close the gap between what people meant to do — and what actually gets done.

Built With

Share this project:

Updates