Alfred, On-device Voice AI Companion

Alfred: Voice AI agent that learns you & automates tasks. Summarize emails, code, manage calendar, take meet notes, you name any task and Alfred does it - on-device! Because even you need an Alfred.

Comment

Inspiration

In today's fast-paced digital world, we often find ourselves overwhelmed by the myriad of tasks we need to accomplish on our devices. Switching between applications, managing emails, scheduling appointments, and staying updated with the latest information can be time-consuming and mentally draining. We envisioned a solution that could seamlessly integrate these tasks, learn from our behavior, and act proactively to assist us—much like a personal assistant. This aspiration led us to create Alfred, an on-device AI companion designed to simplify and enhance the way we interact with our laptops.

What it does

Alfred is an AI-powered assistant that resides on your device, capable of handling virtually anything you would do on your laptop. By learning from your user behavior—such as search history, recent activities, and file usage—Alfred indexes all your information to execute tasks efficiently. Whether it's:

  • Managing Emails: Checking your unread emails and providing concise summaries.
  • Note-taking: Listening to your conversations (with your permission) to jot down important notes.
  • Information Retrieval: Answering questions about the latest news or specific data.
  • Coding Assistance: Writing code based on your requirements or making changes to existing files.
  • Browser Automation: Performing tasks like setting calendar invites, reading emails, or generating reports from your data analytics platforms.

Alfred acts as an all-in-one agent that continuously learns and adapts from your interactions. With a sophisticated queuing mechanism, it can handle multiple concurrent requests. For example, while Alfred is summarizing your emails, you can still engage in conversation or assign new tasks. Once a task is completed, Alfred will promptly notify you and offer the results, ensuring a smooth and uninterrupted workflow.

How we built it

  1. Frontend Development: We used React to create a dynamic and responsive user interface that allows users to interact with Alfred effortlessly.

  2. Backend Development: The backend was developed using Python, which facilitated the integration of various AI models and ensured robust performance.

  3. Agentic Workflow with Function Calling: We implemented an agentic workflow that allows Alfred to perform tasks autonomously by calling specific Agents as needed. The agents are equipped with RAG to retrieve the relevant context to offer the best responses.

  4. AI Technologies: We integrated Whisper for speech recognition, LLM to ensure the conversational flow and provide context to the agents during function calling, and text-to-speech capabilities to enable seamless voice interactions. We setup an Agentic RAG system for getting the context to the Agents.

  5. Communication Protocol: WebRTC was used to handle real-time communication, ensuring smooth and efficient interactions between the user and Alfred.

Challenges we ran into

  • Concurrent Task Handling: Implementing an efficient queuing system that can manage multiple tasks without conflicts or delays required careful planning and optimization. Ensuring that Alfred could handle several requests simultaneously while maintaining responsiveness was crucial. This involved implementing subprocess with event handling.

  • Quick responses: One of the hardest problems to solve for was the quick response times to offer the best User Experience. We solved this by relying on faster inference engines like Groq and using APIs to reduce load on the device.

Accomplishments that we're proud of

We were proud of the range of features we were able to handle.

  • On-device AI Processing: Achieving on-device processing that safeguards user privacy while maintaining high performance.
  • Seamless Multitasking: Developing a concurrent task management system that allows Alfred to handle multiple requests smoothly.

What we learned

Understood the complexities involved in scaling such a system across different devices and user bases.

What's next for Alfred, On-device Voice AI Companion

  • Third-party Integrations: Incorporating more apps and services to broaden the range of tasks Alfred can handle.
  • Enhanced Learning Algorithms: Improving the behavioral learning models for even more accurate predictions and personalized assistance.
  • Developer SDK: Offering a software development kit (SDK) for developers to build custom plugins and functionalities for Alfred.

Built With

Share this project:

Updates