Inspiration

The inspiration behind Kasumi-AI came from a simple question: What if you could FaceTime your own AI assistant and have it control your computer remotely?

We wanted to create something beyond a traditional AI chatbot or voice assistant. Our vision was to develop an intelligent AI companion that could communicate naturally with users while performing real tasks on their laptops or desktops.

Instead of relying on traditional remote desktop applications, we wanted users to simply make a FaceTime call, give a voice command, and let their AI assistant handle the task.

Inspired by intelligent virtual assistants such as JARVIS and FRIDAY, we began developing Kasumi-AI as part of Project Yukari, combining real-time communication, artificial intelligence, voice technology, and computer automation.

Our goal was to make interacting with AI feel more natural, accessible, and personal.

What it does

Kasumi-AI is an AI-powered virtual companion that allows users to control their Windows computers through FaceTime using natural voice commands.

By integrating Chert, LiveKit, Gemini, Hermes, and ElevenLabs, Kasumi-AI can receive a FaceTime call, understand spoken instructions, execute commands on a connected computer, and respond using a synthesized voice.

During our testing, Kasumi-AI successfully performed several tasks, including:

Opening YouTube, Spotify, Discord, and Google.

Retrieving and reporting the laptop's battery percentage.

Executing computer commands remotely through Hermes.

Responding to spoken questions using AI-generated speech.

Handling multiple commands throughout the same FaceTime conversation.

Displaying an animated visual interface while communicating with the user.

For example, a user can call Kasumi-AI from their iPhone and say, "Kasumi, open Spotify on my laptop."

The assistant processes the request, communicates with the Hermes backend, opens Spotify, and verbally confirms the action.

We designed Kasumi-AI to demonstrate how AI assistants can move beyond answering questions and begin interacting directly with computers.

How we built it

We developed Kasumi-AI by integrating several technologies into a single AI assistant system.

Google Gemini serves as the primary AI reasoning model, allowing Kasumi-AI to understand natural language instructions and generate responses.

Hermes provides the backend integration responsible for executing authorized commands on the user's Windows computer.

ElevenLabs powers the assistant's voice and real-time speech recognition, allowing users to communicate with Kasumi-AI through spoken conversations.

Chert enables the FaceTime integration, allowing users to call and interact with the AI assistant directly from their iPhones.

LiveKit handles real-time audio and video communication, connecting the AI agent with the FaceTime communication system.

We used JavaScript and Node.js to develop the real-time agent, integrate the services, manage conversations, and generate the assistant's animated video interface.

Following HackUTA, we also began expanding Project Yukari with a Python-based FastAPI bridge and a React/TypeScript web dashboard.

The project's architecture connects speech recognition, Gemini's language processing, Hermes computer automation, and ElevenLabs voice generation to create a functioning conversational AI system.

Challenges we ran into

One of our biggest challenges was getting multiple independent technologies to communicate reliably.

During development, we encountered temporary service availability issues with Chert, which interrupted our ability to test the FaceTime connection.

Once the service became available again, we continued troubleshooting and successfully established communication between FaceTime, LiveKit, and our AI agent.

Another challenge involved response latency and background noise.

During testing in a quiet environment, Kasumi-AI generally responded within approximately 1–3 seconds. However, in louder environments, response times sometimes increased to 4–8 seconds.

We also worked through technical difficulties involving voice recognition, API communication, real-time audio processing, and connecting Hermes with Windows computer automation.

Despite these challenges, we successfully developed and demonstrated a functioning prototype.

Accomplishments that we're proud of

Our greatest accomplishment was turning our idea into a working AI companion capable of performing real computer tasks through FaceTime.

We successfully demonstrated Kasumi-AI opening applications, retrieving system information, and responding to multiple voice commands during an active FaceTime call.

Another memorable achievement occurred during HackUTA 2026.

While evaluating the Chert Track, one of Chert's co-founders personally visited our table and watched our live demonstration. He also viewed our Devpost demonstration and shared positive feedback about the project.

He expressed surprise at what we had accomplished using Chert's technology and mentioned that other judges were impressed by the demonstration.

Although we did not finish among the top three projects in the Chert Track, receiving that feedback was an encouraging moment for our team.

We are especially proud that we accomplished this as a two-person team and created a functioning prototype that we can continue improving beyond the hackathon.

What we learned

Developing Kasumi-AI taught us valuable lessons about artificial intelligence, software integration, and real-time communication.

We learned how to connect a conversational AI model to FaceTime using Chert and LiveKit, integrate speech recognition and voice generation through ElevenLabs, and execute computer commands using Hermes.

We also gained experience troubleshooting API integrations, managing latency, working with asynchronous processes, and developing a voice-enabled AI assistant.

Perhaps our biggest lesson was that building an AI assistant involves much more than connecting a language model to a chatbot.

Creating an assistant that can listen, understand, execute commands, and communicate naturally requires several technologies to work together reliably.

HackUTA gave us the opportunity to put those skills into practice.

What's next for Kasumi AI

HackUTA was only the beginning of Project Yukari.

We plan to continue developing Kasumi-AI into a more capable, customizable, and accessible AI companion.

Our upcoming development goals include:

1) Improved Voice Recognition and Response Time

Reduce latency, improve background noise filtering, and create more natural conversations with fewer interruptions.

2) Advanced Computer Automation

Expand the assistant's ability to open, manage, and interact with applications, files, and computer settings through secure, authorized commands.

3) Screen Sharing and Computer Vision

Allow Kasumi-AI to examine a user's shared computer screen, understand visible information, and assist with troubleshooting or other tasks.

4) Interactive 3D AI Avatar

Introduce an animated 3D character with facial expressions, lip synchronization, and natural movements during FaceTime conversations.

5) Long-Term Memory

Develop an optional memory system that allows Kasumi-AI to remember user preferences, projects, and previous conversations.

6) Cross-Platform Compatibility

Expand compatibility beyond Windows and iPhone to include Android, additional desktop operating systems, and browser-based communication.

7) Open-Source Developer Community

We intend to make Kasumi-AI available as an experimental developer preview on GitHub.

Our goal is to allow developers and AI enthusiasts to experiment with the project, connect their own supported services, contribute improvements, and provide feedback.

We believe the future of personal AI assistants involves more than answering questions. It involves creating intelligent systems that can communicate naturally and assist users with real-world tasks.

Built With

Share this project:

Updates

posted an update —

We're going to be releasing V0.1.9 on GitHub on Nov 25th, 2026 at 11:00 PM CST for the public! This is a open source project and we will love yall to leave feedback! We will be improving Kasumi Ai every week! Enjoy Kasumi AI with Yukari as your voice and AI agent! Any questions contact us at [email protected]

Submission history