heyMisu - Speak It, Consider It Done

• 

• 

• 

Inspiration

I built heyMisu because I wanted an AI assistant that felt less like a chatbot and more like someone quietly helping me throughout the day.

Most AI tools are useful only after you stop what you are doing, open an app, explain the context, and ask the right question. They may answer well, but they rarely remember what you were working on, help you follow through, or turn a conversation into an actual outcome.

As a full time employee and part time solo developer managing projects, meetings, ideas, reminders, and everyday tasks, I often know what I need to do but lose track of the smaller details between different apps and conversations. I wanted to be able to speak naturally and have an assistant capture what matters, remember it, and take action without interrupting my workflow.

That became heyMisu, a voice first personal AI assistant for macOS that helps you remember, organise, research, and get things done.

What it does

heyMisu lives on the Mac and can be activated instantly with a keyboard shortcut. Instead of opening a traditional chat window, the user can speak naturally and ask Misu to help with what they are doing.

It can:

• Create tasks, reminders, and calendar events
• Search the web and compare information
• Open websites and supported applications
• Capture notes and important details from conversations
• Remember useful context for future interactions
• Review unfinished work and prepare a daily brief
• Understand the current screen when screen context is intentionally activated
• Record meetings and turn them into transcripts, summaries, decisions, and action items
• Transform conversations and research into structured Insights using GPT 5.6 models.

The Insights experience is designed for requests that need more than a quick spoken answer. For example, a user can ask Misu to compare several products, analyse a meeting transcript, organise research, or turn a collection of ideas into a clear plan.

GPT 5.6 then reasons across the available context and produces a structured result that can include summaries, recommendations, comparison tables, action items, and supporting information. These Insights are saved so the user can revisit them later instead of losing useful answers inside a temporary conversation.

The goal is not to make users spend more time talking to AI. The goal is to help them reach a useful outcome faster.

How we built it

heyMisu was built as a voice first macOS application with a compact interface that stays available without taking over the user’s screen.

The app combines different AI systems based on the type of task being performed.

GPT realtime 2.1 mini handles low latency conversation, intent detection, and tool calls. This allows Misu to respond naturally while also triggering actions such as creating reminders, searching online, opening applications, or retrieving saved context.

GPT-5.6 is used as the deeper reasoning layer. When a task requires research synthesis, planning, transcript analysis, comparison, or a polished deliverable, the app passes the relevant context to GPT 5.6. The result is then converted into a structured Insight and stored locally inside the app.

This separation was intentional. The real time model acts as Misu’s ears and voice, while GPT 5.6 acts as the reasoning engine for more complex work.

Codex was a major part of the development process. I used it throughout the project to:

• Plan and refine the app architecture
• Implement new macOS features and interface components
• Connect voice interactions to tool calls
• Debug audio sessions, state management, and asynchronous actions
• Improve prompts and structured model outputs
• Refactor code as the project expanded
• Investigate bugs and iterate quickly on ideas that would have taken much longer to prototype manually

Rather than using Codex only to generate isolated code snippets, I treated it as a development partner. It helped me move between product design, implementation, testing, and debugging while keeping the project small enough for one person to manage.

Challenges we ran into

One of the biggest challenges was making a voice assistant feel useful without making it intrusive.

A real personal assistant needs enough context to help, but users should always understand when audio or screen information is being captured. I designed normal interactions around an intentional keyboard shortcut, with clearer session based modes planned for longer activities such as meetings and presentations.

Another challenge was deciding which model should handle each task. Real time voice interactions need speed, while meeting analysis, research, and planning require deeper reasoning. Trying to make one model handle everything would have made the experience slower and less reliable.

Tool execution also required careful design. Asking an AI to open an application is relatively simple. Allowing it to perform actions inside applications requires dedicated integrations, permissions, safety rules, and reliable feedback. I therefore focused on a smaller set of actions that could be performed clearly and predictably.

Long meeting recordings created a different problem. Sending an entire live audio session through the conversational model would be inefficient. The better workflow was to record the meeting, produce a transcript, and then use GPT 5.6 to analyse the text and create a structured Insight.

The final challenge was scope. There are many directions a personal assistant could take, but adding more features does not automatically make it more helpful. I had to focus on the core loop: speak, understand, act, remember, and produce an outcome.

Accomplishments that we're proud of

I am proud that heyMisu feels like one connected assistant rather than a collection of unrelated AI features.

Voice commands, screen context, memories, meetings, tasks, research, daily planning, and GPT-5.6 Insights all support the same purpose: helping the user continue from where they left off and turn thoughts into completed actions.

I am also proud of building the project as a solo developer. Codex allowed me to prototype, troubleshoot, and iterate much faster, but I still had to make the product decisions, design the experience, test the workflows, and decide what the assistant should and should not do.

The most important accomplishment is that heyMisu does not stop at answering a question. It can remember the result, perform an action, or transform the conversation into something useful that remains available later.

What we learned

I learned that the difference between a chatbot and a personal assistant is not simply voice.

A useful assistant needs continuity. It should understand what the user is doing, remember what matters, recognise unfinished work, and help move things forward.

I also learned that different AI models are better suited to different parts of the experience. Real-time models are ideal for natural interaction and fast tool use. GPT-5.6 is more valuable when the task requires judgment, synthesis, structure, and deeper reasoning.

Another important lesson was that users need visible control. Listening states, tool activity, permissions, and saved memories should never feel hidden. The assistant should make its actions understandable without constantly interrupting the user.

Most importantly, AI becomes much more valuable when it creates outcomes instead of producing disposable responses.

What's next for heyMisu

The next step is to make heyMisu more proactive while keeping the user fully in control.

I plan to add live co-pilot sessions for presentations and meetings. A user will be able to prepare talking points, start an intentional listening session, and have Misu quietly track what has been covered. Instead of interrupting, it can provide subtle visual prompts and only speak when requested or when there is a natural pause.

I also want to expand Insights into richer visual briefs with interactive comparisons, charts, research sources, meeting decisions, and project plans.

A visible activity panel will show what Misu is currently doing, such as researching, analysing, creating an Insight, or saving a result. This will make agentic work feel clearer and more trustworthy than an unexplained loading spinner.

Future versions will include deeper personal memory controls, more integrations with macOS applications, stronger automation workflows, and additional on device processing options for privacy-sensitive information.

The long term vision is for heyMisu to become a personal AI that understands the continuity of your day. Not an assistant you have to manage, but one that helps you remember what matters and turns your words into progress.

Built With

• codex