yuna nano

A live agent can help you with everything on your PC.

Comment

Inspiration

We wanted to build an AI tutor that feels less like a chatbot and more like a real study companion. A lot of tools can answer questions, but very few can listen to you, understand visual context, and respond in a way that feels natural and immediate. That idea led us to Yuna Nano: a multimodal learning assistant that can hear, see, explain, and stay present as a lightweight desktop buddy.

What it does

Yuna Nano is a browser-first and desktop-friendly multimodal tutoring agent. It connects to Gemini Live for real-time conversation, streams microphone audio, accepts camera, screen-share, and uploaded images as context, and can generate explainer diagrams to support learning. We also built a desktop mode where Yuna can live as a floating face and expand into a full chat window when needed.

How we built it

We built Yuna Nano as a TypeScript monorepo. The frontend uses React and Vite for the tutoring interface, the backend uses Express to mint Gemini Live ephemeral tokens and handle tool endpoints, and Electron powers the desktop overlay experience. We also added shared contracts between apps so session state, artifacts, and tools stay consistent across the stack.

On the AI side, we integrated Gemini Live for real-time voice interaction and multimodal understanding. We also built a diagram-generation flow that can create and persist teaching visuals, so Yuna is not limited to plain text responses. To make the system more reliable, we added fallback behavior when image generation is unavailable, which kept the full experience testable even under quota or provider issues.

Challenges we ran into

One of the biggest challenges was making the desktop experience feel smooth on Windows. Transparent Electron windows are tricky, especially when you want a small floating dock face that can switch into a larger chat window. We also had to solve real-time state synchronization between the web app and the desktop dock so Yuna’s expressions matched what was actually happening.

Another challenge was multimodal reliability. Audio capture, screen capture, live streaming, and assistant playback all need to work together in real time. We also ran into subtle UX bugs, like making sure Yuna only looked like she was speaking when the assistant was talking, not when the user’s own speech was being transcribed.

Accomplishments that we're proud of

We’re proud that Yuna Nano works end-to-end as a real multimodal tutor instead of just a static demo. It can handle live voice interaction, visual context, generated diagrams, and a desktop companion mode in one connected experience. We’re also proud of the amount of polish in the desktop overlay, especially the floating dock face, state-based animations, and the transition between dock mode and full chat.

Another accomplishment is the architecture itself. The project is split cleanly across web, API, shared contracts, and desktop, which made it much easier to iterate quickly while keeping the product coherent.

What we learned

We learned that building a good multimodal product is not only about model capability, but also about interaction design, latency, and state management. Even small expression or animation mistakes can make an assistant feel confusing or unnatural. We also learned that desktop UX brings a completely different set of engineering constraints than browser UX, especially around transparency, window management, and platform-specific behavior.

Most importantly, we learned that an AI tutor becomes much more engaging when it can combine conversation, visual context, and a clear personality into one experience.

What's next for yuna nano

Next, we want to make Yuna Nano more personalized and more useful for long-term learning. That includes better memory, more structured tutoring flows, smarter lesson planning, and richer teaching artifacts beyond diagrams. We also want to keep improving the desktop companion so it feels even more natural as an always-available study assistant.

Longer term, we want Yuna Nano to become a true multimodal learning partner: something that can help explain problems, react to what you are working on, and support students in a way that feels immediate, visual, and human-centered.

Built With

Share this project:

Updates

Submission history