Cookie & Cream AI

Language learning for intermediate+ levels with an AI conversation partner that teaches you the word you don't know

Comment

Inspiration

Most language learning tools focus on vocabulary. But I found that the best way to improve is to speak to my friends in the language and let them correct you or teach you in the conversation. This tool lets you speak to an AI as practice and also teaches you new vocabulary as you speak.

What it does

Cookie & Cream is a real-time AI speaking coach built to help people recover mid-sentence without breaking the conversation.

In the demo, the user speaks live, the system listens in real time, and if the user slips or gets stuck, Cookie gives a quick whisper-style repair. Once the user is back on track, Cream continues the conversation naturally. That creates a much smoother loop than stopping everything for a correction.

The current demo also includes recap and translation support, but the core product moment is the live recovery loop: speak, get just-in-time help, recover, and continue.

How we built it

We built Cookie & Cream as a voice-first web app using React and TypeScript on the frontend and a TypeScript backend for the live AI pipeline.

Amazon Nova powers the main experience. Nova 2 Sonic handles the live voice loop, including streaming speech understanding and spoken response playback. Nova 2 Lite handles the durable reasoning layer, including turn routing, recap generation, and translation support. We designed the product around two distinct AI roles: Cookie is the lightweight recovery coach, and Cream is the natural conversation partner.

On the product side, a big part of the build was not just model integration, but interaction design. We had to make the coaching feel fast, helpful, and minimal so that it supported the conversation instead of taking it over.

Challenges we ran into

The hardest challenge was latency and timing. A speaking product lives or dies by whether the help arrives at the right moment. If the feedback comes too late, the user has already lost the thread of the conversation. If it comes too aggressively, it feels like an interruption.

Another challenge was balancing live responsiveness with a reliable durable turn flow. We wanted the experience to feel immediate, while still preserving clean turn handling, recap support, and fallback behavior when needed.

We also spent a lot of time shaping the coaching behavior itself. The system had to avoid becoming a lecture. Cookie needed to say the smallest useful thing possible, then disappear so the conversation could continue naturally.

Accomplishments that we're proud of

We are proud that Cookie & Cream has a clear, memorable product idea instead of being a generic chatbot demo. The two-agent whisper-recovery loop gives the project a distinct interaction that judges can see and understand immediately.

We are also proud that the demo is truly voice-first. The live experience is not just text chat with speech layered on top. The user speaks, gets real-time support, retries, and continues the interaction in one loop.

Finally, we are proud that we kept the project honest and runnable. The app has working live behavior, recap and translation support, and a codebase that builds, tests, and demonstrates the core concept clearly.

What we learned

We learned that in live AI products, the smallest intervention is often the most valuable one. A short, timely repair can be more useful than a long, smart explanation if the goal is to keep the user in flow.

We also learned that separating roles makes the experience feel better. By giving Cookie the recovery job and Cream the conversation job, the system feels more natural than a single assistant trying to teach and chat at the same time.

Most of all, we learned that strong AI demos are not just about model capability. They are about designing the right moment. In our case, that moment is recovering before the conversation breaks.

What's next for Cookie & Cream AI

Next, we want to expand Cookie & Cream from a strong demo into a more complete speaking product. That includes broader language support, better personalization, and progress tracking such as successful recoveries, retry patterns, and time spent staying in flow.

We also want to improve the live experience with even smoother turn-taking, more natural full-duplex interaction, and more user testing to measure how well the whisper-recovery mechanic helps people keep going in real conversations.

Longer term, we think the idea can extend beyond language practice. Cookie & Cream points toward a broader class of real-time AI speaking tools that help people stay confident, present, and conversational in the moment.

If you want, I can turn this into a tighter “more likely to win” version next by making each section punchier and a little more judge-oriented.

Built With

Share this project:

Updates