Pix Museum

• 

  pix museum pipeline diagram

Inspiration

things around us, from the phones in our pocket, to computers we use, to airplanes in the sky, and to rockets soaring to the space, all have stories of human inspiration, struggle, creativity, and discovery behind them. our education right now is lacking in good storytelling, and i want to leverage ai to bring those inspiring, real-world stories to kids.

What it does

pix museum currently has 60+ virtual exhibits. kids can select any exhibits, and pix, the ai companion, will tell the stories about the exhibits to kids using a combination comics, voice interaction, and quizzes. users can also create and explore custom exhibits.

How we built it

i used:

• gemini live api for pix's realtime voice interaction
• gemini 3 flash with google search grounding for text content generation and maintaining the memory about the user
• nano banana for generating both the stories and custom exhibits

Challenges we ran into

• gemini live api websocket crashing when interruptions were not gracefully handled
• customizability/instruction following ability of gemini live api. i could not get gemini live api to reliably trigger certain functions or follow certain instructions. this issue was worsened by the fact that sometimes gemini live api got overwhelmed by too much context (during story narration, for example, even with google search grounding enabled, gemini live api could produce factually wrong responses that it would not produce during a normal session). this could potentially be solved by some kind of 2-stage process that involves context filtering first?
• how to properly update the memory about the user, and which parts of the memory should the agent's response be based on? still far from satisfactory

Accomplishments that we're proud of

despite the many flaws it has, it looks like a thing.

What we learned

building high-quality ai voice experience is hard. our auditory information bandwidth is so much lower than our visual bandwidth, so we're less tolerant of audio ai slop :(

What's next for Pix Museum

• fix voice interaction bugs
• improve the agent's memory system/instruction following
• add database and make it social
• put the app in front of kids and iterate based on their feedback

i would be happy even if the app ends up being just another toy - but a toy that opens a window to stories that inspire :)

Built With

• css
• firebase
• gemini
• genai
• html
• javascript
• vite