Night at the Rizzeum

Title Page

Inspiration

Over the winter, the two questions we got asked the most were: 1.) "When are you bringing back a boyfriend/girlfriend?" 2.) "When are you going to make good use of your degree?"

Hence, upon seeing the problem statement for the PINUS Hack, we decided that the most logical option was to create a dating sim where you can date paintings and the paintings can "look" back at you (obviously).

In addition to that, we were also inspired by the game "Date Everything!", which is a dating sim where you get to date random stuff in the household.

What it does

Rizzeum is a dating sim x product discovery x agentic commerce application that does the following: 1.) Allows admin users to publish new artwork products 2.) Converts said artwork into hot, conversable people 3.) Has a Matchmaker Aunty in the main interface that talks to you to figure out what you like 4.) Allows a bunch of users to join in on "live auctions" for the humanized paintings 5.) Allows the users to speak to the humanized paintings, to: i) Find out more about them ii) Flirt with them, to unlock the ability to buy them iii) Flirt with them, to lower their price to an admin-set minimum (or close to) 6.) Purchase the paintings via a Stripe checkout dialog box 7.) View their purchased paintings in a 3D viewer 8.) Preview what their painting would look like in their house

How we built it

We used Gemini Live API for the voice agents, using some little hacks to re-use it as a transcription model. Some basic system prompting was used to inject the personalities and voices used in these agents, and Nano Banana (also part of the Gemini ecosystem) was used for avatar design.

All backend web stuff was done with Python FastAPI and Websockets (for all of the live stuff).

Frontend stuff was made with Next, Three.js, Tailwind.

Challenges we ran into

Biggest challenge was definitely working with the 3D scenes. In hindsight, no clue why we decided we needed a 3D scene for the frontend, but dangit does it look cool.

Second biggest challenge was probably working with the live API and making the Gemini Live API work natively with our custom-built agent tooling. It was hard to get Aunty to switch artworks!

Accomplishments that we're proud of

1.) Orchestrating 2 live, multimodal, self-looping and goal-seeking agentic systems 2.) Getting to work a lot of new tech (e.g. our Stripe system, our avatar gen system)

What we learned

We should probably keep track of time better - a lot of the issues in the making of this project were more of project management issues rather than technical issues (though there were many).

What's next for Night at the Rizzeum

A lot of bug fixes, but hopefully a partnership with independent artists or even museums / art galleries in Singapore looking to sell their artwork or just display it in a fun and engaging way.