Talkitecture

Inspiration

Cities are full of stories that often go unnoticed, with people rushing past important areas without knowing their history or cultural meaning. Inspired by AR applications like Stellarium and Pokemon, we wanted to overlay fun and expressive characters into city settings.

What it does

Talkitecture brings buildings to life using augmented reality, allowing users to point their camera at certain monuments to create an animated character that tells about its history, cultural context, and artistic significance. It transforms everyday urban exploration into an interactive, educational experience.

How we built it

We used computer vision to detect buildings and anchor AR content directly onto them, overlaying contextual information and visual elements in real time. Once a monument is recognized, we use Gemini API to retrieve educational information about the building, and use ElevenLabs for text-to-speech, finding characters that match the vibe and cultural context of the location.

Challenges we ran into

Some of our main challenges were accurately detecting buildings at different angles and overlaying good-looking facial features.

Accomplishments that we're proud of & what we learned

It was really cool to be able to connect the code to our phone cameras and try it out on buildings around us. We learned how to do object detection, use APIs, and act as project managers, breaking down tasks into smaller pieces.

What's next for Talkitecture

Integrating GPS and phone orientation for quicker identification of buildings; better looking building face models; ability for multiple building characters to appear at once; more animated/expressive character designs.

Built With

• elevenlabs
• gemini
• javascript
• opencv
• pytorch