Inspiration
We were inspired to use hardware to explore the possibility of entering into a new world, determined to show the user an engaging experience.
What it does
You control a Cutebot with a handheld controller, which holds a phone to see the buggy's surroundings. The phone's camera feed gets fed into a laptop and at this point an API call is made to Decart.ai, transforming the feed into a new world. If music is playing at the same time, a call is made to Gemini to determine what kind of world should be shown.
How we built it
We used a python localhost and used a websocket to connect the phone's feed.
Challenges we ran into
- Being stuck with one micro:bit
- Me (Jonathan) not having any internet connection on my laptop and having to hotspot
- Shaky connection between the phone and laptop
- Running out of API credits for Gemini and Decart
Accomplishments that we're proud of
Getting it all working
What we learned
How to connect a phone's feed to a laptop and transform it in real-time
What's next for World Overlay Mapper
- Make the world interactive through the laptop
- Pre-rendering - This would reduce the amount of calls we would have to make to Decart, but would require more powerful hardware