Cyberpunk Tales

Picture of the app
Arduino Two Button Code (Not completed)
The Hardware buttons (Not completed)

Inspiration

We wanted to create an application that had heavy use of Artificial Intelligence. We also wanted to build an application that allowed users to be able to interact with the app through facial recognition. The idea is to create an application that could be used by individuals with disabilities. Given the theme of Cyberpunk, we worked on creating a Cyberpunk game that that utilized AI.

What it does

It is a web app set in the pastel theme of Cyberpunk, that utilizes the user's camera and Gemini API. First, it displays an image of the user, with a marker for their mouth. When the user either smiles, or opens their mouth, the website captures that expression and sends a signal to website. Second, there some text dialog in the form of a story, and two buttons that indicate a choice the player can make in the story. The users can either choose by using facial recognition or by selecting the buttons on the website. Then, it will make a Gemini API call which will update the story, and update the buttons. At the end of 10 story questions, we use Gemini API to rate the entire story and give a score. Finally, we use generative ai to display an image that closely corresponds to the text that is generated, to add flare to the application.

How we built it

We divided our team into 4 groups. Ani built the website using a React Framework. Chris worked on implementing the text using API keys. That code was closely tied to the website Framework. Edwin worked on generating the images for aesthetics, and Zhou worked on implementing the facial recognition piece. Once the pieces were mostly working, Ani helped to pull all the pieces together to a coherent app. Chris worked on making Gemini responses quick enough to be tolerable for a game. Edwin worked on building the Arduino button interface (not completed) and Zhou worked on the MongoDB (not completed).

Once we had the skeleton of the app built, we used Gemini AI browser and Claude AI to help us develop and refine many of the pages. Claude AI was also used to help make the Gemini Story prompts run in a timely manner.

Challenges we ran into

We had some serious issues with our versioning. During several times at night, we felt like we were "programming backwards", meaning our code was reverting further and further back to older incomplete versions of the application. We also had several merge conflicts that set us back several hours.

We had some trouble getting the Arduino to integrate with the react web application. We got the Arduino to recognize button presses, but we were unable to get the signal to the web application in time.

Another issue we ran into was keeping up the moral of the team during the early hours of the hackathon. The lack of sleep made coding way more difficult as we tried to press on with our ambitious app development.

Accomplishments that we're proud of

We really nailed the theme of the competition. With that, we were really able to learn how to implement facial recognition and get that part of the project running smoothly. It is one of the more impressive parts of the app.

The second thing we were able to do was to get the Generative AI to run very quickly and smoothly. We feel the app is really impressive in its ability to generate images on the fly, and to be able to quickly develop an AI story.

What we learned

We had one member of the team who is not natively a programmer; he is a mechanical engineering major. He got the chance to work on wiring the buttons and developing hardware. We also had two members of the group who were relatively new to git, so they got the chance to see and learn what git did and how it really helps with versioning of the apps.

Two members were able to refine their skills in facial recognition software. There were able to adjust the math to be able to further refine the facial recognition so that it responded better to input.

Finally, there was quite a bit of learning implementing and refining Generative AI APIs.

What's next for Cyberpunk Tales

There are several things we would do to make the project better.

1) We worked on creating an external button interface for another alternative to selecting buttons. However, we ran into some issues with getting the Arduino to easily and reliably talk to the website.

2) We would have liked to have implemented Solana in order to capture the scores and stories. It would have been nice to be able to see all the stories that were created for a history, and to have a leaderboard. Alternatively, we thought about implementing MongoDB, but we were unable to polish it for the final release.

3) We would have liked to have generated music through Generative AI. We looked at implementing ElevenLabs for the music, but that was a bit much for us to implement.