The Pen is mighter than the Sword

Dive into "The Pen is Mightier," a VR 2-player hand-tracked painting game blending creativity & competition. Replicate Dall-E images, enhance with Stable Diffusion, and let CLIP AI decide the winner.

Comment

Inspiration - 💡

Our inspiration for "The Pen is Mightier" is threefold: 1) We want to merge artistic fun and competitive madness in an enjoyable, immersive environment, with added fitness benefits while painting away; 2) We aim to make the art creation process easier by integrating advanced AI technologies and VR hand-tracking; and 3) We seek to foster group-bonding, bringing people closer together through shared artistic experiences.

What it does - 💭

In this two-player Virtual Reality game, the players compete to replicate an AI-generated image from DALL-E. They are given 20 seconds to create as accurately of a replica as they can. After which, it is sent to be refined by Stable Diffusion and then the CLIP AI model judges the winner based on the best resemblance to the original image.

How we built it - 🔨

Our team of six undergraduates from Stanford, Duke, and Rice University collaborated to develop this VR experience using the Unity game engine and its XR Toolkit alongside Rendering software such as Blender and several AI's. After brainstorming and gaining a collective plan, we decided to integrate Dall-E for image generation, Stable Diffusion for image enhancement, and CLIP for AI judging, all the while being sure to have built an engaging virtual environment to immerse users while they interact with the game.

Challenges we ran into - 🩹

Like many others, we faced many challenges over this 36-hour long Hackathon. From as complicated as optimizing the Virtual Reality experience for the various VR devices in the industry, to as difficult as synchronizing the different AI technologies to ensure smooth performance, and as unexpected as making sure all our team members were on the same schedule as we worked in 3 different Time Zones (PST, CDT, EDT).

Accomplishments that we are proud of - 😆

Some of the achievements our team is proud of were outreaching to students from different schools to gain a wider experience-base, teaching each other our own tricks, and brainstorming this mess of a project and somehow gluing it all together. As--by far the biggest accomplishment--we're happy about is how we successfully combined Dall-E, Stable Diffusion, and CLIP all into a single immersive VR experience. Not only does it fosters greater creativity and enhance artistic skill development, it is also enjoyable and accessible to a wide audience.

What we learned - 📙

Over this Hackathon, we learned the importance of teamwork, communication, and time management in creating a successful VR project. It was essential for us to correctly spread out our roles, crucial that we helped each other out, and even more so vital that all worked together towards the same goal. Through challenging ourselves this long-long weekend, we have gained valuable experience in integrating various AI technologies, building immersive and seamless VR experience, and more than anything learned why Teamwork makes the Dream work.

What's next for: "The Pen is Mightier" - 📈

Some of the plans we discussed but didn't get to were additional features such as customizable time limits, difficulty levels, custom prompts/image categories, and more game modes. We also aim to potentially transform the art creation process from 2-dimensional to 3-dimensional, adding a whole new level of complexity to our project.

Built With

Share this project:

Updates