RealityPaint

Inspiration

We were inspired by digital art in the physical world. We wanted to come up with a unique way to combine computers with art, and empower people to transform their spaces. We came up with this idea as a result of hours of brainstorming, and we're really excited to see where it goes in the future!

What it does

Turn any surface into your digital canvas with RealityPaint using a projector and a camera. RealityPaint is an interactive augmented reality interface that blends digital art with the real world using projection mapping. A projector and a vision system dynamically map real-time graphics onto physical surfaces and mask out objects. There are many applications to this, from displaying art on unique surfaces, to giving presentations without glare, to drawing on physical surfaces.

How we built it

We use a Microsoft Kinect sensor with an off the shelf projector, open source cross-platform drivers for the Kinect, and a custom Python wrapper. The image processing is done in OpenCV, and the image generation by DALL-E 3.

Challenges we ran into

We initially started our project with a webcam running multiple algorithms to detect objects in the foreground and background. We kept optimizing our algorithms, thinking we would have good performance, but on Saturday night we realized this approach wouldn't work. It caused a feedback loop with the projected images, and failed to reliably and accurately mask objects, especially those that were moving. We decided to switch to a Microsoft Kinect over night, and this caused lots of issues. We had to find drivers that worked on macOS and Windows and Python wrappers for them. We ended up changing some of the source code for the Python wrappers as it was unfinished and relied on deprecated packages.

Accomplishments that we're proud of

We're really proud of all the work we've done, and how much we've learned this past weekend about computer vision and sensing. We think this idea is a great one, and are really happy to have built a system like this, even if it didn't work perfectly.

What we learned

Computer vision is really, really hard! Debugging/visualizing matrix operations is extremely challenging, and working with old hardware with deprecated packages is not fun! There are so many challenges to getting accurate, reliable readings, from changes in systems, drivers, and lighting conditions.

What's next for RealityPaint

Improving our depth masking algorithms and adding more features for user interaction like a painting/graphics drawing mode, and being able to detect objects and label them in the physical space.