[S19] - Chameleon

Inspiration

Lucas had the chance to test the Logitech MX Ink on the Cologne XR Hack where he was a mentor, and was inspired by the tactile tip nature of the Pen, which allows for new kinds of interactions when put together with Mixed Reality. Just the idea of being able to use the pen to touch a real object to "extract" its color was exciting and enough for Lucas to decide to go to the Stockholm hackathon at the last minute and build the idea, where was joined by Behnaz and Nas.

What it does

The app brings familiar editing software tools such as color picker, clone tool, and AI fill to the real world around you. You can use the Pen and touch your favorite plant to get that nice green shade, mark an imperfection of your room that you want to erase for content creation, or use AI fill to replace artwork on your walls or change the style of your carpet.

How we built it

Using Unity, Meta's presence platform passthrough features, and the Logitech MX Ink tracking and tip pressure detection, we were able to get a pixel array of what the pen is touching in the real world, averaging a color, or cloning its texture as needed.

Challenges we ran into

Passthrough camera feed access is not accessible for developers, so we had to use a mix of different techniques such as the AI Capture Kit, and trev3D QuestDisplayAccessDemo repository which uses Android's MediaProjector API. After getting access to the camera feed, the challenge was to actually convert a world position within Unity to the position of the 1k 2D texture that we had access to.

Accomplishments that we're proud of

We are VERY proud of being able to touch a real object with the Logitech pen and "extract" its color or texture.

What we learned

We learned how to use the Logitech MX Ink developer package for Unity, which easily exposed all the important variables of the pen that we needed, such as button input but most importantly tip pressure value.

What's next for Chamaleon: Reality Editor

The next step is to keep building the rest of the tools that would make Chameleon the Reality Editor that we envision. Using Meta's depth API to accurately paint and replace colors and textures in the real world around you and implementing AI services that would allow for quick texture swapping, object removal and reskinning.