Paint By Fingers

Paint By Fingers is insipired by the popular "paint by numbers" concept - now in XR, 3D and hand tracking! Paint By Fingers is compatible with Travel Mode on airplanes and trains - relaxing painting time wherever you go ✈️

What it does

User can pick a model from the model collection (3 example models in the hackathon demo), and pre-selected paints appear. Pick a color and find the spots to paint with it! A reference image of the ready-painted model can be found next to the model. 🖼️

How I built it

While creating a more freeform drawing application, an idea of paint-by-numbers variation of the app came up, and Paint By Fingers got born for the hackathon. Meta's hand tracking technology is used through open standards - OpenXR & Unity XR hands, and the turntable interaction is using code from Ultraleap's open source hand tracking component (Apache 2.0 - https://github.com/ultraleap/UnityPlugin). Significant portion of the application is coded with Cursor & GPT-5, which were really helpful during the creation.

The experience is reusing interactions and assets created earlier for still unpublished more freeform drawing & painting application, which were adapted to a new mechanic & application concept during the hackathon time.

Challenges I ran into

User's hands and the digital position don't quite match in mixed reality mode, which is causing confusion for inexperienced users. That is why the user is started in the VR mode despite the application being perfect for MR. This needs to be looked at with more time, to be able to mitigate the issues coming from the visual mismatch.

Accomplishments that I'm proud of

The interactions with hands feel natural and playful! I'm an indie developer with background in UX, and I'm proud reaching the level of being able to create whole apps from scratch :) Peter has helped me modelling some of the 3D art.

What I learned

AI was an amazing help in creating great UX animations quickly. Hand interactions also feel very natural and are definitely the future of inputs!

What's next for Paint by Fingers

It'd be amazing to find funding for creating more 3D models to the catalog, and release Paint By Fingers with model packs. There's also opportunities with artist collaborations and daily art, where the user gets a new model daily to paint. Peter Nancollis helpfully modelled a few example models for the experience to get started!

A few UX things that can definitely be tweaked further, and there's lots of ideas for polish - e.g. giving a hint feature on where the color is, creating exciting shaders for the paints and could even have model-specific environments, assets and sounds when the model is switched!