[S23] - Paintings

Inspiration

We have been building skills on WebXR for the last 6 months and we designed and developed a new framework that wraps Babylon.js and enables widely adopted Meta React technology developers to build XR immersive experiences. The application built during the hackaton is the first use case that we worked on with our framework, counts 700 lines of code and is deployed on cloud as PWA (Progressive Web Application). This technology can be used across different devices but is built for Meta Quest 3.

What it does

A user that has setup a physical space on his Meta Quest 3 headset can decorate the rooms with AI generated paintings that are also animated. The user asks with his/her own voice what to generate directly through the microphone. Multiple paintings can be generated with different layouts (portrait, landscape and square).

How we built it

We have implemented a PWA with JavaScript and React with web technologies leveraging our proprietary framework. The backend API are build with Node.js and consume Stability AI web services. We have used similar features of Meta Presence Platform provided by Babylon.js and WebXR abstraction layer.

Challenges we ran into

Meta Quest 3 provides restricted and experimental support of Meta Presence Platform through WebXR technology layer. Also some web technologies needed to be replaced and adapted. Deployment and packaging is specific for PWA and extremely lightweight.

Accomplishments that we're proud of

We have competed with other teams that use more mature Unity technology but we have proved WebXR is a viable technology too to produce application for final users.

What we learned

A lot.

What's next for [S23] DVM

We will add ability to move the paintings after creation through DnD and we will support multiple devices (Apple Vision Pro, other smartphone and tablets).

Built With

• azure
• babylon.js
• docker
• javascript
• nginx
• node.js
• react
• reactylon(proprietary)
• stability.ai
• typescript
• webpack
• webxr