SKRIEN

Inspiration

The inspiration goes to my 8yo daughter. She watches youtube videos and tries to run with people doing exercises on TV. And she likes games, so this is a combination of both, to have fun with the family on TV.

What it does

SKRIEN is an endless adventure game, no level play is the same, it could go continuously forever in theory. Control a 3d character in the TV with real life movements. Play it in any living room, or any 2m space in front of TV, with your family, kids or by yourself for an exercise. Possible to pair with phone's camera for improved motion control.

How I built it

The game itself is rendered using Three.js on WebGL. It generates everything on the go - the course/path, timings, scoring, session, obstacles etc. The motion detection uses Google MediaPipe Pose Landmarker, on the web app opened on the phone or laptop. It is a light model, utilizing WebAssembly under the hood, it can turn 33 body landmarks into game actions. A light web relay server running on Node.js that connects the controller phone with TV via secret code and send commands via websocket. The video is neither recorded, nor leaves the phone. Everything is decoded and analyzed on the phone, and only commands with numeric values leave the phone, like "run 0.8", "jump". All of this is built for the Vega OS, using "vegaWebview" template, a small React Native wrapper for the app itself.

Challenges I ran into

The tricky part is how to actually control the character movements. One way is with the remote control, but it does not really give the excitement after some time. However, actually running on place, sliding, ducking and jumping gets you moving. Then, how to capture the person movement - camera. But TV usually do not have cameras, thus need a separate device, laptops are big, phones however are perfect. and almost everyone has one. So, the challenge was to connect phone's camera, track human body pose and movement patterns, then analyze and convert them into set of command to send over to the TV's app of SKRIEN. This was done with Computer Vision, decoding the landmarks coordinate changes and via patterns sending single command to the app via websocket conenction, for lower latency. Also, the running detection as a whole from the front was tricky to at start.

Accomplishments that I am proud of

I managed to have a lot fun with my daughters while testing the movement patterns and calibrating them. Tehcnically - I managed to hook everything up, also the faster running on place increases the character's pace in the gameplay, this was quite a challenging pattern to detect properly.

What I learned

I knew a lot of web development, mobile app development, but this was first experience with developing for TVs. It would say it is quite similar to app development, but it was fun and a learning process, nevertheless.

What's next for SKRIEN

Increase the amount of different levels, add more intense-building music and sound effects. After that could try to introduce multi person mode for 2 person competitions.

Built With

• javascript
• mediapipe
• node.js
• react-native
• three.js
• vega
• vegaos
• webassembly
• webgl
• websocket