Memento

• 

  You can have dynymic scenes that follow the camera

• 

  You can have dynymic scenes that follow the camera (part 2)

• 

  The Machine

• 

  Various perspectives are achiveable

• 

  Various perspectives are achiveable

• 

  Allows intergration with traditional rendering processes

• 

  Allows intergration with traditional rendering processes

• 

  The Guy

Inspiration

We wanted a project that could include "growth" in some way, that also included a fairly new technology that we were interested in called "Gaussian Splatting", which is a relatively fast way to create a highly realistic 3d scene that can run in real time.

What it does

Using a webcam and openCV / mediapipe, we track the player's face. Using their IPD, we can get an estimate for their position in 3D space relative to it. We use a neural network to generate the Gaussian splatters from a video we take, showing many angles of a scene - we then use a tool to implement these scenes in GODOT. We obtain the user's head position from the Python script using websockets, and use it as an input control for the position of the camera in the game. The game is essentially a visual novel / tech demo, which shows off various scenes using this new technology.

Challenges we ran into

Selecting particular scenes for "splatting" was quite difficult - a lot of the time it didn't generate a good scene because the camera angles chosen from our footage weren't enough for it to calibrate a point field of the scene. Another problem was that our original plan was to do some complicated vector / matrix maths to try and project a specific scene into the viewport so that it would seem as if the screen was a "window" to this 3d scene. We achieved a similar, much less complicated effect however.

Accomplishments that we're proud of

The realism of the scenes is really cool, and we're very happy with the fact that we were able to use a brand new technology that we haven't tried before and make something unique with it.

Built With

• godot
• python