This is a proof-of-concept for a scene builder using only the user's voice.

How to use

The app consists of a virtual assistant to whom you can tell instructions to create and move objects in a 3D scene.

Sample instructions (parts in italics are optional):

  • create a {object description}, named {object name}, {object position}
  • move the {object description/name} {position}

An object description can be anything, as long as it has a visual representation (it searches for images on Google). So, get creative!

Objects can be moved by specifying a direction and distance (e.g. "3 meters to the right") or a relative object and anchor (e.g. "above the tree").

Example instructions (used in the video):

  • create a blue car on the ground
  • create a green tree at the left of the blue car
  • move it 1 meter to the left
  • make it bigger
  • create a red car at the right of the green tree
  • move the red car 2 meters to the right
  • create an amazing guy named Max
  • place Max above the tree

How it works

The web app was built with A-Frame, which allows running it on WebVR-enabled devices such as the Oculus Quest.

The voice recognition uses the Web Speech API, and for speech synthesis it uses the SpeechSynthesis API.

Commands are processed and analysed using a trained model, in order to create, place, move and scale objects.

Images are searched using Google's Custom Search API (you may need an API key if you want to try your own).

What's next

I believe virtual reality and voice recognition have a great potential together for more efficient interactions, so I'm really looking forward to enhancing this prototype and hopefully see this kind of technology integrated into future VR apps.

Built With

Share this project: