VR is all about immersion. When make I an in-app purchase in VR, I don't want to be confronted with a floating menu. I want to be able to talk to an in-game character to buy products. For example, if I'm playing a monster-hunting game and need a sword, I should be able to go up to a blacksmith and ask him for a sword.

Market Need

VR is the next big thing and in-app purchases are the fastest growing monetization source for game developers. Floating menus completely break the immersion of the VR experience. Gamers are more likely to buy items if they can just talk to a blacksmith character, or a clothing designer character, or a gunsmith character etc., because it's more realistic. Reading menus also causes massive eyestrain. Bottom line: virtual reality is intended to mimic reality so players should be able to simply talk to game characters to make purchases.

What It Does - Example

Let's say we have a hypothetical game called Legends of the Amazon. It's a virtual reality fantasy role-playing game. We're about to fight the first boss of the game, so we need to buy a new sword. Let's also say we're a famous streamer, so lots of people are watching us play.

While exploring, we come across a swordsmith going about her day. With VR Vendor we can have a natural conversation with her. We say, "I want to buy a sword." She says, "Ok, what metal should I make it out of?" She asks a few more questions and then makes the sword (see video). The game charges our stored credit card automatically.

Example game

While we keep playing, the VR Vendor platform automatically sends a message to our steam chat, telling our audience about the item we just bought and giving them a link to buy it too (if they have the game). This link could be a referral link for us, the streamer.

Example game

Our viewers see how awesome the silver sword is when we slay the boss (a dragon, obviously). So they all go buy the sword, increasing profit 10x for the game company.

Sword html

How it Works

VR Vendor is currently integrated with Unreal Engine 4 (UE4) , a popular commercial game engine. I've also partially integrated it with Amazon Lumberyard game engine, but it only has text-based support.* Here's the workflow:

Front-end: The client-side magic all happens in one python script. The player approaches a vendor character and talks into their mic. Pyaudio records the player until they are done speaking then automatically writes to a wave file. Then, I use AWS's boto3 lex-runtime interface and make a post_content() request to my Lex bot with the wave file as argument. The lex bot responses are all one word names such as "cancel", "hello", or "sword_metal" that correspond to local .wav file names. I append ".wav" to the end, and tell the game engine to play the file (e.g. "cancel.wav") at the location vector of the vendor character's mouth.

Back-end: Lex communicates to a lambda function that has three purposes: slot validation, communicating with our stream, and payment. Right now, for demo purposes, I constrain the player to only buying an iron sword with a leather handle. Slot validation works just like the AWS's default "Order Flowers" example. When "ReadyForFulfillment" is true, I use python's socket library to send a message to our stream. You could imagine this being the streamer's Amazon referral link so they can get a cut. Payment is handled by Amazon Pay, which is in sandbox mode right now so it doesn't do anything.

*Lumberyard is still in beta and currently has no method to take audio input.

Future Potential

VR Vendor is currently an alpha demo. It is intended to demonstrate the desirability and potential to make VR in-app purchases seamlessly using speech interactions with in-game characters. The production version of the platform will need an SDK for game developers to integrate it into their Unreal Engine games.

Testing. Please follow the GitHub Setup instructions. You will need to create a bot that has outputs corresponding to the names of my .wav files. The bot is just two slots for "handle material" and "sword material". If you like, please contact me requesting access to my bot at jbecke at wharton dot upenn dot edu.

Built With

Share this project: