Tobii is the world leader in eye tracking technology. Founded in 2001, the company has built its reputation on providing millions of families with the tools it needs in order to better communicate. Eye tracking was born as a research tool, but soon exploded in the specialty market where users were able to communicate using our products (Watch Sebastian Story). Since then we have uncovered countless uses for this technology.

The inspiration for Tobii Ghost came about whilst watching speech tablet users, We soon realized that we were trying to guess what they were trying to say just before they actually said it, giving us a visual representation of user intention.

With this thought formed and being avid gamers ourselves, we began toying with the idea with the belief that this had great potential ONLY IF we were able to have a good visualization tool to display exactly where and what a person is looking at.

Challenges we ran into

Back then we didn’t have a good visualization tool for eye movements on screen. However, there did exist a lot of experiments such as Dwell-Free that allowed us to trace a succession of actions based on heat maps. Whilst innovative, it wasn’t really suited to gaming. We needed something less intrusive and based on what we experienced, heatmaps were a step in the right direction in terms of achieving fluency.

So, we accelerated the decay of the heatmap and that created a kind of a waterdrop effect in the screen (The first bubble!). We then created a different variation of that technique that ended up being the customization feature in our extension. After that things gathered steam. We showcased the functionality to Eleague and we were promoted in a "Getting Technical" segment that became quite popular in the show. After that (by content creators/streamers request), we made this a streaming tool.

Accomplishments that we are proud of

The 2nd challenge came when we had the option to transform this streaming tool as an extension, by that time twitch was working in a new version of their extension store showcasing a new way of visualizing extensions in categories, that will boost discovery for newcomers. Of course, we did not want to lose this opportunity, but we had 3 weeks to research how to build this and actually do it!

Also, we have no branding at all, we call it internally the "Gaze overlay", so we had to have that in mind too and what was more difficult by then: "How we can help our users to explain to their viewers that the bubble means eye tracking?". It was a huge challenge, a very exciting one with a lot of office hour but also with a ton of fun since we were doing something that no one has seen before in gaming and/or eye tracking :)

We have seen an amazing reception that been translated into a great exponential organic growth. This has not only allowed us to expand in streamers numbers but also has open doors in esports events as a broadcasting tool: Rainbow Six US Invitationals

What it does

Empowers the viewer with the ability to see where the streamer is looking in the screen. Watch Demo

How we built it

There are various components involved in transferring streamers gaze on their screen to a stream on Twitch. The easiest way is, of course, to burn the gaze bubble into the stream. However, this won't offer the best user experience for the stream viewers as they don't have any say in the whole setup.

We started to develop the Twitch extension based on the Tobii streaming gaze application which we already had. It now needed to push the streamer's gaze data to the viewers, luckily Twitch's PubSub API fitted that purpose. From the viewers perspective, they would get streamers gaze data and could customize it according to their preferences or turn it off altogether.

Tobii Ghost windows application passes the gaze point data to Twitch using products offered by Amazon Web Services, then Tobii Ghost extension on Twitch will receive the gaze point data and will try to visualize it with the help of WebGL.

What we learned

That eye tracking is a great addition to streaming for 3 main reasons:

a) To allow viewers to understand the streamer decision-making process while they play.

b) To allow viewers to see streamers reaction time, been able to measure Streamer eye-hand coordination with their own.

c) An extra tool for educational streamers to showcase their game awareness and what they focus is in the different stages of the game.

in addition, being able to have access to all the data that streamer generates with AWS has made us learn a lot about our users. What we initially thought was a great product for the battle royale and overall shooter genre, has given us an amazing response in some other communities. That has been the case with OZU!, Tetris 99, Guitar Hero, Chess and Poker and IRL.

We have also learned how to create a community out of this specific extension. What started as a support channel for new users, today is a place where streamers are constantly discovering new ways to use their eye tracker to connect deeply with their viewers. We are very aware that we have only just started and there is a long path to follow in terms of development, but for now, we have achieved great things and one of them is celebrating user content. We have done that not only by placing them into the spotlight.

What's next for Tobii Ghost - Eye Tracking Twitch Extension

There is a lot cooking, so going forward we intend to make sure we continue the development of this tool by listening to our community feedback since we believe that user-centered design is the best way to explore the future of eye tracking in streaming.

Based on that we are looking forward to working together with Twitch in order to fix time sync. Right now that is a process that needs to be done manually for every streamer. We believe that the outcome of this will be a big revolution since we will be able to provide the social aspect of eye tracking allowing viewers to make clips including Tobii Ghost.

We are also looking forward to extending our experience to more devices, with mobile as a big target. Lastly to be able to give streamers way more control and data over how their viewers interact with the extension, so they can see what decisions boost engagement.

+ 16 more
Share this project: