There are many data analysis tools available (Tableau, Excel, etc.) but these products take time to learn. We imagine a natural and rich conversational experience where students can simply converse with a bot to find meaning in data and observe real-time visualizations in augmented reality.

What it does

Our hack visualizes data through voice commands. A user can say things such as "plot the data", "find the line of best fit", or "find the max", and these actions will be completed on the 3D plot. The 3D plot is visualized through augmented reality.

How I built it

We built the application using Unity and DialogFlow. Unity was used for producing the plot in augmented reality. Using the AR platform Vuforia, we read in data from a CSV file, and generated spherical game objects ("dots") to represent the data points. We then generated a cylindrical game object for the best fit line.

We used DialogFlow to build natural and rich conversational experiences for the student, our user base, to interact with the plot. Dialogflow incorporates Google's machine learning expertise to take phrases such as "find best fit" and expand to understand variations on this phrase. We connected this to Unity using websockets, and plot the points or compute the best fit based on what phrase is detected.

Challenges I ran into

We ran into issues with plotting a line of best fit in Unity and with connecting DialogFlow to the Unity gaming platform. When plotting the line of best fit in Unity, most of the methods available in C# were only for 2D plotting. Thus, we ended up using line of best fit equations in order to get the correct equation.

When connecting DialogFlow to Unity gaming platform, we spent the entire afternoon determining how to integrate the two platforms. To fix this issue, we modified a deprecated library that corrected the communication issues.

Accomplishments that I'm proud of

We're proud of full integration between Google Cloud DialogFlow and the Unity gaming platform and visualizing data points in Unity.

What I learned

We learned more about DiagramFlow and how to connect to Unity. We also gained further experience with Unity and the Vuforia platform.

What's next for Connect DOT-AR (Data On-demand Teller in Augmented Reality)

Next steps for DOT-AR would be to 1.) Allow users to enter different data sets, 2.) Expand upon available commands for the data (extrapolate data, best fit lines other than linear, etc.) 3.) Add additional phrases to DiagramFlow to make speech understanding more robust

Built With

Share this project: