Leverage new technologies to elevate the artist experience.
What it does
Start a creative session by interacting with the virtual assistant bot which suggests art themes and displays creative pieces by other artists that fit in the user selected theme. Based on this inspiration, the artist can input their drawing on the Synaptics Touch Screen. See the drawing come to life on the Hololens in real time and become three dimensional with integrated model mapping. Interact with various paintbrush modification features between the Touch Screen and Hololens with a wide array of gestures. Then capture your drawing, augmented onto reality, and interact with the virtual assistant bot again. The bot will try to identify what you draw and optionally posts to Social Media.
How we built it
- Bot - Microsoft Bot Framework
- Bot recommendations based on art themes - Bing Search from Microsoft Cognitive Services
- User input - Synaptics Touch Screen
- Augmented reality - Microsoft Hololens
- Machine learning drawing identification with custom training - Nanonets
- Backend synchronization - Firebase
- Social Media - Facebook Graph
Challenges we ran into
Creating a valid input stream from the Synaptics touch screen to create a consistent output on the Hololens drawing canvas.
What's next for HoloSketch
Expand database of custom trained objects for drawing identification.