Hardware, the "Penn"
Image drawn by user
Output from Neural Net
With the rapid rise in popularity of AR and VR, artistic applications that allow the user to draw in 3D space are also gaining traction, the most popular of which is Tilt brush, however there's quite a big barrier to entry: namely the cost and requirement of artistic skill. So we thought, what if we made a small and compact digital pen that allowed the user to draw in 3D space but implement some software to stylize and beautify the drawing so that more people are encouraged to explore art in 3D without the need of an expensive and bulky VR system to get started?
What it does
A digital "Penn" which allows the user to draw freely in the air to create a 3D image that is subsequently beautified in software with AI.
With the recent advances in neural networks for image processing, the application of artistic styles from famous paintings to ordinary drawings and pictures has become both possible and visually pleasing in certain situations. We wanted to apply this stylization to 3D drawings made with our AirPenn. Since there is a lack of training data for 3D "pictures" or sculptures, we decided to utilize the power of existing training sets and networks for 2D images. Our goal was to take the 3D drawings of the user, which we assumed would be likely embedded in a curved plane around them (as in many Oculus/Vive videos users draw most in the immediate area), so a non-linear dimensionality reduction such as Laplacian Eigenmaps could preserve local relationships to a relatively high degree. Then we would have a 2D image from a 3D drawing, stylize that image, and output it back to the user where we can reverse the transform to get a stylized 3D drawing. Unfortunately we ran into challenges with such a demanding form of dimensionality reduction and were unable to implement it.
How I built it
The "Penn": A 9 DoF motion sensor talking via I2C to a Bluetooth LE microcontroller which sends the acceleration data via a BLE link to the computer.
Software: A local flask application using a python BLE library that receives acceleration data and calculates relative position which are then sent to a web interface for rendering with three.js. We take the drawn 3D image and de-dimensionalizes it into a 2D image to feed into a deep learning neural network (VGG 19) that was pre-trained on a variety of artistic styles to produce a beautified image.
Challenges I ran into
Lines drawn with three.js cannot have vertices added dynamically. Instead, we made a new line for each data point. The neural network takes a lot of fiddling to work well and its many libraries are a hassle to install. Double integration of acceleration to obtain position data is non-trivial and requires advanced filtering to perform well. Originally, we planned to use the Web Bluetooth API to directly access BLE data from the web browser but Google has put the technology in a transition period just now and thus is currently unavailable.
What I learned
three.js as a flexible JS graphics library BLE and Bluetooth Web (which no longer works) Python I squared C for chip to chip communication
What's next for AirPenn
Implement better filtering and integration math to obtain position data Include rotation data to the "Penn" so that it can be rotated in the air while getting correct positional data Utilize gyroscope to help filter the accelerometer data Better hardware/ergonomics by making a sleeker design Finished dimensionality reduction and restoration for stylization with neural networks Optimize the parameters for the Neural Network such as number of phases and iterations Give the user more options More colors and brushes for the Penn