Sliver Maestro

Inspiration

A human learns how to draw with simple shapes and sketching. At first, we just try to copy it by following the image pixel by pixel and we don’t need a demonstration or a hard-coded drawing steps to achieve this. However, for robots, it’s not the case and we would like to democratize art by enabling self-learning for robots.

What it does

Sliver Maestro is a simulated artistic robot and its expertise is doodling! Sliver Maestro applies sketching by just one look at an image and experiencing how a human would draw it.

How we built it

We used DeepMind’s Deep Recurrent Attentive Writer model and Quick, Draw!'s simplified binary dataset to generate sequential images and extract stroke movements. Draw network is a recurrent auto encoder that uses attention mechanisms. Attention mechanism focus on a small part of the input data in each time step, and iteratively generates image that is closer to the original image. The network is trained with stochastic gradient descent and the objective function is variational upper bound on the log likelihood of the data. The advantage of DRAW to other image generation approaches is that the model generates the entire scene by reconstructing images step by step where parts of a scene are created independently from others, and approximate sketches are successively refined. The Quick, Draw! dataset is a collection of 50 million drawings across 345 categories, contributed by players of the game Quick, Draw!. Files are simplified 28x28 grayscale bitmaps in numpy format. Our experimental results of DRAW model on Quick Draw! dataset can be accessed from here.

We simulated drawings by using both PyGame and a robot simulation environment, CoppeliaSim simulator with Baxter robot model and Remote API for Python. In the post-processing, we first convert outputs into binary images and then into svg files. We use an svg parser to convert into coordinates and Sliver Maestro draws the generated images with successive refinements provided by the model.

As shown in the animation in the image gallery, the advantage of DRAW to other image generation approaches is that the model generates the entire scene by reconstructing images step by step where parts of a scene are created independently from others, and approximate sketches are successively refined.

An illustration of DRAW's refinements which is obtained by subtracting consecutive images is also given in the gallery.

Challenges we ran into

Reconstructing images which are independently created in a way that the robotic arm can follow the lines and draw the image smoothly.
Humans tend to draw with a combination of multiple strokes even if it is a simple doodling. So, we had to translate images generated by multiple strokes into a trajectory that is achievable by only one stroke.

Accomplishments that we're proud of

We are proud of successfully applying our skills into different platforms and their integration. It was a good practice for us to integrate a computer vision project with a robotic simulation. We are also proud of Baxter for being that much easy going and patient in our simulation trials.