Doodle with AI

Inspiration

Learning made fun and interactive.

What it does

Autocompletes vector graphics from strokes and completes a doodle. Once you start doodling, the model comes up with all possible ways to finish it.

How we built it

Backend - (Python, node.js, OpenCV, RNN)

1. The backend captures frames from the default webcam and applies color thresholding to detect a pointer.
2. It then uses contour detection to identify and draw a circle around the most prominent contour.
3. The center of the contour is used to track the movement of the pointer, and the detected points are stored in a deque. These are then sent to the front end to plot.
4. The program terminates after several consecutive frames without detecting a pointer. Frontend - (React, p5.js, ml5.js , Tensorflow.js)
5. The code includes functionality to clear the canvas and draw lines of different colors.
6. Interactive user UI to help with remaining strokes to complete a doodle. ## Challenges we ran into
7. Sending continuous streams of data from python to node.js
8. Integrating p5.js, ml5.js and react.

Accomplishments that we're proud of

1. Interactive UI.
2. Differentiating a particular color's HSL values to capture movement/Strokes.

What we learned

1. Development life cycle
2. Integrating various components of frontend and backend
3. RNN

What's next for Doodle with AI

1. Coloring doodles.
2. Image to Image Translation using stable diffusion

Built With

• ml5
• node.js
• opencv
• ps5
• python
• react
• rnn
• tensorflow