DoodleTales

What inspired us to create DoodleTales?

In our current time, technology makes appearances in our everyday lives however, there are (and still) many concerns about the use of technology with very young children. One of the most common worries people have brought up for example: Can children use technology while still being active behind the screen? With DoodleTales, we aim to achieve this, by making children draw fun and creative drawings or choose to create fun stories with engaging features.

What does DoodleTales do?

DoodleTales uses your device webcam to scan a drawing of a character, animal, scene, etc., or pictures that you take and uses that information to create stories based on what you drew. Scanning your first drawing will start a story, which is also accompanied by text-to-speech narration. You may add on to the story by scanning more drawings/pictures or choose between 2 generated prompts for what happens next, and once you decide to finish your story, you can save it as a PDF file along with the scanned images that were used to generate it. You also have the ability to start a new story anytime.

How we built it

We built the main structure and UI using HTML5 and TailWind CSS, the HTML5 and JavaScript API was utilized to stream the webcam and capture the image, Gemini was used for the scanning and analyzing of images, and generation of stories, and implemented the browser-native speech API to allow the narrative features.

Challenges we ran into

Some challenges we ran into while working on DoodleTales were a Gemini model expiring, fixing the UI to make it look nice and interactive for kids, making DoodleTales run fast and smoothly, and giving it the ability to continue stories smoothly when new images are added onto the same story.

What we learned

We learned how powerful Gemini could be when it used as an I/O engine instead of the usual chatbot. We also learned how to utilized HTML5's and JavaScript API to allow webcam streaming and capturing of images.