StoryScribble

App Icon
App Loading Doc

Inspiration

According to a census from the National Literacy Institute, 40% of students across the nation struggle to read at a basic level and almost 70% of low-income fourth graders are in the same situation. Reading is one of the hardest subjects to learn from scratch, and according to Sliwerski from Ignite Reading, it’s because the human brain hasn’t evolved to make reading as instinctive as speaking. That’s where creativity comes in! Research shows that art is often the primary form of expression for students still learning to read. By blending creative expression with literacy development, we can give children the opportunity to improve their reading skills outside the classroom. Using art to engage them helps make learning more relatable and fun. So we decided to create a game that encourages all of these areas!

What it does

The workflow of this app begins with three user inputs: name, age, and picture(s). Users can either upload drawings they have already drawn or use the built-in canvas to draw their own images. From there, the saved drawings are spliced together into a comic format and the user’s age is used for the model prompt to provide a story with appropriate vocabulary. Both the spliced image and the engineered prompt are fed to the Image-Text-to-Text model which then creates a cohesive story that follows each drawing as a new development. As our stories are for children, we limit the model to one sentence per drawing and adjust the vocabulary of that story based on age. The children can then read their story and save it to their library which lets them access all of their stories at any time.

How we built it

Our tech stack consisted of using Flask with Python as our web framework, React.js with Javascript for frontend design, SQLite as our database, and Figma for our UI/UX mockup. Additionally, to generate stories for users based on their drawings, we used an Image-Text-to-Text model that uses a prompt as context for the images and builds an appropriate story for each page with vocabulary based on the user’s age.

Challenges we ran into

First, we needed an AI model that could interpret children's images. We initially turned to Hugging Face for free models but faced significant challenges in generating coherent stories. Many models required GPU access, which we did not have. Once we found the right model, we encountered another issue: it struggled with understanding the chronological order and context of separate images. We resolved this by developing an image splicer that arranged the images sequentially, separated by a large black bar. In terms of frontend design, we aimed to incorporate more animations; however, due to time constraints, we couldn’t fully realize this in our final product. We plan to focus on this aspect in the future.

Accomplishments that we're proud of

We are proud of our frontend development, particularly in translating our concept into a Figma mockup. Our goals and ideas are clearly organized and presented through these visuals. By combining hand-drawn elements with graphic design skills, we created a well-structured prototype that effectively caters to the childhood stage of life. In terms of backend, we are proud of how meticulously we worded the prompt so it would accept any image no matter how vague or specific the drawing was and produce a story out of it. It took a lot of trial and error testing to ensure that it would output what we wanted it too.

What we learned

We learned the syntax on how to use open ai models as this was our first time implementing an open ai model in code. Additionally when creating the library we realized that we would need a decently complicated database for the library that couldn’t be solved by just a list. This is because the library has stories and stories and every page has a picture and a description. So in order to store this data effectively we utilized SQLite to hold the data.

What's next for StoryScribble

Our primary objective is to advance the development of the frontend and successfully deploy our application. As a motivated team, we have found great satisfaction in addressing this challenge creatively, and we believe this app has the potential to make a meaningful impact on children's learning experiences. In addition to deployment, we have outlined several key goals for the app's enhancement: Achievement System: Implement an achievement framework that promotes user engagement and motivates children to reach their reading goals. Reading Goal Development: Introduce features that facilitate the setting and tracking of individual reading goals to encourage continuous growth and gameplay. Text-to-Speech Integration: Incorporate text-to-speech functionality to assist young users who may benefit from auditory support, enhancing their comprehension of the text.