We wanted to augment the experience of storytelling by providing visual and audio aid to help engage and immerse the users of our project. We recognise that it can be harder for some children to enjoy the experience of reading, but we hope to make these challenges lesser.
What it does
As someone narrates a story, our program converts speech to text, then analyses the words to generate relevant sound and image to augment the experience.
How we built it
We built the project using a basic JS and HTML front end which would use a users microphone along with a speech to text API to send text to the backend wrote in Flask. The backend would send the text to a set of python scripts to process the text, create the images, and choose any appropriate audio files. The backend would send this back to the front end to be displayed in real time for the user.
Challenges we ran into
Accomplishments that we're proud of
-The project worked as intended!
What we learned
-Readers are leaders! - Shanai (5am) -Using flask -Using stable diffusion
What's next for Inkheart
-Spending more time on refining image generation