Social storyboard for adults with special needs

People with special needs can struggle with new social situations. A social storyboard helps them know what to expect and reduces their anxiety when experiencing the situation in real life.

Comment

Inspiration

A social story is a visual representation a real world activity and is used by people with special needs to anticipate what is going to happen, what they might see and do.

I was inspired to build this by my wife who works with non-verbal autistic children that can get very anxious and overwhelmed with new situations. Preparing a social story manually takes time, increasing her already stretched workload. Having a GenAI powered solution means she has more time to spend working with the children directly.

You can read more about the background science to social stories here:

https://www.autism.org.uk/advice-and-guidance/topics/communication/communication-tools/social-stories-and-comic-strip-coversations

What it does

The tool takes a short description of the target situation and activity. From this it identifies the 6 key steps that might typically be experienced along the way. For each of these steps, then then generates a simple cartoon image of that action/situation that the carer can use to help the person prepare for.

How we built it

I remixed the RPG app shared in the Hackathon instructions. This was a good starting point that helped me understand the linkage between chat input and generating a prompt to then feed into an image generation widget. With this base, I edited the prompts and expanded the widgets so that it would generate the 6 steps in one go making it faster for the carer to use.

Challenges we ran into

The original intention was to have the tool used for children but the image generation widget >75% refused to generate an image as it believed the input prompt had words on the 'filter' list. After many iterations I was unable to get around this, so I came to the conclusion there is a block on generating images with children in (for understandable reasons). To get around this I altered the app to generate the social story from the perspective of an adult and after this the images generated without any problem.

Accomplishments that we're proud of

I showed the app to my wife and on her first use she was able to produce a social storyboard that she was able to use with one of her children.

What we learned

I was already familiar with PartyRock but my biggest learning in this project was using a text generation widget as a prompt to an image generation widget. Previously I had been trying to write the prompt myself.

What's next for Social storyboard for adults with special needs

Test test test! My wife is going to use it for the next few weeks and I'll continue to iterate upon it to make it more effective (e.g. the images aren't always reflecting what's in the prompt, so this needs to be fine tuned). Once improved, I then plan to contact some Autism charities to get it more widely shared and hopefully some of those users might even remix my PartyRock app as a starting point!

Built With

  • partyrock
Share this project:

Updates