We are Toddler Numbers, we've created a counting game to help pre-school kids learn how to count. The game can be played on chat or through Alexa devices. It's interactive and expandable so kids don't get bored. Just type "I want to play the counting game".
My daughter is at the age where I feel that she can start learning to count. She is really fascinated by Alexa and loves to interact with it. Using this hackathon as an opportunity, we came up with the idea to create a counting game to help kids learn using technology.
What it does
It helps kids to learn how to count interactively by showing them images and asking them to answer simple counting questions.
How we built it
We used Amazon Lex to create a chatbot which connects to Lambda. Our Lambda function pulls images from S3 which are annotated with counts using user-defined meta data. The images then get run through Rekognition and the function formulates questions that get asked to the user via the chatbot.
Bonus: We exported this chatbot to create an Alexa skill which allows you to play the game on any Alexa device.
Challenges we ran into
Getting the Lambda function to be able to read from an S3 bucket required specific roles/policies to be activated. Similarly, calling Rekognition from our Lambda function needed additional permissions as well. While the documentation of AWS services is vast, sometimes it was hard to find the exact settings we needed to make things work together.
Integration with Slack
When we created the Chatbot under Channels for Slack, we thought the integration was not working, but after some trial and error, we realized that an older version of the chatbot interaction model was integrated within Slack. So we learned that every time we updated the interaction model, we needed to publish a new version of the Lex bot and update the postback and OAuth urls on the Slack side.
We expected that the Alexa skill would be modelled in a similar way to the Lex chatbot. So we went about searching for similar JSON request/response samples to build the Alexa skill. While this worked and we got something going, we later learned that Alexa has a skill builder API that makes this process easier.
Learning Curve on AWS
This is the first time we are working with AWS services and just general development took sometime to get a hang of. For example, knowing that the logs show up in CloudWatch, and permissioning our Lambdas to correctly write logs in CloudWatch. Also, working with Lambdas in the AWS Console and then learning to optimize the feedback loop by creating a script to push up the Lambda code and working locally in our editors.
Accomplishments that we're proud of
It's quite easy for us to keep this game interesting. All we need to do is add more images in the S3 bucket and annotate the images with counts of things in the images. Once we have done this, the Bot will automatically pick up on these new questions.
We have used the different AWS services quite efficiently. We skipped having a database by using meta-data in S3 images. We generate public URLs for our S3 images so Rekognition can read them without needing any uploads. We also cache all the S3 images before starting the game so we can move quickly from one question to the next thus keeping the game moving.
What we learned
Dynamic Responding Chatbot
When we began, it wasn't absolutely clear to us how we would have the bot dynamically generate the questions based on image recognition. But by researching a bunch online and trial and error, we were able to come up with a solution that worked. We used the dialogAction to customize how we asked the user for slot information.
Exporting to an Alexa Skill
Once we had the Lex bot working, we wanted to have another friendly way to interact with our bot. So we decided to create an Alexa skill. The documentation makes it seem really easy to export a Lex bot to an Alexa skill but we soon realized that this was easy for only part of the process. The export/import feature allowed us to port the interaction model for the bot i.e. the intents, the slots, and prompts, but the request/response format in which Alexa and Lex operate is quite different.
What's next for Toddler Numbers
We want to expand this game to have more themes. Currently, we have only collected pictures of fruits, but we would like to have animals and vegetables as part of the image sets as well. We can then add options to play in a random format or by categories.
We would like to expand the reach of this game to speakers of other languages. We thought we could use other AWS services like Amazon Translate to help us with this goal.
We realized that Slack requires images uploaded in the chat to be under a certain limit. This forced us to reduce the quality of one of our images (the blueberry one), which caused Rekognition to stop correctly recognizing the image as blueberries and rather recognizing them as accessories. We can fix this by getting another image of another fruit.