Lost and Found is a common use case, yet it is so hard to recover a lost item. With the technology that is available today, we have built a social app that lets consumers use a familiar interface to recover lost items by leveraging natural language and a deep learning chatbot. We address privacy by brokering all conversations between item owner and finder. With 2 billion users that use the facebook platform, we can locate a lost item anywhere across the globe!
What it does
lofoBot is powered by AWS Lex and Rekognition. A user who finds an item chats with lofoBot to submit details of the item. Rekognition is used to detect labels in the item image so that it can be tagged with these additional labels, thus increasing the likelihood of match when the item owner searches it by a different name. For e.g., an item reported as a bag by the finder will also be tagged as backpack. When the owner searches for this item using a phrase - I lost my backpack, we can locate it. Subsequently, item owner provides identification information to claim the item which need to be verified by item finder. Once item finder is satisfied with the information provided, claim instructions are sent to the owner to complete the fulfillment process.
You can also play the screencast video with speakers turned on to get more information on lofoBot.
How we built it
We have implemented the following intents:
This intent is provided so that user can naturally learn how to start the dialog with chatbot by typing hello.
This intent handles the dialog when a user says - I found an item.
This intent handles the dialog when a user says - I lost an item.
This intent handles the verification of identification details provided by item owner.
This intent been implemented as per the recommended guidelines by AWS so that slot values can be gracefully emptied.
Note: These are just a few of the bigger set of utterances that have been used to train Lex chatbot.
We use Claudia Bot to provide a rich user interface as it natively supports all the facebook messenger templates. We use generic template to give user a mechanism to select the item and also use quick reply mechanism for answering the prompts. Claudia Bot is actually a proxy that invokes LEX using the REST API. This allows for cleaner separation of concerns where we keep presentation aspect isolated from the natural language processing of user utterances.
We use Google Maps API to geocode the address provided by user so that we can search for the item in a predetermined radius rather than the street name. This api is very powerful, has a global reach and can also geocode an intersection.
We use Facebook push notification mechanism to send asynchronous messages to users. For e.g., the item finder may not be online when the claimant locates an item and provides item identification information. This message is available to the finder just like any other message when finder logs onto facebook.
Images are stored in S3 and item information is persisted in DynamoDB.
Challenges we ran into
Lex Facebook connector that is available out of the box is very limited in functionality and has minimal support for generic template. We also want the users to upload images from their smartphone, a feature that Lex Facebook Connector lacks. As such, we had to rethink our architecture and use Claudia as a proxy to Lex.
Accomplishments that we're proud of
It is very fulfilling to see lofoBot as a consumer to consumer app. We got hands on exposure to working with Lex that is a rewarding experience in itself. Our presentation accounts for the technical details as well as the value that we plan to provide to our users.
What we learned
We got familiar with Lex, Rekognition, DynamoDB as well as Facebook Messenger Platform. We learnt what we did not know before by collaborating with like minded contestants as well as mentors from AWS who were prompt in getting us the answers we sought. This is our first screencast, we also learnt how to use Mac QuickTime player and mix audio into our presentation.
This is not a public facebook application yet. If you are interested in beta testing it, you will need to provide your facebook id so that we can assign you the tester role.
What's next for lofoBot - The Lost and Found ChatBot !
- Influence AWS Lex team to add more features to Facebook Connector
- Launch a pilot to test the waters and eventually make it the go to service for lost and found items
- Make a presentation at AWS re:Invent 2018