Detect Animals and Birds in Picture

Solution Architecture
Correctly identifying multiple giraffes in the picture
Twitter Interaction Screenshot
Twitter Response Screenshot

Inspiration

Artificial Intelligence is influencing every part of our lives. It has great potential to improve lot of things in infinite ways but AI must be used with extreme caution. Even slightest carelessness can lead to dangerous consequences.

I have always been interested in learning about risks and challenges of taking a ML model to a production environment. I have created an AI augmented SaaS solution for detecting animals and birds in the picture tweeted by a user.

What it does

A user can tweet a picture with #DetectAnimalInPic hash tag. The solution will detect the tweet in real time, use machine learning models to infer the contents of the picture and send the results in a tweet back to the user.

How I built it

The following are the major components :

Tweet Streaming service
AWS DynamoDB based database to store meta data of the relevant tweets
If uploaded image is safe, AWS Lambda function to orchestrate the inference from two different sources, ensemble the results and send it back to the user via teet.
YOLOv3 object detection detection ML model from AWS Marketplace
AWS Rekognition service for detecting unsafe images and for detect objects in the images.

If image contents are not safe, no inference is done. User is informed via tweet that image had inappropriate contents.

Challenges I ran into

The biggest challenge was user safety and trying to figure out how to avoid misuse of the service. It led me to add check for images having unsafe contents.

Intelligent ensembling of results from multiple inference sources is a unique challenge. For instance, if Rekognition service detects 8 elephants in the picture but YOLO model detects 7 elephants, how would you merge the results? There is no correct answer, but it is an interesting problem to explore.

The solution heavily depends on reliability of the Twitter platform. There were few instances, where I was unable to send a tweet with picture. The issue resolved automatically but as an end user, I had no course for remediation.

Accomplishments that I'm proud of

I was able to wireframe the solution in relatively short time frame days. I am particularly proud of the fact that I was able to use AWS Polly to narrate my demo video. I am very impressed with end results.

What I learned

AWS has really lowered the barrier for any business to use AI models to improve their products or services. You don't even have to know anything about AI, yet you can improve your solution dramatically by incorporating AI components.

The solution is organically scalable and robust by design.

Where to next ?

The solution need to be more fault tolerant. For example, the response is unpredictable, if a video or gif file is uploaded.
The solution need to do a better job of handling hiccups on Twitter platforms.
The solution can also be transformed into a chat bot based or messaging platform based solution.
The solution can currently detect 10 different categories. It can be expanded to detect more species among animal, birds and sea mamals category. A large amount of good quality data is the only realistic challenge to the potential of this or any ML based idea.