The Secret Life of Animals

Classify animal behaviour from video data in real time or upload videos to find out how the animals you are interested in spend their days without having to spend hours staring at a screen!

Comment

Inspiration

Video of behaving animals contains a wide breadth of detailed information about the behaviour of the animals. Many hours are spent by zoos, researchers and the public alike looking and analysing videos and feeds of animals trying to classify what behaviours they are performing. With current ML and CV advances, we feel this is a waste of everyone's time so wanted to put an end to this. So far, computer-vision techniques have only been used in ethology tracking animal trajectories and counting animals, but no solution has been proposed for classifying behaviours of animals, which is key information for animal welfare, activity timings in zoos and the like as well as broader biological studies.

What it does

  1. Takes in video data from user (zoos monitoring their animals, visitors filming wildlife/animal in captivity, scientists studying animal behaviour...)
  2. Detects animal(s) in video
  3. Classifies what behaviour the animal is performing
  4. Computes the percentage and total amount of time the animal has spent performing each type of behaviour since the start of the feed/video

How we built it

  • Azure Microsoft Bing Image Search
  • Microsoft's Object Detection
  • Action Recognition Neural Network

Challenges we ran into

  1. Training dataset:

    • Could not find readily available datasets with labeled video
    • Opted for using online image search results instead, but searches often returned non-animal related image results as well as incorrect labels

  2. Behaviour labelling:

    • eating behaviour was particularly hard to label correctly - model had to be retrained several times
    • frames where animal is transitioning between behaviours hard to classify

  3. Website implementation

    • Handling uploading of big files

Accomplishments that we are proud of

  • Achieved good detection accuracy at low computing costs for placing tight bounding boxes around the animals (tigers used for initial training).
  • >80% accuracy achieved for labelling behaviours into 7 categories (resting, walking, running, eating, grooming, sleeping, swimming)

What we learned

  • A lot of non-animal related results come up when looking for tigers!
  • Behaviours are often hard to classify even by humans

What's next for The Secret Life of Animals

  1. Increasing range of animals and behaviours classified
  2. Open for use by researchers. Use any further label corrections they make to improve prediction accuracy of model.
  3. Adding conditional connections to other devices based on behaviour classification (based on initial user feedback)
    • eg when animal is sleeping, alert zookeeper to clean enclosure
    • if animal has spent >40% of its day eating, reduce amount of food allocated to animal by 20%
  4. Monetising
    • Partner/sell to zoos and other stakeholders to monetise
    • Continue to add further features and connections

Built With

  • bingimagesearchapi
  • microsoftcognitiveservices
  • python
  • tensorflow
Share this project:

Updates