Inspiration
What it does
The application is divided into two parts.
- Image analysis using tactile interface
- Gesture controlled directional image analysis
Workflow
- A user can interact with iBuddy using three buttons, Red, Green, and Yellow.
- The screen preview camera view and also shows the output of the analysis##### Use cases of buttons
- Yellow Button: Click a picture of the camera view
- Red Button: Answer No
- Green Button: Answer Yes
Tactile Interface for human face analysis
- User clicks a picture of image with a human face, using Yellow Button.
- The image is being sent to Azure Face API and we get the overall analysis in the returned json.
- User can listen to overall description of the image, and also see on the screen.
- iBuddy will then ask the user if he/she wants to know the emotion detected in the image.
- User can answer Yes/No using Green & Red Button.
- iBuddy will prompt the response accordingly.
Description: A man looking at camera
Emotion: disgusted
Text:
----------------------
Tactile Interface for OCR
- User clicks a picture of image with using Yellow Button.
- The image is being sent to Azure Face API and we get the overall analysis in the returned json.
- User can listen to overall description of the image, and also see on the screen.
- iBuddy will then ask the user if he/she wants to know the text recognised in the image.
- User can answer Yes/No using Green & Red Button.
- iBuddy will prompt the response accordingly.
Description: white board with black text
Text: NOTICE NO EATING OR DRINKING IN THIS AREA
----------------------
Gesture Recognition
- The user can point to any of the four parts of the image and make a small circle with a finger pointed in that direction.
- Leap detects the gesture and iBuddy crops the image in that part
- That part of the image is sent for analysis and appropriate results are returned.
Gesture Recognition
How we built it
Technologies used
- Python
- Leap Motion
- Raspberry-pi
- Azure
- Natural Language Processing
- Computer Vision
- Ngrok
- Socket programming
- Python text-to-speech
Challenges we ran into
- Setting up R-pi
- Integrating leap motion
Accomplishments that we are proud of
- Working product
- Successful gesture controlled directional analysis
- Low latency
- Good Accuracy
What's next for iBuddy
- 360-degree image analysis
- Live video stream analysis
- Better physical device and integration
Built With
- azure
- computer-vision
- espeak
- leap-motion
- natural-language-processing
- python
- raspberry-pi
Log in or sign up for Devpost to join the conversation.