GrabThisSpot

The bossy owner exploiting his dog
Petting my cuttie doggie

Inspiration

🌟 Our inspiration for "GrabThisSpot" came from a desire to bridge the gap between humans and robots. We envisioned a world where controlling robots wasn't just a technical feat, but a magical experience. Imagine feeling like a wizard, commanding a robot with a simple wave or a flick of the wrist. Our love for innovation and a bit of sci-fi imagination fueled our passion to create this project.

What it does

🤖 "GrabThisSpot" is your portal to the future of robotics. It allows you to control a Boston Dynamics SPOT robot through intuitive gestures. Raise your hand to make it move forward, lower it to go backward, swing it left or right to steer – it's like controlling the robot with a Jedi's finesse. With AI and computer vision, it recognizes your face and your unique gestures, ensuring it responds to you, and only you. Plus, it calculates the distance between you and the robot for safety.

How we built it

🔨 We combined cutting-edge hardware, software, and a dash of creativity. We integrated the SPOT robot, programmed the gesture recognition system, and fine-tuned the AI to understand the user's intent. Our team worked tirelessly to ensure a seamless user experience, where technology feels like magic.

Challenges we ran into

🌦️ We faced challenges like a dog chasing its tail. Fine-tuning the gesture recognition system to ensure accuracy and real-time responsiveness was a tough bone to chew. Dealing with different lighting conditions and user variability added to the complexity. But our determination and teamwork helped us overcome these hurdles.

Accomplishments that we're proud of

🏆 We're ecstatic to have turned our vision into reality. Creating a project that allows anyone to effortlessly control a robot with gestures is a big win for us. We're proud of the seamless integration of hardware and software, as well as our dedication to user safety by implementing distance detection.

What we learned

🧠 This journey taught us the power of innovation, teamwork, and the joy of creating something that wows people. We deepened our understanding of computer vision, AI, and the immense potential of human-robot interaction.

What's next for GrabThisSpot

🚀 The journey doesn't end here. We're eager to refine our project further and explore new applications for gesture-controlled robotics. Our vision is to make technology even more accessible, interactive, and fun. Watch out for more magic with "GrabThisSpot" in the future!

Built With

boston-dynamics-sdk
css3
flask
html5
javascript
opencv
pipemedia
python

Submitted to

Cal Hacks 10.0

Created by

I mostly worked on the computer vision and AI experimentation part of this project. I implemented the AI pipeline to detect various hand gestures and body movements to interact with the robot. The pipeline takes the video stream as the input, detects body posture key points, calculates various angles and comes up with the appropriate action to take which is then converted to a signal that the robot can understand and act upon.

Yash khandelwal
My contributed was mainly focused on building the frontend for the WebAPP, and the huge pipeline that triggers the robot responses based on the 'wasd' characters inputs.

Then my mate Yash worked on triggering this wasd characters though body gestures and we merged everything.

I also allowed voice control by recording audio stream from the frontend using javascript, sending to backend and running Whisper for getting the transcript, and then LLMs to get the mapping from the text to the 'wasd' commands for moving forward, left, backwards and right respectively.

Julen Ferro

Updates

Julen Ferro started this project — Oct 29, 2023 01:56 PM EDT

Leave feedback in the comments!

Log in or sign up for Devpost to join the conversation.