Pyro Machitis

architecture

Inspiration

In emergency situations like fires, every second counts, and clear communication is crucial. We were inspired by the potential of Boston Dynamics' Spot to assist in rescue operations by bridging language barriers and navigating hazardous environments, made possible with flame-retardant polyurethane foam used to make Spot. The idea of a multilingual robotic assistant that could locate victims and aid firefighters motivated us to create Pyro Machitis.

What We Learned

Developing Pyro Machitis taught us the importance of interdisciplinary collaboration. We gained hands-on experience with robotics integration, real-time language processing, and user interface design. We also learned how to use multiple APIs together to make a functioning application.

How We Built the Project

Hardware Integration: Enhanced the Boston Dynamics Spot robot with high-resolution cameras to detect victims in a fire.
Software Development: Utilized the Bosdyn API to program movement software controlled remotely and relay the video and audio to the controller.

Multilingual Communication: Leveraged Groq's AI accelerators and Google Cloud Platform to enable real-time translation and communication in over 15 languages.

User Interface: Created an intuitive UI using HTML, CSS, and Flask for operators to monitor and control the robot effectively.

Backend Systems: Integrated systems using Bosdyn for robot control, Groq for AI processing, and GCP for data management.

Challenges We Faced

New Technology: Working with Boston Dynamics was a first for all of us, and it required us to read and understand complex documentation and conduct extensive testing to make it work as intended. Accessing Spot's hardware had complex requirements that we had to overcome.

Integration Complexity: Combining hardware and software from multiple platforms required extensive testing and problem-solving, especially since the hardware can function remotely without being physically connected to the front end.

Built With

css
fetch
flask
gcp
groq
html
javascript
python
transcribe

Submitted to

Cal Hacks 11.0

Created by

I worked on the backend gRPC service to get video streaming in the frontend. Moreover, I worked with Groq and Google Cloud services to host the gRPC service and use their text-to-speech API which was then sent back to the robot for conversing with the user

Ansh Dholakia
I worked on the Boston Dynamics SDK in Python enabling movement from our UI to Spot using Fetch API.

Sushant Gupta
I contributed to the development of the Flask backend and REST API, enabling integration between the robot's controls and the frontend interface for remote operation.

Collaborating with the team, we utilized the Boston Dynamics SDK to create custom RobotControl functions. These functions enabled us to receive audio from the robot's microphone, send audio commands to its speaker, and control its movement all through Python based implementations tailored to our specific use case.

Krishan Kanji
Jason Wan