Inspiration
Search and Rescue involves time-critical operations and human risk. Robots have been used to reduce human risk, but their real-world training continues to be expensive and time-consuming. That's where SARAH comes in: Search And Rescue Automated Helper. Assisting the rangers in helping train models for SAR operations in a virtual environment.
Many AI applications focus on text-based work: summarizing documents, organizing tasks, or giving feedback. We wanted to explore something else: what happens when an LLM becomes the core multi-modal AI agent that perceives the environment and takes actions to interact with it? Particularly, we wanted to take into account what is called the “Bitter Lesson” in AI. That human-informed structure always ends up being worse in the long run than just raw knowledge and computation that AI has full control over. So while traditional SAR robots use human-informed knowledge like separating perception, routing, obstacle avoidance, and more, our agent is built with the principle of the bitter lesson at heart — minimize structure and let the compute do its thing.
What it does
Our project has two main parts: SARAH and the training ground. On top of cameras, thermal and LiDAR sensors help SARAH find the lost hiker. The training ground is a procedurally generated grove meant to contain many distraction points for SARAH such as false light sources, objects that block its path, and animals that throw off its thermal sensors. SARAH can also be directed by a ranger remotely to speed up the SAR operation.
How we built it
- Three.js creates and renders SARAH’s 3D forest environment, including terrain, obstacles, and the robot’s visual field.
- Gemini powers SARAH’s decision-making. It interprets the robot’s observations and mission context to choose actions such as navigating around obstacles, investigating evidence, or backtracking from dead ends. Gemini is multimodal, meaning it can combine visual, spatial, and text-based inputs. We also used a single agent to minimize latency
- Azure deploys and hosts the web application and supports secure server-side communication with the AI services.
- GoDaddy provides and manages the project’s custom domain.
Challenges we ran into
- Rate limiting ceilings for Gemini API free tier → alleviated by cycling through multiple models in the same project
- Multiple models lose context window → let Gemini handle its own freeform memory with tool calls to a text file. (In the spirit of the Bitter Lesson)
- SARAH misjudging distances → rework LiDAR system to encode depth-info as the hue of a pixel in an image
Accomplishments that we're proud of
- UI design looking the way we want it to, with two different view setups which each have multiple options.
- Analyzing a large codebase and learning new technologies (Three.js, Azure, Gemini API, etc.)
- Having the robot successfully interact with its environment
- Overcoming a lot of obstacles that we didn’t think were solvable at first, like rate limiting
What we learned
- Deployment is harder than it seems! Many more things involved: server settings, IAM permissions, domain records.
- We experimented with new softwares like next.js and three.js .
- LLMs are are hard to code with even with an extensive planning document
What's next for SARAH and the Training Ground
- Tuning SARAH so that it turns more quickly, searches more efficiently
- Local LLM support for the Training Ground
- More customization options in the Training Ground including cosmetic changes (better graphics options)
- Physics engine for the Training Ground so that physical robots can be simulated in the Training Ground, beyond just collisions
- Saveable and shareable simulations via databases
- having the LLM map its successful route for unseen terrain and finding the most optimal path
Log in or sign up for Devpost to join the conversation.