ADAM - Autonomous Disaster Assistance Manager

A humanoid robot and coordination dashboard for post-disaster triage. adamrobotics.xyz - ADAM Robotics official website

Comment

Inspiration

The first hour after traumatic injury is known as “Golden Hour”, because the sixty-minutes after an injury are often the difference between life or death. Fast intervention not only improves survival rates, but also reduces risk of preventable complications like disability, infection, or limb loss.

However, when mass disasters like earthquakes occur, these environments are often too dangerous or unstable for first responders, and manpower is at a premium regardless. This problem is close to our hearts, as one of us had a loved one pass away in the Szechuan Earthquake.

This led us to our core question: How can we bridge the gap between a trapped victim and a medical professional without risking more human lives?

Our answer is ADAM.

What it does

With the progress of Computer Vision and VLMs, we knew we wanted to do more than just a monitoring system. We wanted an agent capable of traversing dangerous terrain, locating human survivors, and relaying information back to a remote responder. Essentially, a robot first-responder that could locate and diagnose the injuries of victims in the aftermath of a disaster.

We built ADAM. It has two parts.

First, the robot. Using vision models, ADAM can detect injured humans across varied terrain and low-visibility conditions. Through an audio classification pipeline, it can recognize distress signals like calls for “HELP!” and moves toward the person. ADAM records visual and audio. Most importantly, as a proof-of-concept, ADAM can find and clear debris off victims trapped under objects.

Second, the command interface. ADAM is meant to be the eyes and ears of responders, who we imagine piloting our robots to safely search for victims. Drawing from real-world emergency response dashboards, we built an interface for responders, complete with a live camera feed and a text transcription of the victim speaking. The interface uses LLMs to assess victim injury and create a preliminary medical report for human first-responders.

How we built it

We built ADAM as an end-to-end system spanning robotics control, perception, AI reasoning, and a real-time command interface. There were five main parts of this build, which we progressed through more-or-less in in the following order.

Robot Movement We deployed ADAM on a K1 Robot, with humanoid hardware from Humans in Motion, using the Booster Robotics SDK to code the robot’s actuators and sensors. Low-level joint control and keyframe sequences allowed us to manipulate actions and ensure walking.

Specifically, we use inverse kinematics to coordinate arm movements for picking up a piece of debris off a human victim.

Perception and Human Detection We implemented a vision pipeline using YOLOv8 and YOLO-World for real-time human detection and scene understanding across a variety of conditions. These models allow the robot to locate victims, track objects, and interpret its surroundings.

For more ambiguous situations, we incorporated a multimodal reasoning model to provide additional context beyond raw detections.

Medical Assessment After capturing images and interacting with the victim, we used a frontier multimodal model to generate a preliminary assessment of injuries and severity signals, structured into a draft report to support triage. Command Interface and Infrastructure

The operator dashboard is deployed on Vercel for low-latency updates. It streams video, audio, and telemetry while supporting speech transcription, report generation, and operator messaging. This setup maintains a tight feedback loop between the robot and responders monitoring remotely.

Challenges we ran into

The Booster Robotics K1 robot has 22 joints. Controlling these joints was very difficult, even with the help of key-frames, especially the transitions between modes. Automating these transitions and movements presented two main difficulties. First, transitions often created stiffness and balance issues which would cause the robot to alter course. Second, the physical movements were very difficult to get consistent, for instance.

But it also opened our eyes to how there will be tremendous improvements in robotics hardware and software, and we want to sit in the middle as an orchestration layer that organizes these robots in an immediately useful way.

We also learned to not open instagram when hacking as a single person on valentines day.

Accomplishments that we're proud of

Throughout the hackathon, we ran into multiple challenges with programming the Booster K1 robot. Initially, we struggled to get the keyframes to load and play on Adam because of the limitations around the upper body versus the lower body. At the end of Friday night, we were able to get Adam to successfully keyframe and throw a punch which was a big win for us.

The next day on Saturday, we spent the whole morning attempting to get Adam to squat because we wanted him to be able to bend down and pick up rubble. However, we had to pivot once we realized that the lower body was incapable of balancing by itself with the current Booster robotics SDK due to the kCustom mode forcing a “switch mode” state where the entire robot would tense up and “plank” sending it off balance. We then created a custom punch capture sequence that allowed us to save a “debris” capture. That was a huge win for our team when Adam was able to pick up a piece of rubble.

Additional milestones included when we got the audio working, and vision capture features.

What we learned

Hardware is hard, so adapt. There are just so many extra considerations in the world of atoms vs bits. Different lighting interfering with the CV, changing conditions on the ground causing traction issues, a noisy space making it harder to segment the specific audio of the person looking for help. These are so many things in the environment you can’t count on, and that's why adaptability is the only way through.

Edge compute isn’t free A disaster zone is not going to have perfectly reliable wifi, and so we put all the inference we could on the robot processor itself. But unlike the magical resource of “compute,” flops on device are scarce, and we had to aggressively control the amount of processing on device. We ended up being able to run almost everything on device except the medical processing.

Help the robot out with kinematics If we asked the LLM to directly control all joint parameters, the robot would probably take an hour to walk a few steps and would certainly make a wrong movement somewhere on the kinematic chain. To rectify this, we used standard SDK kinematics for basic movement, and created our own action presets the AI agent could choose from.

What's next for ADAM - Autonomous Disaster Assistance Manager

We strongly believe in the potential of this concept, as unlike most other AI era projects, this platform only becomes more powerful as robotics hardware and software advances.

Software improvements, primarily moving from vision models to world models. This alone would make our clock speed much faster, and just overall contribute to much higher adroitness in the model’s locomotion and manipulation.

Hardware improvements, more powerful robots, perhaps robots with non-humanoid form factors. This is something that we could do with vision models, but it would require lots of individual tuning. Adopting a world model would allow us to rapidly introduce new robot models without compromising performance consistency.

Control plane enhancement; two things here. First, teleoperation. It would be incredibly helpful if a dashboard viewer could be prompted about a robot being underconfident in its actions, and then be given the opportunity to take over control for a particularly difficult section. Secondly, swarm behavior. Right now our current system treats the drones solely as individual units, when there are some emergent behaviors that could be explored, like group box searches of an area or a combined lift on an object too heavy for one robot.

Security

  • End to end encryption
  • SOC II and HIPAA compliance
  • Endpoint security management and multi-tiered permission to access the dashboard

Built With

+ 5 more
Share this project:

Updates