Inspiration

When firefighters enter a burning building, they don't get to choose how much of the room they can see. Smoke can obscure furniture, doors, and walls, and they may have to navigate an unfamiliar layout while searching for people.

We started EchoSense with a question: could we build a useful map of a room without relying on a camera?

Our first idea was to build an array of ultrasonic sensors. But when we couldn't get the hardware we needed during the hackathon, we had to rethink the approach. We realized that an iPhone Pro already had a LiDAR sensor capable of collecting depth measurements, so we pivoted to using that as our primary source of spatial information.

That solved one problem, but introduced another. LiDAR doesn't detect every surface reliably. Glass can be difficult to measure, and areas outside the sensor's field of view can leave gaps in the map. We decided to keep an ultrasonic sensor in the system as a second way to measure nearby obstacles, particularly where LiDAR might struggle.

As we developed the project, we became interested in something beyond reconstructing the room itself: how do we show what the system doesn't know?

A map can look complete even when an entire corner has never been scanned. For someone using that map to understand an unfamiliar space, the distinction matters. We wanted EchoSense to preserve what it had measured, identify what remained unobserved, and avoid presenting missing information as empty space.

What it does

EchoSense is an experimental indoor mapping system designed with low-visibility search-and-rescue operations in mind. It combines iPhone LiDAR, ultrasonic ranging, and machine-learning-based room analysis to build an interactive picture of an indoor environment.

An iPhone Pro streams LiDAR depth measurements and motion data to a laptop. Instead of using camera images or camera-based tracking, the mapping backend estimates the phone's movement from depth and motion information, gradually building a 3D reconstruction as the operator scans the room.

The browser viewer turns that reconstruction into something easier to interpret. It identifies surfaces such as floors, walls, ceilings, and objects, and produces a floor-plan view with room measurements. It also distinguishes areas that have been observed from areas that haven't.

Unobserved doesn't mean clear. That distinction is central to the project.

Alongside the LiDAR system, we built a physical ultrasonic sensing setup using an HC-SR04 sensor. The mapper can compare its distance readings with LiDAR measurements in the corresponding sensing direction and flag possible discrepancies. This gives us a way to investigate obstacles that may not be represented accurately in the LiDAR map, including transparent surfaces.

We also built session recording, replay, map export, and historical-map preservation. A captured room doesn't disappear when the sensor disconnects, but the interface distinguishes old measurements from a live scan.

EchoSense is a research prototype, not an operational firefighting device. It is intended to explore how complementary sensors and clearer representations of uncertainty could improve spatial awareness.

How we built it

We divided EchoSense into several components that could be developed independently and connected through a common mapping pipeline.

iPhone depth capture

We built a Swift application that streams LiDAR depth frames and phone orientation data over a binary WebSocket connection.

We deliberately avoided using camera images for localization. Instead, the mapping pipeline works with depth and motion measurements, which meant implementing our own approach to tracking the phone as it moves through the environment.

Mapping backend

The Python mapper uses NumPy and SciPy to turn incoming depth frames into 3D points.

To estimate the phone's movement, we use motion-sensor rotation as an initial estimate and refine alignment using Iterative Closest Point (ICP). As new frames arrive, their points are incorporated into a voxelized room map with a 2 cm resolution.

The mapper also supports recording sessions and exporting point clouds as PLY files. Recording was especially useful during development because it let us replay sensor data without requiring the phone to be connected for every test.

Ultrasonic sensing

For the second sensing method, we used an HC-SR04 ultrasonic sensor connected to a Seeed XIAO nRF52840 Sense microcontroller.

We wrote Arduino C++ firmware to read distance measurements at approximately 15 Hz and transmit them over USB serial.

The Python pipeline can compare the sonar distance with the LiDAR measurements in the sensor's approximate field of view. Rather than simply averaging the two readings, we preserve possible disagreements so they can be inspected.

We also built a separate WebSocket bridge for sonar data, with recording and replay support. This made it possible to develop and test the sonar pipeline independently of the iPhone mapper.

Browser viewer and room analysis

We built the viewer using JavaScript and WebGL, without a frontend framework.

It displays the live 3D point cloud, supports floor-plan analysis, and identifies surfaces such as floors, walls, ceilings, and objects.

For classification, we integrated a trained random-forest model into a browser worker. Running inference in a worker helps keep the interface responsive while analysis is performed. We also retained a rule-based geometric classifier as a fallback.

The viewer stores saved maps using IndexedDB, allowing previous scans to be inspected without repeating the original capture.

We also implemented capture-state tracking. If the mapper remains connected but stops receiving fresh depth frames, the viewer can mark the map as historical rather than continuing to present it as live.

Databricks and data processing

We developed a data-processing workflow around recorded scan sessions, with the goal of making captured spatial data available for further analysis and model development.

Additional research

We also explored a separate simulation-based sonar pipeline involving 40 kHz ultrasonic sensing, DBSCAN segmentation, and random-forest and PyTorch classifiers.

This work was simulation-based and separate from the physical HC-SR04 demonstration. It gave us a way to investigate additional approaches to interpreting sonar measurements without presenting simulated results as real hardware performance.

How we used Databricks

Databricks is where every scan ends up. We wanted one place where the data lands, gets checked, trains the model and feeds the demo.

  • Scans in. The mapper exports each recorded session as Parquet: one row per depth frame with the raw depth bytes and the phone's position, plus the finished map and the session labels. The files upload to a Unity Catalog volume, and a file-arrival trigger starts the job.
  • Bronze, silver, gold. A Lakeflow Declarative Pipeline builds the tables. Bronze keeps the raw frames as the phone sent them. Silver adds integrity checks on every frame and keeps map points seen at least twice. Gold has a per-session summary, a quality-by-lighting table and a 5-second timeline.
  • Training. For the sonar research track, Spark runs the simulator in parallel, MLflow tracks each training run, and the model is registered in Unity Catalog under the alias @champion, which scoring reads.
  • Why it mattered. Raw depth stays in bronze, so we can recompute positions when the mapper improves. Expectations stop bad frames before they reach training. The quality-by-lighting table is how we will compare lights-on and lights-off scans.

Both pipelines are Databricks Asset Bundles in the repo and are deployed on Databricks Free Edition. The sonar pipeline ran end to end on 200 simulated scenes.

Challenges we ran into

Pivoting when the original hardware plan fell through

We initially wanted to build around an ultrasonic sensor array. When the necessary hardware wasn't available, we had to change direction in the middle of the hackathon.

Switching to iPhone LiDAR allowed us to build a much richer room map, but it also meant implementing an entirely different capture and mapping pipeline. We kept a single ultrasonic sensor because we still believed that combining sensing methods was more interesting than relying on LiDAR alone.

Tracking movement without camera-based localization

One of our biggest technical challenges was estimating the phone's position without using ARKit's camera-based tracking, meshing, or RoomPlan.

We used depth-frame alignment and motion-sensor orientation instead. ICP helped align successive measurements, but tracking remained difficult during fast movements or when the sensor faced surfaces with little geometric detail, such as blank walls.

This taught us that collecting depth measurements is only part of mapping. We also need to know where those measurements belong in a consistent coordinate system.

Getting the ultrasonic hardware working

The HC-SR04 integration involved more than connecting a sensor to a board.

We ran into firmware compilation and board-detection problems with the XIAO nRF52840 Sense. We had to fix the USB serial configuration and pin selection before we could reliably collect readings.

We also had to account for the HC-SR04's 5 V echo signal when interfacing with the microcontroller's 3.3 V logic, using a resistor divider to protect the board.

Once the hardware was working, we still had to build the serial ingestion, streaming, and replay components needed to connect it to the rest of EchoSense.

Making different sensors agree about the room

LiDAR and sonar don't measure the environment in the same way.

LiDAR provides spatial depth information, while an ultrasonic sensor returns a range measurement over a relatively broad sensing region. Comparing the two requires knowing the sonar's position and orientation relative to the phone.

We couldn't simply treat a sonar reading as another LiDAR point. We needed to preserve the distinction between the measurements and avoid implying that a disagreement automatically identified a particular material or obstacle.

Handling incomplete and stale maps

This became one of the most important engineering problems we encountered.

A map can remain visible even when no new measurements are arriving. In an early implementation, that could make a historical scan look like an active one.

We added provenance and capture-freshness handling so the interface can distinguish live, simulated, replayed, and historical data. The mapper tracks when a valid depth frame was last received, rather than treating outgoing map updates as evidence that capture is still active.

The viewer preserves the last useful map when capture stops, but it no longer needs to pretend that the map is still updating.

Integrating everything before the deadline

Each part of EchoSense was being developed by a different team member: the viewer, the iPhone mapper, the sonar bridge, and the firmware.

Getting those pieces into one working pipeline meant resolving differences in data formats, timing, coordinate systems, and connection behavior. A component passing its own tests didn't necessarily mean it would work when connected to the rest of the system.

A significant amount of our final development time went into integration, testing, and making the demonstration reproducible rather than adding new features.

Accomplishments that we're proud of

We built and tested a real iPhone LiDAR mapping pipeline. The system has captured room geometry from a physical iPhone and displayed the resulting map through our Python backend and browser viewer.

We got the ultrasonic sensor working on physical hardware. The HC-SR04 produced distance readings at approximately 15 Hz and tracked a moving hand over a reported range of roughly 0.1 m to 1.9 m. We connected its measurements to the software pipeline and built independent recording and replay support.

We made unobserved space visible. EchoSense distinguishes geometry that has actually been measured from areas that remain unknown. An unscanned region is not treated as confirmed empty space.

We built more than a point-cloud viewer. The browser includes trained surface classification, a geometric fallback, floor-plan analysis, room measurements, saved maps, and export functionality.

We worked on reliability as part of the product. EchoSense preserves historical maps, distinguishes sensor-data sources, and can identify when live depth capture has stalled. We also built automated tests covering the mapper, viewer, and sonar bridge.

We connected the live sonar to the mapper and put our data on Databricks. With the real HC-SR04 streaming and simulated phone frames, the mapper compared sonar and LiDAR distances and sent the result to the viewer. On Databricks, we deployed two Asset Bundles on Free Edition, and the sonar pipeline ran end to end on 200 simulated scenes.

What we learned

One of our biggest takeaways was that a detailed map isn't necessarily a trustworthy map.

LiDAR can generate thousands of measurements and still miss a surface. Glass may transmit or reflect its near-infrared light in ways that make depth measurements unreliable. Dense smoke can also interfere with optical depth sensing, which means LiDAR alone isn't a complete solution for the environments we're interested in.

We also learned that disagreement between sensors can be useful information. If sonar reports an obstacle where LiDAR doesn't, simply averaging the measurements could hide the discrepancy. It may be more useful to flag the difference and investigate why it happened.

The hardware pivot taught us another lesson: building a system around interchangeable components makes it easier to recover when a particular sensor or implementation doesn't work out.

Finally, we learned how much engineering happens between a working sensor and a useful application. Localization, data streaming, coordinate transformations, classification, browser rendering, recording, and failure handling all affect whether the final result can be understood and trusted.

What's next

Our next priority is collecting more real-world measurements and testing EchoSense under controlled conditions.

We want to run full-room scans with the lights off, evaluate how well depth-based tracking holds up without visible light, and test the system around glass, reflective surfaces, and other difficult geometry.

We also want to improve the calibration between the phone and ultrasonic sensor and evaluate the surface classifier against more real scans. The current model was trained on simulated room data, so measuring its performance on real environments is an important next step.

Longer term, we'd like to investigate thermal or mmWave sensing for environments where smoke makes optical depth measurements unreliable.

We also want to keep improving how the viewer communicates uncertainty. A useful map should make it easy to distinguish what has been observed, what is inferred, and what remains unknown.

EchoSense began as an attempt to map a room without relying on a camera. By the end of the hackathon, we had learned that the harder problem wasn't just building the map. It was figuring out how to communicate what the map could—and couldn't—tell us.


EchoSense is an experimental prototype for spatial mapping and simulated search-and-rescue planning. It has not been validated for operational firefighting, does not see through walls or dense smoke, and should not be relied upon for life-safety decisions.

Share this project:

Updates

Submission history