HAWKEYE

HAWKEYE Inspiration

Software knows where digital objects are, but the physical world lacks a shared system for identifying and tracking places and their changing states. A fridge, a shelf, or a table has meaning to people, but devices have no common way to recognize the same spot and share what they observe there. We were inspired to bridge this gap by creating a shared, continuously updated view of the physical world.

What it does

HAWKEYE creates geospatial checkpoints for real-world locations, allowing devices to recognize places and share their current state. By saying, “Hey Hawkeye, remember this,” a user can turn a physical spot into a checkpoint stored in a FalkorDB graph.

Every device shares the same checkpoints through a mesh of observers. When one observer detects a change, the shared state is updated so other observers can access the latest information. Checkpoints are recognized visually rather than relying solely on GPS coordinates, making them useful indoors and even for specific locations as small as a single shelf.

Users can also ask questions about places remotely, such as, “How many Coke cans are left on the table?” HAWKEYE can provide the current count and relevant context about observed changes, helping users understand not just what is there, but what happened.

How we built it

We built HAWKEYE around three core ideas: visual checkpoint recognition, shared state, and a distributed mesh of observers.

Geospatial checkpoints: Physical locations are saved as recognizable checkpoints that can be identified across devices, including indoor environments where GPS is unreliable. FalkorDB graph: Checkpoints and their current states are stored in a shared graph, providing a common source of truth for observers. Observer mesh: Each phone acts as an observer, contributing new observations and helping keep checkpoint information up to date across devices. Natural-language interaction: Users can interact with HAWKEYE through simple voice-based commands and questions, making physical-world information accessible without manually searching through records.

Together, these components create a system in which observations from individual devices contribute to a shared understanding of the physical world.

Challenges we ran into

One of our biggest challenges was finding a reliable way to identify the same physical location across different devices without depending entirely on GPS. Indoor spaces and small areas require much more precise recognition than traditional geolocation can provide.

Another challenge was keeping checkpoint information current as objects moved, disappeared, or were added. A single observation is not enough when multiple people and devices interact with the same environment.

We also had to think about how to make observations from different devices contribute to one consistent shared state, rather than leaving each device with an isolated view of the world. Finally, connecting these capabilities to a natural conversational interface meant making complex tracking and retrieval feel simple for the user.

Accomplishments that we're proud of

We are proud of the idea of turning ordinary physical locations into shared, recognizable checkpoints that can be accessed across devices.

HAWKEYE brings together visual recognition, a graph-based representation of the world, and a mesh of observers to make physical spaces more understandable and queryable. Instead of treating every camera as an isolated system, we designed an approach where each observation can contribute to a collective understanding.

We are especially excited about the ability to ask questions about a location without being physically present. This opens the door to practical applications such as inventory tracking, shared spaces, household organization, and understanding how objects change hands over time.

What we learned

We learned that understanding the physical world requires more than recognizing objects in an image. A useful system must also understand where an observation happened, whether that location has been seen before, and how the observed state has changed over time.

We also learned the importance of shared state. Individual observations become much more valuable when they can be connected to a common checkpoint and made available to other devices.

Finally, we saw how natural-language interaction can make complex systems feel intuitive. People should be able to ask straightforward questions about their surroundings rather than navigate databases or manually maintain inventories.

What's next for HAWKEYE

Our next step is to make HAWKEYE's observer mesh more robust, scalable, and useful in everyday environments. We want to improve checkpoint recognition across different devices and viewing conditions, strengthen the consistency and freshness of shared state, and expand the system's ability to track changes over time.

We also envision richer histories for checkpoints, allowing users to understand not only what is currently present but also when changes occurred and which observer reported them. Additional applications could include smart homes, offices, warehouses, laboratories, and other shared spaces where knowing the current state of physical objects matters.

Our long-term vision is to give the physical world a shared, continuously updated layer of context — a world where any observer can contribute what it sees, every device can build on those observations, and anyone can ask questions about places and get meaningful answers.