EarthLens

12:00 AM EarthLens uses on-device AI, GPS, and sensors to identify mountains, lakes, islands, and cities, even offline, while refusing to guess when evidence is uncertain.

Comment

Inspiration

People often see a mountain, lake, island, or distant city without knowing what they are looking at. Most discovery tools assume a reliable internet connection, require manual map searching, or upload images to cloud services.

Those assumptions break down on trails, scenic overlooks, flights, and in low-connectivity communities. EarthLens explores a different approach: use the sensors and computing power already available on the phone, keep the core experience functional offline, and be honest when the available evidence is insufficient.

What it does

EarthLens turns an Android phone into an offline-aware geographic discovery lens.

Users point the camera toward the landscape, and EarthLens projects nearby mountains, lakes, islands, and cities into the live camera view. It calculates where each feature belongs using:

  • GPS position
  • Native Android heading
  • Phone tilt
  • Geographic bearing
  • Feature distance and elevation
  • Camera field of view

Mountains, water features, and cities use distinct marker colors. Users can select a feature to view its distance, elevation, bearing, and geographic details.

The core application, fonts, landmark data, radar, and AI model are bundled inside the APK. EarthLens can therefore launch and provide its core experience in airplane mode. When connectivity is available, OpenStreetMap provides optional map tiles and additional nearby geographic data.

How I built it

EarthLens uses React and TypeScript for the interface, Vite for the web build, and Capacitor to package the application as a native Android APK.

A native Java bridge uses Android rotation-vector sensors to determine the direction of the rear camera. GPS establishes the observer's position, while geographic calculations determine the bearing and distance of each candidate feature.

The horizontal relationship between a feature and the camera is calculated as a normalized angular difference:

[ \Delta\theta = ((\theta_{feature} - \theta_{camera} + 540) \bmod 360) - 180 ]

A feature is displayed only when this difference falls inside the camera field of view.

For AI analysis, EarthLens captures a reduced camera frame and runs Google ML Kit Image Labeling directly on the device. The model produces semantic labels and confidence scores for concepts such as mountains, landscapes, water, buildings, and urban scenes.

EarthLens then combines the visual evidence with structured geographic context:

  • Visual compatibility
  • Camera alignment
  • Field-of-view membership
  • Candidate distance
  • Ambiguity between competing candidates

If the evidence is weak or multiple candidates receive similar scores, EarthLens abstains instead of forcing an identification. Camera frames are processed locally and are not uploaded for inference.

AI that performs real work

The AI is not a chatbot or a single cloud API call.

The bundled vision model analyzes the live scene. EarthLens then performs multimodal evidence fusion across computer vision, GPS, native orientation sensors, and geographic candidates. The system ranks candidates, measures ambiguity, and decides whether to return a result or reject the identification.

The current general-purpose model understands broad scene categories. It does not independently recognize a uniquely named mountain from its silhouette. Named results depend on the combination of visual and geographic evidence, and EarthLens explicitly communicates uncertainty.

Challenges

Mobile orientation

Device orientation was the most difficult engineering challenge. Browser orientation events, Android screen rotation, magnetic heading, geographic bearings, and rear-camera direction use different coordinate systems and sign conventions.

We replaced incremental heading offsets with a native Android rotation-vector implementation and added automated tests for north, south, east, west, landscape orientation, and wraparound at 0 and 360 degrees.

Reliable offline behavior

We had to separate features that were truly offline from optional network enhancements. The complete application shell now loads from assets bundled inside the APK rather than a remote website.

EarthLens also handles camera permissions, location permissions, app backgrounding, expired network requests, and unavailable OpenStreetMap services without leaving users on blank or infinite-loading screens.

Honest AI results

General image labeling is not the same as unique landmark recognition. We designed the evidence-fusion and abstention system so geographic proximity is not presented as visual certainty.

What I learned

We learned that geographic augmented reality depends as much on coordinate systems, sensor quality, lifecycle handling, and uncertainty as it does on interface design.

We also learned that offline-first behavior must be architectural. Caching a few files is not enough: startup, permissions, data, inference, failure states, and navigation must all be designed to work without a server.

Finally, we learned that a useful AI system should be able to say “I do not have enough evidence.” Refusing an uncertain result can be more valuable than displaying a confident but incorrect answer.

What's next

Next steps include:

  • Landmark-specific image embeddings
  • Reference-image retrieval
  • Larger downloadable geographic data packs
  • Offline regional map tiles
  • Improved altitude calibration
  • Field testing across different phones and landscapes
  • A larger labeled evaluation dataset
  • Accuracy, false-positive, abstention, and latency metrics

My goal is to make EarthLens useful for outdoor education, park interpretation, field research, travel, and geographic discovery wherever connectivity is limited.

EarthLens: Point. Understand. Stay offline.

Built With

Share this project:

Updates

Submission history