Life Lens Mobile app

Inspiration

We're surrounded by things we can't quite name. A gadget on a shelf with no price tag, a plant that's slowly dying, a plate of food with unknown calories, a document in a language we don't read. The knowledge exists online ? but the gap between seeing something and understanding it is a dozen taps: unlock, search, type a clumsy description, sift results.

I wanted to close that gap to a single gesture: point your camera at anything, and know everything. The Qwen Cloud hackathon's EdgeAgent track was the perfect fit ? a device that perceives the world through its sensors, reasons in the cloud, and acts locally. That's exactly the loop LifeLens runs.

What it does

LifeLens turns your phone camera into a universal "what is this?" agent. You point, and it:

Identifies the subject with Qwen-VL and returns a structured result ? title, category, confidence, and salient attributes.
Adapts the result to what you scanned. A CategoryHandler policy routes each object type to its own view:
- ? Food ? nutrition & macros with a portion stepper
- ? Electronics / products ? spec sheet + a live price range and where to buy
- ? Plants ? a care card (light, water, pet-safety)
- ? Documents ? transcribed text
- ? Books / ? clothing ? tailored details
Grounds prices in real listings by querying several search engines and always surfaces a "Where to buy" list (Google/Bing/DuckDuckGo Shopping + Amazon, eBay, Walmart, Best Buy, and more) ? localized to your currency when you share location.
Runs autonomously with an Auto-scan mode that identifies a steady subject with no shutter tap.
Lets you interrogate the object. An "Ask about this" thread seeds a Qwen chat with the identification context, so you can ask follow-ups conversationally.
Keeps working when the network doesn't ? an offline last-result fallback and graceful degradation instead of dead-ends.

Everything is saved to a private, on-device library you can search, favorite, and revisit.

How we built it

Qwen is the single brain. One multimodal model ? qwen-vl-max, served by Alibaba Cloud Model Studio (Qwen Cloud / DashScope) through its OpenAI-compatible endpoint ? does both the vision understanding and the natural-language synthesis of grounded search results and follow-up answers. The agent's control loop is perceive ? reason ? route ? act ? (optionally) converse.

The app is a 15-module Android codebase built on modern Android:

Jetpack Compose + Material 3 for a fully custom, edge-to-edge UI
Hilt for DI, Coroutines + Flow with a StateFlow unidirectional data flow
CameraX for capture and live framing
Retrofit + OkHttp + kotlinx.serialization for the Qwen calls, with defensive JSON parsing
Room (history) + DataStore (settings/keys), Coil for images
build-logic convention plugins and a Gradle version catalog to keep modules consistent
150+ unit & Compose tests that run entirely on the JVM via Robolectric, so the whole thing is verifiable without a device

Search grounding is an AggregatingSearchClient that fans out to multiple providers concurrently and merges the results, so pricing survives any single engine failing.

Challenges we ran into

Getting strict JSON out of an LLM. Qwen occasionally returns values the schema didn't expect ? an array where a string was declared ("Ingredients": ["rice","water"]), numbers as strings, extra prose around the JSON. I built a tolerant parser that extracts the JSON object and coerces every attribute value to a display string instead of crashing.
"No price found" was unacceptable. Keyless search scraping is inherently flaky (rate limits, shifting markup, consent walls). Instead of dead-ending, I made the buy screen always generate a deterministic list of search engines and retailers to open ? so there's guaranteed a path to buy, with live scraped prices layered on top when available.
Edge-to-edge polish. Handling the status bar, navigation bar, and the soft keyboard correctly across a draggable result sheet and text inputs took real inset work (nav-bar ? IME) so nothing ever hides behind the keyboard.
Edge-cloud orchestration. Designing graceful degradation ? offline fallback, location-optional currency, and a default key that works out of the box without ever being exposed ? so the agent is robust in the messy real world.

Accomplishments that we're proud of

A genuine EdgeAgent loop where a single Qwen-VL model powers perception and reasoning and conversation ? not a bolt-on.
The category-adaptive result modal: the same camera gesture yields nutrition, specs, plant care, or a transcription depending on what you point at.
It just works ? a bundled default key and keyless search mean you can install and scan immediately, yet everything degrades gracefully offline.
A fully JVM-testable multi-module architecture with a large, green test suite.

What we learned

Treat LLM output as untrusted input: parse defensively and design for the model surprising you.
Reliability beats cleverness for grounding ? a guaranteed set of buy links serves the user better than a fragile scraper that sometimes returns a perfect price and often returns nothing.
One capable multimodal model can collapse what used to be several services (vision, NLP, synthesis) into one clean agent loop.

What's next for LifeLens

Barcode & label reading and multi-item scenes in a single frame.
Live translation of signs and menus (OCR + translate) as another CategoryHandler.
Price tracking & alerts on saved items, powered by the existing refresh pipeline.
On-device history intelligence ? ask questions across everything you've ever scanned.

Built with Qwen-VL on Alibaba Cloud Model Studio, Kotlin, Jetpack Compose, Hilt, CameraX, and Room.

link to download the app: https://drive.google.com/file/d/1rQUHAEhdUZattOlfVl0koNiHNpeJ7Olj/view?usp=sharing