pov-agent

• 

  first menu - download models to use locally

• 

  settings

• 

  offline usage example

Inspiration

This project started after a trip with my friends. We hiked around 15 kilometers into a forest looking for an unlocked shelter, but never found it. We tried to start a fire with the materials around us, failed, and had no internet connection to ask for help.

That made me think about a different kind of AI assistant: not another cloud chatbot, but a calm companion that can understand the approximate situation around you and remain useful when there is no connection.

What it does

Local AI Observer is a private, object-aware assistant that runs directly on the phone. Internet access is required only once to download the models.

The camera is continuously analyzed by YOLO. Instead of sending every frame to the language model, the application converts noisy detections into a stable scene containing object labels, approximate positions, and session-scoped IDs.

Qwen uses that structured scene to:

• generate short observations about the surroundings;
• answer typed questions with current visual context;
• answer hands-free questions after the wake phrase;
• speak responses using local text-to-speech.

Camera frames, audio, prompts, and conversations are never uploaded or saved. Qwen does not receive the raw image—only the stabilized object description.

How I built it

The application is built with Flutter and follows feature-oriented architecture with clear presentation, application, domain, and data boundaries.

The local AI pipeline combines:

• YOLO for real-time object detection;
• a custom scene stabilizer that filters detection noise;
• Qwen3-0.6B Q4_K_M through a pinned llama.cpp runtime and a narrow Dart FFI layer;
• Sherpa-ONNX for streaming speech recognition;
• Piper for offline speech synthesis;
• BLoC for observation, dialogue, lifecycle, and concurrency orchestration.

A shared model manager downloads, verifies, caches, loads, and releases every model. Checksums are validated before a model can be used, making subsequent launches fully offline.

I used Codex as an implementation and review partner throughout the project. I also built a Flutter Agentic Harness that encodes my architectural and quality conventions. Every milestone passed deterministic checks first and then a semantic agent review with finding verification. Besides catching bugs, the harness made the codebase easier to understand because every feature follows the same rules and ownership model.

Challenges

The hardest part was not calling the models—it was making several native runtimes coexist on a mobile device.

YOLO, Qwen, ASR, and TTS compete for memory and compute, especially on an iPhone 11 with 4 GB of RAM. I had to explicitly control when models are loaded and released, prevent overlapping generations, cancel obsolete work, and ensure that camera and audio resources are always released during lifecycle transitions.

Another challenge was turning unstable frame-by-frame detections into useful context. An object must appear in several recent frames before it enters the stable scene, so isolated YOLO errors do not constantly change the prompt.

The Flutter application and architecture target both iOS and Android. However, the submitted MVP package is currently iOS-only because of an Android YOLO/runtime model compatibility issue discovered during final integration. Android packaging and real-device validation remain the next engineering step.

What I learned

Building useful on-device AI is primarily a systems-engineering problem. Memory ownership, concurrency, lifecycle handling, model verification, and honest capability boundaries matter as much as model quality.

I also learned that a small language model does not always need raw video. A compact, stabilized scene description is cheaper, more predictable, and easier to keep private.

What's next

Next, I want to complete Android packaging and real arm64 device testing, improve scene tracking, and explore an optional local visual-language-model path for details that object labels cannot represent—while preserving the same offline and privacy guarantees.

Built With

• android
• bloc
• computer-vision
• dart
• ffi
• flutter
• gguf
• ios
• llama.cpp
• metal
• offline-ai
• on-device-ai
• onnx
• openai
• piper-tts
• qwen3
• sherpa-onnx
• streaming-asr
• ultralytics
• yolo