Herbei

Herbei: Your On-Device Second Brain

Comment

🚀 Herbei — Project Submission

💡 Inspiration

We live in an age of information overload. We take dozens of photos, write quick notes, and capture moments every day—yet finding a specific detail later feels like finding a needle in a haystack. Traditional keyword search fails when you can't remember the exact words you used, and cloud-based AI solutions raise major privacy concerns when analyzing personal photos and daily thoughts.

We built Herbei to solve this. Inspired by the concept of a "Second Brain," Herbei is a privacy-first, 100% on-device AI assistant that automatically understands images, creates semantic memories ("Neurons"), and lets you search your life using natural language—without a single byte leaving your phone.

📱 What It Does

Herbei is an intelligent, privacy-first Semantic Memory Assistant that acts as your personal "Second Brain." Here is how it works:

  • Visual Understanding: Capture a photo or pick an image from your gallery, and Herbei's local Vision AI (SmolVLM2) automatically analyzes the image and generates a detailed description.
  • Semantic Embedding ("Neurons"): Converts images, text notes, and tags into 384-dimensional vector embeddings, storing them as interconnected memory nodes ("Neurons").
  • Natural Language Search: Find anything using conceptual queries instead of rigid keywords. Searching for "where did I leave my prescription" or "items from my trip to the park" instantly finds the right memory even if those exact words weren't typed.
  • Memory & Node Exploration: Provides an interactive dashboard to explore past memories, search history, and tag networks with instant swipe-to-manage controls.
  • 100% Offline & Private: All AI inference, image processing, and database storage happen strictly on your device. Your personal moments never touch a cloud server.

🛠️ How We Built It

Herbei is engineered as a modern, local-first Android application built with a Feature-based Clean Architecture:

  • On-Device Vision AI: Integrated SmolVLM2-500M using Google’s LiteRT (TensorFlow Lite) Engine to analyze and describe photos locally.
  • Semantic Search Engine: Utilized ONNX Runtime with the Sentence Transformer (MiniLM-L6-v2) model to convert descriptions and tags into normalized 384-dimensional vector embeddings.
  • Vector Matching: Implemented a custom, high-performance Cosine Similarity algorithm in Kotlin to rank and retrieve memories semantically in real time.
  • Local Data Persistence: Built with Room Database, using custom Little-Endian ByteBuffer converters to store high-dimensional float vectors compactly and efficiently.
  • Modern Android Stack: Built entirely in Jetpack Compose (Material 3), Jetpack Navigation 3, Koin for Dependency Injection, and Ktor for chunked binary model downloads with real-time progress reporting.

🚧 Challenges We Ran Into

  1. Edge AI Memory & Hardware Limits: Running both a Vision-Language Model (VLM) and a Sentence Transformer on a mobile device required strict memory lifecycle management to avoid OutOfMemoryError during inference.
  2. Efficient Vector Storage: Standard JSON/string serialization for 384-element float vectors was slow and bloated Room database reads. We solved this by implementing direct binary ByteBuffer conversion, cutting storage overhead by 75% and speeding up vector loads.
  3. Large Model Onboarding: Distributing ~500MB of AI model weights required building a robust streaming downloader using Ktor channel buffers to prevent memory spikes and handle connection interrupts gracefully.

🏆 Accomplishments That We're Proud Of

  1. Running Multimodal AI 100% On-Device: Successfully executing both a Vision-Language Model (SmolVLM2) and a Sentence Transformer model locally on an Android device with minimal battery overhead and fast inference times.
  2. Sub-Millisecond Semantic Vector Search: Implementing custom Kotlin-native Cosine Similarity vector matching that searches through hundreds of high-dimensional embeddings instantly without UI jank.
  3. Ultra-Efficient Binary Vector Storage: Bypassing slow JSON parsing by engineering a custom ByteBuffer binary converter in Room, cutting vector storage size by 75% and making database reads lightning-fast.
  4. Polished Onboarding & Architecture: Crafting a smooth initial download flow for 500MB+ model weights using Ktor streaming channels, backed by a clean Jetpack Compose & Navigation 3 architecture.

🎓 What We Learned

  • The Power of On-Device AI: Modern Edge AI frameworks (LiteRT & ONNX Runtime) are now mature enough to build rich, private, cloud-independent applications directly on phones.
  • Vector Mathematics in Kotlin: Deepened our understanding of high-dimensional vector spaces, embedding normalization, and optimizing matrix similarity math for mobile CPUs.
  • Cutting-Edge Jetpack Libraries: Mastered the new Jetpack Navigation 3 API and state synchronization patterns for Jetpack Compose.

🔮 What's Next for Herbei

  • Encrypted Cloud Backup & Multi-Device Sync: Implement End-to-End Encrypted (E2EE) cloud backup, allowing users to securely back up their memory database to a server and sync across devices without compromising zero-knowledge privacy.
  • Encrypted Secure Vault for Sensitive Info: Integrate biometric authentication (Fingerprint / Face Unlock) and AES-256 hardware encryption to let users safely store sensitive documents, personal credentials, and private notes.
  • Interactive 3D Knowledge Graph: Build a visual 2D/3D graph view of "Neurons" so users can visually explore clusters of connected memories, tags, and ideas.
  • System Widgets & AI Agent Shortcuts: Add homescreen widgets and Android AppFunctions integration to allow instant one-tap visual capture and voice-activated memory search without opening the app.

Built With

  • ai
  • android
  • android-studio
  • clean-architecture
  • coroutines
  • flows
  • herbei
  • image-processing
  • jetpack-compose
  • kotlin
  • multi-module-model
  • mvvm
  • room
  • sementic-search
Share this project:

Updates

Submission history