Nura

Child health screening, even without a signal.

Comment

🩺👶 Nura App: On-Device & Hybrid AI for Early Stunting & Malnutrition Prevention

Nura App is an innovative, privacy-first mobile application built with React Native (Expo SDK 57) designed for the early detection, screening, and tracking of stunting and childhood malnutrition in remote areas.

By utilizing a Hybrid AI architecture, Nura merges the computational depth of Cloud AI (Gemini 2.0 Flash / 1.5 Pro & Groq/DeepSeek) with the offline availability of On-Device Deep Learning (MobileNetV2 running on Microsoft ONNX Runtime), creating a persistent, localized healthcare safeguard.

💡 Inspiration

In developing countries like Indonesia, stunting and childhood malnutrition remain severe national healthcare priorities. Early detection is crucial: if nutritional deficiencies are identified within the first 1,000 days of a child's life, the effects of stunting can be mitigated or even reversed.

However, real-world deployment faces two critical bottlenecks:

  1. Connectivity Deserts: The rural and remote villages (such as Posyandu outposts) where screening is most needed frequently suffer from weak, unstable, or non-existent internet access.
  2. Clinical Complexity & Panic: Existing medical apps either use dry, confusing interfaces that frighten parents, or require expensive diagnostic equipment.

We wanted to create a solution that operates entirely offline when necessary, guides users through a friendly, non-frightening physical scan using standard smartphone cameras, and establishes a "Local Learning Loop" that tracks patient progression over time rather than returning a scary, single-point-in-time verdict. Thus, Nura App was born.

⚙️ What it does

Nura App acts as a digital companion for parents and local healthcare workers, delivering a comprehensive screening workflow:

  1. Age-Adaptive Dynamic Questionnaire: Instead of static forms, Nura leverages DeepSeek AI to formulate age-appropriate, dynamic screening questions tailored to the child's exact age in months.
  2. 3-Step Guided Camera Scan: Guides users via real-time on-screen camera overlays to capture physical indicators of malnutrition:
    • 👁️ Eyes (Conjunctiva): To check for paleness, indicating anemia.
    • 💅 Nails (Nailbed): To evaluate capillary refill and nutritional markers.
    • 👤 Face: To check for symmetry and physical symptoms of wasting/malnutrition.
  3. Hybrid AI Diagnostic Engine:
    • Online Mode: Transmits compressed images to Gemini 2.0 Flash / 1.5 Pro and Groq Llama 3.2 Vision for high-precision visual screening, with DeepSeek API performing the final integrated clinical diagnosis.
    • Offline Mode: Bypasses the network entirely, running an on-device MobileNetV2 model locally via ONNX Runtime directly on the phone's GPU/CPU.
  4. Local Learning Loop (SQLite): Stores patient profiles and screening sessions locally. Upon subsequent scans, the app fetches previous sessions to determine if the child is improving (membaik), stagnant (belum membaik), or worsening (memburuk).
  5. Healthcare Finder (Faskes Screen): Uses map integration to direct parents to the nearest physical clinics, hospitals, or health centers.
  6. Educational Library: A curated, offline-first digital bookshelf containing illustrated guides, nutrition tips, and local developmental milestones.

🛠️ How we built it

Nura App was engineered with performance, responsiveness, and strict adherence to our design principles:

  • Mobile Framework: Built using React Native (Expo SDK 57) to ensure rapid, unified development compiled to native Android performance.
  • On-Device Inference: Integrated Microsoft's ONNX Runtime (onnxruntime-react-native) to load and run a MobileNetV2 model (mobilenetv2-12.onnx, ~45MB) locally. We developed a custom sideloading pipeline (modelDownloader.ts) that shows real-time progress indicators when downloading the model.
  • Cloud AI Integration: Connected to Gemini 2.0 Flash / 1.5 Pro (via Google AI SDK), Groq, and DeepSeek API using secure, decoupled server layers.
  • Data & Local Persistence: Used Expo SQLite (expo-sqlite) to maintain a fast, relational schema for profiles and history tracking without needing a cloud database.
  • Smart WebP Image Compression: Designed a client-side preprocessing pipeline using HTML5 Canvas (web) and expo-image-manipulator (mobile) that resizes images to a maximum of 600px width at 60% quality, converting ~5MB RAW camera frames into highly detailed ~20KB WebP payloads in milliseconds.
  • Premium Design System: Developed Plus Jakarta Sans (humanist sans-serif) and Inter typographies. Designed standard pastel/solid color tokens, touch-first inputs (min 44x44px), glassmorphic components, and linear gradients to ensure maximum readability under direct sunlight and high aesthetic appeal.

⚠️ Challenges we ran into

  • ONNX Runtime on React Native: Linkage and environment setup of onnxruntime-react-native with the latest Expo SDK 57 presented deep package conflicts. We solved this by creating specific native configurations, leveraging patch-package to handle minor runtime discrepancies, and optimizing image-to-tensor data transformations.
  • High-Resolution Camera Lag: Initial drafts caused UI stuttering when capturing 12MP images from modern cameras. We solved this by offloading the image manipulation (resizing, WebP compression, and base64 conversions) to asynchronous background threads, resulting in instantaneous, freeze-free captures.
  • Graceful Network Failover: Building a state machine that handles sudden cellular drops without crashing was a major hurdle. We designed a parser (cleanErrorMessage) that catches network issues, rate limits (HTTP 429), or server faults, alerts the user, and automatically falls back to local ONNX runtime offline.
  • Age-Calculations & Dynamic Forms: Synchronizing age calculation down to the exact month with dynamic API questions required rigorous validation to prevent questionnaire generation delays.

🎉 Accomplishments that we're proud of

  • Fully Offline Visual Inference: Running a convolutional neural network (MobileNetV2) on a budget phone entirely offline, processing image inputs in less than 150ms.
  • 99% Bandwidth Reduction: Shrunk camera payloads from ~5MB to ~20KB via WebP compression, while retaining the sharp color gradients of the conjunctiva and fingernails necessary for medical screening.
  • Local History Evolution Loop: Created a system that tracks a child's clinical journey chronologically rather than treating diagnostics as a series of isolated events.
  • Unified Brand Experience: Built a stunning, responsive layout that matches the NURA Design System across both mobile viewports and web desktop frames.

🧠 What we learned

  • On-Device Tensor Manipulation: Transforming camera pixel buffers into standardized 3-channel floats expected by ONNX models on React Native.
  • Clinical Interface UX: The importance of non-frightening text copy, soft illustrations, and transparent warning indicators (e.g., using "Belum membaik" instead of critical red alerts to prevent parent panic while encouraging medical consultation).
  • Agile Mobile Development: Harnessing modern AI tooling (Antigravity by Google DeepMind) to refactor type definitions, debug native Gradle builds, and resolve state synchronization issues.

🔮 What's next for Nura

  1. Lightweight Custom VLM Fine-Tuning: Train a custom, lightweight Vision-Language Model specifically on stunting and pediatric clinical indicators (conjunctival anemia, wasting face, koilonychia) optimized for local ONNX runtimes.
  2. National Health Registry Sync (Posyandu Connect): Integrate sync bridges with national health databases (like ePPMGBM in Indonesia) to securely upload stored SQLite records once internet connection is established.
  3. AR Height & Length Verification: Build an augmented reality (AR) camera ruler to help parents measure child height and length dynamically, eliminating manual tape-measurement errors.
  4. Local Dialect Translation: Add voice-over translation support in Indonesian local languages (Javanese, Sundanese, Balinese) for greater accessibility in remote outposts.

🚀 Getting Started

Prerequisites

  • Node.js (v18+)
  • npm or yarn
  • Expo Go (for running on physical test devices) or Android Studio (for emulator setup)

Installation

  1. Clone the repository: bash git clone https://github.com/mrshaadiq/Nura-App.git cd Nura-App
  2. Install dependencies: bash npm install
  3. Set up your API Keys in ai/env.ts (create the file if it does not exist): typescript export const GEMINI_API_KEY = "YOUR_GEMINI_API_KEY"; export const DEEPSEEK_API_KEY = "YOUR_DEEPSEEK_API_KEY"; export const GROQ_API_KEY = "YOUR_GROQ_API_KEY";

Running the App

  • Start the Expo development server: bash npm start
  • Run on an Android device/emulator: bash npm run android
Share this project:

Updates