BawalVision

Deterministic, zero-copy computer vision pipeline in pure C++.

Comment

Inspiration

Across the globe, large-scale school meal initiatives—such as Indonesia's nationwide Makan Bergizi Gratis (MBG) program serving tens of millions of school children daily—face critical logistical and safety hurdles. In high-volume central kitchens and distributed school cafeterias, verifying food freshness, microbial thermal safety, and nutritional portion compliance is still performed manually or not at all. Food left in the microbial "danger zone" ($4^\circ\text{C} - 60^\circ\text{C}$) leads to severe foodborne contamination risks, while subjective visual inspection lacks verifiable accountability. We envisioned BawalVision: an edge-to-cloud automated computer vision and multi-modal telemetry system that inspects meal trays in milliseconds. Our mission was to combine high-speed edge hardware, deterministic SIMD computer vision, and quantized deep learning to ensure that every student receives safe, fresh, and nutritious food with an immutable digital audit trail

What it does

BawalVision provides real-time, multi-modal quality control across 9+ Indonesian school food categories (banana, chicken, egg, orange, rice, spinach, tempeh, tofu, water_spinach, and apple):

  • Real-Time Tray Detection: Accurately localizes, classifies, and bounds all food components on the tray using an AVX2-accelerated YOLOv8s INT8 neural network.
  • Non-Contact Thermal Safety: Synchronously measures food surface temperatures via contactless infrared sensing (MLX90614) to flag microbiological danger zones before meals are distributed.
  • Microsecond Spoilage & Freshness Evaluation: Applies vectorized colorimetric and textural variance analysis (HSV/Lab color spaces) to detect chemical discoloration and wilting.
  • Ultra-Low Latency Ingestion: Sustains a solid $15.7\text{ FPS}$ continuous video stream with an end-to-end latency of just $57.5\text{ ms}$ over zero-copy POSIX shared memory.
  • High-Throughput Batch Inspection: Features an OpenMP-accelerated batch ZIP processor capable of inspecting up to 50 trays simultaneously at $4.41\text{ trays/second}$.
  • Cryptographic Audit Certificates: Automatically issues compliance certificates in compiled PDF and raw LaTeX formats, complete with 64-bit FNV-1a cryptographic integrity checksums.
  • Unified Web HMI & Telemetry Fleet Management: A responsive SvelteKit 5 web dashboard offering live annotated camera streams, host USB Wi-Fi auto-provisioning, and integrated Grafana fleet analytics.

How we built it

BawalVision is architected as an edge-to-cloud polyglot monorepo engineered across multiple hardware and software layers:

  1. Edge Firmware (edge/):

    • Built on an ESP32-S3-WROOM-1 (16MB Flash, 8MB Octal PSRAM) running FreeRTOS with 100% static memory allocation (xTaskCreateStaticPinnedToCore), eliminating heap fragmentation during 24/7 operation.
    • Direct DMA parallel DVP capture with the OV5640 camera (VGA $640\times 480$, 20 FPS) and I2C polling for the MLX90614 infrared thermal sensor.
    • Zero-overhead streaming: skips bulky JSON heartbeats during video transmission and multiplexes sensor telemetry directly into HTTP Keep-Alive frame headers (X-ESP-Temp, X-ESP-CPU-Load).
    • Broadcasts UDP auto-discovery (BAWAL_DISCOVER_V1 on port 8765) to dynamically lock onto host backend IPs.
    • Local operator dashboard rendered on a $128\times 160$ SPI TFT (ST7789) via LVGL 8.

  2. C++20 Vision Engine & Backend (services/):

    • Powered by Crow HTTP/WebSocket, OpenCV 4, ONNX Runtime C++, and Microsoft's mimalloc 2.1 (which reduced heap contention churn by $11.7\times$ compared to standard glibc).
    • Zero-Copy IPC: Uses a decoupled POSIX Shared Memory (/dev/shm/bawal_shm) ring-buffer with monotonic sequence locks, isolating network I/O from compute-intensive neural inference ($0.22\text{ ms}$ IPC transfer).
    • SIMD AVX2 Acceleration: Implemented hand-crafted 8-lane AVX2 intrinsics (_mm256_sad_epu8, SSSE3 planar shuffles) to vectorize BGR channel unpacking and spoilage evaluation, executing in just $0.0058\text{ ms}$ ($20.97\times$ faster than scalar C++).
    • Strictly adheres to cross-platform portability by replacing non-portable -march=native flags with runtime CPU dispatching and explicit compiler flags (-mavx2, -mfma, OpenMP).

  3. Machine Learning Pipeline (ml/):

    • Anti-Data Leakage Splitting: Multi-threaded 70/15/15 dataset splitter utilizing MD5 and 64-bit perceptual difference hashing (dHash with Hamming distance $\le 4$) to guarantee zero test leakage.
    • Hard-Negative Scraper (dish_scraper.py): Crawls empty trays and plates, injecting 0-byte label files into YOLO training to suppress tableware false positives.
    • Zero-Dummy Relabeling (relabel_dataset_tight.py): Integrates zero-shot YOLO-World v2 with LAB border-background color saliency contours to generate tight multi-box ground truth without static dummy box fallbacks.
    • YOLOv8s Cold Training (train_AI1.py): Trained with AdamW, cosine scheduling, and mosaic disablement during the final 15 epochs (close_mosaic=15).
    • Incremental Transfer Learning (finetune_AI1.py): Enforces 10-layer backbone freezing (--freeze 10) and gentle learning rates ($lr_0 = 0.0002$) to absorb hard negatives and new food classes without catastrophic forgetting.
    • AVX2 PTQ INT8 Quantization (quantize_int8.py): Generates an INT8 QDQ graph calibrated for the AVX2 VPMADDUBSW instruction set (unsigned activation $\times$ signed weights). Dynamically excludes 50–80 detection head layers to preserve FP32 bounding box precision, yielding a $44.2\%$ weight reduction ($25.0\text{ MB}$) with confidence drops bounded within $\le 8\%$.
    • C++20 NCNN Generator: Synthesizes Real-ESRGAN $2\times$ sub-pixel super-resolution .param graphs and .bin weights directly without Python deployment overhead.

  4. Frontend & Orchestration (frontend/, deploy/):

    • Modern web HMI built with SvelteKit 5 (Runes), TypeScript, Tailwind CSS, and DaisyUI.
    • Multi-stage Docker Compose ecosystem hosting the C++ backend, PostgreSQL 16 (deploy/postgres/shema.sql), and pre-provisioned Grafana analytics.
    • Serverless AWS Lambda C++ container deployment (Dockerfile.lambda) with cold-start pre-warming.

Challenges we ran into

  • Tableware & Empty Tray False Positives: Early models memorized the textures of steel baki dividers and white ceramic plates as food items. We solved this by developing dish_scraper.py to collect thousands of empty plates and automatically inject 0-byte negative label files, reducing background false positives from $>1000$ to $<50$.
  • Catastrophic Forgetting in Transfer Learning: Retraining the model when introducing new items (such as apples or empty tray negative samples) caused classification degradation on previously trained categories. We designed finetune_AI1.py with 10-layer backbone freezing (locking layers 0–9) and a 5x lower learning rate ($lr_0 = 0.0002$) to adapt the detection head while keeping base feature extractors intact.
  • Quantization OOM Crashes & Accuracy Degradation: Standard entropy calibration consumed over $10\text{ GB}$ of RAM, triggering system OOM crashes. Furthermore, naive 7-bit reduction caused confidence scores to plummet from $70\%$ to $<40\%$. We overcame this by implementing MinMax calibration ($<400\text{ MB}$ RAM footprint) and programmatically traversing the ONNX graph backward from output0 to exempt the detection head from quantization.
  • Edge Memory Fragmentation: Sustained 20 FPS VGA video streaming repeatedly crashed the ESP32-S3 after hours of operation due to heap fragmentation. We eliminated all dynamic heap allocations in the FreeRTOS tasks, transitioning to 100% static task stacks and static queue buffers, while binding camera DMA buffers directly to Octal PSRAM.
  • Ingress Bottlenecks: High-frequency HTTP JPEG uploads blocked vision worker threads. We decoupled the system using a lockless/sequence-locked POSIX Shared Memory ring-buffer, achieving sub-millisecond IPC latency.

Accomplishments that we're proud of

  • Industrial Real-Time Latency: Achieving $57.5\text{ ms}$ end-to-end latency and sustained $15.7\text{ FPS}$ continuous streaming at VGA resolution purely on CPU.
  • Vectorized CV Speedup: Clocking a $20.97\times$ speedup ($0.0058\text{ ms}$) using branchless 8-lane AVX2 bitmasks for food spoilage analysis.
  • Compact Model Footprint: Quantizing YOLOv8s from $44.8\text{ MB}$ (FP32) down to $25.0\text{ MB}$ (INT8) with negligible confidence loss.
  • High-Throughput Batch Processing: Evaluating 30 trays in $6.81\text{ s}$ ($4.41\text{ trays/s}$) utilizing 12-core OpenMP parallelization.
  • Deterministic Edge Firmware: Zero heap memory leaks or fragmentation crashes over continuous multi-hour streaming runs.
  • Portability Guarantee: 100% compliant with cross-platform compilation rules without any non-portable -march=native shortcuts.

What we learned

  • Balancing Neural Networks & Classical SIMD CV: While deep learning (YOLO) is exceptional at answering what is on the plate, hand-optimized SIMD computer vision (HSV/Lab variance) remains vastly superior for answering how fresh it is in microseconds with zero GPU overhead.
  • Data Hygiene Trumps Model Tweaks: Perceptual dHash deduplication and strategic hard-negative injection produced far greater precision gains than simply scaling network depth.
  • Embedded Determinism is Invaluable: In mission-critical edge deployments, static memory allocation and static FreeRTOS primitives provide an unmatched level of operational stability.
  • Modern C++20 Ergonomics: Leveraging C++20 concepts, std::span, std::format, and spaceship operators enabled us to build blazing-fast vision pipelines that are maintainable, clean, and type-safe.

What's next for BawalVision

  • Edge NPU On-Device Acceleration: Porting our quantized models to run directly on the ESP32-S3's internal vector instructions (ESP-NN) and dedicated micro-NPUs (such as Kendryte K230 / Google Coral) for completely offline edge inference.
  • RGB-D Volumetric Portion Estimation: Integrating stereo vision or Time-of-Flight (ToF) depth cameras to measure food mound volume ($cm^3$) for automated macronutrient and caloric estimation.
  • ESP-NOW Central Kitchen Mesh: Establishing local wireless mesh networking between multiple inspection nodes across large institutional meal assembly lines.
  • Field Pilots in Regional Schools: Deploying physical BawalVision inspection stations to regional Indonesian central kitchens to safeguard national school meal distribution with automated transparency.

========

Built With

Share this project:

Updates

Submission history