BZML

High-Performance Generative Digital Asset Management Engine

Comment

Inspiration

The generative AI explosion has created a new problem for enterprises and media agencies: Asset Sprawl and Provenance Tracking. When a company generates thousands of images, videos, and voiceovers, tracking how an asset was made, which model generated it, and what prompt was used becomes nearly impossible.

Most existing AI tools are fragile, single-threaded Python scripts that lack infrastructure-grade reliability. We wanted to build something completely different: a High-Assurance, Enterprise-Grade Generative Digital Asset Management (DAM) engine. We were inspired to combine the raw, uncompromising safety of Rust, the immutable and cost-effective storage of Backblaze B2, and the multi-model orchestration power of the Backblaze Genblaze SDK.

What it does

BZML (GenDAM) is an ultra-fast media library and AI generation engine that gives users a sub-millisecond interface to search, manage, and transform digital assets.

  • Lightning Fast Search: Finds assets in < 0.8ms using an L1 RAM cache.
  • Hybrid AI Transformations: Users can select assets and apply batch transformations (e.g., 16:9 Outpainting, 5s Video Teasers, AI Voiceovers).
  • Smart Intent Routing: BZML dynamically routes requests. If Cloud API keys are present, it spins up an isolated Python worker using the Genblaze SDK to orchestrate OpenAI/ElevenLabs/Runway. If running entirely offline, it uses Native Rust Local AI (Candle + Tokenizers) for 1-step CPU diffusion.
  • Immutable Provenance: Every time an asset is generated, BZML automatically creates a .lineage.json sidecar audit log containing the SHA-256 digest, latency, prompt, and AI provider. Both the media and the sidecar are pushed directly to Backblaze B2 Cloud Storage.
  • Visual Lineage Inspector: Features an interactive HTML5 Canvas modal to visualize the parent-child relationships of generated assets.

How we built it

We built BZML focusing heavily on Production Readiness and System Architecture:

  • The Core Engine (Control Plane): Written in 100% Safe Rust (#![forbid(unsafe_code)]) using Axum and Tokio. We used a dynamic batching Actor pattern with bounded MPSC channels to protect the system from memory exhaustion during traffic spikes (Backpressure).
  • Multi-Tier Caching: We implemented quick_cache (L1 RAM) backed by a fjall LSM-Tree database (L2 Disk) that warms up automatically on boot.
  • Backblaze B2 S3 API: We deeply integrated the aws-sdk-s3 crate, utilizing ListObjectsV2 pagination for bucket syncing, HeadObject for fast metadata retrieval, and time-limited Presigned URLs for secure asset viewing.
  • Python Genblaze Sandbox: We kept the Python environment strictly isolated. It acts solely as an execution sandbox for the official genblaze_core SDK, ensuring that even if an AI model runs out of memory, the Rust web server stays at 100% uptime.
  • DevOps & IaC: The entire system is containerized via a multi-stage Docker build (~60MB footprint), automatically provisioned using Terraform (to create the B2 buckets and scoped keys), and validated via GitHub Actions CI/CD.

Challenges we ran into

  1. The Python/Rust Boundary: Mixing a Python ML ecosystem with Rust's strict safety was challenging. We solved this by using Zero-IPC concepts where possible and keeping Python strictly as a subprocess worker for the Genblaze SDK, while shifting all local Edge AI fallbacks to pure Rust.
  2. Sub-Millisecond Search Bottlenecks: Initially, filtering through thousands of assets by scanning the L2 LSM-Tree disk database caused UI stuttering. We solved this by implementing an on-boot memory warmup into our L1 cache, dropping search latency to under 1 millisecond.
  3. CI/CD Out-of-Memory Errors: Compiling heavy Rust AI libraries in GitHub Actions caused OOM crashes. We engineered a solution by switching compiler profiles to lto = "thin" and codegen-units = 16, which slashed compilation time by 80% and reduced peak RAM usage by 4x.

Accomplishments that we're proud of

  • Zero Panic Invariants: The Rust core strictly uses Result<T, E> for domain logic without a single unwrap() in the hot path.
  • The Smart Intent Router: Our engine analyzes prompt keywords in microseconds and dynamically decides whether to route the task to a photorealism model, a complex-prompt model, or a fast 1-step CPU model.
  • Cross-Platform Readiness: We successfully bundled the Axum server into a Tauri v2 wrapper, allowing BZML to be distributed as a standalone Native Desktop application for Windows, macOS, and Linux.
  • The Final Binary Size: Stripped and optimized, the entire core engine (including an embedded web server, database, and AVX-512 SIMD vectorization) compiles down to just ~21 MB.

What we learned

  • Deep S3 API Mechanics: We learned how to efficiently utilize Backblaze B2 through the S3-compatible API, specifically mastering continuation tokens for pagination and the nuances of Presigned GET requests.
  • Advanced AI Orchestration: We discovered the power of the Genblaze SDK. Building modular Pipeline and Step definitions allowed us to abstract away provider-specific boilerplate and focus purely on the application's business logic.
  • Rust Build Optimization: We learned how to significantly optimize LLVM compilation times and binary sizes for production Docker containers.

What's next for BZML

  1. Compliance & Object Lock: We plan to implement Backblaze B2 Governance Mode (Object Lock) specifically for the .lineage.json sidecars, making the AI audit trails mathematically tamper-proof for enterprise compliance.
  2. Native GPU Acceleration in Tauri: Expanding the desktop application to natively interface with Apple Metal and NVIDIA CUDA for localized generative workflows.
  3. More Genblaze Providers: Adding support for Genblaze's Agent Loops and Streaming Progress features to display real-time generation feedback in the HTMX interface.

Built With

  • aws-sdk-s3
  • axum
  • backblaze
  • backblaze-b2
  • docker
  • elevenlabs
  • genblaze
  • github-actions
  • htmx
  • huggingface
  • openai
  • python
  • pytorch
  • rust
  • stable-diffusion
  • tailwindcss
  • tauri
  • terraform
  • tokio
Share this project:

Updates