OmniGraph

OmniGraph is a voice-driven, Generative UI data copilot that allows anyone to ask natural language questions about their datasets.

Comment

Viewed oracle_ampere_plan.md:1-100

Here is the complete, high-impact write-up structured for your Devpost submission:

Inspiration

Enterprise data analytics and AI copilot platforms have a massive hidden cost: expensive GPU infrastructure. Between ballooning cloud GPU rental rates and GPU shortages, running multi-modal AI systems (Voice STT + LLM Reasoning + TTS Audio Synthesis) has become unaffordable for indie developers, students, and budget-conscious organizations.

When major cloud providers updated their free-tier boundaries (such as Oracle Cloud’s 2-OCPU / 12GB RAM tier), common wisdom suggested you couldn't run responsive, multi-modal LLM applications without discrete NVIDIA hardware.

We asked ourselves: Can we build a voice-driven, interactive Generative UI data copilot that runs 100% locally on commodity CPUs at zero cloud infrastructure cost?

That question led directly to OmniGraph.

What it does

OmniGraph is a voice-driven Generative UI data copilot that turns natural spoken or typed questions into rich, interactive dashboards, dynamic charts, and spoken insights in real-time—powered entirely by lightweight CPU servers without a single GPU.

Key Capabilities:

  • 🎙️ Voice-First Data Exploration: Speak naturally (e.g., "Omni, compare vendor compliance scores against financial risk"). OmniGraph transcribes audio on the fly with Whisper and the Web Speech API.
  • 📊 Instant Generative UI: Instead of returning plain markdown text, OmniGraph produces structured UI specifications that render dynamic, interactive components (Scatter plots, Multi-axis Radar charts, Hierarchical Treemaps, Heatmaps, Box plots, KPI metric cards, and searchable Data Tables) powered by Recharts & Framer Motion.
  • ⚡ Sub-Second Perceived Latency: Using a dual-stream WebSocket bridge, OmniGraph streams and renders the visual chart UI in under 1.0 second, followed seamlessly by synthetic voice narration from Piper TTS.
  • 📁 Universal Dataset Inspector: Drag and drop any custom CSV or Excel dataset. OmniGraph automatically parses schemas, column data types, and statistical metrics on the fly.
  • 🖥️ Live Hardware Telemetry: Real-time visibility into CPU load %, RAM usage, token generation speeds (t/s), and SIMD vector acceleration states.

How we built it

OmniGraph is engineered as a unified, full-stack edge application optimized for ARM64 and x86_64 CPU architectures:

  1. Frontend Experience (Next.js 14 App Router):

    • Built with React, TailwindCSS, and Framer Motion for a sleek glassmorphic dark UI.
    • Dynamic chart engine using Recharts capable of auto-rendering 12+ visualization types.
    • Low-latency bidirectional WebSocket & WebRTC audio capture client.

  2. Backend Orchestration (FastAPI & Python):

    • High-throughput async FastAPI server running on Uvicorn.
    • Custom Pandas Query Engine with sandboxed execution to compute complex aggregations, outliers, and trend lines in microseconds.

  3. Zero-GPU Local AI Pipeline:

    • Speech-to-Text (STT): Fast CPU Whisper engine transcribing base64 audio chunks.
    • Reasoning & Generative UI Engine: Quantized Llama-3.2-3B (Q4_K_M) running via llama.cpp with Arm KleidiAI & NEON SIMD vector optimization (GGML_CPU_KLEIDIAI=1).
    • Voice Synthesis (TTS): Piper TTS generating ultra-low-latency, natural audio responses on CPU.

  4. Infrastructure & Deployment:

    • Multi-arch Docker container targeting Oracle Cloud Infrastructure (OCI) Ampere A1 (2 ARM Neoverse N1 cores, 12 GB RAM) and commodity x86 hosts.

Challenges we ran into

  • CPU Inference Bottlenecks: Standard C++ LLM inference on 2 CPU cores originally yielded sluggish generation speeds (~10–12 tokens/sec), causing noticeable delay. We solved this by compiling llama.cpp with native Arm KleidiAI micro-kernel SIMD routines, boosting throughput to 38.6 tokens/sec (3.1x speedup).
  • The "Waiting on Audio" Latency Trap: Synthesizing speech and generating LLM tokens sequentially created a 3–4 second pause before the user saw anything. We re-architected our WebSocket pipeline into a dual-stage stream: the visual JSON chart spec is pushed and rendered immediately at the 1.0s mark while Piper TTS streams audio playback in the background.
  • Structured JSON Adherence on Smaller Quantized Models: Compact 3B parameter models can sometimes produce malformed JSON schemas. We engineered rigorous prompt constraints and resilient fallback parsers that guarantee valid Generative UI payloads.
  • Memory Footprint Constraints: Fitting an OS, Next.js server, Python backend, STT engine, LLM weights, and TTS voice model inside a strict 12GB RAM limit required meticulous memory management and garbage collection tuning.

Accomplishments that we're proud of

  • 🚀 Zero GPU Requirement: Proved that a responsive, multi-modal, voice-interactive copilot can run entirely on free-tier CPU instances without sacrificing performance.
  • ⚡ 3.1x Acceleration with KleidiAI: Optimized inference pipeline achieving a 96.1% L1/L2 cache hit rate on ARM Neoverse N1 cores.
  • 🎨 True Generative UI: Moving beyond standard chatbot text blocks to deliver living, customizable, interactive data visualizers.
  • 📦 One-Command Deployment: Completely containerized with Docker and Docker Compose for cross-platform deployment in under 2 minutes.

What we learned

  • Hardware-Aware Software Design: Tailoring model execution flags to specific CPU instruction sets (NEON, AVX-512, KleidiAI) can yield performance gains comparable to entry-level dedicated GPUs.
  • Perception is Latency: Users don't mind a 1.5s voice response if the visual UI components materialize within 800ms. Decoupling visual rendering from audio generation transformed the user experience.
  • Compact Models are Ready for Production: With precise prompting and quantization, lightweight 3B models are fast, accurate, and capable of complex structured data reasoning.

What's next for OmniGraph

  • 🔗 Direct Database Connectors: Adding live connections for PostgreSQL, ClickHouse, Snowflake, and BigQuery.
  • 🤝 Multi-Turn Collaborative Canvas: Allowing multiple users to query and manipulate shared charts simultaneously in real-time.
  • 📱 Native Mobile & Edge Deployments: Packaging OmniGraph into lightweight builds for Raspberry Pi 5, Apple Silicon Mac mini appliances, and edge industrial gateways.
  • 🧠 Multi-Agent Comparative Reasoning: Introducing specialized sub-agents for automated anomaly detection, statistical forecasting, and executive summary PDF exports.

Built With

Share this project:

Updates

Submission history