GraviTalk Console

Private cloud AI console & API optimized for ARM64 Graviton CPUs. Runs quantized LLMs on CPU-only servers with 80% cost savings, stateless doc analysis, and browser-side voice dictation.

Comment

posted an update —

Evolving GraviTalk Console: From CLI Prototype to a Fully Featured CPU-Optimized Web Console & API Stack! I'm excited to share a major update on how GraviTalk Console has evolved! What started as a lightweight terminal benchmark script is now a fully featured, private Cloud AI console and API service natively optimized to run on CPU-only ARM64 servers (such as AWS Graviton instances).

Here are the key enhancements and features that have been built and optimized:

  1. Premium Glassmorphic Web Dashboard I designed and built a modern Single-Page Application (SPA) dashboard in templates/index.html utilizing a glassmorphic dark theme (using Google Font families Inter and Orbitron).

Real-time Streaming: Response tokens stream into the chat canvas with smooth, responsive micro-animations. Telemetry Gauge Panel: Includes live-updating gauges for Host CPU %, Host RAM usage, and Model RSS RAM usage, alongside instant printouts of Time to First Token (TTFT) and Tokens/Second (TPS) after each generation.

  1. Stateless Document Q&A (Upload Context) To support deep interactive analysis, I added a stateless document attachment feature:

Users can attach text-based files (logs, code, CSVs, JSON, data sheets) under 50 KB. The file is read entirely client-side using the HTML5 FileReader API and injected dynamically into the context window, preventing unnecessary uploads or heavy vector database indexing.

  1. Zero-Overhead Voice Dictation (Speech-to-Text) To keep the host CPU cycles strictly dedicated to raw model inference, I avoided heavy server-side speech models (like Whisper) and integrated the browser's native Web Speech API (webkitSpeechRecognition). This allows users to dictate prompts hands-free with zero server CPU overhead.

  2. Optimized On-Demand Benchmarking Running full benchmarks on CPU can be slow and run into HTTP timeout bottlenecks. I optimized the benchmarking suite by limiting model predictions (num_predict: 50) and shortening testing prompts. The benchmark now runs a full 3-trial test directly from the UI in under 12 seconds, auto-rendering a comparison table and outputting a copy-pasteable Markdown table.

  3. Cross-Platform Automated Setup I created two direct installers to make testing seamless:

Linux/macOS ( install_ollama.sh ): Installs Ollama, polls ports to ensure availability, pulls phi3:mini (or falls back to qwen2.5:1.5b on low-memory instances), and configures the virtual environment. Windows ( setup_windows.bat ): Automatically bootstraps the virtual environment and packages dependencies with a single double-click.

Log in or sign up for Devpost to join the conversation.