• 

  running uname -m showing aarch64 on AWS Graviton.

• 

  Run Benchmark Suite to see the Markdown report generated live.

GraviTalk Console

💡 Inspiration

Deploying modern Large Language Models (LLMs) has historically been synonymous with high-cost, high-power GPU infrastructure. This creates a massive entry barrier for indie developers and small teams, while raising significant carbon footprint concerns. Furthermore, sending sensitive queries to proprietary cloud API endpoints compromises user privacy in highly regulated sectors like finance and healthcare.

I built GraviTalk Console to prove that highly responsive, private, and cost-effective AI is fully viable on CPU-only infrastructure. By combining 4-bit integer quantization with ARMv8-A hardware acceleration on energy-efficient cloud compute architectures like AWS Graviton, I aimed to create a private AI console that is sustainable, secure, and incredibly cheap to host.

🛠️ How I Built It

GraviTalk is engineered as a complete, lightweight, cloud-native AI console and developer API stack:

• The Inference Core: Powered by Ollama, running highly compiled C++ engines natively optimized for CPU. It utilizes 4-bit quantized GGUF weights of Phi-3-mini (3.8B parameters) or Qwen-2.5 (1.5B/7B), squeezing the active memory footprint down to ~2.2 GB.
• The Web Backend: Built using Python Flask. It interfaces with the local Ollama API and implements Server-Sent Events (SSE) to stream response tokens to the browser in real time with zero polling latency.
• The Interactive Frontend: A premium glassmorphic dark-mode single-page application built with vanilla JS and CSS. It features real-time dials visualizing CPU usage, system memory, model RSS memory, Time to First Token (TTFT), and token generation speed.
• Low-Resource Edge & Architecture Optimizations:
  1. Stateless Document Q&A: Reads text files (logs, code, CSVs) client-side using browser FileReader APIs and injects them directly into the context window, completely avoiding disk I/O bottlenecks on the server.
  2. Client-Side Speech-to-Text: Utilizes the browser's native Web Speech API (webkitSpeechRecognition) to dictate prompts. This offloads heavy voice transcription models (like Whisper) from the cloud CPU, keeping all server cores dedicated entirely to LLM inference.

. Functionality / Output

What the Project Does

GraviTalk installs a local AI host on an Arm64 server, serves a quantized model, and spins up a lightweight Flask server serving a premium Single-Page Application (SPA) dashboard.

The dashboard provides:

1. Interactive Chat Canvas: Real-time token streaming with micro-animations.
2. Dynamic Instrumentation Dial Panel: Immediately reads out Time to First Token (TTFT) and Tokens/Second (TPS) after each generation.
3. System Telemetry: Background polling of active host CPU and System RAM utilization.
4. On-Demand Benchmark Suite: Executes a standardized 3-trial runner directly from the UI, renders the comparison table on screen, and generates a copyable Markdown table for your submission.
5. Contextual Document Q&A (Stateless Upload): Supports uploading custom text-based documents (up to 50 KB, such as logs, source code, data sheets, CSVs, or JSON files) to feed directly into the prompt context for querying, summarization, and interactive analysis.
6. Voice Dictation (Speech-to-Text): Integrates the browser's native Web Speech API to transcribe spoken prompts in real-time. Because translation happens client-side, it places zero processing load on the CPU-only cloud host, keeping CPU cycles dedicated to LLM inference.

Final Output & Benchmarks

Below are representative benchmark results captured on a standard AWS Graviton t4g.medium instance running Phi-3-mini (3.8B parameter, 4-bit Quantized):

Performance Metrics Table (MIT-Licensed Stack on Arm64)

• Model Used: phi3:mini (4-bit Quantized GGUF)
• Host Architecture: Arm64 / aarch64 (AWS Graviton)
• Total System RAM: 4.00 GB

Metric	Trial 1 (Short Prompt)	Trial 2 (Medium Prompt)	Trial 3 (Long Prompt)	Average
Prompt Char Length	33	74	120	75
Tokens Generated	12	148	185	115
Time to First Token (TTFT)	0.452 s	0.420 s	0.435 s	0.436 s
Tokens / Second (Ollama Engine)	16.42 t/s	15.80 t/s	15.54 t/s	15.92 t/s
Peak Ollama Process RAM	2.23 GB	2.25 GB	2.26 GB	2.25 GB
Peak Total System RAM	2.85 GB	2.88 GB	2.90 GB	2.88 GB

🚧 Challenges I Faced

Initial Loading Timeout Bottlenecks: The first request sent to a cold model requires Ollama to copy ~2.2 GB of weights from storage into system RAM. On standard CPU-only instances, this initial transfer frequently exceeded default 30-second HTTP timeouts. I resolved this by implementing an active port-polling check and increasing request read limits to 180 seconds across the Python client, benchmark engine, and web backend.

CPU Math Speed Bottlenecks: Standard CPU floating-point calculations are traditionally too slow for LLM matrix transformations. I solved this by configuring the local engine to leverage hardware-specific ARM NEON vector instructions and ARMv8.2 dot-product extensions. By vectorizing matrix math operations directly on the Graviton hardware pipeline, I achieved stable speeds of ~16 tokens/sec and reduced the average latency (TTFT) to just 0.44 seconds.

🧠 What I Learned

I discovered that highly optimized, Small Language Models (SLMs) running on CPU-only ARM64 architectures are incredibly viable for production workloads. By carefully balancing client-side tasks (like local voice transcription and file parsing) with server-side tasks (like model inference), you can build highly scalable architectures on incredibly cost-effective compute instances. This project opened my eyes to the immediate future of localized, private, and green AI deployment in the cloud.

---

. Setup Instructions

Follow these step-by-step commands to clone, install, run, and validate GraviTalk on your own Arm64 device or AWS Graviton instance.

Prerequisites

• Linux/macOS: Python 3.8+ and curl.
• Windows: Python 3.8+ and Ollama for Windows installed.

Step 1: Clone the Repository

Clone the project directory to your target machine:

git clone https://github.com/billyphosly4/GraviTalk-Console.git


cd GraviTalk-Console

Step 2: Install Dependencies & Download Model

On Ubuntu / Debian Linux:

Run the automated installation script. This script installs Ollama (optimized for Arm64), starts the service, pulls the default model (phi3:mini), sets up a virtual environment, and installs Python packages:

chmod +x install_ollama.sh
./install_ollama.sh

On macOS (Apple Silicon) or other Linux distributions:

1. Install Ollama:
   • For macOS: Download and run the desktop app from Ollama's website or run brew install ollama. Make sure to launch the application.
   • For other Linux distributions: Install Ollama using: bash curl -fsSL https://ollama.com/install.sh | sh
2. Download Model: bash ollama pull phi3:mini
3. Configure Python Virtual Environment: Run the setup script which will automatically skip package updates/system installs but configure your Python environment: bash chmod +x install_ollama.sh ./install_ollama.sh (Alternatively, configure it manually: python3 -m venv venv && source venv/bin/activate && pip install -r requirements.txt)

On Windows:

1. Double-click the setup_windows.bat file (or run it in Command Prompt). This will configure your virtual environment and install the required dependencies from requirements.txt.
2. Open Command Prompt and pull the default model: cmd ollama pull phi3:mini

Step 3: Run the Web Dashboard

On Linux / macOS:

Activate the environment and start the web server:

source venv/bin/activate
python3 app.py

On Windows:

Activate the environment and start the web server:

call venv\Scripts\activate.bat
python app.py

By default, the server runs on port 5000 (http://localhost:5000).

Step 4: Run CLI Client or Performance Benchmark (Optional)

If you prefer running terminal-based clients or validating performance without starting the web dashboard:

• Interactive CLI Chat Client: Activate the environment and run: bash python3 chat_client.py
• Standardized Performance Benchmark: Activate the environment and run: bash python3 benchmark_llm.py

Step 5: Secure Access (SSH Port Forwarding)

If you are running the dashboard on a remote cloud VPS (like AWS Graviton EC2), you can access the dashboard securely in your local browser without opening public ports. Open a new terminal on your local machine and run:

ssh -i arm64-key.pem -L 5000:localhost:5000 ubuntu@<YOUR_INSTANCE_IP>

Now, simply navigate to http://localhost:5000 on your local computer to access the GraviTalk Console!

---

📱 Mobile Device Support (Cross-Platform / Mobile AI)

GraviTalk's architecture supports mobile deployment in two ways:

1. Client-Side Web Access

The dashboard is designed with mobile-first responsive CSS grids. When the server is hosted on a cloud instance, you can access the dashboard on any iOS or Android device by opening Safari, Chrome, or Firefox and navigating to: http://<YOUR_INSTANCE_IP>:5000 (Ensure port 5000 is open in your cloud security group settings).

2. Native On-Device Inference (Mobile AI Track Potential)

Because smartphones run on powerful ARM64 system-on-chips (Apple Silicon, Snapdragon, MediaTek), the underlying precompiled inference engine can be ported directly:

• Android: Can run local CPU-optimized GGUFs natively using Llama.cpp inside Termux or using third-party wrappers like PocketPal AI.
• iOS: Can leverage LLM Farm or PocketPal AI to run GGUFs locally, utilizing Apple's unified memory architecture and Neural Engine.

Built With

• 24-pull-requests
• arm64
• aws-graviton
• css3
• flask
• html5
• javascript
• llama.cpp
• ollama
• psutil
• python