LLM Inference on Edge Hardware

Efficient inference for Large Language Models (LLMs) via FPGA acceleration

Comment

Inspiration

Modern AI adoption has outpaced hardware efficiency. Most computers still rely on the Von Neumann architecture, where 90% of energy is wasted moving data between memory and the CPU, rather than doing the actual math. We realized that while LLMs are eating the world, the hardware running them is stuck in the past. We didn't want to just run an API; we wanted to solve the "Dark Silicon" problem by building a custom hardware accelerator that proves AI can be both accessible and energy-efficient.

What it does

Q-Tensor is a heterogeneous inference engine. It performs "brain surgery" on a running GPT-2 model, surgically removing its most computationally expensive organ (the Matrix Multiplication Kernel) and forcing it to run on a custom-built FPGA accelerator. By offloading these dense calculations to a hardware-based Systolic Array, we bypass the operating system entirely, executing neural network layers with bare-metal efficiency.

How we built it

We adopted a hardware-software co-design approach:

The Silicon (Verilog): We physically programmed an FPGA (Field Programmable Gate Array) to instantiate a Systolic Array. Unlike a CPU that fetches-decodes-executes, our array pumps data through a grid of Processing Elements (PEs) like an assembly line, maximizing arithmetic intensity.

The Driver (Python): We hacked picoGPT (a NumPy inference engine) to intercept linear layer calculations.

The Optimization (Quantization): We implemented a custom Int8 Quantization scheme, compressing 32-bit floating-point weights into 8-bit integers. This allowed us to fit 4x more data into our bandwidth budget and reduce the energy cost per operation by over 10x.

Challenges we ran into

The Interface Bottleneck: Bridging the gap between high-level Python and low-level Verilog via UART was painful. We had to write a custom serial driver to handle packet fragmentation and ensure bit-perfect data transfer.

Timing Closure: Getting our Systolic Array to tick synchronously without signal propagation delays (setup/hold violations) required careful pipelining of the accumulator logic.

Quantization Accuracy: converting continuous FP32 weights to discrete Int8 bins without turning the LLM's output into gibberish required fine-tuning our scaling factors.

Accomplishments that we're proud of

Functional Hybrid Inference: We successfully generated coherent text where the CPU handled the logic and the FPGA handled the math.

Massive Efficiency Gains: By switching to Int8, we achieved a 4x reduction in memory bandwidth requirements.

Architecture Validation: We proved that our custom Systolic Array executes operations with ~10x fewer clock cycles per matrix block compared to a standard iterative CPU loop, validating the same architectural principles used in Google's TPUs.

What we learned

Data Movement > Compute: We learned the hard way that moving data is expensive (energy and time), while doing math is cheap. This shifted our mindset from "optimizing code" to "optimizing data flow."

The Reality of Hardware: Verilog is not code; it is a circuit description. We gained a deep appreciation for clock domains, signal integrity, and the physical limitations of silicon.

Full Stack Engineering: True "Full Stack" isn't React-to-SQL; it's Python-to-Transistor.

What's next for LLM Inference on Edge Hardware

Phase 1: Breaking the Bottleneck (Connectivity)

  • Current State: Constrained by UART (115.2 kbps).
  • The Upgrade: Implement Peripheral Component Interconnect Express (PCIe) with DMA (Direct Memory Access).
  • Impact: Increases data transfer speeds from kilobits to Gigabits per second, saturating the compute array and making real-time inference possible.

Phase 2: Scaling the Core (Architecture)

  • Current State: 4x4 Systolic Array (Proof of Concept).
  • The Upgrade: Scale to 64x64 or 128x128 Arrays.
  • Impact: Exponentially increases computational density. By utilizing the full DSP slices of the FPGA (currently using <5%), we can parallelize entire attention heads.

Phase 3: "Net Zero" Autonomy (Sustainability)

  • Current State: Tethered to a host Mac.
  • The Upgrade: Hostless Inference.
  • Store the entire GPT-2 model in the FPGA's on-board DDR3/DDR4 RAM.
  • Run the control logic on the FPGA's soft-core CPU (Nios II / MicroBlaze).
  • The Vision: A standalone, solar-powered "AI Brick" that performs inference for pennies of electricity per year.

Built With

Share this project:

Updates

Submission history