SplitSys (Splittable Systolic Array)

Inspiration

LLM accelerators are built around big systolic arrays. During decode, a model generates one token at a time, so the matrix multiplies are one row tall (M = 1). An 8-row array can then keep at most 1 of its 8 rows busy. Some LLM-specific chip makers have publicly described splittable systolic arrays as a response. I wanted to measure that tradeoff myself at small scale, with hardware counters rather than slides, and see whether a simple cycle model could predict every counter.

What it does

SplitSys is a GEMM accelerator on a RealDigital Urbana board (AMD Spartan-7 XC7S50).

• Two shapes from one array. It has 64 int8 processing elements, output-stationary, one DSP slice each. Per job, the array runs either as one 8×8 tile (FULL) or, by muxing at row 4, as a 4×16 tile (STRIP). Switching modes needs no reconfiguration.
• Bit-exact checking. A Rust host sends LLM-shaped matmuls over UART and checks every output bit-exactly against a Rust golden model.
• Counters and a model. Twelve hardware counters report cycles per phase, useful MACs per quadrant, and tiles. A cycle model predicts every one of them, and can pick the faster shape per job.
• A real language model on the board. Every linear layer of stories260K, a small public model, runs on the FPGA.
• A live dashboard shows the PE grid lighting up, model-vs-measured utilization, and the language model generating text.

How we built it

• Spec first. One spec (spec/spec.toml) generates both the SystemVerilog package and the Rust constants, so RTL and software cannot drift apart. Every ambiguity was resolved in a numbered decision log.
• SystemVerilog RTL: PE, skew units, array with FULL/STRIP modes, BRAM buffers, sequencer, requantizer, counters, UART packet layer, self-test, and board I/O.
• Verification: cocotb + Verilator unit suites with expected values only from Rust, plus a system simulation. In co-simulation, the real host binary talks to the simulated RTL over a TCP-to-UART bridge.
• Rust: a bit-exact golden model (GEMM, requantization, packing, CRC-8, Fletcher-32), a test-vector generator, a cycle model that predicts all 12 counters, and the host (UART driver, benchmark runner, dashboard server).
• Builds: Vivado 2025.2.1 builds, timing-clean at 100 MHz. A second build with STRIP compiled out measures what the split costs.

Challenges we ran into

• Creation of the actual design, and deciding tests against the FULL vs STRIP models.
• A test harness that lied. Under make -j, cocotb could skip a simulation and still exit 0. I caught it, fixed it so every run must produce fresh results, and re-ran every suite.
• No datapath surprise on hardware. The first bit-exact hardware run matched the model on every counter, so there was nothing to reproduce in simulation. The surprises were board I/O: LEDs glowing from a pull-up setting copied out of the vendor constraints file, and display digits in reverse order. I read off on the board which display digit is which and fixed the RTL default. Then I rebuilt the system simulation's display check from that board-confirmed rule: it fails 7 of 8 cases on the old RTL and passes on the fix. On the board, the fixed build reads in the right order, and the LEDs are dark.

Accomplishments that we're proud of

• On the board: 773 GEMMs, a measured 2.02–2.07× decode utilization gain from base FULL model, and zero stall cycles.
• A real language model with every linear layer on my array, bit-exact on 2,146 of 2,146 GEMMs [HW].

What we learned

• Utilization is set by mapping first, overhead second. At M = 1, the mapping caps FULL at 12.5% and STRIP at 25% . Fixed per-tile setup, flush, and drain cycles then bring the model down to 7.0% and 14.4%.
• Flexibility costs bandwidth. A shorter, wider array needs more operand bytes per MAC: +25% here [S]. The cost of a feature hides in its neighbours. Most of the split's area was the output path that STRIP's 16-wide tiles need, not the muxes.
• Shape beats mode. STRIP wins on most shapes, but its 4×16 tile padding made it slower on 31 of 97 random shapes. A good mode selector has to count tiles.

What's next for SplitSys

• Making four independent 4×4 jobs, for per-head attention, which would also move attention onto the physical board.
• DDR3 weight streaming, so the language model's weights stay next to the array and memory-bound decode can be studied. There FULL's lower bandwidth need could make it the right choice, and auto mode would need a bandwidth term.

Built With

• claude
• cocotb
• matplotlib
• python
• rust
• spartan-7
• systemverilog
• verilator
• vivado