dspark-edge

Data flow of dspark-edge with client and server
Speculative decoder and predictions running on QNX

Inspiration

DeepSeek's speculative decoding strategy, DSpark, requires a very low amount of information for a speculator to make predictions. Being familiar with mostly transformer architectures, due to interacting primarily with LLMs, we were thinking of practical applications of LLMs used on edge devices.

What it does

With a patched version of llama.cpp with support for DSpark, we added a gRPC shim inside of llama.cpp that talks to another client to do speculative decoding. Essentially:

Server, main LLM, sends hidden state calculated from prompt to Client
Client (Raspberry Pi running QNX) uses the hidden state to make a prediction
Client displays prediction on screen
Client sends prediction to LLM on Server to verify
Server then accepts or denies the prediction
Server calculates hidden state and sends layers to repeat the process

How we built it

First, we created the server+client making golden tests compared to DSpark without the gRPC layer. Using both Qwen and experimental Gemma branch for testing.

Then, we created a port to QNX, testing that it worked distributed between a Raspberry Pi and desktop computer.

We used the following technologies:

C++
Python
Claude+Kimi+GLM for coding.

Challenges we ran into

QNX's version of llama.cpp required a special patched version, so we had to go through QNX's ports repo to find the appropriate PR with the necessary patches.
We wanted to integrate this into a general assistant system with tool calling, but we ran out of time for that.

Accomplishments that we're proud of

Applying a relatively new LLM technology in a unique way to see how inference can be done with a potential distributed load / potentially faster early prediction.
Patching llama.cpp for new unique functionality for our own special use case

What we learned

We learned more about how DSpark worked and how it predicts using only the hidden state layers.
We learned how to use the QNX compiler tools as well as how to search for ports / examples for existing projects.

What's next for dspark-edge

Unfortunately, this is a project that with the current hardware, is not very practical. It was more of an experiment to see how we can distribute loads of LLM between devices, especially if one is interested in earlier predictions on the edge (although, they may be incorrect)