NOVA (Networked On-demand Virtual Acceleration))

Turn idle GPUs into instant, distributed compute.

Comment

Inspiration

Most compute infrastructure assumes homogeneous hardware: CUDA clusters, cloud GPUs, or machines managed by one provider. Real-world hardware is the opposite. I might have an Apple Silicon Mac running Metal, while someone else has an NVIDIA GPU running CUDA or AMD hardware using ROCm.

I wanted to build infrastructure that could turn every individual computer into a usable compute resource, regardless of vendor, operating system, or runtime.

That became NOVA: a universal heterogeneous compute platform where applications can be built on top of a shared pool of machines. NOVA discovers available nodes, benchmarks their computational performance, distributes work according to their capabilities, and automatically recovers when machines disconnect.

Text-to-image generation is only one example of an application that can be built on top of NOVA.

For the demo, I use something judges can understand immediately: 64 prompts → 64 SD-Turbo images → multiple heterogeneous GPUs → one live gallery. If I disconnect any worker while the job is running, the remaining nodes automatically take over and finish its work.

What it does

NOVA — Networked On-demand Virtual Acceleration — is a coordinator-led peer-to-peer distributed compute runtime that turns independent computers into a shared heterogeneous compute cluster.

I run the same NOVA client on every machine:

nova start

Each node detects and advertises its available compute backend — CUDA, Metal, ROCm, or another supported runtime — and joins the cluster.

Before scheduling workloads, I benchmark each node to estimate its computational performance. NOVA then combines those benchmark results with observed execution times to determine how jobs should be sharded across the cluster.

Instead of splitting a job evenly, faster machines receive proportionally more work while slower machines receive less. This prevents the overall job from being bottlenecked by the weakest node and maximizes cluster utilization.

Applications can then be built on top of this infrastructure.

My demo application is distributed text-to-image generation. I submit a gallery of prompts, and workers dynamically request tasks:

WORK_REQUEST → TASK_OFFER

Each GPU generates a 512×512 SD-Turbo image and uploads the result to the job owner. The gallery fills live, with every output tagged by the backend that produced it:

CUDA / Metal / ROCm

NOVA continuously updates its estimate of each node's performance using real task execution times, allowing the scheduler to rebalance work while the job is running.

I also designed NOVA around redundancy.

A task is never permanently tied to one computer. Instead, tasks are assigned using leases. If a worker disconnects, crashes, or runs out of memory, its unfinished work is returned to the cluster and another available machine can complete it.

I use fenced lease generations so stale results from disconnected nodes cannot overwrite newer results after a task has been reassigned.

This means the cluster does not depend on any individual machine.

How I built it

I built NOVA as a general distributed-compute infrastructure layer rather than a Stable Diffusion-specific system.

The core stack uses Python, FastAPI, PyTorch, and a real-time dashboard.

Control plane

I use Pear / Hyperswarm for peer discovery and communication, with TCP as a fallback.

The control plane handles:

  • node discovery
  • capability advertisement
  • worker availability
  • job state
  • task leasing
  • heartbeats
  • persistent node identity through .nova/node.json

Performance-aware scheduling

I benchmark every node before using it for scheduling.

The initial benchmark gives NOVA an estimate of the node's computational capability. I then refine that estimate using actual task execution times as the cluster runs.

For the SD-Turbo demo, I separate first-image compilation time from steady-state generation time. The scheduler uses the second image and subsequent generations to estimate real throughput instead of incorrectly ranking nodes based on cold-start latency.

I maintain an exponential moving average of generation time after each task.

NOVA then uses these measurements to shard workloads dynamically.

A high-performance GPU can process significantly more tasks than a slower GPU instead of every node receiving the same amount of work.

The scheduler can also decide not to assign a remaining task to a slow worker if a faster worker is expected to become available and complete it sooner. This prevents straggler nodes from increasing the total wall-clock time.

Fault tolerance

I designed task execution around fenced leases rather than permanent ownership.

Each task tracks values such as:

lease_gen
attempt_count
max_attempts

When a node disappears, its unfinished lease becomes invalid and the task returns to the global work queue.

Another worker can immediately take over.

If the original worker later sends a result, NOVA checks its lease_gen. If the generation is stale, the result is rejected.

OOMs and other execution failures also cause the task to be requeued instead of failing the entire job.

The important property is that any individual node can disappear and the overall job can still be completed by the remaining cluster.

Application layer

I intentionally designed NOVA so the infrastructure itself is workload-agnostic.

An application only needs to define how a larger job can be divided into tasks, how those tasks execute on supported backends, and how results are returned.

My first application is distributed SD-Turbo image generation, but the same infrastructure could be used for workloads such as:

  • video rendering
  • model inference
  • simulation
  • batch data processing
  • compilation
  • scientific computation
  • distributed evaluation

Stable Diffusion is therefore not the project itself. It is a visual demonstration of the distributed infrastructure underneath it.

Challenges I ran into

The biggest problem was connecting heterogeneous machines across LAN and WAN environments.

My Mac might be on an RFC1918 local network while a CUDA worker is running remotely on RunPod. Pear does not automatically solve that NAT boundary.

For example, advertising:

http://10.x.x.x:8080

works perfectly on the LAN but is unreachable from a cloud worker.

I ended up separating the control plane from the data plane. Control traffic can travel through an ngrok TCP tunnel, while generated assets are uploaded through a public HTTPS endpoint.

Another problem was coordinator routing.

Initially, each nova start process could send HELLO and WORK_REQUEST messages to its own local state. This caused situations where the CUDA worker was effectively asking itself for work while the Metal node completed the entire batch.

I fixed this by broadcasting work requests across the control plane and maintaining authoritative job state at the coordinator.

Liveness detection also caused issues.

I originally used ping latency as part of the online/offline decision. That worked on a LAN but failed across ngrok because RTT could exceed 1.5 seconds. A perfectly healthy CUDA worker could therefore be marked offline before receiving a task.

I moved failure detection to heartbeats and actual connection state instead.

Another challenge was performance measurement.

A cold H100 can spend 15–20 seconds compiling its first image while still having vastly more steady-state throughput than an Apple GPU. Ranking machines by first-image latency therefore gave completely misleading scheduling decisions.

I separated compile time from generation time and use warm generation performance for scheduling.

Finally, I had to deal with stragglers. Giving every node exactly one task whenever it was free sounds fair, but it can actually increase total completion time. If a slow Mac accepts the final image while a much faster GPU becomes available moments later, the entire cluster waits for the Mac.

I changed the scheduler to make assignments based on predicted completion time instead of simple availability.

Accomplishments that I'm proud of

I built NOVA so that heterogeneous machines can participate in the same workload without pretending they are identical.

The demo can run CUDA and Metal workers inside the same distributed job, with each machine contributing according to its measured performance.

I also implemented automatic failover. If I Ctrl+C a worker during generation, its unfinished task is requeued and another node completes it.

The system fails closed: a worker cannot claim a backend it does not actually have, stale leases cannot overwrite completed work, and failed or OOM tasks return to the scheduler.

I also made the dashboard expose what is actually happening rather than hiding the infrastructure. It shows workers, backends, task ownership, contribution counts, performance measurements, compile time, generation time, failures, and reassignment events.

Most importantly, the project is not tied to SD-Turbo. The image gallery demonstrates the scheduler and distributed runtime, but NOVA itself is infrastructure for building arbitrary distributed applications across heterogeneous consumer hardware.

What I learned

The biggest thing I learned is that heterogeneous distributed computing is less about making every machine look identical and more about understanding their differences.

RTT is a measurement, not automatically a failure.

A private HTTP address that works locally may be completely unreachable from a cloud node.

Two processes running on one physical machine do not represent two independent compute resources.

First-image latency is not the same thing as steady-state throughput.

And equal task distribution is not necessarily efficient task distribution.

For a heterogeneous cluster to work well, the scheduler has to understand the actual performance, availability, and failure state of every node instead of assuming all hardware behaves the same way.

What's next for NOVA

I want to push NOVA further toward a general platform for turning arbitrary computers into shared computational resources.

The next step is separating:

nova dashboard
nova worker

so the machine displaying the dashboard does not also have to participate in computation.

I also want to improve WAN discovery so nodes can join without manually copying tunnel URLs.

The scheduler can be expanded beyond the current workload model with different policies such as maximum throughput, quotas, cost-aware scheduling, memory-aware scheduling, and hardware-specific task placement.

Most importantly, I want to make the application interface generic enough that developers can build different workloads on NOVA without modifying the underlying networking, discovery, scheduling, benchmarking, or fault-tolerance infrastructure.

The long-term idea is simple:

Any computer can become a resource. NOVA figures out what it can do, how fast it can do it, and how to use it efficiently as part of a larger cluster.

Built With

Share this project:

Updates

Submission history