Inspiration
Reducing GPU memory is only useful if the required result stays correct. Pairwise operations can create an N × M distance tensor even when an application needs only one nearest-neighbor index or one radius count per query. A full distance matrix has a different output contract and cannot be replaced by those reductions. Headroom makes that distinction explicit before it recommends a bounded repair.
What it does
The workbench lets a user choose nearest-neighbor lookup, radius counting, or a full distance matrix, then set memory and latency limits. A separate attention comparison measures full attention, Python tiling, and PyTorch SDPA. The demo presents recorded GPU measurements separately from live Nemotron strategy suggestions. Nemotron can select only a named strategy; it does not write or execute code, and an independent validator makes the acceptance decision.
How we built it
The product uses Next.js, React, and TypeScript for the interface, plus a Python and PyTorch runner for fixed, bounded workloads. The local CUDA runner checks outputs, measures latency and peak allocation, and stores reproducible evidence. A server-side route calls NVIDIA Nemotron through Nebius Token Factory. The public workbench does not accept user code or personal data, and model output is never executed.
Measured GPU evidence
On an NVIDIA GeForce RTX 3050 Ti Laptop GPU, a 1,536 × 1,536 nearest-neighbor case reduced peak additional allocation from 65.012 MB to 0.463 MB, about 99% less. The tiled candidate took 36.061 ms versus 1.394 ms for the baseline, so the memory savings came with a substantial latency cost. It stayed within the demo’s configured 8 MiB additional-allocation and 1,000 ms latency limits. A separate attention run showed why measuring matters: Python tiling used 1.205 MiB and took 91.221 ms, while PyTorch SDPA used 0.75 MiB and took 0.55 ms; both passed the sampled output checks.
The reproducible correctness gauntlet passed 490 of 490 finite comparisons across 14 uneven shapes, three seeds, five tile sizes, duplicate-heavy and clustered inputs, and tile-boundary cases. It also caught two deliberately injected bugs. This is strong evidence for the tested implementations, not a proof for every input. The exact reports and reproduction steps are in the GitHub repository.
Nemotron evaluation and limits
We ran three separately frozen 12-case wording sets for the same fixed allocation and output-contract task. Nemotron scored 11/12, 10/12, and 12/12; the latest run completed all 12 requests. The simple keyword rule scored 5/12, 6/12, and 4/12 on those respective sets. These small evaluations measure strategy selection under varied wording. They do not test unfamiliar executable workloads, GPU correctness, or generalization, and they do not establish superiority over a stronger baseline.
Links
Built With
- cuda
- nebius
- next.js
- python
- pytorch
- react
- typescript
Log in or sign up for Devpost to join the conversation.