RTLoop

AI that writes hardware, tests it, and fixes its own bugs

Comment

Inspiration

I'm an electronic and electrical engineering student, and I built RTLoop to learn hardware design and FPGA-style development hands-on, using NVIDIA's Nemotron models to write circuit designs and a real simulator to verify them.

Language models can write Verilog, but code that looks right isn't necessarily right, and the only way to know is to simulate it. NVIDIA Research's VerilogEval benchmark gives a public way to measure that. So I asked a practical question: how far can a closed loop with a real simulator take a small, cheap model, and when is it worth paying for a bigger one?

What it does

RTLoop is an AI agent that designs digital circuits and checks its own work.

You give it a hardware spec in plain English. A Nemotron model writes the circuit in Verilog, the language used to describe digital hardware. RTLoop then runs the design in a simulator and compares it with a known-correct reference circuit. If it fails, the simulator's error goes back to the model for another try. If a small model keeps failing, RTLoop hands the problem to a bigger one: Nemotron 3 Nano, then Super, then Ultra. Every call logs its tokens, time and cost.

The live demo lets you watch it happen:

  • Attempt log: the result, tokens, time and cost of every call.
  • Waveforms: the reference circuit's signals next to the AI's, with mismatches shaded red. You can see exactly where a design goes wrong, then see the fixed version match.
  • Market view: for the trading problem, a chart of bid and ask prices with the crossed-book periods shaded.
  • Benchmark tab: results for all 151 problems, coloured by which model solved each one.

I also wrote a problem of my own: a top-of-book quote tracker. It keeps the latest bid and ask, flags a crossed book, and reports the spread. To prove its test is strict, I wrote six deliberately buggy versions of the circuit, and the test catches all six.

How we built it

  • A harness I verified before trusting it. A Python runner compiles and simulates each candidate and reports pass, compile error, mismatch or timeout. Reference solutions must pass, and deliberately broken circuits must fail.
  • Honest scope. 5 of the benchmark's 156 problems fail even with NVIDIA's own reference solution on the Icarus Verilog version I could run on Windows, so I excluded them. Everything is measured on 151 problems.
  • The agent. A retry loop that feeds simulator output back to the model, a model cascade, a spend cap, and per-call logging.
  • The models. NVIDIA Nemotron 3 Nano, Super and Ultra, through Nebius Token Factory's API. The API follows the OpenAI request format, so I called it with the standard openai Python library. No OpenAI models are involved.
  • The demo. A Gradio app in a Docker container on Render's free tier. The API key stays on the server, and visitors are limited to preset problems, with rate limits and a daily spend cap.
  • AI assistance. I built this with an AI coding assistant (Claude Code). I chose the problem, directed the design, ran and checked everything, and decided what to measure and what to claim.

Results

Every number is measured on the same 151 problems, with an 8,192-token output cap per call. I ran each configuration three times, because the models' sampling is random.

Giving the model the simulator's feedback, then a bigger model when stuck, solved more problems at a cost of cents.

  • Nano, one attempt: a mean of 90.0 of 151 solved (59.6%), about $0.09 per run.
  • Nano, up to 3 attempts with simulator feedback: a mean of 115.3 (76.4%), about $0.21 per run.
  • Nano, then Super: a mean of 124.7 (82.6%), about $0.42 per run. Super was only called on problems Nano couldn't solve.
  • Nano, then Super, then Ultra: 133 solved (88.1%) in a single run. Nano solved 109, Super 17 and Ultra 7. It's one run, so I treat the gap to the two-stage cascade as indicative, not proven. The Nano and Super calls cost $0.43; Ultra has no published price, so I don't show its cost.
  • Ultra on its own: given the 25 problems the first cascade run couldn't solve, it solved 14.

The gaps between configurations (about 90, 115 and 125 solved) are much larger than the run-to-run spread inside each (2 to 4 problems). Two caveats: the attempt budgets aren't identical across the rows, and 7 of the 25 problems the first cascade run didn't solve were replies cut off at the token limit.

Challenges we ran into

  • Making the measurement trustworthy. It would have been easy to report a flattering number. Instead I found that five benchmark problems can't be verified on my setup, that some replies hit the token limit before producing any code, and that identical runs vary by a few problems. I excluded, disclosed and measured each of those, and ran every configuration three times so the results come with a range.
  • Setting up the toolchain. My laptop couldn't run WSL because virtualisation was off in the BIOS, so I used a native Windows build of Icarus Verilog and avoided an installer my antivirus flagged. Setting up prerequisites across several platforms took a large share of my time.
  • Hosting. The platform I planned to use for a free demo had put Gradio and Docker hosting behind a paid plan, so I moved to Render's free tier. It sleeps when idle, so the first load can take about a minute.
  • A public demo that can't hurt me. The demo spends real credits, so I limited it to preset problems and added rate limits and a daily spend cap.

Accomplishments that we're proud of

  • A working, deployed, measured system, built in about a week alongside my coursework. It runs live at https://rtloop.onrender.com, and the code is public at https://github.com/KartaHughes/RTLoop.
  • A clear result, with the evidence behind it. Simulator feedback lifted a small model from 59.6% to 76.4% of problems solved, and escalating to a bigger model when stuck took it to 82.6%, at around 40 cents for a full run. Every number comes from raw logs committed to the repo.
  • A test I can trust. The harness passes all the reference solutions and rejects deliberately broken circuits, and my own trading problem's testbench catches all six bugs I planted in it.
  • A way to see what the AI got wrong. The waveform view shows the AI's circuit next to the reference with the mismatches in red, and the market view shows the trading problem as bid and ask prices, so anyone can follow what happened without reading Verilog.
  • Honest reporting. I excluded the five problems I couldn't verify, ran each configuration three times so results come with a range, and labelled the single-run Ultra result as a single run.
  • My first deployed project. Setting up hardware simulation tools, calling model APIs, and putting a safe, rate-limited demo on the internet were all new to me.

What we learned

I learned Verilog and how hardware is verified: how a testbench works, why you test the tests with deliberately broken designs, and how a design can look right and still be wrong.

I learned to work with APIs and AI models in practice: when retrying with feedback is enough, when escalating to a larger model pays off, and how to measure cost per solved problem.

I learned that one run can mislead. Two identical runs of the same setup differed by several problems, which is why every result here is repeated and reported as a range.

I also learned how to turn a prototype into something that holds up in public, with a deployment, spend limits, and honest reporting of what the numbers do and don't show.

What's next for RTLoop

For RTLoop itself, there are three things I want to test:

  • a higher output-token limit, to see how many of the cut-off replies would have been solved;
  • whether a larger model does better when it starts fresh or when it sees the earlier failed attempts, which my data raised but didn't settle;
  • a clean, repeated three-stage run once Ultra has a published price.

For me, the next step is a real FPGA project that carries the same simulate-and-verify approach onto actual hardware.

Built With

Share this project:

Updates

Submission history