Verilog AI Tutor

Write a circuit and the simulator finds the first wrong sample. Nemotron then coaches you to the fix without giving away the answer. From a single wire to a 32-bit adder.

Comment

Inspiration

Beginners learning Verilog get stuck in the same place. The online judge says "37 mismatches" and shows a waveform, and they cannot tell why. A human teaching assistant would point at the wrong sample, ask which line produced it, and stop short of the fix.

We wanted that teaching assistant for the first 28 problems of the popular HDLBits problem path, ending at a 32-bit adder-subtractor. We also wanted every hint to stay grounded in what the simulator actually saw.

NVIDIA's own RTL work showed the pattern we needed. Its VerilogEval benchmark is adapted from these same problems, and its ACE-RTL agent loops generate → test → reflect. We turned that loop around: the learner generates, Icarus Verilog tests, and Nemotron reflects.

What it does

  • 28 problems in 8 stages, from a single wire to an adder-subtractor. 19 come from NVIDIA's VerilogEval (MIT). Our authoring agent wrote the other 9, which VerilogEval does not cover.
  • A real simulator grades every submission. Icarus Verilog compiles and simulates it against a reference.
    • The first wrong sample, with its inputs and its expected and actual outputs, comes straight from the waveform and is drawn as a timing diagram.
    • Unsafe code (system tasks, include directives, force, references into the testbench) is rejected before it runs.
  • A four-step hint ladder. L0 names the concept, L1 gives the counterexample, L2 points to the lines that drive the wrong output, and L3 gives the structure in words.
  • "Ask the coach" sends Nemotron 3 Ultra the problem, the learner's numbered code, the simulator's facts and a hidden reference solution. Nemotron writes one hint at the next level and names the line it blames.
  • Every model hint passes a leak check before the learner sees it. A hint that writes the fixed line goes back to the model; if it leaks again, the learner gets the rule-based hint instead.

How we built it

  • Grader. Icarus Verilog 12 with VerilogEval's flags and failure categories. A small VCD reader recovers the exact counterexample.
  • Testbench generator. New problems get testbenches generated from their interface, so a model can never ship a testbench that checks nothing. It accepts and rejects the same designs as VerilogEval's own testbenches.
  • Authoring agent (Nemotron 3 Ultra). It writes the spec, the provided modules, a behavioural reference, a learner-style solution, directed test vectors and hints. Tools then decide whether the problem is right:
    • port checks;
    • simulation with the required submodules;
    • mutation testing, in which every beginner-style bug must be caught;
    • Yosys equivalence, to separate harmless mutants from weak tests.

Every failure goes back to the model.

  • Coach. A rule layer derives facts from the simulator. Nemotron 3 Ultra turns them into one hint, a leak check filters the hint, and a cache plus per-IP and daily limits keep costs bounded.
  • Web app. FastAPI, a CodeMirror editor, a stage map and a waveform view with dec/hex/bin. It runs in Docker on Render's free tier. Every model call goes through Nebius Token Factory.

Challenges we ran into

  • Keeping the model honest. Without the simulator's numbers, models confidently explained the wrong mechanism. On one question about XORing a 32-bit b with a 1-bit sub, all three Nemotron models were wrong with reasoning turned off. With reasoning on, they were right. So every hint now starts from simulator facts, and reasoning stays on.
  • Hints that do not give the answer away. The leak check compares each hint with the reference statements the learner is missing. Naming an operator is fine; writing the fixed expression is not.
  • The reasoning budget. Reasoning tokens count against max_tokens, and one hint was cut off mid-JSON at 2,048. We raised the cap and retry cut-off replies.
  • Testbench quality. A testbench that never exercises a carry passes broken adders. Mutation testing and Yosys equivalence caught this before any learner did.

Accomplishments that we're proud of

  • The authoring agent wrote and verified all 9 missing problems in 17 rounds for about $0.15. In 7 of the 9, the first draft failed a check and the tool feedback fixed it.
  • Mutation testing across all 28 problems: 219 beginner-style mutants, 149 caught, 0 missed, 46 proven equivalent by Yosys.
  • Coach evaluation on 173 buggy submissions. Nemotron 3 Ultra blamed the injected line in 97.8% of bugs that have one; the rule layer alone managed 48.2%. In bugs where a statement was deleted, it named the unassigned signal every time. Zero hints leaked, at L2 or at L3.
  • About $0.003 and a 3-second median per hint. All development and evaluation cost about $1.50 of Token Factory credit.

What we learned

  • Tools make better judges than prompts. Simulation, mutation testing and formal equivalence turned "the model says it is right" into "it is right".
  • For RTL explanations, model reasoning is worth its latency. It was the difference between a right and a wrong explanation.
  • A hint ladder needs discipline at every level. Our first L1 hints already named the cause, so we had to tell the model to trace values only.

What's next

  • Extend the path past the adders: multiplexers, K-maps, flip-flops and FSMs, all authored and verified the same way.
  • Generate a fresh variant of a problem when a learner gets stuck, verified like every other problem.
  • Show a schematic of the learner's circuit next to the reference.
  • A classroom mode, so teachers can see which concepts a group is stuck on.

Built With

Share this project:

Updates

Submission history