Codex Can Do It

Inspiration

Designing a neural-network hardware accelerator is rarely a single coding task. It requires understanding a software reference model, translating nested loops and quantization equations into cycle-accurate RTL, debugging timing-sensitive memory interfaces, and validating the final design on real FPGA hardware.

For students and small engineering teams, this process can be especially difficult. A single signed arithmetic error, incorrect BRAM latency assumption, or mismatched tensor layout can produce an output that appears almost correct while still failing byte-exact verification.

We started this project with a simple question:

Can Codex act as a practical engineering partner throughout the entire hardware-development workflow—not just generate isolated code snippets?

Codex Can Do It explores how OpenAI Codex can support the development of a quantized CNN accelerator from software specification to verified FPGA execution.

What it does

Our project implements and optimizes the core data path of a quantized convolutional neural-network accelerator.

The system includes:

An IM2COL engine that converts input feature maps into a matrix representation for efficient convolution
A GEMM engine that performs quantized matrix multiplication
A requantization pipeline that converts 32-bit accumulated values back into 8-bit activations
Support for convolutional and fully connected layers
Byte-exact comparison against a software reference model
End-to-end LeNet inference validation
FPGA board-level verification

Codex was used as an engineering copilot throughout the process. It helped us analyze unfamiliar code, trace data movement across multiple RTL modules, translate C reference functions into Verilog, generate debugging strategies, interpret simulation failures, and identify opportunities for performance optimization.

The project is not intended to show that AI can replace hardware engineers. Instead, it demonstrates how an engineer can use Codex to investigate a complex system faster while retaining control over architecture, correctness, and verification.

How we built it

We began with a C software reference implementation that defined the expected behavior of the accelerator.

For convolution, the input tensor is transformed using IM2COL:

$$ K = C_{\text{in}} \times K_H \times K_W $$

$$ N = H_{\text{out}} \times W_{\text{out}} $$

The resulting matrix is multiplied by a weight matrix:

$$ [M \times K] \times [K \times N] = [M \times N] $$

Because the network uses quantized values, each accumulated output must also be corrected and requantized. A simplified form of the computation is:

$$ t = mm - z_x S_w - z_w S_x + b $$

$$ q = \left(t \times M_0\right) \gg R $$

The result is then rounded, shifted, offset by the output zero point, and clamped to the valid 8-bit range.

We divided the implementation into three major RTL components:

1. IM2COL compute engine

The software reference used deeply nested loops over channels, kernel positions, and output coordinates. We converted these loops into a hardware finite-state machine with counters.

The engine also handles padding, stride, dilation, input zero points, byte packing, and synchronous BRAM reads.

2. GEMM and requantization engine

The GEMM engine reads packed activation and weight data, performs parallel multiply-accumulate operations, calculates input and weight sums, and sends the results to multiple requantization lanes.

The design supports both convolution-style matrix multiplication and fully connected layers with transposed weight storage.

3. Verification and optimization workflow

We created a layered validation process:

Compare software intermediate values against expected results
Compare RTL outputs against the software reference
Run byte-exact tests for individual operators
Execute complete LeNet inference
Verify the design on an FPGA board
Profile cycle counts and identify bottlenecks

Codex assisted at each stage by explaining the existing architecture, proposing assertions and debug signals, tracing mismatches across files, and reviewing optimization ideas before implementation.

Challenges we ran into

Translating software loops into hardware control

A C loop executes sequentially and can access memory as though data were immediately available. RTL must explicitly represent every counter, state transition, handshake, and memory delay.

The IM2COL engine was especially challenging because several logical loop dimensions had to be mapped onto a finite-state machine while preserving the exact output order expected by the GEMM engine.

Handling BRAM latency

Our block RAM interface had a one-cycle read latency. The address issued in the current cycle did not correspond to valid data until the following cycle.

Initially, this caused shifted or duplicated values. We solved the problem by separating request and consume states and tracking whether returned data was valid.

Signed arithmetic and requantization

Quantized inference combines unsigned activations, signed weights, signed offsets, 32-bit accumulation, and 64-bit multiplication.

Small mistakes in sign extension, arithmetic shifting, rounding, or clamping caused byte-level differences in the final output. Codex helped us trace expressions back to the C reference and inspect the width and signedness of each intermediate value.

Debugging almost-correct results

One of the hardest failure modes occurred when most outputs were correct but a small number differed by one or two values.

These errors could originate from rounding rules, accumulator width, address generation, pipeline alignment, or parameter indexing. We learned to avoid debugging only the final output and instead compare intermediate tensors, sums, accumulators, and requantization inputs.

Balancing performance and correctness

After achieving functional correctness, we began reducing cycle count. However, increasing parallelism or overlapping memory operations could introduce hazards and make debugging significantly harder.

We therefore treated the verified implementation as a protected baseline and evaluated each optimization independently.

What we learned

The most important lesson was that AI-assisted engineering works best when the model is given a verifiable contract.

Codex was most effective when we provided:

The exact C reference function
RTL interface definitions
Memory timing assumptions
Expected tensor dimensions
Simulation logs
Intermediate reference values
Explicit resource and cycle constraints

We also learned that asking Codex to “fix the RTL” was much less effective than asking focused questions such as:

Which cycle does this BRAM response correspond to?
Is this expression signed at every intermediate stage?
Which counter combination generates this output address?
Where can the software and RTL tensor layouts diverge?
Which states contribute most to the total cycle count?

Codex significantly reduced the time needed to understand unfamiliar modules and formulate debugging hypotheses. However, simulations, byte-exact tests, synthesis reports, and FPGA measurements remained the source of truth.

Our final takeaway is simple:

Codex is most powerful not as an automatic code generator, but as a reasoning partner inside a rigorous engineering and verification loop.

What's next

We plan to continue improving the accelerator by:

Reducing redundant memory-access cycles
Increasing overlap between computation and data movement
Evaluating different processing-element configurations
Improving timing and resource utilization
Expanding validation to additional neural-network layers and models
Automating more of the software-to-RTL comparison workflow

Ultimately, we hope this project can serve as a reproducible example of how Codex can support complex embedded-system and semiconductor-design workflows.