FPGA协同GPU

Inspiration

What i# From PyTorch to FPGA: My Learning Journey in MLIR, HLS, and Heterogeneous Computing

1. Why I Started This Journey

My initial interest was in collaborative computing between FPGA and GPU.

GPUs excel at large-scale parallel computation and have a mature software ecosystem. FPGAs, on the other hand, can construct customized data paths tailored to specific algorithms, offering advantages in pipeline parallelism, low latency, and energy efficiency. However, after working on related projects, I realized that the biggest challenge in heterogeneous computing is not just hardware design, but:

How can we gradually transform high-level neural networks described in PyTorch into hardware structures suitable for FPGA implementation?

In traditional workflows, algorithm engineers handle PyTorch models, while hardware engineers are responsible for HLS C++, RTL, and FPGA implementation. There is a clear abstraction gap between the two.

A model typically goes through the following stages to become hardware:

PyTorch Model
    ↓
Computation Graph and Tensor Operations
    ↓
Loops, Memory Access, and Data Dependencies
    ↓
HLS C++ and Hardware Instructions
    ↓
RTL Circuits
    ↓
FPGA Implementation

If these transformations rely heavily on manual effort, developers must not only understand the model structure but also deal with loop unrolling, pipelining, array partitioning, on-chip buffering, dataflow partitioning, and resource constraints.

This led me to ask:

Can a compiler automatically handle the transformation and optimization from AI models to hardware implementations?

MLIR provided a clear direction.

MLIR is not just a fixed intermediate representation for machine learning, but a flexible multi-level compiler infrastructure. It allows different dialects to represent programs at different abstraction levels, and uses passes and lowering to gradually transform them. One of its goals is to bridge compilers, domain-specific languages, and heterogeneous hardware.

2. My First Stage: Truly Understanding MLIR

When I first encountered MLIR, my biggest challenge was not writing code, but understanding why it requires so many layers of intermediate representation.

In traditional compilers, programs are usually transformed from source code into a unified IR, and then into machine code. However, for AI and hardware compilation, a single IR cannot effectively represent:

Tensor operations in neural networks;
Loop and index computations;
Memory layout and data movement;
Pipeline and parallel structures;
Hardware modules, registers, and timing relationships.

The core idea of MLIR is: not forcing all optimizations to happen at the same abstraction level.

For example, a matrix multiplication can first exist as a tensor-level operation:

[ C_{ij}=\sum_{k=0}^{K-1}A_{ik}B_{kj} ]

Then it can be gradually transformed into loops, memory accesses, and hardware execution structures.

Following MLIR tutorials, I completed the following practices:

Reading and modifying MLIR examples;
Understanding Operation, Region, Block, and SSA Value;
Learning how to define Dialect, Type, and Attribute;
Using TableGen to define new operations;
Writing and registering custom passes;
Using Pattern Rewrite to modify IR;
Performing Dialect Conversion and Lowering;
Using test files to verify outputs at each stage.

I gradually realized that a compilation pipeline is not simply about converting one text into another, but about preserving program semantics at every stage.

I summarized my understanding as:

\text{Representation} + \text{Transformation} + \text{Verification} ]

Where:

Representation determines what the compiler can express;
Transformation determines what the compiler can optimize;
Verification ensures the correctness of transformations.

3. From Software Compilation to Hardware Compilation

After understanding MLIR basics, I started learning CIRCT.

CIRCT applies MLIR and LLVM methodologies to hardware design tools, using different hardware dialects to represent combinational logic, sequential logic, module hierarchy, scheduling, and hardware structures.

This stage changed my understanding of HLS.

Previously, I thought HLS was simply:

C/C++ → RTL

Later, I realized that a complete HLS compiler must address at least four aspects:

Program Analysis: identifying loops, dependencies, and memory behavior;
Scheduling: determining when operations execute;
Resource Binding: deciding how hardware resources are allocated and reused;
Hardware Generation: constructing datapaths, control logic, and interfaces.

Thus, a more accurate formulation is:

\text{Program Analysis} + \text{Scheduling} + \text{Resource Binding} + \text{Hardware Generation} ]

Through CIRCT tools and passes, I observed how hardware IR evolves and how high-level computations are gradually transformed into modules, ports, registers, and combinational logic.

This was the first time I clearly saw the boundary between software semantics and hardware semantics.

4. Learning AI Engine and Specialized Accelerator Compilation

Beyond FPGA HLS, I also explored mlir-aie and tpu-mlir.

mlir-aie targets AMD AI Engine devices, mapping designs onto AI Engine arrays and handling tiles, data movement, kernels, and runtime configuration.

This helped me understand that heterogeneous computing is not simply running the same program on different devices, but requires task partitioning:

Control-intensive tasks        → CPU
Massively parallel computation → GPU
Fixed dataflow, low-latency    → FPGA
Array-based vector compute     → AI Engine / NPU

Different devices have different memory hierarchies, parallel models, and communication mechanisms. Therefore, compilers must answer:

Which operators should run on which compute units?
How should data move across devices and memory hierarchies?
Can computation and communication overlap?

This led me to a key insight:

The essence of heterogeneous collaboration is not to use all devices simultaneously, but to assign each device the tasks it is best suited for, while minimizing data movement.

5. Entering MLIR-based HLS through ScaleHLS

The project that truly connected MLIR and HLS for me was ScaleHLS.

ScaleHLS is built on MLIR and can take HLS C/C++ or PyTorch models as input, generating optimized HLS C++ code, which is then passed to tools like Vitis HLS for RTL generation. It performs optimizations across multiple IR levels.

I completed the full pipeline from model input to HLS output:

PyTorch Model
    ↓
Model Graph
    ↓
Tensor / Operator-level IR
    ↓
Linalg / Affine / SCF / MemRef IR
    ↓
Loop and Memory Optimization
    ↓
HLS-specific Optimization
    ↓
HLS C++ Generation
    ↓
C Simulation
    ↓
C/RTL Co-simulation
    ↓
Synthesis and Implementation
    ↓
FPGA Deployment

This process taught me:

Successful compilation is only the beginning—it does not guarantee performance or correctness.

A design must pass multiple validations:

Output consistency with PyTorch;
Semantic equivalence across MLIR transformations;
Successful HLS C++ compilation;
Correct C simulation;
Correct RTL co-simulation;
Reasonable latency, initiation interval, and resource usage;
Correct FPGA execution;
Efficient host–FPGA data transfer.

6. How I Built My Own Project

To avoid staying at the level of running examples, I structured my project like a full compiler system:

project/
├── models/
├── frontend/
├── mlir/
├── dialect/
├── passes/
├── hls/
├── scripts/
├── test/
├── reports/
├── runtime/
└── README.md

I divided development into stages:

Stage 1: Minimal Working Pipeline

Model Import → MLIR → HLS C++ → Simulation

Focus: correctness, not performance.

Stage 2: Saving IR at Each Level

00_input.mlir
01_canonicalized.mlir
02_linalg.mlir
03_affine.mlir
04_memref.mlir
05_hls_optimized.mlir
06_output.cpp

This helps locate errors precisely.

Stage 3: Automated Regression

For each pass:

Validate input conditions;
Check output structure;
Verify numerical correctness;
Ensure proper error handling;
Test repeated execution stability.

Stage 4: Performance Optimization

Loop Tiling
Loop Interchange
Loop Unrolling
Pipeline
Array Partition
Dataflow
Buffer reuse
Memory optimization
Parallel compute units

7. Lessons from HIDA and Stream-HLS

Building on ScaleHLS, I studied HIDA and Stream-HLS.

HIDA focuses on hierarchical dataflow optimization, decomposing large problems into multiple levels.

Stream-HLS emphasizes multi-kernel applications, global scheduling, and automatic dataflow architecture generation.

I realized:

Adding HLS pragmas alone is not enough for high performance.

For multi-kernel systems, total latency is not simply additive:

[ T_{\text{total}} \approx T_{\text{fill}} + (N-1)\times \max(II_1,II_2,\ldots,II_m) + T_{\text{drain}} ]

Key factors:

Matching production and consumption rates;
FIFO depth;
Avoiding stalls and backpressure;
Memory bandwidth;
Continuous data streaming.

8. Systematic HLS Practice with CODO

Using CODO, I completed a more systematic MLIR-to-HLS workflow.

CODO emphasizes:

Coarse and fine-grained dataflow optimization;
On-chip/off-chip data movement;
Automatic scheduling balancing performance and resources.

It unified my understanding:

MLIR
  ↓
HLS Compilation
  ↓
Dataflow Architecture
  ↓
FPGA Validation

CODO showed me how a modern HLS compiler should handle dataflow systematically.

9. Major Challenges

9.1 Environment and Version Compatibility

Issues included:

LLVM version mismatch;
API changes;
CMake configuration errors;
Missing registrations;
Linking failures;
TableGen issues;
Python binding mismatches;
Vitis incompatibility.

Solutions:

Fix LLVM commit;
Record versions;
Use isolated builds;
Save build commands;
Start from minimal examples;
Run regression after updates.

9.2 Cross-Dialect Transformation

Lowering must correctly handle:

Types;
Indices;
Memory layout;
Control flow;
Dependencies.

Errors can cause:

Shape mismatch;
Incorrect indexing;
Buffer issues;
Data corruption;
Loop errors;
Non-synthesizable code.

9.3 Correctness vs Performance

Floating-point operations are not associative:

[ (a+b)+c \neq a+(b+c) ]

Thus, optimizations may introduce numerical differences.

I evaluate:

Absolute error;
Relative error;
Precision tolerance;
Quantization effects;
Model accuracy.

9.4 Compute Optimization ≠ System Optimization

Actual performance:

[ P_{\text{actual}} = \min(P_{\text{compute}}, P_{\text{memory}}, P_{\text{communication}}) ]

Even with more compute units, bandwidth limits can stall execution.

Key insight:

FPGA optimization is about the entire system, not isolated kernels.

10. How I Evaluate HLS Results

Correctness

Matches PyTorch;
Passes simulations;
Works on FPGA.

Performance

[ \text{Throughput} = \frac{f_{\text{clk}}}{II} ]

Resources

LUT, FF, DSP, BRAM, URAM;
Memory bandwidth.

Timing

Clock period;
WNS/TNS;
Critical path.

Engineering Quality

Reproducibility;
Error clarity;
Testability;
Extensibility;
Platform portability.

Optimization goal:

[ \min T_{\text{end-to-end}} ]

subject to resource and accuracy constraints.

11. What This Journey Changed

Before:

PyTorch
MLIR
HLS
CIRCT
FPGA
GPU

Now:

Algorithm
    ↓
Graph
    ↓
IR
    ↓
Optimization
    ↓
Dataflow
    ↓
Hardware
    ↓
Runtime
    ↓
System Performance

I now see compilers as bridges between algorithms and hardware.

12. Next Steps

Future directions:

Custom MLIR dialects;
Reusable HLS passes;
Design space exploration;
Auto-tuning;
FPGA–GPU partitioning;
Communication optimization;
Compiler–hardware feedback loop;
Reproducible pipelines.

Goal:

PyTorch Model
    ↓
Automatic Analysis
    ↓
MLIR Optimization
    ↓
Dataflow Optimization
    ↓
HLS C++
    ↓
Simulation & Reports
    ↓
FPGA Deployment
    ↓
Heterogeneous Execution

Conclusion

What excites me most is seeing:

A high-level algorithm transformed step by step into a real FPGA circuit.

From reading MLIR IR to writing passes, from generating HLS code to FPGA deployment, I have built a complete technical perspective.

The journey is long and challenging, but every time an abstract IR becomes a real hardware pipeline, I am reminded why I chose this path.

I want to continue exploring compilers, HLS, and heterogeneous computing—and help bridge the gap between AI algorithms and custom hardware. t does