GPU Architecture: Nvidia vs AMD

This page moves from beginner fundamentals to advanced, performance-engineering-level detail — jump to whatever you need from the right-hand outline. Later sections assume the vocabulary introduced earlier (SM/CU, warp/wavefront), but nothing here requires having written a kernel yet. Two companion pages: CUDA & HIP for the actual launch syntax and index arithmetic, and GPU Optimization for what to do with this mental model once you have it.

A CPU spends its transistor budget on a handful of cores plus big caches, deep out-of-order pipelines, and branch predictors — it is optimized to run one thread as fast as possible. A GPU spends the same budget on thousands of simple ALUs and shallow control logic — it is optimized to hide latency by keeping many threads in flight, not to make any single thread fast. That single bet (throughput over latency) is the root cause of almost every GPU programming rule that follows: keep everything busy, tolerate stalls with more parallelism, and never assume one thread's speed matters on its own.

A GPU "card" is really three things bolted together: the die (the silicon full of SMs/CUs), a stack of high-bandwidth memory sitting right next to or on top of the die (HBM on datacenter cards, GDDR on consumer cards), and a PCIe — or NVLink/Infinity Fabric — connector that plumbs it into the host system. The die and its memory talk over a very wide, very short on-package bus, which is exactly why GPU memory has so much more bandwidth than a CPU's DIMMs, at the cost of much smaller capacity. From the host's point of view the whole card shows up as a PCIe device; the OS/driver treat it much like an NVMe SSD or NIC, just with a far bigger BAR (memory-mapped address window) behind it.

Code that runs on a GPU is a kernel: a function launched not once, but once per thread, across up to millions of threads at the same time. Threads are grouped into blocks (CUDA) / workgroups (HIP), and blocks are grouped into a grid covering the whole problem. This software-side grouping maps directly onto the hardware below: a block is scheduled onto exactly one SM/CU for its entire lifetime and never migrates mid-flight; the threads inside it are further split into warps/wavefronts, the hardware's actual unit of execution. Getting this mapping right — enough blocks to fill every SM/CU, enough threads per block to fill whole warps — is the first and most basic performance lever, before anything in GPU Optimization applies.

Nvidia GPUs are built from Streaming Multiprocessors (SMs); AMD GPUs are built from Compute Units (CUs), or Workgroup Processors (WGPs) on RDNA. Each one is a self-contained scheduler + ALUs + register file + shared memory block; a full GPU is dozens to over a hundred of these tiled across a die.

Threads execute in lockstep groups: a warp on Nvidia is 32 threads, a wavefront on AMD's CDNA (compute-focused, e.g. MI-series) is 64 threads, while RDNA (gaming/consumer) supports wave32 as well. This width matters directly for performance — it is the unit that suffers from divergence (see GPU Optimization) and the unit memory coalescing rules are defined around.

Both are dedicated matrix-multiply-accumulate units sitting next to the regular ALUs, built specifically because GEMM (general matrix multiply) dominates AI compute. Nvidia calls them Tensor Cores (since Volta); AMD calls them Matrix Cores on CDNA (MI200/MI300 series). They natively support the reduced-precision formats used in AI training/inference — see FP16/BF16/FP8 in GPU Optimization — and get most of a GPU's advertised "AI TFLOPS" from these units, not the regular shader cores.

Same shape on both vendors, different names: registers (per-thread, fastest) → shared memory/LDS (per-block, on-chip, programmer-managed) → L2 cache (chip-wide) → global/HBM memory (off-chip, largest, slowest). The entire discipline of GPU optimization is about keeping data as high in this hierarchy as possible and hiding the latency of the levels below with parallelism.

On Nvidia, shared memory and the L1 cache are carved out of the same physical block per SM, with a configurable split between the two. AMD's RDNA (consumer) line adds a large Infinity Cache between L2 and GDDR as an extra hit-rate booster not present on the CDNA (datacenter) line, which leans on HBM bandwidth instead.

Every kernel is either compute-bound (limited by how many FLOPs the ALUs/tensor cores can issue) or memory-bound (limited by how fast data can be streamed from HBM). The roofline model plots achievable performance against arithmetic intensity (FLOPs per byte moved) — below a hardware-specific "knee," you are memory-bound and more compute won't help; above it, you are compute-bound and more bandwidth won't help. Most AI training/inference kernels today are memory-bound, which is why memory bandwidth growth (HBM3, HBM3e) matters as much as raw FLOPs growth for AI progress.

Every Nvidia GPU has a compute capability (e.g. sm_90) and every AMD GPU has a gfx ISA target (e.g. gfx942) — the version of the instruction set and hardware features the compiler targets with -arch= / --offload-arch= (see CUDA & HIP). Compiling for the wrong target either fails outright or silently falls back to slower generic code, so knowing which generation you're actually running on is not just trivia.

Each SM/CU holds far more resident warps/wavefronts than it can execute in a single cycle; a warp scheduler picks among the ones that are ready (not stalled on a memory access or a dependency) and issues an instruction each cycle — this is the actual mechanism behind "occupancy hides latency" (see GPU Optimization). Modern SMs have several independent scheduler partitions — four per SM on recent Nvidia architectures — each with its own register-file slice, so multiple warps genuinely issue in parallel rather than merely interleaving. Since Volta, Nvidia SMs also support independent thread scheduling: threads within a diverged warp each keep their own program counter, letting the hardware interleave both sides of a branch more flexibly than the strict all-then-all reconvergence of older SIMT designs. It softens some divergence penalties, but doesn't eliminate the fundamental cost described under warp divergence.

A single GPU's HBM bandwidth doesn't help multi-GPU training much if the GPUs can only reach each other over PCIe — which is why datacenter GPUs ship with a dedicated, much faster GPU-to-GPU interconnect instead of routing peer traffic through the host. Nvidia's is NVLink, with NVSwitch chips fanning it out to all-to-all connectivity across 8+ GPUs in a node; AMD's equivalent is Infinity Fabric, branded xGMI for the GPU-to-GPU links on MI-series cards. Both sit underneath the collective libraries — NCCL and RCCL respectively (see GPU Libraries) — so most code never calls them directly, but their bandwidth and topology (fully-connected vs ring vs hierarchical) is what decides whether all-reduce time in distributed training actually scales the way you'd hope.