Lesson 4 · The Physical Memory Hierarchy

Why memory — not arithmetic — is almost always the thing that's actually slow, and how the chip is built to fight it.

Through Lessons 1–3 one phrase kept recurring: "a memory load takes hundreds of cycles." We hid that latency behind swarms of warps. But latency was only half the story. Now we open the second map — memory — and meet the cost that can't be hidden by more warps: bandwidth. This is the lesson that explains why fast GPU code is, overwhelmingly, about data movement, not math.

The pyramid: four levels of storage

Storage on a GPU is a hierarchy. Near the lanes: tiny, blindingly fast. Far from them: enormous, slow. Every step toward the core is roughly an order of magnitude faster — and an order of magnitude smaller. Here it is on your RTX 5080:

Registers256 KB/SM · ~1 cycle · per-thread

Shared memory / L1~128 KB/SM · ~30 cycles · per-SM (on-chip SRAM)

L2 cache64 MB · ~200 cycles · whole GPU

Global memory — GDDR7 DRAM16 GB · ~400–800 cycles · 960 GB/s · whole GPU

↑ faster, smaller, closer to the lanebigger, slower ↓

Level	Scope (who shares it)	Latency	Bandwidth	Managed by
Registers	one thread	~1 cycle	astronomical (many operands/cycle/lane)	the compiler
Shared / L1	one SM (shared: one block)	~30 cycles	tens of TB/s aggregate	you (shared) · auto (L1)
L2 cache	whole GPU	~200 cycles	several TB/s	automatic
Global (DRAM)	whole GPU + the CPU	~400–800 cyc	0.96 TB/s	you (cudaMalloc)

You've already met the top two: registers are the SM's register file (Lesson 2's occupancy story); "shared memory" is a programmer-controlled slice of the same on-chip SRAM as L1. What's new is the bottom two — the L2 that every SM shares, and global memory, the 16 GB of GDDR7 that also happens to be the only memory your CPU can hand data to.¹

Latency you can hide. Bandwidth you cannot.

This is the crux of the whole lesson, so slow down here. A trip to global memory costs you two different things:

Latency — the wait per access

~400–800 cycles before your value arrives.
Hideable. While one warp waits, the scheduler runs others (Lesson 2). With enough warps, the lanes never notice.
Fixed cost per access, paid in time.

Bandwidth — total bytes/second

960 GB/s, shared by all 10,752 lanes at once.
Not hideable. It's a hard ceiling on the whole chip. More warps don't raise it — they just queue for the same pipe.
A budget, paid in bytes. Spend it twice, wait twice.

The insight the hierarchy exists to serve

More warps hide latency but do nothing for bandwidth. Once every lane is fed and DRAM is saturated at 960 GB/s, the only way to go faster is to stop going to DRAM — reuse data from the fast on-chip tiers instead. The whole pyramid is a tool for one job: touch global memory as little as possible.

Why almost every kernel is memory-bound

Put the two headline numbers side by side. Your RTX 5080 can do 56 TFLOPS of FP32 but read only 0.96 TB/s from DRAM. Divide them:

56.3e12 FLOP/s ÷ 0.96e12 byte/s ≈ 59 FLOP per byte
// To keep the math units fed, a kernel must do ~59 FLOPs
// for every byte it pulls from global memory (~234 per float).

That's the break-even ratio — the arithmetic intensity a kernel needs just to not starve the compute units. Now measure a real kernel against it. Vector add, c[i] = a[i] + b[i]:

So a GPU running vector-add uses well under 1% of its arithmetic power. It spends virtually the whole time waiting on the 960 GB/s pipe. It is memory-bound — and so is the vast majority of real code. The 56 TFLOPS on the box is a number you only approach if you feed the lanes from on-chip memory, not DRAM.²

This is the roofline mental model: a kernel's ceiling is either compute (56 TFLOPS) or bandwidth (0.96 TB/s × its arithmetic intensity), whichever is lower. Below ~59 FLOP/byte you're under the slanted "bandwidth roof"; above it, under the flat "compute roof." Knowing which roof you're under tells you what to optimize.

The move the hierarchy enables: load once, reuse many times

Here's how you actually beat the bandwidth wall, and it's the reason shared memory exists as a thing you control. Take multiplying two N×N matrices. The naive version re-reads each input value from global memory N times — catastrophic bandwidth waste. The fast version:

This is tiling, and it is the archetypal GPU optimization. You'll do it for real later; today the point is only why it works — you are climbing the pyramid, trading slow-and-shared DRAM for fast-and-local SRAM. Every serious GPU optimization is a variation on this one idea.

Teaser: how you spend bandwidth well — coalescing

Even when you must hit DRAM, how a warp's 32 threads address it matters enormously. If they touch 32 consecutive addresses, the hardware fuses them into one wide memory transaction. If they scatter, it issues many — wasting most of every transfer. That's memory coalescing; it decides how much of your 960 GB/s you actually get. We'll give it a full treatment in the optimization lessons.

Check yourself

ConceptYou add far more resident warps to a kernel that is saturating DRAM at 960 GB/s. The likely effect on its speed is —

ApplyA kernel does 2 FLOPs per byte it reads from global memory. On a chip that breaks even at ~59 FLOP/byte, this kernel is —

TransferTiling a matrix multiply into shared memory speeds it up chiefly because it —

Primary source to read next (≈20 min)

CUDA C++ Best Practices Guide — "Memory Optimizations." NVIDIA's own ranking of what matters for performance, and it is almost entirely about the hierarchy you just learned: minimizing global-memory traffic, coalescing, using shared memory. Read it here. For measured latencies, the Dissecting Blackwell microbenchmark paper.

One lesson to go. Lesson 5 flips this to the logical side: CUDA's memory spaces — global, shared, local, constant, registers — the keywords you'll actually write, and which physical tier from today backs each one. Good threads to pull first: why is "local memory" actually in DRAM? · what makes shared memory fast — and what's a "bank conflict"? · how does data even get from my CPU into that 16 GB? Ask in chat.