GPU Architecture · Foundations
Before any term makes sense, one reframe: a GPU is not a faster CPU. It is a different bargain.
You already have a rock-solid mental model of a CPU: a handful of powerful cores, each racing through one instruction stream as fast as physically possible. That instinct is exactly what makes GPU terminology confusing — because a GPU throws that priority away. This whole course hangs on the trade it makes instead. Get it now, and warp, SM, block, and coalescing will all fall out of it naturally.
A CPU is a latency machine: its job is to finish one task as soon as possible. So it spends most of its silicon not on arithmetic, but on hiding the wait — large caches, branch predictors, out-of-order execution — all to keep one thread sprinting.
A GPU is a throughput machine: its job is to finish an enormous pile of similar tasks per second, and it does not care if any single one is slow. So it spends its silicon the opposite way: mostly on arithmetic units, with tiny caches and simple control.1
A CPU avoids waiting. A GPU embraces waiting — and hides it by having so much other work queued up that there is always something ready to run. This is called latency hiding through massive parallelism, and it is the single fact the rest of the architecture is built to serve.2
Reading a value from a GPU's main memory can cost hundreds of clock cycles. A CPU fights this with big caches so it rarely has to wait. A GPU takes the other road entirely:
When one group of threads stalls waiting on memory, the hardware instantly parks it and runs a different group that is ready — then another, then another. With thousands of threads resident at once, the arithmetic units almost never sit idle, even though every individual memory access is slow. The latency never went away; it got covered up by other work.
Here is the framing you asked for, and the spine of everything ahead. GPU vocabulary lives in two parallel hierarchies that describe the same machine from two sides:
| Physical — the silicon (Lessons 2, 4) | Logical — the CUDA model (Lessons 3, 5) |
|---|---|
| The GPU chip | A grid (one kernel launch) |
| A Streaming Multiprocessor (SM) | A block of threads runs here |
| A warp scheduler + its lanes | A warp — 32 threads in lockstep |
| A CUDA core / lane | One thread |
You wrote that these two levels are "related, but not the same." Exactly right — and that gap is where most GPU confusion lives. Logical terms are what you write in code; physical terms are what the hardware does with it. The whole skill you're building is fluently translating between the two columns. Today you don't need the rows yet — just the shape: two maps of one machine.
A "CUDA core" is not a CPU core. A CPU core is an independent processor that fetches and runs its own instruction stream. A "CUDA core" is just a single arithmetic lane — it can't fetch instructions on its own; it does what its warp scheduler tells all 32 of its neighbours to do, together. When a spec sheet says "16,384 CUDA cores," read it as "16,384 arithmetic lanes," not "16,384 CPUs." We'll unpack this in Lesson 2.
Answer from memory — the retrieval is what makes it stick. One click locks each answer.
RecallA GPU devotes far more of its transistor budget to arithmetic units than to caches and control logic. It is built above all to maximize —
Throughput — work completed per second across the whole pile — is the GPU's design goal. Single-thread speed, big caches, and branch predictors are exactly the latency-oriented things a CPU invests in and a GPU largely gives up.
MechanismA thread needs a value from GPU main memory, which costs hundreds of cycles. Rather than stall, the hardware —
This is latency hiding: with thousands of threads resident, a stalled group is parked and a ready one runs, so the arithmetic units stay busy. The GPU doesn't avoid the wait (no big caches, no value prediction) — it covers it with other work.
TransferOf these four tasks, the one that maps worst onto a GPU is —
A pointer chase is inherently sequential — each step depends on the last, so there's no pile of independent work to hide latency with, and 99% of the lanes sit idle. The other three do the same operation over millions of independent items: the GPU's sweet spot.
CUDA C++ Programming Guide — §1 "Introduction / The Benefits of Using GPUs." NVIDIA's own framing of the latency-vs-throughput trade, with the classic figure of how each chip spends its transistors. This is the canonical statement of today's keystone idea. Read it here.
I'm your teacher — ask me anything. If the "two maps" framing feels abstract, or you want the CPU/GPU trade pushed further (why can't a CPU just add more cores and become a GPU?), ask. Questions are how we find the edge of your understanding and aim the next lesson. Just tell me in chat.