GPU Architecture · Physical Map
Zoom into the middle of the physical map — the Streaming Multiprocessor — and watch a warp actually execute.
In Lesson 1 you placed the GPU on two maps. Now we open the physical one and look at its central organ. A GPU is, structurally, just an array of Streaming Multiprocessors (SMs) sharing an L2 cache and DRAM. Your RTX 5080 has 84 SMs; an H100 has 132. Nearly everything interesting happens inside one SM — so if you understand one, you understand the compute engine.
An SM is not one big processor. It's divided into 4 sub-partitions (NVIDIA calls them processing blocks), each a near-independent little engine. This is the picture to burn in:
ONE STREAMING MULTIPROCESSOR (SM)
Count it up: 4 sub-partitions × 32 lanes = 128 CUDA cores per SM. Across the RTX 5080's 84 SMs that's 84 × 128 = 10,752 CUDA cores — the exact number on the box.1 Now you know what that number is: not 10,752 processors, but 10,752 arithmetic lanes, in groups of 32, each group fed by one scheduler.
That "64 KB register file slice" in each sub-partition is the thing from your last questions. Four slices × 64 KB = the SM's 256 KB register file. A thread's registers come from the slice of the sub-partition it's running on. Full detail in the register & occupancy reference card.
Here is the concept everything pivots on. The hardware does not schedule threads one at a time. It schedules them in fixed groups of 32, called a warp. A warp is the smallest unit a scheduler can act on — you never get 1 thread on the hardware, you get 32.2
And the 32 threads of a warp are not independent. They execute in lockstep: the scheduler issues one instruction, and all 32 lanes run that same instruction at the same time, each on its own data and its own registers. NVIDIA's name for this is SIMT — Single Instruction, Multiple Threads.
One instruction — say c = a + b — issued once, executed by all 32 lanes simultaneously, each on its own a, b, c. That's SIMT.
Why 32? It's a fixed hardware constant NVIDIA has kept for every GPU: wide enough to amortize the cost of one shared scheduler/decoder across many lanes, narrow enough that groups stay flexible. You'll see the number 32 everywhere in CUDA, and it always traces back to the warp.
Each sub-partition's scheduler holds many resident warps at once — dozens, drawn from the blocks assigned to this SM. Its job each clock cycle is tiny and relentless:
Look at all my resident warps. Pick one whose next instruction is ready (its inputs have arrived). Issue that one instruction to my 32 lanes. Repeat next cycle.
This is exactly the latency hiding from Lesson 1, now with a mechanism. When a warp issues a memory load and must wait hundreds of cycles for the value, the scheduler doesn't wait with it — it simply picks a different ready warp next cycle. With enough resident warps, there is always someone ready, and the lanes stay busy. The wait is real; it's just hidden behind other warps' work. (This is why occupancy — how many warps are resident — matters.)
Lockstep is cheap and fast — until the 32 threads want to do different things. What happens at a branch where some threads go one way and some the other?
if (threadId % 2 == 0)
x = expensive_A(); // even lanes want this
else
x = expensive_B(); // odd lanes want thisThe lanes can't each take their own branch — there's only one instruction stream for the warp. So the hardware runs both paths, one after the other, masking off the lanes that shouldn't be active for each:
Running path A: even lanes active, odd lanes masked off (idle, struck through). Then it flips — odd lanes run path B while even lanes sit idle. The two halves run serially, not in parallel.
This is warp divergence, and it's the first real performance rule you can act on: branches that split threads within a warp cost you, because the paths serialize and half your lanes idle. Branches where an entire warp goes the same way are free. Writing GPU code that keeps a warp's 32 threads on the same path is a recurring theme — and now you know exactly why.
An SM issues work in warps of 32 threads that share one instruction (SIMT); its schedulers hide memory latency by swapping among many resident warps; and its weak spot is divergence, when threads in a warp split paths and serialize.
From memory — one click locks each answer.
RecallThe smallest group of threads a warp scheduler can issue an instruction to is —
The warp — a fixed 32 threads — is the quantum of scheduling. You never get a single thread on the hardware; the scheduler always issues one instruction to all 32 lanes together (SIMT). Block size is yours to choose, but the block is still executed as warps of 32.
MechanismWhen a resident warp issues a slow memory load, its warp scheduler on the very next cycle —
Latency hiding, mechanized: the scheduler just picks a different ready warp next cycle. With dozens of warps resident there's almost always one ready, so the lanes stay busy while the load is in flight. Nothing stalls, migrates, or changes clock.
TransferInside one warp, half the threads take the if and half take the else. The hardware handles this by —
Warp divergence: with only one instruction stream per warp, the two paths run serially, each with the non-participating lanes masked off and idle. That serialization — and the idle lanes — is the cost. Keep a warp's 32 threads on the same path and branches are free.
NVIDIA Ampere GA102 Whitepaper — the SM diagram & description (pp. 8–12). The canonical, diagram-heavy view of exactly the SM you just learned: sub-partitions, warp schedulers, CUDA-core lanes. Compare its figure to the one above. Open the PDF. For the precise semantics, CUDA C++ Programming Guide §"Hardware Implementation."
Ask me anything. Good threads to pull: if a block is bigger than 32 threads, how do its warps map onto the 4 sub-partitions? · what exactly makes a warp "ready" vs "stalled"? · are Tensor Cores just more CUDA cores? Any of these is a great next question — just ask in chat.