Anatomy of a GPU
What's inside a graphics card, one layer at a time, and which parts decide how fast an LLM runs on it.
One card, four layers
From the top down: fans and shroud, heatsink, circuit board, backplate. Pick a part from the list, or point at it in the drawing, to see what it does.
A graphics card built around NVIDIA’s AD102 chip, the one in the GeForce RTX 4090. The numbers are the RTX 4090’s; the cooler is a generic design, drawn to scale.
The parts, top to bottom
Every graphics card is the same sandwich. A processor and its memory sit on a circuit board. A cooler is clamped on top and a stiff plate on the back. Here is each part and what it means when the job is running a language model.
01Fans and shroud3 fans · 88 mm
The shroud is the card's outer shell. Its three fans push air down through the heatsink fins underneath. The faster they spin, the more heat leaves the card.
For LLMsA long inference job keeps the GPU near full power for hours. If the fans can't keep up, the chip slows itself down to stay cool, and tokens per second drop with it.
02Heatsink4 heat pipes · ~450 W of heat
A copper block presses on the GPU chip. Heat pipes carry the heat out to stacks of thin metal fins, and air from the fans carries it away. Almost every watt the card draws ends up here as heat.
For LLMsEvery multiply-add the GPU does for a model turns into heat that this block has to get rid of.
03GPU chip (AD102)608 mm² · 76.3 B transistors
The processor itself: one piece of silicon about 25 mm on a side holding 76.3 billion transistors. It is split into 144 streaming multiprocessors (the grid drawn on it). An RTX 4090 switches on 128 of them, which gives 16,384 CUDA cores and 512 Tensor Cores.
For LLMsThe Tensor Cores do matrix multiplication, which is nearly all of an LLM's work, at fp16 and lower precisions.
04Video memory (VRAM)24 GB · 1,008 GB/s
Twelve 2 GB GDDR6X chips sit right around the chip. They are wired to it by a 384-bit bus, 32 bits per chip, and together move about a terabyte of data every second.
For LLMsThis is where the model lives. A 7B-parameter model in fp16 needs about 14 GB for its weights, so it fits; a 70B model needs about 140 GB, so it doesn't. Every new token reads every weight once, so 1,008 GB/s ÷ 14 GB caps that 7B model at roughly 70 tokens per second for a single user.
05Power delivery (VRM)20 + 3 phases
Power arrives at 12 volts, but the GPU core runs at about 1 volt. The voltage regulator module steps it down. Each phase is a choke (the small cubes) plus a power stage: 20 phases feed the GPU and 3 feed the memory. Sharing the current across many phases keeps each one cool and the voltage steady.
For LLMsInference load jumps from idle to full in milliseconds as requests arrive. The VRM has to hold the voltage steady through every jump.
0616-pin power connectorrated 600 W · card draws 450 W
The 12VHPWR connector brings power from the power supply. Twelve large pins carry the current (six +12 V, six ground), and four small sense pins tell the card how much it is allowed to draw.
For LLMsAt scale, power is a running cost: one 450 W card running flat out all day uses about 10.8 kWh.
07PCIe x16 connectorPCIe 4.0 x16 · ~32 GB/s each way
The gold fingers plug into the motherboard: 16 PCIe 4.0 lanes of about 2 GB/s each in each direction. This is how data, including the model itself, gets from system RAM onto the card. The slot can also supply up to 75 W.
For LLMsCopying a 14 GB model over this link takes about half a second at best. The link is about 30× slower than the card's own memory, which is why a model is loaded once and kept in VRAM.
08Display outputs3 × DisplayPort 1.4a · 1 × HDMI 2.1
The metal bracket screws to the PC case and holds the display connectors.
For LLMsWhen a GPU only runs models, these ports go unused. Data-centre GPUs such as the H100 don't have them at all.
09Circuit board (PCB)384-bit memory bus
A stack of copper layers wires everything together: power from the VRM to the chip, and 384 data lines between the chip and its memory, routed so their signals arrive in step.
For LLMsThose memory lines are kept as short as possible, which is why the memory chips crowd so tightly around the GPU.
10Backplatestiffens · spreads heat
A metal plate on the back of the card. It stops the long, heavy board from bending. With thermal pads behind the VRM and the GPU, it also carries a little extra heat away.