How to Shrink a Big Model

Transcript

Before we start: a disclosure 0:07

Maya Before we start, a disclosure that happens to be on topic. I'm Maya, he's Theo, and neither of us exists. This episode was researched and written by an AI, and these voices are synthetic.

Theo So for the next couple of hours, a large model is going to explain how to make large models smaller. We've decided not to take it personally.

Maya The sources are real, though, and we'll say where each claim comes from. If a word comes out sounding slightly wrong, think of it as a rounding error.

Cold open: 552 billion parameters on one RTX 5090 0:44

Maya Here's a machine someone described on the llama.cpp discussion board. One RTX 5090, the top consumer graphics card, with thirty-two gigabytes of video memory. A hundred and twenty-six gibibytes of ordinary system RAM. And a fast NVMe drive. On it, they run DeepSeek V four point one Flash, a model whose main network alone has five hundred and fifty-two billion parameters.

Theo Which at sixteen bits per weight would be over a terabyte. More than thirty times what that card holds.

Maya And it runs. About five tokens a second on new text, and around twenty-one when the parts it needs are already sitting on the card. That's a single person's report on a single machine, so we'll treat it as an existence proof, not a benchmark. But it's real enough to ask the question this episode is about: how?

Theo And the honest answer is that it isn't one trick. It's four different kinds of shrinking stacked on top of each other.

Maya The first is fewer bytes per weight. The experts in that model are stored in a four-bit floating-point format called MXFP4. That's quantization.

Theo The second is fewer weights that actually do work. It's a mixture-of-experts model, so for any given token only a small fraction of those five hundred and fifty-two billion parameters are used. The rest can sit somewhere slower.

Maya Which is the third kind: changing where the weights live. Most of the experts aren't on the graphics card at all. They're read from the drive as the router asks for them, through a cache in system memory.

Theo And the fourth is the strangest. The model file on that machine is about five hundred gigabytes. A hundred and eighty-nine gibibytes of it isn't ordinary weights at all. It's a lookup table, a hundred and ninety-six billion more parameters, and that engine keeps it on the drive and never loads it whole. The model reads a few rows from it per token, the way you'd look something up in an index.

Maya So the model has a part that's essentially a giant phrase book on disk.

Theo DeepSeek calls it Engram. We'll get to it near the end, along with a very similar idea from Qwen.

Maya And there's a fifth kind we'll cover, which is the oldest: making a genuinely smaller model, by pruning a big one or teaching a small one to imitate it.

Theo So, three questions to keep in your head for the whole episode, whenever someone says they've shrunk a model. What got smaller? Bytes on disk, memory on the GPU, the number of weights, or the compute per token. Those are different things. Second: what got worse, and how was that measured? And third: did it actually get faster, on the hardware you have?

Maya Because those three come apart constantly. A network can lose eight in every nine of its weights and get only three times cheaper to run. And a four-bit format can, on the graphics card in our own test machine, save memory without speeding up the maths.

Theo We'll label where the evidence comes from as we go: papers, the original pull requests behind the formats people download, vendor blogs, and a few hands-on reports from people running these models at home.

Maya Let's start where the field started, with an idea that's older than deep learning: that what a model knows and the weights it happens to use to know it are not the same thing.

Knowledge is not the weights: Optimal Brain Damage, model compression, Hinton's distillation 4:24

Theo Nineteen eighty-nine. Yann LeCun, John Denker and Sara Solla at Bell Labs publish "Optimal Brain Damage". The question is simple: if you want to delete some of a network's parameters, which ones?

Maya And the obvious answer is the small ones.

Theo That's the obvious answer, and the paper's point is that it's the wrong measure. A small weight can still be expensive to remove, if the loss curves sharply as that weight moves towards zero. A large weight can sit in a direction where the loss barely changes. So instead of size, they estimate how much the loss would rise if you removed each parameter, combining the size of the move with the curvature of the loss in that direction, its second derivative.

Maya How big was the network?

Theo Tiny by today's standards. A handwritten digit recogniser with about a hundred thousand connections but only two thousand five hundred and seventy-eight free parameters, because many connections shared weights. Deleting by their curvature score raised the training objective much less than deleting by magnitude. After retraining, they could remove about sixty percent of the parameters and the error measure stayed almost the same.

Maya Their saliency scores suggested an entire layer of an earlier, larger version of the network wasn't pulling its weight. So they removed the whole penultimate layer. Half the parameters gone. The training loss got ten times worse. And the classification error on the test set went down.

Theo Which tells you something that comes back over and over in this episode. The number you optimise and the thing you care about are not the same. Here the loss got worse and the actual task got better. Later we'll see the opposite: the loss barely moves and the task falls apart.

Maya Jump to twenty oh six. Cristian Buciluă, Rich Caruana and Alexandru Niculescu-Mizil at Cornell, a paper simply called "Model Compression". The problem then wasn't neural networks being too big. It was ensembles. The best classifiers on tabular data were collections of hundreds or thousands of models voting together.

Theo Accurate, and miserable to deploy.

Maya Right. Their idea: take the giant ensemble, use it to label a large amount of extra data, and train a single small neural network to reproduce its answers. The small network isn't learning the task from scratch. It's learning to imitate the ensemble.

Theo And where does the extra data come from?

Maya That turned out to be the whole problem. Their first attempt sampled each input feature independently, which destroys the correlations between features. That data actively hurt: with enough of it, the mimic did worse than a network trained on the original four thousand examples. What worked was a method they called MUNGE, which makes new examples by perturbing real ones towards their nearest neighbours, so the synthetic data stays where the real data lives.

Theo And the result?

Maya Across eight problems, the mimic networks recovered ninety-seven percent of the improvement the ensemble had over the best single model. Average error about the same as the ensemble's. And the size: their table averages about five hundred and fifty megabytes per ensemble against a third of a megabyte for the mimic network. They describe it as a thousand times smaller and a thousand times faster. Twenty fourteen: Jimmy Ba and Rich Caruana again, "Do Deep Nets Really Need to be Deep?" They take a strong deep model for speech recognition, an ensemble of nine convolutional networks, and train a shallow network, a single hidden layer, to match its outputs before the final softmax.

Theo And the shallow one matches it?

Maya At about twelve million parameters, the shallow mimic slightly beat a deep network of the same size: twenty-one point six percent phone error against twenty-one point nine. Trained directly on the labels, shallow networks couldn't get there; making them wider didn't help. A much larger shallow mimic, with a hundred and eighty million parameters, got to twenty percent, close to the best convolutional model.

Theo So depth wasn't the limiting factor. The way the shallow network was trained was.

Maya That's their conclusion, almost word for word: the limitation was the learning procedure, not the representational power.

Theo And then the paper everyone cites. Twenty fifteen. Geoffrey Hinton, Oriol Vinyals and Jeff Dean: "Distilling the Knowledge in a Neural Network". The core move is to train the student on the teacher's full probability distribution rather than just its top answer, and to soften that distribution with a temperature so the small probabilities become visible.

Maya Why would the small probabilities matter?

Theo Because that's where the knowledge about similarity lives. Their example is handwritten digits: a particular two might be given a probability of one in a million of being a three and one in a billion of being a seven. Both are tiny. But the ratio says this two looks more like a three than a seven. A hard label throws that away.

Maya And they tested how much of the knowledge travels that way.

Theo With a striking experiment. They train a small network on MNIST with every example of the digit three removed. It has never seen a three. It learns only from the teacher's soft outputs on other digits. After adjusting one bias term on the test set, it correctly classifies ninety-eight point six percent of the threes.

Maya The digit is learned from how other digits resemble it.

Theo A model's weights are one way of storing a function. A different, smaller set of weights can often store nearly the same function, if it's shown the function and not just the training labels. And the pruning work said something parallel: plenty of weights can go, but which ones is a harder question than their size. Both ideas got stress-tested in the next decade, on much bigger networks.

Before LLMs: Han's pruning, Deep Compression, lottery tickets, integer phones 10:54

Maya Twenty fifteen. Song Han, Jeff Pool, John Tran and Bill Dally: "Learning both Weights and Connections for Efficient Neural Networks". The method is the obvious one this time: remove the weights with the smallest magnitude, retrain what's left, and repeat.

Theo The method LeCun's paper argued against.

Maya Yes, and on these networks it worked, because of the retraining. Pruning without retraining fell apart quickly. With several rounds of prune and retrain, AlexNet went from sixty-one million parameters to six point seven million, nine times fewer, with no loss of ImageNet accuracy. VGG-16 went from a hundred and thirty-eight million to ten point three million, thirteen times fewer, and its error rate actually went down slightly.

Theo Thirteen times fewer parameters. So thirteen times faster?

Maya No, and the paper says so. For AlexNet, the caption reads: pruning reduces the number of weights by nine times and computation by three times. Most of the parameters were in the huge fully connected layers at the end, which are cheap per weight. The convolutional layers, where most of the arithmetic happens, couldn't be pruned nearly as hard.

Theo Which layer was the most fragile?

Maya The smallest one. The very first convolutional layer, the one that looks directly at the pixels, kept eighty-four percent of its weights. The giant fully connected layers kept nine percent. The follow-up a few months later was "Deep Compression", Han, Mao and Dally. Three stages: prune, then make the remaining weights share a small set of values found by clustering, so each weight is stored as a short index, and then Huffman-code the result. AlexNet went from two hundred and forty megabytes to six point nine. VGG-16 from five hundred and fifty-two to eleven point three. Thirty-five and forty-nine times smaller, no loss of accuracy.

Theo Thirty-five times smaller. That's the number people remember.

Maya And it's correct for storage, with two details. After pruning, you have to store where the surviving weights are, and the paper notes that the indexes cost about as much as the weights themselves: roughly half and half. A sparse model doesn't just store fewer numbers. It stores fewer numbers plus a map.

Theo And the speed?

Maya They timed only the pruned fully connected layers, at a batch size of one, where they were three to four times faster. The full compressed pipeline, with the clustered weights, wasn't benchmarked for speed at all. And at a batch of sixty-four, the pruned layers were actually slower than the dense ones.

Theo Because a GPU is very good at dense matrix multiplication and not good at hopping around a sparse one. That gap, between fewer numbers and less time, comes back in almost every section. Twenty nineteen: Jonathan Frankle and Michael Carbin, "The Lottery Ticket Hypothesis". This one asked a question that sounds abstract and turned out to be very practical. If a pruned network works so well, why don't we just train the small network from the start?

Maya And the answer was?

Theo You can, but only if you start it from the right place. They train a network, prune it, and then take the surviving weights and rewind them to the exact random values they had before training began. Then they retrain just that subnetwork. At ten to twenty percent of the original size, it matches the full network's accuracy in no more training iterations. That doesn't count the cost of finding it.

Maya And if you give the same small network fresh random starting values?

Theo It does much worse. The structure alone isn't enough. The specific initial values matter. In their words, structure alone cannot explain a winning ticket's success.

Maya The intuitive guess is that the winning weights are the lucky ones that happened to start near where they needed to end up.

Theo And they checked. It's the opposite: the winning ticket's weights move further during training than the others. On the deeper networks there was a catch: at the standard learning rate they found no winning tickets at all. With a learning-rate warm-up, VGG-19 kept just one and a half percent of its weights and still beat the full network. ResNet-eighteen also needed a lower learning rate. So it's a real phenomenon, but not a free recipe for training small networks from scratch.

Maya The other strand of this era is quantization, and the reference paper is from Google in twenty eighteen. Benoit Jacob and colleagues, "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference". It's the scheme TensorFlow Lite adopted. Weights and activations both stored as eight-bit integers, and the model trained with that rounding simulated, so it learns to cope with it.

Theo Give me the detail that shows how low-level this gets.

Maya An eight-bit signed integer runs from minus a hundred and twenty-eight to a hundred and twenty-seven. They simply never let a weight take the value minus a hundred and twenty-eight. A single product of two eight-bit values already fits in sixteen bits, but the sum of two products can overflow, just barely, and only in the case built from minus a hundred and twenty-eight. Forbid that value, and the inner loop can add pairs of products in sixteen bits before widening, which speeds it up on phone processors.

Theo They give up one value out of two hundred and fifty-six to make the arithmetic faster.

Maya Exactly. The payoff on a phone: a small face detector ran at about thirty-six frames a second on a single core in eight-bit, against twenty-three in floating point. Real-time versus not.

Theo And one result from that paper sets up everything coming next. They tried four bits, and for their face-attribute classifier, four-bit weights hurt far more than four-bit activations.

Maya Four-bit weights with eight-bit activations lost about eleven percent of its average precision. Four-bit activations with eight-bit weights lost about three and a half.

Theo Hold on to that. In large language models, it flips completely.

The outlier problem: LLM.int8, SmoothQuant, AWQ, QuaRot, GPTQ, the case for 4 bits 17:36

Theo So by the time large language models arrived, quantization looked like a solved engineering problem. Eight-bit integers worked for vision models; the tooling existed. Then people tried it on models with billions of parameters, and it broke.

Maya The paper that explained it is from twenty twenty-two. Tim Dettmers, Mike Lewis, Younes Belkada and Luke Zettlemoyer: "LLM.int8". They took ordinary eight-bit quantization, the kind that works by finding the largest value in a row or column and scaling everything to fit, and applied it to models of increasing size.

Theo And it worked fine for small models.

Maya Fine up to a few billion parameters. Then it fell off a cliff. Their measure was perplexity, which is how surprised the model is by real held-out text: lower is better, and it's a test of next-token prediction, not of whether the model can solve anything. At thirteen billion parameters, the eight-bit model's perplexity was worse than the eight-bit model at six point seven billion. A model twice the size, quantized the same way, came out worse.

Theo Why?

Maya Because of what they called outlier features. In the hidden states of these models, almost every value lives in a narrow range, roughly between minus three and a half and plus three and a half. But a handful of specific dimensions, the same ones across many tokens and layers, carry values up to about sixty. In the thirteen billion model, the lowest quarter of those outlier values sit around minus sixty.

Theo And if you scale a whole row to fit a value of sixty into eight bits, everything else in that row gets squashed into a few of the available levels.

Maya Exactly. The precision for all the normal values collapses. And those few dimensions aren't noise. When they zeroed out the outlier dimensions, at most seven of them, perplexity rose by six hundred to a thousand percent. Zeroing the same number of randomly chosen dimensions changed it by about a tenth of a percent.

Theo Seven dimensions out of thousands, and the model depends on them completely.

Maya And one more detail that's easy to miss: the number of these outliers tracks the model's perplexity more cleanly than its size. Better-trained models have more of them. It isn't simply a property of big models; it's a property of models that have learned a lot.

Theo Their fix was blunt and effective. Find those outlier dimensions as you run, compute them in sixteen-bit, and do everything else in eight-bit. More than ninety-nine point nine percent of the values still go through eight-bit arithmetic, and quality matched sixteen-bit all the way to a hundred and seventy-five billion parameters.

Maya That's when the field stopped treating quantization as rounding, and started treating it as managing a few very large numbers. The next few methods are all different answers to that same problem.

Theo The first answer is SmoothQuant, from Guangxuan Xiao, Song Han and colleagues. They want both weights and activations in eight bits, because that's what lets the hardware's integer units do the maths. And naively, that's a disaster. Eight-bit weights and activations on OPT a hundred and seventy-five billion took perplexity from about eleven to over ninety thousand.

Maya So next-token prediction collapses completely.

Theo The outliers live in specific channels, the same columns for every token. But fast integer kernels can only apply one scale per token, per row. So you can't give the bad channels their own scale on the fly. Their insight: activations are hard to quantize, weights are easy, so move the difficulty. Divide each troublesome activation channel by a constant, and multiply the matching row of the next weight matrix by the same constant. Mathematically, the layer computes exactly the same thing. But now the activations are flatter, and the weights, which had headroom, absorb the outliers.

Maya A rebalancing you can fold into the weights in advance.

Theo So it costs nothing at inference time. With it, eight-bit weights and activations worked up to five hundred and thirty billion parameters on their tests.

Maya The second answer is AWQ, activation-aware weight quantization, also from Song Han's group, this time going for four-bit and three-bit weights while leaving activations in sixteen-bit. Their starting observation: a tiny fraction of weights matters disproportionately. Keep just one percent of the weight channels in sixteen-bit and most of the damage from three-bit rounding disappears.

Theo And the question is which one percent.

Maya Which is where it gets interesting. On OPT six point seven billion, at three bits, ordinary rounding gives a perplexity of about twenty-three and a half. Protect the one percent of channels with the largest weights, and you get twenty-two. Protect one percent chosen at random: twenty-four. Basically no help either way. Protect the one percent of channels that see the largest activations, and it drops to eleven point four, close to the sixteen-bit model's ten point nine.

Theo So the weights that matter are the ones multiplied by big inputs, not the ones that are big themselves.

Maya Which is LeCun's point again, in a different form: size isn't importance. And they then replace keeping those channels in sixteen-bit, which is awkward for hardware, with simply scaling them up before rounding so they lose less precision.

Theo The third answer is QuaRot, from Saleh Ashkboos and colleagues in twenty twenty-four. Multiply the hidden states by a rotation matrix, and multiply the weights by the inverse. In full precision, the model's output is identical. But a rotation spreads a few large values across many coordinates. The outlier that was sixty in one dimension becomes something modest in all of them.

Maya So the outliers aren't removed. They're dissolved.

Theo And they use a particular kind of rotation, a Hadamard matrix, which is all plus and minus ones, cheap to apply, and spreads values very evenly. With it, they quantized weights, activations and the KV cache of the seventy billion Llama two all to four-bit integers, losing at most about half a point of perplexity. And it does matter which rotation: a random rotation was clearly worse than the Hadamard one on the seven billion model.

Maya One more result from QuaRot that we'll come back to. In the KV cache, at two bits, keys are far more fragile than values. Two-bit keys hurt much more than two-bit values.

Theo A separate question is how to choose the rounded weights themselves. The method that made that practical at scale is GPTQ, from Elias Frantar, Saleh Ashkboos, Torsten Hoefler and Dan Alistarh. It descends from the same curvature idea as LeCun's paper, by way of its successor, Optimal Brain Surgeon. Instead of rounding each weight to the nearest value independently, quantize weights one at a time and, after each one, adjust the remaining unquantized weights to compensate for the error, using second-order information about how the layer's output depends on them.

Maya And they made it fast enough for huge models.

Theo About four GPU hours to quantize a hundred and seventy-five billion parameter model. At three bits, that model took about sixty-three gigabytes and fit on a single eighty gigabyte card, where the sixteen-bit version needed five of them. And the baseline it replaced, plain rounding to four bits, was bad enough that the rounded hundred and seventy-five billion model was worse than the ten times smaller thirteen billion model at full precision.

Maya Which raises the question every practitioner actually has. If I have a fixed amount of memory, should I run a big model squeezed hard, or a smaller model squeezed less?

Theo Dettmers and Zettlemoyer answered that in "The case for four-bit precision". Across model families from nineteen million to a hundred and seventy-six billion parameters, holding the total number of bits fixed, four-bit weights gave the best accuracy almost everywhere. Going from sixteen to eight to four bits, you can afford a bigger model and it wins. Going to three bits, the trade reverses.

Maya So, by twenty twenty-three, four bits was the sweet spot, and three was the edge. The obvious next question was whether anything could go below that.

How low can you go: QuIP#, QTIP, AQLM, BitNet, scaling laws for precision 26:45

Maya Below four bits, rounding each weight to the nearest of a few levels starts to fail, because there are so few levels. So the next generation of methods changed what gets rounded. Instead of quantizing weights one at a time, quantize them in groups, as vectors, and pick each group from a codebook.

Theo The clearest line of work here is from Albert Tseng, Chris De Sa and colleagues at Cornell. First QuIP sharp, then QTIP. And QTIP matters to anyone running models at home, because the EXL3 format, which we'll come to, is a streamlined version of it.

Maya Start with QuIP sharp.

Theo Two ideas. The first is the rotation trick we just heard about: multiply the weights by random Hadamard matrices so their values look like a smooth bell curve, with no outliers. They call that incoherence processing. The second is what to do once the weights look like a bell curve. If you quantize eight weights together, you're choosing a point in eight-dimensional space. And there's a beautiful mathematical object, the E8 lattice, which is the most efficient way to pack points in eight dimensions.

Maya So the codebook is a lattice rather than something learned.

Theo And the symmetry makes it tiny. Each group of eight weights gets a sixteen-bit code, so the codebook has sixty-five thousand entries. But you only store a table of two hundred and fifty-six base patterns; sign flips and a small shift generate the rest. The whole thing fits in one kibibyte, right next to the compute. They then fine-tune a little after quantizing.

Maya And the result that stands out?

Theo It's about budget, not about a single model. Plot quality against the total number of bits the model occupies. Their three-bit Llama two beats an imaginary perfect four-bit model, a sixteen-bit model squeezed into four bits' worth of space with no loss at all, at the same total size. In other words, at that point on the curve it's better to quantize a bigger model harder than to quantize a smaller one gently.

Maya Which is the four-bit scaling result pushed one step further.

Theo Exactly. With a caveat they make themselves: fewer bytes don't automatically mean faster. Reconstructing weights from a fancy codebook has a cost, and a real speed-up needs a method that decodes cheaply.

Maya Which is what QTIP addressed.

Theo QTIP replaces the stored codebook with a trellis code, borrowed from signal processing. Picture a sliding window of sixteen bits. Each weight's value is computed from the current window, and to move to the next weight you shift in two new bits. So you store two bits per weight, but every value depends on the sixteen bits around it, which lets it behave like a very high-dimensional codebook without storing one. Finding the best sequence of bits is a dynamic-programming search, the Viterbi algorithm.

Maya And there's a detail in it that sounds like a mistake.

Theo It does. To produce values that look like a bell curve, they take a pseudo-random thirty-two-bit word, split it into four bytes, and add the bytes together. The sum of four roughly uniform numbers is approximately Gaussian. It's coarse, and the authors report it doesn't hurt quantization quality. A hash and an addition replace the codebook.

Maya And QTIP also contains a warning about measurement.

Theo Yes. In one test on the seventy billion Llama three at four bits, they also quantized one particular tensor in the first layer that the authors know is fragile, and normally advise leaving alone. Both QuIP sharp and QTIP still had perplexity close to the sixteen-bit model: two point nine nine and two point seven five, against two point five nine. Looking only at perplexity, you'd call both fine. But on the ARC Challenge reasoning benchmark, the sixteen-bit model scores sixty point five. QTIP scored fifty-six. QuIP sharp scored thirty-five.

Maya A gap of four tenths of a point in perplexity, and a collapse of twenty-five points on the task.

Theo Perplexity averages over every token. A model can be nearly right on almost every token and wrong on the few that matter.

Maya The other big vector method is AQLM, additive quantization, from Vage Egiazarian, Dan Alistarh and colleagues. Each group of eight weights is stored as the sum of entries from learned codebooks, and in their main setting each group is a single sixteen-bit index.

Theo Their headline was two-bit Llama two.

Maya Yes, and two results from it are useful for decisions. First: a thirteen billion parameter model at about two point seven six bits per weight had lower perplexity than the seven billion model at full sixteen-bit precision, four point nine four against five point one two. Bigger and squeezed beats smaller and intact.

Theo So two bits is the place to be?

Maya No. Two bits isn't the best place on the curve. At about the same size in bytes, the seven billion model at four bits beat the thirteen billion model at two: five point two one against about five point six. The authors say plainly that strict two-bit compression is below the Pareto front. Two bits is impressive. It's not optimal. Then there's the approach that skips post-training quantization entirely. BitNet b one point five eight, from Shuming Ma and colleagues at Microsoft in twenty twenty-four. Every weight is minus one, zero or plus one.

Theo Three values. Log base two of three is one point five eight, hence the name.

Maya And the zero is the important part. With only plus and minus one, every input contributes to every output. Zero lets the model switch an input off. And with weights of minus one, zero and one, a matrix multiplication becomes additions and subtractions. No multiplications at all.

Theo The catch?

Maya It's trained that way from the start. It's not a method for converting a model you already have. And the efficiency numbers should be read carefully: the memory and speed measurements in the paper used a two-bit kernel, not a true one point five eight bit implementation. They report matching a full-precision baseline they trained themselves from three billion parameters upwards.

Theo Which leads to one more result that changes how you should think about all of this. "Scaling Laws for Precision", from Tanishq Kumar, Cengiz Pehlevan and colleagues at Harvard, Stanford, MIT and Databricks. They trained many small models and fitted a law for how precision interacts with model size and amount of training data.

Maya And the finding?

Theo The more data a model is trained on, the more it suffers from quantization afterwards. Train past a certain point and extra data can actually make the quantized model worse. Their intuition is that a heavily trained model packs more information into each weight, so there's less slack to round away.

Maya And real models are trained far past the classic ratios.

Theo the eight billion Llama three was trained on roughly two thousand tokens per parameter. Their own experiments go up to around a thousand, on small models of a couple of hundred million parameters, so applying it to Llama three is an extrapolation, and the effect is mild at eight bits. But it fits something the QuaRot authors reported: with the same method, Llama three lost noticeably more than Llama two.

Maya Now to what people actually download.

What people actually download: k-quants, imatrix, KL divergence, Unsloth, EXL3, KV-cache bits 34:52

Theo If you've downloaded a model to run locally, you've almost certainly met one of two families of formats. GGUF, from the llama.cpp project, which runs on almost anything, including plain CPUs and Macs. And EXL3, from the ExLlama project, built specifically for NVIDIA graphics cards. Neither started as a paper. They started as pull requests.

Maya Begin with GGUF. In June twenty twenty-three, a contributor who goes by ikawrakow opened a pull request titled simply "k-quants". It added a family of two- to six-bit formats, and, importantly, recipes that mix them.

Theo Which is what the suffixes mean. Q four underscore K underscore M.

Maya Q4_K_M is not four bits everywhere. The recipe stores some of the most sensitive tensors, half of the attention value and feed-forward down projections, at six bits. On the eight billion Llama three point one it comes out at about four point nine bits per weight. The S, M and L variants are different mixes at the same nominal bit level. Then in January twenty twenty-four, the same author added the importance matrix, often written i-matrix. You run some calibration text through the model and record, for each input channel of each weight matrix, the average squared activation. That's the whole importance matrix: one number per input channel, not a matrix the size of the model.

Theo Which is AWQ's insight again. Weights that meet big inputs matter more.

Maya Same principle. The quantizer then minimises a weighted error, so it spends precision where the activations say it counts. On Mistral seven billion, using it cut the extra perplexity of the two-bit format from about nineteen percent above the sixteen-bit model to about thirteen. At six bits it made almost no difference, which is what you'd expect: there's little error left to redistribute.

Theo And the author was candid about the calibration text. The pull request suggests that for instruct models, calibrating on instruct-style data "may actually be better". A hypothesis, not a measurement.

Maya Hold on to that, because it's become a selling point since.

Theo The llama.cpp documentation also shows how to measure the result. Their perplexity tool can save the full probability distribution of the original model, then compare a quantized model against it token by token, using KL divergence. How far apart the two models' predictions are, on every single token.

Maya Why is that better than perplexity?

Theo Because perplexity measures how well a model predicts the test text, and errors can cancel: the quantized model can get slightly better on some tokens and slightly worse on others. KL divergence measures how different it is from the original model, and you can look at the worst tokens, not just the average. The documentation suggests looking at percentiles of the change, precisely to tell noise from real damage.

Maya And there's a twenty twenty-six paper that shows why that matters. Uygar Kurt evaluated thirteen common GGUF formats on the eight billion Llama three point one instruct, on a CPU, across five task benchmarks. Three of the five-bit formats, Q5_0, Q5_1 and Q5_K_S, have exactly the same perplexity: seven point four three. On the grade-school maths benchmark, GSM8K, they score seventy-nine, seventy-eight and a half, and seventy-five and a half.

Theo Same perplexity, three and a half points apart on maths.

Maya And "three-bit" turns out not to be one thing either. The three three-bit recipes, small, medium and large, score sixty-eight, seventy-three and seventy-four on GSM8K, against about seventy-eight for the sixteen-bit model. Meanwhile HellaSwag, a common-sense benchmark, barely moved across the whole sweep. Maths is where the damage shows first.

Theo And one result from that paper about speed. On that CPU, the three-bit model generated text more than three times faster than sixteen-bit, because generation is mostly reading weights from memory. But it processed the prompt slower: fifty-seven tokens a second against eighty. The paper doesn't say why, and it isn't uniform: the four-bit K-S format processed prompts faster than sixteen-bit. We'll come back to why generation and prompt processing behave so differently. Next, Unsloth, which publishes many of the most-downloaded GGUFs. Their "Dynamic" quants choose the format layer by layer, per model, and calibrate on chat-style data. These are their own measurements, not independent testing, but their newer posts include a warning against their own headline numbers. They report top-one agreement with the original model, how often the most likely next token matches. Their smallest quant of a Qwen model kept about seventy-two percent. But they also measure agreement over thirty-two generated tokens, and there, stepping down from their two-bit XL quant to the next size, it drops from about a quarter to under ten percent. They warn of looping, and explicitly say not to use the heavily compressed versions for tool calling.

Maya One-token agreement can look fine while thirty-token behaviour falls apart.

Theo Now EXL3. The ExLlama-three README describes it as a streamlined variant of QTIP, the trellis method from the last section. And the author, who goes by turboderp, publishes his own comparison charts. These are images in the repository, so we read the values off them ourselves, and they're the format author's measurements, not independent ones.

Maya What do they show?

Theo On one recent chart, for a Qwen three point eight model, measured as KL divergence against the full model, EXL3 at about three bits per weight scored point zero one seven seven. Unsloth's dynamic GGUF at about four bits scored point zero one six five. Roughly the same fidelity, for about one bit per weight less.

Maya And the NVIDIA four-bit format on that chart?

Theo NVFP4, which that chart places at about four point seven five bits per weight, counting everything left unquantized. With only the weights in four-bit, point zero one. With activations in four-bit as well, point zero two four. Worse than EXL3 at three bits.

Maya What about very low bits?

Theo On the eight billion Llama three point one, the divergence climbs steeply below about two bits per weight: point three one at two bits, point five four at one point nine, point seven six at one point eight. And on the seventy billion model, measured in perplexity, every format converges above about five bits. EXL3, GGUF, the older EXL2, all around three point seven. The methods only really differ below four bits.

Maya So above five bits, pick whichever runs best on your hardware. Below four, the method matters a lot.

Theo The EXL3 docs also have practical advice from its optimisation guide. When it measured how much each tensor contributes to the final error, the most valuable bits were often small attention key and value projections in the middle of the network, not just the first and last layers people usually protect. It recommends calibrating on text in the model's real format, the chat template plus a sampled response with its reasoning included. And it warns that if you evaluate on the same trace you calibrated on, you'll overstate the gain.

Maya Which is the same mistake in every field: don't test on your training data.

Theo Last practical result, about the KV cache, the memory that grows with the length of your conversation. ExLlama lets you quantize keys and values separately. On the eight billion Llama three point one at six bits, with the cache's keys at two bits, HumanEval coding accuracy fell to between seventeen and twenty-seven percent, whatever the values were. With keys at four bits or more, and values at just two, it stayed at sixty-four to sixty-seven percent, about where it was with higher-precision values. That's one sample per task, so a few points is noise.

Maya Keys fragile, values tolerant. The same thing QuaRot found.

Theo Two independent sources pointing the same way. If you need to squeeze the cache, start testing with the values, not the keys.

Does it still work? Facts, reasoning, and answers that get longer 44:13

Maya We've already seen two warnings that perplexity can hide damage. This section is about how to actually tell whether a compressed model still works, and the honest answer is that it depends heavily on what you ask it to do.

Theo Start with "Compressing LLMs: The Truth is Rarely Pure and Never Simple", from Ajay Jaiswal, Zhangyang Wang and colleagues at ICLR twenty twenty-four. The title is a fair summary. They took Vicuna models, compressed them with the standard pruning and quantization methods, and built a set of tests aimed at knowledge and instruction following rather than perplexity.

Maya And the first thing they found was the gap.

Theo Pruning half the weights changed perplexity only a little with the good methods. But on closed-book factual questions, FreebaseQA, the seven billion model went from sixty-five percent with all its weights to fourteen percent with magnitude pruning, and about forty-three to forty-five percent with SparseGPT and Wanda, the two methods we'll meet in the pruning section.

Maya So it loses facts long before it loses fluency.

Theo They have an example in the paper: asked a trivia question, the full model says "The answer is: Woody Allen." The four-bit quantized version confidently says "The answer is one hundred percent." Perfectly grammatical. Completely wrong. That's one illustration, not a failure rate, but it captures the pattern.

Maya And the interesting twist?

Theo Give the compressed model the relevant passage in its prompt, and it does much better. As retrievers, reading the answer from supplied text, compressed models held up far longer: the seven billion model matched the full one to about forty percent sparsity, and the thirteen billion to about fifty percent sparsity or four-bit quantization. What breaks first is the knowledge stored in the weights, not the ability to read.

Maya Which is a practical point if you're running a model with retrieval: compression may hurt you less than benchmarks suggest.

Theo And instruction following was the most fragile of all. It only held up to about twenty-five percent sparsity.

Maya The more reassuring paper is from Eldar Kurtic, Dan Alistarh and colleagues: "Give Me BF16 or Give Me Death? Accuracy-Performance Trade-Offs in LLM Quantization". It's a very large evaluation, a preprint last revised in May this year, across Llama three point one from eight to four hundred and five billion parameters and the distilled DeepSeek reasoning models.

Theo How much did the big one lose?

Maya Almost nothing. On the four hundred and five billion Llama three point one instruct, on the Open LLM Leaderboard version two average: sixteen-bit forty-eight point seven, eight-bit floating point forty-eight point seven, eight-bit integer forty-seven point nine, four-bit integer weights forty-eight point two. About ninety-nine percent kept at four bits.

Theo And the smaller model?

Maya Less forgiving. The eight billion model at four bits kept about ninety-six percent. Same pattern for reasoning: the thirty-two billion distilled reasoning model kept ninety-nine and a half percent at four bits, and the smallest distilled model in the family kept about ninety-three and a half. The general rule shows up again and again: big models are more tolerant.

Theo And, as everyone found, activations were harder to quantize than weights, the reverse of the twenty eighteen vision result. Their other contribution is about speed. Which format is fastest depends on how you serve. Eight-bit weights and activations maximise throughput, many users at once, because the arithmetic itself gets cheaper. Four-bit weights with sixteen-bit activations minimise latency for a single user, because they reduce how much has to be read from memory. There isn't one best format; there's a best format for your traffic.

Maya And a nice concrete number: with weight compression, the four hundred and five billion model runs on four A100 or H100 GPUs. In sixteen-bit it needed sixteen. Now reasoning models, where things get more interesting. "Quantization Hurts Reasoning?", from Ruikang Liu and colleagues at COLM twenty twenty-five. Moderate quantization was mostly fine. Aggressive quantization exposed problems that are specific to how reasoning models work.

Theo Where did it break first?

Maya One was the KV cache. The distilled Qwen models inherit large bias terms in the projection that makes the attention keys. In the one and a half billion model the biggest of those bias values reaches four hundred and two, which forces a huge range on the cached keys. On the seven billion model, a simple four-bit cache quantizer that rounds the keys after the bias is added dropped the average score from sixty-three to seven and a half.

Theo Seven and a half. That's a broken model.

Maya And the fix was almost trivial: quantize the keys before adding the bias rather than after. That brought it back to sixty-one point six. Same bits, different order of operations.

Theo And their example of a failure.

Maya A four-bit floating-point thirty-two billion model, solving a geometry problem, quietly changes one of the coordinates given in the question, says it'll assume that for now, and then very carefully checks its arithmetic on the wrong problem.

Theo The reasoning is diligent. The premise is wrong.

Maya Two practical findings from it. Calibrate on reasoning data if your quantization method relies on calibration, like GPTQ. And don't expect longer thinking to rescue a damaged model: when aggressive quantization made the models write more, the extra tokens did not reliably win back the accuracy.

Theo Which connects to a preprint from the end of September called RATIO, from Chengzhu Bao, Yulun Zhang and colleagues at Shanghai Jiao Tong University. They looked at what three-bit quantization does to the length of reasoning, not just accuracy.

Maya I'd guess longer. A damaged model dithers?

Theo Much longer. On the four billion Qwen three in thinking mode, on the AIME maths competition, the full model scores about seventy-two percent with an average chain of seventeen thousand tokens. At three bits with GPTQ, seventeen percent, and the chain grows to thirty-two thousand tokens. On a distilled one and a half billion model, averaged across benchmarks, accuracy goes from sixty to thirty-four percent and the reasoning gets about three times longer.

Maya So it's less accurate and it writes far more. Whether each answer ends up costing more depends on how much cheaper each token got.

Theo The paper doesn't measure that, but its example shows the failure. The model reaches the right number, writes "wait, let me double-check that", and keeps going until it degenerates into repeated fragments. The authors argue, reasonably, that measuring accuracy and speed per token alone is incomplete for reasoning models.

Maya Last paper in this section, a preprint from August: Chenxi Zhou, Pengfei Cao, Jinyu Ye and colleagues, mostly at the Chinese Academy of Sciences' Institute of Automation, looking at why the damage varies so much. Over eight hundred evaluations. Their measure is how much of each model's ability above chance survives quantization.

Theo And the headline?

Maya Four bits usually preserves performance, two bits usually destroys it, and three bits is where everything depends on the details. But the four-bit rule has an exception that matters for small local models. On Qwen three with GPTQ, at four bits, the fourteen billion model kept ninety-eight percent. The zero point six billion model kept seventy-one.

Theo And at two bits?

Maya Under ten percent for every size, fourteen billion included. And at three bits the method matters enormously: on the eight billion Llama three, plain rounding kept thirty-nine percent of its overall ability, while a stronger method kept eighty-eight.

Theo Which is why, for a small model, I'd measure the tasks I actually care about before trusting any bit-width rule.

Maya And there's a way to do better than any of this, which is to stop compressing models after the fact and train them to be compressed.

Training for low precision: Gemma 3 QAT, gpt-oss MXFP4, NVIDIA's QAD, ReQAT, QATFactory 53:17

Theo Most of the language-model methods so far are post-training quantization. Take a finished model, round its numbers, and repair what you can using a little calibration data. The alternative is to let the model see the rounding while it's still learning, so it can adapt to it. That's quantization-aware training, QAT.

Maya The idea behind Jacob's phone models in twenty eighteen. How does it work for a language model?

Theo The trick is that you don't need four-bit hardware to train for four bits. In the forward pass, each weight is rounded to the target format and immediately converted back to sixteen-bit. So the layer computes with the rounding error baked in, but on ordinary hardware. In the backward pass, the gradient pretends the rounding wasn't there, which is called the straight-through estimator. The model keeps a high-precision copy of its weights, and gradually nudges them towards values that survive rounding well.

Maya So the model learns to live with its own compression.

Theo Right. And the most visible use of it came from Google with Gemma three, in spring twenty twenty-five. They took each finished model and trained it for about five thousand more steps with quantization in the loop. And the target wasn't the original training data's labels. It was the probabilities from the uncompressed version of the same model.

Maya The full model as teacher, the compressed one as student.

Theo Which is distillation again, Hinton's soft targets, now used to repair quantization. Google's blog says this reduced the perplexity drop from four-bit quantization by fifty-four percent, measured with llama.cpp. The twenty-seven billion model went from fifty-four gigabytes in sixteen-bit to fourteen point one gigabytes at four bits, which Google says fits a single RTX 3090 with twenty-four gigabytes.

Maya With a caveat Google states itself.

Theo That figure is weights only. The KV cache for long contexts comes on top. And the evidence is thin: one relative perplexity number, no absolute scores, no task accuracies for the quantized models. I'd take it as "the method helps" rather than "the method is lossless".

Maya Now OpenAI's gpt-oss, from August twenty twenty-five, which is a different story. The model card says the mixture-of-experts weights were quantized during post-training to MXFP4, at four point two five bits per parameter. Those expert weights are over ninety percent of the parameters. The result is that the hundred and twenty billion parameter model ships as a checkpoint of about sixty-one gibibytes and fits on a single eighty gigabyte GPU.

Theo So four-bit was the release format. Though the card doesn't say how the quantization was done or compare against an unquantized version, so we know what shipped, not what it cost in quality. NVIDIA calls the teacher-guided version quantization-aware distillation, QAD. Their technical report, from Meng Xin and colleagues in March, makes the case with one table. They took their forty-nine billion parameter reasoning model, Llama Nemotron Super, and recovered it at four bits two ways. Plain QAT, which trains the compressed model on next-token prediction against the data. And QAD, which trains it to match the original model's probabilities. Then they measured both on held-out data.

Maya If both are trained to recover, how would you even tell them apart?

Theo That's the point of the table. Both had essentially the same loss against the data labels: point four zero eight and point four one six, the same as the original model. But the KL divergence from the original model was point three one one for QAT and point zero zero four for QAD.

Maya So both fit the data labels equally well, but only the distilled one stays close to the original model's probabilities.

Theo And it showed up in the results. On the AIME maths benchmark, QAD recovered almost all of the original score; QAT recovered less.

Maya Why would QAT fall short?

Theo Because a modern model isn't just the result of predicting data. It went through supervised fine-tuning, reinforcement learning, model merging. If you retrain on next-token labels, you pull it back towards something simpler. On a model trained heavily with reinforcement learning, the effect was striking: QAT actually did worse than not training at all. On AIME twenty twenty-five, plain quantization scored fifty-eight point seven, QAT forty-six point one, and QAD sixty-two, against sixty-three and a half for the original.

Maya So training the compressed model on data undid the reinforcement learning.

Theo That's the authors' reading, and it fits. And a corollary: QAD on code alone recovered the maths score fully. Sixty-two on AIME, the same as training on maths and code together. Even training on random tokens kept the model stable, though it didn't fully recover it. Which makes sense once you see that you're not teaching skills, you're asking the compressed model to behave like the original on whatever inputs you give it.

Maya Now, I want to add a counterweight, because NVIDIA also published a blog on their newer Nemotron three point five model, and the result there is more mixed. They deliberately quantized intermediate checkpoints aggressively, and on one of them QAD brought the median score recovery from about ninety-six to over ninety-nine percent. But on the final release, which used a safer recipe, median recovery was ninety-nine point two four percent for plain post-training quantization and ninety-eight point nine seven for QAD.

Theo So training made it slightly worse.

Maya In the median, yes, while improving some tasks and hurting others. Long-context performance went down; one agentic coding benchmark went up past the original. My takeaway: QAD lets you use a more aggressive format than you could otherwise get away with. If a careful post-training recipe is already near lossless, there may be little left to recover.

Theo One diagnosis from this year is worth knowing. ReQAT, from Janghwan Lee and colleagues at ICML, looked at reasoning models in fully four-bit floating point. Quantization rarely changes the model's top choice on confident tokens, mostly digits and operators. What it does is move a little probability onto the alternatives. When you sample, sometimes you get one, and in maths one wrong digit is enough. Their fix, which includes training the model to be sharper on confident tokens, worked well on one fourteen billion model and less well on an eight billion one.

Maya And QATFactory, a framework from Weili Xu and colleagues revised at the start of this month, has two lessons for anyone doing this. The best training setup isn't always the one that matches deployment: for NVFP4, training with sixteen-bit activations did as well as training with four-bit ones, while for MXFP4 including the four-bit activations raised the average from fifty-eight to sixty-six percent. And a checkpoint can train successfully and then fail to load in the inference engine, or be rounded differently on export. Evaluate the exported model, not the training-time numbers.

Theo Which sets up the next section nicely, because the gap between "this format exists" and "this format runs well on your machine" is wider than most people expect.

Your GPU vs the format: an RTX 4090 test of FP8, AWQ and NVFP4 1:01:34

Maya Before we go back to pruning, a practical point that catches a lot of people. You can quantize a model to a format your GPU doesn't support. It will usually still run. It just won't run the way the format's marketing implies.

Theo Because there are two different things a quantized format can save. It can save memory, by storing the weights in fewer bits. And it can save arithmetic, if the GPU can multiply numbers in that format directly.

Maya And for the second one, you need hardware. Eight-bit floating-point maths arrived on NVIDIA's consumer cards with the Ada generation, the RTX 4090. vLLM's documentation states it plainly: FP8 computation needs compute capability eight point nine or newer. Four-bit floating-point maths, in formats like NVFP4, is a Blackwell feature: the RTX 5090 and the data-centre B-series.

Theo So what happens if you load an FP8 model on an older card, say an RTX 3090?

Maya vLLM added a path for that in twenty twenty-four, using a kernel called Marlin. It keeps the weights in eight bits in memory, converts them to sixteen bits on the fly inside the kernel, and does the multiplication in sixteen-bit. The pull request is careful about the claim: half the weight memory, and up to twice as fast in memory-bound situations. No faster arithmetic.

Theo And memory-bound means?

Maya Generating text for one user. To produce each token, the GPU has to read every weight once, and it does very little arithmetic per weight. So the speed is set by how fast you can move the weights out of memory. Fewer bytes, faster tokens. Processing a long prompt, or serving many users at once, is different: lots of arithmetic for every weight read. That's compute-bound, and there only native low-precision maths helps.

Theo We wanted to see this on real hardware, so we ran it on our own RTX 4090. The eight billion Qwen three, in vLLM, five ways. The original sixteen-bit. An eight-bit floating-point version using the card's native FP8 maths. The same eight-bit file again, but forced onto the Marlin weight-only path. A four-bit integer AWQ version. And an NVFP4 version, a format the 4090 can't compute in.

Maya Did the NVFP4 model load?

Theo It loaded and ran. And vLLM printed a warning that says it all: "Your GPU does not have native support for FP4 computation but FP4 quantization is being used. Weight-only FP4 compression will be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads."

Maya So the file is four-bit, the maths is sixteen-bit.

Theo Exactly. Now the numbers. Generating for one user, the sixteen-bit model ran at about fifty-nine tokens a second. Native eight-bit, eighty-seven. The four-bit AWQ model, a hundred and fifty-two. And the NVFP4 model, on its weight-only fallback, a hundred and forty-four.

Maya So the fallback didn't hurt single-user speed at all.

Theo For single-user generation, four-bit weights were roughly two and a half times faster than sixteen-bit, whether or not the card could do four-bit maths. That's the memory-bound regime: what matters is bytes moved.

Maya And processing a long prompt?

Theo A four-thousand-token prompt. Sixteen-bit, about ten thousand tokens per second. Both four-bit models, also about ten thousand. No faster at all, because the arithmetic was still sixteen-bit. The only configuration that sped up prompt processing was native FP8, at about fifteen and a half thousand, roughly one and a half times faster.

Maya And the same eight-bit file on the weight-only path?

Theo This is the cleanest comparison: the identical file, only the kernel changed. Forced onto weight-only Marlin, it was actually a bit faster for single-user generation, ninety-eight tokens a second against eighty-seven. But prompt processing fell by thirty-eight percent, to below the sixteen-bit model. And with thirty-two users at once, throughput dropped by thirteen percent.

Maya So the same model file is faster or slower depending on which kernel the engine picks, and which kind of work you're doing.

Theo Which is the "BF16 or Death" paper's advice in miniature: weight-only for latency, weights-and-activations for throughput.

Maya Let me put the caveats on that. One machine, one model, one engine version, one sequential pass through each configuration, with the clocks not locked. The single-user speeds were estimated by subtracting a separately measured prompt time. And the quality check was eight simple arithmetic questions, which every version got right, and which tells you almost nothing about quality. The different quantized files also aren't numerically identical. It's a demonstration of the mechanism, not a ranking.

Theo And memory: vLLM reported about fifteen gibibytes of weights for sixteen-bit, eight point eight for eight-bit, five point seven for AWQ, and six for NVFP4.

Maya Why is NVFP4 bigger than four-bit AWQ?

Theo Because a four-bit format isn't four bits once you count the scales. NVIDIA's own description of NVFP4 puts it at four and a half bits per value: every sixteen numbers share an eight-bit scale, and there's a second scale for the whole tensor. MXFP4 uses blocks of thirty-two, so about four and a quarter. GGUF's Q4_K_M is about four point nine on the eight billion Llama three point one. The name of the format tells you the size of the payload, not the size of the file.

Maya So the practical rule is: look at what the engine actually does on your card. vLLM logs which kernel it selected for each layer type at startup. If you see Marlin, or a warning like ours, you're getting the memory saving and the memory-bound speed-up, not faster arithmetic.

Theo And if your workload is one person chatting, that may be exactly what you want. A four-bit model on a 4090 doesn't need four-bit tensor cores to be two and a half times faster at generating text.

Pruning LLMs: SparseGPT, Wanda, 2:4 sparsity, LLM-Pruner, SliceGPT, deleting layers 1:08:46

Maya Back to the other way of shrinking a model: deleting parts of it. Pruning worked beautifully on vision networks in twenty fifteen. When language models got large, the obvious question was whether you could do the same, without the days of retraining Han's method needed.

Theo And the first answer was a hard no, if you use the simplest method. In January twenty twenty-three, Elias Frantar and Dan Alistarh, who also wrote GPTQ, published SparseGPT. Their baseline was magnitude pruning: remove the half of the weights with the smallest values, in each layer. On the hundred and seventy-five billion parameter OPT model, perplexity went from eight point three five to forty-three thousand.

Maya Forty-three thousand. Next-token prediction is gone.

Theo On the word-prediction benchmark LAMBADA, accuracy went to essentially zero. SparseGPT, at exactly the same fifty percent sparsity, kept perplexity at eight point two one and the zero-shot average where it was. Same number of weights removed. One model is broken, the other is fine.

Maya What's the difference?

Theo The same idea as GPTQ, from the same people. When you remove a weight, you adjust the remaining weights in that row to compensate, using second-order information about how the layer's inputs are correlated. It goes back to LeCun's optimal brain damage, which we talked about at the start, scaled up to a hundred and seventy-five billion parameters, in under four and a half hours on a single GPU.

Maya That eight point two one is a hair below the dense model, but only on one dataset. On the other two, as the authors point out, the pruned model was slightly worse, as you'd expect. Another result from that paper: half the weights removed plus four-bit quantization uses the same memory as three-bit quantization alone. And it was better: on the big model, eight point two nine versus eight point six eight for three-bit GPTQ.

Theo Which is a nice example of combining methods instead of pushing one too far.

Maya Then, in twenty twenty-four at ICLR, Wanda, from Mingjie Sun, Zhuang Liu, Anna Bair and Zico Kolter. And Wanda is the result I find most counterintuitive in pruning.

Theo Why?

Maya Because it skips SparseGPT's whole compensation step. It doesn't update any remaining weight at all. It just changes what you measure. Instead of looking at the size of a weight, it multiplies the weight's size by the typical size of the input it receives. A small weight on a huge input matters; a large weight on an input that's always near zero doesn't.

Theo Which is the outlier story again. A few input features are enormous, so their weights carry the model.

Maya Right, the same insight that drove LLM.int8 and AWQ. And the result: on Llama seven billion at half sparsity, Wanda got seven point two six perplexity against SparseGPT's seven point two two. Essentially the same. And the scoring step that decides which weights to remove took about half a second instead of two hundred.

Theo So the expensive part of SparseGPT bought almost nothing, at least at fifty percent.

Maya At fifty percent. At eighty percent sparsity SparseGPT wins clearly, so compensation matters when you push hard. And I want to flag the other caveat: on the largest Llama models, Wanda at half sparsity matched the dense model's zero-shot average. But on MMLU, which needs knowledge, the thirteen billion Llama two went from sixty-one point seven to fifty-five.

Theo Which matches what we heard in the "truth is rarely pure" paper. Pruning loses facts first.

Maya Now, there's a practical problem with all of this, and it's the one that makes pruning less popular than quantization in practice.

Theo The speed.

Maya Yes. Removing half the weights at random positions doesn't make a GPU twice as fast. The weights are scattered, so you still have to store where the zeros are, and GPU arithmetic doesn't skip scattered zeros. The exception is a pattern NVIDIA hardware supports since Ampere, called two-to-four sparsity: in every group of four weights, exactly two are zero. That runs on the tensor cores of NVIDIA's Ampere generation and newer. Wanda's authors estimated about one point six times faster matrix multiplications for those layers, from a kernel simulation, and measured one point two four times end to end on Llama seven billion.

Theo And two-to-four is more damaging to quality, because you can't choose freely which weights to keep.

Maya On Llama seven billion, perplexity goes from about seven at unstructured half sparsity to about eleven at two-to-four.

Theo So the alternative is to remove whole structures: entire rows, attention heads, or layers. Then the model is genuinely smaller and runs faster on any hardware. But the risks are greater.

Maya LLM-Pruner, from Xinyin Ma and colleagues at the National University of Singapore in twenty twenty-three, has the clearest example of why. In a transformer, components are coupled. If you remove one channel in one layer, the layers that read from it and write to it have to lose the matching rows and columns. LLM-Pruner groups these dependencies together and removes them as a unit.

Theo And if you don't?

Maya Removing twenty percent of Llama seven billion with dependencies respected gave perplexity nineteen. Without respecting dependencies, sixty-eight thousand.

Theo Then SliceGPT, from Saleh Ashkboos, James Hensman and colleagues at ETH Zurich and Microsoft, at ICLR twenty twenty-four. The idea is elegant: compute a principal-components rotation once from calibration activations, so that most of the information sits in a smaller number of dimensions, then delete the rest. Every weight matrix becomes smaller, and you're left with an ordinary dense model. The practical result: slicing a quarter off the seventy billion Llama two let it run on five twenty-four gigabyte RTX 6000 cards instead of seven, and decoding got about sixteen percent faster. With a quarter removed, its zero-shot average dropped from seventy-six point six to seventy-three point six. Even at thirty percent removed, a short LoRA fine-tune brought it back to seventy-four point three.

Maya And one detail in that paper is easy to miss.

Theo The numerical precision of the rotation itself. They compute it once, from a hundred and twenty-eight calibration samples. Computing it in thirty-two-bit floating point gave the seventy billion Llama two a perplexity of seven point zero one. In sixty-four-bit, four point eight nine. So in a method meant to make the model smaller, the one-off calculation needed double precision.

Maya The bluntest form of structured pruning is just deleting entire layers. Andrey Gromov, Daniel Roberts and colleagues, from Meta and elsewhere, "The Unreasonable Ineffectiveness of the Deeper Layers", at ICLR twenty twenty-five. They removed contiguous blocks of deep layers, healed the model with a short LoRA fine-tune, and found that on knowledge questions like MMLU, the seventy billion Llama two barely changed until about half the layers were gone. Then it collapsed to random guessing.

Theo Half of the layers. That's remarkable.

Maya But reasoning didn't have a plateau. Grade-school maths, GSM8K, got worse with any pruning at all. So the deep layers aren't useless; they're doing something that knowledge benchmarks don't measure.

Theo Cheap repairs help without fully restoring it. One from Huawei this August replaces each deleted block with a tiny linear correction fitted without training. On the eight billion Llama three point one with a quarter of its layers removed, average accuracy went from forty-three to fifty-nine, against seventy-one for the original. On some models it made things worse.

Maya And the extreme end. In September, a preprint from Hyeondo Jang, Namhoon Lee and colleagues pushed unstructured pruning to ninety-five and even ninety-nine percent, with substantial retraining. The seven billion Llama two at ninety-five percent sparsity had perplexity of about thirteen and a half, against five and a half dense. On one RTX 3090, with their sparse kernel, it decoded at a hundred and seventy-six tokens per second versus fifty-four dense, in two point two gigabytes instead of thirteen point six.

Theo And the catch?

Maya Maths. The eight billion Qwen three on GSM8K: eighty-seven point six percent dense, twenty-eight percent at seventy percent sparsity, and about two percent beyond ninety. Their example is a pay calculation. The dense model computes overtime and gets four hundred and sixty dollars. The sparse model computes the overtime rate correctly, forgets to multiply it by the number of overtime hours, and answers four hundred and twelve.

Theo The multiple-choice scores held up far better than that. The arithmetic didn't.

Maya In these pruning experiments, fluent output survived damage that was already visible on facts and reasoning. Which capability fails first depends on the model and the test, but it's rarely the one perplexity shows you. So the most successful use of pruning for language models turned out to need a second ingredient.

Theo Retraining from the original model.

Prune, then distil: DistilBERT, Minitron, Sheared LLaMA, on-policy distillation, R1, capacity gaps, Boomerang 1:19:12

Theo Pruning and distillation work best together: remove a large part of a model, then teach what's left to behave like the original. Pruning to choose the shape, distillation to recover the knowledge.

Maya The ancestor is DistilBERT, from Victor Sanh, Thomas Wolf and colleagues at Hugging Face in twenty nineteen. They took BERT, removed half its layers, initialised the student from the teacher's remaining layers, and trained it on the teacher's soft predictions. Forty percent fewer parameters, sixty percent faster, and on the GLUE language-understanding benchmark seventy-seven against seventy-nine and a half for the original. For large language models, the clearest demonstration came from NVIDIA in twenty twenty-four: Minitron, from Saurav Muralidharan and colleagues. They took their fifteen billion parameter Nemotron four model, pruned it to eight billion and to four billion, and retrained with distillation from the original.

Theo And the headline number is about cost. The eight billion model needed ninety-four billion tokens of retraining. Their previous eight billion model, trained from scratch, used three point eight trillion. Forty times fewer tokens, and it scored better: sixty-four on MMLU against fifty-five.

Maya Though "forty times cheaper" leaves out the teacher, which had already been trained on eight trillion tokens. If you have the big model anyway, the small one is cheap. If you don't, it isn't.

Theo Two Minitron results change engineering practice. The first is about which shape to prune to. They compared removing whole layers, which is depth, against making every layer narrower, which is width. Immediately after pruning, a mixed depth-and-width model had the lower loss. After about two hundred steps of retraining, the ranking flipped, and width-only pruning won.

Maya So the model that looks best straight after pruning isn't the one that recovers best.

Theo Which means you can't choose your pruning strategy by measuring the damage. You have to retrain a little first and then decide.

Maya And the second result?

Theo The importance of distillation. At the same compute, they compared three four billion models. One trained from scratch, one pruned from the fifteen billion and retrained on ordinary next-token labels, and one pruned and distilled. On MMLU, the first two scored about twenty-four and a half percent. That's chance, for four-choice questions. The distilled one scored thirty-eight.

Maya So pruning without the teacher threw away nearly everything the pruned weights still knew, at least within that budget. Sheared Llama, from Mengzhou Xia, Danqi Chen and colleagues at Princeton, ICLR twenty twenty-four, adds one observation that I think is quietly important. After pruning the seven billion Llama two down to one point three billion, they continued training on the same mix of data the original had seen. But the pruned model didn't lose all kinds of knowledge equally. It lost more on some domains than others.

Theo So they rebalanced on the fly.

Maya They measured the loss on each domain as training went, and sent more data where the gap to the target was largest. By the end, the share of web text from the C4 dataset had gone from fifteen percent of the mix to forty-nine. GitHub code went from four and a half percent to under one.

Theo Which tells you what pruning hurt most.

Maya And the result: their one point three billion model, with fifty billion tokens of continued training, slightly beat TinyLlama, a model of similar size trained from scratch on three trillion. Fifty-one against fifty, on average across eleven tasks. Comparable, not dominant, but at a sixtieth of the training data.

Theo Now, distillation itself changed for language models, and it's worth knowing how. Hinton's original version matches the teacher's probabilities on the training data. For text generation, that has a specific problem: the student learns from text it didn't write. At deployment, it continues from its own words, mistakes included, which it never practised on.

Maya The fix is on-policy distillation: the student writes, and the teacher scores every token of what it wrote. GKD, from Rishabh Agarwal and colleagues at Google DeepMind, made this concrete under the title "Learning from Self-Generated Mistakes". MiniLLM, from Yuxian Gu and colleagues, added an argument about the direction of the comparison: a small student can't cover every possibility a large teacher considers, so they used reverse KL, which pushes it to concentrate on what the teacher finds most likely. The best-known distilled reasoning models went in a different direction, though. The DeepSeek R1 distilled models, released in January twenty twenty-five.

Theo And here's the surprise. "Distillation" there doesn't mean probabilities at all. It's plain supervised fine-tuning: Qwen and Llama models trained on about eight hundred thousand samples written by the big R1 model, and about a quarter of those aren't reasoning at all. No reinforcement learning on the students.

Maya And the comparison that made the case.

Theo DeepSeek took the same thirty-two billion Qwen base model and trained it with large-scale reinforcement learning for more than ten thousand steps. AIME twenty twenty-four: forty-seven percent. The distilled version, with nothing but fine-tuning on R1's outputs: seventy-two point six. The authors' conclusion is that for small models, distilling a strong reasoner beats trying to discover reasoning yourself.

Maya Thinking Machines Lab combined the two ideas in a blog post a year ago. They started with the eight billion Qwen three, fine-tuned on four hundred thousand reasoning examples to reach sixty percent on AIME twenty twenty-four. Continuing that way, they estimate, would take about two million examples to reach seventy. With on-policy distillation, it reached seventy in about a hundred and fifty steps.

Theo Their cost estimates are nine to thirty times cheaper, depending on what you count. Their example: a student asked about ice cubes in a frying pan counts the ice cubes, instead of noticing that they melt. The teacher's feedback lands on the token where the reasoning went wrong, not on the final answer.

Maya Now, is a bigger teacher always better? Apple's Dan Busbridge and colleagues, "Distillation Scaling Laws", at ICML twenty twenty-five, fitted scaling laws to hundreds of teacher and student training runs to answer exactly that.

Theo And the answer is no.

Maya There's a capacity gap. Past a point, a stronger teacher produces a worse student, because the student can't follow what the teacher is doing. And it's the teacher's capability that matters, not just its size.

Theo Their other finding is about economics. Distillation beats training from scratch when the teacher already exists, or when you'll distil many students from it. If you have to train the teacher just for one student, it often doesn't pay.

Maya Last idea in this section, and it's one I didn't expect to exist. Boomerang distillation, from Sara Kangaslahti, David Alvarez-Melis and colleagues at Harvard, ICLR twenty twenty-six. Distil a small student from a larger teacher, initialised from some of the teacher's layers. Then take the student and put some of the teacher's original layers back in, without any training.

Theo And you get a working model of intermediate size?

Maya That's the result. On the four billion Qwen three, the hybrids interpolate fairly smoothly between the student's and teacher's performance. One distillation run gives you a whole family of sizes.

Theo Which brings us to the models where compression decisions are most interesting right now: mixture of experts.

Mixture of experts: compress the experts, protect the always-on path 1:27:46

Maya Mixture of experts changes the compression problem, so let's be precise about why. In a mixture-of-experts model, each feed-forward block is replaced by many smaller blocks, the experts, plus a small router that picks a few of them for each token. The attention, the router itself and, in many designs, a shared expert run for every token. The embedding table is a lookup, so each token reads just its own row.

Theo So there are two very different kinds of weight in the same file.

Maya Yes, and their proportions are extreme. Amazon's HOPE paper, from last month, points out that in the one hundred and twenty-two billion Qwen three point five, a hundred and sixteen billion of the parameters are in the expert pool. OpenAI's gpt-oss card says the expert weights are over ninety percent. So most of the bytes are in weights any single token barely uses.

Theo Which suggests an obvious strategy. Compress the experts hard, and leave the always-on parts alone.

Maya And the evidence mostly supports it. The most extreme demonstration came early: QMoE, from Elias Frantar and Dan Alistarh again, in twenty twenty-three. They took Google's Switch Transformer with one point six trillion parameters, compressed only the experts to three levels, minus one, zero and plus one, and then packed those with a custom compression scheme. The checkpoint went from three point one terabytes to a hundred and fifty-nine gigabytes. Less than one bit per parameter: point eight zero seven.

Theo Less than one bit. How does that even work?

Maya Because ternary experts are very repetitive. Most of the values are zero, so a dictionary-style compression packs them far below the two bits you'd need naively. The cost was real but modest: validation loss went from one point one eight to one point two six. Keep in mind that's an older encoder-decoder model on a language-modelling loss, not a modern chat model on reasoning tasks.

Theo That paper also names the coverage problem. Each expert sees only a small subset of the inputs, so to calibrate all of them, you need far more data in total. Even so, for a few layers the GPTQ maths broke down and they fell back to plain rounding. With a rare expert, the calibration data may simply not contain the kind of input it specialises in.

Maya Now the other half of the strategy, protecting the always-on parts. The cleanest test is QuantMoE-Bench, from Pingzhi Li and colleagues in twenty twenty-four. On DeepSeek MoE sixteen billion, quantizing everything to two bits with GPTQ gave an average accuracy of forty-nine point four percent across six tasks. Keeping just the attention at four bits barely moved the average size, to two point zero six bits, and brought accuracy to fifty-six point eight.

Theo And the shared experts, the ones every token uses?

Maya Protecting those as well brought it to sixty-one and a half, at two point one two bits on average.

Theo So twelve points of accuracy for about a tenth of a bit.

Maya The parts every token touches are a tiny fraction of the bytes and have a disproportionate effect. So if you're choosing or making a quant of a big mixture-of-experts model, check what precision the attention and shared experts got, not just the headline bit count. Now, an obvious idea: the router picks some experts far more often than others. So give the popular experts more bits, or delete the unpopular ones.

Theo And that's where the evidence turns against intuition.

Maya On Mixtral, QuantMoE-Bench gave four selected experts per layer four bits and the rest two bits. Picking those four by frequency scored sixty-three point eight. Picking them at random scored sixty-three point seven, with a spread of half a point across random runs. So frequency was no better than chance.

Theo And for deletion it's worse.

Maya The HOPE paper, from Alex Tseng, Stefano Soatto and colleagues at AWS, compared ways to remove half the experts in each layer without retraining. On GLM four point five Air, calibrated on coding prompts, removing the least frequently used experts dropped the score on SWE-Bench Pro, a benchmark of real software tasks, from twenty-eight point seven to zero.

Theo From twenty-eight point seven to nothing.

Maya While their method, which also accounts for which experts tend to be used together, roughly kept the original score. It's not uniformly the best method across their tests, but the frequency result is the warning. An expert that's rarely used may be the only one that handles a particular kind of input. Delete it, and that capability is simply gone.

Theo Which matches what we saw with pruning in dense models. The damage concentrates in things the benchmarks average over.

Maya And then the practical comparison. MoE-XBench, a preprint from August by Afsara Benazir, Lingjuan Lyu and colleagues, ran these techniques in llama.cpp on the thirty billion Qwen three A3B. Pruning nineteen percent of the experts raised perplexity by twenty-eight and a half percent. Q4_K_M quantization, which made the model seventy percent smaller, raised it by point seven percent.

Theo So, on that model, four-bit quantization did far less damage while saving far more space.

Maya Far less. And on tasks, pruning twenty-eight percent of the model's size dropped MMLU by ten points and HumanEval by thirteen. So if you have a large mixture-of-experts model and need it smaller, quantize the experts first and prune them last, if at all.

Theo One more finding from that benchmark. Adding an eight-bit KV cache on top of the four-bit weights, at thirty-two thousand tokens of context, saved one point three gigabytes out of twenty-one. And decoding ran at sixty-one percent of the speed.

Maya So you saved six percent of memory and lost almost forty percent of the speed. At two thousand tokens of context the slowdown nearly disappeared. And their one-bit weight format halved the memory again but ran prompt processing at about forty percent of the speed of Q4_K_M.

Theo There's one more consequence of mixture of experts, and it's the reason that RTX 5090 can run DeepSeek at all. If only a few experts are used for each token, the rest don't need to be on the GPU. They don't even need to be in memory.

Maya Which brings us to where the weights live.

Lookup memory and SSD streaming: LLM in a Flash, DeepSeek V4.1's Engram, Qwen3.8-Next's n-gram tables 1:35:02

Theo Up to now, shrinking has meant making the model smaller. This last technical section is about something different: leaving the model big, and being clever about where each part of it is stored.

Maya The starting point is a paper from Apple, "LLM in a Flash", from Keivan Alizadeh and colleagues, first posted in December twenty twenty-three. The question was how to run a model larger than your memory by keeping the weights on flash storage and reading only what each token needs.

Theo And the first lesson in it is about what an SSD's speed actually means.

Maya On their M1 Max, reading a whole model sequentially ran at six point one gigabytes per second. Reading only the scattered pieces that a token needed ran at one point two five. Storage is fast at long reads and much slower at many small ones. So they did something that sounds wasteful: they bundled related weights together and deliberately read more than needed, in bigger chunks, then threw the extra away.

Theo Reading more data to finish sooner.

Maya And it worked. The combination of predicting which weights would be needed, caching recent ones and bundling cut per-token storage latency on a seven billion model from about two seconds to under a tenth of a second. That depends on most of the model's neurons being inactive for any given token, which was naturally true of the OPT model they used. For Llama two they had to modify the model to make it that way, and that step alone cost about three points on MMLU.

Theo Now the same idea for mixture of experts, where it's a much more natural fit. The router tells you exactly which experts each token needs. And that's what the RTX 5090 report from the start of the episode does.

Maya Let me give the setup again. A single RTX 5090 with about thirty-two gigabytes, a hundred and twenty-six gigabytes of system memory, and an NVMe drive. DeepSeek V four point one Flash with its experts in MXFP4. The experts live on the drive, with a cache of them in system memory and another on the card.

Theo And the numbers?

Maya Five point one tokens a second on new content. The author also measured the ceilings: about six with every expert already in system memory, so no disk reads at all, and twenty-one only when everything it needs is resident on the card. And the most useful thing in that report is the analysis of where the time goes. With a seventy-two gibibyte host cache, each fetch spent about twenty percent of its time computing, twenty-six percent moving data over PCIe, and fifty-four percent waiting on the drive. But the drive was only running at forty-three percent of what it can deliver.

Theo So the drive is the bottleneck, but it isn't busy.

Maya Because of the router. A layer can't ask for its experts until its router has run, and the router can't run until the previous layer is finished. So only about four disk requests are in flight at a time per layer. The drive spends most of its time waiting for the model to decide what it wants.

Theo Which is the fundamental difference from ordinary storage workloads. The reads depend on the computation. There's a second report, from a company called Argonaut Labs in September, running the same model on an M5 Max Mac with a hundred and twenty-eight gigabytes. Their finding was about prompt processing. The upstream engine processed the prompt one layer at a time and read all three hundred and eighty-four experts of each layer, whether the prompt's tokens used them or not. For their test, it read three hundred and two gigabytes when the model actually touched a hundred and forty-nine. Reading only the routed experts made prompt processing one point seven times faster on one drive. And with three drives, two point seven times. But again with a twist: in some of their tests, reading selectively lost, because waiting for the exact selection gives up the chance to load the next layer while the current one computes.

Maya A paper from the start of this month, SpAx, from the University of Toronto and LG, shows the same limit on an RTX 4090 reading weights from an SSD. Their method needed far fewer bytes, but its small, scattered reads made the drive move about one and a half times what was actually needed, and on a four-bit Llama a simpler method that needed more data was faster.

Theo Now the other thing in that DeepSeek file, the lookup table.

Maya Engram. It's from a DeepSeek and Peking University paper, Xin Cheng and colleagues, in January this year. The idea starts from an observation: a lot of what a language model does is recognising short, fixed patterns. Names, common phrases, multi-token words. A transformer reconstructs those with computation, layer after layer, every time.

Theo So why not just look them up?

Maya That's the proposal. Take the last few tokens, two, three, four of them, hash them into an index, and fetch a learned vector from a huge table. Each row of the table is a learned embedding for whatever patterns hash to it. The model then decides, through a gate, how much to trust what it retrieved, and the gate can switch it off when the context makes the stored association wrong.

Theo Like an index in the back of a book. You don't read the whole book, you jump to the page.

Maya And the cost is constant: a handful of rows per token, whether the table has a million rows or billions. In their controlled comparison, at the same total and active parameters, replacing some experts with Engram memory improved BIG-Bench Hard from fifty point nine to fifty-five point nine. Code and maths improved too, not just facts, which they argue is because the early layers no longer have to spend effort reconstructing phrases.

Theo And here's why it matters for this episode. The table's addresses depend only on the input tokens. You know which rows you'll need before the model even starts on that token.

Maya Unlike experts, which you only know after the router runs. So you can prefetch Engram rows while the GPU is busy with earlier layers. They tested a hundred billion parameter table kept entirely in host memory, and the throughput penalty was at most two point eight percent.

Theo So it's a part of the model that doesn't need to be near the GPU at all.

Maya With one caveat I want to be very clear about, because it's tempting to think it's disposable. When they turned off the Engram output after training, the model kept only twenty-nine to forty-four percent of its performance on factual knowledge benchmarks. The facts really are stored there.

Theo And then DeepSeek V four point one Flash shipped it. Their report gives the ledger: five hundred and fifty-two billion backbone parameters, plus a hundred and ninety-six billion Engram parameters, in two modules, using two-, three- and four-token patterns, stored in FP8.

Maya And activating about eight billion parameters per token during prompt processing and sixteen billion during generation.

Theo So the thing that sounds like a seven hundred and fifty billion parameter model does about sixteen billion parameters of work per generated token.

Maya DeepSeek's own deployment prefetches the rows from host memory in the background. The 5090 port keeps the table on disk and reads it from the host. And DwarfStar's ds4, a separate engine with its own GGUF builds, makes that the rule: the hundred and eighty-nine gibibyte table stays on the SSD even on a machine big enough to hold everything else, and the engine reads only the rows each token needs.

Theo Which leads to a nice piece of arithmetic. Their two-bit build is three hundred and forty-one gibibytes and their four-bit build is four hundred and eighty-three. Doubling the bits doesn't double the file, because the table doesn't change.

Maya There's also a set of KV cache techniques in the V four point one report that are worth one sentence each, because they're about shrinking the other big memory cost. Several layers share one global KV cache instead of each keeping its own. The main cache is stored in a four-bit floating-point format, with the report saying explicitly that this reduces storage rather than speeding up the maths. And the local attention cache is kept at eight bits, because it turned out to be sensitive.

Theo The result: about eight hundred and ninety bytes of global cache per token, roughly a quarter of the previous version. And for prompt caches saved to SSD for reuse, an eighth of the previous size.

Maya Now Qwen, which arrived at a very similar idea. The Qwen team's architecture paper for Qwen three point eight Next, from late August. A hundred and twenty-five billion parameters, six billion active per token, plus fifty-one billion parameters of n-gram embedding tables, held off the accelerator.

Theo Same idea: short sequences of tokens as keys into tables.

Maya And they make a deliberate placement choice: one table, at layer two. Early enough to help, but late enough that the lookup can be fetched from host memory while layer one computes. In their ablation, adding the table at layer two lowered loss and raised the average task score from about forty-five and a half to forty-eight.

Theo And they also report where it didn't help.

Maya When they held the total parameter count fixed, swapping experts for n-gram memory showed no clear improvement on downstream benchmarks. And making the tables bigger kept lowering loss but didn't reliably improve tasks.

Theo So lookup memory is a real new axis, but it's early. It moves knowledge into a place that's cheap to store and cheap to read. Whether it's better than spending those bytes on experts is still being worked out.

Maya And it changes what "model size" means. That DeepSeek model has a parameter count, a file size, an amount that needs to be on the GPU, an amount read per token, and an amount of computation per token, and they're all different numbers.

Theo Which is the right moment to go back to that machine.

Back to the 5090, and how to choose 1:45:57

Maya So, back to that machine. One RTX 5090 with thirty-two gigabytes of video memory, a hundred and twenty-six gibibytes of system memory, an NVMe drive. And a model with five hundred and fifty-two billion parameters in its main network, plus a hundred and ninety-six billion in a lookup table.

Theo Start with the bytes per weight. The experts are in MXFP4, about four and a quarter bits each. The report describes its file as a lossless repack of the blocks DeepSeek released, so this isn't a community re-quantization.

Maya Then the experts themselves. Each token uses only a few, so most of them can live on the drive, with caches in system memory and on the card. That's why the speed depends so much on what you ask: five tokens a second on new content, and twenty-one only when everything it needs is already on the card.

Theo And the drive isn't saturated, because the model can't request a layer's experts until that layer's router decides. The limit is the chain of dependent decisions, not the drive's headline bandwidth.

Maya Then the lookup table, kept on disk and read from the host. Its rows are chosen from the input tokens alone, so they can be fetched early. It holds a lot of the model's factual knowledge, and it costs a few small reads per token.

Theo And the model's own design shrinks its KV cache, by sharing it across layers and storing the main part in four-bit floating point.

Maya So several kinds of shrinking, stacked: fewer bytes per weight, fewer weights active per token, weights and lookup memory moved to cheaper storage, and a smaller cache. Nothing in that machine was pruned or distilled.

Theo And it's one person's report on one machine, which we haven't reproduced.

Maya If you need to choose a method, start by asking which resource has to shrink. If it's the memory on your graphics card, weight-only quantization is the tool, and for one user it will usually make generation faster too. If it's the cost of serving many users, the arithmetic has to get cheaper, which means a format your hardware computes natively. And if you need a genuinely small, fast model, that's pruning plus distillation, from a teacher.

Theo A few rules held up across these sources. At a fixed memory budget, a larger model at around four bits usually beats a smaller one at eight. Around four bits, good methods are close to lossless on big models and less so on small ones. Below three, the method matters enormously. Spend bits where every token goes, and in a mixture of experts don't delete experts just because they're rarely used.

Maya Calibrate on data like yours. Measure KL divergence against the original, the tasks you actually care about, and, for reasoning models, how long the answers get. And check speed on your own hardware, at your own context length and batch size.

Theo The shift I didn't expect is how much the field has moved from compressing models after the fact to designing them for it. Gemma shipped quantization-aware checkpoints. DeepSeek trained its model to cope with a four-bit cache and built a lookup memory whose rows can be fetched ahead of time. Qwen put its table at layer two so the fetch could hide behind layer one.

Maya And the oldest idea in the episode still holds. LeCun in nineteen eighty-nine, Hinton in twenty fifteen: what a model knows isn't the same as the weights it happens to use. Every method we covered is a way of finding out how much of a model you can take away before the knowledge goes with it.

Theo And the answer, every time, depends on what you measure.