Demonstrates that a 2016-era Xeon E5-2680 v4 with no AVX-512 or AMX can serve as an effective MTP drafter for Gemma 3 27B. Argues that because the 4B-active-parameter drafter is memory-bandwidth-bound rather than compute-bound, the Broadwell chip's quad-channel DDR4 (~60-70 GB/s) is sufficient to feed a modern GPU verifier and net 2-3x end-to-end throughput.
Walks through the arithmetic showing a 4B int8 model requires ~4GB of weight reads per token, capping the Xeon at ~15 tokens/sec theoretical based on its 60 GB/s memory subsystem. The point is that newer compute features (AVX-512, AMX) are irrelevant when the workload is dominated by reading weights from RAM, which is why ancient server CPUs remain competitive for drafter roles.
Frames the result as evidence that Multi-Token Prediction (popularized by DeepSeek-V3 and now in Gemma) makes mixed-hardware inference practical: cheap used server CPUs handle proposal while expensive GPUs only verify. This shifts the cost calculus for self-hosted LLM rigs, since the drafter no longer needs to live on the same GPU as the verifier.
A post on point.free titled "A 10 year old Xeon is all you need" demonstrates running a Multi-Token Prediction (MTP) drafter for a 27B-parameter Gemma model on a 2016-era Intel Xeon — specifically the E5-2680 v4, a Broadwell-EP chip you can buy used for the price of a decent dinner. The trick: the drafter has roughly 4B active parameters (the "A4B" in 26B-A4B MTP), small enough that CPU inference is dominated by memory bandwidth rather than compute. The GPU still runs verification on the full 27B model, but the speculative path — the part that proposes the next several tokens — happens entirely on the Xeon.
The benchmark headline: a decade-old server CPU, with no AVX-512 and no AMX, can keep a modern GPU fed with draft tokens fast enough to make speculative decoding pay off. The author measured the E5-2680 v4's DDR4 quad-channel memory subsystem and found it delivers enough effective bandwidth — around 60-70 GB/s in practice — to push the small drafter model through at speeds that beat the GPU's own naive token-by-token verification cadence for the full 27B model.
The post walks through the math: a 4B-parameter draft model at int8 weights is roughly 4GB of weight reads per token. At 60 GB/s, that's a theoretical ceiling of ~15 tokens/sec on the CPU. In practice, with KV-cache and overhead, the author lands in the single-digit-to-low-teens tokens/sec range for the drafter — which is exactly the sweet spot where the GPU verifier can batch-validate 4-8 proposed tokens per forward pass and net out 2-3x end-to-end throughput vs. non-speculative decoding.
Speculative decoding has been the quiet headline of the past 18 months of inference work. The technique — propose tokens with a small fast model, verify with the big slow one — was published by DeepMind in 2022 and went mainstream in vLLM and TGI through 2024. Multi-Token Prediction (MTP), popularized by DeepSeek-V3 and now reproduced in Gemma's recipe, takes it further: the drafter is *trained alongside* the main model to predict multiple tokens ahead, getting acceptance rates that hand-rolled drafters never reach.
What this post nails is the unsexy implication: once the drafter is good enough, you stop needing a second GPU for it. Every prior speculative-decoding production setup assumed you'd run the drafter on a smaller GPU instance — a T4, an L4, an old A10. That's $0.50-$1.50/hour of compute sitting there proposing tokens. A used dual-socket Xeon node, fully amortized, is closer to $0.02/hour of electricity. The economics flip.
The community reaction on HN (303 points, top of front page) split predictably. The skeptics: "this only works because Gemma's MTP drafter is unusually small and well-trained — try it with Llama and you'll get nothing." Fair. The optimists: "this is the first real signal that inference disaggregation will happen across heterogeneous hardware, not just across GPU classes." Also fair. Both camps agree on the underlying point: memory bandwidth, not FLOPs, is the constraint that matters for the small-model half of speculative inference.
There's a deeper architectural point hiding here. The standard mental model for LLM serving is "GPU does everything." That made sense when models were monolithic and inference was a single forward pass. But MTP + speculative decoding splits the workload into two genuinely different shapes: a compute-bound verification pass (batch-friendly, GPU-native) and a latency-bound proposal pass (sequential, memory-bound, CPU-friendly). They want different hardware. Treating them as the same workload on the same accelerator is leaving performance and money on the table.
This also reframes what "old hardware" means. The E5-2680 v4 was a server CPU from 2016. It has 14 cores, DDR4-2400 quad-channel, and roughly the same memory bandwidth as a 2023 desktop Ryzen. The reason it works isn't that it's secretly fast — it's that the drafter workload is structured to extract bandwidth, not FLOPs, and bandwidth-per-dollar on used server hardware is excellent. Apple Silicon's unified memory gets a lot of attention for the same reason: it's bandwidth-rich for the price.
If you're running inference at any scale, the immediate audit: are you paying for drafter GPU time you don't need? Check your vLLM or TGI config. If you're using a speculative-decoding drafter and it's an MTP-style small head (Gemma 3, DeepSeek-V3, Qwen 2.5 MTP variants), there's a real chance the drafter could move to a CPU NUMA node on the same box, freeing the small GPU for actual inference or eliminating it entirely.
The second audit: your hardware procurement assumptions. Used dual-socket Xeon nodes — E5-2680 v4, E5-2690 v4, even Cascade Lake Xeons from 2019 — are absurdly cheap right now because hyperscalers are dumping Skylake-and-older inventory. If your inference workload has a bandwidth-bound component (drafter, embedding lookup, retrieval index in RAM), these boxes are stupidly cost-effective. The catch is they're power-hungry — 120-150W per socket — so total-cost-of-ownership math depends on your electricity price and rack density.
Third, and the one most teams will miss: this changes what "edge inference" can mean. A factory-floor box with a single consumer GPU and a beefy CPU can now run speculative decoding on a 27B-class model with usable throughput. The architectural separation between drafter and verifier is what makes that possible. If your roadmap includes on-prem or edge LLM deployment, the question is no longer "can we afford an H100?" — it's "can we afford an RTX 5090 plus a used Xeon?"
The broader trend is heterogeneous inference: different stages of the same model running on different hardware classes, coordinated by a serving framework that understands the workload shape. We're at the start of that — vLLM and SGLang are adding it piecemeal, and posts like this one are the empirical evidence that it works. Expect MTP-style drafters to ship with every frontier open model by mid-2026, and expect the serving frameworks to treat CPU-drafter + GPU-verifier as a first-class deployment topology by end of year. The 2016 Xeon isn't a curiosity — it's a preview.
Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.