The author demonstrates that a $40 decade-old Xeon E5-2680 v4 can run Gemma 3 27B at interactive speeds (~7-9 tok/sec) using MTP speculative decoding with stock llama.cpp flags. Their core argument is that the unlock isn't exotic engineering but an algorithmic shift — a 4B drafter proposes tokens that the 27B verifier checks in parallel, effectively making depreciated hardware viable for modern LLM inference.
The editorial frames the 20-40× bandwidth gap between HBM (1-3 TB/s) and DDR4 (~70 GB/s) as the historical reason CPU inference was unusable. Speculative decoding inverts the geometry by letting the small drafter live in L3 cache and batching verification into a single forward pass, meaning the weight-streaming cost is amortized across multiple tokens rather than paid per-token.
The author deliberately keeps the setup boring: DDR4-2400 ECC RAM, Q4_K_M quantization, stock llama.cpp build. The argument is that reproducibility and accessibility matter more than peak benchmark numbers — anyone with a used server CPU can replicate this without hand-tuned AVX kernels or specialized expertise.
A post on point.free titled "A 10 year old Xeon is all you need (for 26B-A4B MTP Drafters without GPU)" hit HN's front page with 435 points. The author runs Gemma 3 27B on a 2016-era Xeon E5-2680 v4 — a 14-core, 35MB-L3 server CPU you can buy used for under $40 — and gets interactive token rates without touching a GPU. The unlock isn't quantization tricks or hand-tuned AVX kernels. It's Multi-Token Prediction speculative decoding, where a small 4B-active drafter model proposes several tokens ahead and the 27B verifier checks them in a single parallel forward pass.
The configuration is deliberately mundane: DDR4-2400 ECC RAM, llama.cpp built with stock flags, Q4_K_M quantization for the verifier, a smaller MTP head as the drafter. No NUMA pinning gymnastics, no custom BLAS. The author reports ~7–9 tokens/sec on prose and faster on code, well past the threshold where a reader stops waiting on the model and starts waiting on themselves.
What makes the post resonate isn't the throughput number — plenty of people have run quantized LLMs on CPUs. It's the demonstration that algorithmic progress in the past 18 months has effectively retroactively upgraded hardware that was already paid off and depreciated.
The dominant story about LLM inference for two years has been: buy H100s, or rent them. Speculative decoding rewrites that story because it attacks the actual bottleneck on consumer-grade hardware — memory bandwidth, not FLOPS.
When a 27B model autoregresses one token at a time, every token requires streaming the full weight set through the cache hierarchy. A modern GPU has 1–3 TB/s of HBM bandwidth; a 10-year-old Xeon has maybe 70 GB/s of DDR4 across four channels. That 20–40× gap is why CPU inference traditionally felt like watching paint dry. Speculative decoding inverts the geometry: the drafter is small enough to live mostly in L3 cache and produces a candidate sequence, then the verifier does one bandwidth-heavy pass to check K tokens at once. Acceptance rates of 60–80% mean you amortize that expensive pass across multiple output tokens, and effective throughput rises by the acceptance multiplier.
The HN thread argues over the right baseline. One commenter points out that an M2 Max with unified memory hits similar numbers at lower power; another notes that a single used 3090 still beats this setup at multi-user serving. Both are correct and beside the point. The relevant comparison isn't to an optimal new machine — it's to the dual-Xeon workstation already sitting in someone's basement, or the $300 refurb on eBay, or the 2017 Dell PowerEdge a small business stopped depreciating last year. Hardware that was written off as too slow for modern AI is now, with the right decoding algorithm, fast enough for a single developer to live on.
There's also a quieter implication for the open-weight ecosystem. Google released Gemma 3 with the MTP head trained alongside the main model — not bolted on afterward. That's a deliberate bet that drafter-verifier pairs become the default deployment topology. Meta did similar work for Llama 3's speculative variants. The frontier labs are quietly turning every model release into a hardware-class-broadening event.
Finally, the security and sovereignty angle. A 27B-class model running locally on hardware you own, with no API call leaving the box, is the cheapest version of data residency that has ever existed. For consultants handling NDA'd code, lawyers on discovery, doctors on PHI, or anyone in a jurisdiction where US-hosted inference is a compliance problem, this is the first time the answer to "can we just run it ourselves?" is genuinely yes.
If you've been pricing GPU boxes for a side project or a small internal tool, stop and benchmark first. A used dual-socket Xeon with 128GB of DDR4 costs $200–400 on eBay. The total cost of ownership over two years — including electricity at US commercial rates — is roughly one month of a 4090-class cloud instance. For a single developer, the throughput is sufficient for coding assistants, document Q&A, log triage, and most internal RAG pipelines.
The sharp edge is concurrency. Speculative decoding gives you single-stream latency, not multi-stream throughput. If you put this behind a team of 20 engineers all hitting it simultaneously, the wall-clock arithmetic collapses — each request still demands a full memory-bandwidth pass per verification step, and there's no SM parallelism to hide behind. Treat the Xeon-plus-MTP stack as a personal compute unit, not a shared service, and the architecture decisions get much clearer.
For production multi-user inference, the H100 / MI300 / Trainium economics still dominate. But "production" was always a smaller slice of LLM use than the cloud bills suggested. The shadow LLM workload — devs running Continue against a local model, ops engineers using a coding assistant in airgapped environments, researchers iterating on prompts without metering — has been waiting for hardware that doesn't require a permission slip from finance. That hardware is now ten years old and sitting in surplus warehouses.
Practical recipe if you want to try this weekend: grab a Xeon E5-2680 v4 or better (anything with AVX2 and at least 8 cores), 64GB+ of DDR4 across all memory channels, pull the latest llama.cpp, download Gemma 3 27B Q4_K_M plus the MTP drafter, and set `--draft-max 8 --draft-min 4`. Expect ~8 tokens/sec on prose, more on structured output. Increase draft length until acceptance rate drops below 50%, then back off.
The pattern here will repeat. Every 18 months, an algorithmic advance — quantization, FlashAttention, paged KV cache, now speculative decoding with native drafters — lowers the hardware floor by roughly an order of magnitude. The half-life of "you need a new GPU for that" is now shorter than the depreciation schedule of the GPU itself. Plan accordingly: the model you can't run today on the box under your desk, you'll be running on it next year, and the year after that you'll wonder why anyone ever paid for the API.
Top 10 dev stories every morning at 8am UTC. AI-curated. Retro terminal HTML email.