Google's TurboQuant algorithm slashes AI memory costs by 6x, erasing $100B in chip stock value during a single trading session.

$100B Wiped: Google’s TurboQuant Cuts AI Memory Cost 6x

by
💡 Key Takeaways

  • Google’s TurboQuant algorithm compresses AI key-value cache from 16 bits to 3 bits, slashing memory usage 6x with zero accuracy loss
  • On Nvidia H100 GPUs, 4-bit TurboQuant delivers up to 8x faster attention computation — and the paper wiped roughly $100 billion from memory chip stocks
  • Open-source implementations in Python, Rust, and Apple Silicon are already live, with a vLLM integration pull request under review

One hundred billion dollars. That is how much vanished from memory chip stocks in a single trading session. The trigger was not a sanctions package, a supply chain disruption, or a catastrophic earnings miss.

It was a research paper. Google’s TurboQuant algorithm, formally presented at ICLR 2026, compresses the key-value cache — the working memory that large language models use to recall prior tokens — from 16 bits down to 3 bits. Memory usage drops by a factor of six, and according to Google’s own benchmarks, accuracy loss is zero.

Led by research scientist Amir Zandieh and VP and Google Fellow Vahab Mirrokni, TurboQuant requires no training and no fine-tuning. It applies instantly to existing models as a zero-shot compression technique.

The question now is whether a single algorithm can reshape the economics of AI inference — and what that means for the $100 billion already erased from chip valuations. The answer depends on how quickly the industry adopts it, and the early signals are moving fast.

Three Bits Rewrote the Economics of AI Memory

The bottleneck in running large language models is not compute — it is memory. When a model processes a long context window, it stores a key-value pair for every token it has already seen. An 8-billion-parameter model handling a 32K-token context burns through roughly 4.6 GB of VRAM on the KV cache alone.

Scale the model up and stretch the context longer, and the numbers spiral. The KV cache consumes the majority of available GPU memory, throttling the number of concurrent requests a server can handle.

The downstream effect is economic. When KV cache monopolizes GPU memory, cloud providers must deploy more hardware to serve the same number of users. That drives up cost per query and makes long-context applications — the features users increasingly demand — disproportionately expensive to run.

The key here is what TurboQuant does to that bottleneck. By compressing each value in the KV cache from 16 bits to just 3, it cuts memory consumption by 6x without measurable accuracy degradation.

The algorithm is zero-shot — no retraining, no fine-tuning, no model-specific adjustments. Deploy it on an existing model, and the memory bill shrinks immediately.

Key point: An 8B-parameter model at 32K context devours 4.6 GB of VRAM just for KV cache. TurboQuant compresses it 6x with zero accuracy loss and zero retraining — a deploy-and-save proposition.
TurboQuant AI memory compression visualization
Compressing 16-bit data into 3-bit representations — TurboQuant marks a turning point in AI memory economics — Generated by OpenAI DALL-E 3

PolarQuant Plus QJL: Inside the Two-Stage Pipeline

TurboQuant is not a single trick. It is a two-stage compression pipeline, each stage published as a standalone contribution before being fused together. The first stage, PolarQuant (presented at AISTATS 2026), applies a Walsh-Hadamard transform to randomly rotate the data vectors, then uses Lloyd-Max scalar quantization to reduce bit-width while preserving as much information as possible.

The second stage deploys QJL — Quantized Johnson-Lindenstrauss, originally published at AAAI 2025. QJL contributes just 1 additional bit per coordinate to correct the residual error left behind by PolarQuant. PolarQuant handles the heavy lifting of bit reduction, and QJL cleans up the margins with minimal overhead.

A critical finding is that keys and values should not receive equal bit allocations. The optimal configuration assigns 4 bits to keys and 2 bits to values — an asymmetric split that overturns the default assumption in prior quantization work.

Most previous quantization approaches treated keys and values symmetrically, allocating the same number of bits to each. TurboQuant demonstrated that keys carry more retrieval-critical information and benefit from higher precision, while values tolerate more aggressive compression. This asymmetry is not a minor tuning detail — it is the difference between accuracy preservation and noticeable degradation.

In simpler terms: Think of compressing a high-resolution photo by removing only the details invisible to the human eye. The image looks identical, but the file is six times smaller. TurboQuant does this for AI memory, and it does not need to study the photo first.

8x Faster Inference on H100 GPUs

Compression alone would be impressive. The speed gains make TurboQuant remarkable.

Running at 4-bit precision on Nvidia H100 GPUs, the algorithm accelerates attention computation by up to 8x compared to the 32-bit baseline. Less memory means less data movement, and less data movement means faster execution — a compounding benefit.

Zandieh and Mirrokni validated TurboQuant across three major open-weight model families: Gemma, Mistral, and Llama. They ran benchmarks on LongBench, Needle In A Haystack (NIAH), and ZeroSCROLLS — the standard gauntlet for long-context performance. At 6x compression, TurboQuant scored perfectly on the NIAH retrieval test, meaning the compressed model found every hidden fact in a sea of irrelevant text without a single miss.

Why this matters: on models with 3 billion parameters or more, 4-bit TurboQuant produces output indistinguishable from full-precision inference. This is not a research curiosity that works only on toy models. It performs at production quality on the architectures that power real-world AI services today.

Key point: 8x speed gain on H100, perfect NIAH scores at 6x compression across Gemma, Mistral, and Llama — 3B+ models produce results indistinguishable from full precision.
TurboQuant GPU semiconductor circuit board
TurboQuant achieved 8x speed gains on H100 GPUs — compressing memory and accelerating computation simultaneously — Photo by Harrison Broadbent on Unsplash
💡 Related Posts
$5.6M, 1 Trillion Parameters: DeepSeek V4 Drops Nvidia

$100 Billion Wiped: Wall Street Reacts to a Research Paper

Within hours of the paper’s publication, memory chip stocks cratered. The combined market capitalization of Samsung, SK Hynix, Kioxia, Micron, and SanDisk fell by approximately $100 billion. A single academic paper triggering a selloff of that magnitude is virtually unprecedented.

Individual stocks told the story in sharper detail: Micron dropped 3%, Western Digital fell 4.7%, and SanDisk plunged 5.7%. Wells Fargo analyst Andrew Rocha framed the threat bluntly: the algorithm “directly attacks the cost curve for memory in AI systems.”

Rocha’s assessment struck at a deeper anxiety: if AI models can do the same work with one-sixth of the memory, the entire demand forecast for high-bandwidth memory chips faces revision. That question alone was enough to rattle valuations across the sector.

The bears, however, face a counterargument. Compression algorithms have existed for years without fundamentally altering procurement volumes. Memory constitutes only one component of total data center costs, and AI infrastructure demand remains historically strong.

What the market priced in was not a revenue decline — it was a recalibration of future demand expectations. No actual reduction in memory shipments has materialized. For investors, the key distinction matters: the selloff was driven by a shift in forward-looking narrative, not by deteriorating fundamentals.

From vLLM to pip install: Open-Source Adoption Accelerates

Google has not yet released an official implementation. The open-source community did not wait. Independent developers have shipped working implementations in Python, Rust, and optimized Apple Silicon variants, and pip-installable packages compatible with HuggingFace models are already available.

The most consequential development is a pull request currently under review to integrate TurboQuant into vLLM — the most widely used open-source framework for serving large language models. vLLM powers inference infrastructure across organizations of every size, from early-stage startups to major cloud providers.

If the PR is merged, the memory savings would propagate rapidly across the AI industry. Organizations running vLLM would gain access to 6x memory compression through a configuration change rather than a hardware upgrade — a fundamentally different cost equation for scaling AI services.

In short, the community mobilization began before TurboQuant’s formal ICLR 2026 presentation. That timeline signals an industry that sees this algorithm not as another academic exercise but as a production-ready tool with immediate deployment value.

For smaller AI teams, the implications are particularly significant. Long-context inference has historically required expensive GPU clusters that only large technology companies could afford. A 6x reduction in memory footprint means the same hardware can handle substantially more concurrent requests, lowering the barrier to competitive AI services.

AI data center server rack
Open-source TurboQuant implementations are already spreading across AI serving infrastructure — Photo by Marc PEZIN on Unsplash
💡 Related Posts
$14.2B Intel Buyback: Paying $3B Extra to Reclaim Fab 34

Jevons Paradox: When Lower Costs Drive Higher Demand

Memory chip stocks dropped, but the long-term picture is more nuanced. Meta has committed $27 billion to Nebius for compute capacity, and Google, Microsoft, and Amazon are collectively planning hundreds of billions of dollars in data center spending.

Against that backdrop, the premise that memory demand will simply decline looks incomplete. The sheer scale of planned infrastructure investment dwarfs the potential savings from any single compression algorithm.

Industry analysts characterize TurboQuant as “evolutionary, not revolutionary” — and then invoke Jevons Paradox. The paradox, first observed in 19th-century coal economics, holds that when a resource becomes more efficient to use, total consumption tends to increase rather than decrease.

Applied to AI inference, the logic runs as follows: lower memory costs per query make previously uneconomical workloads viable. Those new workloads generate fresh demand for hardware, potentially exceeding the savings from compression. The net effect is more GPUs deployed, not fewer — running more efficiently, but running in greater numbers.

The takeaway: in the short term, memory semiconductor demand forecasts face a question mark. In the longer term, the probability that total AI infrastructure spending declines is low. Whether TurboQuant evolves from a conference paper into an industry standard will depend on two milestones — the vLLM integration PR being merged, and Google releasing an official reference implementation.

References

Stay Ahead of the Curve

Get the latest insights on AI, tech, and markets — delivered in both Korean and English.

Explore More Posts