New KV cache compaction technique cuts LLM memory 50x without accuracy loss

MIT researchers developed Attention Matching, a KV cache compaction technique that compresses LLM memory by 50x in seconds — ...

Nvidia shrinks LLM memory 20x without changing model weights

Nvidia's KV Cache Transform Coding (KVTC) compresses LLM key-value cache by 20x without model changes, cutting GPU memory costs and time-to-first-token by up to 8x for multi-turn AI applications.
Semiconductor Engineering

Freeing Up Near-Memory Capacity For Cache Using Compression Techniques In A Flat Hybrid-Memory Architecture

A technical paper titled “HMComp: Extending Near-Memory Capacity using Compression in Hybrid Memory” was published by researchers at Chalmers University of Technology and ZeroPoint Technologies.
Nature

Cache Performance and Memory Hierarchy Optimization

The dynamic interplay between processor speed and memory access times has rendered cache performance a critical determinant of computing efficiency. As modern systems increasingly rely on hierarchical ...
Android Authority

What is cache memory - Gary explains

System-on-a-Chip (SoC) designers have a problem, a big problem in fact, Random Access Memory (RAM) is slow, too slow, it just can’t keep up. So they came up with a workaround and it is called cache ...
Electronic Design

Hardware Compression Works at the Memory Cache Level

How lossless data compression can reduce memory and power requirements. How ZeroPoint’s compression technology differs from the competition. One can never have enough memory, and one way to get more ...
Hackaday

Spectre And Meltdown: How Cache Works

The year so far has been filled with news of Spectre and Meltdown. These exploits take advantage of features like speculative execution, and memory access timing. What they have in common is the fact ...
Houston Chronicle

How Important Is a Processor Cache?

In the early days of computing, everything ran quite a bit slower than what we see today. This was not only because the computers' central processing units – CPUs – were slow, but also because ...