A 900KB AI Model That Compresses 100MB Files to 7MB. How It Actually Works

A developer built an experiment: train a tiny transformer model — just 900KB — to memorize a specific file, then use it to compress that file with surprising efficiency. A 100MB CSV of NYC taxi data compressed to 7MB. A 100MB slice of enwik9 compressed to 21MB. It’s slow and impractical for most real-world use, but it reveals something interesting about how transformers actually work as compression systems.

The Basic Idea

The concept is straightforward — almost suspiciously so:

Instead of training a model to generalize (the normal goal in machine learning), you train it to memorize a specific file. The model learns to predict the next byte in that file, byte by byte. Those predictions aren’t perfect, but they’re good enough that the prediction errors — the gaps between what the model predicted and what actually came next — are highly compressible using arithmetic coding.

The arithmetic coder converts the model’s probability distributions into a compact bit stream. The model weights themselves are small (900KB), and the arithmetic-coded residuals are what actually hold the compressed data.

This is fundamentally different from traditional compression (gzip, zstd) because the “dictionary” is a neural network trained specifically on your data, not a fixed algorithm.

The Results

From the developer’s own benchmarks:

| Dataset | Original Size | Compressed | Ratio | Bits/Byte |
|———|————-|————|——-|———–|
| NYC Taxi CSV | 100MB | 7MB | ~14:1 | 0.5 |
| enwik9 | 100MB | 21MB | ~5:1 | 1.68 |

For comparison, gzip typically achieves 2-3:1 on text, and zstd can reach 3-5:1 on structured data. The 14:1 result on tabular data is genuinely impressive.

The tradeoff: each compressed file requires its own model. You can’t compress a new file with the same model — you have to train a new one. And training takes 20-30 minutes, with another 45 minutes each for compression and decompression.

What the Hacker News Discussion Revealed

The thread was lively. Several points stood out:

“Fabrice Bellard did this seven years ago.”
One commenter noted that the legendary French programmer (known for the fastest JIT compilers, BPG image format, and QEMU) published similar experiments in 2018. The developer acknowledged the prior art.

“What about ensembling?”
Someone suggested bagging multiple overfitted transformers to improve compression. Gwern — a well-known ML researcher — shot back that ensembling is terrible for compression efficiency: “This is related to why people train ever larger LLMs like a 10T-parameter model rather than 100 GPT-3-scale LLMs.” Larger, specialized models compress better than many small ones.

The real question: practical applications
Most conventional compression tools handle general data better and faster. The neural approach only wins on specific data types where the model can find strong patterns — structured tabular data, repetitive text, domain-specific corpora. And the slow decode speed is a significant barrier.

Why This Matters Beyond the Benchmark

The interesting part isn’t the compression ratio. It’s what the experiment reveals about transformers as compression systems.

Large language models are, in a formal sense, compression systems. They predict the next token given previous tokens, which is the same problem as arithmetic coding for compression. The key insight from this experiment: the compression efficiency depends on whether the model can actually fit the data distribution.

On structured tabular data (the NYC taxi CSV), a tiny model can capture the patterns efficiently because the data is relatively low-entropy. On natural language (enwik9), you need much more capacity because language is higher-entropy — there’s more unpredictability in the next word.

This connects to ongoing debates about whether AI models “understand” or “compress.” A compression-centric view suggests that what we call “understanding” might just be very efficient compression of patterns in training data.

When Neural Compression Actually Makes Sense

Despite the hype around this experiment, neural compression has real practical applications:

Archival compression for specific corpora — if you’re compressing millions of similar documents (customer support logs, legal filings, medical records), training a domain-specific model once could pay off significantly.

Structured data where patterns are predictable — tabular data, time series, and semi-structured formats tend to compress well with neural methods because the underlying distributions are learnable.

Scientific data with known physical constraints — if you know the data must obey certain physical laws, a model that learns those laws can compress far beyond what general-purpose algorithms achieve.

For everything else — general-purpose file compression, backup systems, everyday use — traditional algorithms are faster, more flexible, and don’t require a GPU to decompress.

Focus Keyword
`900KB transformer compression`, `neural file compression AI`, `transformer compressor open source`, `AI data compression tutorial`, `compress files with machine learning`

Tags
AI compression, transformer, machine learning, file compression, neural networks, open source AI, AI experiment, enwik9, NYC taxi data