Run a 744-Billion Parameter AI Model on Your Laptop. For Real.
There is a dividing line in AI right now that does not get enough attention. On one side: trillion-parameter models that require $100,000-plus GPU clusters and a team of infrastructure engineers to run. On the other: everyone else, running quantized versions that lose something in translation, or paying API fees that add up fast.
colibri is a project that punches straight through that line.
It is a pure-C inference engine that runs GLM-5.2, a 744-billion-parameter Mixture-of-Experts model, on a consumer laptop with roughly 25 GB of RAM and a decent SSD. No GPU. No Python runtime. No cluster. One person built the whole thing on a 12-core laptop over what appears to have been a focused few months.
The numbers are not a marketing claim. They are on a public GitHub repo, with community benchmarks, and you can verify them yourself.
The Hardware Gap This Closes
Large language models have followed a predictable arc: bigger, more capable, more expensive to run. MoE architectures changed the math somewhat—a 744B model only activates around 40B parameters per token—but that is still an enormous amount of compute to move, and the standard approach has been to throw GPU clusters at it.
colibri takes the sparse nature of MoE seriously as a systems problem, not just an architecture diagram. The dense part of GLM-5.2 (attention layers, shared experts, embeddings) sits in RAM at int4 quantization: roughly 9.9 GB, manageable on most modern machines. The routed experts—21,504 of them, one per token routing decision—are stored on disk, roughly 370 GB total, and loaded on demand.
This is the part that sounds impossible when you sit down and think through the math. Only about 11 GB of model weights change from one token to the next. The disk read per token is real, but it is bounded and streaming, not random access across the entire model.
The Key Technical Pieces
The engine is approximately 1,300 lines of C in a single file, plus small headers. There is no BLAS, no Python at runtime, no GPU dependency. The README describes the architecture in enough detail that reading it feels like a compact graduate seminar in efficient inference.
The compressed KV-cache is the first thing that stands out. GLM-5.2 uses Multi-head Latent Attention (MLA), and colibri implements a version where the cache is 576 floats per token instead of the standard 32,768. That is a 57x reduction, and it matters because the KV cache is that makes long conversations tractable in memory.
The speculative decoding is the second piece worth understanding. GLM-5.2 has a built-in multi-token prediction (MTP) head at layer 78. colibri uses this head to draft multiple tokens at once, then verifies them in a single forward pass. At int8 precision for the head, acceptance rates run 39-59%, delivering 2.2-2.8 tokens per forward step. The catch: at int4 the acceptance collapses to near zero, so the head must be quantized to int8 while the rest of the model stays at int4. The project ships a converter that handles this automatically.
The learning cache is the third piece that is genuinely novel. The engine records which experts your actual usage routes to, stores that in a local file, and at startup automatically pins the hottest ones in available RAM. The more you use the model, the faster it gets, because the disk bottleneck shrinks as the working set fits in cache. This is not a benchmark optimization—it is a runtime behavior that compounds over normal usage.
Reading the Numbers
The README has an honest performance table that is refreshing to read. The developer tested on deliberately underpowered hardware: WSL2 on a laptop, 12 cores, 25 GB RAM, NVMe through a virtual disk that caps random read performance at roughly 1 GB/s. Cold decode on that setup runs 0.05-0.1 tokens per second.
That sounds slow. It is slow. But the README makes a point that is easy to miss: this is a 744B frontier-class model answering questions correctly on a machine that costs less than a single H100 fan. The comparison is not to GPT-5 running on a GPU cluster. It is to the alternative of not having that model accessible at all.
Community benchmarks tell a more interesting story. Real hardware tells the real picture.
| Machine | Disk Speed | RAM | Config | Speed |
|---|---|---|---|---|
| WSL2 dev box (VHDX) | ~1 GB/s | 25 GB | default | 0.05-0.1 tok/s |
| M5 Max MacBook (128 GB) | 14.2 GB/s | 128 GB | MTP off | 1.06 tok/s |
| Native Linux + PCIe4 NVMe | 3-5 GB/s | 32 GB | – | 0.5-1 tok/s |
| PCIe5 NVMe + 64 GB RAM | 8-12 GB/s | 64 GB | PIN ~40 GB | 2-4 tok/s |
The binding constraint on small-RAM machines is not the disk. It is RAM. When the expert cache is auto-capped at 2 slots per layer because the system only has 24 GB available, the cache hit rate stays low regardless of how fast the disk is. -topp 0.7 alone bought a 1.6x end-to-end speedup on one test machine by routing to fewer experts per token. These are the kinds of knobs that matter in practice.
When the expert cache is auto-capped at 2 slots per layer because the system only has 24 GB available, the cache hit rate stays low regardless of how fast the disk is. `-topp 0.7` alone bought a 1.6x end-to-end speedup on one test machine by routing to fewer experts per token. These are the kinds of knobs that matter in practice.
What You Need to Run It
Minimum: Linux or WSL2, gcc with OpenMP, AVX2 support, 16 GB RAM, and roughly 370 GB of local NVMe storage (ext4 filesystem—network mounts will not work). Python is only needed during the one-time conversion step to go from the FP8 checkpoint to the int4 container the engine expects.
| Component | Requirement |
|---|---|
| OS | Linux or WSL2 |
| Compiler | gcc with OpenMP, AVX2 |
| RAM | 16 GB minimum |
| Storage | ~370 GB NVMe (ext4, not network mount) |
| Python | Required for conversion step only |
| Disk space for conversion | Only one shard at a time (resumable) |
The converter is also resumable. It downloads one shard at a time, converts it, and deletes the original. You never need 756 GB of free space for the FP8 checkpoint—the full conversion happens shard by shard and the source data does not pile up.
The engine itself is pure C at runtime. The setup script builds it, runs a self-test that validates token-exact match against a transformers oracle, and leaves you with a working `./coli` binary.
There is a Hugging Face repo with a pre-converted int4 model ready to download: jlnsrk/GLM-5.2-colibri-int4. Point `COLI_MODEL` at it and you skip the conversion step entirely.
Where It Makes Sense
| Use Case | Why colibri Fits |
|---|---|
| Local development | No cloud bill, private, no rate limits |
| Research and education | Real 744B MoE architecture on real hardware |
| Offline/edge workflows | No network dependency on capable workstations |
| Production workloads | Not suitable — use A100s or similar |
colibri is not trying to replace GPU inference for serious workloads. If you are running a production service, paying for A100s is still the right answer. The places where this genuinely matters are more specific.
Local development and experimentation is the obvious one. Running a 744B model without a cloud bill changes how you can iterate. You have the model weight and you have the machine—the session is free and private and there is no rate limit.
Education and research is another real use case. Understanding how MoE sparsity works in practice, testing quantization strategies, running experiments on model behavior—having a real 744B architecture on real hardware makes that concrete in a way that reading papers does not.
Offline or edge-adjacent scenarios have a place too. A sufficiently equipped workstation can run this without any network dependency, which matters for certain development and testing workflows.
The honest limitation is quality. No one has published accuracy benchmarks for the int4 container against the full-precision model. The developer explicitly notes this and has built the benchmark harness—it just takes a long time on slow storage. MMLU and HellaSwag numbers on the original GLM-5.2 run in the 85-95% range. Whether int4 lands within a few points of that is unknown. If you need precise model quality guarantees, wait for those numbers or run the benchmarks yourself.
What This Represents
The README ends with a description of why the project is named colibri: the hummingbird weighs a few grams, hovers in place, and visits a thousand flowers a day. The engine keeps a 744-billion-parameter giant alive on hummingbird rations.
That is a more honest framing than most AI announcements manage. The point is not that this is the fastest way to run a large model. It is that it is possible at all on the hardware you already have, and the gap between possible and practical is smaller than it looks.
The disk reads are real. The memory requirements are real. The 1 tok/s ceiling on a MacBook is not going to confuse anyone for GPT-4 Turbo. But a 744B model that answers questions correctly, on a laptop, offline, for free, is a meaningfully different thing than a 744B model that only exists behind an API.
If you have the hardware and the interest, the barrier to finding out what that difference feels like is now a `git clone` and a conversion script.
Hardware requirements aside, the more lasting interest from a reader perspective is probably in understanding how the technical pieces fit together—the MoE sparsity exploited as a systems problem instead of just an architectural curiosity, the compressed KV cache, the speculative decoding via the model’s own MTP head. Those ideas transfer. The specific engine runs on Linux today. The concepts behind it will show up in whatever comes next.
