Llama-modchip claims to run 284B DeepSeek model on 64GB RAM, no GPU

A GitHub project called Llama-modchip, published by Synchopate, says it runs DeepSeek-V4-Flash, a 284-billion-parameter model, entirely in the memory of a 64GB machine using only the CPU and no graphics card. The developer’s README puts the compressed model at 56.56 GiB on disk and its peak memory use at 57.61 GiB with a 256K-token context, with text generation at 9.1 to 14.7 tokens per second.

All of those figures are the developer’s own. The project’s README is the only primary source available, and none of the sources contain an independent test of the claims.

What Llama-modchip says it does

The repository, now at version 2.4.0, frames its goal as breaking “the 64 GB RAM Wall”. Its central claim is that a full frontier-scale mixture-of-experts (MoE) model fits on an ordinary workstation. A mixture-of-experts model splits its network into many “expert” sub-networks and activates only a few for each token. According to the README, the model fits “100% in RAM, zero NVMe swap”, meaning no part of it is paged out to disk during inference.

The developer says the unmodified DeepSeek-V4-Flash checkpoint is 76.87 GiB, which “crashes in OOM on 64 GB Linux”. OOM means the system runs out of memory. Llama-modchip’s version is 56.56 GiB on disk and uses 57.61–58.9 GiB of resident memory at the full 256K context.

The project also covers smaller models. Its main example is NVIDIA-Nemotron-3.5-30B-A3B, compressed to 14.35 GiB, which the README says runs on 16–32 GB laptops and mini-PCs. It also lists Qwen3-30B for hybrid CPU-plus-GPU setups.

How the 284B model is squeezed under 59 GiB

The README names a stack of techniques, several under the project’s own labels:

• SwiGLU-H2 compression. The README describes this as “closed-form Analytical SwiGLU-H2 Second-Order Hessian Compression”. SwiGLU is the type of feed-forward block used inside many large language models. The developer credits this step alone with saving 20.31 GiB.
• Fractal-64 quantization. Quantization stores weights at lower numeric precision. The project uses a custom “Q2_0” format at 3.75 bits per weight, applied with what it calls MLA transcoding.
• Whole-model tiling across the CPU’s memory hierarchy. The README describes four levels of tiling: AVX-512BW vector registers, a 32 KB L1 data cache, a 1 MB L2 cache and a 64 MB L3 cache.
• MoE pruning. The README refers to “$L_2$-Quadrature MoE Pruning”. This implies that some expert capacity is removed, though the README excerpt does not say how much.

The repository includes a paper folder with a preprint PDF, plus benchmarks, reviews and an architecture document. The contents of those files are not part of the source text reviewed here.

Claimed speeds for DeepSeek and 30B models

The README reports two kinds of speed. Prefill is the rate at which the model reads the prompt. Decode, or token generation, is the rate at which it writes its answer. For the 284B model, the developer claims 44.36 tokens per second of prefill and 9.06–10.74 tokens per second of pure CPU decode. Decode rises to 11.5–14.7 tokens per second with “PVS”, the project’s speculative drafting system.

Speculative drafting uses a small, fast model to guess upcoming tokens, which the large model then checks. Llama-modchip’s drafter is a 1.64 MB component it calls “Stockfish-PVS v2 + RouteCrack L2/L3 Drafter”. For the 30B setup, the developer reports a 96.4% acceptance rate for its guesses.

Setup (developer’s figures)	RAM	Model	Claimed speed
Developer laptop / mini-PC	16–32 GB	Nemotron-3.5-30B-A3B (14.35 GiB)	25.0–60.4 tok/s
64 GB desktop / server	64 GB	DeepSeek-V4-Flash 284B (56.56 GiB)	44.4 tok/s prefill; 9.1–14.7 tok/s generation
Hybrid desktop (CPU + 16 GB GPU)	32–64 GB	Nemotron-30B / Qwen3-30B	80.80 tok/s
8-channel Threadripper workstation	128 GB+	Nemotron-30B	111.62 tok/s

Source for all rows: the project README.

For the 30B Nemotron model, the README gives additional figures: 25.0–60.4 tokens per second on a 16-core Zen 5 processor and 44.97 tokens per second on AWS Graviton3 ARM64. It also lists 80.8 tokens per second in an AMD Vulkan hybrid mode and 111.6 tokens per second on a Threadripper. In the hybrid mode, the attention layers that are used for every token run on the GPU, while the less frequently used routed experts stay in the CPU’s DDR5 memory.

The README excerpt does not name the exact processor used for the 284B DeepSeek numbers. It describes that tier only as “a 64 GB desktop / server”. There is also a difference worth checking: the quick-start command for the DeepSeek model sets a 32,768-token context (-c 32768), while the headline memory figure is for 256K context.

A Claude Code mode that restores context in milliseconds

The third feature the developer highlights targets AI coding agents. A command, ./modchip claude, runs Claude Code against the local model, which the README calls “Zero-API”. Coding agents resend long prompts full of tool definitions and file context. On a CPU, the developer says, re-evaluating that prompt causes a “45-second” delay.

The project’s fix stores the model’s KV cache in shared memory (/dev/shm) and restores it instead of recomputing it. The KV cache is the saved intermediate state for text the model has already processed. The README reports restore times of 18.9 ms on Nemotron-30B and 13.1–32.6 ms on DeepSeek-284B, “with all 40 Claude Code tools active”.

For builders, this is the most practical part of the pitch. A local model that answers an agent within a fraction of a second, rather than after a long pause, would make offline coding agents usable on CPU-only hardware. The figures come from the developer, though, and none of the sources include a test of agent task quality on the compressed models.

How to get it, and what it will not open in

The README offers a one-command setup on Linux and macOS (./modchip chat). It also provides a direct-run path that downloads compressed GGUF model files from Hugging Face repositories under the Synchopate name, such as Synchopate/DeepSeek-V4-Flash-0731-Fractal-64GB-CPU-GGUF. GGUF is the model file format used by llama.cpp. On Windows 10 and 11, a batch file detects CPU cores, free RAM and GPU memory, downloads official llama.cpp CUDA 12.4 or Vulkan binaries, picks how to split the experts between CPU and GPU, and opens a local web chat. The repository also has folders for Apple Silicon and Fedora.

Other commands include a terminal chat with live speed telemetry, a web dashboard, and ./modchip bench. The README says that command runs a 30-second speed test and prints a shareable “CPU Benchmark Scorecard”.

There is a major compatibility caveat. The README warns that the Fractal-64 models use a custom tensor type (GGML_TYPE_Q2_0, ID 40) and custom kernels. Opening them in unpatched llama.cpp, Ollama or LM Studio “will fail with unknown tensor type 40”. Users need the project’s own launcher or a standalone pre-compiled Linux x86_64 binary. The README says that binary has no shared-library dependencies and is bundled in the Hugging Face repositories and the v2.4.0 GitHub release. Running a pre-built binary from a new project means trusting that project’s build. The repository contains a LICENSE file, but its terms are not visible in the source text.

How this compares with the usual GPU requirements

The claim stands out because local LLM inference has usually been framed around graphics memory. One GPU cloud provider’s guide to running models through Ollama lists at least 16 GB of video memory for DeepSeek R1 14B. It lists at least 24 GB, with an NVIDIA RTX 4090 or higher, for DeepSeek Coder V2 16B. Those models are small fractions of the 284B model Llama-modchip targets.

Further back, the original LLaMA reference code ran its example on “a single or multi-gpu node”. Its 65B model required a model-parallel setting of 8. That code also lists “CUDA Out of memory errors” among its FAQ topics. Against that history, a 284B model generating around 10 tokens per second on CPU and system RAM would be a large shift for hobbyists and small teams, if independent users can reproduce it.

The comparison has limits. The GPU guides describe standard model files, while Llama-modchip relies on aggressive compression and pruning. A 3.75-bit-per-weight model with pruned experts is not necessarily equivalent to the original DeepSeek-V4-Flash. The README text reviewed here gives no accuracy or quality scores to show how much is lost.

What is unverified, and what to watch next

The open questions for builders, based on what the sources do and do not show:

• Independent reproduction. Every number comes from the developer. The included benchmark scorecard tool means community results could appear quickly. Results from outside testers on named hardware would be the first real check.
• Quality after compression. The README describes Hessian-based compression, 2-bit-class quantization and MoE pruning but, in the text available, no evaluation of output quality. Builders should test the models on their own tasks before relying on them.
• Hardware details for the 284B figures. The CPU behind the DeepSeek speeds is not specified in the excerpt, and the quick-start example uses a 32K context rather than the advertised 256K.
• Ecosystem support. Because the format uses a custom tensor type, it works only with the project’s own tools for now. Whether its kernels reach mainstream llama.cpp, Ollama or LM Studio will decide how widely it spreads.
• The preprint. The repository links a preprint in its paper folder. Outside scrutiny of the method would carry more weight than the README’s headline numbers.

Until independent results arrive, Llama-modchip is best treated as a promising, self-reported claim. If the numbers hold up, a 64GB CPU-only box could serve a frontier-scale open model and a local coding agent with no GPU budget at all.