NVIDIA's Dory 4B model reuses its layers to trade compute for accuracy

NVIDIA has published Dory 1.0 4B, a small open-weight language model that runs one shared block of its layers several times per token, so a single checkpoint can be run lighter or heavier depending on the compute available. The model, listed on Hugging Face as NVIDIA-Nemotron-Labs-Dory1.0-4B, appeared on October 16, 2026, without an accompanying blog post.

It is the first Dory release NVIDIA has published. The design is the news: rather than adding parameters, Dory lets the user choose how many times its middle layers run. At launch, the practical catch was software. Released versions of common local runtimes did not read the new architecture, and NVIDIA’s own instructions pointed developers to unmerged code branches.

How Dory’s looped layers work

The model card, as quoted by ModelFit, describes Dory as a looped (latent-recurrent) normalized-GPT with interleaved sliding-window and full attention. In plain terms, “latent-recurrent” means the model repeats part of its computation on its own internal representation rather than on output text. “Sliding-window attention” lets a layer look only at a recent span of tokens, which is cheaper than full attention over the whole context.

The layer stack is split into three groups:

• Input block: 8 layers that run once.
• Recurrent block: 20 shared layers that are applied several times, with the original input reinjected into every loop.
• Output block: 8 layers that run once.

The number of times the shared block runs is the loop count. Changing the setting dory.num_loops trades compute per token for accuracy without downloading a second checkpoint. The card’s wording, per ModelFit, is that one checkpoint serves several inference compute budgets, with no retraining and no model swap.

The two reports differ on which loop counts are available. ModelFit lists values of 1, 2, 3 and 4, while AICoder describes the shared block running 1, 2 or 4 times. AICoder reports that two loops is the recommended default and four is suggested for agentic work.

Running layers more than once would normally multiply the memory held in the KV cache, the store of past attention data a model keeps while generating. AICoder reports a last-loop KV-cache mode that keeps cache memory at one-loop size regardless of loop count.

The idea of reusing layers has appeared elsewhere recently: SenseTime released an open 9B model that runs its layers twice, covered in an earlier Unicorn API article.

Dory 1.0 4B specs from the model card and file header

ModelFit read the shape of the model directly from the header of the published Q5_K_M GGUF file. Combined with the card details, the reported specs are:

Spec	Value
Parameters	4B
Layer groups	8 input, 20 shared recurrent, 8 output
Blocks in file header	72
Embedding length	2,560
Attention heads	16, with 4 key-value heads of 256 elements each
Attention layers	36, of which 9 use full attention
Sliding window	1,024 tokens
Maximum context	262,144 tokens (up to 256K)
Language	English only
Licence	OpenMDW 1.1

On training, AICoder reports that Dory was pre-trained on 6T tokens, natively in NVFP4 and then in BF16, followed by 300B tokens of context extension, and post-trained with multi-environment GRPO. NVFP4 is NVIDIA’s 4-bit floating-point number format; training natively in it is part of what the company is demonstrating here.

NVIDIA’s reported scores, and what they leave open

All benchmark numbers available so far are NVIDIA’s own, and AICoder notes there is no independent replication yet. In NVIDIA’s reported table:

• SWE-bench Verified (a test of fixing real software issues): 50.3 at one loop, 59.5 at two loops and 61.8 at four loops, compared with 61 for Qwen3.5-9B in the same table.
• Terminal-Bench 2.1 (command-line agent tasks): 22.2 at four loops, well below the 43 listed for Qwen3.6-35B-A3B.
• IFBench (instruction following): 79.

The SWE-bench numbers are the clearest picture of what looping buys in NVIDIA’s evaluation: the gap from one loop to two is larger than the gap from two to four. That pattern fits the recommendation of two loops as the default. The Terminal-Bench result is a reminder that the comparison with larger models is not uniform across tasks.

On speed, AICoder reports that the NVFP4 build runs about 2.17 to 2.44 times faster than BF16 on NVIDIA’s DGX Spark. That figure also comes from NVIDIA and covers only NVIDIA hardware.

Licence, formats and download sizes

Weights ship in BF16, MXFP8, NVFP4 and Q5_K_M GGUF builds under the OpenMDW-1.1 licence, which AICoder describes as commercially usable. ModelFit measured each repository through the Hugging Face API on October 19:

Build	Format	Repository size
Dory1.0-4B-GGUF	Q5_K_M GGUF plus card	3.675 GB
Dory1.0-4B-NVFP4	NVFP4 safetensors	3.221 GB
Dory1.0-4B-MXFP8	MXFP8 safetensors	4.770 GB
Dory1.0-4B-BF16	BF16 safetensors	7.969 GB

The GGUF file itself, the one NVIDIA’s Mac instructions expect, is 3,673,288,864 bytes, or 3.673 GB; the repository total also counts the card, images and pointer files. The card publishes a SHA-256 checksum for that file, which lets developers confirm they have the exact build the instructions assume.

Running Dory 4B locally: the llama.cpp branch and other runtimes

Memory is not the obstacle. By ModelFit’s calculation, which allows a model 70% of unified memory on Macs of 32GB and below, the GGUF fits on every current Mac. On an 8GB Mac that budget is 5,600 MB, leaving 1,927 MB after the weights for the KV cache and runtime buffers: enough for a 4,096-token context at ModelFit’s settings, not for 256K. ModelFit rates 16GB machines and up as comfortable.

Software is the obstacle. The status below is as ModelFit found it on October 19, 2026, with AICoder reporting the same picture two days earlier. Neither report covers what has happened since.

• llama.cpp: pull request 30306, titled “dory support for llama.cpp”, was open and in draft. It was opened on October 10, last updated on October 17, and stood at 4 commits touching 17 files. Until it merges, no llama.cpp release, including the 0.4.x line, loads a Dory GGUF.
• NVIDIA’s Mac path: the documented route for Apple Silicon is to build from a pinned revision of a contributor fork.
• Ollama: no library tag existed under dory, nemotron-labs-dory or nemotron-dory, so there was no ready-made run command.
• vLLM: pull request 60149, “Add Dory support”, was open and unmerged, and the card tells users to install vLLM from that pull request to run the NVFP4 and MXFP8 builds.
• MLX: mlx-lm had no Dory model implementation.

Fine-tuning tooling is also unsettled in the available record. NVIDIA’s NeMo AutoModel model support log, which tracks models its training library supports, lists entries through November 7, 2026, including NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-BF16 on October 23, but its visible list shows no Dory entry.

What Dory means for builders, and its limits

For developers shipping local or on-device features, the appeal is a single 4B file that can be dialled between a cheaper and a more capable mode at inference time. An app could, in principle, use fewer loops for quick replies and more for harder tasks without managing two models. The last-loop KV-cache mode, if it performs as described, keeps memory flat as the loop count rises.

The limits the sources raise:

• English only. The card lists English as the only language.
• Vendor-only evidence. Every score is NVIDIA-reported, with no independent replication reported.
• Uneven task results. The Terminal-Bench 2.1 score sits well below a larger rival in NVIDIA’s own table.
• Unreleased runtime support. As of the October reports, running Dory meant building llama.cpp or vLLM from unmerged branches or a pinned fork. Whether those branches have since merged is not confirmed by the sources, so developers should check the pull requests’ current status before relying on them in production.
• Loop-count ambiguity. The reports disagree on whether three loops is a supported setting.

What to watch next for NVIDIA Dory

The first signal is whether llama.cpp pull request 30306 and vLLM pull request 60149 merge into released versions, which would remove the need for fork builds and open the door to an Ollama tag. The second is independent testing: third-party runs of SWE-bench Verified and Terminal-Bench at different loop counts would show whether the trade-off holds outside NVIDIA’s evaluation. The third is whether NVIDIA follows this first Dory release with larger looped models, and whether its training tools add Dory support for fine-tuning.