Google open-sources TPU Sync to speed up long AI sessions on its chips

Google Cloud on 9 November 2026 released TPU Sync, an open-source library that moves the working memory of large AI models between its TPU chips, host computers, storage and the data center network. Google’s headline claim is that it cuts time to first token (the wait before a model starts answering) by an average of 90% for large open-weight mixture-of-experts models in prefix-caching and disaggregated-serving setups, compared with running without it.

The release lands in the year Google split its eighth-generation TPU line into a training chip, TPU 8t, and an inference chip, TPU 8i, both pitched at bigger models and longer contexts. TPU Sync is the software side of the same problem: keeping enormous amounts of model state close enough to the chip that the accelerator does not sit idle.

What Google released on 9 November

The announcement, written by Google Cloud Distinguished Engineer Fenghui Zhang and Senior Product Manager Weiren Yu, frames the problem plainly. As AI moves from single-turn prompts to “million-token agents, disaggregated serving, and continuous reinforcement learning (RL)”, Google argues the main bottleneck has shifted “from raw compute to orchestrating live model state”. That state comes in two forms:

• Contextual state: the key-value (KV) cache, the stored intermediate results a model keeps for every token it has already read, so it does not have to recompute them.
• Parameter state: model weights that change during RL training and must be sent from trainers to the rollout samplers that generate new outputs.

Google says TPU Sync was designed as “the model-state layer across TPU clusters”, built on the idea that accelerator High Bandwidth Memory (HBM), host DRAM, shared storage and the data center network (DCN) should behave as one co-designed memory hierarchy. It is a native C++ library that moves TPU-tiled tensors across that hierarchy with asynchronous DMA (direct memory access) transfers, so data moves while the chip keeps computing.

“Whenever this state is trapped inside a single chip’s memory or stalled on network transfers, AI accelerators sit idle.”

That sentence, from Google’s post, sums up the pitch.

How TPU Sync handles KV cache and long contexts

For inference, Google says TPU Sync turns host DRAM, local SSDs and Google Cloud Managed Lustre (a managed shared file system) into an automated, multi-tier KV cache, and decouples TPU compute from network backpressure. Google lists four main inference and training scenarios:

1. Long-context agents and multi-turn chats. In sessions of 256K to more than 1M tokens, the KV cache keeps growing. TPU Sync moves older KV blocks out of TPU HBM into large pools of host RAM, with experimental tiering to shared secondary storage.
2. Prefix caching. When thousands of queries share the same system prompt or documents, TPU Sync extends the prefix cache into distributed host memory so concurrent sessions can reuse precomputed KV blocks instead of repeating the prefill step.
3. Disaggregated prefill/decode serving. Here compute-bound machines process the prompt (prefill) and memory-bound machines generate the answer token by token (decode). TPU Sync acts as a Host-to-Host streaming pipeline between them, which Google says delivers an over 90% reduction in data transfer overhead.
4. Reinforcement learning (experimental). Covered in the next section.

The technical core, according to Google, is that TPU Sync reads the underlying JAX PJRT buffers or PyTorch tensors in place, rather than copying or locking them, and moves slices directly between TPU HBM and page-locked host memory in a single hardware DMA, keeping tensors in their native TPU layout. Google’s own analogy compares the TPU to a chef on a tiny countertop: without TPU Sync, prepped ingredients get thrown out and chopped again (prefill recomputation); with it, runners swap trays between the counter, a side pantry and shared cold storage while the chef keeps cooking.

Reinforcement learning weight transfer, still experimental

For RL, TPU Sync exposes low-level DMA and DCN primitives that higher-level RL frameworks can use to stream updated weights from trainers to samplers. Google says this includes automatic resharding across different trainer and sampler parallelism layouts, which matters because training and sampling jobs often split a model across chips in different ways. The goal is to cut the time rollout workers spend waiting for weight updates.

This part carries a clear caveat in Google’s own post: it was “validated on internal RL workloads” and ships in the repository as an experimental module. TPU Sync provides primitives here, not a complete RL framework; Google names Pathways and multi-controller JAX (McJAX) RL pipelines as the intended consumers.

Google’s performance claims and what backs them

All of the numbers come from Google and compare TPU Sync with running the same workloads without it.

Claim	Google’s figure	Conditions stated
KV transfer latency	Reduced by over 90%	Prefix-caching and disaggregated-serving workloads
Time to first token	Reduced by an average of 90%	Open-weight MoE models of 235B to 480B parameters
Prefill-to-decode data transfer overhead	Over 90% reduction	Disaggregated serving

The sources reviewed do not name the specific models tested, the TPU generation used, the context lengths, or the baseline configuration beyond “without TPU Sync”. There are no third-party benchmarks yet. A 90% cut against a setup that recomputes every prefix is plausible in workloads with heavy prompt reuse, but builders should expect the gain to depend on how much context their traffic actually shares.

Where it fits with TPU 8t, TPU 8i and existing tools

Google says TPU Sync plugs into existing stacks rather than replacing them. It integrates with PyTorch through TorchTPU and with JAX, serves as the TPU KV cache connector for vLLM, SGLang and llm-d disaggregated serving, and provides transfer primitives for Pathways and McJAX. That list matters for portability: a Google Cloud developer advocate noted in late 2025 that vLLM already supported many models on both TPUs and GPUs, and that JAX runs on TPUs, GPUs and CPUs.

The hardware context is Google’s eighth TPU generation. In April 2026, Google unveiled two separate chips, one for training and one for inference, a return to a split design after Trillium and Ironwood used largely a single design, Network World reported. Google’s stated figures, as reported, compare the new chips with Ironwood:

Spec (Google’s figures)	Ironwood	TPU 8t (training)	TPU 8i (inference)
Chips per pod	9,216 (256 in the inference comparison)	9,600	Up to 1,152
Compute per pod	42.5 exaflops (1.2 in the inference comparison)	121 exaflops	11.6 exaflops
Memory	Not stated	Not stated	288GB HBM plus 384MB on-chip SRAM
Scale-up bandwidth	Not stated	19.2 Tbps per chip (double Ironwood)	Not stated
Scale-out networking	Not stated	400 Gbps (four times Ironwood)	Not stated

Google also claimed 2x better performance per watt than Ironwood and tighter integration with its Axion Arm-based CPU hosts, according to Network World. Omdia analyst Alexander Harrowell said the larger on-chip SRAM in TPU 8i should keep active model data closer to the processor and reduce inference latency. HyperFRAME Research’s Stephen Sopko tied the design to mixture-of-experts and long-context models: “Large RAM and pod sizes are now required to keep trillion-parameter models and million-token context windows memory-resident during serving”.

TPU Sync addresses the same pressure from the other direction: instead of only adding memory on the chip, it pushes cold context out to host RAM and storage. Google’s post, in the portion reviewed, does not say which TPU generations TPU Sync supports, so it should not be assumed to require TPU 8t or 8i.

What it means for builders

Most developers who call models through an API will never touch TPU Sync directly. Its effect, if Google’s numbers hold, shows up indirectly: faster first tokens on long agent sessions and lower serving cost for prompts that share long system instructions or documents. Teams that self-host open-weight models on Google Cloud TPUs with vLLM, SGLang or llm-d are the ones who can try it now.

Points to weigh:

• Lock-in. As of December 2025, TPUs were available only through Google Cloud Platform, according to a Google Cloud developer advocate, who added that moving from TPUs to GPUs is usually easier than the reverse. A Network World headline from October 2026 reports Oracle adding Google TPUs to its cloud, but the details are not covered in the sources reviewed. TPU Sync is open source, but it is built around TPU memory layouts and Google Cloud services such as Managed Lustre.
• Maturity. RL weight transfer and tiering to shared secondary storage are both labelled experimental.
• Evidence. Every performance number is Google’s own, measured against a no-TPU-Sync baseline.
• Cost logic. Analysts quoted by Network World expect split training and inference chips to help companies avoid paying training-grade prices for inference-heavy work; Forrester’s Charlie Dai made that case. TPU Sync’s prefix-cache reuse fits the same cost argument, though no pricing data is part of this release.

What to watch next

Three things will show how much TPU Sync matters. First, independent measurements from teams running vLLM or SGLang on TPUs, with the model, context length and baseline spelled out. Second, whether the experimental RL module and storage tiering graduate to stable status. Third, general availability of TPU 8t and TPU 8i, which Google said in April would arrive later this year as part of its AI Hypercomputer platform, and whether Google publishes TPU Sync results on those chips. This is the first Unicorn API article in the Google eighth-generation TPU thread; later updates will build on it.