NVIDIA Details Full-Stack Codesign to Tackle AI Memory Bottlenecks
Frontier AI models · AI assistants · Enterprise AI
NVIDIA outlined its memory strategy addressing GPU memory pressure from agentic inference and large context windows. Instead of merely increasing HBM capacity, NVIDIA leverages full-stack codesign across GPUs, Vera CPUs, NVLink-C2C, Dynamo inference software, and tiered CMX context storage to optimize memory usage.
Facing rising GPU memory pressure driven by larger model sizes, extended context windows, and multi-turn agentic workflows, NVIDIA stated that adding raw high-bandwidth memory (HBM) capacity alone is constrained by memory bandwidth limits. According to NVIDIA, addressing this requires a full-stack codesign across hardware, interconnects, and software to manage a hierarchical memory architecture spanning GPU HBM, CPU LPDDR memory, local SSDs, and tiered external storage. To optimize memory utilization, NVIDIA highlighted several core techniques: reducing model weights using the 4-bit NVFP4 precision format and distributing them across NVL72 configurations; applying NVFP4 to the key-value (KV) cache; and leveraging NVIDIA Dynamo software alongside NVIDIA CMX context memory storage to offload inactive KV context across servers. Additionally, NVIDIA indicated that Nemotron Retriever models and the cuVS library keep RAG embeddings and indexing within CPU and storage tiers, ensuring enterprise knowledge bases bypass GPU HBM until relevant passages are needed.
Key takeaways
- Agentic trajectories processing up to 3 million tokens and RAG workloads are driving up HBM requirements during inference.
- NVIDIA uses NVFP4 4-bit precision to reduce footprints for both model weights and KV cache compared to FP8.
- Dynamo software coordinates KV cache across HBM, Vera CPU memory, and shared CMX storage to offload inactive context.
- NVIDIA claims the Vera Rubin NVL72 achieves up to 30x higher throughput per dollar than GB300 NVL72 on the SemiAnalysis AgentX benchmark.
Source：NVIDIA Blog Read the original ↗
Prepared, checked and automatically published by AI; not individually reviewed by a human.