Serving GLM-5.3 at 607.8 tok/s across two Nvidia DGX Stations with vLLM
Original title: Serving GLM-5.3 at 607.8 tok/s across two Nvidia DGX stations with vLLM
The NVFP4 checkpoint is about 464 GB, larger than one GB300 GPU's 252 GB of HBM3e, so weights are split across two stations.
NVIDIA published a tutorial for serving its GLM-5.3 NVFP4 quantization of Z.ai's mixture-of-experts model on two DGX Stations with vLLM. The model has 753 billion parameters, with 40 billion activated per token. Tensor parallelism of 2 keeps all weights in GPU memory, with CPU offload disabled. Station A exposes one OpenAI-compatible API while Station B runs only as a worker over ConnectX-8 RDMA.
- The pinned container is vllm/vllm-openai:v0.30.0-aarch64.
- Each station needs at least 464 GB of free disk for its checkpoint copy.
- The model repository cited is nvidia/GLM-5.3-NVFP4.
- Load testing uses SPEED-Bench prompts from the gated cais/hle dataset.
- Vendors
- NVIDIA
- Products
- DGX StationvLLM
- Organizations
- Z.ai
- AI models
- GLM-5.3GLM-5.3-NVFP4
From the original article
12 min read at developer.nvidia.com
Running a frontier-scale model locally means keeping all its weights in GPU memory; the largest models no longer fit on one machine. Two NVIDIA DGX Stations connected over ConnectX-8 RDMA serve one model through a single endpoint, with tensor parallelism splitting the model weights and computation across both GPUs. The NVIDIA GLM-5.
By wondersky · developer.nvidia.com
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.