Skip to content
NewA calmer, faster ZeroHour

Serving GLM-5.3 at 607.8 tok/s across two Nvidia DGX Stations with vLLM

Original title: Serving GLM-5.3 at 607.8 tok/s across two Nvidia DGX stations with vLLM

Hacker News · AIby wondersky ()12 min read1
AI summary · x-ai/grok-4.7Summaries by Vani AI

The NVFP4 checkpoint is about 464 GB, larger than one GB300 GPU's 252 GB of HBM3e, so weights are split across two stations.

NVIDIA published a tutorial for serving its GLM-5.3 NVFP4 quantization of Z.ai's mixture-of-experts model on two DGX Stations with vLLM. The model has 753 billion parameters, with 40 billion activated per token. Tensor parallelism of 2 keeps all weights in GPU memory, with CPU offload disabled. Station A exposes one OpenAI-compatible API while Station B runs only as a worker over ConnectX-8 RDMA.

Key facts
  • The pinned container is vllm/vllm-openai:v0.30.0-aarch64.
  • Each station needs at least 464 GB of free disk for its checkpoint copy.
  • The model repository cited is nvidia/GLM-5.3-NVFP4.
  • Load testing uses SPEED-Bench prompts from the gated cais/hle dataset.
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

12 min read at developer.nvidia.com

Running a frontier-scale model locally means keeping all its weights in GPU memory; the largest models no longer fit on one machine. Two NVIDIA DGX Stations connected over ConnectX-8 RDMA serve one model through a single endpoint, with tensor parallelism splitting the model weights and computation across both GPUs. The NVIDIA GLM-5.

Hacker News · AI

By wondersky · developer.nvidia.com

Read the full article at Hacker News · AI

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.