Serving GLM-5.3 Across Two Nvidia DGX Stations With vLLM Reaches 607.8 Tokens per Second

Engineers have demonstrated a high-throughput deployment serving the massive GLM-5.3 artificial intelligence model across two Nvidia DGX stations using the open-source vLLM inference engine. By distributing the workload across two networked systems, the setup achieved an inference throughput of 607.8 tokens per second. The milestone highlights how multi-node clustering can run state-of-the-art open models without relying on slow central processing unit offloading.

The GLM-5.3 architecture represents a massive mixture-of-experts model containing roughly 753 billion total parameters. Even when quantized to the compact NVFP4 numeric format, the model checkpoint requires approximately 464 gigabytes of memory. While only around 40 billion parameters are active for any given token, the sparse routing mechanism requires the entire parameter footprint to remain resident in high-bandwidth memory.

This substantial memory requirement creates a strict barrier for single-machine serving environments. Top-tier enterprise accelerators, such as Nvidia GB300 systems featuring 252 gigabytes of high-bandwidth HBM3e memory, cannot host the complete 464-gigabyte checkpoint on an individual unit. Traditional workarounds that swap inactive weights to host system memory introduce severe latency bottlenecks that degrade real-time generation speeds.

To bypass the memory ceiling, the engineering team linked two separate DGX stations using high-speed ConnectX-8 Remote Direct Memory Access interconnects. This dedicated fabric provides the ultra-low latency and wide bandwidth required for constant data exchange between the physical chassis. The resulting cluster pools the memory of both systems into a unified execution environment capable of holding the full model.

Rather than employing pipeline parallelism, which processes model layers sequentially across nodes, the deployment implemented distributed tensor parallelism with a tensor parallel size of two. Under this configuration, individual matrix calculations and weight tensors are divided evenly between the two machines. Both stations actively compute every incoming prompt simultaneously and continuously synchronize intermediate activations across the network.

The software stack relies on vLLM to manage the distributed execution flow and memory paging. The inference framework coordinates the multi-node worker processes through communication backends while presenting a single, standard OpenAI-compatible application programming interface endpoint. Clients interact with the cluster through one primary gateway node without needing custom networking code.

Reaching 607.8 tokens per second demonstrates that network communication overhead does not cripple real-time generation when modern interconnects are paired with optimized tensor kernels. For large mixture-of-experts models, rapid token throughput is essential for interactive chat applications, code generation, and autonomous agent loops. The benchmark proves that distributed serving can deliver production-grade responsiveness for models previously restricted to cloud-scale data centers.

The architecture offers a practical blueprint for enterprise research laboratories and localized data centers seeking to deploy frontier-tier open models independently. Organizations can utilize compact workstation clusters in office or lab environments instead of provisioning full supercomputing rows. This flexibility lowers infrastructure barriers for teams experimenting with highly capable open weights.

As open-source foundational models continue to scale beyond hundreds of billions of parameters, distributed multi-node serving will become a standard operational requirement. Continued improvements in open inference frameworks and low-latency networking protocols will likely streamline multi-node deployment workflows further. Developers and systems architects are expected to build on these techniques to accommodate the next generation of massive multimodal models.