Ran Shu

Network simulation plays a critical role in improving the efficiency of large-scale AI clusters for design validation, parameter tuning, and protocol development. However, high-fidelity network simulation becomes prohibitively slow at scale, especially when running large batches of experiments on topologies with tens or hundreds of thousands of accelerators. We observe that a key bottleneck comes from the control plane. Existing network simulators typically compute routes and install forwarding tables at initialization, which can consume hundreds of GB of memory before packet-event execution begins and limit overall simulation throughput. In this paper, we present Nüwa, which views routing as a compilation problem, it leverages the hierarchical and symmetric structure common in AI fabrics and compiles a declarative topology description together with routing policies into compact forwarding artifacts that are fast to generate and efficient to look up. Evaluations show that Nüwa can reduce simulation initialization time from hours to only 25 seconds for a 65,536-GPU cluster. For end-to-end simulation time, Nüwa takes only 20% of that required by existing approaches in a 40K+ GPU cluster, and Nüwa can scale to a 221,184-GPU cluster.

Remote Direct Memory Access (RDMA) is increasingly used as a shared communication substrate across datacenter workloads with very different scheduling needs, from request-response services and storage fan-out to AI training collectives. Proper request scheduling can reduce communication time, but in practice, no RDMA flow scheduling is enabled in datacenters, leaving traffic to simple fair sharing. We present STORM, a NIC-level scheduler for all types of RDMA workloads using NIC-only information: the known RDMA request size, and per-queue-pair backlog. STORM converts these signals into a small number of extra priority levels on the wire and prioritizes requests that are either near completion or blocking queued dependent work. STORM requires no application hints and works with both in-order RoCEv2 and newer RDMA stacks that tolerate reordering. We prototype STORM on an FPGA NIC with negligible overhead. Across representative cloud and LLM training workloads, STORM reduces training iteration time by up to 12% and reduces average and P99 flow completion slowdown by up to 90% compared to fair scheduling.

As RDMA becomes increasingly tolerant to out-of-order delivery, fine-grained packet-level multipathing is emerging as a practical design for datacenter fabrics. Packet spraying and related schemes improve load distribution for large flows, but they also force RDMA flows onto multiple paths whose conditions can differ at short timescales due to randomized traffic placement. For short flows, even one packet sent on a temporarily slower path can delay the entire flow. As a result, fine-grained load balancing can hurt, rather than help, small multi-packet flows. We present PIN, a lightweight wrapper that layers on top of any existing fine-grained RDMA load balancer. PIN applies less multipathing to short flows, which benefit from it least: it pins short flows to one path for their lifetime and leaves larger flows to the baseline mechanism. The design is simple, compositional, and requires only a size threshold. We show analytically that practical thresholds preserve overall load-balance fairness while reducing short-flow exposure to temporarily slower paths. Large-scale simulations across a variety of workloads show that PIN consistently improves both mean and tail completion time for short flows across multiple RDMA load-balancing baselines.