1. AI Data Center Design and Optimization — 5%

What: The architecture and communication patterns of GPU-based AI infrastructure.

Why: Network capacity and topology affect accelerator utilization and distributed job completion.

How: Identify system components, map scale-up/scale-out paths, and reason about rails, collectives, and communication overhead.

1.1 Describe an AI factory networking architecture and its components

What: The coordinated compute, scale-up, scale-out, storage, and management components that deliver AI training and inference.

Why: Network or service bottlenecks can leave GPUs idle and reduce job throughput even when accelerator capacity is available.

How: Diagram each component and traffic plane, mark their connections and boundaries, and explain the role and failure impact of every component.

An AI factory is designed to turn data into trained or served models. Its compute, storage, power, cooling, and network capacity must be planned as one system. Large GPU jobs create many synchronized flows; a slow or oversubscribed network can leave expensive accelerators waiting instead of computing.

Think of the architecture as several cooperating planes rather than one undifferentiated network:

Component	Role in the AI factory	Networking perspective
GPU and compute node	Execute model training or inference; nodes may contain multiple GPUs	Exchanges data within the node and with remote nodes; GPU count alone does not determine network bandwidth
NVLink and NVSwitch	Connect GPUs within supported systems	Provide high-bandwidth scale-up communication; distinct from the data-center scale-out fabric
NIC / SuperNIC	Connect a host to an Ethernet network and support data movement	Provides host-facing ports and, depending on platform, RDMA and hardware offloads
NVIDIA BlueField DPU	Programmable data-processing unit with Arm compute and networking interfaces	Can accelerate or isolate infrastructure services such as networking, security, and storage; exact functions depend on the BlueField generation and deployed software
Leaf switches	Attach servers and provide access into the fabric	Aggregate host links and connect to spine or higher-tier switches
Spine switches	Interconnect leaf blocks in a scalable fabric	Provide paths between leaves; count, port speed, and topology determine available aggregate capacity
Scalable unit (SU)	Repeatable building block used to grow an AI factory	Treat its exact contents and scale as design-specific; identify its compute, switch, cabling, and inter-unit boundaries from the reference architecture
Storage network	Connects compute to training datasets, checkpoints, and other storage services	Capacity and path design affect data loading and checkpoint traffic; may share or use separate infrastructure depending on the architecture
Management / service network	Supports provisioning, monitoring, and operations	Carries control and operational traffic and should be understood separately from GPU payload traffic

The scale-up domain connects components within a server or tightly coupled system, commonly using GPU interconnects. The scale-out domain connects servers across the data-center fabric using Ethernet/RoCE or InfiniBand. Storage and management traffic have their own requirements. One physical infrastructure may carry more than one traffic type, but sharing it does not make the traffic's performance and isolation needs identical.

Scalable units and leaf-spine design

What: A scalable unit is a repeatable design building block; leaf-spine is a multi-path fabric connecting endpoint-facing leaves through spines.

Why: Repeatable units simplify growth, while topology and link ratios determine path diversity, aggregate bandwidth, and potential bottlenecks.

How: Identify what a reference design repeats, draw a leaf-spine path, and calculate offered downlink capacity versus uplink capacity.

A scalable unit (SU) is a repeatable building block in a given reference architecture. Replicating a validated unit can simplify capacity planning, cabling, deployment, and fault isolation. Do not assume every design uses the same SU definition: identify which compute, network, storage, and service components it contains, and which links connect it to other units.

In a basic leaf-spine fabric, each host attaches to a leaf, and each leaf connects to every spine. Paths between hosts in different leaf blocks usually have a predictable number of switch hops. Multiple equal-cost paths can provide aggregate capacity and resilience, provided routing and link utilization are working as intended.

Compute / GPU nodes              Fabric
  Node A ---- Leaf A ===== Spine 1 ===== Leaf B ---- Node C
  Node B ----   |   ===== Spine 2 =====   |   ---- Node D
                |                          |
            local hosts                 local hosts

The double lines represent multiple links in a simplified drawing.
Actual link count, speed, and topology are design-specific.

Draw a small topology and trace traffic between two hosts on one leaf and on different leaves. Identify the links shared by many flows. Calculate a simple oversubscription ratio as total offered downlink bandwidth divided by total uplink bandwidth; a ratio above 1 means not all host ports can run at line rate simultaneously toward the rest of the fabric.

1.2 Describe rail-optimized topologies for high-performance AI workloads

What: A topology that aligns corresponding GPU/NIC positions across servers into parallel network rails.

Why: Independent rails can distribute collective traffic and reduce contention, provided they remain balanced and sufficiently independent.

How: Map GPU-to-NIC-to-switch-to-uplink for every rail, then trace collectives and evaluate one-link and one-rail failures.

In a rail-optimized GPU cluster, network paths are organized into parallel rails and GPU/NIC positions are connected consistently across servers. For example, traffic associated with GPU/NIC position 0 can use rail 0 across nodes, while position 1 uses rail 1. This can give distributed GPU collectives multiple parallel paths and limit competition between traffic assigned to different rails.

                Rail 0                         Rail 1
Node A: GPU 0 -> NIC 0 -> Leaf A0       GPU 1 -> NIC 1 -> Leaf A1
Node B: GPU 0 -> NIC 0 -> Leaf B0       GPU 1 -> NIC 1 -> Leaf B1
                         \  spine paths  /                \ spine paths /

Illustrative only: real GPU/NIC counts and rail-to-switch mappings vary.

Rails provide useful parallelism only when the end-to-end mapping is correct: GPU-to-NIC affinity, adapter ports, switch connectivity, routing, and application/library behavior must agree. Map each GPU to its local NIC, switch port, leaf, and uplink. Then trace a collective across multiple nodes. If one rail is unavailable, determine whether traffic fails over, loses capacity, or becomes imbalanced; do not assume automatic failover or linear bandwidth scaling. Cabling, routing, congestion, and shared uplinks can still become bottlenecks.

Rail switches are the first switching tier for GPU-facing network links in many rail-optimized designs. A rail groups corresponding endpoint links across servers so their traffic can use a consistent parallel path into the fabric. Some designs use a separate switch per rail; others integrate rail behavior into a leaf layer. Follow the reference topology and map each GPU/NIC port to its switch, uplink, and inter-unit path. A rail alone does not guarantee non-blocking bandwidth, tenant isolation, or automatic failover; those depend on capacity, topology, routing, and configuration.

1.3 Describe GPU-to-GPU communications

What: Data movement between GPUs, either within a node over supported GPU interconnects or between nodes over an RDMA-capable network.

Why: Distributed training depends on communication patterns such as all-reduce and all-to-all; a slow rank or congested path can delay all participants.

How: Trace the complete path from source GPU through software, adapter, fabric, and destination GPU; account for topology, message pattern, completion, and synchronization.

GPU-to-GPU communication can stay within one node or cross the scale-out network. The communication library/runtime and topology determine which available paths are used; an application-level “GPU-to-GPU” operation is not necessarily a single direct physical link.

Communication scope	Typical path	What to understand
Within a node	GPU ↔ NVLink/NVSwitch ↔ GPU, where supported	Link topology and bandwidth differ by platform; identify local GPU connectivity and its failure/monitoring domain
Across nodes	GPU ↔ host communication stack / RDMA ↔ NIC ↔ Ethernet/RoCE or InfiniBand fabric ↔ remote NIC ↔ GPU	The complete path includes host software, adapter, network topology, congestion behavior, and destination placement
To/from storage	GPU/node ↔ host and storage software ↔ storage network/service	Data staging, storage throughput, and checkpointing can compete with or be separate from GPU collective traffic

Collective communication patterns explain why AI workloads can stress a network differently from independent request/response flows:

• All-reduce: ranks contribute values and receive a reduced result. Training frameworks commonly use it to aggregate gradients. The algorithm may form rings, trees, or other schedules; each has different traffic and sensitivity to topology.
• All-gather / reduce-scatter: move or reduce different portions of data across ranks and are often combined to implement larger collectives.
• All-to-all: each rank exchanges data with many or all other ranks. It can create many simultaneous flows and expose oversubscription or path imbalance.

For an inter-node transfer, trace the sending GPU, communication library, RDMA operation and buffers, local NIC, fabric path, remote NIC, destination memory/GPU, and completion/synchronization. With supported GPUDirect RDMA, payload can move between adapter and GPU memory without a CPU staging copy; the supported hardware/software stack and correct configuration are required. RDMA reduces CPU data-path work but still requires setup, permissions, and software coordination.

Collectives often synchronize participants. A slow rank, congested rail, or imbalanced path may delay completion for the whole operation. Thus, aggregate link speed alone is not a performance guarantee: consider topology, number of active flows, message size, congestion, endpoint behavior, and application overlap.

RDMA and GPUDirect concepts

What: RDMA accesses registered remote memory with reduced CPU data-path involvement; GPUDirect RDMA can enable supported NIC-to-GPU-memory transfers.

Why: These mechanisms can reduce copies and CPU overhead for large-scale GPU communication, but only on compatible, correctly configured systems.

How: Verify hardware/software support, memory registration and permissions, adapter configuration, and that the workload actually uses the intended transfer path.

Remote Direct Memory Access (RDMA) lets a registered memory region on one host be accessed by a remote peer with less CPU involvement than a conventional socket-copy data path. RDMA does not mean “no software”: memory registration, queue setup, permissions, addressing, and reliable transport behavior still matter.

GPUDirect RDMA enables supported adapters to transfer data to or from GPU memory without routing every payload through a CPU-owned staging buffer. Confirm compatibility across GPU, driver, adapter, firmware, and software versions; then validate that the workload is actually using the intended data path. Remember that Ethernet RoCE needs a correctly engineered congestion and loss strategy, while InfiniBand uses its own link and fabric mechanisms.

Review questions: Which components are intra-node vs. inter-node? What creates oversubscription? Trace one GPU-to-GPU transfer across hosts. Why can RDMA improve CPU efficiency without eliminating configuration requirements?