2. NVIDIA Spectrum Networking — 30%¶
What: Ethernet networking concepts and NVIDIA platform capabilities used to connect AI workloads, including RoCE, QoS, routing, and operations.
Why: This high-weight domain tests how to configure, validate, and diagnose AI traffic across an Ethernet fabric.
How: Trace the host-to-host path, validate configuration end to end, and correlate host and switch telemetry under representative load.
2.1 Configure NVIDIA Spectrum-X switches for RoCE¶
What: RoCE carries RDMA traffic over Ethernet; Spectrum-X is NVIDIA's Ethernet platform for AI networking.
Why: The endpoint RDMA stack and Ethernet fabric must work together for reachability, congestion handling, and predictable performance.
How: Validate the host adapter/driver, addressing, routes, MTU, QoS, and congestion behavior end to end; test performance beyond basic IP reachability.
RoCE (RDMA over Converged Ethernet) carries RDMA traffic over Ethernet. RoCEv2 uses IP/UDP encapsulation, so normal Ethernet/IP reachability and the RDMA configuration both matter. A successful ping is not proof that RDMA is configured, and a working RDMA session is not proof that the fabric is performing well under load.
Study the entire path: host adapter and driver, addressing and routing, switching, MTU consistency, traffic classification, congestion control, and the application’s RDMA settings. Establish a baseline with a single pair of hosts before adding scale or multiple traffic classes. Check product and software-version documentation for exact commands and supported features; switch syntax and defaults can vary.
Spectrum-X is NVIDIA's Ethernet platform for AI workloads. Learn the roles of switches, adapters, and software as an end-to-end system rather than assuming that one switch feature alone guarantees performance. Be ready to diagnose asymmetric paths, mismatched MTUs, incorrect priorities, and congestion points.
RoCE bring-up and validation workflow¶
What: A staged process validates the switch, host adapter, and Ethernet/RDMA configuration as one end-to-end path.
Why: RoCE depends on compatible endpoint and fabric behavior; IP reachability alone does not establish RDMA operation or performance.
How: Confirm platform, firmware, driver, and software compatibility; verify physical links, addressing, routes, and end-to-end MTU; check host RDMA device and traffic-class settings; align switch QoS and congestion configuration; test one host pair before adding concurrent flows; record counters and a rollback plan before scaling changes. Use release-specific NVIDIA documentation for commands and supported features.
Benchmarking with CloudAI Benchmark¶
What: CloudAI Benchmark measures AI networking behavior under defined, supported test conditions.
Why: Link counters and basic point-to-point checks do not necessarily predict performance for an application's communication pattern or a multi-node workload.
How: Record topology, versions, workload, scale, and baseline; run supported tests and correlate throughput and latency with endpoint and switch telemetry. Compare like-for-like runs after changing one variable. Follow current documentation; a score alone is not a root-cause diagnosis.
2.2 Enable and verify QoS, ECN, PFC, and advanced features¶
What: QoS classifies traffic, ECN marks congestion for endpoint reaction, and PFC pauses a selected priority on a link.
Why: Consistent traffic treatment and carefully designed congestion response help protect latency-sensitive and RDMA flows under load.
How: Inspect classification, queue mapping, ECN marks, PFC pause counters, drops, and endpoint response together; do not use PFC as a substitute for congestion control.
Quality of Service (QoS) classifies traffic and maps it to queues or priorities. The mapping must be consistent across hosts and switches; otherwise traffic can enter the fabric in an unexpected class. Preserve capacity for control and management traffic so bulk data does not starve essential protocols.
Explicit Congestion Notification (ECN) allows a congested queue to mark packets before it drops them. A capable receiver/adapter can react by reducing its sending rate. Priority Flow Control (PFC) pauses a selected priority on a link when buffers approach a threshold. PFC can limit loss locally, but poorly designed pause propagation can cause head-of-line blocking or congestion spreading. It is not a substitute for congestion control or correct buffer and queue design.
Learn the intended sequence: classify the flow, observe queue buildup, signal congestion with ECN, and use PFC only as a carefully engineered safeguard where the design requires it. Validate priority-to-queue mappings, ECN thresholds, PFC counters, pause duration, drops, and end-host reaction together. Never copy thresholds from another deployment without checking platform guidance and traffic characteristics.
Adaptive routing and telemetry¶
What: Adaptive routing can select among paths using fabric conditions; telemetry exposes utilization, congestion, latency, and events.
Why: Static path hashing may imbalance synchronized AI flows, while telemetry helps distinguish fabric congestion from endpoint or application delays.
How: Compare per-path and per-queue measurements on synchronized timestamps and confirm actual path selection and platform feature support.
Equal-cost multipath (ECMP) commonly selects a path using flow attributes. Several large synchronized flows can hash onto the same path while other paths remain underused. Adaptive routing can select among available paths based on current conditions, but actual behavior depends on platform support, topology, configuration, and flow type.
Telemetry makes the behavior observable. Compare per-link and per-queue utilization, congestion marks, drops, latency, and retransmission or recovery indicators over the same time window. Distinguish a congested link from a slow endpoint or an application waiting on synchronization. Learn how the chosen NVIDIA monitoring tools collect and display fabric state, and verify sampling intervals and time synchronization before correlating events.
2.3 Configure multi-tenancy with BGP EVPN¶
What: EVPN distributes overlay reachability using BGP; VTEPs encapsulate and decapsulate VXLAN; VNIs identify overlay segments.
Why: Together they enable tenant segmentation over a routed, shared underlay and provide control-plane endpoint learning.
How: Verify underlay VTEP reachability, BGP EVPN routes and policy, VNI mapping, and host attachment; trace both inner and outer packet headers.
VXLAN encapsulates an overlay Ethernet frame inside an IP/UDP packet to carry Layer 2 segments across a routed underlay. A VXLAN Tunnel Endpoint (VTEP) encapsulates and decapsulates that traffic. A VXLAN Network Identifier (VNI) identifies an overlay segment. EVPN uses BGP to distribute reachability information so VTEPs can learn which remote endpoint is reachable through which tunnel.
Keep the underlay and overlay separate in your mental model. The underlay must provide IP reachability between VTEP addresses; the overlay provides tenant connectivity. Trace a packet from a host through its local VTEP, across the underlay, and to the remote VTEP. Know where MAC/IP learning occurs, how a VNI maps to a tenant segment, and why route-target/import policy and correct addressing matter. For L3 services, study the applicable EVPN model and its control-plane routes from official documentation.
Practice: draw two tenants on shared switches, each with isolated VNIs. Explain how you would check VTEP reachability, BGP session state, route advertisement, VNI mapping, and host attachment when one tenant cannot reach a remote host.
Microsegmentation case study: Two tenants share the physical Ethernet fabric but must remain isolated. Verify the source access attachment maps to the intended VLAN/VNI and VRF, EVPN route-target policy imports only expected tenant routes, and ACL or service policy permits only the intended flows. Test both allowed same-tenant communication and denied cross-tenant communication. If isolation fails, inspect route learning, VNI/VRF mapping, VTEP reachability, policy counters, and both inner and outer packet headers; successful connectivity alone does not prove tenant separation.
2.4 Use NVIDIA Air to simulate network environments¶
What: A network simulation environment for supported NVIDIA networking topologies and configurations.
Why: Simulation allows safe practice and configuration validation before changes are attempted on physical infrastructure.
How: Build a supported lab, record topology and assumptions, test normal and failure cases, and separately validate physical behavior on real hardware.
NVIDIA Air is a network simulation environment useful for learning and validating supported topologies and configurations without first changing a production fabric. Treat simulation results as evidence for the simulated features only: it cannot prove physical cabling, optics, actual hardware capacity, or behavior of features not represented by the environment.
Use a lab to practice configuration changes, expected state, and failure scenarios. Save the topology, assumptions, commands, and before/after output with your notes. Confirm current access and feature availability in NVIDIA's documentation.
2.5 Diagnose congestion or packet loss with in-band telemetry and WJH¶
What: WJH helps investigate switch-level packet events; NetQ provides broader fabric operations and health visibility, depending on deployment.
Why: One narrows a specific event while the other helps determine whether the condition spans devices or the wider fabric.
How: Correlate timestamps, device/port/queue context, counters, and host symptoms; use both as evidence rather than assuming either alone is root cause.
What Just Happened (WJH) helps investigate switch events such as packet drops and their reported causes. Use it to narrow a specific event to a port, reason, or relevant hardware context, then correlate its timestamp and counters with the host and other switches. A reported drop reason is a clue to validate, not by itself a complete end-to-end root cause.
In-band telemetry, where supported and enabled, can expose path and queue observations for selected traffic. Correlate those records with WJH, switch counters, and host symptoms; check timestamps, sampling interval, and telemetry scope.
2.6 Use NetQ for real-time network monitoring¶
What: NVIDIA NetQ provides fabric-wide operational visibility, including topology, health, and supported congestion or latency measurements depending on deployment and release.
Why: Fabric context helps determine whether a problem is isolated to a device or link, follows a path, or spans the wider network.
How: Confirm devices report to NetQ, inspect topology and health, then compare time-aligned latency, congestion, link, and event data for the affected path. Correlate findings with switch, WJH, telemetry, host RDMA, and workload evidence. Metrics vary by version; a dashboard alone does not prove root cause.
NetQ provides broader fabric operations and visibility, including topology and health information depending on the deployed version and configuration. Use it to identify whether an issue is isolated or spans multiple devices, and to compare configuration or operational state across the fabric. Know when to use event-level switch evidence (WJH) versus fabric-wide context (NetQ); neither replaces checking endpoint drivers, application symptoms, and physical links.
2.7 Install NVIDIA DOCA¶
What: DOCA is NVIDIA's software development framework for supported networking platforms; SuperNICs are high-performance adapters for demanding network workloads.
Why: Software frameworks and adapter offloads affect how data-plane, RDMA, congestion, and telemetry functions are implemented.
How: Identify the host/DPU component and product generation, verify version compatibility, and inspect supported offloads and operational state using current product documentation.
DOCA installation workflow: Identify the BlueField generation, operating mode, host OS, firmware, and required services. Select a supported DOCA release and follow its host- or DPU-side installation guide, including prerequisites and any documented reboot or firmware steps. Verify the installed version, device and driver state, required services, and the specific application or offload. Do not mix packages from incompatible releases or assume that installation enables every feature.
DOCA is NVIDIA's software development framework for supported DPUs and networking platforms. It includes APIs, libraries, and tools; it is not a single switch command or a synonym for all networking software. Understand which host or DPU component runs a given service and which device/driver versions it requires.
2.8 Configure NVIDIA SuperNIC functionality¶
What: NVIDIA SuperNICs provide high-performance host networking and, depending on product and software support, accelerated packet processing, RDMA, congestion management, and telemetry.
Why: Adapter offloads can improve data-path efficiency and cooperate with the Ethernet fabric to manage AI traffic at scale.
How: Verify adapter model, firmware, driver, link and RDMA state, then validate only the packet-processing and congestion features documented for that product and release. Compare a controlled baseline under representative load using adapter counters, switch queues, ECN/PFC statistics, and workload measurements.
SuperNIC refers to high-performance adapters designed for demanding accelerated-computing network workloads. Depending on product and deployment, hardware and software can support packet processing, RDMA, congestion management, and telemetry. Learn the role of the adapter in the end-to-end data path and how to check link state, firmware/driver compatibility, and relevant offload configuration from current product documentation.
Review questions: Why is ping insufficient to verify RoCE? How do ECN and PFC differ? What does each of VTEP, VNI, and EVPN do? Which tool would you use first for a single switch drop versus a fabric-wide health question?