3. NVIDIA InfiniBand Networking — 30%

What: InfiniBand fabric components and behavior, from subnet management and endpoint RDMA to partitioning, QoS, and monitoring.

Why: This high-weight domain requires distinguishing fabric control, permissions, physical health, and data-path performance.

How: Trace a QP operation across the fabric and validate SM state, HCA ports, PKeys, paths, and counters in layers.

Fabric bring-up and Subnet Manager high availability

What: Fabric initialization discovers endpoints and switches and uses a Subnet Manager (SM) to configure subnet state; high availability provides standby management capability.

Why: InfiniBand depends on valid fabric discovery and path configuration, and an unavailable SM can prevent required management changes or recovery.

How: Verify cabling and active links first, then SM master/standby state, discovered devices, addressing/path state, and controlled failover behavior.

InfiniBand relies on a Subnet Manager (SM) to discover the fabric, assign identifiers and paths, and configure switches and endpoints. Before diagnosing software state, verify the physical layer: correct port-to-port cabling, link state, supported speed/width, and error counters. A link light alone does not prove that the port is active and configured in the fabric.

During bring-up, confirm that the intended SM is active, that switches and host channel adapters (HCAs) are discovered, and that addressing and routing state are consistent. High availability typically uses a master SM with one or more standby candidates; the exact election and failover behavior depends on the SM implementation and configuration. Test failover in a controlled environment and verify the resulting fabric state rather than assuming that a standby is ready merely because its process is running.

InfiniBand packet anatomy and end-to-end flow

What: The RDMA operation path from application work request and QP, through the source HCA and fabric, to destination memory and completion queue.

Why: It separates application setup, host transport, fabric forwarding, partition policy, and physical-link behavior for diagnosis.

How: Trace endpoints, addressing, QP/transport, PKey, service level/virtual lane, switch path, destination operation, and completion. Keep SM configuration separate from per-packet forwarding.

Separate the control/management plane from the data path. The Subnet Manager discovers the subnet and configures information such as identifiers and forwarding paths. Once the fabric is configured, the SM is not an additional switch hop for every application packet.

An RDMA application typically uses the verbs interface to create a protection domain, register memory, create a queue pair (QP), and post work requests. Memory registration establishes the permissions and keys used for remote access. A completion queue (CQ) reports completion events to the application. The exact setup and transport behavior depends on the application and selected InfiniBand transport.

Application
  -> verbs work request on a QP
  -> source HCA reads registered source buffer
  -> packet traverses links and switches on the configured path
  -> destination HCA validates and performs the requested operation
  -> destination memory is updated
  -> completion is placed on a CQ and consumed by the application

For a simplified packet walk, the source HCA forms transport and network headers for the operation. The destination QP and packet sequence information help the destination transport endpoint identify and process the traffic. Switches forward within a subnet using the configured InfiniBand path and addressing; a routed multi-subnet design can involve additional global address/routing information. Do not assume every fabric uses the same addressing or routing mode.

At each physical link, InfiniBand uses credit-based flow control: a sender must have available receive-buffer credits before transmitting on that virtual lane. QoS service levels can map traffic to virtual lanes, allowing classes of traffic to be treated separately. Credits protect the local link from overrunning the receiver's buffers; they do not prove that the end-to-end path is uncongested or that an application has enough bandwidth.

PKeys also participate in communication permission checks. A healthy physical link and a valid route are not sufficient if endpoint partition membership does not permit the exchange. Trace both endpoints' PKey configuration and membership when only a subset of peers can communicate.

Walkthrough exercise: select two endpoints and draw every HCA, switch, link, and (if present) router between them. Record the source and destination identifiers, relevant QP/transport, PKey, service level/virtual lane, and expected forwarding path from your lab documentation. Then predict what you would observe if (1) a link is down, (2) a PKey is mismatched, (3) a receive buffer has no credits, or (4) the RDMA operation completes but the application does not make progress.

Diagnosis order: confirm local HCA/port state; confirm fabric discovery and SM/path configuration; verify endpoint addressing and PKey membership; inspect link errors and VL/credit-related counters supported by the platform; then run a controlled reachability and RDMA performance test. Use iblinkinfo, ibdiagnet, UFM, and host-side tools as appropriate, and compare timestamps rather than inferring a cause from a single counter.

Partition keys (PKeys)

What: PKeys identify InfiniBand partition membership and constrain which ports may communicate under the configured membership rules.

Why: Correct routing and link state cannot make a communication succeed when endpoint partition membership does not permit it.

How: Compare PKey values and full/limited membership on both endpoints and verify the active fabric configuration before changing routes.

PKeys provide partition-based access control for InfiniBand communication. Ports use PKey membership to determine which partition traffic they may exchange. Full and limited membership are different: in general, full members can communicate with other permitted members more broadly, while limited members have more restricted communication rights. Check the exact PKey rules and operational behavior in the relevant InfiniBand/NVIDIA documentation.

Think of PKeys as a fabric-level isolation mechanism, not as a drop-in synonym for Ethernet VLANs. Verify the PKey assigned to each relevant port and the configured membership on both communicating endpoints. When communication fails only for some nodes, compare membership and partition configuration before changing routing.

QoS, virtual lanes, and adaptive routing

What: InfiniBand QoS maps service levels to virtual lanes; adaptive routing selects among available paths based on supported fabric behavior.

Why: Traffic classes may need separation, and alternate paths can help avoid congestion when the topology and configuration support them.

How: Inspect effective SL-to-VL mapping, switch/SM capabilities, route state, and per-link evidence; verify there is a usable alternate path.

InfiniBand QoS uses service levels and virtual lanes (VLs) to map traffic into separate link-level queues. VLs can help isolate traffic classes and avoid one blocked class holding up unrelated traffic, subject to correct configuration and available resources. Learn how service levels map to VLs in the deployed fabric and how to inspect the effective state.

Adaptive routing can select paths based on fabric conditions rather than always relying on one static route choice. Understand that the SM and switch capabilities/configuration influence available paths. It cannot route around every failure if the topology has no alternate path; confirm topology, supported features, and route state.

Unified Fabric Manager (UFM)

What: UFM is a fabric-management solution for supported InfiniBand deployments, with topology, health, and operational views.

Why: It helps establish whether an incident is local or fabric-wide and where to focus physical or host-side investigation.

How: Scope the event in UFM, inspect affected devices/links and counters, then corroborate with HCA state and targeted diagnostics at matching timestamps.

UFM is NVIDIA's fabric management and monitoring solution for supported InfiniBand environments. Use it to view topology and device/link status, monitor performance and health, and help locate problematic links or devices. Know the difference between a visualization/alert and the underlying counters or state that support a diagnosis.

For a fabric incident, record the scope, affected endpoints, time window, link state, error counters, and recent changes. Compare UFM's view with host-side HCA information and targeted fabric diagnostics. Familiarize yourself with the available UFM features in the version used in your lab.

Review questions: What fabric state does the SM establish? What is the difference between PKey membership levels? How do VLs relate to QoS? What evidence would you collect before replacing a cable or changing a partition?