5. Troubleshooting Tools — 20%¶
What: Host, switch, fabric, and benchmark tools used to investigate connectivity and performance symptoms.
Why: Choosing a tool that measures the wrong layer can create false confidence or hide the actual failure domain.
How: Scope the incident, select the narrowest useful check, correlate results and timestamps, and escalate from link state to end-to-end performance.
A reliable troubleshooting sequence¶
What: A scoped, evidence-driven sequence from symptom definition through physical/link, addressing, routing/policy, congestion, transport, and application checks.
Why: Layered diagnosis reduces guesswork and avoids changing configuration before the failure domain is understood.
How: Record endpoints, direction, timestamps, scope, and recent changes; test one hypothesis at a time and capture before/after evidence.
Start by defining the symptom: affected workload, endpoints, direction, start time, frequency, and recent changes. Scope whether it is one host, one link, one rack, one tenant, or the whole fabric. Move from the physical and link layers toward addressing, routing/partitioning, congestion, transport, and application behavior. Change one variable at a time and capture before/after state.
For every diagnostic command, know what device it runs on, what layer it checks, what a healthy result looks like, and what it cannot prove. Command options and output can vary by OS, driver, and software version; confirm syntax with the installed tool's help and vendor documentation.
Ethernet and switch diagnostics¶
What: Switch-level tools and telemetry for investigating drops, resource limits, configuration state, and fabric-wide health.
Why: Host symptoms alone may not reveal the switch port, queue, or hardware event responsible for an Ethernet/RoCE problem.
How: Select WJH for event context, NetQ for broader fabric context, and
cl-resource-query for supported switch resource questions; correlate with
host and port counters.
cl-resource-query: inspect supported switch resource allocation and limits when a configuration or forwarding resource appears exhausted. Pair its output with the specific feature/configuration and switch logs; it is not a general end-to-end connectivity test.- WJH: investigate reported packet-drop events and their switch-side reason/context. Correlate the event with the affected port, queue, time, and host symptoms.
- NetQ: use fabric-wide state and health views to determine whether configuration, topology, or operational issues are isolated or widespread.
InfiniBand discovery and link tools¶
What: Host and fabric utilities that inspect HCA ports, discovered nodes, reachability, topology, and link details.
Why: Different commands answer different questions; discovery, reachability, and physical health are not interchangeable proofs.
How: Start with local ibstat, then use ibnodes, ibping,
iblinkinfo, ibdiagnet, or UFM according to the symptom and operational
scope.
ibstat: inspect local HCA and port state, including link properties reported by the installed stack. Start here when one host cannot join or use the fabric.ibnodes: check which nodes are visible to the fabric tools. A missing node suggests discovery, link, SM, or configuration issues; it does not alone identify which one.ibping: perform a basic InfiniBand path/reachability check between endpoints when configured and permitted. It is a narrow test, not a throughput benchmark.iblinkinfo: inspect fabric link state and reported link details to narrow a topology-wide symptom to a port or connection.ibdiagnet: run a broader fabric diagnostic scan to discover topology and report supported errors or inconsistencies. Review impact and runtime before using extensive scans on a production fabric.- UFM health views: combine fabric-wide monitoring with targeted checks on the links and devices implicated by the symptom.
RDMA performance tools¶
What: Perftest utilities such as ib_write_bw and ib_write_lat measure
controlled RDMA bandwidth and latency between endpoints.
Why: They help determine whether an RDMA path meets an expected baseline and narrow performance issues beyond simple reachability.
How: Use compatible client/server options, record message size and other settings, run repeatable baselines, and compare like-for-like results while monitoring host and fabric counters.
ib_write_bw and ib_write_lat are perftest utilities for measuring RDMA
write bandwidth and latency between configured endpoints. Use matching
software and compatible options on both sides, follow the tool's server/client
workflow, and record message size, queue depth, iterations, transport, and
other relevant settings. Run a baseline before drawing conclusions.
Benchmarks can consume resources and results depend on hardware, topology, concurrency, CPU placement, and configuration. Compare like with like, test both directions when relevant, and avoid treating one synthetic result as a guarantee of application performance.
Symptom-to-tool decision practice¶
What: A mapping from common symptoms to a first diagnostic check and evidence-driven follow-up.
Why: It provides a disciplined starting point without assuming every symptom has the same root cause.
How: Choose the row matching the observed scope, run the first checks, and select the next test from the evidence rather than blindly executing every tool.
| Symptom | First checks | Follow-up |
|---|---|---|
| One host has no InfiniBand link | ibstat, host logs, cable/port |
iblinkinfo, UFM, SM state |
| Expected endpoint is absent | ibnodes, host link state |
ibdiagnet, UFM topology, cabling |
| Reachability fails between IB peers | local port state, addressing/partition | ibping, route/SM and PKey checks |
| Link is up but RDMA is slow | baseline perftest, counters, host errors | ib_write_bw, ib_write_lat, congestion and topology |
| Ethernet packets are reported dropped | switch port/queue and host counters | WJH event details, NetQ fabric context |
| Switch configuration/resource issue suspected | config state and logs | cl-resource-query, supported resource limits |
Treat this table as a starting point. Choose the next test based on evidence and write down what each result rules in or out.
Practice: choose a fictional symptom, write the first three commands or views you would use, predict healthy and unhealthy findings, and state the next branch for each result.