Troubleshooting FSx for Lustre: How to Perform EFA to TCP Failover for kefalnd_abort_tx Failures

This is a troubleshooting guide for FSx for Lustre connectivity failures caused by EFA (Elastic Fabric Adapter) data plane issues. The guide covers how to identify EFA-specific Lustre mount drops, fall back to Transmission Control Protocol (TCP) as a short-term mitigation, and restore EFA once the root cause is resolved. This document will cover diagnostic commands, a step-by-step fallback procedure, and a decision framework for when to use this approach.

Amazon FSx for Lustre combined with Elastic Fabric Adapter (EFA) is the recommended architecture for high-performance I/O in large-scale ML training on AWS, to include SageMaker HyperPod, Amazon EKS, and AWS ParallelCluster. EFA provides OS-bypass networking that delivers lower latency and higher throughput than standard TCP.

However, EFA data plane failures can cause Lustre mounts to drop on a subset of nodes in a training cluster and how falling back to TCP on affected clients restored service while the EFA team investigated the root cause. This guide is meant to capture this case and allow for faster customer resolution in the event you come across a similar issue.

Test all commands in a non-production environment before unmounting and changing network transport configurations. Coordinate with workload owners before executing.

---

How FSx for Lustre Uses EFA

FSx for Lustre communicates with compute nodes through a client-side networking stack called LNET (Lustre Networking). At the bottom of that stack sits a Lustre Network Driver (LND). An LND is a pluggable module that connects Lustre to the physical network.

[ ML Training Workload (PyTorch, etc.) ]
                    ↕
[ POSIX / VFS Layer ]
                    ↕
[ Lustre Client (llite) ]
                    ↕
[ LNET (routing) ]
                    ↕
[ LND (Network Driver) ]
        ┌───────────┴───────────┐
  [ kefalnd ]             [ tcp LND ]
  (EFA path)              (TCP/IP path)

Available LND Modules

Lustre has pluggable network drivers and switching between them, specifically kefalnd and tcp, is architecturally supported. This means in the event EFA breaks, you can fall back to TCP because Lustre was designed to support multiple transports without having to rebuild your clusters or filesystems. FSx servers accept connections over both paths.

LND Module	Network Type	Typical Throughput
kefalnd	AWS EFA (OS-bypass)	12.5+ GBps (100+ Gbps)
tcp	Standard TCP/IP (kernel)	Up to 12.5 GBps (100 GBps) *
o2iblnd	InfiniBand (RDMA)	Varies

*FSx for Lustre documented per-client TCP ceiling (AWS Blog, Nov 2024). Actual throughput also limited by instance network bandwidth.

Why Doesn't Lustre Failover Automatically?

FSx Lustre-over-EFA multi-rail is designed for bandwidth aggregation (16 EFA NIDs in parallel), not cross-transport redundancy. Automatic EFA→TCP failover does not occur because:

• UDSP (User-Defined Selection Policy) pins EFA as priority 0: LNet will not choose TCP while UDSP is active

• kefalnd does not report failures to LNet's health framework: EFA NI health stays at 1000 (maximum) even during active abort_tx failures, so LNet thinks EFA is healthy

• Heartbeats/pings travel exclusively over EFA once connected: TCP is only used during the initial mount handshake

• Server eviction timeout (227s) is shorter than the time needed for LNet to accumulate enough health decrements to trigger transport switching

This is by design; manual intervention is required to switch transports.

---

Recognizing EFA-Related Lustre Failures

Lustre mount problems can be due to memory pressure, server-side faults, or misconfiguration. While symptoms may be similar, the following indicators help confirm if EFA is the issue before executing a transport fallback:

• Lustre mounts dropping on a subset of nodes (not all nodes)

• dmesg showing kefalnd_abort_tx errors

• Peer addresses resolving to 0.0.0.0@efa (kefalnd probing an unresolved peer during reconnection, indicating the driver cannot resolve the target peer's EFA address)

• OST evictions (FSx servers kicking clients for unresponsiveness)

• OSC connections cycling repeatedly (disconnect → reconnect loops)

• No OOM (Out of Memory) events on the affected nodes

Differential Diagnosis

Symptom	Likely Cause
All nodes lose mount simultaneously	FSx server-side issue
Subset of nodes with OOM in dmesg	Memory pressure → client crash
Subset of nodes with kefalnd errors, no OOM	**EFA data plane issue ← this article
Mount fails on fresh boot	Configuration / security group issue

Diagnostic Commands

# Check LNET network interfaces and transport
lctl list_nids
lnetctl net show

# Check peer connectivity
lnetctl peer show

# Look for EFA-specific errors
dmesg | grep -i "kefalnd\|efa\|lnet"

# Check Lustre mount status
mount | grep lustre

# Check OSC connection state
lctl get_param osc.*.import | grep "state:" | sort | uniq -c

# Check for OOM events (rule out memory pressure)
dmesg | grep -i "oom\|out of memory\|killed process"

Server-side indicator (if you have access to STLI or CloudWatch kernel logs):

kefalnd_tx_complete() Device[efa_0] QP[0] received TX[CONN_PROBE] completion 
with err. opcode[0] status[15] vendor[9] peer_ni[x.x.x.x@efa]

status[15] = EFA_IO_COMP_STATUS_LOCAL_ERROR_UNREACH_REMOTE : confirms the server cannot reach the client's EFA NID.

Interpretation

If you see kefalnd_abort_tx errors with 0.0.0.0@efa as the destination, and affected nodes have no OOM history, this is most likely an EFA data plane issue. The null destination indicates the EFA LND cannot resolve the target peer's EFA address, causing all outbound I/O to fail. FSx servers then evict the unresponsive clients (after 227 seconds without heartbeat), and the resulting reconnect cycling creates a feedback loop that Lustre cannot break on its own. This confirms the issue is below the filesystem layer and that a transport-level change (EFA → TCP) is the appropriate short-term mitigation.

---

A Real-World Enterprise Example

A customer running a large SageMaker HyperPod cluster with FSx for Lustre over EFA reported a cascade of failures: hung_task kernel messages, EFA-LND reporting no available networks, OST RPC timeouts, and a collapse in read throughput. HyperPod detected and labeled the affected nodes but did not trigger automated remediation. The nodes remained schedulable despite being non-functional. Production ML training was blocked for an extended period.

Errors observed:

Error Signature	What It Indicates	What It Rules Out
kefalnd_abort_tx to 0.0.0.0@efa	EFA LND cannot resolve peer address, probing unresolved peer during reconnection	Server-side issue
OST evictions	FSx servers evicting clients that apprear unresponsive	Client-side config error
OSC connection cycling	Connections dropping and restoring in a feedback loop	Transient network blip

OOM was initially suspected because it is one of the most common causes of Lustre mount drops. When a node runs out of memory, the kernel kills the Lustre client process and the mount is lost. However, kernel logs on the affected nodes showed zero OOM events, definitively ruling this out.

If you reboot all server hosts to clear any server-side state corruption, but mounts continue to drop, then you have a strong confirmation that the issue is neither memory-related nor server-side. This then resulted in the team inspecting the EFA driver layer which is invisible to standard filesystem diagnostics.

---

The Fallback Procedure: EFA → TCP

When to use:

• Active production pain is present (revenue loss, productivity halted)

• Config and OOM is NOT the root cause

• Confirmation that the workload throughput is within TCP-achievable range

Do NOT use when:

• The issue is on the server-side (all clients affected)

• The workload genuinely requires EFA-level latency

• Per-node throughput exceeds ~10 GBps (80 Gbps)

Step-by-Step: Failover to TCP

# 1. Unmount the filesystem (kill processes using it first if needed)
sudo fuser -km /fsx  # Optional: kill processes using the mount
sudo umount /fsx

# 2. Shut down LNET cleanly (flushes in-flight messages, drops peer connections)
sudo lctl network down

# 3a. Surgical removal: remove only the EFA network interface (preferred):
sudo lnetctl net del --net efa --if <interface>
# Find interface names with: lnetctl net show | grep "interfaces" -A1

# 3b. Full teardown (if 3a fails due to module refcount or bad LNet state):
sudo lustre_rmmod

# 4. Disable the EFA configure service (prevents EFA on reload)
sudo systemctl disable --now configure-efa-fsx-lustre-client.service

# 5. Reload LNet with TCP only (specify your network interface)
sudo modprobe lnet networks="tcp(enp72s0)"  # replace enp72s0 with your interface
sudo modprobe lustre

# Verify Lustre port 988 connectivity before mount:
nc -zv <fsx-mount-target-ip> 988

# 6. Remount (LNet will use TCP since only TCP NI is configured)
sudo mount -t lustre <fsx-dns>@tcp:/<fs-name> /fsx

# 7. Verify
lctl list_nids        # Should show only: 10.2.x.x@tcp
lnetctl net show      # Should show only tcp net type, no efa
mount | grep lustre   # Should show the mount

Important Notes

• This changes only the client-side transport; FSx servers support both EFA and TCP simultaneously

• NCCL (NVIDIA Collective Communications Library) / GPU-to-GPU communication still uses EFA; only Lustre I/O traffic moves to TCP

• This is a per-node operation: you can do it on just the affected subset without disrupting healthy nodes

• For SageMaker HyperPod: Update the lifecycle script to disable the EFA configure service and load LNet with TCP only

Server-side stale state: After disabling EFA on clients, FSx servers may retain stale EFA peer NID references. If clients experience connectivity issues after switching to TCP, coordinate with AWS Support.

---

TCP Performance Expectations

Falling back to TCP adds ~3-5x latency on small I/O and caps throughput at instance network bandwidth, but most ML workloads (bursty checkpoints, sequential reads, metadata ops) stay well under the 12.5 GBps (100 Gbps) per-client TCP ceiling. If per-node throughput is below ~10 GBps (80 Gbps), TCP fallback is viable.

TCP will NOT suffice for: frequent small random I/O (e.g., RL environment resets), sustained throughput exceeding instance network capacity, or latency-sensitive synchronous writes in tight training loops. For those profiles, pursue EFA remediation or new hardware instead.

---

Switching Back to EFA

Only do this after the root cause is confirmed fixed (e.g., faulty instance replaced, EFA driver patched, or FSx servers rebooted to clear stale state).

# 1. Unmount the filesystem
sudo umount /fsx

# 2. Shut down LNET cleanly
sudo lctl network down

# 3. Remove all Lustre/LNet modules
sudo lustre_rmmod

# 4. Re-enable the EFA configure service
sudo systemctl enable configure-efa-fsx-lustre-client.service

# 5. Start the EFA configure service (adds EFA NIs, sets UDSP, enables discovery)
sudo systemctl start configure-efa-fsx-lustre-client.service
# This performs: lnetctl net add --net efa --if <device> (x16),
# lnetctl udsp add --src efa --priority 0,
# lnetctl set discovery 1

# If the service is unavailable, manually configure:
# sudo modprobe lnet
# sudo modprobe lustre
# sudo lnetctl net add --net efa --if <device> --peer-credits 32  # for each EFA device
# (--peer-credits 32 per AWS docs: 
# https://docs.aws.amazon.com/fsx/latest/LustreGuide/configure-efa-clients.html)
# sudo lnetctl udsp add --src efa --priority 0
# lnetctl global show  # verify discovery: 1; if reset after lustre_rmmod, run: sudo lnetctl set discovery 1
# Find NID/interface values with: ls /sys/class/infiniband/rdmap*/
# NOTE: No need to remove/re-add TCP as it should still be present from the failover state.

# 6. Verify EFA NIs are configured
lctl list_nids
# Should show: 10.2.x.x@tcp + 16x @efa NIDs

lnetctl net show
# Should show: lo, tcp, efa

lnetctl udsp show
# Should show: src efa priority 0

# 7. Remount
sudo mount -t lustre <fsx-dns>@tcp:/<fs-name> /fsx

# 8. Verify EFA is being used (check peer send counts after some IO)
lnetctl peer show | grep -A20 "primary nid: <server_tcp_nid>" | grep "send_count"
# EFA peer NID should show increasing send_count

# Generate I/O to confirm EFA carries traffic:
dd if=/fsx/testfile of=/dev/null bs=1M count=100 2>/dev/null
# Then re-check send_count ; EFA peer NIDs should increment

Pre-flight checks before failback:

1. ✅ Faulty instance replaced or hardware issue resolved (DHC passed)

2. ✅ FSx server fleet rebooted (to clear stale EFA NID references)

3. ✅ EFA driver version confirmed safe (modinfo efa | grep version)

4. ✅ Validate with a small I/O benchmark before resuming full training

5. ✅ Monitor dmesg for 15-30 minutes for any recurrence of kefalnd errors

Important Notes

• NCCL is separate: Falling back Lustre to TCP does NOT affect GPU collective communication. NCCL will still use EFA for allreduce / allgather operations.

• Parallel paths: Always pursue "restore service" (TCP fallback) and "root cause" (EFA investigation) simultaneously. Don't wait for root cause before offering mitigation.

• HyperPod lifecycle scripts: Remember to revert lifecycle scripts after EFA is restored, or new nodes will continue mounting via TCP.

• discovery: 1 side effect: Enabling discovery allows LNet to probe all peers for additional NIDs. On long-running nodes, this may probe @tcp1 management peers that don't have EFA, generating benign peer X@tcp1 has no available nets errors in dmesg. These do not affect data-path connectivity.

• Update lifecycle scripts back to EFA once confirmed stable

---

Quick Command Reference Card

Step	Command & Action
Diagnose	dmesg | grep kefalnd
Check transport	lctl list_nids / lnetctl net show
Check OSC state	lctl get_param osc.*.import | grep "state:"
Unmount	sudo fuser -km /fsx && sudo umount /fsx
Graceful LNET shutdown	sudo lctl network down
Full module teardown	sudo lustre_rmmod
Disable EFA service	sudo systemctl disable --now configure-efa-fsx-lustre client.service
Reload with TCP	sudo modprobe lnet networks="tcp(enp72s0)" && sudo modprobe lustre
Remount	sudo mount -t lustre <fsx-dns>@tcp:/<fs-name> /fsx
Verify TCP only	lctl list_nids (should show only @tcp)
Failback to EFA	umount > lctl network down > lustre_rmmod > systemctl enable+start EFA service > mount

---

Common Errors & Roadblocks

1. "TCP can't handle our throughput"

This is the most common objection. To evaluate it correctly, you must distinguish between aggregate filesystem throughput and per-node throughput. These are fundamentally different numbers:

Metric	What it means	TCP Ceiling
Aggregate throughput	Total I/O across ALL nodes to the filesystem	Limited only by FSx server capacity (hundreds of GBps)
Per-node throughput	Max I/O from a SINGLE client to FSx	~12.5 GBps (100 Gbps) per client over TCP

Example: If CloudWatch shows 15 GBps aggregate throughput across 51 nodes, the per-node average is only ~300 MBps (trivial for TCP). Even if checkpoint bursts are 10x the average, that's still only ~3 GBps per node, well below the 12.5 GBps TCP ceiling.

How to validate before proposing TCP fallback:

1. Pull FSx CloudWatch metrics: DataReadBytes and DataWriteBytes (total filesystem throughput)

2. Divide by number of active client nodes to get average per-node throughput

3. Check peak checkpoint window (typically 1-5 min bursts) for max per-node I/O

4. Compare per-node peak against 12.5 GBps (100 Gbps) TCP ceiling

5. If per-node peak < 10 GBps → TCP fallback is viable

6. If per-node peak > 12 GBps → TCP will bottleneck; consider partial fallback or wait for EFA fix

2. Customer rejects without testing

The customer may say "TCP won't work for AI training" based on theoretical concern rather than data. Counter with:

• Present the actual throughput data (aggregate vs per-node breakdown)

• Propose a 1-node trial: "Let's switch one node to TCP and run a mini-benchmark. If it fails, we revert in 5 minutes"

• Emphasize NCCL/GPU comms still use EFA so that only storage I/O moves to TCP

• Frame as temporary: "This buys us time while EFA team investigates, not a permanent change"

3. "We can't unmount due to jobs that are running"

If the mount is already in a hung state (which is likely if kefalnd is broken), the jobs are already failing. Unmounting allows recovery:

# Kill processes using the mount point
sudo fuser -km /fsx 2>/dev/null

# Unmount
sudo umount /fsx

# If unmount hangs (D-state processes), use lazy unmount
sudo umount -l /fsx

# After lazy unmount, modules may not unload immediately.
# Wait up to 3 minutes, or use:
timeout 180 sudo lustre_rmmod

# If still hung after timeout → reboot the node

4. HyperPod lifecycle scripts override manual changes

On SageMaker HyperPod, if a node is replaced or rebooted, the lifecycle script will re-mount with the original configuration (EFA). You must update the lifecycle script to persist the TCP fallback, or nodes will revert on restart.

---

Examples: How to Use This Article

Example 1: Subset of nodes losing mount (classic EFA failure)

Scenario: Engineer team reports 13 of 51 nodes have lost their Lustre mount. Training jobs are stalled.

Steps:

1. SSH into an affected node and run: dmesg | grep kefalnd

2. Confirm kefalnd_abort_tx errors with 0.0.0.0@efa destination

3. Check CloudWatch: FSx aggregate throughput = 8 GBps across 51 nodes = ~160 MBps/node

4. Per-node throughput is far below TCP ceiling (12.5 GBps) → TCP fallback is safe

5. On ONE affected node, execute the failover procedure above

6. Verify mount is stable for 5 minutes, run a quick I/O test

7. If successful, expand to remaining 12 affected nodes

8. Notify EFA team for root cause investigation in parallel

Example 2: Customer pushes back on TCP performance

Scenario: Customer or CTO says: "TCP can't handle our checkpoint writes. GPUs need fast burst throughput."

Steps:

1. Pull CloudWatch FSx metrics for past 7 days

2. Identify peak checkpoint window (look for periodic write spikes)

3. Calculate: peak write throughput ÷ number of nodes = per-node burst

4. Present data: "Your peak per-node burst is X GBps. TCP ceiling is 12.5 GBps. You have Y GBps of headroom."

5. Propose single-node trial: "Let's test on one node for 10 minutes. If checkpoint fails, we revert immediately."

6. If customer still declines → document the refusal and escalate to capacity provisioning as the alternative path