Summary
In the installed libfabric 2.3.1, the CXI provider's RMA op table wires the writedata
entry points to the no-op stubs:
cxip_ep_rma_ops.writedata = fi_no_rma_writedata
cxip_ep_rma_ops.injectdata = fi_no_rma_injectdata
confirmed by inspecting the library binary. There is no cxip_rma_writedata symbol at
all. Every neighbouring entry — read, readv, readmsg, write, writev, writemsg,
inject — is a real cxip_rma_* implementation.
The consequence is that a consumer requesting FI_RMA negotiates successfully, opens its
endpoints, reports itself ready, and then fails on the first actual transfer with:
NET/OFI fi_writedata failed; RC: -38, Error: Function not implemented
-38 is -ENOSYS. Failing at first transfer rather than at capability negotiation is the
substance of this report: nothing in fi_getinfo or endpoint setup indicates the operation
is missing, so the consumer has no way to select a different transport up front.
Who this breaks
aws-ofi-nccl's RDMA transport uses fi_writedata for its 4 bytes of immediate data, so RDMA
can never work on this fabric regardless of plugin version. With aws-ofi-nccl 1.20.0 the
sequence is: plugin logs Using transport protocol RDMA (user set), no ENODATA, endpoints
open — then the first transfer returns RC: -38 and the job dies.
This is a distinct failure from the FI_MR_ENDPOINT negotiation issue that stops 1.19.2
earlier. For completeness, that one is: CXI mandates FI_MR_ENDPOINT (it returns
mr_mode 0x260 = FI_MR_ENDPOINT|FI_MR_PROV_KEY|FI_MR_ALLOCATED), and 1.19.2's RDMA
get_hints omits it, yielding -FI_ENODATA / "No eligible providers were found". Adding
FI_MR_ENDPOINT alone flips it to eligible — we relaxed threading, cq_data_size=4,
FI_HMEM, FI_RMA, PROV_KEY, mr_key_size and FI_CONTEXT2 individually and none of
them mattered. That part is arguably aws-ofi-nccl's to fix, and 1.20.0 has fixed it; the
writedata stub is what remains.
How it was determined
Direct fi_getinfo bisection against the cxi provider, reproduced on both a login node and
a 4-NIC compute node, plus symbol inspection of the installed libfabric binary.
Environment
- libfabric 2.3.1,
FI_PROVIDER=cxi, HPE Slingshot, 4 NICs per node
- Only 2.3.1 is actually installed: the
1.20.1 and 1.22.0 trees under
/opt/cray/libfabric contain nothing but pkgconfig files, so we could not test whether a
newer CXI provider implements it.
What would help
Either implement cxip_rma_writedata / cxip_rma_injectdata, or — if immediate data cannot
be supported on this hardware — have the provider decline to advertise the capability during
fi_getinfo so consumers fall back at negotiation time instead of mid-transfer.
If a later CXI provider already implements this, saying so is equally useful; we cannot test
it from here.
Summary
In the installed libfabric 2.3.1, the CXI provider's RMA op table wires the
writedataentry points to the no-op stubs:
confirmed by inspecting the library binary. There is no
cxip_rma_writedatasymbol atall. Every neighbouring entry —
read,readv,readmsg,write,writev,writemsg,inject— is a realcxip_rma_*implementation.The consequence is that a consumer requesting
FI_RMAnegotiates successfully, opens itsendpoints, reports itself ready, and then fails on the first actual transfer with:
-38is-ENOSYS. Failing at first transfer rather than at capability negotiation is thesubstance of this report: nothing in
fi_getinfoor endpoint setup indicates the operationis missing, so the consumer has no way to select a different transport up front.
Who this breaks
aws-ofi-nccl's RDMA transport uses
fi_writedatafor its 4 bytes of immediate data, so RDMAcan never work on this fabric regardless of plugin version. With aws-ofi-nccl 1.20.0 the
sequence is: plugin logs
Using transport protocol RDMA (user set), noENODATA, endpointsopen — then the first transfer returns
RC: -38and the job dies.This is a distinct failure from the
FI_MR_ENDPOINTnegotiation issue that stops 1.19.2earlier. For completeness, that one is: CXI mandates
FI_MR_ENDPOINT(it returnsmr_mode 0x260=FI_MR_ENDPOINT|FI_MR_PROV_KEY|FI_MR_ALLOCATED), and 1.19.2's RDMAget_hintsomits it, yielding-FI_ENODATA/ "No eligible providers were found". AddingFI_MR_ENDPOINTalone flips it to eligible — we relaxed threading,cq_data_size=4,FI_HMEM,FI_RMA,PROV_KEY,mr_key_sizeandFI_CONTEXT2individually and none ofthem mattered. That part is arguably aws-ofi-nccl's to fix, and 1.20.0 has fixed it; the
writedatastub is what remains.How it was determined
Direct
fi_getinfobisection against the cxi provider, reproduced on both a login node anda 4-NIC compute node, plus symbol inspection of the installed libfabric binary.
Environment
FI_PROVIDER=cxi, HPE Slingshot, 4 NICs per node1.20.1and1.22.0trees under/opt/cray/libfabriccontain nothing but pkgconfig files, so we could not test whether anewer CXI provider implements it.
What would help
Either implement
cxip_rma_writedata/cxip_rma_injectdata, or — if immediate data cannotbe supported on this hardware — have the provider decline to advertise the capability during
fi_getinfoso consumers fall back at negotiation time instead of mid-transfer.If a later CXI provider already implements this, saying so is equally useful; we cannot test
it from here.