NFS over SoftRoCE setup
From Linux NFS
Revision as of 03:49, 11 August 2021 by Jianhong Yin (Talk | contribs)
Soft ROCE is a software implementation of RoCE that allows RDMA to be used on any Ethernet adapter. It is available in Linux kernels after 4.8. The following instructions are based on Arch Linux
Contents |
Client and Server Common Setup
- Check for the
rdma_rxe
kernel module. If you don't have it, then enable the following Kconfig options and rebuild your kernel
CONFIG_INFINIBAND CONFIG_INFINIBAND_RDMAVT CONFIG_RDMA_RXE
- Install the
iproute2
package and use therdma link add
command to load the module and start an RDMA interface. Note that this does not survive reboots. - Comment:
rxe_cfg
has been deprecated from 01/2020, userdma
command instead.
[foo@linux-bar ~]$ sudo rdma link add rxe0 type rxe netdev eth0 [foo@linux-bar ~]$ sudo rdma link link rxe0/1 state ACTIVE physical_state LINK_UP netdev eth0
Ping Test
- Start an
rping
server on one machine
[foo@linux-bar ~]$ sudo rping -s -v -C 3 server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst server DISCONNECT EVENT... wait for RDMA_READ_ADV state 10
- Now check that the connection works from another machine
[foo@linux-bor ~]$ sudo rping -c -a 192.168.122.40 -v -C 3 ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst client DISCONNECT EVENT...
NFS Setup
Server
- Install the
nfs-utils-rdma-server
package and enable thenfs-utils-rdma-server.service
service through systemd
Client
- Install the
nfs-utils-rdma-client
package and enable thenfs-utils-rdma-client.service
service through systemd - Mount the server to check that it worked
anna@client ~ % sudo mount -o rdma,port=20049,vers=4.1 192.168.100.215:/export anna@client ~ % mount | grep proto=rdma 192.168.100.215:/home/anna/tests/tests/anna/cthon on /nfs type nfs4 (rw,relatime,vers=4.1,rsize=524288,wsize=524288,proto=rdma,port=20049)