NFS over SoftRoCE setup

From Linux NFS

You do not have permission to edit this page, for the following reason:

You can view and copy the source of this page:

Soft ROCE is a software implementation of [https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet RoCE] that allows RDMA to be used on any Ethernet adapter. It is available in Linux kernels after 4.8. The following instructions are suitable for most of the latest linux distributions. = Client and Server Common Setup = * Check for the <code>rdma_rxe</code> kernel module. If you don't have it, then enable the following Kconfig options and rebuild your kernel CONFIG_INFINIBAND CONFIG_INFINIBAND_RDMAVT CONFIG_RDMA_RXE * Install the <code>iproute2</code> package and use the <code>rdma link add</code> command to load the module and start an RDMA interface. Note that this does not survive reboots. * Comment: <code>rxe_cfg</code> has been deprecated from 01/2020, use <code>rdma</code> command instead. [foo@linux-bar ~]$ sudo rdma link add rxe0 type rxe netdev eth0 [foo@linux-bar ~]$ sudo rdma link link rxe0/1 state ACTIVE physical_state LINK_UP netdev eth0 == Ping Test == * Start an <code>rping</code> server on one machine [foo@linux-bar ~]$ sudo rping -s -v -C 3 server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst server DISCONNECT EVENT... wait for RDMA_READ_ADV state 10 * Now check that the connection works from another machine [foo@linux-bor ~]$ sudo rping -c -a 192.168.122.40 -v -C 3 ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst client DISCONNECT EVENT... = NFS Setup = == Server == * Install the <code>nfs-utils</code> package and enable rdma from <code>/etc/nfs.conf</code> and restart service nfs-server [foo@linux-bar]$ sudo sed -i '/rdma/{s/^#//; s/rdma=n/rdma=y/}' /etc/nfs.conf [foo@linux-bar]$ grep -v ^# /etc/nfs.conf [general] [exportfs] [gssd] use-gss-proxy=1 [lockd] [mountd] [nfsdcld] [nfsdcltrack] [nfsd] rdma=y rdma-port=20049 [statd] [sm-notify] [foo@linux-bar]$ sudo mkdir -p /expdir [foo@linux-bar]$ sudo bash -c 'echo "/expdir *(rw,no_root_squash)" >/etc/exports' [foo@linux-bar]$ sudo systemctl restart nfs-server == Client == * Install the <code>nfs-utils</code> package and do nfs mount * Mount the server to check that it worked [foo@linux-bor]$ sudo mount -o rdma,port=20049,vers=4.2 192.168.122.40:/expdir /mnt/nfsmp [foo@linux-bor]$ mount | grep proto=rdma 192.168.122.40:/expdir on /mnt/nfsmp type nfs4 (rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=192.168.122.161,local_lock=none,addr=192.168.122.40) = See Also = * [https://community.mellanox.com/docs/DOC-2184 Mellanox's softRoCE setup guide] * [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_infiniband_and_rdma_networks/configuring-roce_configuring-and-managing-networking#configuring-soft-roce_configuring-roce Red Hat's softRoCE setup guide] * [https://github.com/linux-rdma/rdma-core/blob/master/Documentation/rxe.md Rdma-core's documentation] * [https://man7.org/linux/man-pages/man8/rdma.8.html The rdma man page] * [https://www.systutorials.com/docs/linux/man/8-rxe_cfg/ The rxe_cfg man page] * [https://manpages.debian.org/testing/rdma-core/rxe_cfg.8.en.html rxe_cfg deprecated]

Return to NFS over SoftRoCE setup.