NFS over SoftRoCE setup

From Linux NFS

(Difference between revisions)
Jump to: navigation, search
(Client and Server Common Setup)
 
(3 intermediate revisions not shown)
Line 1: Line 1:
-
Soft ROCE is a software implementation of [https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet RoCE] that allows RDMA to be used on any Ethernet adapter.  It is available in Linux kernels after 4.8.  The following instructions are based on [https://www.archlinux.org/ Arch Linux]
+
Soft ROCE is a software implementation of [https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet RoCE] that allows RDMA to be used on any Ethernet adapter.  It is available in Linux kernels after 4.8.  The following instructions are suitable for most of the latest linux distributions.
= Client and Server Common Setup =
= Client and Server Common Setup =
Line 29: Line 29:
= NFS Setup =
= NFS Setup =
== Server ==
== Server ==
-
* Install the <code>nfs-utils-rdma-server</code> package and enable the <code>nfs-utils-rdma-server.service</code> service through systemd
+
* Install the <code>nfs-utils</code> package and enable rdma from <code>/etc/nfs.conf</code> and restart service nfs-server
 +
[foo@linux-bar]$ sudo sed -i '/rdma/{s/^#//; s/rdma=n/rdma=y/}' /etc/nfs.conf
 +
[foo@linux-bar]$ grep -v ^# /etc/nfs.conf
 +
[general]
 +
[exportfs]
 +
[gssd]
 +
  use-gss-proxy=1
 +
[lockd]
 +
[mountd]
 +
[nfsdcld]
 +
[nfsdcltrack]
 +
[nfsd]
 +
  rdma=y
 +
  rdma-port=20049
 +
[statd]
 +
[sm-notify]
 +
[foo@linux-bar]$ sudo mkdir -p /expdir
 +
[foo@linux-bar]$ sudo bash -c 'echo "/expdir *(rw,no_root_squash)" >/etc/exports'
 +
[foo@linux-bar]$ sudo systemctl restart nfs-server
== Client ==
== Client ==
-
* Install the <code>nfs-utils-rdma-client</code> package and enable the <code>nfs-utils-rdma-client.service</code> service through systemd
+
* Install the <code>nfs-utils</code> package and do nfs mount
* Mount the server to check that it worked
* Mount the server to check that it worked
-
  anna@client ~ % sudo mount -o rdma,port=20049,vers=4.1 192.168.100.215:/export
+
  [foo@linux-bor]$ sudo mount -o rdma,port=20049,vers=4.2 192.168.122.40:/expdir /mnt/nfsmp
-
  anna@client ~ % mount | grep proto=rdma
+
  [foo@linux-bor]$ mount | grep proto=rdma
-
  192.168.100.215:/home/anna/tests/tests/anna/cthon on /nfs type nfs4 (rw,relatime,vers=4.1,rsize=524288,wsize=524288,proto=rdma,port=20049)
+
192.168.122.40:/expdir on /mnt/nfsmp type nfs4 (rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=192.168.122.161,local_lock=none,addr=192.168.122.40)
= See Also =
= See Also =
* [https://community.mellanox.com/docs/DOC-2184 Mellanox's softRoCE setup guide]
* [https://community.mellanox.com/docs/DOC-2184 Mellanox's softRoCE setup guide]
-
* [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/networking_guide/sec-configuring_soft-_roce Red Hat's softRoCE setup guide]
+
* [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_infiniband_and_rdma_networks/configuring-roce_configuring-and-managing-networking#configuring-soft-roce_configuring-roce Red Hat's softRoCE setup guide]
* [https://github.com/linux-rdma/rdma-core/blob/master/Documentation/rxe.md Rdma-core's documentation]
* [https://github.com/linux-rdma/rdma-core/blob/master/Documentation/rxe.md Rdma-core's documentation]
 +
* [https://man7.org/linux/man-pages/man8/rdma.8.html The rdma man page]
* [https://www.systutorials.com/docs/linux/man/8-rxe_cfg/ The rxe_cfg man page]
* [https://www.systutorials.com/docs/linux/man/8-rxe_cfg/ The rxe_cfg man page]
 +
* [https://manpages.debian.org/testing/rdma-core/rxe_cfg.8.en.html rxe_cfg deprecated]

Latest revision as of 05:27, 11 August 2021

Soft ROCE is a software implementation of RoCE that allows RDMA to be used on any Ethernet adapter. It is available in Linux kernels after 4.8. The following instructions are suitable for most of the latest linux distributions.

Contents

Client and Server Common Setup

  • Check for the rdma_rxe kernel module. If you don't have it, then enable the following Kconfig options and rebuild your kernel
CONFIG_INFINIBAND
CONFIG_INFINIBAND_RDMAVT
CONFIG_RDMA_RXE
  • Install the iproute2 package and use the rdma link add command to load the module and start an RDMA interface. Note that this does not survive reboots.
  • Comment: rxe_cfg has been deprecated from 01/2020, use rdma command instead.
[foo@linux-bar ~]$ sudo rdma link add rxe0 type rxe netdev eth0
[foo@linux-bar ~]$ sudo rdma link
link rxe0/1 state ACTIVE physical_state LINK_UP netdev eth0

Ping Test

  • Start an rping server on one machine
[foo@linux-bar ~]$ sudo rping -s -v -C 3
server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
server DISCONNECT EVENT...
wait for RDMA_READ_ADV state 10
  • Now check that the connection works from another machine
[foo@linux-bor ~]$ sudo rping -c -a 192.168.122.40 -v -C 3
ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
client DISCONNECT EVENT...

NFS Setup

Server

  • Install the nfs-utils package and enable rdma from /etc/nfs.conf and restart service nfs-server
[foo@linux-bar]$ sudo sed -i '/rdma/{s/^#//; s/rdma=n/rdma=y/}' /etc/nfs.conf
[foo@linux-bar]$ grep -v ^# /etc/nfs.conf
[general]
[exportfs]
[gssd]
 use-gss-proxy=1
[lockd]
[mountd]
[nfsdcld]
[nfsdcltrack]
[nfsd]
 rdma=y
 rdma-port=20049
[statd]
[sm-notify]
[foo@linux-bar]$ sudo mkdir -p /expdir
[foo@linux-bar]$ sudo bash -c 'echo "/expdir *(rw,no_root_squash)" >/etc/exports'
[foo@linux-bar]$ sudo systemctl restart nfs-server

Client

  • Install the nfs-utils package and do nfs mount
  • Mount the server to check that it worked
[foo@linux-bor]$ sudo mount -o rdma,port=20049,vers=4.2 192.168.122.40:/expdir /mnt/nfsmp
[foo@linux-bor]$ mount | grep proto=rdma
192.168.122.40:/expdir on /mnt/nfsmp type nfs4 (rw,relatime,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,proto=rdma,port=20049,timeo=600,retrans=2,sec=sys,clientaddr=192.168.122.161,local_lock=none,addr=192.168.122.40)

See Also

Personal tools