NFS over SoftRoCE setup

From Linux NFS

Revision as of 03:49, 11 August 2021 by Jianhong Yin (Talk | contribs)
Jump to: navigation, search

Soft ROCE is a software implementation of RoCE that allows RDMA to be used on any Ethernet adapter. It is available in Linux kernels after 4.8. The following instructions are based on Arch Linux

Contents

Client and Server Common Setup

  • Check for the rdma_rxe kernel module. If you don't have it, then enable the following Kconfig options and rebuild your kernel
CONFIG_INFINIBAND
CONFIG_INFINIBAND_RDMAVT
CONFIG_RDMA_RXE
  • Install the iproute2 package and use the rdma link add command to load the module and start an RDMA interface. Note that this does not survive reboots.
  • Comment: rxe_cfg has been deprecated from 01/2020, use rdma command instead.
[foo@linux-bar ~]$ sudo rdma link add rxe0 type rxe netdev eth0
[foo@linux-bar ~]$ sudo rdma link
link rxe0/1 state ACTIVE physical_state LINK_UP netdev eth0

Ping Test

  • Start an rping server on one machine
[foo@linux-bar ~]$ sudo rping -s -v -C 3
server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
server DISCONNECT EVENT...
wait for RDMA_READ_ADV state 10
  • Now check that the connection works from another machine
[foo@linux-bor ~]$ sudo rping -c -a 192.168.122.40 -v -C 3
ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
client DISCONNECT EVENT...

NFS Setup

Server

  • Install the nfs-utils-rdma-server package and enable the nfs-utils-rdma-server.service service through systemd

Client

  • Install the nfs-utils-rdma-client package and enable the nfs-utils-rdma-client.service service through systemd
  • Mount the server to check that it worked
anna@client ~ % sudo mount -o rdma,port=20049,vers=4.1 192.168.100.215:/export
anna@client ~ % mount | grep proto=rdma
  192.168.100.215:/home/anna/tests/tests/anna/cthon on /nfs type nfs4 (rw,relatime,vers=4.1,rsize=524288,wsize=524288,proto=rdma,port=20049)

See Also

Personal tools