PNFS Block Server Setup Instructions

From Linux NFS

(Difference between revisions)
Jump to: navigation, search
(blkmapd)
 
(24 intermediate revisions not shown)
Line 1: Line 1:
How to guide to setup the pNFS Block Layout server based on sPNFS
How to guide to setup the pNFS Block Layout server based on sPNFS
-
This page describes about  compiling and setting up the  pNFS Block Layout Server. This is based on the Rick McNeal's how to guide sent to me.  
+
This page describes the setup of the  pNFS Block Layout Server. This is based on the Rick McNeal's how to guide. Please note that Fedora 11 was used to setup the server, some of the content you see might be specific to Fedora ( for e.g yum).  
-
 
+
 +
Note that this is an early development prototype, and recently has not been actively maintained; thus this is recommended for developers only.
==Building the code==
==Building the code==
Line 12: Line 12:
'''1) Building the kernel source'''
'''1) Building the kernel source'''
-
  -  Obtain the code from Linux pNFS git. pNFS Block Layout server is currently a part of the pNFS git.
+
Obtain the code from Linux pNFS git. pNFS Block Layout server is currently a part of the pNFS git.
-
     git clone git://linux-nfs.org/~bhalevy/linux-pnfs.git
+
     '''git clone git://linux-nfs.org/~bhalevy/linux-pnfs.git'''
-
  -  CONFIG_SPNFS_BLOCK should be enabled before the compilation of the code. 
+
-
  -  This page doesn't discuss anything about kernel compilation.
+
-
 
+
-
'''2) Building the usespace daemon'''
+
-
There is a userspace daemon and  this is required to be started before client access the block network.
+
Use the pnfs-all-latest branch and configure the kernel as follows:
-
Source code an be obtained from the following git.
+
-
'''git clone git://git.linux-nfs.org/projects/rmcneal/ctl.git'''
+
CONFIG_NFSD=m
 +
CONFIG_NFSD_V4=y
 +
CONFIG_PNFSD=y
 +
# CONFIG_PNFSD_LOCAL_EXPORT is not set
 +
CONFIG_PNFSD_BLOCK=y
-
To compile this code , parted, parted-devel and libevent packages should be installed on the machine.
+
Note: CONFIG_PNFSD_LOCAL_EXPORT must be disabled for pnfsd-block since
-
    yum insall parted
+
it currently takes precedence over it in run-time.
-
    yum insall parted-devel
+
-
    yum insall libevent
+
-
    yum insall libevent-devel
+
-
You might see couple or compilattion errors. ''' Please note that below mentioned is just a temporary work around. '''
+
This page doesn't discuss anything about kernel compilation.
 +
 
 +
'''2) Building the nfsutils and utils/blkmapd'''
-
In file included from ctl.c:12:
+
The nfs-utils-1.2.5-3.fc16.x86_64 Fedora RPM package already includes the blkmapd service
-
/usr/include/parted/device.h:140: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
+
-
In file included from /usr/include/asm/types.h:4,
+
-
from efi.h:26,
+
-
from ctl.c:16:
+
-
<ROOT_OF_SRC>/include/asm-generic/int-ll64.h:11:29: error: asm/bitsperlong.h: No such file or directory
+
-
make: *** [ctl.o] Error 1
+
 +
Here the instructions for building the utilities for older distributions:
-
1) To get past the first error , the work around is I have updated the  "/usr/include/parted/device.h" file , I have add an ifdef
+
Obtain the "nfs-utils" source code.
-
like the following.  
+
    '''git clone git://linux-nfs.org/~bhalevy/pnfs-nfs-utils.git'''
-
Line 140 in device.h.
+
Run the autogen.sh to generate the "configure" file. If you are trying to build the code first time several
 +
packages are required. I have either installed or updated the following packages.
-
#ifdef notdef
+
A one-liner command to install all prerequisites on Fedora 15:
-
extern PedConstraint* ped_device_get_constraint (PedDevice* dev);
+
yum install autoconf automake libtool libtirpc{,-devel} tcp_wrappers{,-devel} libevent{,-devel} \
-
#endif
+
            libnfsidmap{,-devel} openldap-devel libgssglue{,-devel} krb5-devel libblkid{,-devel} \
-
+
            device-mapper-devel libcap{,-devel}
-
2) I have created a symbolic link to  <ROOT_OF_SRC>/include/asm-generic/bitsperlong.h  in <ROOT_OF_SRC>/include/asm-generic.
+
 +
Or run selectively:
 +
yum install autoconf automake
 +
yum install libtool
 +
yum install libtirpc-devel
 +
yum install tcp_wrappers-devel
 +
yum install libevent-devel
 +
yum install libnfsidmap-devel
 +
yum install nfs-utils-lib-devel (deprecated in Fedora 15)
 +
yum install openldap-devel
 +
yum install libgssglue-devel
 +
# Fedora 12 and later require also:
 +
yum install libblkid-devel
 +
yum install device-mapper-devel
 +
# Fedora 13 and later require also:
 +
yum install krb5-devel
-
'''3) Building the nfsutils'''
+
== blkmapd ==
-
- Obtain the "nfs-utils" source code.
+
The blkmapd daemon should be run on the pnfs client to map the block devices
-
git clone  git://linux-nfs.org/~bhalevy/pnfs-nfs-utils
+
according the pnfs device information.
 +
See utils/blkmapd/etc/blkmapd.conf and utils/blkmapd/etc/initd/initd.redhat for
 +
more information about its setup.
-
- Run the autogen.sh to generate the "configure" file. If you are trying to build the code first time several
+
For more instructions on setting up the client, see [[Fedora pNFS Client Setup]].
-
pacakages are required. I have either installed or updated the following packages.
+
-
 
+
-
yum update libtirpc
+
-
yum install libtirpc-devel
+
-
yum install tcp_wrappers-devel
+
-
yum insall libevent
+
-
yum insall libevent-devel
+
-
yum insall libnfsidmap
+
-
yum insall libnfsidmap-devel
+
-
yum insall nfs-utils-lib
+
-
yum insall nfs-utils-lib-devel
+
-
yum insall libgssglue
+
-
yum insall libgssglue-devel
+
== Exporting the filesystem ==  
== Exporting the filesystem ==  
-
- For the block access to work properly  the disks must have a signature.  
+
For the block access to work properly  the disks must have a signature. Partitioned the disks using "parted". Disks partitioned with "fdisk" doesn't have the signatures.
-
- Partetioned the disks using "parted". Disks partetioned with "fdisk" doesn't have the signatures.
+
 
-
- I have followed the below mentioned steps.
+
I have followed the below mentioned steps.
# parted /dev/sdb
# parted /dev/sdb
-
(parted) mklabel gpt
+
(parted) mklabel gpt
-
(parted) mkpart 1 <Provide start and end of the partetions>
+
(parted) mkpart 1 <Provide start and end of the partetions>
-
(parted) print  
+
(parted) print
-
(parted) print
+
Model: VMware Virtual disk (scsi)
 +
Disk /dev/sdb: 53.7GB
 +
Sector size (logical/physical): 512B/512B
 +
Partition Table: gpt
 +
Number  Start  End    Size    File system  Name  Flags
 +
1      17.4kB  53.7GB  53.7GB  ext3        1    msftres
-
Model: VMware Virtual disk (scsi)
+
I have tested with ext4 filesystem, create  ext4 filesystem with 4K block size.
-
Disk /dev/sdb: 53.7GB
+
-
Sector size (logical/physical): 512B/512B
+
-
Partition Table: gpt
+
-
Number  Start  End    Size    File system  Name  Flags
 
-
1      17.4kB  53.7GB  53.7GB  ext3        1    msftres
 
-
- I have tested with ext4 , create  ext4 filesystem with 4K block size.
+
  # mkfs.ext4 -b 4096 /dev/sdb1  
-
 
+
-
# mkfs.ext4 -b 4096 /dev/sdb1  
+
== Setting up the BLOCK storage / SAN ==  
== Setting up the BLOCK storage / SAN ==  
 +
I haven't setup the block storage and metadata server on the same machine. You may setup them on two different machines , but client and metadata sever should see the same disks.
-
- I have used ISCSI to setup the block storage , "scsi-target-utils" is required to setup the iscsi target.
+
I have used iSCSI to setup the block storage , "scsi-target-utils" is required to setup the iscsi target.
-
- One key thing is when adding a LUN to the target , don't add the disk partettion (/dev/sdb1) , instead add the entire disk(/dev/sdb).
+
One key thing is when adding a LUN to the target , don't add the disk partition (/dev/sdb1), instead add the entire disk(/dev/sdb).
-
  The disk signatures are not visible when if you add the disk partetion to the target.
+
 
 +
The disk signatures are not visible when if you add the disk partetion to the target.
== Export Options ==
== Export Options ==
/mnt  *(rw,sync,fsid=0,insecure,no_subtree_check,no_root_squash,pnfs)
/mnt  *(rw,sync,fsid=0,insecure,no_subtree_check,no_root_squash,pnfs)
 +
 +
== ctl ==
 +
 +
The "ctl" daemon should be run on the pnfs server to map the block devices
 +
according the pnfs device information.
 +
 +
Obtain the "ctl" source code.
 +
    '''git clone git://linux-nfs.org/~bhalevy/pnfsd-block-ctl.git'''
 +
 +
Simply run "make" to build the daemon.
 +
 +
Run the daemon on the server using the following command:
 +
# ctl -u
== How to Start the server ==  
== How to Start the server ==  
-
- I have used a scrpit to start the server. Script is attached.  
+
I have used the following script to start the server
 +
 
 +
#/bin/bash
 +
# UMOUNT /mnt
 +
umount /mnt
 +
#start the service
 +
service tgtd restart
 +
sleep 8
 +
# Create iSCSI target
 +
tgtadm --lld iscsi --op new --mode target --tid 1 -T iqn.1992-05.com.emc:openblock
 +
# Expose LUN as iSCSI target
 +
tgtadm --lld iscsi --mode logicalunit --op new --tid 1 --lun 1 --backing-store /dev/sdb
 +
# Allow acces of all initiator
 +
tgtadm --lld iscsi --mode target --op bind --tid 1 --initiator-address ALL
 +
# show all the details
 +
tgtadm --lld iscsi --op show --mode target
 +
# mount the partetion
 +
mount /dev/sdb1 /mnt
 +
sleep 3
 +
# start the nfs server
 +
service nfs restart
 +
sleep 3
 +
# start the deamon
 +
cd <CTL_SRC>/ctl/
 +
./ctl -u &
== Mount from the client ==  
== Mount from the client ==  
-
  - mount -t nfs4 -o minorversion=1 SN:/ /mnt/ob
+
# modprobe blocklayoutdriver
 +
# mount -t nfs4 -o minorversion=1 SN:/ /mnt/ob
== How to verify ==  
== How to verify ==  
Line 123: Line 158:
   - tcpdump/wireshark  is the best way to see what is happening.
   - tcpdump/wireshark  is the best way to see what is happening.
   - The other way is after mounting the export,on the client check /proc/self/mountstats.
   - The other way is after mounting the export,on the client check /proc/self/mountstats.
 +
  - nfsstat also provides operations counters
 +
  - echo 32767 > /proc/sys/sunrpc/nfs_debug (and nfsd_debug, correspondingly) provides debug printouts in /var/log/messages

Latest revision as of 00:15, 16 March 2012

How to guide to setup the pNFS Block Layout server based on sPNFS

This page describes the setup of the pNFS Block Layout Server. This is based on the Rick McNeal's how to guide. Please note that Fedora 11 was used to setup the server, some of the content you see might be specific to Fedora ( for e.g yum).

Note that this is an early development prototype, and recently has not been actively maintained; thus this is recommended for developers only.

Contents

Building the code



1) Building the kernel source

Obtain the code from Linux pNFS git. pNFS Block Layout server is currently a part of the pNFS git.

    git clone git://linux-nfs.org/~bhalevy/linux-pnfs.git

Use the pnfs-all-latest branch and configure the kernel as follows:

CONFIG_NFSD=m
CONFIG_NFSD_V4=y
CONFIG_PNFSD=y
# CONFIG_PNFSD_LOCAL_EXPORT is not set
CONFIG_PNFSD_BLOCK=y

Note: CONFIG_PNFSD_LOCAL_EXPORT must be disabled for pnfsd-block since it currently takes precedence over it in run-time.

This page doesn't discuss anything about kernel compilation.

2) Building the nfsutils and utils/blkmapd

The nfs-utils-1.2.5-3.fc16.x86_64 Fedora RPM package already includes the blkmapd service

Here the instructions for building the utilities for older distributions:

Obtain the "nfs-utils" source code.

    git clone git://linux-nfs.org/~bhalevy/pnfs-nfs-utils.git

Run the autogen.sh to generate the "configure" file. If you are trying to build the code first time several packages are required. I have either installed or updated the following packages.

A one-liner command to install all prerequisites on Fedora 15:

yum install autoconf automake libtool libtirpc{,-devel} tcp_wrappers{,-devel} libevent{,-devel} \
            libnfsidmap{,-devel} openldap-devel libgssglue{,-devel} krb5-devel libblkid{,-devel} \
            device-mapper-devel libcap{,-devel}

Or run selectively:

yum install autoconf automake
yum install libtool
yum install libtirpc-devel
yum install tcp_wrappers-devel
yum install libevent-devel
yum install libnfsidmap-devel
yum install nfs-utils-lib-devel (deprecated in Fedora 15)
yum install openldap-devel
yum install libgssglue-devel
# Fedora 12 and later require also:
yum install libblkid-devel
yum install device-mapper-devel
# Fedora 13 and later require also:
yum install krb5-devel

blkmapd

The blkmapd daemon should be run on the pnfs client to map the block devices according the pnfs device information. See utils/blkmapd/etc/blkmapd.conf and utils/blkmapd/etc/initd/initd.redhat for more information about its setup.

For more instructions on setting up the client, see Fedora pNFS Client Setup.

Exporting the filesystem

For the block access to work properly the disks must have a signature. Partitioned the disks using "parted". Disks partitioned with "fdisk" doesn't have the signatures.

I have followed the below mentioned steps.

  1. parted /dev/sdb
(parted) mklabel gpt
(parted) mkpart 1 <Provide start and end of the partetions>
(parted) print
Model: VMware Virtual disk (scsi)
Disk /dev/sdb: 53.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Number  Start   End     Size    File system  Name  Flags
1      17.4kB  53.7GB  53.7GB  ext3         1     msftres

I have tested with ext4 filesystem, create ext4 filesystem with 4K block size.


 # mkfs.ext4 -b 4096 /dev/sdb1 


Setting up the BLOCK storage / SAN

I haven't setup the block storage and metadata server on the same machine. You may setup them on two different machines , but client and metadata sever should see the same disks.

I have used iSCSI to setup the block storage , "scsi-target-utils" is required to setup the iscsi target. One key thing is when adding a LUN to the target , don't add the disk partition (/dev/sdb1), instead add the entire disk(/dev/sdb).

The disk signatures are not visible when if you add the disk partetion to the target.

Export Options

/mnt *(rw,sync,fsid=0,insecure,no_subtree_check,no_root_squash,pnfs)

ctl

The "ctl" daemon should be run on the pnfs server to map the block devices according the pnfs device information.

Obtain the "ctl" source code.

    git clone git://linux-nfs.org/~bhalevy/pnfsd-block-ctl.git

Simply run "make" to build the daemon.

Run the daemon on the server using the following command:

# ctl -u

How to Start the server

I have used the following script to start the server

#/bin/bash 
# UMOUNT /mnt
umount /mnt
#start the service
service tgtd restart
sleep 8
# Create iSCSI target
tgtadm --lld iscsi --op new --mode target --tid 1 -T iqn.1992-05.com.emc:openblock
# Expose LUN as iSCSI target
tgtadm --lld iscsi --mode logicalunit --op new --tid 1 --lun 1 --backing-store /dev/sdb
# Allow acces of all initiator
tgtadm --lld iscsi --mode target --op bind --tid 1 --initiator-address ALL
# show all the details
tgtadm --lld iscsi --op show --mode target
# mount the partetion
mount /dev/sdb1 /mnt
sleep 3
# start the nfs server
service nfs restart
sleep 3
# start the deamon
cd <CTL_SRC>/ctl/
./ctl -u &

Mount from the client

# modprobe blocklayoutdriver 
# mount -t nfs4 -o minorversion=1 SN:/ /mnt/ob

How to verify

 - tcpdump/wireshark  is the best way to see what is happening.
 - The other way is after mounting the export,on the client check /proc/self/mountstats.
 - nfsstat also provides operations counters
 - echo 32767 > /proc/sys/sunrpc/nfs_debug (and nfsd_debug, correspondingly) provides debug printouts in /var/log/messages
Personal tools