scst

Differences between revisions 8 and 9
Revision 8 as of 2010-05-09 13:44:22
Size: 1573
Editor: brmn-4db70f7e
Comment:
Revision 9 as of 2010-07-23 18:16:34
Size: 7314
Editor: bas4-toronto21-1176312462
Comment:
Deletions are marked like this. Additions are marked like this.
Line 19: Line 19:
"" lsmod |grep ib_"" will show you the avalaible Infiniband kernel modules. Load any modules you need with ""modprobe ib_<name> "" "" lsmod |grep ib_"" will show you the available Infiniband kernel modules. Load any modules you need with ""modprobe ib_<name> ""
Line 31: Line 31:

After installing the packages from the PPA above, reboot your machine and verify that your Infiniband connections are working with ibstat, ibhosts, ibswitches, and/or iblinkinfo. More information about those commands can be found here: [[http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-4.html|debian OFED HowTo]]

== Ok, my infiniband mesh is up. What now? ==

Next, you will want to set up your '''scst target'''.

||Disclaimer: I am simply documenting my experiences to get a working SCST setup on my particular hardware and according to my storage requirements. '''There are many security options which have not been covered here. This should not be considered an acceptable setup for secure environments.''' It would be great if a storage expert could further develop this guide.||

There are two methods of setting up scst targets, either manually through the kernel's /proc interface, or through an automated tool known as scstadmin. If you are interested in using the /proc interface, look here for some pointers: [[http://iscsi-scst.sourceforge.net/iscsi-scst-howto.txt|iscs-scst howto]]. We will use the scstadmin tool in this guide.

My particular storage configuration was a 6TB softraid storage array running OCFS2 on Mellanox ConnectX2 Infiniband HBAs. You'll need to have some kind of parallel-write safe filesystem (OCFS2, GFS2, etc.) when using scst unless you know what you're doing. '''Ext4 and friends will not appreciate being mounted read/write on multiple machines at a time.''' There's sufficient documentation about how to setup these filesystems elsewhere, so I won't go into detail about it. For an OCFS2 filesystem (which currently provides the highest performance among shared-disk filesystems I believe), you can get documentation from the [[http://www.google.com/url?sa=t&source=web&cd=2&ved=0CBgQFjAB&url=http%3A%2F%2Foss.oracle.com%2Fprojects%2Focfs2%2Fdist%2Fdocumentation%2Fv1.4%2Focfs2-1_4-usersguide.pdf&ei=0dhJTNDuEpjrnQfT4fHiDQ&usg=AFQjCNGmABKQ-1epABHe1cj9ul7L57Fplw&sig2=c-J6I2jWAJUyh3j_B2PpTA|Oracle OCFS2 v1.4 User's Guide]]

The next few steps were taken from the scst howto page on sourceforge. They will set up your target name.
----
''2. Set up /etc/iscsi/ininitiatorname.iscsi

The most convenient way to set up this file is to install the open-iscsi
package provided by your Linux distro first and then to run the shell commands
shown below. Verify the contents of the generated file.

  /etc/init.d/open-iscsi stop
  { echo "InitiatorName=$(if [ -e /usr/sbin/iscsi-iname ]; then /usr/sbin/iscsi-iname; else /sbin/iscsi-iname; fi)";
                          echo "InitiatorAlias=$(hostname)"; } >/etc/iscsi/initiatorname.iscsi
  /etc/init.d/open-iscsi start
  cat /etc/iscsi/initiatorname.iscsi

3. Set up /etc/iscsi-scstd.conf

You can do this by e.g. running the following shell commands:

  echo "Target $(sed -n 's/InitiatorName=//p' /etc/iscsi/initiatorname.iscsi):storage" >/etc/iscsi-scstd.conf
  cat /etc/iscsi-scstd.conf
''
----
We will now set up your target disks or files.

Clear your config files first with
   ''scstadmin -clearConfig /etc/scst.conf''

Next, if you are connecting to a virtual disk, such as a linux softraid device or file, add the device to your kernel's scsi handler using
   ''scstadmin -adddev <vdisk01> -path </dev/md0> -handler vdisk -options <BLOCKIO>''
Replace the options in angled brackets with information relevant to your system. The BLOCKIO option provides block level access to any srp initiators, bypassing the target's kernel pagecache. This is suitable for any softraid devices etc. Note that scstadmin --help erroneously calls this the BLOCK_IO option.

Scst defines LUNs are a part of security groups. The command to do this in scstadmin is
   ''scstadmin -assigndev <vdisk01> -group Default -lun <0>''
Again, you can replace the options in angled brackets as necessary.

Lastly, commit your changes to disk with
   ''scstadmin -WriteConfig /etc/scst.conf''

By default, the scst service will start on boot and be detectable by SRPs on your network. However, you need to use lsmod to verify that the '''ib_srp and ib_srpt''' kernel modules are loaded. You be unable to establish any srp connections if they are not. Put the modules into /etc/modules.conf if needed.

You can use /etc/initiators.allow and /etc/initiators.deny to add a bit more security to your setup.

That's it for your target configuration. We'll be moving on to '''initiator configuration''' now.

On your initiator, run
   ''ibsrpdm -c''
to see whether your initiator is able to establish an srp connection to your target. If you get output like
   ''id_ext=50001ff10005052a,ioc_guid=50001ff10005052a,dgid=fe8000000000000050001ff10005052a,pkey=ffff,service_id=2a050500f11f0050''
then you're golden. If not, double check your infiniband connection.

Next, edit your /etc/srp_daemon.conf file and add an entry with your id_ext and ioc_guid. For example, from the ipsrpdm output above, you would add the line
   ''a id_ext=50001ff10005052a,ioc_guid=50001ff10005052a''
to your srp_daemon.conf file. Note the 'a' is for allow. For security reasons, the last line in the file should be 'd' to deny all other connections.

You will also need to edit /etc/defaults/srptools to allow connections on your infiniband adapter. You will need to have the syntax
   PORTS="<your_ib_card>:<your_ib_card_port>".
My mellanox card's entry was
   PORTS="mlx4_0:1"

Lastly, restart the srptools service in /etc/init.d/srptools and you should see entries in dmesg about a new block device being discovered. fdisk -l should also show the new block device. Simply add it to your fstab and you're set!

SCST - Infiniband, SRP Target and Ubuntu Lucid

This Wiki will describe how to setup an Infiniband based SRP target on Ubuntu Lucid (10.04).

A Word of Warning

Configuring, designing and implementing an Infiniband based setup is not for the faint-hearted. It will take days or even weeks to understand its quirks, bottlenecks, complexities and you'll even end up doubting yourself why you had chosen Infiniband over 10GB ethernet.

Before we jump ahead

First of all: Check your BIOS settings! You may end up with really bad PCIe performance if not configured properly. Especially server boards tend to be configured very conservatively. "Optimized Defaults" may be your friend but your mileage may vary.

Where do I start?

The kernel module for your Infiniband adapter will already be loaded (mine was ib_mtcha). Now you need to install/load the corresponding Infiniband-libs and modules:

"" apt-get install libibcm1 libibcommon1 libibmad1 libibumad1 libibverbs1 librdmacm1 rdmacm-utils ibverbs-utils ""

"" lsmod |grep ib_"" will show you the available Infiniband kernel modules. Load any modules you need with ""modprobe ib_<name> ""

Where do I get the appropriate Ubuntu packages?

Initiator: You can find srptools and opensm in this PPA:

https://launchpad.net/~ast/+archive/srp

Target: You can find patched Kernels and SCST-Admin in this PPA:

https://launchpad.net/~ast/+archive/scst

After installing the packages from the PPA above, reboot your machine and verify that your Infiniband connections are working with ibstat, ibhosts, ibswitches, and/or iblinkinfo. More information about those commands can be found here: debian OFED HowTo

Ok, my infiniband mesh is up. What now?

Next, you will want to set up your scst target.

Disclaimer: I am simply documenting my experiences to get a working SCST setup on my particular hardware and according to my storage requirements. There are many security options which have not been covered here. This should not be considered an acceptable setup for secure environments. It would be great if a storage expert could further develop this guide.

There are two methods of setting up scst targets, either manually through the kernel's /proc interface, or through an automated tool known as scstadmin. If you are interested in using the /proc interface, look here for some pointers: iscs-scst howto. We will use the scstadmin tool in this guide.

My particular storage configuration was a 6TB softraid storage array running OCFS2 on Mellanox ConnectX2 Infiniband HBAs. You'll need to have some kind of parallel-write safe filesystem (OCFS2, GFS2, etc.) when using scst unless you know what you're doing. Ext4 and friends will not appreciate being mounted read/write on multiple machines at a time. There's sufficient documentation about how to setup these filesystems elsewhere, so I won't go into detail about it. For an OCFS2 filesystem (which currently provides the highest performance among shared-disk filesystems I believe), you can get documentation from the Oracle OCFS2 v1.4 User's Guide

The next few steps were taken from the scst howto page on sourceforge. They will set up your target name.


2. Set up /etc/iscsi/ininitiatorname.iscsi

The most convenient way to set up this file is to install the open-iscsi package provided by your Linux distro first and then to run the shell commands shown below. Verify the contents of the generated file.

  • /etc/init.d/open-iscsi stop

    { echo "InitiatorName=$(if [ -e /usr/sbin/iscsi-iname ]; then /usr/sbin/iscsi-iname; else /sbin/iscsi-iname; fi)";

    /etc/init.d/open-iscsi start cat /etc/iscsi/initiatorname.iscsi

3. Set up /etc/iscsi-scstd.conf

You can do this by e.g. running the following shell commands:

  • echo "Target $(sed -n 's/InitiatorName=//p' /etc/iscsi/initiatorname.iscsi):storage" >/etc/iscsi-scstd.conf cat /etc/iscsi-scstd.conf


We will now set up your target disks or files.

Clear your config files first with

  • scstadmin -clearConfig /etc/scst.conf

Next, if you are connecting to a virtual disk, such as a linux softraid device or file, add the device to your kernel's scsi handler using

  • scstadmin -adddev <vdisk01> -path </dev/md0> -handler vdisk -options <BLOCKIO>

Replace the options in angled brackets with information relevant to your system. The BLOCKIO option provides block level access to any srp initiators, bypassing the target's kernel pagecache. This is suitable for any softraid devices etc. Note that scstadmin --help erroneously calls this the BLOCK_IO option.

Scst defines LUNs are a part of security groups. The command to do this in scstadmin is

  • scstadmin -assigndev <vdisk01> -group Default -lun <0>

Again, you can replace the options in angled brackets as necessary.

Lastly, commit your changes to disk with

By default, the scst service will start on boot and be detectable by SRPs on your network. However, you need to use lsmod to verify that the ib_srp and ib_srpt kernel modules are loaded. You be unable to establish any srp connections if they are not. Put the modules into /etc/modules.conf if needed.

You can use /etc/initiators.allow and /etc/initiators.deny to add a bit more security to your setup.

That's it for your target configuration. We'll be moving on to initiator configuration now.

On your initiator, run

  • ibsrpdm -c

to see whether your initiator is able to establish an srp connection to your target. If you get output like

  • id_ext=50001ff10005052a,ioc_guid=50001ff10005052a,dgid=fe8000000000000050001ff10005052a,pkey=ffff,service_id=2a050500f11f0050

then you're golden. If not, double check your infiniband connection.

Next, edit your /etc/srp_daemon.conf file and add an entry with your id_ext and ioc_guid. For example, from the ipsrpdm output above, you would add the line

  • a id_ext=50001ff10005052a,ioc_guid=50001ff10005052a

to your srp_daemon.conf file. Note the 'a' is for allow. For security reasons, the last line in the file should be 'd' to deny all other connections.

You will also need to edit /etc/defaults/srptools to allow connections on your infiniband adapter. You will need to have the syntax

  • PORTS="<your_ib_card>:<your_ib_card_port>".

My mellanox card's entry was

  • PORTS="mlx4_0:1"

Lastly, restart the srptools service in /etc/init.d/srptools and you should see entries in dmesg about a new block device being discovered. fdisk -l should also show the new block device. Simply add it to your fstab and you're set!


CategoryDocumentation

scst (last edited 2010-10-08 00:34:46 by eth2083)