scst

Differences between revisions 10 and 11
Revision 10 as of 2010-07-30 21:14:37
Size: 7571
Editor: dhcp-108-120
Comment:
Revision 11 as of 2010-07-30 21:29:07
Size: 8135
Editor: dhcp-108-120
Comment:
Deletions are marked like this. Additions are marked like this.
Line 36: Line 36:
Next, you will want to set up your '''scst target'''. ||Disclaimer: I am simply documenting my experiences to get a working SCST setup on my particular hardware and according to my storage requirements. '''There are many security options which have not been covered here. This should not be considered an acceptable setup for secure environments.''' It would be great if a storage expert could further develop this guide.||
Line 38: Line 38:
===scst target=== Next, you will want to set up your '''scst target''' and all of your '''SRP initiators'''.
Line 40: Line 40:
||Disclaimer: I am simply documenting my experiences to get a working SCST setup on my particular hardware and according to my storage requirements. '''There are many security options which have not been covered here. This should not be considered an acceptable setup for secure environments.''' It would be great if a storage expert could further develop this guide.|| === scst target ===

||This step is run once on your scst target.||
Line 44: Line 46:
My particular storage configuration was a 6TB softraid storage array running OCFS2 on Mellanox ConnectX2 Infiniband HBAs. You'll need to have some kind of parallel-write safe filesystem (OCFS2, GFS2, etc.) when using scst unless you know what you're doing. '''Ext4 and friends will not appreciate being mounted read/write on multiple machines at a time.''' There's sufficient documentation about how to setup these filesystems elsewhere, so I won't go into detail about it. For an OCFS2 filesystem (which currently provides the highest performance among shared-disk filesystems I believe), you can get documentation from the [[http://www.google.com/url?sa=t&source=web&cd=2&ved=0CBgQFjAB&url=http%3A%2F%2Foss.oracle.com%2Fprojects%2Focfs2%2Fdist%2Fdocumentation%2Fv1.4%2Focfs2-1_4-usersguide.pdf&ei=0dhJTNDuEpjrnQfT4fHiDQ&usg=AFQjCNGmABKQ-1epABHe1cj9ul7L57Fplw&sig2=c-J6I2jWAJUyh3j_B2PpTA|Oracle OCFS2 v1.4 User's Guide]] My particular storage configuration was a 6TB softraid storage array running OCFS2 on Mellanox ConnectX2 Infiniband HBAs. You'll need to have some kind of parallel-write safe filesystem (OCFS2, GFS2, etc.) when using scst unless you know what you're doing. '''Ext4 and friends will not appreciate being mounted read/write on multiple machines at a time.''' There's sufficient documentation about how to setup these filesystems elsewhere, so I won't go into detail about it. For an OCFS2 filesystem (which currently provides the highest performance among shared-disk filesystems I believe), you can get documentation from the [[http://www.google.com/url?sa=t&source=web&cd=2&ved=0CBgQFjAB&url=http%3A%2F%2Foss.oracle.com%2Fprojects%2Focfs2%2Fdist%2Fdocumentation%2Fv1.4%2Focfs2-1_4-usersguide.pdf&ei=0dhJTNDuEpjrnQfT4fHiDQ&usg=AFQjCNGmABKQ-1epABHe1cj9ul7L57Fplw&sig2=c-J6I2jWAJUyh3j_B2PpTA|Oracle OCFS2 v1.4 User's Guide]]. If you use OCFS2, beware of a bug currently in Lucid which brings up the o2cb cluster manager before the infiniband mesh has time to come up. A temporary workaround is to append a 'post-up /etc/init.d/o2cb start ocfs2' under the infiniband entry in /etc/network/interfaces. It should be fixed in mavrick.
Line 90: Line 92:
===SRP initiator=== === SRP initiator ===

||This step will need to repeated on each of your SRP initiators.||
Line 108: Line 112:
For some reason, the PORTS=ALL option does not work, despite being present in the config file.

SCST - Infiniband, SRP Target and Ubuntu Lucid

This Wiki will describe how to setup an Infiniband based SRP target on Ubuntu Lucid (10.04).

A Word of Warning

Configuring, designing and implementing an Infiniband based setup is not for the faint-hearted. It will take days or even weeks to understand its quirks, bottlenecks, complexities and you'll even end up doubting yourself why you had chosen Infiniband over 10GB ethernet.

Before we jump ahead

First of all: Check your BIOS settings! You may end up with really bad PCIe performance if not configured properly. Especially server boards tend to be configured very conservatively. "Optimized Defaults" may be your friend but your mileage may vary.

Where do I start?

The kernel module for your Infiniband adapter will already be loaded (mine was ib_mtcha). Now you need to install/load the corresponding Infiniband-libs and modules:

"" apt-get install libibcm1 libibcommon1 libibmad1 libibumad1 libibverbs1 librdmacm1 rdmacm-utils ibverbs-utils ""

"" lsmod |grep ib_"" will show you the available Infiniband kernel modules. Load any modules you need with ""modprobe ib_<name> ""

Where do I get the appropriate Ubuntu packages?

Initiator: You can find srptools and opensm in this PPA:

https://launchpad.net/~ast/+archive/srp

Target: You can find patched Kernels and SCST-Admin in this PPA:

https://launchpad.net/~ast/+archive/scst

After installing the packages from the PPA above, reboot your machine and verify that your Infiniband connections are working with ibstat, ibhosts, ibswitches, and/or iblinkinfo. More information about those commands can be found here: debian OFED HowTo

Ok, my infiniband mesh is up. What now?

Disclaimer: I am simply documenting my experiences to get a working SCST setup on my particular hardware and according to my storage requirements. There are many security options which have not been covered here. This should not be considered an acceptable setup for secure environments. It would be great if a storage expert could further develop this guide.

Next, you will want to set up your scst target and all of your SRP initiators.

scst target

This step is run once on your scst target.

There are two methods of setting up scst targets, either manually through the kernel's /proc interface, or through an automated tool known as scstadmin. If you are interested in using the /proc interface, look here for some pointers: iscs-scst howto. We will use the scstadmin tool in this guide.

My particular storage configuration was a 6TB softraid storage array running OCFS2 on Mellanox ConnectX2 Infiniband HBAs. You'll need to have some kind of parallel-write safe filesystem (OCFS2, GFS2, etc.) when using scst unless you know what you're doing. Ext4 and friends will not appreciate being mounted read/write on multiple machines at a time. There's sufficient documentation about how to setup these filesystems elsewhere, so I won't go into detail about it. For an OCFS2 filesystem (which currently provides the highest performance among shared-disk filesystems I believe), you can get documentation from the Oracle OCFS2 v1.4 User's Guide. If you use OCFS2, beware of a bug currently in Lucid which brings up the o2cb cluster manager before the infiniband mesh has time to come up. A temporary workaround is to append a 'post-up /etc/init.d/o2cb start ocfs2' under the infiniband entry in /etc/network/interfaces. It should be fixed in mavrick.

The next few steps were taken from the scst howto page on sourceforge. They will set up your target name.


2. Set up /etc/iscsi/ininitiatorname.iscsi

The most convenient way to set up this file is to install the open-iscsi package provided by your Linux distro first and then to run the shell commands shown below. Verify the contents of the generated file.

  • /etc/init.d/open-iscsi stop

    { echo "InitiatorName=$(if [ -e /usr/sbin/iscsi-iname ]; then /usr/sbin/iscsi-iname; else /sbin/iscsi-iname; fi)";

    /etc/init.d/open-iscsi start cat /etc/iscsi/initiatorname.iscsi

3. Set up /etc/iscsi-scstd.conf

You can do this by e.g. running the following shell commands:

  • echo "Target $(sed -n 's/InitiatorName=//p' /etc/iscsi/initiatorname.iscsi):storage" >/etc/iscsi-scstd.conf cat /etc/iscsi-scstd.conf


We will now set up your target disks or files.

Clear your config files first with

  • scstadmin -clearConfig /etc/scst.conf

Next, if you are connecting to a virtual disk, such as a linux softraid device or file, add the device to your kernel's scsi handler using

  • scstadmin -adddev <vdisk01> -path </dev/md0> -handler vdisk -options <BLOCKIO>

Replace the options in angled brackets with information relevant to your system. The BLOCKIO option provides block level access to any srp initiators, bypassing the target's kernel pagecache. This is suitable for any softraid devices etc. Note that scstadmin --help erroneously calls this the BLOCK_IO option.

Scst defines LUNs are a part of security groups. The command to do this in scstadmin is

  • scstadmin -assigndev <vdisk01> -group Default -lun <0>

Again, you can replace the options in angled brackets as necessary.

Lastly, commit your changes to disk with

By default, the scst service will start on boot and be detectable by SRPs on your network. However, you need to use lsmod to verify that the ib_srp and ib_srpt kernel modules are loaded. You be unable to establish any srp connections if they are not. Put the modules into /etc/modules.conf if needed.

You can use /etc/initiators.allow and /etc/initiators.deny to add a bit more security to your setup, but I would advise coming back to this step later. In the event your iscsi connection has issues, you should keep things simple first.

That's it for your target configuration. We'll be moving on to initiator configuration now.

SRP initiator

This step will need to repeated on each of your SRP initiators.

Make sure that the ib_srp kernel module is loaded on your initiator.

On your initiator, run

  • ibsrpdm -c

to see whether your initiator is able to establish an srp connection to your target. If you get output like

  • id_ext=50001ff10005052a,ioc_guid=50001ff10005052a,dgid=fe8000000000000050001ff10005052a,pkey=ffff,service_id=2a050500f11f0050

then you're golden. If not, double check your infiniband connection.

Next, edit your /etc/srp_daemon.conf file and add an entry with your id_ext and ioc_guid. For example, from the ipsrpdm output above, you would add the line

  • a id_ext=50001ff10005052a,ioc_guid=50001ff10005052a

to your srp_daemon.conf file. Note the 'a' is for allow. For security reasons, the last line in the file should be 'd' to deny all other connections.

You will also need to edit /etc/defaults/srptools to allow connections on your infiniband adapter. You will need to have the syntax

  • PORTS="<your_ib_card>:<your_ib_card_port>".

My mellanox card's entry was

  • PORTS="mlx4_0:1"

For some reason, the PORTS=ALL option does not work, despite being present in the config file.

Lastly, restart the srptools service in /etc/init.d/srptools and you should see entries in dmesg about a new block device being discovered. fdisk -l should also show the new block device. Simply add it to your fstab and you're set!


CategoryDocumentation

scst (last edited 2010-10-08 00:34:46 by eth2083)