  Infiniband HOWTO
  Guy Coates


  This document describes how to install and configure the OFED infini-
  band software on Debian.
  ______________________________________________________________________

  Table of Contents



  1. Introduction
     1.1 The latest version
     1.2 What is OFED?

  2. Installing the OFED Software
     2.1 Installing prebuilt packages
     2.2 Building packages from source
        2.2.1 Install the prerequisites development packages
        2.2.2 Checkout the svn tree
        2.2.3 Install the upstream source (optional)
        2.2.4 Build the packages.

  3. Install the kernel modules
     3.1 Building new kernel modules

  4. Setting up a basic infiniband network
     4.1 Upgrade your Infiniband card and switch firmware
     4.2 Physically Connect the network
     4.3 Choose a Subnet Manager
     4.4 Load the kernel modules
     4.5 (optional) Start opensm
     4.6 Check network health
     4.7 Check the extended network connectivity
     4.8 testing connectivity with ibping
     4.9 Testing RDMA performance

  5. IP over Infiniband (IPoIB)
     5.1 List the network devices
     5.2 IP Configuration
     5.3 Connected vs Unconnected Mode
     5.4 TCP tuning
     5.5 ARP and dual ported cards

  6. OpenMPI
     6.1 Configure IPoIB
     6.2 Load the modules
     6.3 Check permissions and limits
     6.4 Install the mpi test programs
     6.5 Configure Host Access
     6.6 Run the MPI PingPong benchmark

  7. SDP
     7.1 Configuration
     7.2 Example Using SDP with Netpipe

  8. SRP
     8.1 Configuration
     8.2 SRP daemon configuration
        8.2.1 Determine the IDs of presented devices
        8.2.2 Configure srp_deamon to connect to the devices
     8.3 Multipathing, LVM and formatting

  9. Building Lustre against OFED
     9.1 Check Compatibility
     9.2 Build a lustre patched kernel
     9.3 Build OFED modules for the lustre patched kernel
     9.4 Configure lustre

  10. Troubleshooting
     10.1 Genernal fabric troubleshooting
     10.2 ib_query_gid() failed errors on mlx4 platforms
     10.3 Missing XRC support

  11. Tips and Tricks
     11.1 Descriptive node names

  12. Further Information


  ______________________________________________________________________

  11..  IInnttrroodduuccttiioonn

  This document describes how to install and configure the OFED
  infiniband software on Debian. This document is intended to show you
  how to configure a simple Infiniband network as quickly as possible.
  It is not a replacement for the details documentation provided in the
  ofed-docs package!

  11..11..  TThhee llaatteesstt vveerrssiioonn

  The latest version of the howto can be found on the pkg-ofed alioth
  webite:

  http://pkg-ofed.alioth.debian.org/howto/infiniband-howto.html
  <http://pkg-ofed.alioth.debian.org/howto/infiniband-howto.html>

  Source is kept in the SVN repository:

  http://svn.debian.org/wsvn/pkg-ofed/ <http://svn.debian.org/wsvn/pkg-
  ofed/>

  11..22..  WWhhaatt iiss OOFFEEDD??

  OFED (OpenFabric's Enterprise Distribution) is the defacto Infiniband
  software stack on Linux. OFED provides a consistent set of kernel
  modules and userspace libraries which have been tested together.

  Further details of the Openfabrics Alliance and OFED can be found here
  http://www.openfabrics.org <http://www.openfabrics.org/>

  22..  IInnssttaalllliinngg tthhee OOFFEEDD SSooffttwwaarree

  Before you can use your infiniband network you will need to install
  the OFED software on your infiniband client machines.  You can choose
  to use the pre-build packages on alioth, or build your own packages
  straight from the alioth SVN repository.

  22..11..  IInnssttaalllliinngg pprreebbuuiilltt ppaacckkaaggeess

  Add the following lines to your sources.list file:


       deb http://pkg-ofed.alioth.debian.org/apt/ofed ./
       deb-src http://pkg-ofed.alioth.debian.org/apt/ofed ./



  and run:


       aptitude update
       aptitude install ofed



  22..22..  BBuuiillddiinngg ppaacckkaaggeess ffrroomm ssoouurrccee

  If you wish to build the OFED packages from the alioth svn repository,
  use the following procedure.
  22..22..11..  IInnssttaallll tthhee pprreerreeqquuiissiitteess ddeevveellooppmmeenntt ppaacckkaaggeess



       aptitude install svn-buildpackage build-essential devscripts



  22..22..22..  CChheecckkoouutt tthhee ssvvnn ttrreeee


       svn co svn://svn.debian.org/pkg-ofed/


  22..22..33..  IInnssttaallll tthhee uuppssttrreeaamm ssoouurrccee ((ooppttiioonnaall))

  The upstream source tarballs need to be available if you want to build
  pukka debian packages suitable for inclusion upstream. If you are
  simply building packages for your own use, you can ignore this step.


       cd pkg-ofed
       mkdir tarballs



  Original source tarballs can be downloaded from the repository:


         apt-get source libibverbs



  Alternatively, you can grab the source code directly from upstream.

  http://www.openfabrics.org/downloads/OFED/

  Upstream source is distributed via SRPMS; you can use alien to convert
  them into tarballs.

  22..22..44..  BBuuiilldd tthhee ppaacckkaaggeess..

  cd into the package you wish to build. eg for libibcommon,

       cd pkg-ofed/libibcommon


  Link in the upstream tarballs directory (optional)

       ln -s -f ../tarballs .


  Run svn-buildpackage from within the trunk directory.


        cd pkg-ofed/libibcommon/trunk
        svn-buildpackage -uc -us -rfakeroot



  The build process will generate a deb in the build-area directory.

  Repeat the process for the rest of the packages. Note that some
  packages have build dependancies on other OFED packages. The suggested
  build order is:


        libibverbs
        libnes
        libcxgb3
        libipathverbs
        libmlx4
        libmthca
        librdmacm
        libibcm
        libibcommon
        libibumad
        libibmad
        libsdp
        dapl
        opensm
        infiniband-diags
        ibutils
        mstflint
        perftest
        qlvnictools
        qperf
        rds-tools
        sdpnetstat
        srptools
        tvflash
        ibsim
        mpitests
        ofed-docs
        ofa_kernel
        ofed



  33..  IInnssttaallll tthhee kkeerrnneell mmoodduulleess

  You now need to build a set of OFED kernel modules which match the
  version of the OFED software you have installed.

  The Debian kernel contains a set of OFED infiniband drivers, but they
  may not match the OFED userspace version have installed.  Consult the
  table below to determine what OFED version the Debian kernel contains.



       Debian Kernel Version      OFED Version
       <=2.6.26                       1.3
       >=2.6.27                       1.4



  If the debian kernel modules are the incorrect version, you can build
  a new set of modules using the ofa-kernel-source package.  If your
  kernel already includes the correct OFED kernel modules you can skip
  the rest of this section. If you are in doubt, you should build a new
  set of modules rather than relying on the modules shipped with the
  kernel.



  33..11..  BBuuiillddiinngg nneeww kkeerrnneell mmoodduulleess

  You can build new kernel modules using module-assistant.


       aptitude install module-assistant



  Ensure you have the ofa-kernel-source package installed, and then run:


        module-assistant prepare
        module-assistant clean ofa-kernel
        module-assistant build ofa-kernel



  This procedure will create an ofa-kernel-modules deb in /usr/src. You
  can the install the deb using dpkg or by running:


        module-assistant install ofa-kernel



  The deb can also be copied to your other infiniband hosts and
  installed using dpkg.

  As the deb contains replacements for existing kernel modules you will
  need to either manually remove any infiniband modules which have
  already been loaded, or reboot the machine, before you can use the new
  modules.

  The new kernel modules will be installed into /usr/lib/<kernel-
  version>/updates. They will not overwrite the original kernel modules,
  but the module loader will pick up the modules from the updates
  directory in preference. You can verify that the system is using the
  new kernel modules by running the modinfo command.



       # modinfo ib_core
       filename:       /lib/modules/2.6.22.19/updates/kernel/drivers/infiniband/core/ib_core.ko
       author:         Roland Dreier
       description:    core kernel InfiniBand API
       license:        Dual BSD/GPL
       vermagic:       2.6.22.19 SMP mod_unload



  Note that if you wish to rebuild the kernel modules for any reason,
  (eg for a new kernel version or to continue an interrupted build) then
  you must issue the "module-assistant clean" command before trying a
  new build.

  44..  SSeettttiinngg uupp aa bbaassiicc iinnffiinniibbaanndd nneettwwoorrkk

  This sections describes how to set up a basic infiniband network and
  test its functionality.


  44..11..  UUppggrraaddee yyoouurr IInnffiinniibbaanndd ccaarrdd aanndd sswwiittcchh ffiirrmmwwaarree

  Before proceeding you should ensure that the firmware in your switches
  and infiniband cards is at the latest release.  Older firmware
  versions may cause interoperability and fabric stability issues. Do
  not assume that just because your hardware has come fresh from the
  factory that it has the latest firmware on it.

  You should follow the documentation from your vendor as to how the
  firmware should be updated.

  44..22..  PPhhyyssiiccaallllyy CCoonnnneecctt tthhee nneettwwoorrkk

  Connect up to your hosts and switches.

  44..33..  CChhoooossee aa SSuubbnneett MMaannaaggeerr

  Each infiniband network requires a subnet manager. You can choose to
  run the OFED opensm subnet manager on one of the Linux clients, or you
  may choose to use an embedded subnet manager running on one of the
  switches in your fabric. Note that not all switches come with a subnet
  manager; check your switch documentation.

  44..44..  LLooaadd tthhee kkeerrnneell mmoodduulleess

  Infiniband kernel modules are not loaded automatically. You should
  adding them to /etc/modules so that they are automatically loaded on
  machine bootup. You will need to include the hardware specific modules
  and the protocol modules.

  /etc/modules:

  # Hardware drivers
  # Choose the apropriate modules from
  # /lib/modules/<kernel-version>/updates/kernel/drivers/infiniband/hw
  #
  #mlx4_ib  # Mellanox ConnectX cards
  #ib_mthca # some mellanox cards
  #iw_cxgb3 # Chelsio T3 cards
  #iw_nes # NetEffect cards
  #
  # Protocol modules
  # Common modules
  rdma_ucm
  ib_umad
  ib_uverbs
  # IP over IB
  ib_ipoib
  # scsi over IB
  ib_srp
  # IB SDP protocol
  ib_sdp



  44..55..  ((ooppttiioonnaall)) SSttaarrtt ooppeennssmm

  If you are going to use the opensm suetnet manager, edit
  /etc/default/opensm and add the port GUIDs of the interfaces on which
  you wish to start opensm.

  You can find the port GUIDs of your cards with the ibstat -p command:



  # ibstat -p
  0x0002c9030002fb05
  0x0002c9030002fb06



  /etc/default/opensm:


       PORTS="0x0002c9030002fb05 0x0002c9030002fb06"



  Note if you want to start opensm on all ports you can use the
  PORTS="ALL" keyword.

  Start opensm:


  #/etc/init.d/opensm start



  If opensm has started correctly you should see SUBNET UP messages in
  the opensm logfile (/var/log/opensm.<PORTID>.log).


  Mar 04 14:56:06 600685 [4580A960] 0x02 -> SUBNET UP



  Note that you can start opensm on multiple nodes; one node will be the
  active subnet manager and the others will put themselves into standby.

  44..66..  CChheecckk nneettwwoorrkk hheeaalltthh

  You can now check the status of the local IB link with the ibstat
  command. Connected links should be in the "LinkUp" state. The
  following output is from a dual ported card, only one of which (port1)
  is connected.



  # ibstat
  CA 'mlx4_0'
          CA type: MT25418
          Number of ports: 2
          Firmware version: 2.3.0
          Hardware version: a0
          Node GUID: 0x0002c9030002fb04
          System image GUID: 0x0002c9030002fb07
          Port 1:
                  State: Active
                  Physical state: LinkUp
                  Rate: 20
                  Base lid: 2
                  LMC: 0
                  SM lid: 1
                  Capability mask: 0x02510868
                  Port GUID: 0x0002c9030002fb05
          Port 2:
                  State: Down
                  Physical state: Polling
                  Rate: 10
                  Base lid: 0
                  LMC: 0
                  SM lid: 0
                  Capability mask: 0x02510868
                  Port GUID: 0x0002c9030002fb06



  44..77..  CChheecckk tthhee eexxtteennddeedd nneettwwoorrkk ccoonnnneeccttiivviittyy

  Once the host is connected to the infiniband network you can check the
  health of all of the other network components with the ibhosts,
  ibswitches and iblinkinfo commands.

  ibhosts displays all of the hosts visible on the network.



       # ibhosts
       Ca      : 0x0008f1040399d3d0 ports 2 "Voltaire HCA400Ex-D"
       Ca      : 0x0008f1040399d370 ports 2 "Voltaire HCA400Ex-D"
       Ca      : 0x0008f1040399d3fc ports 2 "Voltaire HCA400Ex-D"
       Ca      : 0x0008f1040399d3f4 ports 2 "Voltaire HCA400Ex-D"
       Ca      : 0x0002c9030002faf4 ports 2 "MT25408 ConnectX Mellanox Technologies"
       Ca      : 0x0002c9030002fc0c ports 2 "MT25408 ConnectX Mellanox Technologies"
       Ca      : 0x0002c9030002fc10 ports 2 "MT25408 ConnectX Mellanox Technologies"



  ibswitches will display all of the switches in the network.


       # ibswitches
       Switch  : 0x0008f104004121fa ports 24 "ISR9024D-M Voltaire" enhanced port 0 lid 1 lmc 0



  iblinkinfo will show the status and speed of all of the links in the
  network.



  #iblinkinfo.pl
  Switch 0x0008f104004121fa ISR9024D-M Voltaire:
        1    1[  ]  ==( 4X 5.0 Gbps Active /   LinkUp)==>       2    1[  ] "MT25408 ConnectX Mellanox Technologies" (  )
        1    2[  ]  ==( 4X 5.0 Gbps Active /   LinkUp)==>      13    1[  ] "MT25408 ConnectX Mellanox Technologies" (  )
        1    3[  ]  ==( 4X 5.0 Gbps Active /   LinkUp)==>       4    1[  ] "MT25408 ConnectX Mellanox Technologies" (  )
        1    4[  ]  ==( 4X 5.0 Gbps Active /   LinkUp)==>      26    1[  ] "MT25408 ConnectX Mellanox Technologies" (  )
        1    5[  ]  ==( 4X 5.0 Gbps Active /   LinkUp)==>      27    1[  ] "MT25408 ConnectX Mellanox Technologies" (  )
        1    6[  ]  ==( 4X 5.0 Gbps Active /   LinkUp)==>      24    1[  ] "MT25408 ConnectX Mellanox Technologies" (  )
        1    7[  ]  ==( 4X 5.0 Gbps Active /   LinkUp)==>      28    1[  ] "MT25408 ConnectX Mellanox Technologies" (  )
        1    8[  ]  ==( 4X 5.0 Gbps Active /   LinkUp)==>      25    1[  ] "MT25408 ConnectX Mellanox Technologies" (  )
        1    9[  ]  ==( 4X 5.0 Gbps Active /   LinkUp)==>      31    1[  ] "MT25408 ConnectX Mellanox Technologies" (  )
        1   10[  ]  ==( 4X 5.0 Gbps Active /   LinkUp)==>      32    1[  ] "MT25408 ConnectX Mellanox Technologies" (  )
        1   11[  ]  ==( 4X 5.0 Gbps Active /   LinkUp)==>      33    1[  ] "MT25408 ConnectX Mellanox Technologies" (  )
        1   12[  ]  ==( 4X 5.0 Gbps Active /   LinkUp)==>      29    1[  ] "MT25408 ConnectX Mellanox Technologies" (  )
        1   13[  ]  ==( 4X 5.0 Gbps Active /   LinkUp)==>      30    1[  ] "MT25408 ConnectX Mellanox Technologies" (  )
            14[  ]  ==( 4X 2.5 Gbps   Down /  Polling)==>             [  ] "" (  )
        1   15[  ]  ==( 4X 5.0 Gbps Active /   LinkUp)==>       3    1[  ] "Voltaire HCA400Ex-D" (  )
        1   16[  ]  ==( 4X 5.0 Gbps Active /   LinkUp)==>      10    1[  ] "Voltaire HCA400Ex-D" (  )
            17[  ]  ==( 4X 2.5 Gbps   Down /  Polling)==>             [  ] "" (  )
            18[  ]  ==( 4X 2.5 Gbps   Down /  Polling)==>             [  ] "" (  )
        1   19[  ]  ==( 4X 5.0 Gbps Active /   LinkUp)==>       7    2[  ] "Voltaire HCA400Ex-D" (  )
        1   20[  ]  ==( 4X 5.0 Gbps Active /   LinkUp)==>       6    2[  ] "Voltaire HCA400Ex-D" (  )
        1   21[  ]  ==( 4X 5.0 Gbps Active /   LinkUp)==>       5    2[  ] "Voltaire HCA400Ex-D" (  )
        1   22[  ]  ==( 4X 5.0 Gbps Active /   LinkUp)==>      21    1[  ] "Voltaire HCA400Ex-D" (  )
        1   23[  ]  ==( 4X 5.0 Gbps Active /   LinkUp)==>       9    2[  ] "Voltaire HCA400Ex-D" (  )
        1   24[  ]  ==( 4X 5.0 Gbps Active /   LinkUp)==>       8    1[  ] "Voltaire HCA400Ex-D" (  )



  44..88..  tteessttiinngg ccoonnnneeccttiivviittyy wwiitthh iibbppiinngg

  ibping is an infiniband equivalent to the icmp ping command. Choose a
  node on the fabric and run a ibping server:

       #ibping -S


  Choose another node on your network, and then ping the port GUID of
  the server. (ibstat on the server will list the port GUID).



       #ibping -G 0x0002c9030002fc1d
       Pong from test.example.com (Lid 13): time 0.072 ms
       Pong from test.example.com (Lid 13): time 0.043 ms
       Pong from test.example.com (Lid 13): time 0.045 ms
       Pong from test.example.com (Lid 13): time 0.045 ms



  44..99..  TTeessttiinngg RRDDMMAA ppeerrffoorrmmaannccee

  You can test the latency and bandwidth of a link with the ib_rdma_lat
  commands.

  To test the latency, start the server on a node:

       #ib_rdma_lat


  and then start a client on another node, giving it the hostname of the
  server.


  #ib_rdma_lat  hostname-of-server
     local address: LID 0x0d QPN 0x18004a PSN 0xca58c4 RKey 0xda002824 VAddr 0x00000000509001
    remote address: LID 0x02 QPN 0x7c004a PSN 0x4b4eba RKey 0x82002466 VAddr 0x00000000509001
  Latency typical: 1.15193 usec
  Latency best   : 1.13094 usec
  Latency worst  : 5.48519 usec



  You can test the bandwith of the link using the ib_rdma_bw command.

       #ib_rdma_bw


  and then start a client on another node, giving it the hostname of the
  server.


       #ib_rdma_bw  hostname-of-server
       855: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | iters=1000 | duplex=0 | cma=0 |
       855: Local address:  LID 0x0d, QPN 0x1c004a, PSN 0xbf60dd RKey 0xde002824 VAddr 0x002aea4092b000
       855: Remote address: LID 0x02, QPN 0x004a, PSN 0xaad03c, RKey 0x86002466 VAddr 0x002b8a4e191000


       855: Bandwidth peak (#0 to #955): 1486.85 MB/sec
       855: Bandwidth average: 1486.47 MB/sec
       855: Service Demand peak (#0 to #955): 1970 cycles/KB
       855: Service Demand Avg  : 1971 cycles/KB



  The perftest package contains a number of other similar benchmarking
  programs to test various aspects of your network.

  55..  IIPP oovveerr IInnffiinniibbaanndd ((IIPPooIIBB))

  The OFED stack allows you to run TCP/IP over your infiniband network,
  allowing you to run non-infiniband aware applications across your
  network. Several native infiniband applications also use IPoIB for
  host resolution (eg Lustre and SDP).

  55..11..  LLiisstt tthhee nneettwwoorrkk ddeevviicceess

  Check that the IBoIP modules is loaded.


       #modprobe ib_ipoib


  You will now have an "ib" network interface for each of your infini-
  band cards.



  #ifconfig -a

  <snip>
  ib0       Link encap:UNSPEC  HWaddr 80-06-00-48-FE-80-00-00-00-00-00-00-00-00-00-00
            BROADCAST MULTICAST  MTU:2044  Metric:1
            RX packets:0 errors:0 dropped:0 overruns:0 frame:0
            TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:256
            RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

  ib1       Link encap:UNSPEC  HWaddr 80-06-00-49-FE-80-00-00-00-00-00-00-00-00-00-00
            BROADCAST MULTICAST  MTU:2044  Metric:1
            RX packets:0 errors:0 dropped:0 overruns:0 frame:0
            TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:256
            RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
  <snip>



  55..22..  IIPP CCoonnffiigguurraattiioonn

  You can now configure the ib network devices using
  /etc/network/interfaces.


       auto ib0
       iface ib0 inet static
         address 172.31.128.50
         netmask 255.255.240.0
         broadcast 172.31.143.255



  Bring the network device up, as normal.

       ifup ib0


  55..33..  CCoonnnneecctteedd vvss UUnnccoonnnneecctteedd MMooddee

  IPoIB can run over two infiniband transports, Unreliable Datagram (UD)
  mode or Connected mode (CM). The difference between these two modes
  are described in:

  RFC4392 - IP over InfiniBand (IPoIB) Architecture
  RFC4391 - Transmission of IP over InfiniBand (IPoIB) (UD mode)
  RFC4755 - IP over InfiniBand: Connected Mode


  ADDME: Pro/cons of these two methods?

  You can switch between these two mode at runtime with:



        echo datagram > /sys/class/net/ibX/mode
        echo connected > /sys/class/net/ibX/mode



  The default is datagram (UD) mode. If you with to use CM then you can
  add a script to /etc/network/interfaces/if-up.d to automatically set
  CM mode on your interfaces when they are configured.

  55..44..  TTCCPP ttuunniinngg

  In order to obtain maximum IPoIB throughput you may need to tweak the
  MTU and various kernel TCP buffer and window settings.  See the
  details in the ipoib_release_notes.txt document in the ofed-docs
  package.

  55..55..  AARRPP aanndd dduuaall ppoorrtteedd ccaarrddss

  If you have a dual ported card with both ports on the same IB subnet,
  but different IP subnets, you will need to tweak the ARP settings for
  the IPoIB interfaces. See ipoib_release_notes.txt in the ofed-docs
  package for a full discussion of this issue.



          sysctl -w net.ipv4.conf.ib0.arp_ignore=1
          sysctl -w net.ipv4.conf.ib1.arp_ignore=1



  66..  OOppeennMMPPII

  The section describes how to configure OpenMPI to use Infiniband.

  66..11..  CCoonnffiigguurree IIPPooIIBB

  OpenMPI uses IPoIB for job startup and tear-down. You should configure
  IPoIB on all of your hosts.

  66..22..  LLooaadd tthhee mmoodduulleess

  Ensure the rdma_ucm module is loaded.

       modprobe rdma_ucm


  66..33..  CChheecckk ppeerrmmiissssiioonnss aanndd lliimmiittss

  Uses who want to run MPI jobs will need to have write permissions for
  the following devices:


        /dev/infiniband/uverbs*
       /dev/infiniband/rdma_cm*



  The simplest way to do this is to add the users to the rdma group. If
  that is not suitiable for your site, you can change the permissions
  and ownership of these devices by editing the following udev rules:


       /etc/udev/rules.d/50-udev.rules
       /etc/udev/rules.d/91-permissions.rules



  OpenMPI will need to pin memory. Edit /etc/security/limits.conf and
  add the line:

  * hard memlock unlimited


  66..44..  IInnssttaallll tthhee mmppii tteesstt pprrooggrraammss

  Check the mpitests package is installed.

       aptitude install mpitests


  66..55..  CCoonnffiigguurree HHoosstt AAcccceessss

  OpenMPI uses ssh to spawn jobs on remote hosts. You should configure a
  public/private keypair to ensure that you can ssh between hosts
  without entering a password. You should also ensure that your login
  process is silent.

  66..66..  RRuunn tthhee MMPPII PPiinnggPPoonngg bbeenncchhmmaarrkk

  We will use the MPI PingPong benchmark for our testing. By default,
  openmpi should use inifiniband networks in preference to any tcp
  networks it finds. However, we will force mpi to ignore tcp networks
  to ensure that is using the infiniband network.


  #!/bin/bash
  #Infiniband MPI test program
  #Edit the hosts below to match your test hosts
  cat > /tmp/hostfile.$$.mpi <<EOF
  hostA slots=1
  HostB slots=1
  EOF

  mpirun --mca btl_openib_verbose 1 --mca btl ^tcp -n 2 -hostfile /tmp/hostfile.$$.mpi IMB-MPI1 PingPong



  If all goes well you should see openib debugging messages from both
  hosts, together with the job output.



  <snip>
  # PingPong
  [HostB][0,1,1][btl_openib_endpoint.c:992:mca_btl_openib_endpoint_qp_init_query] Set MTU to IBV value 4 (2048 bytes)
  [HostB][0,1,1][btl_openib_endpoint.c:992:mca_btl_openib_endpoint_qp_init_query] Set MTU to IBV value 4 (2048 bytes)
  [HostA][0,1,0][btl_openib_endpoint.c:992:mca_btl_openib_endpoint_qp_init_query] Set MTU to IBV value 4 (2048 bytes)
  [HostA][0,1,0][btl_openib_endpoint.c:992:mca_btl_openib_endpoint_qp_init_query] Set MTU to IBV value 4 (2048 bytes)

  #---------------------------------------------------
  # Benchmarking PingPong
  # #processes = 2
  #---------------------------------------------------
         #bytes #repetitions      t[usec]   Mbytes/sec
              0         1000         1.53         0.00
              1         1000         1.44         0.66
              2         1000         1.42         1.34
              4         1000         1.41         2.70
              8         1000         1.48         5.15
             16         1000         1.50        10.15
             32         1000         1.54        19.85
             64         1000         1.79        34.05
            128         1000         3.01        40.56
            256         1000         3.56        68.66
            512         1000         4.46       109.41
           1024         1000         5.37       181.92
           2048         1000         8.13       240.25
           4096         1000        10.87       359.48
           8192         1000        15.97       489.17
          16384         1000        30.54       511.68
          32768         1000        55.01       568.12
          65536          640       122.20       511.46
         131072          320       207.20       603.27
         262144          160       377.10       662.96
         524288           80       706.21       708.00
        1048576           40      1376.93       726.25
        2097152           20      1946.00      1027.75
        4194304           10      3119.29      1282.34



  If you encounter any errors read the excellent OpenMPI troubleshooting
  guide. http://www.openmpi.org <http://www.openmpi.org>

  If you want to compare infiniband performance with your ethernet/TCP
  networks, you can re-run the tests using flags to tell openmpi to use
  your ethernet network. (The example below assumes that your test nodes
  are connected via eth0).


  #!/bin/bash
  #TCP MPI test program
  #Edit the hosts below to match your test hosts
  cat > /tmp/hostfile.$$.mpi <<EOF
  hostA slots=1
  HostB slots=1
  EOF
  mpirun --mca btl ^openib --mca btl_tcp_if_include eth0 --hostfile hostfile -n 2 IMB-MPI1 -benchmark PingPong



  You should notice signficantly higher latencies than for the
  infiniband test.



  77..  SSDDPP

  Sockets Direct Protocol (SDP) is a network protocol which provides an
  RDMA accelerated alternative to TCP over infiniband networks. OFED
  provides an LD_PRELOADable library (libsdp.so) which allows programs
  which use TCP to use the more efficient SDP protocol instead.  The use
  of an LD_PRELOADable libary means that the switch in protocol is
  transparent, and does not require the application to be recompiled.

  77..11..  CCoonnffiigguurraattiioonn

  SDP used IPoIB for address resolution, so you must configure IPoIB
  before using SDP.

  You should also ensure the ib_sdp kernel module is installed.

  modprobe ib_sdp



  You can use libsdp in two ways; you can either manually LD_PRELOAD the
  library whilst invoking your application, or create a config file
  which specifies which applications will use SDP.

  To manually LD_PRELOAD a library, simply set the LD_PRELOAD variable
  before invoking your application.

  LD_PRELOAD=libsdp.so ./path/to/your/application ...


  If you which to choose which programs will use SDP you can edit
  /etc/sdp.conf and specify which programs, ports and addresses are eli-
  gible for use.

  77..22..  EExxaammppllee UUssiinngg SSDDPP wwiitthh NNeettppiippee

  The following example shows how to use libsdp to make the TCP
  benchmarking application, netpipe, use SDP rather than TCP.  NodeA is
  the server and NodeB is the client. IPoIB is configured on both nodes,
  and NodeA's IPoIB address is 10.0.0.1

  Install netpipe on both nodes.

  aptitude install netpipe-tcp



  First, run the netpipe benchmark over TCP in order to obtain a
  baseline number.



       nodeA# NPtcp
       nodeB# NPtcp -h 10.0.0.1
       Send and receive buffers are 16384 and 87380 bytes
       (A bug in Linux doubles the requested buffer sizes)
       Now starting the main loop
         0:       1 bytes   2778 times -->      0.22 Mbps in      34.04 usec
         1:       2 bytes   2937 times -->      0.45 Mbps in      33.65 usec
         2:       3 bytes   2971 times -->      0.69 Mbps in      33.41 usec
       <snip>
       121: 8388605 bytes      3 times -->   2951.89 Mbps in   21680.99 usec
       122: 8388608 bytes      3 times -->   3008.08 Mbps in   21276.00 usec
       123: 8388611 bytes      3 times -->   2941.76 Mbps in   21755.66 usec


  Now repeat the test, but force netpipe to use SDP rather than TCP.



       nodeA# LD_PRELOAD=libsdp.so NPtcp
       nodeB# LD_PRELOAD=libsdp.so  NPtcp -h 10.0.0.1
       Send and receive buffers are 16384 and 87380 bytes
       (A bug in Linux doubles the requested buffer sizes)
       Now starting the main loop
         0:       1 bytes   9765 times -->      1.45 Mbps in       5.28 usec
         1:       2 bytes  18946 times -->      2.80 Mbps in       5.46 usec
         2:       3 bytes  18323 times -->      4.06 Mbps in       5.63 usec
       <snip>
       121: 8388605 bytes      5 times -->   7665.51 Mbps in    8349.08 usec
       122: 8388608 bytes      5 times -->   7668.62 Mbps in    8345.70 usec
       123: 8388611 bytes      5 times -->   7629.04 Mbps in    8389.00 usec



  You should see a significant increase in performance when using SDP.

  88..  SSRRPP

  SRP (SCSI Remote protocol or SCSI RDMA protocol) is a protocol that
  allows the use of SCSI devices across infiniband. If you have
  infiniband storage, use can use SRP to acess the devices.

  88..11..  CCoonnffiigguurraattiioonn

  Ensure that your infiniband storage is presented to the host in
  question. Check your storage controller documentation.  Ensure that
  the ib_srp kernel module is loaded and that the srptools package is
  installed.


       modprobe ib_srp


  88..22..  SSRRPP ddaaeemmoonn ccoonnffiigguurraattiioonn

  srp_daemon is responsible for discovering and connecting to SRP
  targets. The default configuration shipped with srp_daemon is to
  ignore all presented devices; this is a failsafe to prevent devices
  from being mounted by accident on the wrong hosts.

  The srp_daemon config file /etc/srp_daemon.conf has a simply syntax,
  and is described in the srp_daemon(1) manpage. Each line in this file
  is a rule which can be either to allow connection or to disallow
  connection according to the first character in the line (a or d
  accordingly) and ID of the storage device.

  88..22..11..  DDeetteerrmmiinnee tthhee IIDDss ooff pprreesseenntteedd ddeevviicceess

  You can determine the IDs of SRP devices presented to your hosts by
  running the ibsrpdm -c command.


       # ibsrpdm -c
       id_ext=50001ff10005052a,ioc_guid=50001ff10005052a,dgid=fe8000000000000050001ff10005052a,pkey=ffff,service_id=2a050500f11f0050



  88..22..22..  CCoonnffiigguurree ssrrpp__ddeeaammoonn ttoo ccoonnnneecctt ttoo tthhee ddeevviicceess

  Once we have the IDs of the devices, we can add them to
  /etc/srp_daemon.conf. You can also specify other srp related options
  for the target, such as max_cmd_per_lun and Max_sect. These are
  storage specific; check your vendor documentation for reccomended
  values.


       # This rule allows connection to our target
       a id_ext=50001ff10005052a,ioc_guid=50001ff10005052a,max_cmd_per_lun=32,max_sect=65535
       # This rule disallows everything else
       d



  Restart the srp_daemon and the storage target should now become visi-
  ble; check the kernel log to see if the disk has been detected.


  /etc/init.d/srptools restart



  In the example kernel log output the disk has been descovered as scsi
  device sdb.


       scsi 3:0:0:1: Direct-Access     IBM      DCS9900          5.03 PQ: 0 ANSI: 5
       sd 3:0:0:1: [sdb] 1953458176 4096-byte hardware sectors (8001365 MB)
       sd 3:0:0:1: [sdb] Write Protect is off
       sd 3:0:0:1: [sdb] Mode Sense: 97 00 10 08
       sd 3:0:0:1: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
       sd 3:0:0:1: [sdb] 1953458176 4096-byte hardware sectors (8001365 MB)
       sd 3:0:0:1: [sdb] Write Protect is off
       sd 3:0:0:1: [sdb] Mode Sense: 97 00 10 08
       sd 3:0:0:1: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
        sdb:<6>scsi4 : SRP.T10:50001FF10005052A
        unknown partition table
       sd 3:0:0:1: [sdb] Attached SCSI disk
       sd 3:0:0:1: Attached scsi generic sg5 type 0



  88..33..  MMuullttiippaatthhiinngg,, LLVVMM aanndd ffoorrmmaattttiinngg

  The newly detected SRP device can be treated as an other scsi device.
  If you have multiple infiniband adapters you can use multipath-tools
  on top of the SRP devices to protects against a network failure. If
  you are not using multipathed IO you can simply format the device as
  normal.

  99..  BBuuiillddiinngg LLuussttrree aaggaaiinnsstt OOFFEEDD

  Lustre is a scalable cluster filesystem popular on high performance
  compute clusters. See http://www.lustre.org <http://www.lustre.org>
  for more information. lustre can use infiniband as one of its network
  transports in order to increase performance. The section describes how
  to compile lustre against the OFED infiniband stack.

  99..11..  CChheecckk CCoommppaattiibbiilliittyy

  Not all lustre versions are compatible with all OFED or kernel
  versions. Read the lustre release notes for which versions are
  supported.

  99..22..  BBuuiilldd aa lluussttrree ppaattcchheedd kkeerrnneell

  Build a lustre patched kernel as per the instructions on the lustre
  wiki. Once you have build the kernel keep the configured source tree.
  It is required for the next step.

  99..33..  BBuuiilldd OOFFEEDD mmoodduulleess ffoorr tthhee lluussttrree ppaattcchheedd kkeerrnneell

  Build OFED modules against the newly build lustre patched kernel.



        module-assistant prepare
        module-assistant clean ofa-kernel
        module-assistant -k/path/to/lustre/patched/kernel build ofa-kernel



  Do not issue a "module-assistant clean" command after the build. The
  ofa-kernel-module source tree is needed for the next step.

  99..44..  CCoonnffiigguurree lluussttrree

  You can now configure lustre to build against the lustre patched
  kernel and the ofa-kernel-module sources.



        cd lustre-source
        ./configure --with-o2ib=/usr/src/modules/ofa-kernel  --with-linux=/path/to/patched/linux/source \
        --other-options



  1100..  TTrroouubblleesshhoooottiinngg

  This section covers general troubleshooting and commonly reported
  problems.

  1100..11..  GGeenneerrnnaall ffaabbrriicc ttrroouubblleesshhoooottiinngg

  The ibdiagnet program can be used to troubleshoot potential issues
  with your infiniband fabric.

       ibdiagnet -r


  1100..22..  iibb__qquueerryy__ggiidd(()) ffaaiilleedd eerrrroorrss oonn mmllxx44 ppllaattffoorrmmss

  ibstat or opensm hangs and the following kernel messages are printed:



       kernel: [   78.170077] ib0: ib_query_gid() failed
       kernel: [   89.272789] ib0: ib_query_port failed



  Fix: Load the mlx4_core module with the msi_x=0 option.


       cat > /etc/modprobe.d/mlx4_core <<EOF
       options mlx4_core msi_x=0
       EOF

       update-initramfs -u



  1100..33..  MMiissssiinngg XXRRCC ssuuppppoorrtt

  If you see error messages pertaining to missing support for XRC, it
  means you have mis-matched kernel modules and userspace libraries.


       mlx4: There is a mismatch between the kernel and the userspace
       libraries: Kernel does not support XRC. Exiting.



  Fix: Make sure that you build and install the OFED kernel modules as
  described in section X.

  1111..  TTiippss aanndd TTrriicckkss

  This section details an assortment of miscellaneous tips.

  1111..11..  DDeessccrriippttiivvee nnooddee nnaammeess

  You can give your hosts descriptive names by echoing text to the
  following file:


       echo `uname -n` > /sys/class/infiniband/<driver>/node_desc



  1122..  FFuurrtthheerr IInnffoorrmmaattiioonn

  Extensive documentation on the OFED software is present in the ofed-
  docs package.

  The openfabrics alliance webpage can be found here:

  http://www.openfabrics.org/ <http://www.openfabrics.org/>

  The following mailing lists are also useful:

  http://lists.alioth.debian.org/mailman/listinfo/pkg-ofed-devel
  <http://lists.alioth.debian.org/mailman/listinfo/pkg-ofed-devel>: pkg-
  ofed-devel: Discussion of debian specific problem or issues.

  http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
  <http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general>: ofa-
  general: General discussion of the OFED software.

  Books:

  Infiniband Network Architecture
  by MindShare, Inc.; Tom Shanley
  Publisher: Addison-Wesley Professional
  Pub Date: October 31, 2002
  Print ISBN-10: 0-321-11765-4



