This document is a quickstart guide for a user connected to a Fibre Channel
Forwarder (FCF) and not a SW target. It covers the configuration of the kernel
and installation and configuration of Data Center Bridging (DCB), the HBA API
wrapper library, libhbalinux and fcoe-utils.

This document was written using Fedora 11 as the installed operating system.
The instructions may need adjustments for them to work on other
distributions.

##
# Dependencies
###############

https://github.com/cleech/libHBAAPI.git
https://github.com/cleech/libhbalinux.git
https://github.com/openSUSE/fcoe-utils.git

##
# Kernel Configuration and Compilation
######################################

1) Download kernel source
   There are a number of good choices regarding the most appropriate
   kernel source for your needs. fcoe-next.git is a repository that may be
   unstable, but will have the latest code. Downloading a released kernel
   from kernel.org will give you the most stable kernel, but you'll need to
   get user space code that matches the kernel version you've chosen. You can
   get user space code that matches stable kernels on the Open-FCoE.org
   "Downloads" page or you can pull directly from the git repositories using
   tags.

2) Configure the kernel
   # make menuconfig

   Select the following:

   Networking Support -> Networking Options
	      Data Center Bridging
	      802.1Q VLAN Support
	      GVRP (GARP VLAN Registration Protocol) support

   Networking Support -> Networking Options -> QoS and/or fair queuing
	      Multi Band Priority Queueing (PRIO)
	      Hardware Multiqueue-aware Multi Band Queuing (MULTIQ)
	      Elementary classification (BASIC)
	      Universal 32bit comparisons w/ hashing (U32)
	      Extended Matches
			U32 key
	      Actions
			SKB Editing

   Device Drivers -> SCSI Device Support -> SCSI Low-level drivers
	      LibFC
	      LibFCoE
	      FCoE

   Enable the block layer
	      Block layer SG support v4

   Device Drivers -> Network Device Support -> Ethernet (10000 Mbit)
	      Intel(R) 10GbE PCI Express adapters support
	      Data Center Bridging (DCB) Support

   [ Replace with desired driver if not using Intel adapter. ]

   Exit Saving Changes

3) Compile the kernel
   # make && make modules_install && make install

   If you're going to build fcoe-utils, you can run 'make headers_install'
   at this stage to skip a later step.

4) Configure Grub
   Change the default kernel so that this new kernel is used when booting
   # reboot


##
# DCB
#######

DEPENDENCIES

* libnl
* libnl-devel
* libconfig
* libconfig-devel
* autoconf
* autotools
* sysconftool
* automake
* libtool
* gcc-c++
* flex
* readline-devel

PROCESS

1) Obtain the latest source

   git clone git://open-lldp.org/open-lldp

   (note: fcoe-utils versions >= 1.0.20 require an lldpad version >= 0.9.43)

2) Obtain libconfig version 1.3.2 or greater included in distribution
   devel packages source can be obtained directly from
   http://www.hyperrealm.com/libconfig/

3) Build and install libconfig -- see libconfig documentation

4) Obtain and install devel netlink library (libnl) version 1.1 or greater
   included in distribution devel packages source can be obtained directly
   from http://www.infradead.org/~tgr/libnl/

5) Bootstrap, configure, make and make install
   # cd open-lldp
   # ./bootstrap.sh
   # rpm --eval "%configure" | sh
   # make
   # make install


##
# HBA API Wrapper Library
##########################

DEPENDENCIES

* autoconf
* autotools
* sysconftool
* automake
* libtool

PROCESS

1) Obtain the latest source

   git clone https://github.com/cleech/libHBAAPI.git

2) Bootstrap, configure, make and make install
   # cd libHBAAPI
   # ./bootstrap.sh
   # rpm --eval "%configure" | sh
   # make
   # make install


##
# libhbalinux
##############

DEPENDENCIES

* HBA API Wrapper

* autoconf
* autotools
* sysconftool
* automake
* libtool
* libpciaccess-devel

PROCESS

1) Obtain the latest source

   git clone https://github.com/cleech/libhbalinux.git

2) Bootstrap, configure, make and make install
   # cd libhbalinux
   # ./bootstrap.sh
   # rpm --eval "%configure" | sh
   # make
   # make install


##
# fcoe-utils
############

DEPENDENCIES

* libnl (see DCB section)
* libhbalinux
* lldpad v0.9.26 or greater (optional configure flag to build without)

* autoconf
* autotools
* sysconftool
* automake
* libtool

PROCESS

1) Obtain the latest source

   git clone https://github.com/openSUSE/fcoe-utils.git

2) Bootstrap, configure, make and make install
   # cd fcoe-utils
   # ./bootstrap.sh
   Either execute the next line for System V init script install
   # rpm --eval "%configure" | sh
   or the following line to install systemd unit scripts
   # rpm --eval "%configure --with-systemdsystemunitdir=/" | sh
   # make
   # make install

##
# Example Configuration
#######################

This example configures interface eth3 to automatically connect to storage over
a discovered VLAN.

1) Configure FCoE on the interface
   # cd /etc/fcoe/
   # cp cfg-ethx cfg-eth3

2) Start lldpad and configure the interface for DCB.
   # service lldpad start
   # dcbtool sc eth3 dcb on
   # dcbtool sc eth3 pfc e:1
   # dcbtool sc eth3 app:fcoe e:1

   As a convenience there is a script that will confirm if DCB has been
   configured correctly for FCoE. The script is run as follows,

   <fcoe-utils source>/debug/dcbcheck.sh eth3
   (note: this is on the root device, not the VLAN)

   Follow the suggestions and repeatedly run the script until it states that
   DCB is configured correctly.

3) Start fcoe
   # service fcoe start
     After a few moments your storage should appear (assuming everything is
     configured correctly on the fabric)

4) Setup lldpad and fcoe to start when booting
    # chkconfig lldpad on
    # chkconfig fcoe on


##
# Reporting Errors
##################

Most steps cannot be skipped. Resolve issues before moving forward.

Please run '<fcoe-utils-src>/debug/fcoedump.sh eth3.<VID>-fcoe &>dump.out'
and provide the dump.out file with the bug report. Defects can be filed
against the distribution used, reported to the Ethernet vendor of the
card in use, or reported to the "Issues" page for each of the packages
on github.


##
# Debugging Tools
###################

1) Discover VLANs manually
   # fipvlan -a
   Fibre Channel Forwarders Discovered
   interface | VLAN | FCF MAC
   ------------------------------------
   eth3      | 101  | 00:0d:ec:a3:3a:80

   VLANs should be automatically discovered by the FCoE start up process which
   adds a VLAN to the real device. The VLAN can be identified by the name
   ethx.vid-fcoe.

2) fcping can be used to ping targets to verify connectivity.

   # fcping -h eth3.<VID>-fcoe  -F E10AEF -c 3
     sending echo to 0xE10AEF
     echo    1 REJECT received                 0.116 ms
     echo    2 REJECT received                 0.222 ms
     echo    3 REJECT received                 0.226 ms
     3 frames sent, 3 received 3 errors, 0.000% loss, avg. rt time 0.188 ms
