# Performing the azimutal integration from shell scripts

There are three main tools for performing azimuthal integration from the command line. Two of them specicalized for SAXS and WAXS and the last which is common to the GUI but we will see how to use it in command line mode (CLI).

They all require 3 mandatory parameters:

- The PONI-file containing the geometry of the experimental setup
- The number of bins used for the histogram
- The list of input files

The three tools differ only by the default options selected to fit the best the needs of a given user community.

## Azimuthal integration for powder diffraction

The pyFAI-waxs which stands for wide angle scattering produces integrated patterns in \(2\theta\) units by default.

Note: while this cookbook is performed in a Jupyter notebook, all commands are run in the command line interpreter (Bash), not in Python.

    [1]:

    pyFAI-waxs -p LaB6_29.4keV.poni -n 1000 LaB6_29.4keV.tif

    Detector Detector        Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline  PixelSize= 5.168e-05, 5.126e-05 m
    Wavelength= 4.217150e-11m
    SampleDetDist= 1.182208e-01m    PONI= 5.394843e-02, 5.551600e-02m       rot1=0.006974  rot2= -0.003313  rot3= -0.000000 rad
    DirectBeamDist= 118.224mm       Center: x=1066.839, y=1036.336 pix      Tilt=0.442 deg  tiltPlanRotation= -154.594 deg
    Mask: None      Method: splitpixel
    Integrating LaB6_29.4keV.tif --> LaB6_29.4keV.xy,     reading: 0.022s         1D integration: 1.789s.

    [2]:

    #Let's have a look at the header of the produces ascii file:
    head -n 50 LaB6_29.4keV.xy

    # == pyFAI calibration ==
    # Distance Sample to Detector: 0.118220810284 m
    # PONI: 5.395e-02, 5.552e-02 m
    # Rotations: 0.006974 -0.003313 -0.000000 rad
    #
    # == Fit2d calibration ==
    # Distance Sample-beamCenter: 118.224 mm
    # Center: x=1066.839, y=1036.336 pix
    # Tilt: 0.442 deg  TiltPlanRot: -154.594 deg
    #
    # Detector Detector      Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline  PixelSize= 5.168e-05, 5.126e-05 m
    #    Detector has a mask: False
    #    Detector has a dark current: False
    #    detector has a flat field: False
    #
    # Wavelength: 4.21714957131e-11 m
    # Mask applied: False
    # Dark current applied: False
    # Flat field applied: False
    # Polarization factor: None
    # Normalization factor: 1.0
    #
    # Headers of the input frame:
    # {
    # "info": {},
    # "photometricInterpretation": 1,
    # "rowsPerStrip": 2048,
    # "nColumns": 2048,
    # "compression": false,
    # "sampleFormat": 3,
    # "imageDescription": "{\nEDF_DataBlockID=0.Image.Psd\nEDF_BinarySize=16777216\nEDF_HeaderSize=1536\nByteOrder=LowByteFirst\nDataType=FloatValue\nDim_1=2048\nDim_2=2048\nImage=0\nHeaderID=EH:000000:000000:000000\nSize=16777216\nmethod=max\nnframes=21\ncutoff=None\nmerged_file_00=ref_lab6_0001.edf\nmerged_file_01=ref_lab6_0002.edf\nmerged_file_02=ref_lab6_0003.edf\nmerged_file_03=ref_lab6_0004.edf\nmerged_file_04=ref_lab6_0005.edf\nmerged_file_05=ref_lab6_0006.edf\nmerged_file_06=ref_lab6_0007.edf\nmerged_file_07=ref_lab6_0008.edf\nmerged_file_08=ref_lab6_0009.edf\nmerged_file_09=ref_lab6_0010.edf\nmerged_file_10=ref_lab6_0011.edf\nmerged_file_11=ref_lab6_0012.edf\nmerged_file_12=ref_lab6_0013.edf\nmerged_file_13=ref_lab6_0014.edf\nmerged_file_14=ref_lab6_0015.edf\nmerged_file_15=ref_lab6_0016.edf\nmerged_file_16=ref_lab6_0017.edf\nmerged_file_17=ref_lab6_0018.edf\nmerged_file_18=ref_lab6_0019.edf\nmerged_file_19=ref_lab6_0020.edf\nmerged_file_20=ref_lab6_0021.edf\nfilename=LaB6_29.4keV.tif\n",
    # "nRows": 2048,
    # "colormap": null,
    # "nBits": 32,
    # "date": "Fri Mar  2 10:02:31 2018",
    # "software": "fabio.tifimage",
    # "compression_type": 1,
    # "stripOffsets": [
    # 1143
    # ],
    # "stripByteCounts": [
    # 16777216
    # ]
    # }
    # --> LaB6_29.4keV.xy
    #       2th_deg             I
      1.668855e-02    2.425633e+00
      5.006564e-02    2.804243e+00
      8.344272e-02    2.121904e+00
      1.168198e-01    2.734025e+00

    [3]:

    pyFAI-waxs -h

    usage: pyFAI-waxs [options] -p ponifile file1.edf file2.edf ...

    Azimuthal integration for powder diffraction.

    positional arguments:
      FILE                  Image files to integrate

    optional arguments:
      -h, --help            show this help message and exit
      -v, --version         show program's version number and exit
      -p PONIFILE           PyFAI parameter file (.poni)
      -n NPT, --npt NPT     Number of points in radial dimension
      -w WAVELENGTH, --wavelength WAVELENGTH
                            wavelength of the X-Ray beam in Angstrom
      -e ENERGY, --energy ENERGY
                            energy of the X-Ray beam in keV
                            (hc=12.3984197396keV.A)
      -u DUMMY, --dummy DUMMY
                            dummy value for dead pixels
      -U DELTA_DUMMY, --delta_dummy DELTA_DUMMY
                            delta dummy value
      -m MASK, --mask MASK  name of the file containing the mask image
      -d DARK, --dark DARK  name of the file containing the dark current
      -f FLAT, --flat FLAT  name of the file containing the flat field
      -P POLARIZATION_FACTOR, --polarization POLARIZATION_FACTOR
                            Polarization factor, from -1 (vertical) to +1
                            (horizontal), default is None for no correction,
                            synchrotrons are around 0.95
      --error-model ERROR_MODEL
                            Error model to use. Currently on 'poisson' is
                            implemented
      --unit UNIT           unit for the radial dimension: can be q_nm^-1, q_A^-1,
                            2th_deg, 2th_rad or r_mm
      --ext EXT             extension of the regrouped filename (.xy)
      --method METHOD       Integration method
      --multi               Average out all frame in a file before integrating
                            extracting variance, otherwise treat every single
                            frame
      --average AVERAGE     Method for averaging out: can be 'mean' (default),
                            'min', 'max' or 'median
      --do-2D               Perform 2D integration in addition to 1D

    pyFAI-waxs is the script of pyFAI that allows data reduction (azimuthal
    integration) for Wide Angle Scattering to produce X-Ray Powder Diffraction
    Pattern with output axis in 2-theta space.

One options which may be of some interest is the –do-2D which allow to perform caking.

## Azimuthal integration for small angle data

Small angle scattering usually prefer integration in a radial unit independant of the wavelength, typically \(q = \frac{4 \pi sin(2\theta/2)}{\lambda}\) in inverse nanometers (\(nm^{-1}\)).

    [4]:

    pyFAI-saxs -p LaB6_29.4keV.poni -n 1000 LaB6_29.4keV.tif

    Detector Detector        Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline  PixelSize= 5.168e-05, 5.126e-05 m
    Wavelength= 4.217150e-11m
    SampleDetDist= 1.182208e-01m    PONI= 5.394843e-02, 5.551600e-02m       rot1=0.006974  rot2= -0.003313  rot3= -0.000000 rad
    DirectBeamDist= 118.224mm       Center: x=1066.839, y=1036.336 pix      Tilt=0.442 deg  tiltPlanRotation= -154.594 deg
    Mask: None      Method: splitpixel
    Integrating LaB6_29.4keV.tif --> LaB6_29.4keV.dat,    reading: 0.022s         1D integration: 1.792s.

    [5]:

    head -n 50 LaB6_29.4keV.dat

    # == pyFAI calibration ==
    # Distance Sample to Detector: 0.118220810284 m
    # PONI: 5.395e-02, 5.552e-02 m
    # Rotations: 0.006974 -0.003313 -0.000000 rad
    #
    # == Fit2d calibration ==
    # Distance Sample-beamCenter: 118.224 mm
    # Center: x=1066.839, y=1036.336 pix
    # Tilt: 0.442 deg  TiltPlanRot: -154.594 deg
    #
    # Detector Detector      Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline  PixelSize= 5.168e-05, 5.126e-05 m
    #    Detector has a mask: False
    #    Detector has a dark current: False
    #    detector has a flat field: False
    #
    # Wavelength: 4.21714957131e-11 m
    # Mask applied: False
    # Dark current applied: False
    # Flat field applied: False
    # Polarization factor: None
    # Normalization factor: 1.0
    #
    # Headers of the input frame:
    # {
    # "info": {},
    # "photometricInterpretation": 1,
    # "rowsPerStrip": 2048,
    # "nColumns": 2048,
    # "compression": false,
    # "sampleFormat": 3,
    # "imageDescription": "{\nEDF_DataBlockID=0.Image.Psd\nEDF_BinarySize=16777216\nEDF_HeaderSize=1536\nByteOrder=LowByteFirst\nDataType=FloatValue\nDim_1=2048\nDim_2=2048\nImage=0\nHeaderID=EH:000000:000000:000000\nSize=16777216\nmethod=max\nnframes=21\ncutoff=None\nmerged_file_00=ref_lab6_0001.edf\nmerged_file_01=ref_lab6_0002.edf\nmerged_file_02=ref_lab6_0003.edf\nmerged_file_03=ref_lab6_0004.edf\nmerged_file_04=ref_lab6_0005.edf\nmerged_file_05=ref_lab6_0006.edf\nmerged_file_06=ref_lab6_0007.edf\nmerged_file_07=ref_lab6_0008.edf\nmerged_file_08=ref_lab6_0009.edf\nmerged_file_09=ref_lab6_0010.edf\nmerged_file_10=ref_lab6_0011.edf\nmerged_file_11=ref_lab6_0012.edf\nmerged_file_12=ref_lab6_0013.edf\nmerged_file_13=ref_lab6_0014.edf\nmerged_file_14=ref_lab6_0015.edf\nmerged_file_15=ref_lab6_0016.edf\nmerged_file_16=ref_lab6_0017.edf\nmerged_file_17=ref_lab6_0018.edf\nmerged_file_18=ref_lab6_0019.edf\nmerged_file_19=ref_lab6_0020.edf\nmerged_file_20=ref_lab6_0021.edf\nfilename=LaB6_29.4keV.tif\n",
    # "nRows": 2048,
    # "colormap": null,
    # "nBits": 32,
    # "date": "Fri Mar  2 10:02:31 2018",
    # "software": "fabio.tifimage",
    # "compression_type": 1,
    # "stripOffsets": [
    # 1143
    # ],
    # "stripByteCounts": [
    # 16777216
    # ]
    # }
    # --> LaB6_29.4keV.dat
    #       q_nm^-1             I
      4.278568e-02    2.427221e+00
      1.283570e-01    2.821874e+00
      2.139284e-01    2.101271e+00
      2.994998e-01    2.708885e+00

    [6]:

    pyFAI-saxs -h

    usage: pyFAI-saxs [options] -n 1000 -p ponifile file1.edf file2.edf ...

    Azimuthal integration for SAXS users.

    positional arguments:
      FILE                  Image files to integrate

    optional arguments:
      -h, --help            show this help message and exit
      -v, --version         show program's version number and exit
      -p PONIFILE           PyFAI parameter file (.poni)
      -n NPT, --npt NPT     Number of points in radial dimension
      -w WAVELENGTH, --wavelength WAVELENGTH
                            wavelength of the X-Ray beam in Angstrom
      -e ENERGY, --energy ENERGY
                            energy of the X-Ray beam in keV
                            (hc=12.3984197396keV.A)
      -u DUMMY, --dummy DUMMY
                            dummy value for dead pixels
      -U DELTA_DUMMY, --delta_dummy DELTA_DUMMY
                            delta dummy value
      -m MASK, --mask MASK  name of the file containing the mask image
      -d DARK, --dark DARK  name of the file containing the dark current
      -f FLAT, --flat FLAT  name of the file containing the flat field
      -P POLARIZATION_FACTOR, --polarization POLARIZATION_FACTOR
                            Polarization factor, from -1 (vertical) to +1
                            (horizontal), default is None for no correction,
                            synchrotrons are around 0.95
      --error-model ERROR_MODEL
                            Error model to use. Currently on 'poisson' is
                            implemented
      --unit UNIT           unit for the radial dimension: can be q_nm^-1, q_A^-1,
                            2th_deg, 2th_rad or r_mm
      --ext EXT             extension of the regrouped filename (.dat)
      --method METHOD       Integration method

    pyFAI-saxs is the SAXS script of pyFAI that allows data reduction (azimuthal
    integration) for Small Angle Scattering with output axis in q space.

## Common tool for azimuthal intration

pyFAI-integrate offers a graphical user interface for integration which exposes most options from pyFAI. Once all options have been configured in the the windows, one can either launch the processing or save the configuration into a JSON file to be able to create script with all the options.

To run pyFAI-integrate as a script, i.e. without the graphical interface, the –no-gui option is required:

    [7]:

    pyFAI-integrate --no-gui -j azimint.json -o LaB6_29.4keV.cli  LaB6_29.4keV.tif

    INFO:root:Enter, port=54321.
    INFO:root:Enter.
    start config ...
    INFO:pyFAI.worker:Detector Detector      Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline  PixelSize= 5.168e-05, 5.126e-05 m
    Wavelength= 4.217150e-11m
    SampleDetDist= 1.182208e-01m    PONI= 5.394843e-02, 5.551600e-02m       rot1=0.006974  rot2= -0.003313  rot3= -0.000000 rad
    DirectBeamDist= 118.224mm       Center: x=1066.839, y=1036.336 pix      Tilt=0.442 deg  tiltPlanRotation= -154.594 deg
    INFO:pyFAI.azimuthalIntegrator:AI.integrate1d: Resetting integrator because init
    INFO:pyFAI.opencl.processing:234.570MB are needed on device which has 12644.188MB
    INFO:pyFAI.opencl.processing:Compiling file ['kahan.cl', 'preprocess.cl', 'memset.cl', 'ocl_azim_CSR.cl'] with options -D NBINS=1000  -D NIMAGE=4194304 -D WORKGROUP_SIZE=32
    INFO:pyFAI:Processing done in 2.433s !

    [8]:

    pyFAI-integrate --help

    INFO:root:Enter, port=54321.
    INFO:root:Enter.
    usage: pyFAI-integrate [options] file1.edf file2.edf ...

    PyFAI-integrate is a graphical interface (based on Python/Qt4) to perform
    azimuthal integration on a set of files. It exposes most of the important
    options available within pyFAI and allows you to select a GPU (or an openCL
    platform) to perform the calculation on.

    positional arguments:
      FILE                  Files to be integrated

    optional arguments:
      -h, --help            show this help message and exit
      -V, --version         show program's version number and exit
      -v, --verbose         switch to verbose/debug mode
      -o OUTPUT, --output OUTPUT
                            Directory or file where to store the output data
      -f FORMAT, --format FORMAT
                            output data format (can be HDF5)
      -s SLOW, --slow-motor SLOW
                            Dimension of the scan on the slow direction (makes
                            sense only with HDF5)
      -r RAPID, --fast-motor RAPID
                            Dimension of the scan on the fast direction (makes
                            sense only with HDF5)
      --no-gui              Process the dataset without showing the user
                            interface.
      -j JSON, --json JSON  Configuration file containing the processing to be
                            done
      --monitor-name MONITOR_KEY
                            Name of the monitor in the header of each input files.
                            If defined the contribution of each input file is
                            divided by the monitor. If the header does not contain
                            or contains a wrong value, the contribution of the
                            input file is ignored. On EDF files, values from
                            'counter_pos' can accessed by using the expected
                            mnemonic. For example 'counter/bmon'.

    PyFAI-integrate saves all parameters in a .azimint.json (hidden) file. This
    JSON file is an ascii file which can be edited and used to configure online
    data analysis using the LImA plugin of pyFAI. Nota: there is bug in debian6
    making the GUI crash (to be fixed inside pyqt) http://bugs.debian.org/cgi-
    bin/bugreport.cgi?bug=697348

    [9]:

    head -n 50 LaB6_29.4keV.cli

    # == pyFAI calibration ==
    # Distance Sample to Detector: 0.118220810284 m
    # PONI: 5.395e-02, 5.552e-02 m
    # Rotations: 0.006974 -0.003313 -0.000000 rad
    #
    # == Fit2d calibration ==
    # Distance Sample-beamCenter: 118.224 mm
    # Center: x=1066.839, y=1036.336 pix
    # Tilt: 0.442 deg  TiltPlanRot: -154.594 deg
    #
    # Detector Detector      Spline= /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/F_K4320T_Cam43_30012013_distorsion.spline  PixelSize= 5.168e-05, 5.126e-05 m
    #    Detector has a mask: False
    #    Detector has a dark current: False
    #    detector has a flat field: False
    #
    # Wavelength: 4.21714957131e-11 m
    # Mask applied: False
    # Dark current applied: False
    # Flat field applied: False
    # Polarization factor: None
    # Normalization factor: 1.0
    # --> /users/kieffer/workspace-400/pyFAI/doc/source/usage/cookbook/LaB6_29.4keV.cli
    #          r_mm             I
      3.913197e-02    2.562033e+00
      1.173959e-01    2.491645e+00
      1.956599e-01    2.354185e+00
      2.739238e-01    2.858854e+00
      3.521877e-01    2.796831e+00
      4.304517e-01    2.496695e+00
      5.087156e-01    2.796695e+00
      5.869795e-01    2.742193e+00
      6.652435e-01    2.834705e+00
      7.435074e-01    2.855904e+00
      8.217714e-01    2.906582e+00
      9.000353e-01    3.021892e+00
      9.782992e-01    3.171136e+00
      1.056563e+00    3.010916e+00
      1.134827e+00    3.027250e+00
      1.213091e+00    3.020955e+00
      1.291355e+00    2.935973e+00
      1.369619e+00    3.207142e+00
      1.447883e+00    2.983296e+00
      1.526147e+00    3.007866e+00
      1.604411e+00    3.052688e+00
      1.682675e+00    2.898143e+00
      1.760939e+00    3.027955e+00
      1.839203e+00    3.147613e+00
      1.917466e+00    3.243755e+00
      1.995730e+00    3.213144e+00
      2.073994e+00    2.948822e+00

## Conclusion

This cookbook exposed the way to perfrom azimuthal integration in scripts with pyFAI using three tools: pyFAI-waxs, pyFAI-saxs and pyFAI-integrate. Using shell scripts is probably not as efficient as writing Python scripts but they may be easier to interface with batch queueing system managing compute-clusters.