See also

- @product in the Ruffus Manual
- Decorators for more decorators

# @product( input, filter, [input2, filter2, …], output, [extras,…] )

> Purpose:

> > Generates the Cartesian product, i.e. all vs all comparisons, between multiple sets of input (e.g. A B C D, and X Y Z),

> > The effect is analogous to the python itertools function of the same name, i.e. a nested for loop.

> >     >>> from itertools import product
> >     >>> # product('ABC', 'XYZ') --> AX AY AZ BX BY BZ CX CY CZ
> >     >>> [ "".join(a) for a in product('ABC', 'XYZ')]
> >     ['AX', 'AY', 'AZ', 'BX', 'BY', 'BZ', 'CX', 'CY', 'CZ']

> > Only out of date tasks (comparing input and output files) will be run

> > output file names and strings in the extra parameters are generated by string replacement via the formatter() filter from the input. This can be, for example, a list of file names or the output of up stream tasks. . The replacement strings require an extra level of nesting to refer to parsed components.

> > > 1. The first level refers to which set in each tuple of input.
> > > 2. The second level refers to which input file in any particular set of input.

> > This will be clear in the following example:

> Example:

> > Calculates the @product of A,B and P,Q and X, Y files

> > If input is three sets of file names

> > >         set1 = [ 'a.start',                         # 0
> > >                  'b.start'])

> > >         set2 = [ 'p.start',                         # 1
> > >                  'q.start'])

> > >         set3 = [ ['x.1_start', 'x.2_start'],        # 2
> > >                  ['y.1_start', 'y.2_start'] ]

> > The first job of:

> > >     @product( input  = set1, filter  = formatter(),
> > >               input2 = set2, filter2 = formatter(),
> > >               input3 = set2, filter3 = formatter(),
> > >               ...)

> > Will be

> > >     # One from each set
> > >     ['a.start']
> > >     # versus
> > >     ['p.start']
> > >     # versus
> > >     ['x.1_start', 'x.2_start'],

> > First level of nesting (one list of files from each set):

> >       ['a.start']                 # [0]
> >       ['p.start']                 # [1]
> >       ['x.1_start', 'x.2_start'], # [2]

> > Second level of nesting (one file):

> >       'a.start'                   # [0][0]
> >       'p.start'                   # [1][0]
> >       'x.1_start'                 # [2][0]

> > Parse filename without suffix

> >       'a'                         # {basename[0][0]}
> >       'p'                         # {basename[1][0]}
> >       'x'                         # {basename[2][0]}

> > Python code:

> > >     from ruffus import *
> > >     from ruffus.combinatorics import *

> > >     #   Three sets of initial files
> > >     @originate([ 'a.start', 'b.start'])
> > >     def create_initial_files_ab(output_file):
> > >         with open(output_file, "w") as oo: pass

> > >     @originate([ 'p.start', 'q.start'])
> > >     def create_initial_files_pq(output_file):
> > >         with open(output_file, "w") as oo: pass

> > >     @originate([ ['x.1_start', 'x.2_start'],
> > >                  ['y.1_start', 'y.2_start'] ])
> > >     def create_initial_files_xy(output_files):
> > >         for o in output_files:
> > >             with open(o, "w") as oo: pass

> > >     #   @product
> > >     @product(   create_initial_files_ab,        # Input
> > >                 formatter("(.start)$"),         # match input file set # 1

> > >                 create_initial_files_pq,        # Input
> > >                 formatter("(.start)$"),         # match input file set # 2

> > >                 create_initial_files_xy,        # Input
> > >                 formatter("(.start)$"),         # match input file set # 3

> > >                 "{path[0][0]}/"                 # Output Replacement string
> > >                 "{basename[0][0]}_vs_"          #
> > >                 "{basename[1][0]}_vs_"          #
> > >                 "{basename[2][0]}.product",     #

> > >                 "{path[0][0]}",                 # Extra parameter: path for 1st set of files, 1st file name

> > >                 ["{basename[0][0]}",            # Extra parameter: basename for 1st set of files, 1st file name
> > >                  "{basename[1][0]}",            #                               2nd
> > >                  "{basename[2][0]}",            #                               3rd
> > >                  ])
> > >     def product_task(input_file, output_parameter, shared_path, basenames):
> > >         print "# basenames      = ", " ".join(basenames)
> > >         print "input_parameter  = ", input_file
> > >         print "output_parameter = ", output_parameter, "\n"

> > >     #
> > >     #       Run
> > >     #
> > >     #pipeline_printout(verbose=6)
> > >     pipeline_run(verbose=0)

> > This results in:

> > >     >>> pipeline_run(verbose=0)

> > >     # basenames      =  a p x
> > >     input_parameter  =  ('a.start', 'p.start', 'x.start')
> > >     output_parameter =  /home/lg/temp/a_vs_p_vs_x.product

> > >     # basenames      =  a p y
> > >     input_parameter  =  ('a.start', 'p.start', 'y.start')
> > >     output_parameter =  /home/lg/temp/a_vs_p_vs_y.product

> > >     # basenames      =  a q x
> > >     input_parameter  =  ('a.start', 'q.start', 'x.start')
> > >     output_parameter =  /home/lg/temp/a_vs_q_vs_x.product

> > >     # basenames      =  a q y
> > >     input_parameter  =  ('a.start', 'q.start', 'y.start')
> > >     output_parameter =  /home/lg/temp/a_vs_q_vs_y.product

> > >     # basenames      =  b p x
> > >     input_parameter  =  ('b.start', 'p.start', 'x.start')
> > >     output_parameter =  /home/lg/temp/b_vs_p_vs_x.product

> > >     # basenames      =  b p y
> > >     input_parameter  =  ('b.start', 'p.start', 'y.start')
> > >     output_parameter =  /home/lg/temp/b_vs_p_vs_y.product

> > >     # basenames      =  b q x
> > >     input_parameter  =  ('b.start', 'q.start', 'x.start')
> > >     output_parameter =  /home/lg/temp/b_vs_q_vs_x.product

> > >     # basenames      =  b q y
> > >     input_parameter  =  ('b.start', 'q.start', 'y.start')
> > >     output_parameter =  /home/lg/temp/b_vs_q_vs_y.product

> Parameters:

> - input = tasks_or_file_names
>     can be a:
>     1. Task / list of tasks.
>          File names are taken from the output of the specified task(s)
>     2. (Nested) list of file name strings.
>          File names containing `*[]?` will be expanded as a glob.
>            E.g.:`"a.*" => "a.1", "a.2"`

> - filter = formater(…)
>     a formatter indicator object containing optionally a python regular expression (re).

Additional input and filter as needed:

> - input2 = tasks_or_file_names
> - filter2 = formater(…)

> - output = output
>     Specifies the resulting output file name(s) after string substitution

> - extras = extras

>     Any extra parameters are passed verbatim to the task function

>     If you are using named parameters, these can be passed as a list, i.e. `extras= [...]`

>     Any extra parameters are consumed by the task function and not forwarded further down the pipeline.