edu.isi.pegasus.planner.selector.site

## Class NonJavaCallout

- java.lang.Object
- - edu.isi.pegasus.planner.selector.site.Abstract
  - - edu.isi.pegasus.planner.selector.site.AbstractPerJob
    - - edu.isi.pegasus.planner.selector.site.NonJavaCallout

All Implemented Interfaces:
  SiteSelector

    public class NonJavaCallout
    extends AbstractPerJob

This is the class that implements a call-out to a site selector which is an application or executable script. In order to use the site selector implemented by this class, the property `pegasus.selector.site` must be set to value `NonJavaCallout`.

This site selector implements a `popen()` like call to an external application that conforms to the API laid out here. The name of the application to run is specified by the property `pegasus.selector.site.path`. Its value points to a locally available application.

If the external executable requires certain environment variables to be set for execution, these can be specified in the property files, using the prefix `pegasus.selector.site.env`. The name of the environment variable is obtained by stripping the prefix. For example, to set the variable PATH to a certain value, use the following entry in your user property file:

       pegasus.selector.site.env.PATH = /usr/bin:/bin:/usr/X11R6/bin

The site selector populates the environment of the external application with the following default properties, which can be overwritten by user-specified properties:

key | value
--- | ---
PEGASUS_HOME | As set by the system
CLASSPATH | From `java.class.path`
JAVA_HOME | From `java.home`
USER | From `user.name`, if present
LOGNAME | From `user.name`, if present
HOME | From `user.home`, if present
TMP | From `java.io.tmpdir`, if present
TZ | From `user.timezone`, if present

The user can specify the environment variables, by specifying the properties with the prefix pegasus.selector.site.env. prefix. for e.g user can override the default user.name property by setting the property pegasus.selector.site.env.user.home .

The external application is invoked with one commandline argument. This argument is the name of a temporary file. The temporary file is created for each invocation anew by the site selecting caller. Being temporary, the file is deleted after the site selector returns with success. The deletion of the file is governed by the property pegasus.selector.site.keep.tmp. It can have a tristate value with the valid values being

                  ALWAYS
                  NEVER
                  ONERROR

The external application is expected to write one line to stdout. The line starts with the string `SOLUTION:`, followed by the chosen site handle. Optionally, separated by a colon, the name of a jobmanager for the site can be provided by the site selector. Two examples for successful site selections are:

       SOLUTION:mysite:my.job.mgr/jobmanager-batch
       SOLUTION:siteY

Note, these are two examples. The site selector only returns one line with the appropriate solution. If no site is found to be eligble, the poolhandle should be set to NONE by the site selector.

The temporary file is the corner stone of the communication between the site selecting caller and the external site selector. It is a collection of key-value pairs. Each pair is separated by an equals (=) sign, and stands on a line of its own. There are no multi-line values permitted.

The following pairs are generated for the siteselector temporary file:

# | key | value
--- | --- | ---
1 | version | The version of the site selector API, currently 2.0
1 | transformation | The fully-qualified definition identifier for the TR, ns::id:vs.
1 | derivation | The fully-qualified definition identifier for the DV, ns::id:vs.
1 | job.level | The job's depth in the DFS tree of the workflow DAG
1 | job.id | The job's ID, as used in the DAX file.
N | resource.id | A pool handle, followed by a whitespace, followed by a gridftp server. Typically, each gridftp server is enumerated once, so you may have multiple occurances of the same site.
M | input.lfn | An input LFN, optionally followed by a whitespace and filesize.
1 | wf.name | The label of the DAX, as found in the DAX's root element.
1 | wf.index | The DAX index, which is incremented for each partition.
1 | wf.time | The mtime of the workflow.
1 | wf.manager | The name of the workflow manager to be used, e.g. `dagman`.
1 | vo.name | unused at present, name of the virtual organization who runs this WF.
1 | vo.group | unused at present, usage not clear .

In order to detect malfunctioning site selectors, a timeout is attached with each site selector, see property `pegasus.selector.site.timeout`. By default, a site selector is given up upon after 60 s.

Version:
  $Revision$
Author:
  Karan Vahi, Jens Vöckler
See Also:
  `Runtime`, `Process`

### Field Summary

Fields
`static int` `KEEP_ALWAYS` The state denoting always to keep the temporary files.
`static int` `KEEP_NEVER` The state denoting never to keep the temporary files.
`static int` `KEEP_ONERROR` The state denoting to keep the temporary files only in case of error.
`private ADag` `mAbstractDag` The abstract DAG.
`private static java.lang.String` `mDescription` The description of the site selector.
`private java.util.Map` `mEnvVar` The map that contains the environment variables including the default ones that are set while calling out to the site selector unless they are overridden by the values set in the properties file.
`private int` `mKeepTMP` The tristate value for whether keeping the temporary files generated or not.
`private java.lang.String` `mSiteSelectorPath` The path to the site selector.
`private int` `mTimeout` The timeout value in seconds after which to timeout, in the case where the external site selector does nothing (nothing on stdout nor stderr).
`static java.lang.String` `PREFIX_PROPERTIES` The prefix of the property names that specify the environment variables that need to be set before calling out to the site selector.
`static java.lang.String` `PREFIX_TEMPORARY_FILE` The prefix to be used while creating a temporary file to pass to the external siteselector.
`static java.lang.String` `SOLUTION_PREFIX` The prefix that the site selector writes out on its stdout to designate that it is sending a solution.
`static java.lang.String` `SUFFIX_TEMPORARY_FILE` The suffix to be used while creating a temporary file to pass to the external siteselector.
`static java.lang.String` `VERSION` The version number associated with this API of non java callout site selection.

### Fields inherited from class edu.isi.pegasus.planner.selector.site.Abstract

`mBag, mLogger, mProps, mSiteStore, mTCMapper`

### Fields inherited from interface edu.isi.pegasus.planner.selector.SiteSelector

`SITE_NOT_FOUND`

### Constructor Summary

Constructors
Constructor and Description
`NonJavaCallout()` The default constructor.

### Method Summary

All Methods Static Methods Instance Methods Concrete Methods
`java.lang.String` `description()` Returns a brief description of the site selection technique implemented by this class.
`private java.lang.String[]` `getEnvArrFromMap()` Generates an array of environment variables.
`private int` `getKeepTMPValue(java.lang.String value)` Returns the int value corresponding to the string value passed.
`private java.lang.String` `getTempFilename()` Creates a temporary file and obtains its name.
`void` `initialize(PegasusBag bag)` Initializes the site selector.
`private void` `loadEnvironmentVariables()` Initializes the internal hash that collects environment variables.
`static void` `main(java.lang.String[] args)` The main program that allows you to test.
`void` `mapJob(Job job, java.util.List sites)` Calls out to the external site selector.
`void` `mapWorkflow(ADag workflow, java.util.List sites)` Maps the jobs in the workflow to the various grid sites.
`private boolean` `parseStdOut(Job job, java.lang.String s)` Extracts the chosen site from the site selector's answer.
`private java.io.File` `prepareInputFile(Job job, java.util.List pools)` Writes job knowledge into the temporary file passed to the external site selector.

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Field Detail

#### PREFIX_TEMPORARY_FILE

    public static final java.lang.String PREFIX_TEMPORARY_FILE

The prefix to be used while creating a temporary file to pass to the external siteselector.

See Also:
  Constant Field Values

#### SUFFIX_TEMPORARY_FILE

    public static final java.lang.String SUFFIX_TEMPORARY_FILE

The suffix to be used while creating a temporary file to pass to the external siteselector.

#### PREFIX_PROPERTIES

    public static final java.lang.String PREFIX_PROPERTIES

The prefix of the property names that specify the environment variables that need to be set before calling out to the site selector.

See Also:
  Constant Field Values

#### SOLUTION_PREFIX

    public static final java.lang.String SOLUTION_PREFIX

The prefix that the site selector writes out on its stdout to designate that it is sending a solution.

See Also:
  Constant Field Values

#### VERSION

    public static final java.lang.String VERSION

The version number associated with this API of non java callout site selection.

See Also:
  Constant Field Values

#### KEEP_NEVER

    public static final int KEEP_NEVER

The state denoting never to keep the temporary files.

See Also:
  Constant Field Values

#### KEEP_ONERROR

    public static final int KEEP_ONERROR

The state denoting to keep the temporary files only in case of error.

See Also:
  Constant Field Values

#### KEEP_ALWAYS

    public static final int KEEP_ALWAYS

The state denoting always to keep the temporary files.

See Also:
  Constant Field Values

#### mDescription

    private static final java.lang.String mDescription

The description of the site selector.

See Also:
  Constant Field Values

#### mEnvVar

    private java.util.Map mEnvVar

The map that contains the environment variables including the default ones that are set while calling out to the site selector unless they are overridden by the values set in the properties file.

#### mTimeout

    private int mTimeout

The timeout value in seconds after which to timeout, in the case where the external site selector does nothing (nothing on stdout nor stderr).

#### mKeepTMP

    private int mKeepTMP

The tristate value for whether keeping the temporary files generated or not.

#### mSiteSelectorPath

    private java.lang.String mSiteSelectorPath

The path to the site selector.

#### mAbstractDag

    private ADag mAbstractDag

The abstract DAG.

### Constructor Detail

#### NonJavaCallout

    public NonJavaCallout()

The default constructor.

### Method Detail

#### initialize

    public void initialize(PegasusBag bag)

Initializes the site selector.

Specified by:
  `initialize` in interface `SiteSelector`
Overrides:
  `initialize` in class `Abstract`
Parameters:
  `bag` - the bag of objects that is useful for initialization.

#### mapWorkflow

    public void mapWorkflow(ADag workflow,
                            java.util.List sites)

Maps the jobs in the workflow to the various grid sites. The jobs are mapped by setting the site handle for the jobs.

Specified by:
  `mapWorkflow` in interface `SiteSelector`
Overrides:
  `mapWorkflow` in class `AbstractPerJob`
Parameters:
  `workflow` - the workflow.
  `sites` - the list of `String` objects representing the execution sites that can be used.

#### description

    public java.lang.String description()

Returns a brief description of the site selection technique implemented by this class.

Returns:
  a self-description of this site selector.

#### mapJob

    public void mapJob(Job job,
                       java.util.List sites)

Calls out to the external site selector. The method converts a `Job` object into an API-compliant temporary file. The file's name is provided as single commandline argument to the site selector executable when it is invoked. The executable, representing the external site selector, provides its answer on stdout. The answer is captures, and returned.

Specified by:
  `mapJob` in class `AbstractPerJob`
Parameters:
  `job` - is a representation of the DAX compute job whose site of execution need to be determined.
  `sites` - the list of `String` objects representing the execution sites that can be used. FIXME: Some site selector return an empty string on failures. Also: NONE could be a valid site name.
See Also:
  `org.griphyn.cPlanner.classes.Job`

#### prepareInputFile

    private java.io.File prepareInputFile(Job job,
                                          java.util.List pools)

Writes job knowledge into the temporary file passed to the external site selector. The job knowledge derives from the contents of the DAX job's `Job` record, and the a list of site candidates. The format of the file is laid out in the class's introductory documentation.

Parameters:
  `job` - is a representation of the DAX compute job whose site of execution need to be determined.
  `pools` - is a list of site candidates. The items of the list are `String` objects.
Returns:
  the temporary input file was successfully prepared. A value of `null` implies that an error occured while writing the file.
See Also:
  `getTempFilename()`

#### parseStdOut

    private boolean parseStdOut(Job job,
                                java.lang.String s)

Extracts the chosen site from the site selector's answer. Parses the stdout sent by the selector, to see, if the execution pool and the jobmanager were sent or not.

Parameters:
  `job` - the job that has to be mapped.
  `s` - is the stdout received from the site selector.
Returns:
  boolean indicating if the stdout was succesfully parsed and job populated.

#### getTempFilename

    private java.lang.String getTempFilename()

Creates a temporary file and obtains its name. This method returns the absolute path to a temporary file in the system's TEMP directory. The file is guarenteed to be unique for the current invocation of the virtual machine. FIXME: However, since we return a filename and not an opened file, race conditions are still possible.

Returns:
  the absolute path of a newly created temporary file.

#### loadEnvironmentVariables

    private void loadEnvironmentVariables()

Initializes the internal hash that collects environment variables. These variables are set up to run the external helper application. Environment variables come from two source.

1. Default environment variables, fixed, hard-coded.
2. User environment variables, from properties.

#### getEnvArrFromMap

    private java.lang.String[] getEnvArrFromMap()

Generates an array of environment variables. The variables are kept in an internal map. Converts the environment variables in the map to the array format.

Returns:
  array of enviroment variables set, or `null` if the map is empty.
See Also:
  `loadEnvironmentVariables()`

#### getKeepTMPValue

    private int getKeepTMPValue(java.lang.String value)

Returns the int value corresponding to the string value passed.

Parameters:
  `value` - the string value for keeping the temporary files.
Returns:
  the corresponding int value.
See Also:
  `KEEP_ALWAYS`, `KEEP_NEVER`, `KEEP_ONERROR`

#### main

    public static void main(java.lang.String[] args)

The main program that allows you to test. FIXME: Test programs should have prefix Test.....java

Parameters:
  `args` - the arguments