edu.isi.pegasus.planner.refiner

## Class DeployWorkerPackage

- java.lang.Object
- - edu.isi.pegasus.planner.refiner.Engine
  - - edu.isi.pegasus.planner.refiner.DeployWorkerPackage

    public class DeployWorkerPackage
    extends Engine

The refiner that is responsible for adding - setup nodes that deploy a worker package on each deployment site at start of workflow execution - cleanup nodes that undeploy a worker package on each deployment site at end workflow execution

Version:
  $Revision: 538 $
Author:
  Karan Vahi, Gaurang Mehta

### Field Summary

Fields
`static java.lang.String` `BASE_BUILD_DIRECTORY_URL` The base directory URL for the builds.
`static java.lang.String` `CLEANUP_PREFIX` Constant suffix for the names of the deployment nodes.
`static java.lang.String` `COMPLETE_TRANSFORMATION_NAME` The complete TC name for pegasus worker package.
`static java.lang.String` `COMPLETE_UNTAR_TRANSFORMATION_NAME` The complete TC name for untar.
`static java.lang.String` `DEFAULT_REFINER` The default transfer refiner name.
`static java.lang.String` `DEPLOY_WORKER_PREFIX` Constant suffix for the names of the deployment nodes.
`static java.lang.String` `DERIVATION_NAME` The logical name of the transformation for the worker package
`static java.lang.String` `DERIVATION_NAMESPACE` The derivation namespace for the worker package.
`static java.lang.String` `DERIVATION_VERSION` The version number for the derivations for worker package.
`protected ADag` `mCurrentDag` It is a reference to the Concrete Dag so far.
`protected java.util.Map<java.lang.String,FileTransfer>` `mFTMap` The FileTransfer map indexed by site id.
`protected java.lang.String` `mJobPrefix` The job prefix that needs to be applied to the job file basenames.
`protected java.util.Map<java.lang.String,java.lang.Boolean>` `mLocalTransfers` Map that indicates whether we need local setup transfer jobs for a site or not.
`private static java.util.Map<SysInfo.OS,java.lang.String>` `mOSToNMIOSReleaseAndVersion` The map storing OS to corresponding NMI OS platforms.
`private static java.util.regex.Pattern` `mPattern` Stores compiled patterns at first use, quasi-Singleton.
`private static java.lang.String` `mRegexExpression` Store the regular expressions necessary to parse the basename from the worker package url to retrieve the version of pegasus.
`protected Implementation` `mSetupTransferImplementation` The transfer implementation to be used for staging in the data as part of setup job.
`protected java.util.Map<java.lang.String,java.lang.String>` `mSiteToPegasusHomeMap` Maps a site to the the directory where the pegasus worker package has been untarred during workflow execution.
`private static java.util.Set<java.lang.String>` `mSupportedOSReleaseVersions` A set of supported OS release and versions that our build process builds for.
`protected boolean` `mTransferWorkerPackage` Boolean indicating whether user wants the worker package to be transferred or not.
`protected java.lang.String` `mUserSpecifiedSourceLocation` The user specified location from where to stage the worker packages.
`protected boolean` `mUseUserSpecifiedSourceLocation` Boolean indicating whether to use the user specified location or not
`protected boolean` `mWorkerNodeExecution` Boolean indicating worker node execution.
`static java.lang.String` `PACKAGE_NAME` The name of the package in which all the implementing classes are.
`static java.lang.String` `PATH_VALUE` The path to be set for create dir jobs.
`static java.lang.String` `PEGASUS_VERSION` The version of pegasus matching the planner.
`static java.lang.String[][]` `PEGASUS_WORKER_EXECUTABLES` Array storing the names of the executables in the $PEGASUS_HOME/bin directory Associates the transformation name with the executable basenames
`static java.lang.String` `POSTSCRIPT_ARGUMENTS_FOR_ONLY_ROTATING_LOG_FILE` The arguments for pegasus-exitcode when you only want the log files to be rotated.
`static java.lang.String` `REFINER_NAME` The name of the refiner for purposes of error logging
`static java.lang.String` `TRANSFORMATION_NAME` The logical name of the worker package
`static java.lang.String` `TRANSFORMATION_NAMESPACE` The transformation namespace for the worker package
`static java.lang.String` `TRANSFORMATION_VERSION` The version number for the worker package.
`static java.lang.String` `UNTAR_DERIVATION_NAME` The logical name of the transformation for the untar job.
`static java.lang.String` `UNTAR_DERIVATION_NAMESPACE` The derivation namespace for the untar job.
`static java.lang.String` `UNTAR_DERIVATION_VERSION` The version number for the derivations for untar job.
`static java.lang.String` `UNTAR_PREFIX` Constant suffix for the names of the deployment nodes.
`static java.lang.String` `UNTAR_TRANSFORMATION_NAME` The logical name of the worker package
`static java.lang.String` `UNTAR_TRANSFORMATION_NAMESPACE` The transformation namespace for the worker package
`static java.lang.String` `UNTAR_TRANSFORMATION_VERSION` The version number for the worker package.

### Fields inherited from class edu.isi.pegasus.planner.refiner.Engine

`mBag, mLogger, mLogMsg, mOutputPool, mPoolFile, mPOptions, mProps, mRLIUrl, mSiteStore, mTCFile, mTCHandle, mTCMode, REGISTRATION_UNIVERSE, TRANSFER_UNIVERSE`

### Constructor Summary

Constructors
Constructor and Description
`DeployWorkerPackage(PegasusBag bag)` A pratically nothing constructor !

### Method Summary

All Methods Static Methods Instance Methods Concrete Methods
`ADag` `addCleanupNodesForWorkerPackage(ADag dag)` Adds cleanup nodes in the workflow for sites specified.
`private TransformationCatalogEntry` `addDefaultTCEntry(java.lang.String site, java.lang.String pegasusHome, SysInfo sysinfo, boolean useFullPath, java.lang.String name, java.lang.String executable)` Returns a default TC entry to be used in case entry is not found in the transformation catalog.
`protected TransformationCatalogEntry` `addDefaultTCEntryForPegasusWebsite(java.lang.String site, java.lang.String name)` Returns a default TC entry for the pegasus site.
`ADag` `addSetupNodes(ADag dag)` Adds a setup node per execution site in the workflow that will stage the worker node executables to the workdirectory on the sites the workflow has been scheduled to.
`private ADag` `addSetupNodesWithoutUntarNodes(ADag dag, java.util.Set<java.lang.String> deploymentSites)` Adds only the stage worker nodes to the workflow.
`private ADag` `addSetupNodesWithUntarNodes(ADag dag, java.util.Set<java.lang.String> deploymentSites)` Adds untar nodes to the workflow, in addition to the stage worker nodes
`protected java.lang.String` `constructDefaultURLToPegasusWorkerPackage(java.lang.String name, SysInfo sysinfo)` Constructs the default URL's for the pegasus worker package.
`private TransformationCatalogEntry` `defaultUntarTCEntry(SiteCatalogEntry site)` Returns a default TC entry to be used in case entry is not found in the transformation catalog.
`protected java.lang.String` `getBasename(java.lang.String url)` Returns the basename of the URL using substring.
`protected java.lang.String` `getCleanupJobname(ADag dag, java.lang.String site)` It returns the name of the untar job, that is to be assigned.
`protected java.lang.String` `getDeployJobName(ADag dag, java.lang.String site, boolean localTransfer)` It returns the name of the deployment job, that is to be assigned.
`protected java.util.Set` `getDeploymentSites(ADag dag)` Retrieves the sites for which the deployment jobs need to be created.
`protected java.lang.String` `getRootDirectoryNameForPegasus(java.lang.String url)` Does regex magic to figure out the version of pegasus from the url, and use it to construct the name of pegasus directory, when worker package is untarred.
`private java.lang.String` `getStagingSite(java.lang.String site)` Returns the staging site for a particular execution site.
`protected java.lang.String` `getUntarJobName(ADag dag, java.lang.String site)` It returns the name of the untar job, that is to be assigned.
`void` `initialize(ADag scheduledDAG)` Initialize with the scheduled graph.
`static DeployWorkerPackage` `loadDeployWorkerPackage(PegasusBag bag)` Loads the implementing class corresponding to the mode specified by the user at runtime.
`protected Job` `makeUntarJob(java.lang.String site, java.lang.String jobName, java.lang.String wpBasename)` It creates a untar job , that untars the worker package that is staged by the setup transfer job.
`private static java.util.Map<SysInfo.OS,java.lang.String>` `osToOSReleaseAndVersion()` Maps each to OS to a specific OS release for purposes of picking up the correct worker package for a site.
`boolean` `runTransferOnLocalSite(Refiner refiner, java.lang.String site, java.lang.String destinationURL, int type)` Returns whether to run a transfer job on local site or not.
`private static java.util.Set<java.lang.String>` `supportedOSReleaseAndVersions()` A set of OS release and version combinations for which our build processes build Pegasus binaries.

### Methods inherited from class edu.isi.pegasus.planner.refiner.Engine

`addVector, appendArrayList, complainForHeadNodeURLPrefix, complainForHeadNodeURLPrefix, loadProperties, printVector, stringInList, stringInPegVector, stringInVector, vectorToString`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Field Detail

#### DEPLOY_WORKER_PREFIX

    public static final java.lang.String DEPLOY_WORKER_PREFIX

Constant suffix for the names of the deployment nodes.

See Also:
  Constant Field Values

#### UNTAR_PREFIX

    public static final java.lang.String UNTAR_PREFIX

Constant suffix for the names of the deployment nodes.

See Also:
  Constant Field Values

#### CLEANUP_PREFIX

    public static final java.lang.String CLEANUP_PREFIX

Constant suffix for the names of the deployment nodes.

See Also:
  Constant Field Values

#### POSTSCRIPT_ARGUMENTS_FOR_ONLY_ROTATING_LOG_FILE

    public static final java.lang.String POSTSCRIPT_ARGUMENTS_FOR_ONLY_ROTATING_LOG_FILE

The arguments for pegasus-exitcode when you only want the log files to be rotated.

See Also:
  Constant Field Values

#### PEGASUS_WORKER_EXECUTABLES

    public static final java.lang.String[][] PEGASUS_WORKER_EXECUTABLES

Array storing the names of the executables in the $PEGASUS_HOME/bin directory Associates the transformation name with the executable basenames

#### mRegexExpression

    private static final java.lang.String mRegexExpression

Store the regular expressions necessary to parse the basename from the worker package url to retrieve the version of pegasus.

See Also:
  Constant Field Values

#### PATH_VALUE

    public static final java.lang.String PATH_VALUE

The path to be set for create dir jobs.

See Also:
  Constant Field Values

#### DEFAULT_REFINER

    public static final java.lang.String DEFAULT_REFINER

The default transfer refiner name.

See Also:
  Constant Field Values

#### TRANSFORMATION_NAMESPACE

    public static final java.lang.String TRANSFORMATION_NAMESPACE

The transformation namespace for the worker package

See Also:
  Constant Field Values

#### TRANSFORMATION_NAME

    public static final java.lang.String TRANSFORMATION_NAME

The logical name of the worker package

See Also:
  Constant Field Values

#### TRANSFORMATION_VERSION

    public static final java.lang.String TRANSFORMATION_VERSION

The version number for the worker package.

#### UNTAR_TRANSFORMATION_NAMESPACE

    public static final java.lang.String UNTAR_TRANSFORMATION_NAMESPACE

The transformation namespace for the worker package

#### UNTAR_TRANSFORMATION_NAME

    public static final java.lang.String UNTAR_TRANSFORMATION_NAME

The logical name of the worker package

See Also:
  Constant Field Values

#### UNTAR_TRANSFORMATION_VERSION

    public static final java.lang.String UNTAR_TRANSFORMATION_VERSION

The version number for the worker package.

#### COMPLETE_UNTAR_TRANSFORMATION_NAME

    public static final java.lang.String COMPLETE_UNTAR_TRANSFORMATION_NAME

The complete TC name for untar.

#### COMPLETE_TRANSFORMATION_NAME

    public static final java.lang.String COMPLETE_TRANSFORMATION_NAME

The complete TC name for pegasus worker package.

#### DERIVATION_NAMESPACE

    public static final java.lang.String DERIVATION_NAMESPACE

The derivation namespace for the worker package.

See Also:
  Constant Field Values

#### DERIVATION_NAME

    public static final java.lang.String DERIVATION_NAME

The logical name of the transformation for the worker package

See Also:
  Constant Field Values

#### DERIVATION_VERSION

    public static final java.lang.String DERIVATION_VERSION

The version number for the derivations for worker package.

See Also:
  Constant Field Values

#### UNTAR_DERIVATION_NAMESPACE

    public static final java.lang.String UNTAR_DERIVATION_NAMESPACE

The derivation namespace for the untar job.

#### UNTAR_DERIVATION_NAME

    public static final java.lang.String UNTAR_DERIVATION_NAME

The logical name of the transformation for the untar job.

See Also:
  Constant Field Values

#### UNTAR_DERIVATION_VERSION

    public static final java.lang.String UNTAR_DERIVATION_VERSION

The version number for the derivations for untar job.

See Also:
  Constant Field Values

#### PACKAGE_NAME

    public static final java.lang.String PACKAGE_NAME

The name of the package in which all the implementing classes are.

See Also:
  Constant Field Values

#### BASE_BUILD_DIRECTORY_URL

    public static final java.lang.String BASE_BUILD_DIRECTORY_URL

The base directory URL for the builds.

See Also:
  Constant Field Values

#### PEGASUS_VERSION

    public static final java.lang.String PEGASUS_VERSION

The version of pegasus matching the planner.

#### REFINER_NAME

    public static final java.lang.String REFINER_NAME

The name of the refiner for purposes of error logging

See Also:
  Constant Field Values

#### mPattern

    private static java.util.regex.Pattern mPattern

Stores compiled patterns at first use, quasi-Singleton.

#### mOSToNMIOSReleaseAndVersion

    private static java.util.Map<SysInfo.OS,java.lang.String> mOSToNMIOSReleaseAndVersion

The map storing OS to corresponding NMI OS platforms.

#### mSupportedOSReleaseVersions

    private static java.util.Set<java.lang.String> mSupportedOSReleaseVersions

A set of supported OS release and versions that our build process builds for.

#### mCurrentDag

    protected ADag mCurrentDag

It is a reference to the Concrete Dag so far.

#### mJobPrefix

    protected java.lang.String mJobPrefix

The job prefix that needs to be applied to the job file basenames.

#### mSetupTransferImplementation

    protected Implementation mSetupTransferImplementation

The transfer implementation to be used for staging in the data as part of setup job.

#### mFTMap

    protected java.util.Map<java.lang.String,FileTransfer> mFTMap

The FileTransfer map indexed by site id.

#### mLocalTransfers

    protected java.util.Map<java.lang.String,java.lang.Boolean> mLocalTransfers

Map that indicates whether we need local setup transfer jobs for a site or not.

#### mSiteToPegasusHomeMap

    protected java.util.Map<java.lang.String,java.lang.String> mSiteToPegasusHomeMap

Maps a site to the the directory where the pegasus worker package has been untarred during workflow execution.

#### mUserSpecifiedSourceLocation

    protected java.lang.String mUserSpecifiedSourceLocation

The user specified location from where to stage the worker packages.

#### mUseUserSpecifiedSourceLocation

    protected boolean mUseUserSpecifiedSourceLocation

Boolean indicating whether to use the user specified location or not

#### mTransferWorkerPackage

    protected boolean mTransferWorkerPackage

Boolean indicating whether user wants the worker package to be transferred or not.

#### mWorkerNodeExecution

    protected boolean mWorkerNodeExecution

Boolean indicating worker node execution.

### Constructor Detail

#### DeployWorkerPackage

    public DeployWorkerPackage(PegasusBag bag)

A pratically nothing constructor !

Parameters:
  `bag` - bag of initialization objects

### Method Detail

#### osToOSReleaseAndVersion

    private static java.util.Map<SysInfo.OS,java.lang.String> osToOSReleaseAndVersion()

Maps each to OS to a specific OS release for purposes of picking up the correct worker package for a site. The mapping is to be kept consistent with the NMI builds for the releases.

Returns:
  map

#### supportedOSReleaseAndVersions

    private static java.util.Set<java.lang.String> supportedOSReleaseAndVersions()

A set of OS release and version combinations for which our build processes build Pegasus binaries.

Returns:

#### loadDeployWorkerPackage

    public static DeployWorkerPackage loadDeployWorkerPackage(PegasusBag bag)
                                                       throws FactoryException

Loads the implementing class corresponding to the mode specified by the user at runtime.

Parameters:
  `bag` - bag of initialization objects
Returns:
  instance of a DeployWorkerPackage implementation
Throws:
  `FactoryException` - that nests any error that might occur during the instantiation of the implementation.

#### initialize

    public void initialize(ADag scheduledDAG)

Initialize with the scheduled graph. Results in the appropriate population of the transformation catalog with pegasus-worker executables.

Parameters:
  `scheduledDAG` - the scheduled workflow.

#### runTransferOnLocalSite

    public boolean runTransferOnLocalSite(Refiner refiner,
                                          java.lang.String site,
                                          java.lang.String destinationURL,
                                          int type)

Returns whether to run a transfer job on local site or not.

Parameters:
  `site` - the site handle associated with the destination URL.
  `destURL` - the destination URL
  `type` - the type of transfer job for which the URL is being constructed.
Returns:
  true indicating if the associated transfer job should run on local site or not.

#### getRootDirectoryNameForPegasus

    protected java.lang.String getRootDirectoryNameForPegasus(java.lang.String url)

Does regex magic to figure out the version of pegasus from the url, and use it to construct the name of pegasus directory, when worker package is untarred.

Parameters:
  `url` - the url.
Returns:
  basename for pegasus directory

#### addSetupNodes

    public ADag addSetupNodes(ADag dag)

Adds a setup node per execution site in the workflow that will stage the worker node executables to the workdirectory on the sites the workflow has been scheduled to.

Parameters:
  `dag` - the scheduled workflow.
Returns:
  the workflow with setup jobs added

#### addSetupNodesWithUntarNodes

    private ADag addSetupNodesWithUntarNodes(ADag dag,
                                             java.util.Set<java.lang.String> deploymentSites)

Adds untar nodes to the workflow, in addition to the stage worker nodes

Parameters:
  `dag` - the dag
  `deploymentSites` - the sites for which the worker package has to be deployed
Returns:
  the workflow in the graph representation with the nodes added.

#### addSetupNodesWithoutUntarNodes

    private ADag addSetupNodesWithoutUntarNodes(ADag dag,
                                                java.util.Set<java.lang.String> deploymentSites)

Adds only the stage worker nodes to the workflow. This is used when Pegasus Lite is used to launch the jobs on the execution sites.

Parameters:
  `dag` - the dag
  `deploymentSites` - the sites for which the worker package has to be deployed
Returns:
  the workflow in the graph representation with the nodes added.

#### addCleanupNodesForWorkerPackage

    public ADag addCleanupNodesForWorkerPackage(ADag dag)

Adds cleanup nodes in the workflow for sites specified.

Parameters:
  `dag` - the workflow
Returns:
  workflow with cleanup jobs added

#### getDeploymentSites

    protected java.util.Set getDeploymentSites(ADag dag)

Retrieves the sites for which the deployment jobs need to be created.

Parameters:
  `dag` - the dag on which the jobs need to execute.
Returns:
  a Set containing a list of siteID's of the sites where the dag has to be run.

#### getDeployJobName

    protected java.lang.String getDeployJobName(ADag dag,
                                                java.lang.String site,
                                                boolean localTransfer)

It returns the name of the deployment job, that is to be assigned. The name takes into account the workflow name while constructing it, as that is thing that can guarentee uniqueness of name in case of deferred planning.

Parameters:
  `dag` - the workflow so far.
  `site` - the execution pool for which the create directory job is responsible.
  `localTransfer` - whether the transfer needs to run locally or not.
Returns:
  String corresponding to the name of the job.

#### getUntarJobName

    protected java.lang.String getUntarJobName(ADag dag,
                                               java.lang.String site)

It returns the name of the untar job, that is to be assigned. The name takes into account the workflow name while constructing it, as that is thing that can guarentee uniqueness of name in case of deferred planning.

Parameters:
  `dag` - the workflow so far.
  `site` - the execution pool for which the create directory job is responsible.
Returns:
  String corresponding to the name of the job.

#### getCleanupJobname

    protected java.lang.String getCleanupJobname(ADag dag,
                                                 java.lang.String site)

It returns the name of the untar job, that is to be assigned. The name takes into account the workflow name while constructing it, as that is thing that can guarentee uniqueness of name in case of deferred planning.

Parameters:
  `dag` - the workflow so far.
  `site` - the execution pool for which the create directory job is responsible.
Returns:
  String corresponding to the name of the job.

#### makeUntarJob

    protected Job makeUntarJob(java.lang.String site,
                               java.lang.String jobName,
                               java.lang.String wpBasename)

It creates a untar job , that untars the worker package that is staged by the setup transfer job.

Parameters:
  `site` - the execution pool for which the create dir job is to be created.
  `jobName` - the name that is to be assigned to the job.
  `wpBasename` - the basename of the worker package that is staged to remote site.
Returns:
  create dir job.

#### addDefaultTCEntry

    private TransformationCatalogEntry addDefaultTCEntry(java.lang.String site,
                                                         java.lang.String pegasusHome,
                                                         SysInfo sysinfo,
                                                         boolean useFullPath,
                                                         java.lang.String name,
                                                         java.lang.String executable)

Returns a default TC entry to be used in case entry is not found in the transformation catalog. It also attempts to add the transformation catalog entry to the underlying TC store.

Parameters:
  `site` - the site for which the default entry is required.
  `pegasusHome` - the path to deployed worker package
  `sysinfo` - the system information of that site.
  `useFullPath` - boolean indicating whether to use just the basename or the full path
  `name` - the logical name of the transformation
  `executable` - the basename of the executable
Returns:
  the default entry.

#### addDefaultTCEntryForPegasusWebsite

    protected TransformationCatalogEntry addDefaultTCEntryForPegasusWebsite(java.lang.String site,
                                                                            java.lang.String name)

Returns a default TC entry for the pegasus site. The entry points to the http webserver on the pegasus website. It also attempts to add the transformation catalog entry to the TC store.

Parameters:
  `site` - the execution site for which we need a matching static binary.
  `name` - logical name of the transformation
Returns:
  the default entry.

#### constructDefaultURLToPegasusWorkerPackage

    protected java.lang.String constructDefaultURLToPegasusWorkerPackage(java.lang.String name,
                                                                         SysInfo sysinfo)

Constructs the default URL's for the pegasus worker package. If the user has not specified the URL to the source directory in Pegaus Properties then the URL constructed points to the pegasus website. The version of Pegasus retrieved is the one against which the planner is executing.

Parameters:
  `name` - the logical name of the executable, usually worker|binary.
  `sysinfo` - the sysinfo for which the path is required.
Returns:
  url

#### defaultUntarTCEntry

    private TransformationCatalogEntry defaultUntarTCEntry(SiteCatalogEntry site)

Returns a default TC entry to be used in case entry is not found in the transformation catalog.

Parameters:
  `site` - the site for which the default entry is required.
Returns:
  the default entry.

#### getBasename

    protected java.lang.String getBasename(java.lang.String url)

Returns the basename of the URL using substring.

Parameters:
  `url` -
Returns:
  basename

#### getStagingSite

    private java.lang.String getStagingSite(java.lang.String site)

Returns the staging site for a particular execution site. If worker node execution is enabled, then the staging site is the submit directory for the workflow on the local sit.e

Parameters:
  `site` - the execution site.
Returns:
  the staging site