edu.isi.pegasus.planner.refiner

## Class RemoveDirectory

- java.lang.Object
- - edu.isi.pegasus.planner.refiner.Engine
  - - edu.isi.pegasus.planner.refiner.RemoveDirectory

    public class RemoveDirectory
    extends Engine

Ends up creating a cleanup dag that deletes the remote directories that were created by the create dir jobs. The cleanup dag is generated in a sub directory from the main directory containing the submit files of the dag. The dag consists of independant jobs, with each job responsible for deleting directory for a execution pool. The current way of generating the dag is tied to the fact, that the directories in which a job are executed is tied to the pool not the job itself.

Version:
  $Revision$
Author:
  Karan Vahi
See Also:
  `CreateDirectory`

### Field Summary

Fields
`static java.lang.String` `CLEANUP_DAG_PREFIX` The prefix that is attached to the name of the dag for which the cleanup Dag is being generated, to generate the name of the cleanup Dag.
`static java.lang.String` `CLEANUP_PREFIX` Constant suffix for the names of the deployment nodes.
`static java.lang.String` `DERIVATION_NAME` The logical name of the transformation that removes directories on the remote execution pools.
`static java.lang.String` `DERIVATION_NAMESPACE` The derivation namespace for the create dir jobs.
`static java.lang.String` `DERIVATION_VERSION` The version number for the derivations for create dir jobs.
`private ADag` `mConcDag` The concrete dag so far, for which the clean up dag needs to be generated.
`protected java.lang.String` `mJobPrefix` The job prefix that needs to be applied to the job file basenames.
`private java.lang.String` `mSubmitDirectory` The submit directory for the workflow.
`private boolean` `mTransferFromSubmitHost` Boolean indicating whether we need to transfer dirmanager from the submit host.
`static java.lang.String` `REMOVE_DIR_EXECUTABLE_BASENAME` The basename of the pegasus dirmanager executable.
`static java.lang.String` `REMOVE_DIR_SUFFIX` Constant suffix for the names of the remote directory nodes.
`static java.lang.String` `TRANSFORMATION_NAME` The logical name of the transformation that removes directories on the remote execution pools.
`static java.lang.String` `TRANSFORMATION_NAMESPACE` The transformation namespace for the create dir jobs.
`static java.lang.String` `TRANSFORMATION_VERSION` The version number for the derivations for create dir jobs.

### Fields inherited from class edu.isi.pegasus.planner.refiner.Engine

`mBag, mLogger, mLogMsg, mOutputPool, mPoolFile, mPOptions, mProps, mRLIUrl, mSiteStore, mTCFile, mTCHandle, mTCMode, REGISTRATION_UNIVERSE, TRANSFER_UNIVERSE`

### Constructor Summary

Constructors
Constructor and Description
`RemoveDirectory(ADag concDag, PegasusBag bag, java.lang.String submitDirectory)` The overloaded constructor that sets the dag for which we have to generated the cleanup dag for.

### Method Summary

All Methods Static Methods Instance Methods Concrete Methods
`ADag` `addRemoveDirectoryNodes(ADag dag)` Modifies the workflow to add remove directory nodes.
`ADag` `addRemoveDirectoryNodes(ADag workflow, java.util.Set<java.lang.String> sites)` Adds create dir nodes to the workflow.
`private TransformationCatalogEntry` `defaultTCEntry(SiteCatalogEntry site)` Returns a default TC entry to be used in case entry is not found in the transformation catalog.
`private java.lang.String` `getAssociatedCreateDirSite(Job job)` Returns the associated site that job is dependant on.
`static java.lang.String` `getCompleteTranformationName()` A convenience method to return the complete transformation name being used to construct jobs in this class.
`protected java.util.Set` `getCreateDirSites(ADag dag)` Retrieves the sites for which the create dir jobs need to be created.
`private java.lang.String` `getRemoveDirJobName(ADag dag, java.lang.String site)` It returns the name of the remove directory job, that is to be assigned.
`Job` `makeRemoveDirJob(java.lang.String site, java.lang.String jobName)` It creates a remove directory job that creates a directory on the remote pool using the perl executable that Gaurang wrote.
`Job` `makeRemoveDirJob(java.lang.String site, java.lang.String jobName, java.util.List<java.lang.String> files)` It creates a remove directory job that creates a directory on the remote pool using the perl executable that Gaurang wrote.

### Methods inherited from class edu.isi.pegasus.planner.refiner.Engine

`addVector, appendArrayList, complainForHeadNodeURLPrefix, complainForHeadNodeURLPrefix, loadProperties, printVector, stringInList, stringInPegVector, stringInVector, vectorToString`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Field Detail

#### CLEANUP_DAG_PREFIX

    public static final java.lang.String CLEANUP_DAG_PREFIX

The prefix that is attached to the name of the dag for which the cleanup Dag is being generated, to generate the name of the cleanup Dag.

See Also:
  Constant Field Values

#### REMOVE_DIR_SUFFIX

    public static final java.lang.String REMOVE_DIR_SUFFIX

Constant suffix for the names of the remote directory nodes.

See Also:
  Constant Field Values

#### TRANSFORMATION_NAME

    public static final java.lang.String TRANSFORMATION_NAME

The logical name of the transformation that removes directories on the remote execution pools.

See Also:
  Constant Field Values

#### REMOVE_DIR_EXECUTABLE_BASENAME

    public static final java.lang.String REMOVE_DIR_EXECUTABLE_BASENAME

The basename of the pegasus dirmanager executable.

See Also:
  Constant Field Values

#### TRANSFORMATION_NAMESPACE

    public static final java.lang.String TRANSFORMATION_NAMESPACE

The transformation namespace for the create dir jobs.

See Also:
  Constant Field Values

#### TRANSFORMATION_VERSION

    public static final java.lang.String TRANSFORMATION_VERSION

The version number for the derivations for create dir jobs.

#### DERIVATION_NAMESPACE

    public static final java.lang.String DERIVATION_NAMESPACE

The derivation namespace for the create dir jobs.

See Also:
  Constant Field Values

#### DERIVATION_NAME

    public static final java.lang.String DERIVATION_NAME

The logical name of the transformation that removes directories on the remote execution pools.

See Also:
  Constant Field Values

#### DERIVATION_VERSION

    public static final java.lang.String DERIVATION_VERSION

The version number for the derivations for create dir jobs.

See Also:
  Constant Field Values

#### CLEANUP_PREFIX

    public static final java.lang.String CLEANUP_PREFIX

Constant suffix for the names of the deployment nodes.

See Also:
  Constant Field Values

#### mConcDag

    private ADag mConcDag

The concrete dag so far, for which the clean up dag needs to be generated.

#### mTransferFromSubmitHost

    private boolean mTransferFromSubmitHost

Boolean indicating whether we need to transfer dirmanager from the submit host.

#### mSubmitDirectory

    private java.lang.String mSubmitDirectory

The submit directory for the workflow.

#### mJobPrefix

    protected java.lang.String mJobPrefix

The job prefix that needs to be applied to the job file basenames.

### Constructor Detail

#### RemoveDirectory

    public RemoveDirectory(ADag concDag,
                           PegasusBag bag,
                           java.lang.String submitDirectory)

The overloaded constructor that sets the dag for which we have to generated the cleanup dag for.

Parameters:
  `concDag` - the concrete dag for which cleanup is reqd.
  `bag` - the bag of initialization objects
  `submitDirectory` - the submit directory for the cleanup workflow

### Method Detail

#### getCompleteTranformationName

    public static java.lang.String getCompleteTranformationName()

A convenience method to return the complete transformation name being used to construct jobs in this class.

Returns:
  the complete transformation name

#### addRemoveDirectoryNodes

    public ADag addRemoveDirectoryNodes(ADag dag)

Modifies the workflow to add remove directory nodes. The workflow passed is a worklow, where the jobs have been mapped to sites. The strategy involves in walking the graph in a BFS order, and updating a bit set associated with each job based on the BitSet of the parents jobs. The BitSet indicates whether an edge exists from the descendant of a node to the remove dir job. For a node, the bit set is the union of all the parents BitSets. The BFS traversal ensures that the bitsets are of a node are only updated once the parents have been processed.

Parameters:
  `dag` - the workflow to which the nodes have to be added.
Returns:
  the added workflow

#### addRemoveDirectoryNodes

    public ADag addRemoveDirectoryNodes(ADag workflow,
                                        java.util.Set<java.lang.String> sites)

Adds create dir nodes to the workflow. The strategy involves in walking the graph in a BFS order, and updating a bit set associated with each job based on the BitSet of the parent jobs. The BitSet indicates whether an edge exists from the create dir job to an ancestor of the node. For a node, the bit set is the union of all the parents BitSets. The BFS traversal ensures that the bitsets are of a node are only updated once the parents have been processed.

Parameters:
  `workflow` - the workflow
  `sites` - the staging sites the workflow refers to.
Returns:

#### getCreateDirSites

    protected java.util.Set getCreateDirSites(ADag dag)

Retrieves the sites for which the create dir jobs need to be created. It returns all the sites where the compute jobs have been scheduled.

Returns:
  a Set containing a list of siteID's of the sites where the dag has to be run.

#### getRemoveDirJobName

    private java.lang.String getRemoveDirJobName(ADag dag,
                                                 java.lang.String site)

It returns the name of the remove directory job, that is to be assigned. The name takes into account the workflow name while constructing it, as that is thing that can guarentee uniqueness of name in case of deferred planning.

Parameters:
  `dag` - the dag for which the cleanup DAG is being generated.
  `site` - the execution site for which the remove directory job is responsible.
Returns:
  String corresponding to the name of the job.

#### makeRemoveDirJob

    public Job makeRemoveDirJob(java.lang.String site,
                                java.lang.String jobName)

It creates a remove directory job that creates a directory on the remote pool using the perl executable that Gaurang wrote. It access mkdir underneath. It gets the name of the random directory from the Pool handle.

Parameters:
  `site` - the execution pool for which the create dir job is to be created.
  `jobName` - the name that is to be assigned to the job.
Returns:
  the remove dir job.

#### makeRemoveDirJob

    public Job makeRemoveDirJob(java.lang.String site,
                                java.lang.String jobName,
                                java.util.List<java.lang.String> files)

It creates a remove directory job that creates a directory on the remote pool using the perl executable that Gaurang wrote. It access mkdir underneath. It gets the name of the random directory from the Pool handle.

Parameters:
  `site` - the site from where the directory need to be removed.
  `jobName` - the name that is to be assigned to the job.
  `files` - the list of files to be cleaned up.
Returns:
  the remove dir job.

#### defaultTCEntry

    private TransformationCatalogEntry defaultTCEntry(SiteCatalogEntry site)

Returns a default TC entry to be used in case entry is not found in the transformation catalog.

Parameters:
  `site` - the SiteCatalogEntry for the site for which the default entry is required.
Returns:
  the default entry.

#### getAssociatedCreateDirSite

    private java.lang.String getAssociatedCreateDirSite(Job job)

Returns the associated site that job is dependant on. This is site, whose create dir job should be a parent or an ancestor of the job.

Parameters:
  `job` - the job for which we need the associated create dir site.
Returns:
  the site