edu.isi.pegasus.planner.transfer.refiner

## Class Basic

- java.lang.Object
- - edu.isi.pegasus.planner.transfer.AbstractRefiner
  - - edu.isi.pegasus.planner.transfer.MultipleFTPerXFERJobRefiner
    - - edu.isi.pegasus.planner.transfer.refiner.Basic

All Implemented Interfaces:
  Refiner

Direct Known Subclasses:
  BalancedCluster, Bundle, Chain

    public class Basic
    extends MultipleFTPerXFERJobRefiner

The default transfer refiner, that implements the multiple refiner. For each compute job if required it creates the following - a single stagein transfer job - a single stageout transfer job - a single interpool transfer job In addition this implementation prevents file clobbering while staging in data to a remote site, that is shared amongst jobs.

Version:
  $Revision$
Author:
  Karan Vahi

### Field Summary

Fields
`static java.lang.String` `DESCRIPTION` A short description of the transfer refinement.
`protected java.lang.Boolean` `mCreateRegistrationJobs` Boolean indicating whether to create registration jobs or not.
`protected java.util.Map` `mFileTable` A Map containing information about which logical file has been transferred to which site and the name of the stagein transfer node that is transferring the file from the location returned from the replica catalog.
`protected java.lang.String` `mLogMsg` The string holding the logging messages
`protected PPS` `mPPS` The handle to the provenance store implementation.
`protected java.util.Map<java.lang.String,java.util.Set<java.lang.String>>` `mRelationsMap` The map indexed by a node name, where the associated value is the set of child nodes.

### Fields inherited from class edu.isi.pegasus.planner.transfer.AbstractRefiner

`mDAG, mLogger, mPOptions, mProps, mRemoteTransfers, mTPT, mTXInterImplementation, mTXStageInImplementation, mTXStageOutImplementation, mTXSymbolicLinkImplementation, mXMLStore`

### Fields inherited from interface edu.isi.pegasus.planner.transfer.Refiner

`INTER_POOL_PREFIX, LOCAL_PREFIX, REGISTER_PREFIX, REMOTE_PREFIX, STAGE_IN_PREFIX, STAGE_OUT_PREFIX`

### Fields inherited from interface edu.isi.pegasus.planner.refiner.Refiner

`VERSION`

### Constructor Summary

Constructors
Constructor and Description
`Basic(ADag dag, PegasusBag bag)` The overloaded constructor.

### Method Summary

All Methods Instance Methods Concrete Methods
`void` `addInterSiteTXNodes(Job job, java.util.Collection files, boolean localTransfer)` Adds the inter pool transfer nodes that are required for transferring the output files of the parents to the jobs execution site.
`void` `addJob(Job job)` Add a new job to the workflow being refined.
`void` `addRelation(java.lang.String parent, java.lang.String child)` Adds a new relation to the workflow being refiner.
`void` `addRelation(java.lang.String parent, java.lang.String child, java.lang.String site, boolean parentNew)` Adds a new relation to the workflow.
`void` `addStageInXFERNodes(Job job, java.util.Collection<FileTransfer> files, java.util.Collection<FileTransfer> symlinkFiles)` Adds the stage in transfer nodes which transfer the input files for a job, from the location returned from the replica catalog to the job's execution pool.
`void` `addStageInXFERNodes(Job job, java.util.Collection<FileTransfer> files, java.lang.String prefix, Implementation implementation)` Adds the stage in transfer nodes which transfer the input files for a job, from the location returned from the replica catalog to the job's execution pool.
`void` `addStageOutXFERNodes(Job job, java.util.Collection files, ReplicaCatalogBridge rcb, boolean localTransfer)` Adds the stageout transfer nodes, that stage data to an output site specified by the user.
`void` `addStageOutXFERNodes(Job job, java.util.Collection files, ReplicaCatalogBridge rcb, boolean localTransfer, boolean deletedLeaf)` Adds the stageout transfer nodes, that stage data to an output site specified by the user.
`protected void` `appendAttribute(java.lang.StringBuffer xmlFeed, java.lang.String key, java.lang.String value)` Appends an xml attribute to the xml feed.
`protected java.lang.String` `constructFileKey(java.lang.String lfn, java.lang.String siteHandle)` Constructs the key for an entry to the file table.
`protected Job` `createRegistrationJob(java.lang.String regJobName, Job job, java.util.Collection files, ReplicaCatalogBridge rcb)` Creates the registration jobs, which registers the materialized files on the output site in the Replica Catalog.
`void` `done()` Signals that the traversal of the workflow is done.
`java.lang.String` `getDescription()` Returns a textual description of the transfer mode.
`protected int` `getJobPriority(Job job)` Returns the priority associated with a job based on the condor profile key priority .
`protected void` `logRefinerAction(Job computeJob, Job txJob, java.util.Collection files, java.lang.String type)` Records the refiner action into the Provenace Store as a XML fragment.

### Methods inherited from class edu.isi.pegasus.planner.transfer.MultipleFTPerXFERJobRefiner

`loadImplementations`

### Methods inherited from class edu.isi.pegasus.planner.transfer.AbstractRefiner

`addStageInXFERNodes, getWorkflow, getXMLProducer, isSiteThirdParty, logConfigMessages, refinerPreferenceForLocalTransferJobs, refinerPreferenceForTransferJobLocation, runTPTOnRemoteSite, runTransferRemotely`

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Field Detail

#### DESCRIPTION

    public static final java.lang.String DESCRIPTION

A short description of the transfer refinement.

See Also:
  Constant Field Values

#### mLogMsg

    protected java.lang.String mLogMsg

The string holding the logging messages

#### mFileTable

    protected java.util.Map mFileTable

A Map containing information about which logical file has been transferred to which site and the name of the stagein transfer node that is transferring the file from the location returned from the replica catalog. The key for the hashmap is logicalfilename:sitehandle and the value would be the name of the transfer node.

#### mRelationsMap

    protected java.util.Map<java.lang.String,java.util.Set<java.lang.String>> mRelationsMap

The map indexed by a node name, where the associated value is the set of child nodes.

#### mPPS

    protected PPS mPPS

The handle to the provenance store implementation.

#### mCreateRegistrationJobs

    protected java.lang.Boolean mCreateRegistrationJobs

Boolean indicating whether to create registration jobs or not.

### Constructor Detail

#### Basic

    public Basic(ADag dag,
                 PegasusBag bag)

The overloaded constructor.

Parameters:
  `dag` - the workflow to which transfer nodes need to be added.
  `bag` - the bag of initialization objects.

### Method Detail

#### addStageInXFERNodes

    public void addStageInXFERNodes(Job job,
                                    java.util.Collection<FileTransfer> files,
                                    java.util.Collection<FileTransfer> symlinkFiles)

Adds the stage in transfer nodes which transfer the input files for a job, from the location returned from the replica catalog to the job's execution pool.

Overrides:
  `addStageInXFERNodes` in class `AbstractRefiner`
Parameters:
  `job` - `Job` object corresponding to the node to which the files are to be transferred to.
  `files` - Collection of `FileTransfer` objects containing the information about source and destURL's.
  `symlinkFiles` - Collection of `FileTransfer` objects containing source and destination file url's for symbolic linking on compute site.

#### addStageInXFERNodes

    public void addStageInXFERNodes(Job job,
                                    java.util.Collection<FileTransfer> files,
                                    java.lang.String prefix,
                                    Implementation implementation)

Adds the stage in transfer nodes which transfer the input files for a job, from the location returned from the replica catalog to the job's execution pool.

Parameters:
  `job` - `Job` object corresponding to the node to which the files are to be transferred to.
  `files` - Collection of `FileTransfer` objects containing the information about source and destURL's.
  `prefix` - the prefix to be used while constructing the transfer jobname.
  `implementation` - the transfer implementation to use

#### addInterSiteTXNodes

    public void addInterSiteTXNodes(Job job,
                                    java.util.Collection files,
                                    boolean localTransfer)

Adds the inter pool transfer nodes that are required for transferring the output files of the parents to the jobs execution site.

Parameters:
  `job` - `Job` object corresponding to the node to which the files are to be transferred to.
  `files` - Collection of `FileTransfer` objects containing the information about source and destURL's.
  `localTransfer` - boolean indicating that associated transfer job will run on local site.

#### addStageOutXFERNodes

    public void addStageOutXFERNodes(Job job,
                                     java.util.Collection files,
                                     ReplicaCatalogBridge rcb,
                                     boolean localTransfer)

Adds the stageout transfer nodes, that stage data to an output site specified by the user.

Parameters:
  `job` - `Job` object corresponding to the node to which the files are to be transferred to.
  `files` - Collection of `FileTransfer` objects containing the information about source and destURL's.
  `rcb` - bridge to the Replica Catalog. Used for creating registration nodes in the workflow.
  `localTransfer` - boolean indicating that associated transfer job will run on local site.

#### addStageOutXFERNodes

    public void addStageOutXFERNodes(Job job,
                                     java.util.Collection files,
                                     ReplicaCatalogBridge rcb,
                                     boolean localTransfer,
                                     boolean deletedLeaf)

Adds the stageout transfer nodes, that stage data to an output site specified by the user.

Parameters:
  `job` - `Job` object corresponding to the node to which the files are to be transferred to.
  `files` - Collection of `FileTransfer` objects containing the information about source and destURL's.
  `rcb` - bridge to the Replica Catalog. Used for creating registration nodes in the workflow.
  `localTransfer` - boolean indicating that associated transfer job will run on local site.
  `deletedLeaf` - to specify whether the node is being added for a deleted node by the reduction engine or not. default: false

#### createRegistrationJob

    protected Job createRegistrationJob(java.lang.String regJobName,
                                        Job job,
                                        java.util.Collection files,
                                        ReplicaCatalogBridge rcb)

Creates the registration jobs, which registers the materialized files on the output site in the Replica Catalog.

Parameters:
  `regJobName` - The name of the job which registers the files in the Replica Mechanism.
  `job` - The job whose output files are to be registered in the Replica Mechanism.
  `files` - Collection of `FileTransfer` objects containing the information about source and destURL's.
  `rcb` - bridge to the Replica Catalog. Used for creating registration nodes in the workflow.
Returns:
  the registration job.

#### done

    public void done()

Signals that the traversal of the workflow is done. It signals to the Provenace Store, that refinement is complete.

#### getJobPriority

    protected int getJobPriority(Job job)

Returns the priority associated with a job based on the condor profile key priority . Defaults to 0.

Parameters:
  `job` - the job priority
Returns:
  the priority key

#### addJob

    public void addJob(Job job)

Add a new job to the workflow being refined.

Parameters:
  `job` - the job to be added.

#### addRelation

    public void addRelation(java.lang.String parent,
                            java.lang.String child)

Adds a new relation to the workflow being refiner.

Parameters:
  `parent` - the jobname of the parent node of the edge.
  `child` - the jobname of the child node of the edge.

#### addRelation

    public void addRelation(java.lang.String parent,
                            java.lang.String child,
                            java.lang.String site,
                            boolean parentNew)

Adds a new relation to the workflow. In the case when the parent is a transfer job that is added, the parentNew should be set only the first time a relation is added. For subsequent compute jobs that maybe dependant on this, it needs to be set to false.

Parameters:
  `parent` - the jobname of the parent node of the edge.
  `child` - the jobname of the child node of the edge.
  `site` - the execution pool where the transfer node is to be run.
  `parentNew` - the parent node being added, is the new transfer job and is being called for the first time.

#### getDescription

    public java.lang.String getDescription()

Returns a textual description of the transfer mode.

Returns:
  a short textual description

#### logRefinerAction

    protected void logRefinerAction(Job computeJob,
                                    Job txJob,
                                    java.util.Collection files,
                                    java.lang.String type)

Records the refiner action into the Provenace Store as a XML fragment.

Parameters:
  `computeJob` - the compute job.
  `txJob` - the associated transfer job.
  `files` - list of `FileTransfer` objects containing file transfers.
  `type` - the type of transfer job

#### appendAttribute

    protected void appendAttribute(java.lang.StringBuffer xmlFeed,
                                   java.lang.String key,
                                   java.lang.String value)

Appends an xml attribute to the xml feed.

Parameters:
  `xmlFeed` - the xmlFeed to which xml is being written
  `key` - the attribute key
  `value` - the attribute value

#### constructFileKey

    protected java.lang.String constructFileKey(java.lang.String lfn,
                                                java.lang.String siteHandle)

Constructs the key for an entry to the file table. The key returned is lfn:siteHandle

Parameters:
  `lfn` - the logical filename of the file that has to be transferred.
  `siteHandle` - the name of the site to which the file is being transferred.
Returns:
  the key for the entry to be made in the filetable.