edu.isi.pegasus.planner.transfer.sls

## Class Transfer

- java.lang.Object
- - edu.isi.pegasus.planner.transfer.sls.Transfer

All Implemented Interfaces:
  SLS

    public class Transfer
    extends java.lang.Object
    implements SLS

This uses the transfer executable distributed with Pegasus to do the second level staging.

Version:
  $Revision$
Author:
  Karan Vahi

### Field Summary

Fields
`static int` `DEFAULT_NUMBER_OF_THREADS` The default number of threads pegasus-transfer uses
`static java.lang.String` `DERIVATION_NAME` The name of the underlying derivation.
`static java.lang.String` `DERIVATION_NAMESPACE` The derivation namespace for for the transfer job.
`static java.lang.String` `DERIVATION_VERSION` The derivation version number for the transfer job.
`static java.lang.String` `DESCRIPTION` A short description of the transfer implementation.
`static java.lang.String` `EXECUTABLE_BASENAME` The executable basename
`private boolean` `mBypassStagingForInputs` A boolean indicating whether to bypass first level staging for inputs
`protected java.lang.String` `mExtraArguments` Any extra arguments that need to be passed ahead to the s3 client invocation.
`protected LogManager` `mLogger` The handle to the logging manager.
`private PlannerCache` `mPlannerCache` A SimpleFile Replica Catalog, that tracks all the files that are being materialized as part of workflow execution.
`protected PegasusProperties` `mProps` The handle to the properties.
`protected boolean` `mSeqExecGridStartUsed` Boolean to track whether the gridstart used in PegasusLite or not
`protected SiteStore` `mSiteStore` The handle to the site catalog.
`protected boolean` `mStageSLSFile` Boolean to track whether to stage sls file or not
`protected TransformationCatalog` `mTCHandle` The handle to the transformation catalog.
`protected boolean` `mUseSymLinks` This member variable if set causes the destination URL for the symlink jobs to have symlink:// url if the pool attributed associated with the pfn is same as a particular jobs execution pool.
`static java.lang.String` `TRANSFORMATION_NAME` The name of the underlying transformation that is queried for in the Transformation Catalog.
`static java.lang.String` `TRANSFORMATION_NAMESPACE` The transformation namespace for the transfer job.
`static java.lang.String` `TRANSFORMATION_VERSION` The version number for the transfer job.

### Fields inherited from interface edu.isi.pegasus.planner.transfer.SLS

`VERSION`

### Constructor Summary

Constructors
Constructor and Description
`Transfer()` The default constructor.

### Method Summary

All Methods Instance Methods Concrete Methods
`protected void` `complainForHeadNodeURLPrefix(Job job, java.lang.String site)` Complains for head node url prefix not specified
`protected void` `complainForHeadNodeURLPrefix(java.lang.String site)` Complains for head node url prefix not specified
`protected TransformationCatalogEntry` `defaultTCEntry(java.lang.String namespace, java.lang.String name, java.lang.String version, java.lang.String executableBasename, java.lang.String site)` Returns a default TC entry to be used in case entry is not found in the transformation catalog.
`java.util.Collection<FileTransfer>` `determineSLSInputTransfers(Job job, java.lang.String fileName, FileServer stagingSiteServer, java.lang.String stagingSiteDirectory, java.lang.String workerNodeDirectory)` Generates a second level staging file of the input files to the worker node directory.
`java.util.Collection<FileTransfer>` `determineSLSOutputTransfers(Job job, java.lang.String fileName, FileServer stagingSiteServer, java.lang.String stagingSiteDirectory, java.lang.String workerNodeDirectory)` Generates a second level staging file of the input files to the worker node directory.
`boolean` `doesCondorModifications()` Returns a boolean whether the SLS implementation does a condor based modification or not.
`protected java.util.List` `getEnvironmentVariables(java.lang.String site)` Returns the environment profiles that are required for the default entry to sensibly work.
`protected java.lang.String` `getExecutableBasename()` Return the executable basename for transfer executable used.
`java.lang.String` `getSLSInputLFN(Job job)` Returns the LFN of sls input file.
`java.lang.String` `getSLSOutputLFN(Job job)` Returns the LFN of sls output file.
`TransformationCatalogEntry` `getTransformationCatalogEntry(java.lang.String siteHandle)` Retrieves the transformation catalog entry for the executable that is being used to transfer the files in the implementation.
`void` `initialize(PegasusBag bag)` Initializes the SLS implementation.
`java.lang.String` `invocationString(Job job, java.io.File slsFile)` Constructs a command line invocation for a job, with a given sls file.
`boolean` `modifyJobForWorkerNodeExecution(Job job, java.lang.String stagingSiteURLPrefix, java.lang.String stagingSitedirectory, java.lang.String workerNodeDirectory)` Modifies a compute job for second level staging.
`boolean` `needsSLSInputTransfers(Job job)` Returns a boolean indicating whether it will an input file for a job to do the transfers.
`boolean` `needsSLSOutputTransfers(Job job)` Returns a boolean indicating whether it will an output file for a job to do the transfers.

### Methods inherited from class java.lang.Object

`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Field Detail

#### TRANSFORMATION_NAMESPACE

    public static final java.lang.String TRANSFORMATION_NAMESPACE

The transformation namespace for the transfer job.

See Also:
  Constant Field Values

#### TRANSFORMATION_NAME

    public static final java.lang.String TRANSFORMATION_NAME

The name of the underlying transformation that is queried for in the Transformation Catalog.

See Also:
  Constant Field Values

#### TRANSFORMATION_VERSION

    public static final java.lang.String TRANSFORMATION_VERSION

The version number for the transfer job.

#### DERIVATION_NAMESPACE

    public static final java.lang.String DERIVATION_NAMESPACE

The derivation namespace for for the transfer job.

See Also:
  Constant Field Values

#### DERIVATION_NAME

    public static final java.lang.String DERIVATION_NAME

The name of the underlying derivation.

See Also:
  Constant Field Values

#### DERIVATION_VERSION

    public static final java.lang.String DERIVATION_VERSION

The derivation version number for the transfer job.

See Also:
  Constant Field Values

#### DEFAULT_NUMBER_OF_THREADS

    public static final int DEFAULT_NUMBER_OF_THREADS

The default number of threads pegasus-transfer uses

See Also:
  Constant Field Values

#### DESCRIPTION

    public static final java.lang.String DESCRIPTION

A short description of the transfer implementation.

See Also:
  Constant Field Values

#### EXECUTABLE_BASENAME

    public static final java.lang.String EXECUTABLE_BASENAME

The executable basename

See Also:
  Constant Field Values

#### mSiteStore

    protected SiteStore mSiteStore

The handle to the site catalog.

#### mTCHandle

    protected TransformationCatalog mTCHandle

The handle to the transformation catalog.

#### mProps

    protected PegasusProperties mProps

The handle to the properties.

#### mLogger

    protected LogManager mLogger

The handle to the logging manager.

#### mExtraArguments

    protected java.lang.String mExtraArguments

Any extra arguments that need to be passed ahead to the s3 client invocation.

#### mStageSLSFile

    protected boolean mStageSLSFile

Boolean to track whether to stage sls file or not

#### mSeqExecGridStartUsed

    protected boolean mSeqExecGridStartUsed

Boolean to track whether the gridstart used in PegasusLite or not

#### mBypassStagingForInputs

    private boolean mBypassStagingForInputs

A boolean indicating whether to bypass first level staging for inputs

#### mPlannerCache

    private PlannerCache mPlannerCache

A SimpleFile Replica Catalog, that tracks all the files that are being materialized as part of workflow execution.

#### mUseSymLinks

    protected boolean mUseSymLinks

This member variable if set causes the destination URL for the symlink jobs to have symlink:// url if the pool attributed associated with the pfn is same as a particular jobs execution pool.

### Constructor Detail

#### Transfer

    public Transfer()

The default constructor.

### Method Detail

#### initialize

    public void initialize(PegasusBag bag)

Initializes the SLS implementation.

Specified by:
  `initialize` in interface `SLS`
Parameters:
  `bag` - the bag of objects. Contains access to catalogs etc.

#### doesCondorModifications

    public boolean doesCondorModifications()

Returns a boolean whether the SLS implementation does a condor based modification or not. By condor based modification we mean whether it uses condor specific classads to achieve the second level staging or not.

Specified by:
  `doesCondorModifications` in interface `SLS`
Returns:
  false

#### invocationString

    public java.lang.String invocationString(Job job,
                                             java.io.File slsFile)

Constructs a command line invocation for a job, with a given sls file. The SLS maybe null. In the case where SLS impl does not read from a file, it is advised to create a file in generateSLSXXX methods, and then read the file in this function and put it on the command line.

Specified by:
  `invocationString` in interface `SLS`
Parameters:
  `job` - the job that is being sls enabled
  `slsFile` - the slsFile can be null
Returns:
  invocation string

#### needsSLSInputTransfers

    public boolean needsSLSInputTransfers(Job job)

Returns a boolean indicating whether it will an input file for a job to do the transfers. Transfer reads from stdin the file transfers that it needs to do. Always returns true, as we need to transfer the proxy always.

Specified by:
  `needsSLSInputTransfers` in interface `SLS`
Parameters:
  `job` - the job being detected.
Returns:
  true

#### needsSLSOutputTransfers

    public boolean needsSLSOutputTransfers(Job job)

Returns a boolean indicating whether it will an output file for a job to do the transfers. Transfer reads from stdin the file transfers that it needs to do.

Specified by:
  `needsSLSOutputTransfers` in interface `SLS`
Parameters:
  `job` - the job being detected.
Returns:
  true

#### getSLSInputLFN

    public java.lang.String getSLSInputLFN(Job job)

Returns the LFN of sls input file.

Specified by:
  `getSLSInputLFN` in interface `SLS`
Parameters:
  `job` - Job
Returns:
  the name of the sls input file.

#### getSLSOutputLFN

    public java.lang.String getSLSOutputLFN(Job job)

Returns the LFN of sls output file.

Specified by:
  `getSLSOutputLFN` in interface `SLS`
Parameters:
  `job` - Job
Returns:
  the name of the sls input file.

#### determineSLSInputTransfers

    public java.util.Collection<FileTransfer> determineSLSInputTransfers(Job job,
                                                                         java.lang.String fileName,
                                                                         FileServer stagingSiteServer,
                                                                         java.lang.String stagingSiteDirectory,
                                                                         java.lang.String workerNodeDirectory)

Generates a second level staging file of the input files to the worker node directory.

Specified by:
  `determineSLSInputTransfers` in interface `SLS`
Parameters:
  `job` - job for which the file is being created
  `fileName` - name of the file that needs to be written out.
  `stagingSiteServer` - the file server on the staging site to be used for retrieval of files i.e the get operation
  `stagingSiteDirectory` - directory on the head node of the staging site.
  `workerNodeDirectory` - worker node directory
Returns:
  a Collection of FileTransfer objects listing the transfers that need to be done.
See Also:
  `needsSLSInputTransfers( Job)`

#### determineSLSOutputTransfers

    public java.util.Collection<FileTransfer> determineSLSOutputTransfers(Job job,
                                                                          java.lang.String fileName,
                                                                          FileServer stagingSiteServer,
                                                                          java.lang.String stagingSiteDirectory,
                                                                          java.lang.String workerNodeDirectory)

Generates a second level staging file of the input files to the worker node directory.

Specified by:
  `determineSLSOutputTransfers` in interface `SLS`
Parameters:
  `job` - the job for which the file is being created
  `fileName` - the name of the file that needs to be written out.
  `stagingSiteServer` - the file server on the staging site to be used for retrieval of files i.e the put operation
  `stagingSiteDirectory` - the directory on the head node of the staging site.
  `workerNodeDirectory` - the worker node directory
Returns:
  a Collection of FileTransfer objects listing the transfers that need to be done.
See Also:
  `needsSLSOutputTransfers( Job)`

#### modifyJobForWorkerNodeExecution

    public boolean modifyJobForWorkerNodeExecution(Job job,
                                                   java.lang.String stagingSiteURLPrefix,
                                                   java.lang.String stagingSitedirectory,
                                                   java.lang.String workerNodeDirectory)

Modifies a compute job for second level staging. The only modification it does is add the appropriate environment varialbes to the job

Specified by:
  `modifyJobForWorkerNodeExecution` in interface `SLS`
Parameters:
  `job` - the job to be modified.
  `stagingSiteURLPrefix` - the url prefix for the server on the staging site
  `stagingSitedirectory` - the directory on the staging site, where the inp
  `workerNodeDirectory` - the directory in the worker node tmp
Returns:
  boolean indicating whether job was successfully modified or not.

#### getTransformationCatalogEntry

    public TransformationCatalogEntry getTransformationCatalogEntry(java.lang.String siteHandle)

Retrieves the transformation catalog entry for the executable that is being used to transfer the files in the implementation. If an entry is not specified in the Transformation Catalog, then null is returned.

Parameters:
  `siteHandle` - the handle of the site where the transformation is to be searched.
Returns:
  the transformation catalog entry if found, else null.

#### defaultTCEntry

    protected TransformationCatalogEntry defaultTCEntry(java.lang.String namespace,
                                                        java.lang.String name,
                                                        java.lang.String version,
                                                        java.lang.String executableBasename,
                                                        java.lang.String site)

Returns a default TC entry to be used in case entry is not found in the transformation catalog.

Parameters:
  `namespace` - the namespace of the transfer transformation
  `name` - the logical name of the transfer transformation
  `version` - the version of the transfer transformation
  `executableBasename` - the basename of the executable
  `site` - the site for which the default entry is required.
Returns:
  the default entry.

#### getEnvironmentVariables

    protected java.util.List getEnvironmentVariables(java.lang.String site)

Returns the environment profiles that are required for the default entry to sensibly work. Tries to retrieve the following variables

     PEGASUS_HOME
     GLOBUS_LOCATION
     LD_LIBRARY_PATH

Parameters:
  `site` - the site where the job is going to run.
Returns:
  List of environment variables, else empty list if none are found

#### getExecutableBasename

    protected java.lang.String getExecutableBasename()

Return the executable basename for transfer executable used.

Returns:
  the executable basename.

#### complainForHeadNodeURLPrefix

    protected void complainForHeadNodeURLPrefix(java.lang.String site)

Complains for head node url prefix not specified

Parameters:
  `site` - the site handle
Throws:
  `java.lang.RuntimeException` - when URL Prefix cannot be determined for various reason.

#### complainForHeadNodeURLPrefix

    protected void complainForHeadNodeURLPrefix(Job job,
                                                java.lang.String site)

Complains for head node url prefix not specified

Parameters:
  `job` - the related job if any
  `site` - the site handle
Throws:
  `java.lang.RuntimeException` - when URL Prefix cannot be determined for various reason.