edu.isi.pegasus.planner.refiner

## Class ReplicaCatalogBridge

- java.lang.Object
- - edu.isi.pegasus.planner.refiner.Engine
  - - edu.isi.pegasus.planner.refiner.ReplicaCatalogBridge

    public class ReplicaCatalogBridge
    extends Engine

This coordinates the look up to the Replica Location Service, to determine the logical to physical mappings.

Version:
  $Revision$
Author:
  Karan Vahi, Gaurang Mehta

### Field Summary

Fields
`static java.lang.String` `CACHE_READ_ONLY_KEY` The name of the key that disables writing back to the cache file.
`static java.lang.String` `CACHE_REPLICA_CATALOG_IMPLEMENTER` The name of the Replica Catalog Implementer that serves as the source for cache files.
`static java.lang.String` `CACHE_REPLICA_CATALOG_KEY` The name of the source key for Replica Catalog Implementer that serves as cache
`static java.lang.String` `DEFAULT_REGISTRATION_CATEGORY_KEY` Default category for registration jobs
`static java.lang.String` `DIRECTORY_REPLICA_CATALOG_IMPLEMENTER` The name of the Replica Catalog Implementer that serves as the source for cache files.
`static java.lang.String` `DIRECTORY_REPLICA_CATALOG_KEY` The name of the source key for Replica Catalog Implementer that serves as cache
`private ReplicaStore` `mCacheStore` The replica store in which we store all the results that are queried from the cache replica catalogs.
`private ADag` `mDag` The DAG being worked upon.
`private ReplicaStore` `mDAXReplicaStore` The DAX Replica Store.
`private boolean` `mDefaultTCRCCreated` A boolean indicating whether the attempt to create a default tc entry has happened or not.
`private TransformationCatalogEntry` `mDefaultTCRCEntry` The default tc entry.
`private ReplicaStore` `mDirectoryReplicaStore` The replica store where we store all the results that are queried from the input directory specified at the command line.
`private ReplicaStore` `mInheritedReplicaStore` The inherited Replica Store
`private boolean` `mRCDown` A boolean variable to desingnate whether the RLI queried was down or not.
`private ReplicaCatalog` `mReplicaCatalog` The handle to the main Replica Catalog.
`private ReplicaStore` `mReplicaStore` The replica store in which we store all the results that are queried from the main replica catalog.
`protected java.util.Set` `mSearchFiles` The Vector of `String` objects containing the logical filenames of the files whose locations are to be searched in the Replica Catalog.
`private boolean` `mTreatCacheAsRC` A boolean indicating whether the cache file needs to be treated as a replica catalog or not.
`static java.lang.String` `RC_DERIVATION_NAME` The derivation name for the transfer jobs.
`static java.lang.String` `RC_DERIVATION_NS` The derivation namespace for the transfer jobs.
`static java.lang.String` `RC_DERIVATION_VERSION` The version number for the derivations for registration jobs.
`static java.lang.String` `RC_TRANSFORMATION_NAME` The logical name of the transformation used.
`static java.lang.String` `RC_TRANSFORMATION_NS` The transformation namespace for the regostration jobs.
`static java.lang.String` `RC_TRANSFORMATION_VERSION` The logical name of the transformation used.
`static java.lang.String` `REPLICA_CATALOG_URL_KEY` The name of the URL key for the replica catalog impelementer to be picked up.

### Fields inherited from class edu.isi.pegasus.planner.refiner.Engine

`mBag, mLogger, mLogMsg, mOutputPool, mPoolFile, mPOptions, mProps, mRLIUrl, mSiteStore, mTCFile, mTCHandle, mTCMode, REGISTRATION_UNIVERSE, TRANSFER_UNIVERSE`

### Constructor Summary

Constructors
Constructor and Description
`ReplicaCatalogBridge(ADag dag, PegasusBag bag)` The overloaded constructor.

### Method Summary

All Methods Instance Methods Concrete Methods
`protected void` `appendProperty(java.lang.StringBuffer sb, java.lang.String key, java.lang.String value)` Appends a property to the StringBuffer, in the java command line format.
`void` `closeConnection()` To close the connection to replica services.
`private TransformationCatalogEntry` `defaultTCRCEntry()` Returns a default TC entry to be used in case entry is not found in the transformation catalog.
`void` `finalize()` Closes the connection to the rli.
`private java.lang.String` `ftToRC(FileTransfer ft)` Converts a `FileTransfer` to a RC compatible string representation.
`private java.lang.String` `generateMappingsFile(java.lang.String regJob, java.util.Collection files)` Generates the registration mappings in a text file that is generated in the dax directory (the directory where all the condor submit files are generated).
`private java.lang.String` `generateRepJobArgumentString(SiteCatalogEntry site, java.lang.String regJob, java.util.Collection files)` Generates the argument string to be given to the replica registration job.
`private Profile` `getClassPath(java.lang.String home)` Returns the classpath for the default rc-client entry.
`protected java.lang.String` `getCommandLineProperties(PegasusProperties properties)` Returns the properties that need to be passed to the the rc-client invocation on the command line .
`java.lang.String` `getDefaultRegistrationMaxJobsPropertyKey()` Returns the property key that can be used to set the max jobs for the default category associated with the registration jobs.
`ReplicaLocation` `getFileLocs(java.lang.String lfn)` Returns all the locations as returned from the Replica Lookup Mechanism.
`java.util.Set` `getFilesInReplica()` This returns the files for which mappings exist in the Replica Catalog.
`private java.lang.String` `getPathToLocalProxy()` Returns path to the local proxy
`private ReplicaStore` `getReplicaStoreFromDirectory(java.lang.String directory)` Loads the mappings from the input directory
`private ReplicaStore` `getReplicaStoreFromFiles(java.util.Set files)` Ends up loading a Replica Store from replica catalog files
`void` `initialize(ADag dag, PegasusBag bag)` Intialises the refiner.
`void` `initialize(ADag dag, PegasusProperties properties, PlannerOptions options)` Intialises the refiner.
`private void` `loadCacheFiles(java.util.Set cacheFiles)` Ends up loading the cache files so as to enable the lookup for the transient files created by the parent jobs.
`private void` `loadInheritedReplicaStore(java.util.Set files)` Ends up loading the inherited replica files.
`Job` `makeRCRegNode(java.lang.String regJobName, Job job, java.util.Collection files)` It constructs the Job object for the registration node, which registers the materialized files on the output pool in the RLS.
`private ReplicaLocation` `retrieveFromCache(java.lang.String lfn)` Retrieves a location from the cache table, that contains the contents of the cache files specified at runtime.

### Methods inherited from class edu.isi.pegasus.planner.refiner.Engine

`addVector, appendArrayList, complainForHeadNodeURLPrefix, complainForHeadNodeURLPrefix, loadProperties, printVector, stringInList, stringInPegVector, stringInVector, vectorToString`

### Methods inherited from class java.lang.Object

`clone, equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

### Field Detail

#### DEFAULT_REGISTRATION_CATEGORY_KEY

    public static final java.lang.String DEFAULT_REGISTRATION_CATEGORY_KEY

Default category for registration jobs

See Also:
  Constant Field Values

#### RC_TRANSFORMATION_NS

    public static final java.lang.String RC_TRANSFORMATION_NS

The transformation namespace for the regostration jobs.

See Also:
  Constant Field Values

#### RC_TRANSFORMATION_NAME

    public static final java.lang.String RC_TRANSFORMATION_NAME

The logical name of the transformation used.

See Also:
  Constant Field Values

#### RC_TRANSFORMATION_VERSION

    public static final java.lang.String RC_TRANSFORMATION_VERSION

The logical name of the transformation used.

#### RC_DERIVATION_NS

    public static final java.lang.String RC_DERIVATION_NS

The derivation namespace for the transfer jobs.

See Also:
  Constant Field Values

#### RC_DERIVATION_NAME

    public static final java.lang.String RC_DERIVATION_NAME

The derivation name for the transfer jobs.

See Also:
  Constant Field Values

#### RC_DERIVATION_VERSION

    public static final java.lang.String RC_DERIVATION_VERSION

The version number for the derivations for registration jobs.

See Also:
  Constant Field Values

#### CACHE_REPLICA_CATALOG_IMPLEMENTER

    public static final java.lang.String CACHE_REPLICA_CATALOG_IMPLEMENTER

The name of the Replica Catalog Implementer that serves as the source for cache files.

See Also:
  Constant Field Values

#### CACHE_REPLICA_CATALOG_KEY

    public static final java.lang.String CACHE_REPLICA_CATALOG_KEY

The name of the source key for Replica Catalog Implementer that serves as cache

See Also:
  Constant Field Values

#### CACHE_READ_ONLY_KEY

    public static final java.lang.String CACHE_READ_ONLY_KEY

The name of the key that disables writing back to the cache file. Designates a static file. i.e. read only

See Also:
  Constant Field Values

#### DIRECTORY_REPLICA_CATALOG_IMPLEMENTER

    public static final java.lang.String DIRECTORY_REPLICA_CATALOG_IMPLEMENTER

The name of the Replica Catalog Implementer that serves as the source for cache files.

See Also:
  Constant Field Values

#### DIRECTORY_REPLICA_CATALOG_KEY

    public static final java.lang.String DIRECTORY_REPLICA_CATALOG_KEY

The name of the source key for Replica Catalog Implementer that serves as cache

See Also:
  Constant Field Values

#### REPLICA_CATALOG_URL_KEY

    public static final java.lang.String REPLICA_CATALOG_URL_KEY

The name of the URL key for the replica catalog impelementer to be picked up.

See Also:
  Constant Field Values

#### mReplicaCatalog

    private ReplicaCatalog mReplicaCatalog

The handle to the main Replica Catalog.

#### mSearchFiles

    protected java.util.Set mSearchFiles

The Vector of `String` objects containing the logical filenames of the files whose locations are to be searched in the Replica Catalog.

#### mRCDown

    private boolean mRCDown

A boolean variable to desingnate whether the RLI queried was down or not. By default it is up, unless it is set to true explicitly.

#### mReplicaStore

    private ReplicaStore mReplicaStore

The replica store in which we store all the results that are queried from the main replica catalog.

#### mCacheStore

    private ReplicaStore mCacheStore

The replica store in which we store all the results that are queried from the cache replica catalogs.

#### mDirectoryReplicaStore

    private ReplicaStore mDirectoryReplicaStore

The replica store where we store all the results that are queried from the input directory specified at the command line.

#### mDAXReplicaStore

    private ReplicaStore mDAXReplicaStore

The DAX Replica Store.

#### mInheritedReplicaStore

    private ReplicaStore mInheritedReplicaStore

The inherited Replica Store

#### mTreatCacheAsRC

    private boolean mTreatCacheAsRC

A boolean indicating whether the cache file needs to be treated as a replica catalog or not.

#### mDefaultTCRCEntry

    private TransformationCatalogEntry mDefaultTCRCEntry

The default tc entry.

#### mDefaultTCRCCreated

    private boolean mDefaultTCRCCreated

A boolean indicating whether the attempt to create a default tc entry has happened or not.

#### mDag

    private ADag mDag

The DAG being worked upon.

### Constructor Detail

#### ReplicaCatalogBridge

    public ReplicaCatalogBridge(ADag dag,
                                PegasusBag bag)

The overloaded constructor.

Parameters:
  `dag` - the workflow that is being worked on.
  `bag` - of initialization objects.

### Method Detail

#### initialize

    public void initialize(ADag dag,
                           PegasusBag bag)

Intialises the refiner.

Parameters:
  `dag` - the workflow that is being worked on.
  `bag` - the bag of Pegasus initialization objects

#### initialize

    public void initialize(ADag dag,
                           PegasusProperties properties,
                           PlannerOptions options)

Intialises the refiner.

Parameters:
  `dag` - the workflow that is being worked on.
  `properties` - the properties passed to the planner.
  `options` - the options passed to the planner at runtime.

#### closeConnection

    public void closeConnection()

To close the connection to replica services. This must be defined in the case where one has not done a singleton implementation. In other cases just do an empty implementation of this method.

#### finalize

    public void finalize()

Closes the connection to the rli.

Overrides:
  `finalize` in class `java.lang.Object`

#### getFilesInReplica

    public java.util.Set getFilesInReplica()

This returns the files for which mappings exist in the Replica Catalog. This should return a subset of the files which are specified in the mSearchFiles, while getting an instance to this.

Returns:
  a `Set` of logical file names as String objects, for which logical to physical mapping exists.
See Also:
  `mSearchFiles`

#### getFileLocs

    public ReplicaLocation getFileLocs(java.lang.String lfn)

Returns all the locations as returned from the Replica Lookup Mechanism.

Parameters:
  `lfn` - The name of the logical file whose PFN mappings are required.
Returns:
  ReplicaLocation containing all the locations for that LFN
See Also:
  `org.griphyn.cPlanner.classes.ReplicaLocation`

#### getDefaultRegistrationMaxJobsPropertyKey

    public java.lang.String getDefaultRegistrationMaxJobsPropertyKey()

Returns the property key that can be used to set the max jobs for the default category associated with the registration jobs.

Returns:
  the property key

#### makeRCRegNode

    public Job makeRCRegNode(java.lang.String regJobName,
                             Job job,
                             java.util.Collection files)

It constructs the Job object for the registration node, which registers the materialized files on the output pool in the RLS. Note that the relations corresponding to this node should already have been added to the concerned `DagInfo` object.

Parameters:
  `regJobName` - The name of the job which registers the files in the Replica Location Service.
  `job` - The job whose output files are to be registered in the Replica Location Service.
  `files` - Collection of `FileTransfer` objects containing the information about source and destination URLs. The destination URLs would be our PFNs.
Returns:
  Job corresponding to the new registration node.

#### defaultTCRCEntry

    private TransformationCatalogEntry defaultTCRCEntry()

Returns a default TC entry to be used in case entry is not found in the transformation catalog.

Returns:
  the default entry.

#### getClassPath

    private Profile getClassPath(java.lang.String home)

Returns the classpath for the default rc-client entry.

Parameters:
  `home` - the home directory where we need to check for lib directory.
Returns:
  the classpath in an environment profile.

#### generateRepJobArgumentString

    private java.lang.String generateRepJobArgumentString(SiteCatalogEntry site,
                                                          java.lang.String regJob,
                                                          java.util.Collection files)

Generates the argument string to be given to the replica registration job. At present by default it would be picking up the file containing the mappings.

Parameters:
  `site` - the `SiteCatalogEntry` object
  `regJob` - The name of the registration job.
  `files` - Collection of `FileTransfer` objects containing the information about source and destURLs. The destination URLs would be our PFNs.
Returns:
  the argument string.

#### getCommandLineProperties

    protected java.lang.String getCommandLineProperties(PegasusProperties properties)

Returns the properties that need to be passed to the the rc-client invocation on the command line . It is of the form "-Dprop1=value1 -Dprop2=value2 .."

Parameters:
  `properties` - the properties object
Returns:
  the properties list, else empty string.

#### appendProperty

    protected void appendProperty(java.lang.StringBuffer sb,
                                  java.lang.String key,
                                  java.lang.String value)

Appends a property to the StringBuffer, in the java command line format.

Parameters:
  `sb` - the StringBuffer to append the property to.
  `key` - the property.
  `value` - the property value.

#### generateMappingsFile

    private java.lang.String generateMappingsFile(java.lang.String regJob,
                                                  java.util.Collection files)

Generates the registration mappings in a text file that is generated in the dax directory (the directory where all the condor submit files are generated). The pool value for the mapping is the output pool specified by the user when running Pegasus. The name of the file is regJob+.in

Parameters:
  `regJob` - The name of the registration job.
  `files` - Collection of `FileTransfer`objects containing the information about source and destURLs. The destination URLs would be our PFNs.
Returns:
  String corresponding to the path of the the file containig the mappings in the appropriate format.

#### ftToRC

    private java.lang.String ftToRC(FileTransfer ft)

Converts a `FileTransfer` to a RC compatible string representation.

Parameters:
  `ft` - the `FileTransfer` object
Returns:
  the RC version.

#### retrieveFromCache

    private ReplicaLocation retrieveFromCache(java.lang.String lfn)

Retrieves a location from the cache table, that contains the contents of the cache files specified at runtime.

Parameters:
  `lfn` - the logical name of the file.
Returns:
  `ReplicaLocation` object corresponding to the entry if found, else null.

#### loadInheritedReplicaStore

    private void loadInheritedReplicaStore(java.util.Set files)

Ends up loading the inherited replica files.

Parameters:
  `files` - set of paths to the inherited replica files.

#### loadCacheFiles

    private void loadCacheFiles(java.util.Set cacheFiles)

Ends up loading the cache files so as to enable the lookup for the transient files created by the parent jobs.

Parameters:
  `cacheFiles` - set of paths to the cache files.

#### getReplicaStoreFromFiles

    private ReplicaStore getReplicaStoreFromFiles(java.util.Set files)

Ends up loading a Replica Store from replica catalog files

Parameters:
  `files` - set of paths to the cache files.

#### getReplicaStoreFromDirectory

    private ReplicaStore getReplicaStoreFromDirectory(java.lang.String directory)

Loads the mappings from the input directory

Parameters:
  `directory` - the directory to load from

#### getPathToLocalProxy

    private java.lang.String getPathToLocalProxy()

Returns path to the local proxy

Returns:
  path to the local proxy