TOIL(1)

NAME
toil - Toil Documentation

Toil is an open-source pure-Python workflow engine that lets people write better pipelines.

Check out our website <http://toil.ucsc-cgl.org/> for a comprehensive list of Toil's features and read our paper <http://biorxiv.org/content/early/2016/07/07/062497> to learn what Toil can do in the real world. Please subscribe to our low-volume announce <https://groups.google.com/forum/#!forum/toil-announce> mailing list and feel free to also join us on GitHub <https://github.com/BD2KGenomics/toil> and Gitter <https://gitter.im/bd2k-genomics-toil/Lobby>.

If using Toil for your research, please cite
        Vivian, J., Rao, A. A., Nothaft, F. A., Ketchum, C., Armstrong, J., Novak, A., … Paten, B. (2017). Toil enables reproducible, open source, big biomedical data analyses. Nature Biotechnology, 35(4), 314–316. <http://doi.org/10.1038/nbt.3772>

QUICKSTART EXAMPLES

  Running a basic CWL workflow

The Common Workflow Language <http://www.commonwl.org/> (CWL) is an emerging standard for writing workflows that are portable across multiple workflow engines and platforms. Running CWL workflows using Toil is easy.

    1.  Copy and paste the following code block into example.cwl:

                cwlVersion: v1.0
                class: CommandLineTool
                baseCommand: echo
                stdout: output.txt
                inputs:
                  message:
                    type: string
                    inputBinding:
                      position: 1
                outputs:
                  output:
                    type: stdout

    and this code into example-job.yaml:

                message: Hello world!

    2.  To run the workflow simply enter

                $ toil-cwl-runner example.cwl example-job.yaml

    Your output will be in output.txt:

                $ cat output.txt
                Hello world!

Congratulations! You've run your first Toil workflow using the default Batch System <#batchsysteminterface>, single_machine, and the default file job store (which was placed in a temporary directory for you by toil-cwl-runner).

Toil uses batch systems to manage the jobs it creates.

The single_machine batch system is primarily used to prepare and debug workflows on a local machine. Once validated, try running them on a full-fledged batch system (see Batch System API <#batchsysteminterface>). Toil supports many different batch systems such as Kubernetes <https://kubernetes.io/> and Grid Engine; its versatility makes it easy to run your workflow in all kinds of places.

Toil's CWL runner is totally customizable! Run toil-cwl-runner --help to see a complete list of available options.

To learn more about CWL, see the CWL User Guide <https://www.commonwl.org/user_guide/> (from where this example was shamelessly borrowed). For information on using CWL with Toil see the section CWL in Toil <#cwl>. And for an example of CWL on an AWS cluster, have a look at Running a CWL Workflow on AWS.

  Running a basic WDL workflow

The Workflow Description Language <https://software.broadinstitute.org/wdl/> (WDL) is another emerging language for writing workflows that are portable across multiple workflow engines and platforms. Running WDL workflows using Toil is still in alpha, and currently experimental. Toil currently supports basic workflow syntax (see WDL in Toil <#wdl> for more details and examples). Here we go over running a basic WDL helloworld workflow.

    1.  Copy and paste the following code block into wdl-helloworld.wdl:

                workflow write_simple_file {
                  call write_file
                }
                task write_file {
                  String message
                  command { echo ${message} > wdl-helloworld-output.txt }
                  output { File test = "wdl-helloworld-output.txt" }
                }

    and this code into wdl-helloworld.json:

                {
                  "write_simple_file.write_file.message": "Hello world!"
                }

    2.  To run the workflow simply enter

                $ toil-wdl-runner wdl-helloworld.wdl wdl-helloworld.json

    Your output will be in wdl-helloworld-output.txt:

                $ cat wdl-helloworld-output.txt
                Hello world!

This will, like the CWL example above, use the single_machine batch system and an automatically-located file job store by default. You can customize Toil's execution of the workflow with command-line options; run toil-wdl-runner --help to learn about them.

To learn more about WDL in general, see the Terra WDL documentation <https://support.terra.bio/hc/en-us/sections/360007274612-WDL-Documentation> . For more on using WDL in Toil, see WDL in Toil <#wdl>.

  Running a basic Python workflow

In addition to workflow languages like CWL and WDL, Toil supports running workflows written against its Python API.

An example Toil Python workflow can be run with just three steps:

    1.  Install Toil (see Installation <#installation-ref>)

    2.  Copy and paste the following code block into a new file called helloWorld.py:

            from toil.common import Toil
            from toil.job import Job

            def helloWorld(message, memory="1G", cores=1, disk="1G"):
                return f"Hello, world!, here's a message: {message}"

            if __name__ == "__main__":
                parser = Job.Runner.getDefaultArgumentParser()
                options = parser.parse_args()
                options.clean = "always"
                with Toil(options) as toil:
                    output = toil.start(Job.wrapFn(helloWorld, "You did it!"))
                print(output)

    3.  Specify the name of the job store <#jobstoreoverview> and run the workflow:

                $ python3 helloWorld.py file:my-job-store

For something beyond a "Hello, world!" example, refer to Example: sorting.

Toil's customization options are available in Python workflows. Run python3 helloWorld.py --help to see a complete list of available options.

  Example: mitochondrial variant calling

Let's run a more realistic workflow with Toil. This workflow is mitochondrial variant calling: It will take the human reference genome and sequenced reads from an individual and determine how that person's mitochondrial DNA differs from the reference genome.

We will run an example workflow from Dockstore <https://dockstore.org/workflows/github.com/broadinstitute/gatk/MitochondriaPipeline:master?tab=info>. First, grab an example workflow input:

            (venv) $ wget https://toil-datasets.s3.us-west-2.amazonaws.com/MitochondriaInputs.zip && unzip MitochondriaInputs.zip

Then, change your current working directory:

            (venv) $ cd MitochondriaInputs

This workflow will take approximately 30 minutes to run.

Since Toil supports Dockstore TRS IDs that allows the WDL runner to run any workflow on Dockstore, we will run it directly:

            (venv) $ toil-wdl-runner '#workflow/github.com/broadinstitute/gatk/MitochondriaPipeline:master' -i ExampleInputsMitochondriaPipeline.json --logInfo --container docker --quantCheck false --outputFile mitochondria.json

Note:

            •   --logInfo runs the workflow with INFO level logging. For different levels of logging, see --logLevel, --logCritical, --logError, --logWarning, --logDebug, and --logTrace.

            •   --container docker uses Docker as the container backend. By default, Toil will run with Singularity. To set explicitly, use --container singularity.

            •   --outputFile will put the workflow JSON outputs into a file. If omitted, Toil will put the workflow outputs onto the commandline.

            •   --quantCheck false disables certain type checks. This is useful for Cromwell compatibility.

Unless fakeroot support is set up for Singularity, this particular workflow must be run with Docker because it assumes commands in the container will run as root. Additionally, WDL workflows sometimes depend on non-spec compliant behaviors. To see if Toil has an workaround option, see Toil WDL Runner Options <#wdloptions>.

Once the workflow is done running, you can look at your JSON output with jq . mitochondria.json. For example, if we want the out_vcf output from the workflow, we can run jq -r '.["MitochondriaPipeline.out_vcf"]' mitochondria.json to get its path:

            /private/groups/patenlab/toil-dev/mitochondria/wdl-out-c6o9mjop/MitochondriaPipeline.AlignAndCall.FilterContamination/HG02571.GRCh38.chrM.vcf

To open the VCF file, we can run less $(jq -r '.["MitochondriaPipeline.out_vcf"]' mitochondria.json)

Note:
        For outputs that aren't files, their values are directly in the JSON. For example, with jq '.["MitochondriaPipeline.median_coverage"]' mitochondria.json you can fetch the median_coverage output's value:

                    4183.5

Toil uses a jobstore to store all of a workflow's files and to communicate between workers. If not specified, Toil will use an ephemeral directory that is deleted after Toil is done running. To control where those files are placed or allow a workflow to be restarted, you can use the --jobStore option. If you specify a jobstore explicitly, the jobstore will stick around if the workflow fails. To keep the jobstore after a successful completion, use --clean never. To remove the jobstore even after a failing run, use --clean always.

On a cluster, the jobstore must be somewhere accessible to all worker nodes. Here's an example of running the workflow with a specified jobstore:

            (venv) $ toil-wdl-runner MitochondriaPipeline.wdl -i ExampleInputsMitochondriaPipeline.json --logInfo --container docker --outputFile mitochondria.json --jobstore mitochondriaJobstore

Toil supports several batch systems. By default, Toil will use single_machine, which will run everything on the local machine. Other batch systems are available. For example, you can use --batchSystem slurm to run on a Slurm cluster:

            (venv) $ toil-wdl-runner MitochondriaPipeline.wdl -i ExampleInputsMitochondriaPipeline.json --logInfo --container docker --outputFile mitochondria.json --jobStore mitochondriaJobstore --batchSystem slurm

See Running on Slurm <#runningslurm> for more information, including how to specify time limits and partitions.

Sometimes, a workflow may fail. If this is the case, the workflow can be restarted from the point of failure with --restart, as long as you still have the jobstore:

            (venv) $ toil-wdl-runner MitochondriaPipeline.wdl -i ExampleInputsMitochondriaPipeline.json --logInfo --container docker --outputFile mitochondria.json --jobStore mitochondriaJobstore --batchSystem slurm

  Example: sorting

For a more detailed example and explanation, we've developed a sample pipeline that merge-sorts a temporary file. This is not supposed to be an efficient sorting program, rather a more fully worked example of what Toil is capable of.

  Running the example

    1.  Download the example code

    2.  Run it with the default settings:

                $ python3 sort.py file:jobStore

    The workflow created a file called sortedFile.txt in your current directory. Have a look at it and notice that it contains a whole lot of sorted lines!

    This workflow does a smart merge sort on a file it generates, fileToSort.txt. The sort is smart because each step of the process---splitting the file into separate chunks, sorting these chunks, and merging them back together---is compartmentalized into a job. Each job can specify its own resource requirements and will only be run after the jobs it depends upon have run. Jobs without dependencies will be run in parallel.

Note:
        Delete fileToSort.txt before moving on to #3. This example introduces options that specify dimensions for fileToSort.txt, if it does not already exist. If it exists, this workflow will use the existing file and the results will be the same as #2.

    3.  Run with custom options:

                $ python3 sort.py file:jobStore \
                             --numLines=5000 \
                             --lineLength=10 \
                             --overwriteOutput=True \
                             --workDir=/tmp/

    Here we see that we can add our own options to a Toil Python workflow. As noted above, the first two options, --numLines and --lineLength, determine the number of lines and how many characters are in each line. --overwriteOutput causes the current contents of sortedFile.txt to be overwritten, if it already exists. The last option, --workDir, is an option built into Toil to specify where temporary files unique to a job are kept.

  Describing the source code

To understand the details of what's going on inside. Let's start with the main() function. It looks like a lot of code, but don't worry---we'll break it down piece by piece.

            def main(options=None):
                if not options:
                    # deal with command line arguments
                    parser = ArgumentParser()
                    Job.Runner.addToilOptions(parser)
                    parser.add_argument(
                        "--numLines",
                        default=defaultLines,
                        help="Number of lines in file to sort.",
                        type=int,
                    )
                    parser.add_argument(
                        "--lineLength",
                        default=defaultLineLen,
                        help="Length of lines in file to sort.",
                        type=int,
                    )
                    parser.add_argument("--fileToSort", help="The file you wish to sort")
                    parser.add_argument("--outputFile", help="Where the sorted output will go")
                    parser.add_argument(
                        "--overwriteOutput",
                        help="Write over the output file if it already exists.",
                        default=True,
                    )
                    parser.add_argument(
                        "--N",
                        dest="N",
                        help="The threshold below which a serial sort function is used to sort file. "
                        "All lines must of length less than or equal to N or program will fail",
                        default=10000,
                    )
                    parser.add_argument(
                        "--downCheckpoints",
                        action="store_true",
                        help="If this option is set, the workflow will make checkpoints on its way through"
                        'the recursive "down" part of the sort',
                    )
                    parser.add_argument(
                        "--sortMemory",
                        dest="sortMemory",
                        help="Memory for jobs that sort chunks of the file.",
                        default=None,
                    )

                    parser.add_argument(
                        "--mergeMemory",
                        dest="mergeMemory",
                        help="Memory for jobs that collate results.",
                        default=None,
                    )

                    options = parser.parse_args()
                if not hasattr(options, "sortMemory") or not options.sortMemory:
                    options.sortMemory = sortMemory
                if not hasattr(options, "mergeMemory") or not options.mergeMemory:
                    options.mergeMemory = sortMemory

                # do some input verification
                sortedFileName = options.outputFile or "sortedFile.txt"
                if not options.overwriteOutput and os.path.exists(sortedFileName):
                    print(
                        f"Output file {sortedFileName} already exists.  "
                        f"Delete it to run the sort example again or use --overwriteOutput=True"
                    )
                    exit()

                fileName = options.fileToSort
                if options.fileToSort is None:
                    # make the file ourselves
                    fileName = "fileToSort.txt"
                    if os.path.exists(fileName):
                        print(f"Sorting existing file: {fileName}")
                    else:
                        print(
                            f"No sort file specified. Generating one automatically called: {fileName}."
                        )
                        makeFileToSort(
                            fileName=fileName, lines=options.numLines, lineLen=options.lineLength
                        )
                else:
                    if not os.path.exists(options.fileToSort):
                        raise RuntimeError("File to sort does not exist: %s" % options.fileToSort)

                if int(options.N) <= 0:
                    raise RuntimeError("Invalid value of N: %s" % options.N)

                # Now we are ready to run
                with Toil(options) as workflow:
                    sortedFileURL = "file://" + os.path.abspath(sortedFileName)
                    if not workflow.options.restart:
                        sortFileURL = "file://" + os.path.abspath(fileName)
                        sortFileID = workflow.importFile(sortFileURL)
                        sortedFileID = workflow.start(
                            Job.wrapJobFn(
                                setup,
                                sortFileID,
                                int(options.N),
                                options.downCheckpoints,
                                options=options,
                                memory=sortMemory,
                            )
                        )
                    else:
                        sortedFileID = workflow.restart()
                    workflow.exportFile(sortedFileID, sortedFileURL)

First we make a parser to process command line arguments using the argparse <https://docs.python.org/2.7/library/argparse.html> module. It's important that we add the call to Job.Runner.addToilOptions() to initialize our parser with all of Toil's default options. Then we add the command line arguments unique to this workflow, and parse the input. The help message listed with the arguments should give you a pretty good idea of what they can do.

Next we do a little bit of verification of the input arguments. The option --fileToSort allows you to specify a file that needs to be sorted. If this option isn't given, it's here that we make our own file with the call to makeFileToSort().

Finally we come to the context manager that initializes the workflow. We create a path to the input file prepended with 'file://' as per the documentation for toil.common.Toil() <#toil.common.Toil> when staging a file that is stored locally. Notice that we have to check whether or not the workflow is restarting so that we don't import the file more than once. Finally we can kick off the workflow by calling toil.common.Toil.start() <#toil.common.Toil.start> on the job setup. When the workflow ends we capture its output (the sorted file's fileID) and use that in toil.common.Toil.exportFile() <#toil.common.Toil.exportFile> to move the sorted file from the job store back into "userland".

Next let's look at the job that begins the actual workflow, setup.

            def setup(job, inputFile, N, downCheckpoints, options):
                """
                Sets up the sort.
                Returns the FileID of the sorted file
                """
                RealtimeLogger.info("Starting the merge sort")
                return job.addChildJobFn(
                    down,
                    inputFile,
                    N,
                    "root",
                    downCheckpoints,
                    options=options,
                    preemptible=True,
                    memory=sortMemory,
                ).rv()

setup really only does two things. First it writes to the logs using Job.log() and then calls addChildJobFn(). Child jobs run directly after the current job. This function turns the 'job function' down into an actual job and passes in the inputs including an optional resource requirement, memory. The job doesn't actually get run until the call to Job.rv(). Once the job down finishes, its output is returned here.

Now we can look at what down does.

            def down(job, inputFileStoreID, N, path, downCheckpoints, options, memory=sortMemory):
                """
                Input is a file, a subdivision size N, and a path in the hierarchy of jobs.
                If the range is larger than a threshold N the range is divided recursively and
                a follow on job is then created which merges back the results else
                the file is sorted and placed in the output.
                """

                RealtimeLogger.info("Down job starting: %s" % path)

                # Read the file
                inputFile = job.fileStore.readGlobalFile(inputFileStoreID, cache=False)
                length = os.path.getsize(inputFile)
                if length > N:
                    # We will subdivide the file
                    RealtimeLogger.critical(
                        "Splitting file: %s of size: %s" % (inputFileStoreID, length)
                    )
                    # Split the file into two copies
                    midPoint = getMidPoint(inputFile, 0, length)
                    t1 = job.fileStore.getLocalTempFile()
                    with open(t1, "w") as fH:
                        fH.write(copySubRangeOfFile(inputFile, 0, midPoint + 1))
                    t2 = job.fileStore.getLocalTempFile()
                    with open(t2, "w") as fH:
                        fH.write(copySubRangeOfFile(inputFile, midPoint + 1, length))
                    # Call down recursively. By giving the rv() of the two jobs as inputs to the follow-on job, up,
                    # we communicate the dependency without hindering concurrency.
                    result = job.addFollowOnJobFn(
                        up,
                        job.addChildJobFn(
                            down,
                            job.fileStore.writeGlobalFile(t1),
                            N,
                            path + "/0",
                            downCheckpoints,
                            checkpoint=downCheckpoints,
                            options=options,
                            preemptible=True,
                            memory=options.sortMemory,
                        ).rv(),
                        job.addChildJobFn(
                            down,
                            job.fileStore.writeGlobalFile(t2),
                            N,
                            path + "/1",
                            downCheckpoints,
                            checkpoint=downCheckpoints,
                            options=options,
                            preemptible=True,
                            memory=options.mergeMemory,
                        ).rv(),
                        path + "/up",
                        preemptible=True,
                        options=options,
                        memory=options.sortMemory,
                    ).rv()
                else:
                    # We can sort this bit of the file
                    RealtimeLogger.critical(
                        "Sorting file: %s of size: %s" % (inputFileStoreID, length)
                    )
                    # Sort the copy and write back to the fileStore
                    shutil.copyfile(inputFile, inputFile + ".sort")
                    sort(inputFile + ".sort")
                    result = job.fileStore.writeGlobalFile(inputFile + ".sort")

                RealtimeLogger.info("Down job finished: %s" % path)
                return result

Down is the recursive part of the workflow. First we read the file into the local filestore by calling job.fileStore.readGlobalFile(). This puts a copy of the file in the temp directory for this particular job. This storage will disappear once this job ends. For a detailed explanation of the filestore, job store, and their interfaces have a look at Managing files within a workflow <#managingfiles>.

Next down checks the base case of the recursion: is the length of the input file less than N (remember N was an option we added to the workflow in main)? In the base case, we just sort the file, and return the file ID of this new sorted file.

If the base case fails, then the file is split into two new tempFiles using job.fileStore.getLocalTempFile() and the helper function copySubRangeOfFile. Finally we add a follow on Job up with job.addFollowOnJobFn(). We've already seen child jobs. A follow-on Job is a job that runs after the current job and all of its children (and their children and follow-ons) have completed. Using a follow-on makes sense because up is responsible for merging the files together and we don't want to merge the files together until we know they are sorted. Again, the return value of the follow-on job is requested using Job.rv().

Looking at up

            def up(job, inputFileID1, inputFileID2, path, options, memory=sortMemory):
                """
                Merges the two files and places them in the output.
                """

                RealtimeLogger.info("Up job starting: %s" % path)

                with job.fileStore.writeGlobalFileStream() as (fileHandle, outputFileStoreID):
                    fileHandle = codecs.getwriter("utf-8")(fileHandle)
                    with job.fileStore.readGlobalFileStream(inputFileID1) as inputFileHandle1:
                        inputFileHandle1 = codecs.getreader("utf-8")(inputFileHandle1)
                        with job.fileStore.readGlobalFileStream(inputFileID2) as inputFileHandle2:
                            inputFileHandle2 = codecs.getreader("utf-8")(inputFileHandle2)
                            RealtimeLogger.info(
                                "Merging %s and %s to %s"
                                % (inputFileID1, inputFileID2, outputFileStoreID)
                            )
                            merge(inputFileHandle1, inputFileHandle2, fileHandle)
                    # Cleanup up the input files - these deletes will occur after the completion is successful.
                    job.fileStore.deleteGlobalFile(inputFileID1)
                    job.fileStore.deleteGlobalFile(inputFileID2)

                    RealtimeLogger.info("Up job finished: %s" % path)

                    return outputFileStoreID

we see that the two input files are merged together and the output is written to a new file using job.fileStore.writeGlobalFileStream(). After a little cleanup, the output file is returned.

Once the final up finishes and all of the rv() promises are fulfilled, main receives the sorted file's ID which it uses in exportFile to send it to the user.

There are other things in this example that we didn't go over such as Checkpoints <#checkpoints> and the details of much of the Toil Class API <#api>.

At the end of the script the lines

            if __name__ == '__main__'
                main()

are included to ensure that the main function is only run once in the '__main__' process invoked by you, the user. In Toil terms, by invoking the script you created the leader process in which the main() function is run. A worker process is a separate process whose sole purpose is to host the execution of one or more jobs defined in that script. In any Toil workflow there is always one leader process, and potentially many worker processes.

When using the single-machine batch system (the default), the worker processes will be running on the same machine as the leader process. With full-fledged batch systems like Kubernetes the worker processes will typically be started on separate machines. The boilerplate ensures that the pipeline is only started once---on the leader---but not when its job functions are imported and executed on the individual workers.

Typing python3 sort.py --help will show the complete list of arguments for the workflow which includes both Toil's and ones defined inside sort.py. A complete explanation of Toil's arguments can be found in Commandline Options <#commandref>.

  Logging

By default, Toil logs a lot of information related to the current environment in addition to messages from the batch system and jobs. This can be configured with the --logLevel flag. For example, to only log CRITICAL level messages to the screen:

            $ python3 sort.py file:jobStore \
                         --logLevel=critical \
                         --overwriteOutput=True

This hides most of the information we get from the Toil run. For more detail, we can run the pipeline with --logLevel=debug to see a comprehensive output. For more information, see Commandline Options <#workflowoptions>.

  Error Handling and Resuming Pipelines

With Toil, you can recover gracefully from a bug in your pipeline without losing any progress from successfully completed jobs. To demonstrate this, let's add a bug to our example code to see how Toil handles a failure and how we can resume a pipeline after that happens. Add a bad assertion at line 52 of the example (the first line of down()):

            def down(job, inputFileStoreID, N, downCheckpoints, memory=sortMemory):
                ...
                assert 1 == 2, "Test error!"

When we run the pipeline, Toil will show a detailed failure log with a traceback:

            $ python3 sort.py file:jobStore
            ...
            ---TOIL WORKER OUTPUT LOG---
            ...
            m/j/jobonrSMP    Traceback (most recent call last):
            m/j/jobonrSMP      File "toil/src/toil/worker.py", line 340, in main
            m/j/jobonrSMP        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)
            m/j/jobonrSMP      File "toil/src/toil/job.py", line 1270, in _runner
            m/j/jobonrSMP        returnValues = self._run(jobGraph, fileStore)
            m/j/jobonrSMP      File "toil/src/toil/job.py", line 1217, in _run
            m/j/jobonrSMP        return self.run(fileStore)
            m/j/jobonrSMP      File "toil/src/toil/job.py", line 1383, in run
            m/j/jobonrSMP        rValue = userFunction(*((self,) + tuple(self._args)), **self._kwargs)
            m/j/jobonrSMP      File "toil/example.py", line 30, in down
            m/j/jobonrSMP        assert 1 == 2, "Test error!"
            m/j/jobonrSMP    AssertionError: Test error!

If we try and run the pipeline again, Toil will give us an error message saying that a job store of the same name already exists. By default, in the event of a failure, the job store is preserved so that the workflow can be restarted, starting from the previously failed jobs. We can restart the pipeline by running

            $ python3 sort.py file:jobStore \
                         --restart \
                         --overwriteOutput=True

We can also change the number of times Toil will attempt to retry a failed job:

            $ python3 sort.py file:jobStore \
                         --retryCount 2 \
                         --restart \
                         --overwriteOutput=True

You'll now see Toil attempt to rerun the failed job until it runs out of tries. --retryCount is useful for non-systemic errors, like downloading a file that may experience a sporadic interruption, or some other non-deterministic failure.

To successfully restart our pipeline, we can edit our script to comment out line 30, or remove it, and then run

            $ python3 sort.py file:jobStore \
                         --restart \
                         --overwriteOutput=True

The pipeline will run successfully, and the job store will be removed on the pipeline's completion.

  Collecting Statistics

Please see the Status Command <#cli-status> section for more on gathering runtime and resource info on jobs.

  Launching a Toil Workflow in AWS

After having installed the aws extra for Toil during the Installation <#installation-ref> and set up AWS (see Preparing your AWS environment <#prepareaws>), the user can run the basic helloWorld.py script (Running a basic Python workflow) on a VM in AWS just by modifying the run command.

Note that when running in AWS, users can either run the workflow on a single instance or run it on a cluster (which is running across multiple containers on multiple AWS instances). For more information on running Toil workflows on a cluster, see Running in AWS <#runningaws>.

Also! Remember to use the Destroy-Cluster Command <#destroycluster> command when finished to destroy the cluster! Otherwise things may not be cleaned up properly.

    1.  Launch a cluster in AWS using the Launch-Cluster Command <#launchcluster> command:

                $ toil launch-cluster <cluster-name> \
                             --clusterType kubernetes \
                             --keyPairName <AWS-key-pair-name> \
                             --leaderNodeType t2.medium \
                             --nodeTypes t2.medium -w 1 \
                             --zone us-west-2a

    The arguments keyPairName, leaderNodeType, and zone are required to launch a cluster.

    2.  Copy helloWorld.py to the /tmp directory on the leader node using the Rsync-Cluster Command <#rsynccluster> command:

                $ toil rsync-cluster --zone us-west-2a <cluster-name> helloWorld.py :/tmp

    Note that the command requires defining the file to copy as well as the target location on the cluster leader node.

    3.  Login to the cluster leader node using the Ssh-Cluster Command <#sshcluster> command:

                $ toil ssh-cluster --zone us-west-2a <cluster-name>

    Note that this command will log you in as the root user.

    4.  Run the workflow on the cluster:

                $ python3 /tmp/helloWorld.py aws:us-west-2:my-S3-bucket

    In this particular case, we create an S3 bucket called my-S3-bucket in the us-west-2 availability zone to store intermediate job results.

    Along with some other INFO log messages, you should get the following output in your terminal window: Hello, world!, here's a message: You did it!.

    5.  Exit from the SSH connection.

                $ exit

    6.  Use the Destroy-Cluster Command <#destroycluster> command to destroy the cluster:

                $ toil destroy-cluster --zone us-west-2a <cluster-name>

    Note that this command will destroy the cluster leader node and any resources created to run the job, including the S3 bucket.

  Running a CWL Workflow on AWS

After having installed the aws and cwl extras for Toil during the Installation <#installation-ref> and set up AWS (see Preparing your AWS environment <#prepareaws>), the user can run a CWL workflow with Toil on AWS.

Also! Remember to use the Destroy-Cluster Command <#destroycluster> command when finished to destroy the cluster! Otherwise things may not be cleaned up properly.

    1.  First launch a node in AWS using the Launch-Cluster Command <#launchcluster> command:

                $ toil launch-cluster <cluster-name> \
                             --clusterType kubernetes \
                             --keyPairName <AWS-key-pair-name> \
                             --leaderNodeType t2.medium \
                             --nodeTypes t2.medium -w 1 \
                             --zone us-west-2a

    2.  Copy example.cwl and example-job.yaml from the CWL example to the node using the Rsync-Cluster Command <#rsynccluster> command:

                toil rsync-cluster --zone us-west-2a <cluster-name> example.cwl :/tmp
                toil rsync-cluster --zone us-west-2a <cluster-name> example-job.yaml :/tmp

    3.  SSH into the cluster's leader node using the Ssh-Cluster Command <#sshcluster> utility:

                $ toil ssh-cluster --zone us-west-2a <cluster-name>

    4.  Once on the leader node, command line tools such as kubectl will be available to you. It's also a good idea to update and install the following:

                sudo apt-get update
                sudo apt-get -y upgrade
                sudo apt-get -y dist-upgrade
                sudo apt-get -y install git

    5.  Now create a new virtualenv with the --system-site-packages option and activate:

                virtualenv --system-site-packages venv
                source venv/bin/activate

    6.  Now run the CWL workflow with the Kubernetes batch system:

                (venv) $ toil-cwl-runner \
                             --provisioner aws \
                             --batchSystem kubernetes \
                             --jobStore aws:us-west-2:any-name \
                             /tmp/example.cwl /tmp/example-job.yaml

    Tip:
            When running a CWL workflow on AWS, input files can be provided either on the local file system or in S3 buckets using s3:// URI references. Final output files will be copied to the local file system of the leader node.

    7.  Finally, log out of the leader node and from your local computer, destroy the cluster:

                $ toil destroy-cluster --zone us-west-2a <cluster-name>

  Running a Workflow with Autoscaling - Cactus

Cactus <https://github.com/ComparativeGenomicsToolkit/cactus> is a reference-free, whole-genome multiple alignment program that can be run on any of the cloud platforms Toil supports.

Note:
        Cloud Independence:

        This example provides a "cloud agnostic" view of running Cactus with Toil. Most options will not change between cloud providers. However, each provisioner has unique inputs for --leaderNodeType, --nodeType and --zone. We recommend the following:
                Option | Used in | AWS | Google
                --leaderNodeType | launch-cluster | t2.medium | n1-standard-1
                --zone | launch-cluster | us-west-2a | us-west1-a
                --zone | cactus | us-west-2
                --nodeType | cactus | c3.4xlarge | n1-standard-8

        When executing toil launch-cluster with gce specified for --provisioner, the option --boto must be specified and given a path to your .boto file. See Running in Google Compute Engine (GCE) <#runninggce> for more information about the --boto option.

Also! Remember to use the Destroy-Cluster Command <#destroycluster> command when finished to destroy the cluster! Otherwise things may not be cleaned up properly.

    1.  Download pestis.tar.gz

    2.  Launch a cluster using the Launch-Cluster Command <#launchcluster> command:

                $ toil launch-cluster <cluster-name> \
                             --provisioner <aws, gce> \
                             --keyPairName <key-pair-name> \
                             --leaderNodeType <type> \
                             --nodeType <type> \
                             -w 1-2 \
                             --zone <zone>

    Note:
            A Helpful Tip

            When using AWS, setting the environment variable eliminates having to specify the --zone option for each command. This will be supported for GCE in the future.

                        $ export TOIL_AWS_ZONE=us-west-2c

    3.  Create appropriate directory for uploading files:

                $ toil ssh-cluster --provisioner <aws, gce> <cluster-name>
                $ mkdir /root/cact_ex
                $ exit

    4.  Copy the required files, i.e., seqFile.txt (a text file containing the locations of the input sequences as well as their phylogenetic tree, see here <https://github.com/ComparativeGenomicsToolkit/cactus#seqfile-the-input-file>), organisms' genome sequence files in FASTA format, and configuration files (e.g. blockTrim1.xml, if desired), up to the leader node:

                $ toil rsync-cluster --provisioner <aws, gce> <cluster-name> pestis-short-aws-seqFile.txt :/root/cact_ex
                $ toil rsync-cluster --provisioner <aws, gce> <cluster-name> GCF_000169655.1_ASM16965v1_genomic.fna :/root/cact_ex
                $ toil rsync-cluster --provisioner <aws, gce> <cluster-name> GCF_000006645.1_ASM664v1_genomic.fna :/root/cact_ex
                $ toil rsync-cluster --provisioner <aws, gce> <cluster-name> GCF_000182485.1_ASM18248v1_genomic.fna :/root/cact_ex
                $ toil rsync-cluster --provisioner <aws, gce> <cluster-name> GCF_000013805.1_ASM1380v1_genomic.fna :/root/cact_ex
                $ toil rsync-cluster --provisioner <aws, gce> <cluster-name> setup_leaderNode.sh :/root/cact_ex
                $ toil rsync-cluster --provisioner <aws, gce> <cluster-name> blockTrim1.xml :/root/cact_ex
                $ toil rsync-cluster --provisioner <aws, gce> <cluster-name> blockTrim3.xml :/root/cact_ex

    5.  Log in to the leader node:

                $ toil ssh-cluster --provisioner <aws, gce> <cluster-name>

    6.  Set up the environment of the leader node to run Cactus:

                $ bash /root/cact_ex/setup_leaderNode.sh
                $ source cact_venv/bin/activate
                (cact_venv) $ cd cactus
                (cact_venv) $ pip install --upgrade .

    7.  Run Cactus <https://github.com/ComparativeGenomicsToolkit/cactus> as an autoscaling workflow:

                (cact_venv) $ cactus \
                                  --retry 10 \
                                  --batchSystem kubernetes \
                                  --logDebug \
                                  --logFile /logFile_pestis3 \
                                  --configFile \
                                  /root/cact_ex/blockTrim3.xml <aws, google>:<zone>:cactus-pestis \
                                  /root/cact_ex/pestis-short-aws-seqFile.txt \
                                  /root/cact_ex/pestis_output3.hal

    Note:
            Pieces of the Puzzle:

            --logDebug --- equivalent to --logLevel DEBUG.

            --logFile /logFile_pestis3 --- writes logs in a file named logFile_pestis3 under / folder.

            --configFile --- this is not required depending on whether a specific configuration file is intended to run the alignment.

            <aws, google>:<zone>:cactus-pestis --- creates a bucket, named cactus-pestis, with the specified cloud provider to store intermediate job files and metadata. NOTE: If you want to use a GCE-based jobstore, specify google here, not gce.

            The result file, named pestis_output3.hal, is stored under /root/cact_ex folder of the leader node.

            Use cactus --help to see all the Cactus and Toil flags available.

    8.  Log out of the leader node:

                (cact_venv) $ exit

    9.  Download the resulted output to local machine:

                (venv) $ toil rsync-cluster \
                             --provisioner <aws, gce> <cluster-name> \
                             :/root/cact_ex/pestis_output3.hal \
                             <path-of-folder-on-local-machine>

    10. Destroy the cluster:

                (venv) $ toil destroy-cluster --provisioner <aws, gce> <cluster-name>

CWL IN TOIL

The Common Workflow Language (CWL) is an emerging standard for writing workflows that are portable across multiple workflow engines and platforms. Toil has full support for the CWL v1.0, v1.1, and v1.2 standards.

You can use Toil to run CWL workflows <#runcwl> or develop and test new ones.

RUNNING CWL WORKFLOWS

The toil-cwl-runner command provides CWL parsing functionality using cwltool, and leverages the job-scheduling and batch system support of Toil. You can use it to run CWL workflows locally or in the cloud.

  Running CWL Locally

To run in local batch mode, provide the CWL file and the input object file:

            $ toil-cwl-runner example.cwl example-job.yml

For a simple example of CWL with Toil see Running a basic CWL workflow <#cwlquickstart>.

  Note for macOS + Docker + Toil

When invoking CWL documents that make use of Docker containers if you see errors that look like

            docker: Error response from daemon: Mounts denied:
            The paths /var/...tmp are not shared from OS X and are not known to Docker.

you may need to add

            export TMPDIR=/tmp/docker_tmp

either in your startup file (.bashrc) or add it manually in your shell before invoking toil.

  Detailed Usage Instructions

Help information can be found by using this toil command:

            $ toil-cwl-runner -h

A more detailed example shows how we can specify both Toil and cwltool arguments for our workflow:

            $ toil-cwl-runner \
                --singularity \
                --jobStore my_jobStore \
                --batchSystem lsf \
                --workDir `pwd` \
                --outdir `pwd` \
                --logFile cwltoil.log \
                --writeLogs `pwd` \
                --logLevel DEBUG \
                --retryCount 2 \
                --maxLogFileSize 20000000000 \
                --stats \
                standard_bam_processing.cwl \
                inputs.yaml

In this example, we set the following options, which are all passed to Toil:

--singularity: Specifies that all jobs with Docker format containers specified should be run using the Singularity container engine instead of the Docker container engine.

--jobStore: Path to a folder which doesn't exist yet, which will contain the Toil jobstore and all related job-tracking information.

--batchSystem: Use the specified HPC or Cloud-based cluster platform.

--workDir: The directory where all temporary files will be created for the workflow. A subdirectory of this will be set as the $TMPDIR environment variable and this subdirectory can be referenced using the CWL parameter reference $(runtime.tmpdir) in CWL tools and workflows.

--outdir: Directory where final File and Directory outputs will be written. References to these and other output types will be in the JSON object printed to the stdout stream after workflow execution.

--logFile: Path to the main logfile.

--writeLogs: Directory where job logs will be stored. At DEBUG log level, this will contain logs for each Toil job run, as well as stdout/stderr logs for each CWL CommandLineTool that didn't use the stdout/stderr directives to redirect output.

--retryCount: How many times to retry each Toil job.

--maxLogFileSize: Logs that get larger than this value will be truncated.

--stats: Save resources usages in json files that can be collected with the toil stats command after the workflow is done.

  Extra Toil CWL Options

Besides the normal Toil options and the options supported by cwltool, toil-cwl-runner adds some of its own options:

            --bypass-file-store
                Do not use Toil's file store system and assume all paths are accessible in place from all nodes. This can avoid possibly-redundant file copies into Toil's job store storage, and is required for CWL's InplaceUpdateRequirement. But, it allows a failed job execution to leave behind a partially-modified state, which means that a restarted workflow might not work correctly.

            --reference-inputs
                Do not copy remote inputs into Toil's file store and assume they are accessible in place from all nodes.

            --disable-streaming
                Do not allow streaming of job input files. By default, files marked with streamable True are streamed from remote job stores.

            --cwl-default-ram
                Apply CWL specification default ramMin.

            --no-cwl-default-ram
                Do not apply CWL specification default ramMin, so that Toil --defaultMemory applies. This can help jobs get to Slurm with no memory limit assigned.

            --cwl-min-ram BYTES
                Specify a minimum memory allocation for all tasks. If --no-cwl-default-ram is passed, this does not apply to tools that do not specify a memory requirement; --defaultMemory is used for those tools in that case.

  Running CWL in the Cloud

To run in cloud and HPC configurations, you may need to provide additional command line parameters to select and configure the batch system to use.

To run a CWL workflow in AWS with toil see Running a CWL Workflow on AWS <#awscwl>.

  Running CWL workflows with InplaceUpdateRequirement

Some CWL workflows use the InplaceUpdateRequirement feature, which requires that operations on files have visible side effects that Toil's file store cannot support. If you need to run a workflow like this, you can make sure that all of your worker nodes have a shared filesystem, and use the --bypass-file-store option to toil-cwl-runner. This will make it leave all CWL intermediate files on disk and share them between jobs using file paths, instead of storing them in the file store and downloading them when jobs need them.

  Toil & CWL Tips

See logs for just one job by using the full log file

This requires knowing the job's toil-generated ID, which can be found in the log files.

            cat cwltoil.log | grep jobVM1fIs

Grep for full tool commands from toil logs

This gives you a more concise view of the commands being run (note that this information is only available from Toil when running with --logDebug).

            pcregrep -M "\[job .*\.cwl.*$\n(.*        .*$\n)*" cwltoil.log
            #         ^allows for multiline matching

Find Bams that have been generated for specific step while pipeline is running:

            find . | grep -P '^./out_tmpdir.*_MD\.bam$'

See what jobs have been run

            cat log/cwltoil.log | grep -oP "\[job .*.cwl\]" | sort | uniq

or:

            cat log/cwltoil.log | grep -i "issued job"

Get status of a workflow

            $ toil status /home/johnsoni/TEST_RUNS_3/TEST_run/tmp/jobstore-09ae0acc-c800-11e8-9d09-70106fb1697e
            <hostname> 2018-10-04 15:01:44,184 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
            <hostname> 2018-10-04 15:01:44,185 MainThread INFO toil.utils.toilStatus: Parsed arguments
            <hostname> 2018-10-04 15:01:47,081 MainThread INFO toil.utils.toilStatus: Traversing the job graph gathering jobs. This may take a couple of minutes.

            Of the 286 jobs considered, there are 179 jobs with children, 107 jobs ready to run, 0 zombie jobs, 0 jobs with services, 0 services, and 0 jobs with log files currently in file:/home/user/jobstore-09ae0acc-c800-11e8-9d09-70106fb1697e.

Toil Stats

You can get run statistics broken down by CWL file. This only works once the workflow is finished:

            $ toil stats /path/to/jobstore

This will report resource usage information for all the CWL jobs executed by the workflow.

See Stats Command <#cli-stats> for an explanation of what the different fields mean.

Understanding toil log files

There is a worker_log.txt file for each Toil job. This file is written to while the job is running, and uploaded at the end if the job finishes or if running at debug log level. If uploaded, the contents are printed to the main log file and transferred to a log file in the --logDir folder.

The new log file will be named something like:

            CWLJob_<name of the CWL job>_<attempt number>.log

Standard output/error files will be named like:

            <name of the CWL job>.stdout_<attempt number>.log

If you have a workflow revsort.cwl which has a step rev which calls the tool revtool.cwl, the CWL job name ends up being all those parts strung together with .: revsort.cwl.rev.revtool.cwl.

CWL CONFORMANCE

Note: To fully support inplace_update, --bypass-file-store should be enabled. See Running CWL workflows with InplaceUpdateRequirement <#inplaceupdaterequirement>.

The badges below show Toil's support for different aspects of the CWL specification. They describe the current development version of Toil, not necessarily the release documented here.

  CWL 1.2 Required Features

[image: required] [image]

  CWL 1.2 Optional Features

[image: command_line_tool] [image]

[image: conditional] [image]

[image: docker] [image]

[image: expression_tool] [image]

[image: format_checking] [image]

[image: initial_work_dir] [image]

[image: inline_javascript] [image]

[image: inplace_update] [image]

[image: input_object_requirements] [image]

[image: json_schema_invalid] [image]

[image: load_listing] [image]

[image: multiple_input] [image]

[image: networkaccess] [image]

[image: resource] [image]

[image: scatter] [image]

[image: schema_def] [image]

[image: step_input] [image]

[image: timelimit] [image]

[image: workflow] [image]

[image: work_reuse] [image]

WDL IN TOIL

The Workflow Description Language <https://openwdl.org/> (WDL) is a programming language designed for writing workflows that execute a set of tasks in a pipeline distributed across multiple computers. Workflows enable scientific analyses to be reproducible, by wrapping up a whole sequence of commands, whose outputs feed into other commands, into a workflow that can be executed the same way every time.

Toil can be used to run <#runwdl> and to develop <#devwdl> WDL workflows. The Toil team also maintains a set of WDL conformance tests <#conformancewdl> for evaluating Toil and other WDL runners.

RUNNING WDL WORKFLOWS

Toil has beta support for running WDL workflows, using the toil-wdl-runner command. This command comes with the [wdl] extra; see Installing Toil with Extra Features <#extras> for how to install it if you do not have it.

You can run WDL workflows with toil-wdl-runner. Currently, toil-wdl-runner works by using MiniWDL <https://github.com/chanzuckerberg/miniwdl/#miniwdl> to parse and interpret the WDL workflow, and has support for workflows in WDL 1.0 or later (which are required to declare a version, and which use inputs and outputs sections).

Tip:
        The last release of Toil that supported unversioned, draft-2 WDL workflows was 5.12.0 <https://github.com/DataBiosphere/toil/releases/tag/releases%2F5.12.0>.

Toil is, for compatible workflows, a drop-in replacement for the Cromwell <https://github.com/broadinstitute/cromwell#readme> WDL runner. Instead of running a workflow with Cromwell:

            java -jar Cromwell.jar run myWorkflow.wdl --inputs myWorkflow_inputs.json

You can run the workflow with toil-wdl-runner:

            toil-wdl-runner myWorkflow.wdl --input myWorkflow_inputs.json

(We're here running Toil with --input, but it can also accept the Cromwell-style --inputs.)

This will default to executing on the current machine, with a job store in an automatically determined temporary location, but you can add a few Toil options to use other Toil-supported batch systems, such as Kubernetes:

            toil-wdl-runner --jobStore aws:us-west-2:wdl-job-store --batchSystem kubernetes myWorkflow.wdl --input myWorkflow_inputs.json

For Toil, the --input is optional, and inputs can be passed as a positional argument:

            toil-wdl-runner myWorkflow.wdl myWorkflow_inputs.json

You can also run workflows from URLs. For example, to run the MiniWDL self test workflow, you can do:

            toil-wdl-runner https://raw.githubusercontent.com/DataBiosphere/toil/36b54c45e8554ded5093bcdd03edb2f6b0d93887/src/toil/test/wdl/miniwdl_self_test/self_test.wdl https://raw.githubusercontent.com/DataBiosphere/toil/36b54c45e8554ded5093bcdd03edb2f6b0d93887/src/toil/test/wdl/miniwdl_self_test/inputs.json

  Toil WDL Runner Options

--jobStore: Specifies where to keep the Toil state information while running the workflow. Must be accessible from all machines.

-o or --outputDirectory: Specifies the output folder or URI prefix to save workflow output files in. Defaults to a new directory in the current directory.

-m or --outputFile: Specifies a JSON file name or URI to save workflow output values at. Defaults to standard output.

-i, --input, or --inputs: Alternative to the positional argument for the input JSON file, for compatibility with other WDL runners.

--outputDialect: Specifies an output format dialect. Can be cromwell to just return the workflow's output values as JSON or miniwdl to nest that under an outputs key and includes a dir key.

--referenceInputs: Specifies whether input files to Toil should be passed around by URL reference instead of being imported into Toil's storage. Defaults to off. Can be True or False or other similar words.

--container: Specifies the container engine to use to run tasks. By default this is auto, which tries Singularity if it is installed and Docker if it isn't. Can also be set to docker or singularity explicitly.

--allCallOutputs: Specifies whether outputs from all calls in a workflow should be included alongside the outputs from the output section, when an output section is defined. For strict WDL spec compliance, should be set to False. Usually defaults to False. If the workflow includes metadata for the Cromwell Output Organizer (croo) <https://github.com/ENCODE-DCC/croo>, will default to True.

--strict: Specifies whether Toil should immediately exit on a lint warning. By default, this is false.

--quantCheck: Specifies whether quantifier validation type checking should be enabled. Disabling this is useful for workflows which Cromwell can run but don't fully comply with the WDL spec. By default, this is true.

--runImportsOnWorkers: Run file imports on workers. This is useful if the leader is not network optimized and lots of downloads are necessary. By default, this is false.

--importWorkersBatchSize: Requires --runImportsOnWorkers to be true. Specify the target batch size in bytes for batched imports. As many files as can fit will go into each batch import job. This also accepts abbreviations, such as G or Gi.

--importWorkersDisk: Requires --runImportsOnWorkers to be true. Specify the disk size each import worker will get. This usually will not need to be set as Toil will attempt to use file streaming when downloading files. If not possible, for example, when downloading from AWS to a GCE job store, this should be set to the largest file size of all files to import. By default, this is 1 MiB.

Any number of other Toil options may also be specified. For defined Toil options, see Commandline Options <#commandref>.

  Managing Workflow Logs

At the default settings, if a WDL task succeeds, the standard output and standard error will be printed in the toil-wdl-runner output, unless they are captured by the workflow (with the stdout() and stderr() WDL built-in functions). If a WDL task fails, they will be printed whether they were meant to be captured or not. Complete logs from Toil for failed jobs will also be printed.

If you would like to save the logs organized by WDL task, you can use the --writeLogs or --writeLogsGzip options to specify a directory where the log files should be saved. Log files will be named after the same dotted, hierarchical workflow and task names used to set values from the input JSON, except that scatters will add an additional numerical component. In addition to the logs for WDL tasks, Toil job logs for failed jobs will also appear here when running at the default log level.

For example, if you run:

            toil-wdl-runner --writeLogs logs https://raw.githubusercontent.com/DataBiosphere/toil/36b54c45e8554ded5093bcdd03edb2f6b0d93887/src/toil/test/wdl/miniwdl_self_test/self_test.wdl https://raw.githubusercontent.com/DataBiosphere/toil/36b54c45e8554ded5093bcdd03edb2f6b0d93887/src/toil/test/wdl/miniwdl_self_test/inputs.json

You will end up with a logs/ directory containing:

            hello_caller.0.hello.stderr_000.log
            hello_caller.1.hello.stderr_000.log
            hello_caller.2.hello.stderr_000.log

The final number is a sequential counter: if a step has to be retried, or if you run the workflow multiple times without clearing out the logs directory, it will increment.

  Enabling WDL Call Cache

Toil can cache the task and workflow outputs to use outputs of already ran tasks and workflows. This can save time when debugging long running workflows where a later task fails. However, this is only guaranteed for running locally and can use up a considerable amount of disk space.

To use, set the following environment variables before running the workflow:

            export MINIWDL__CALL_CACHE__PUT=True
            export MINIWDL__CALL_CACHE__GET=True
            export MINIWDL__CALL_CACHE__DIR=/absolute_path/to/cache

The path to the cache directory must be an absolute path.

For setting up call cache permanently, see the MiniWDL call cache <https://miniwdl.readthedocs.io/en/latest/runner_reference.html#call-cache> documentation.

DEVELOPING WDL WORKFLOWS

Toil can be used as a development tool for writing and locally testing WDL workflows. These workflows can then be run on Toil against a cloud or cluster backend, or used with other WDL implementations such as Terra <https://support.terra.bio/hc/en-us/sections/360004147011-Workflows>, Cromwell <https://github.com/broadinstitute/cromwell#readme>, or MiniWDL <https://github.com/chanzuckerberg/miniwdl/#miniwdl>.

  Learning WDL

To learn to write WDL workflows in general, you should take the WDL Tutorial <#tutorialwdl>.

There are other tutorials available in the resources section <resourcesWdl>.

  Debugging WDL Workflows

When a WDL workflow breaks, many of the general Toil debugging strategies <debugging>, like identifying failed jobs with toil status --failed or running individual jobs locally with toil debug-job, can be useful. These are easier if you use a manually specified job store, by passing --jobStore to toil-wdl-runner, so that you have the job store path handy for the debugging tools.

There are also WDL-specific debugging strategies.

  Reading WDL Workflow Logs

When a WDL workflow fails, you are likely to see a message like this:

            WDL.runtime.error.CommandFailed: task command failed with exit status 1
            [2023-07-16T16:23:54-0700] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host phoenix-15.prism

This means that the command line command specified by one of your WDL tasks exited with a failing (i.e. nonzero) exit code, which will happen when either the command line command is written wrong, or when the error detection code in the tool you are trying to run detects and reports an error.

Go up higher in the log until you find lines that look like:

            [2024-01-16T20:12:19-0500] [Thread-3 (statsAndLoggingAggregator)] [I] [toil.statsAndLogging] hello_caller.0.hello.stderr follows:

And:

            [2024-01-16T20:12:19-0500] [Thread-3 (statsAndLoggingAggregator)] [I] [toil.statsAndLogging] hello_caller.0.hello.stdout follows:

These will be followed by the standard error and standard output log data from the task's command. There may be useful information (such as an error message from the underlying tool) in there.

If you would like individual task logs to be saved separately for later reference, you can use the --writeLogs option to specify a directory to store them. For more information, see Managing Workflow Logs <#logging>.

  Finding Uploaded Files

If you want to find files that were uploaded from a WDL job, look for lines like this in the job's debug log <debuggingLog>:

            [2023-07-16T15:58:39-0700] [MainThread] [D] [toil.wdl.wdltoil] Virtualized /data/tmp/2846b6012e3e5535add03b363950dd78/cb23/197c/work/bamPerChrs/Sample.chr14.bam as WDL file toilfile:2703483274%3A0%3Afiles%2Ffor-job%2Fkind-WDLTaskJob%2Finstance-b4c5x6hq%2Ffile-c4e4f1b16ddf4c2ab92c2868421f3351%2FSample.chr14.bam/Sample.chr14.bam

If you are using a file job store, you can grab that long URI:

            toilfile:2703483274%3A0%3Afiles%2Ffor-job%2Fkind-WDLTaskJob%2Finstance-b4c5x6hq%2Ffile-c4e4f1b16ddf4c2ab92c2868421f3351%2FSample.chr14.bam/Sample.chr14.bam

Then URL-decode it with, for example, <https://www.urldecoder.io/>, getting this:

            toilfile:2703483274:0:files/for-job/kind-WDLTaskJob/instance-b4c5x6hq/file-c4e4f1b16ddf4c2ab92c2868421f3351/Sample.chr14.bam/Sample.chr14.bam

Then you can take the part after the last colon:

            files/for-job/kind-WDLTaskJob/instance-b4c5x6hq/file-c4e4f1b16ddf4c2ab92c2868421f3351/Sample.chr14.bam/Sample.chr14.bam

That is the path relative to the job store directory where this file can be found.

Another approach would be to use find -iname 'Sample.chr14.bam' on a file job store, if you know the name the file had when it was uploaded.

  Workflow Authoring Tips

Here are some tips for writing WDL workflows. You can also consult the OpenWDL Cookbook <https://github.com/openwdl/cookbook>.

  Deleting Files

WDL doesn't have a built-in way to delete files; if you run a task that deletes a file, it will still exist in Toil's job store storage.

Toil recently gained support for deleting files at the end of WDL workflows. If you have a large file that you only need for part of your workflow, consider writing the part that creates and uses it as a separate child workflow and invoking it with call. Then the file will be cleaned up when the child workflow ends, leaving more space for files created in the parent workflow.

  External WDL Resources

Here are some other resources a WDL developer might find useful.

  Official WDL Tutorials

You can also learn to write WDL workflows for Toil by following the official WDL quickstart guide <https://docs.openwdl.org/getting-started/quickstart.html>.

Once you have your workflow .wdl file and your input .json, you can run the workflow on the inputs with toil-wdl-runner:

            toil-wdl-runner myWorkflow.wdl --input myWorkflow_inputs.json

  Learn WDL Video Tutorials

For people who prefer video tutorials, Lynn Langit has a Learn WDL Video Course <https://www.youtube.com/playlist?list=PL4Q4HssKcxYv5syJKUKRrD8Fbd-_CnxTM> that will teach you how to write and run WDL workflows. The course is taught using Cromwell, but Toil should also be compatible with the course's workflows.

  OpenWDL Cookbook

The OpenWDL Cookbook <https://github.com/openwdl/cookbook> contains example solutions to a variety of tricky workflow authoring problems.

  WDL Specifications

WDL language specifications can be found here: <https://github.com/openwdl/wdl/blob/main/versions/1.1/SPEC.md>

Toil is not yet fully conformant with the WDL specification (see WDL Conformance <#conformancewdl>), but it inherits most of the functionality of MiniWDL <https://github.com/chanzuckerberg/miniwdl/#miniwdl>.

  UCSC Genomics Institute Wiki

The UCSC Genomics Institute (home of the Toil project) has tips on debugging WDL workflows with Toil <https://giwiki.gi.ucsc.edu/index.php?title=Phoenix_WDL_Tutorial#Debugging_Workflows>. They are designed for one particular Slurm cluster environment, but might be applicable elsewhere.

WDL CONFORMANCE

The Toil team maintains a set of WDL Conformance Tests <https://github.com/DataBiosphere/wdl-conformance-tests>. Much like the CWL Conformance Tests <https://github.com/common-workflow-language/cwl-v1.2/blob/main/CONFORMANCE_TESTS.md> for CWL, the WDL Conformance Tests are useful for determining if a WDL implementation actually follows the WDL specification <https://github.com/openwdl/wdl/blob/cb875867d86f868fa08f6eb2be179a50097ba440/versions/1.1/SPEC.md>.

The WDL Conformance Tests include a runner harness that is able to test toil-wdl-runner, as well as Cromwell and MiniWDL, and supports testing conformance with the 1.1, 1.0, and draft-2 versions of WDL.

If you would like to evaluate Toil's WDL conformance for yourself, first make sure that you have toil-wdl-runner installed. It comes with the [wdl] extra; see Installing Toil with Extra Features <#extras>.

Then, you can check out the test repository:

            $ git clone https://github.com/DataBiosphere/wdl-conformance-tests
            $ cd wdl-conformance-tests

Most tests will need a Docker daemon available, so make sure yours is working properly:

            $ docker info
            $ docker run --rm docker/whalesay cowsay "Docker is working"

Then, you can test toil-wdl-runner against a particular WDL spec version, say 1.1:

            $ python3 run.py --runner toil-wdl-runner --versions 1.1

For any failed tests, the test number and the log of the failing test will be reported.

After the tests run, you can clean up intermediate files with:

            $ make clean

For more options, see:

            $ python3 run.py --help

Or, consult the conformance test documentation <https://github.com/DataBiosphere/wdl-conformance-tests/tree/master#wdl-workflow-description-language-spec-conformance-tests>.

WDL TUTORIAL

This tutorial will walk you through writing a workflow in WDL that you can run with Toil <runWdl>. We're going to write a workflow for Fizz Buzz <https://en.wikipedia.org/wiki/Fizz_buzz>.

  Writing the File

Let's go step by step through the process of creating a single-file WDL workflow.

  Version

All WDL files need to start with a version statement (unless they are very old draft-2 files). Toil supports draft-2, WDL 1.0, and WDL 1.1, while Cromwell (another popular WDL runner used on Terra) supports only draft-2 and 1.0.

So let's start a new WDL 1.0 workflow. Open up a file named fizzbuzz.wdl and start with a version statement:

            version 1.0

  Workflow Block

Then, add an empty workflow named FizzBuzz.

            version 1.0
            workflow FizzBuzz {
            }

  Input Block

Workflows usually need some kind of user input, so let's give our workflow an input section.

            version 1.0
            workflow FizzBuzz {
                input {
                    # How many FizzBuzz numbers do we want to make?
                    Int item_count
                    # Every multiple of this number, we produce "Fizz"
                    Int to_fizz = 3
                    # Every multiple of this number, we produce "Buzz"
                    Int to_buzz = 5
                    # Optional replacement for the string to print when a multiple of both
                    String? fizzbuzz_override
                }
            }

Notice that each input has a type, a name, and an optional default value. If the type ends in ?, the value is optional, and it may be null. If an input is not optional, and there is no default value, then the user's inputs file must specify a value for it in order for the workflow to run.

  Body

Now we'll start on the body of the workflow, to be inserted just after the inputs section.

The first thing we're going to need to do is create an array of all the numbers up to the item_count. We can do this by calling the WDL range() function, and assigning the result to an Array[Int] variable.

            Array[Int] numbers = range(item_count)

WDL 1.0 has a wide variety of functions in its standard library <https://github.com/openwdl/wdl/blob/main/versions/1.0/SPEC.md#standard-library>, and WDL 1.1 has even more.

  Scattering

Once we create an array of all the numbers, we can use a scatter to operate on each. WDL does not have loops; instead it has scatters, which work a bit like a map() in Python. The body of the scatter runs for each value in the input array, all in parallel. We're going to increment all the numbers, since FizzBuzz starts at 1 but WDL range() starts at 0.

            Array[Int] numbers = range(item_count)
            scatter (i in numbers) {
                Int one_based = i + 1
            }

  Conditionals

Inside the body of the scatter, we are going to put some conditionals to determine if we should produce "Fizz", "Buzz", or "FizzBuzz". To support our fizzbuzz_override, we use an array of it and a default value, and use the WDL select_first() function to find the first non-null value in that array.

Each execution of a scatter is allowed to declare variables, and outside the scatter those variables are combined into arrays of all the results. But each variable can be declared only once in the scatter, even with conditionals. So we're going to use select_first() at the end and take advantage of variables from un-executed conditionals being null.

Note that WDL supports conditional expressions with a then and an else, but conditional statements only have a body, not an else branch. If you need an else you will have to check the negated condition.

So first, let's handle the special cases.

            Array[Int] numbers = range(item_count)
            scatter (i in numbers) {
               Int one_based = i + 1
               if (one_based % to_fizz == 0) {
                   String fizz = "Fizz"
                   if (one_based % to_buzz == 0) {
                       String fizzbuzz = select_first([fizzbuzz_override, "FizzBuzz"])
                   }
               }
               if (one_based % to_buzz == 0) {
                   String buzz = "Buzz"
               }
               if (one_based % to_fizz != 0 && one_based % to_buzz != 0) {
                   # Just a normal number.
               }
            }

  Calling Tasks

Now for the normal numbers, we need to convert our number into a string. In WDL 1.1, and in WDL 1.0 on Cromwell, you can use a ${} substitution syntax in quoted strings anywhere, not just in command line commands. Toil technically will support this too, but it's not in the spec, and the tutorial needs an excuse for you to call a task. So we're going to insert a call to a stringify_number task, to be written later.

To call a task (or another workflow), we use a call statement and give it some inputs. Then we can fish the output values out of the task with . access, only if we don't make a noise instead.

            Array[Int] numbers = range(item_count)
            scatter (i in numbers) {
               Int one_based = i + 1
               if (one_based % to_fizz == 0) {
                   String fizz = "Fizz"
                   if (one_based % to_buzz == 0) {
                       String fizzbuzz = select_first([fizzbuzz_override, "FizzBuzz"])
                   }
               }
               if (one_based % to_buzz == 0) {
                   String buzz = "Buzz"
               }
               if (one_based % to_fizz != 0 && one_based % to_buzz != 0) {
                   # Just a normal number.
                   call stringify_number {
                       input:
                           the_number = one_based
                   }
               }
               String result = select_first([fizzbuzz, fizz, buzz, stringify_number.the_string])
            }

We can put the code into the workflow now, and set about writing the task.

            version 1.0
            workflow FizzBuzz {
                input {
                    # How many FizzBuzz numbers do we want to make?
                    Int item_count
                    # Every multiple of this number, we produce "Fizz"
                    Int to_fizz = 3
                    # Every multiple of this number, we produce "Buzz"
                    Int to_buzz = 5
                    # Optional replacement for the string to print when a multiple of both
                    String? fizzbuzz_override
                }
                Array[Int] numbers = range(item_count)
                scatter (i in numbers) {
                    Int one_based = i + 1

                    if (one_based % to_fizz == 0) {
                        String fizz = "Fizz"
                        if (one_based % to_buzz == 0) {
                            String fizzbuzz = select_first([fizzbuzz_override, "FizzBuzz"])
                        }
                     }
                    if (one_based % to_buzz == 0) {
                        String buzz = "Buzz"
                    }
                    if (one_based % to_fizz != 0 && one_based % to_buzz != 0) {
                        # Just a normal number.
                        call stringify_number {
                            input:
                                the_number = one_based
                        }
                    }
                    String result = select_first([fizzbuzz, fizz, buzz, stringify_number.the_string]
                }
            }

  Writing Tasks

Our task should go after the workflow in the file. It looks a lot like a workflow except it uses task.

            task stringify_number {
            }

We're going to want it to take in an integer the_number, and we're going to want it to output a string the_string. So let's fill that in in input and output sections.

            task stringify_number {
                input {
                    Int the_number
                }
                # ???
                output {
                    String the_string # = ???
                }
            }

Now, unlike workflows, tasks can have a command section, which gives a command to run. This section is now usually set off with triple angle brackets, and inside it you can use ~{}, that is, Bash-like substitution but with a tilde, to place WDL variables into your command script. So let's add a command that will echo back the number so we can see it as a string.

            task stringify_number {
                input {
                    Int the_number
                }
                command <<<
                   # This is a Bash script.
                   # So we should do good Bash script things like stop on errors
                   set -e
                   # Now print our number as a string
                   echo ~{the_number}
                >>>
                output {
                    String the_string # = ???
                }
            }

Now we need to capture the result of the command script. The WDL stdout() returns a WDL File containing the standard output printed by the task's command. We want to read that back into a string, which we can do with the WDL read_string() function (which also removes trailing newlines <https://github.com/openwdl/wdl/blob/main/versions/1.0/SPEC.md#string-read_stringstringfile>).

            task stringify_number {
                input {
                    Int the_number
                }
                command <<<
                   # This is a Bash script.
                   # So we should do good Bash script things like stop on errors
                   set -e
                   # Now print our number as a string
                   echo ~{the_number}
                >>>
                output {
                    String the_string = read_string(stdout())
                }
            }

We're also going to want to add a runtime section to our task, to specify resource requirements. We're also going to tell it to run in a Docker container, to make sure that absolutely nothing can go wrong with our delicate echo command. In a real workflow, you probably want to set up optiopnal inputs for all the tasks to let you control the resource requirements, but here we will just hardcode them.

            task stringify_number {
                input {
                    Int the_number
                }
                command <<<
                    # This is a Bash script.
                    # So we should do good Bash script things like stop on errors
                    set -e
                    # Now print our number as a string
                    echo ~{the_number}
                >>>
                output {
                    String the_string = read_string(stdout())
                }
                runtime {
                    cpu: 1
                    memory: "0.5 GB"
                    disks: "local-disk 1 SSD"
                    docker: "ubuntu:24.04"
                }
            }

The disks section is a little weird; it isn't in the WDL spec, but Toil supports Cromwell-style strings that ask for a local-disk of a certain number of gigabytes, which may suggest that it be SSD storage.

Then we can put our task into our WDL file:

            version 1.0
            workflow FizzBuzz {
                input {
                    # How many FizzBuzz numbers do we want to make?
                    Int item_count
                    # Every multiple of this number, we produce "Fizz"
                    Int to_fizz = 3
                    # Every multiple of this number, we produce "Buzz"
                    Int to_buzz = 5
                    # Optional replacement for the string to print when a multiple of both
                    String? fizzbuzz_override
                }
                Array[Int] numbers = range(item_count)
                scatter (i in numbers) {
                    Int one_based = i + 1
                    if (one_based % to_fizz == 0) {
                        String fizz = "Fizz"
                        if (one_based % to_buzz == 0) {
                            String fizzbuzz = select_first([fizzbuzz_override, "FizzBuzz"])
                        }
                     }
                    if (one_based % to_buzz == 0) {
                        String buzz = "Buzz"
                    }
                    if (one_based % to_fizz != 0 && one_based % to_buzz != 0) {
                        # Just a normal number.
                        call stringify_number {
                            input:
                                the_number = one_based
                        }
                    }
                    String result = select_first([fizzbuzz, fizz, buzz, stringify_number.the_string]
                }
            }
            task stringify_number {
                input {
                    Int the_number
                }
                command <<<
                    # This is a Bash script.
                    # So we should do good Bash script things like stop on errors
                    set -e
                    # Now print our number as a string
                    echo ~{the_number}
                >>>
                output {
                    String the_string = read_string(stdout())
                }
                runtime {
                    cpu: 1
                    memory: "0.5 GB"
                    disks: "local-disk 1 SSD"
                    docker: "ubuntu:24.04"
                }
            }

  Output Block

Now the only thing missing is a workflow-level output section. Technically, in WDL 1.0 you aren't supposed to need this, but you do need it in 1.1 and Toil doesn't actually send your outputs anywhere yet if you don't have one, so we're going to make one. We need to collect together all the strings that came out of the different tasks in our scatter into an Array[String]. We'll add the output section at the end of the workflow section, above the task.

            version 1.0
            workflow FizzBuzz {
                input {
                    # How many FizzBuzz numbers do we want to make?
                    Int item_count
                    # Every multiple of this number, we produce "Fizz"
                    Int to_fizz = 3
                    # Every multiple of this number, we produce "Buzz"
                    Int to_buzz = 5
                    # Optional replacement for the string to print when a multiple of both
                    String? fizzbuzz_override
                }
                Array[Int] numbers = range(item_count)
                scatter (i in numbers) {
                    Int one_based = i + 1
                    if (one_based % to_fizz == 0) {
                        String fizz = "Fizz"
                        if (one_based % to_buzz == 0) {
                            String fizzbuzz = select_first([fizzbuzz_override, "FizzBuzz"])
                        }
                     }
                    if (one_based % to_buzz == 0) {
                        String buzz = "Buzz"
                    }
                    if (one_based % to_fizz != 0 && one_based % to_buzz != 0) {
                        # Just a normal number.
                        call stringify_number {
                            input:
                                the_number = one_based
                        }
                    }
                    String result = select_first([fizzbuzz, fizz, buzz, stringify_number.the_string]
                }
                output {
                   Array[String] fizzbuzz_results = result
                }
            }
            task stringify_number {
                input {
                    Int the_number
                }
                command <<<
                    # This is a Bash script.
                    # So we should do good Bash script things like stop on errors
                    set -e
                    # Now print our number as a string
                    echo ~{the_number}
                >>>
                output {
                    String the_string = read_string(stdout())
                }
                runtime {
                    cpu: 1
                    memory: "0.5 GB"
                    disks: "local-disk 1 SSD"
                    docker: "ubuntu:24.04"
                }
            }

Because the result variable is defined inside a scatter, when we reference it outside the scatter we see it as being an array.

  Running the Workflow

Now all that remains is to run the workflow! Make an inputs file to specify the workflow inputs:

            echo '{"FizzBuzz.item_count": 20}' >fizzbuzz.json

Then run it with Toil. If you are on a Slurm cluster, and you are currently in a shared directory available on all your nodes, you can run:

            toil-wdl-runner --jobStore ./fizzbuzz_store --batchSystem slurm --slurmTime 00:10:00 --caching false --batchLogsDir ./logs fizzbuzz.wdl fizzbuzz.json -o fizzbuzz_out -m fizzbuzz_out.json

If instead you want to run your workflow locally, you can run:

            toil-wdl-runner fizzbuzz.wdl fizzbuzz.json -o fizzbuzz_out -m fizzbuzz_out.json

  Next Steps

    •   Try breaking your workflow up into multiple files and using import statements.

    •   Publish your workflow on Dockstore <https://docs.dockstore.org/en/stable/getting-started/dockstore-workflows.html>.

    •   Read up on Toil-specific WDL development considerations in Developing WDL Workflows <#devwdl>.

INTRODUCTION

Toil runs in various environments, including locally and in the cloud <#cloudoverview> (Amazon Web Services and Google Compute Engine). Toil also supports workflows written in two DSLs: CWL <#cwl> and WDL <#wdl>, as well as workflows written in Python (see Developing a Python Workflow <#devpython>).

Toil is built in a modular way so that it can be used on lots of different systems, and with different configurations. The three configurable pieces are the

            •   Job Store: A filepath or url that can host and centralize all files for a workflow (e.g. a local folder, or an AWS s3 bucket url).

            •   Batch System: Specifies either a local single-machine or a currently supported HPC environment (lsf, mesos, slurm, torque, htcondor, kubernetes, or grid_engine).

            •   Provisioner: For running in the cloud only. This specifies which cloud provider provides instances to do the "work" of your workflow.

  Job Store

The job store is a storage abstraction which contains all of the information used in a Toil run. This centralizes all of the files used by jobs in the workflow and also the details of the progress of the run. If a workflow crashes or fails, the job store contains all of the information necessary to resume with minimal repetition of work.

Several different job stores are supported, including the file job store and cloud job stores. For information on developing job stores, see Job Store API <#jobstoreinterface>.

  File Job Store

The file job store is for use locally, and keeps the workflow information in a directory on the machine where the workflow is launched. This is the simplest and most convenient job store for testing or for small runs.

For an example that uses the file job store, see Running a basic CWL workflow <#quickstart>.

  Cloud Job Stores

Toil currently supports the following cloud storage systems as job stores:

            •   AWS Job Store <#awsjobstore>: An AWS S3 bucket formatted as "aws:<zone>:<bucketname>" where only numbers, letters, and dashes are allowed in the bucket name. Example: aws:us-west-2:my-aws-jobstore-name.

            •   Google Job Store <#googlejobstore>: A Google Cloud Storage bucket formatted as "gce:<zone>:<bucketname>" where only numbers, letters, and dashes are allowed in the bucket name. Example: gce:us-west2-a:my-google-jobstore-name.

These use cloud buckets to house all of the files. This is useful if there are several different worker machines all running jobs that need to access the job store.

  Batch System

A Toil batch system is either a local single-machine (one computer) or a currently supported cluster of computers (lsf, mesos, slurm, torque, htcondor, or grid_engine) These environments manage individual worker nodes under a leader node to process the work required in a workflow. The leader and its workers all coordinate their tasks and files through a centralized job store location.

See Batch System API <#batchsysteminterface> for a more detailed description of different batch systems, or information on developing batch systems.

  Provisioner

The Toil provisioner provides a tool set for running a Toil workflow on a particular cloud platform.

The Toil Cluster Utilities <#clusterutils> are command line tools used to provision nodes in your desired cloud platform. They allows you to launch nodes, ssh to the leader, and rsync files back and forth.

For detailed instructions for using the provisioner see Running in AWS <#runningaws> or Running in Google Compute Engine (GCE) <#runninggce>.

COMMANDLINE OPTIONS

A quick way to see all of Toil's commandline options is by executing the following on a workflow language front-end:

            $ toil-wdl-runner --help

Or a Toil Python workflow:

            $ python3 example.py --help

For a basic toil workflow, Toil has one mandatory argument, the job store. All other arguments are optional.

  The Config File

Instead of changing the arguments on the command line, Toil offers support for using a configuration file.

Options will be applied with priority:

            1.  Command line options

            2.  Environmental Variables

            3.  Config file values

                a.  Provided config file through --config

                b.  Default config value in $HOME/.toil/default.yaml

            4.  Defaults

You can manually generate an example configuration file to a path you select. To generate a configuration file, run:

            $ toil config [filename].yaml

Then uncomment options as necessary and change/provide new values.

After editing the config file, you can run Toil with its settings by passing it on the command line:

            $ python3 example.py --config=[filename].yaml

Alternatively, you can edit the default config file, which is located at $HOME/.toil/default.yaml

If CLI options are used in addition to the configuration file, the CLI options will overwrite the configuration file options. For example:

            $ python3 example.py --config=[filename].yaml --defaultMemory 80Gi

This will result in a default memory per job of 80GiB no matter what is in the configuration file provided.

  The Job Store

Running Toil workflows requires a file path or URL to a central location for all of the intermediate files for the workflow: the job store. For toil-cwl-runner and toil-wdl-runner a job store can often be selected automatically or can be specified with the --jobStore option; Toil Python workflows generally require the job store as a positional command line argument. To use the Python quickstart <#pyquickstart> example, if you're on a node that has a large /scratch volume, you can specify that the jobstore be created there by executing: python3 HelloWorld.py /scratch/my-job-store, or more explicitly, python3 HelloWorld.py file:/scratch/my-job-store.

Syntax for specifying different job stores:
        Local: file:job-store-name

        AWS: aws:region-here:job-store-name

        Google: google:projectID-here:job-store-name

Different types of job store options can be found below.

  Commandline Options

Core Toil Options Options to specify the location of the Toil workflow and turn on stats collation about the performance of jobs.

            --workDir WORKDIR
                Absolute path to directory where temporary files generated during the Toil run should be placed. Standard output and error from batch system jobs (unless --noStdOutErr is set) will be placed in this directory. A cache directory may be placed in this directory. Temp files and folders will be placed in a directory toil-<workflowID> within workDir. The workflowID is generated by Toil and will be reported in the workflow logs. Default is determined by the variables (TMPDIR, TEMP, TMP) via mkdtemp. For CWL, the temporary output directory is used instead (see CWL option --tmp-outdir-prefix). This directory needs to exist on all machines running jobs; if capturing standard output and error from batch system jobs is desired, it will generally need to be on a shared file system. When sharing a cache between containers on a host, this directory must be shared between the containers.

            --coordinationDir COORDINATION_DIR
                Absolute path to directory where Toil will keep state and lock files. When sharing a cache between containers on a host, this directory must be shared between the containers.

            --noStdOutErr
                Do not capture standard output and error from batch system jobs.

            --stats
                Records statistics about the toil workflow to be used by 'toil stats'.

            --clean=STATE
                Determines the deletion of the jobStore upon completion of the program. Choices: 'always', 'onError','never', or 'onSuccess'. The --stats option requires information from the jobStore upon completion so the jobStore will never be deleted with that flag. If you wish to be able to restart the run, choose 'never' or 'onSuccess'. Default is 'never' if stats is enabled, and 'onSuccess' otherwise

            --cleanWorkDir STATE
                Determines deletion of temporary worker directory upon completion of a job. Choices: 'always', 'onError', 'never', or 'onSuccess'. Default = always. WARNING: This option should be changed for debugging only. Running a full pipeline with this option could fill your disk with intermediate data.

            --clusterStats FILEPATH
                If enabled, writes out JSON resource usage statistics to a file. The default location for this file is the current working directory, but an absolute path can also be passed to specify where this file should be written. This option only applies when using scalable batch systems.

            --restart
                If --restart is specified then will attempt to restart existing workflow at the location pointed to by the --jobStore option. Will raise an exception if the workflow does not exist.

Logging Options Toil hides stdout and stderr by default except in case of job failure. Log levels in toil are based on priority from the logging module:

            --logOff
                Only CRITICAL log messages are shown. Equivalent to --logLevel=OFF or --logLevel=CRITICAL.

            --logCritical
                Only CRITICAL log messages are shown. Equivalent to --logLevel=OFF or --logLevel=CRITICAL.

            --logError
                Only ERROR, and CRITICAL log messages are shown. Equivalent to --logLevel=ERROR.

            --logWarning
                Only WARN, ERROR, and CRITICAL log messages are shown. Equivalent to --logLevel=WARNING.

            --logInfo
                All non-debugging-related log messages are shown. Equivalent to --logLevel=INFO.

            --logDebug
                Log messages at DEBUG level and above are shown. Equivalent to --logLevel=DEBUG.

            --logTrace
                Log messages at TRACE level and above are shown. Equivalent to --logLevel=TRACE.

            --logLevel=LOGLEVEL
                May be set to: OFF (or CRITICAL), ERROR, WARN (or WARNING), INFO, DEBUG, or TRACE.

            --logFile FILEPATH
                Specifies a file path to write the logging output to.

            --rotatingLogging
                Turn on rotating logging, which prevents log files from getting too big (set using --maxLogFileSize BYTESIZE).

            --maxLogFileSize BYTESIZE
                The maximum size of a job log file to keep (in bytes), log files larger than this will be truncated to the last X bytes. Setting this option to zero will prevent any truncation. Setting this option to a negative value will truncate from the beginning. Default=100MiB Sets the maximum log file size in bytes (--rotatingLogging must be active).

            --log-dir DIRPATH
                For CWL and local file system only. Log stdout and stderr (if tool requests stdout/stderr) to the DIRPATH.

            --logColors BOOL
                Enable or disable colored logging. Default=True.

Batch System Options

            --batchSystem BATCHSYSTEM
                The type of batch system to run the job(s) with. Default = single_machine.

            --disableAutoDeployment
                Should auto-deployment of Toil Python workflows be deactivated? If True, the workflow's Python code should be present at the same location on all workers. Default = False.

            --maxJobs MAXJOBS
                Specifies the maximum number of jobs to submit to the backing scheduler at once. Not supported on Mesos or AWS Batch. Use 0 for unlimited. Defaults to unlimited.

            --maxLocalJobs MAXLOCALJOBS
                Specifies the maximum number of housekeeping jobs to run simultaneously on the local system. Use 0 for unlimited. Defaults to the number of local cores.

            --manualMemArgs
                Do not add the default arguments: 'hv=MEMORY' & 'h_vmem=MEMORY' to the qsub call, and instead rely on TOIL_GRIDGENGINE_ARGS to supply alternative arguments. Requires that TOIL_GRIDGENGINE_ARGS be set.

            --memoryIsProduct
                If the batch system understands requested memory as a product of the requested memory and the number of cores, set this flag to properly allocate memory. This can be fairly common with grid engine clusters (Ex: SGE, PBS, Torque).

            --runCwlInternalJobsOnWorkers
                Whether to run CWL internal jobs (e.g. CWLScatter) on the worker nodes instead of the primary node. If false (default), then all such jobs are run on the primary node. Setting this to true can speed up the pipeline for very large workflows with many sub-workflows and/or scatters, provided that the worker pool is large enough.

            --statePollingWait STATEPOLLINGWAIT
                Time, in seconds, to wait before doing a scheduler query for job state. Return cached results if within the waiting period. Only works for grid engine batch systems such as gridengine, htcondor, torque, slurm, and lsf.

            --statePollingTimeout STATEPOLLINGTIMEOUT
                Time, in seconds, to retry against a broken scheduler. Only works for grid engine batch systems such as gridengine, htcondor, torque, slurm, and lsf.

            --batchLogsDir BATCHLOGSDIR
                Directory to tell the backing batch system to log into. Should be available on both the leader and the workers, if the backing batch system writes logs to the worker machines' filesystems, as many HPC schedulers do. If unset, the Toil work directory will be used. Only works for grid engine batch systems such as gridengine, htcondor, torque, slurm, and lsf.

            --mesosEndpoint MESOSENDPOINT
                The host and port of the Mesos server separated by a colon. (default: <leader IP>:5050)

            --mesosFrameworkId MESOSFRAMEWORKID
                Use a specific Mesos framework ID.

            --mesosRole MESOSROLE
                Use a Mesos role.

            --mesosName MESOSNAME
                The Mesos name to use. (default: toil)

            --scale SCALE
                A scaling factor to change the value of all submitted tasks' submitted cores. Used in single_machine batch system. Useful for running workflows on smaller machines than they were designed for, by setting a value less than 1. (default: 1)

            --slurmAllocateMem SLURM_ALLOCATE_MEM
                If False, do not use --mem. Used as a workaround for Slurm clusters that reject jobs with memory allocations.

            --slurmTime SLURM_TIME
                Slurm job time limit, in [DD-]HH:MM:SS format.

            --slurmPartition SLURM_PARTITION
                Partition to send Slurm jobs to.

            --slurmGPUPartition SLURM_GPU_PARTITION
                Partition to send Slurm jobs to if they ask for GPUs.

            --slurmPE SLURM_PE
                Special partition to send Slurm jobs to if they ask for more than 1 CPU. Useful for Slurm clusters that do not offer a partition accepting both single-core and multi-core jobs.

            --slurmArgs SLURM_ARGS
                Extra arguments to pass to Slurm.

            --kubernetesHostPath KUBERNETES_HOST_PATH
                Path on Kubernetes hosts to use as shared inter-pod temp directory.

            --kubernetesOwner KUBERNETES_OWNER
                Username to mark Kubernetes jobs with.

            --kubernetesServiceAccount KUBERNETES_SERVICE_ACCOUNT
                Service account to run jobs as.

            --kubernetesPodTimeout KUBERNETES_POD_TIMEOUT
                Seconds to wait for a scheduled Kubernetes pod to start running. (default: 120s)

            --kubernetesPrivileged BOOL
                Whether to allow Kubernetes pods to run as privileged. This can be used to enable FUSE mounts for faster runtimes with Singularity. When launching Toil-managed clusters, this will be set to true by --allowFuse. (default: False)

            --kubernetesPodSecurityContext KUBERNETES_POD_SECURITY_CONTEXT
                Path to a YAML defining a pod security context to apply to all pods.

            --kubernetesSecurityContext KUBERNETES_SECURITY_CONTEXT
                Path to a YAML defining a security context to apply to all containers.

            --awsBatchRegion AWS_BATCH_REGION
                The AWS region containing the AWS Batch queue to submit to.

            --awsBatchQueue AWS_BATCH_QUEUE
                The name or ARN of the AWS Batch queue to submit to.

            --awsBatchJobRoleArn AWS_BATCH_JOB_ROLE_ARN
                The ARN of an IAM role to run AWS Batch jobs as, so they can e.g. access a job store. Must be assumable by ecs-tasks.amazonaws.com

Data Storage Options Allows configuring Toil's data storage.

            --symlinkImports BOOL
                When using a filesystem based job store, CWL input files are by default symlinked in. Setting this option to True instead copies the files into the job store, which may protect them from being modified externally. When set to False and as long as caching is enabled, Toil will protect the file automatically by changing the permissions to read-only. (Default=True)

            --moveOutputs BOOL
                When using a filesystem based job store, output files are by default moved to the output directory, and a symlink to the moved exported file is created at the initial location. Setting this option to True instead copies the files into the output directory. Applies to filesystem-based job stores only. (Default=False)

            --caching BOOL
                Enable or disable worker level file caching. Set to "true" if caching is desired. By default, caching is enabled on supported batch systems. Does not affect CWL or WDL task caching.

            --symlinkJobStoreReads BOOL
                Allow reads and container mounts from a JobStore's shared filesystem directly via symlink. Can be turned off if the shared filesystem can't support the IO load of all the jobs reading from it at once, and you want to use --caching=True to make jobs on each node read from node-local cache storage. (Default=True)

Autoscaling Options Allows the specification of the minimum and maximum number of nodes in an autoscaled cluster, as well as parameters to control the level of provisioning.

            --provisioner CLOUDPROVIDER
                The provisioner for cluster auto-scaling. This is the main Toil --provisioner option, and defaults to None for running on single_machine and non-auto-scaling batch systems. The currently supported choices are 'aws' or 'gce'.

            --nodeTypes NODETYPES
                Specifies a list of comma-separated node types, each of which is composed of slash-separated instance types, and an optional spot bid set off by a colon, making the node type preemptible. Instance types may appear in multiple node types, and the same node type may appear as both preemptible and non-preemptible.

                Valid argument specifying two node types:
                    c5.4xlarge/c5a.4xlarge:0.42,t2.large

                Node types:
                    c5.4xlarge/c5a.4xlarge:0.42 and t2.large

                Instance types:
                    c5.4xlarge, c5a.4xlarge, and t2.large

                Semantics:
                    Bid $0.42/hour for either c5.4xlarge or c5a.4xlarge instances, treated interchangeably, while they are available at that price, and buy t2.large instances at full price

            --minNodes MINNODES
                Minimum number of nodes of each type in the cluster, if using auto-scaling. This should be provided as a comma-separated list of the same length as the list of node types. default=0

            --maxNodes MAXNODES
                Maximum number of nodes of each type in the cluster, if using autoscaling, provided as a comma-separated list. The first value is used as a default if the list length is less than the number of nodeTypes. default=10

            --targetTime TARGETTIME
                Sets how rapidly you aim to complete jobs in seconds. Shorter times mean more aggressive parallelization. The autoscaler attempts to scale up/down so that it expects all queued jobs will complete within targetTime seconds. (Default: 1800)

            --betaInertia BETAINERTIA
                A smoothing parameter to prevent unnecessary oscillations in the number of provisioned nodes. This controls an exponentially weighted moving average of the estimated number of nodes. A value of 0.0 disables any smoothing, and a value of 0.9 will smooth so much that few changes will ever be made. Must be between 0.0 and 0.9. (Default: 0.1)

            --scaleInterval SCALEINTERVAL
                The interval (seconds) between assessing if the scale of the cluster needs to change. (Default: 60)

            --preemptibleCompensation PREEMPTIBLECOMPENSATION
                The preference of the autoscaler to replace preemptible nodes with non-preemptible nodes, when preemptible nodes cannot be started for some reason. Defaults to 0.0. This value must be between 0.0 and 1.0, inclusive. A value of 0.0 disables such compensation, a value of 0.5 compensates two missing preemptible nodes with a non-preemptible one. A value of 1.0 replaces every missing pre-emptable node with a non-preemptible one.

            --nodeStorage NODESTORAGE
                Specify the size of the root volume of worker nodes when they are launched in gigabytes. You may want to set this if your jobs require a lot of disk space. The default value is 50.

            --nodeStorageOverrides NODESTORAGEOVERRIDES
                Comma-separated list of nodeType:nodeStorage that are used to override the default value from --nodeStorage for the specified nodeType(s). This is useful for heterogeneous jobs where some tasks require much more disk than others.

            --metrics
                Enable the prometheus/grafana dashboard for monitoring CPU/RAM usage, queue size, and issued jobs.

            --assumeZeroOverhead
                Ignore scheduler and OS overhead and assume jobs can use every last byte of memory and disk on a node when autoscaling.

Service Options Allows the specification of the maximum number of service jobs in a cluster. By keeping this limited we can avoid nodes occupied with services causing deadlocks. (Not for CWL).

            --maxServiceJobs MAXSERVICEJOBS
                The maximum number of service jobs that can be run concurrently, excluding service jobs running on preemptible nodes. default=9223372036854775807

            --maxPreemptibleServiceJobs MAXPREEMPTIBLESERVICEJOBS
                The maximum number of service jobs that can run concurrently on preemptible nodes. default=9223372036854775807

            --deadlockWait DEADLOCKWAIT
                Time, in seconds, to tolerate the workflow running only the same service jobs, with no jobs to use them, before declaring the workflow to be deadlocked and stopping. default=60

            --deadlockCheckInterval DEADLOCKCHECKINTERVAL
                Time, in seconds, to wait between checks to see if the workflow is stuck running only service jobs, with no jobs to use them. Should be shorter than --deadlockWait. May need to be increased if the batch system cannot enumerate running jobs quickly enough, or if polling for running jobs is placing an unacceptable load on a shared cluster. default=30

Resource Options The options to specify default cores/memory requirements (if not specified by the jobs themselves), and to limit the total amount of memory/cores requested from the batch system.

            --defaultMemory INT
                The default amount of memory to request for a job. Only applicable to jobs that do not specify an explicit value for this requirement. Standard suffixes like K, Ki, M, Mi, G or Gi are supported. Default is 2.0Gi

            --defaultCores FLOAT
                The default number of CPU cores to dedicate a job. Only applicable to jobs that do not specify an explicit value for this requirement. Fractions of a core (for example 0.1) are supported on some batch systems, namely Mesos and single_machine. Default is 1.0

            --defaultDisk INT
                The default amount of disk space to dedicate a job. Only applicable to jobs that do not specify an explicit value for this requirement. Standard suffixes like K, Ki, M, Mi, G or Gi are supported. Default is 2.0Gi

            --defaultAccelerators ACCELERATOR
                The default amount of accelerators to request for a job. Only applicable to jobs that do not specify an explicit value for this requirement. Each accelerator specification can have a type (gpu [default], nvidia, amd, cuda, rocm, opencl, or a specific model like nvidia-tesla-k80), and a count [default: 1]. If both a type and a count are used, they must be separated by a colon. If multiple types of accelerators are used, the specifications are separated by commas. Default is [].

            --defaultPreemptible BOOL
                Make all jobs able to run on preemptible (spot) nodes by default.

            --maxCores INT
                The maximum number of CPU cores to request from the batch system at any one time. Standard suffixes like K, Ki, M, Mi, G or Gi are supported.

            --maxMemory INT
                The maximum amount of memory to request from the batch system at any one time. Standard suffixes like K, Ki, M, Mi, G or Gi are supported.

            --maxDisk INT
                The maximum amount of disk space to request from the batch system at any one time. Standard suffixes like K, Ki, M, Mi, G or Gi are supported.

Options for rescuing/killing/restarting jobs. The options for jobs that either run too long/fail or get lost (some batch systems have issues!).

            --retryCount INT
                Number of times to retry a failing job before giving up and labeling job failed. default=1

            --stopOnFirstFailure BOOL
                Stop the workflow at the first complete job failure.

            --enableUnlimitedPreemptibleRetries
                If set, preemptible failures (or any failure due to an instance getting unexpectedly terminated) will not count towards job failures and --retryCount.

            --doubleMem
                If set, batch jobs which die due to reaching memory limit on batch schedulers will have their memory doubled and they will be retried. The remaining retry count will be reduced by 1. Currently only supported by LSF. default=False.

            --maxJobDuration INT
                Maximum runtime of a job (in seconds) before we kill it (this is a lower bound, and the actual time before killing the job may be longer).

            --rescueJobsFrequency INT
                Period of time to wait (in seconds) between checking for missing/overlong jobs, that is jobs which get lost by the batch system. Expert parameter.

            --jobStoreTimeout FLOAT
                Maximum time (in seconds) to wait for a job's update to the job store before declaring it failed.

Log Management Options

            --maxLogFileSize MAXLOGFILESIZE
                The maximum size of a job log file to keep (in bytes), log files larger than this will be truncated to the last X bytes. Setting this option to zero will prevent any truncation. Setting this option to a negative value will truncate from the beginning. Default=62.5 K

            --writeLogs FILEPATH
                Write worker logs received by the leader into their own files at the specified path. Any non-empty standard output and error from failed batch system jobs will also be written into files at this path. The current working directory will be used if a path is not specified explicitly. Note: By default only the logs of failed jobs are returned to leader. Set log level to 'debug' or enable --writeLogsFromAllJobs to get logs back from successful jobs, and adjust --maxLogFileSize to control the truncation limit for worker logs.

            --writeLogsGzip FILEPATH
                Identical to --writeLogs except the logs files are gzipped on the leader.

            --writeMessages FILEPATH
                File to send messages from the leader's message bus to.

            --realTimeLogging BOOL
                Enable real-time logging from workers to leader.

Miscellaneous Options

            --disableChaining BOOL
                Disables chaining of jobs (chaining uses one job's resource allocation for its successor job if possible).

            --disableJobStoreChecksumVerification
                Disables checksum verification for files transferred to/from the job store. Checksum verification is a safety check to ensure the data is not corrupted during transfer. Currently only supported for non-streaming AWS files

            --sseKey SSEKEY
                Path to file containing 32 character key to be used for server-side encryption on awsJobStore or googleJobStore. SSE will not be used if this flag is not passed.

            --setEnv NAME, -e NAME
                NAME=VALUE or NAME, -e NAME=VALUE or NAME are also valid. Set an environment variable early on in the worker. If VALUE is omitted, it will be looked up in the current environment. Independently of this option, the worker will try to emulate the leader's environment before running a job, except for some variables known to vary across systems. Using this option, a variable can be injected into the worker process itself before it is started.

            --servicePollingInterval SERVICEPOLLINGINTERVAL
                Interval of time service jobs wait between polling for the existence of the keep-alive flag (default=60)

            --forceDockerAppliance
                Disables sanity checking the existence of the docker image specified by TOIL_APPLIANCE_SELF, which Toil uses to provision mesos for autoscaling.

            --statusWait INT
                Seconds to wait between reports of running jobs. (default=3600)

            --disableProgress
                Disables the progress bar shown when standard error is a terminal.

            --publishWorkflowMetrics PUBLISHWORKFLOWMETRICS
                Whether to publish workflow metrics reports (including unique workflow and task run IDs, job names, and version and Toil feature use information) to Dockstore when a workflow completes. Selecting "current" will publish metrics for the current workflow. Selecting "all" will also publish prior workflow runs from the Toil history database, even if they themselves were run with "no". Note that once published, workflow metrics CANNOT be deleted or un-published; they will stay published forever!

Debug Options Debug options for finding problems or helping with testing.

            --debugWorker
                Experimental no forking mode for local debugging. Specifically, workers are not forked and stderr/stdout are not redirected to the log. (default=False)

            --disableWorkerOutputCapture
                Let worker output go to worker's standard out/error instead of per-job logs.

            --badWorker BADWORKER
                For testing purposes randomly kill --badWorker proportion of jobs using SIGKILL. (Default: 0.0)

            --badWorkerFailInterval BADWORKERFAILINTERVAL
                When killing the job pick uniformly within the interval from 0.0 to --badWorkerFailInterval seconds after the worker starts. (Default: 0.01)

            --kill_polling_interval KILL_POLLING_INTERVAL
                Interval of time (in seconds) the leader waits between polling for the kill flag inside the job store set by the "toil kill" command. (default=5)

  Restart Option

In the event of failure, Toil can resume the pipeline by adding the argument --restart and rerunning the workflow. Toil Python workflows (but not CWL or WDL workflows) can even be edited and resumed, which is useful for development or troubleshooting.

  Running Workflows with Services

Toil supports jobs, or clusters of jobs, that run as services to other accessor jobs. Example services include server databases or Apache Spark Clusters. As service jobs exist to provide services to accessor jobs their runtime is dependent on the concurrent running of their accessor jobs. The dependencies between services and their accessor jobs can create potential deadlock scenarios, where the running of the workflow hangs because only service jobs are being run and their accessor jobs can not be scheduled because of too limited resources to run both simultaneously. To cope with this situation Toil attempts to schedule services and accessors intelligently, however to avoid a deadlock with workflows running service jobs it is advisable to use the following parameters:

    •   --maxServiceJobs: The maximum number of service jobs that can be run concurrently, excluding service jobs running on preemptible nodes.

    •   --maxPreemptibleServiceJobs: The maximum number of service jobs that can run concurrently on preemptible nodes.

Specifying these parameters so that at a maximum cluster size there will be sufficient resources to run accessors in addition to services will ensure that such a deadlock can not occur.

If too low a limit is specified then a deadlock can occur in which toil can not schedule sufficient service jobs concurrently to complete the workflow. Toil will detect this situation if it occurs and throw a toil.DeadlockException exception. Increasing the cluster size and these limits will resolve the issue.

  Setting Options directly in a Python Workflow

It's good to remember that commandline options can be overridden in the code of a Python workflow. For example, toil.job.Job.Runner.getDefaultOptions() <#toil.job.Job.Runner.getDefaultOptions> can be used to get the default Toil options, ignoring what was passed on the command line. In this example, this is used to ignore command-line options and always run with the "./toilWorkflow" directory as the jobstore:

            options = Job.Runner.getDefaultOptions("./toilWorkflow") # Get the options object

            with Toil(options) as toil:
                toil.start(Job())  # Run the root job

However, each option can be explicitly set within the workflow by modifying the options object. In this example, we are setting logLevel = "DEBUG" (all log statements are shown) and clean="ALWAYS" (always delete the jobstore) like so:

            options = Job.Runner.getDefaultOptions("./toilWorkflow") # Get the options object
            options.logLevel = "DEBUG" # Set the log level to the debug level.
            options.clean = "ALWAYS" # Always delete the jobStore after a run

            with Toil(options) as toil:
                toil.start(Job())  # Run the root job

However, the usual incantation is to accept commandline args from the user with the following:

            parser = Job.Runner.getDefaultArgumentParser() # Get the parser
            options = parser.parse_args() # Parse user args to create the options object

            with Toil(options) as toil:
                toil.start(Job())  # Run the root job

We can also have code in the workflow to overwrite user supplied arguments:

            parser = Job.Runner.getDefaultArgumentParser() # Get the parser
            options = parser.parse_args() # Parse user args to create the options object
            options.logLevel = "DEBUG" # Set the log level to the debug level.
            options.clean = "ALWAYS" # Always delete the jobStore after a run

            with Toil(options) as toil:
                toil.start(Job())  # Run the root job

TOIL UTILITIES

Toil includes some utilities for inspecting or manipulating workflows during and after their execution. (There are additional Toil Cluster Utilities <#clusterutils> available for working with Toil-managed clusters in the cloud.)

The generic toil subcommand utilities are:
        stats --- Reports runtime and resource usage for all jobs in a specified jobstore (workflow must have originally been run using the --stats option).

        status --- Inspects a job store to see which jobs have failed, run successfully, etc.

        debug-job --- Runs a failing job on your local machine.

        clean --- Delete the job store used by a previous Toil workflow invocation.

        kill --- Kills any running jobs in a rogue toil.

For information on a specific utility, run it with the --help option:

            toil stats --help

  Stats Command

To use the stats command, a workflow must first be run using the --stats option. Using this command makes certain that toil does not delete the job store, no matter what other options are specified (i.e. normally the option --clean=always would delete the job store, but --stats will override this).

  Running an Example

We can run an example workflow and record stats:

            python3 tutorial_stats.py file:my-jobstore --stats

Where tutorial_stats.py is the following:

            import math
            import time
            from multiprocessing import Process

            from toil.common import Toil
            from toil.job import Job

            def think(seconds):
                start = time.time()
                while time.time() - start < seconds:
                    # Use CPU
                    math.sqrt(123456)

            class TimeWaster(Job):
                def __init__(self, time_to_think, time_to_waste, space_to_waste, *args, **kwargs):
                    self.time_to_think = time_to_think
                    self.time_to_waste = time_to_waste
                    self.space_to_waste = space_to_waste
                    super().__init__(*args, **kwargs)

                def run(self, fileStore):
                    # Waste some space
                    file_path = fileStore.getLocalTempFile()
                    with open(file_path, "w") as stream:
                        for i in range(self.space_to_waste):
                            stream.write("X")

                    # Do some "useful" compute
                    processes = []
                    for core_number in range(max(1, self.cores)):
                        # Use all the assigned cores to think
                        p = Process(target=think, args=(self.time_to_think,))
                        p.start()
                        processes.append(p)
                    for p in processes:
                        p.join()

                    # Also waste some time
                    time.sleep(self.time_to_waste)

            def main():
                options = Job.Runner.getDefaultArgumentParser().parse_args()

                job1 = TimeWaster(0, 0, 0, displayName="doNothing")
                job2 = TimeWaster(10, 0, 4096, displayName="efficientJob")
                job3 = TimeWaster(10, 0, 1024, cores=4, displayName="multithreadedJob")
                job4 = TimeWaster(1, 9, 65536, displayName="inefficientJob")

                job1.addChild(job2)
                job1.addChild(job3)
                job3.addChild(job4)

                with Toil(options) as toil:
                    if not toil.options.restart:
                        toil.start(job1)
                    else:
                        toil.restart()

            if __name__ == "__main__":
                main()

Notice the displayName key, which can rename a job, giving it an alias when it is finally displayed in stats.

  Displaying Stats

To see the runtime and resources used for each job when it was run, type

            toil stats file:my-jobstore

This should output something like the following:

            Batch System: single_machine
            Default Cores: 1  Default Memory: 2097152KiB
            Max Cores: unlimited
            Local CPU Time: 55.54 core·s  Overall Runtime: 26.23 s
            Worker
                Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
                    n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
                    3 |     0.34   10.83   10.80   21.23   32.40 |     0.33   10.43   17.94   43.07   53.83 |     0.01    0.40   14.08   41.85   42.25 |  177168Ki 179312Ki 178730Ki 179712Ki 536192Ki |      0Ki     4Ki    22Ki    64Ki    68Ki
            Job
             Worker Jobs  |     min    med    ave    max
                          |       1      1 1.3333      2
                Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
                    n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
                    4 |     0.33   10.83    8.10   10.85   32.38 |     0.33   10.43   13.46   41.70   53.82 |     0.01    1.68    2.78    9.02   11.10 |  177168Ki 179488Ki 178916Ki 179696Ki 715664Ki |      0Ki     4Ki    18Ki    64Ki    72Ki
             multithreadedJob
                Total Cores: 4.0
                Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
                    n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
                    1 |    10.85   10.85   10.85   10.85   10.85 |    41.70   41.70   41.70   41.70   41.70 |     1.68    1.68    1.68    1.68    1.68 |  179488Ki 179488Ki 179488Ki 179488Ki 179488Ki |      4Ki     4Ki     4Ki     4Ki     4Ki
             efficientJob
                Total Cores: 1.0
                Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
                    n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
                    1 |    10.83   10.83   10.83   10.83   10.83 |    10.43   10.43   10.43   10.43   10.43 |     0.40    0.40    0.40    0.40    0.40 |  179312Ki 179312Ki 179312Ki 179312Ki 179312Ki |      4Ki     4Ki     4Ki     4Ki     4Ki
             inefficientJob
                Total Cores: 1.0
                Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
                    n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
                    1 |    10.38   10.38   10.38   10.38   10.38 |     1.36    1.36    1.36    1.36    1.36 |     9.02    9.02    9.02    9.02    9.02 |  179696Ki 179696Ki 179696Ki 179696Ki 179696Ki |     64Ki    64Ki    64Ki    64Ki    64Ki
             doNothing
                Total Cores: 1.0
                Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
                    n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
                    1 |     0.33    0.33    0.33    0.33    0.33 |     0.33    0.33    0.33    0.33    0.33 |     0.01    0.01    0.01    0.01    0.01 |  177168Ki 177168Ki 177168Ki 177168Ki 177168Ki |      0Ki     0Ki     0Ki     0Ki     0Ki

This report gives information on the resources used by your workflow. Note that right now it does NOT track CPU and memory used inside Docker containers, only Singularity containers.

There are three parts to this report.

  Overall Summary

At the top is a section with overall summary statistics for the run:

            Batch System: single_machine
            Default Cores: 1  Default Memory: 2097152KiB
            Max Cores: unlimited
            Local CPU Time: 55.54 core·s  Overall Runtime: 26.23 s

This lists some important the settings for the Toil batch system that actually executed jobs. It also lists:

    •   The CPU time used on the local machine, in core seconds. This includes time used by the Toil leader itself (excluding some startup time), and time used by jobs that run under the leader (which, for the single_machine batch system, is all jobs). It does not include CPU used by jobs that ran on other machines.

    •   The overall wall-clock runtime of the workflow in seconds, as measured by the leader.

These latter two numbers don't count some startup/shutdown time spent loading and saving files, so you still may want to use the time shell built-in to time your Toil runs overall.

  Worker Summary

After the overall summary, there is a section with statistics about the Toil worker processes, which Toil used to execute your workflow's jobs:

            Worker
                Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
                    n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
                    3 |     0.34   10.83   10.80   21.23   32.40 |     0.33   10.43   17.94   43.07   53.83 |     0.01    0.40   14.08   41.85   42.25 |  177168Ki 179312Ki 178730Ki 179712Ki 536192Ki |      0Ki     4Ki    22Ki    64Ki    68Ki

    •   The Count column shows that, to run this workflow, Toil had to submit 3 Toil worker processes to the backing scheduler. (In this case, it ran them all on the local machine.)

    •   The Real Time column shows satistics about the wall clock times that all the worker process took. All the sub-column values are in seconds.

    •   The CPU Time column shows statistics about the CPU usage amounts of all the worker processes. All the sub-column values are in core seconds.

    •   The CPU Wait column shows statistics about CPU time reserved for but not consumed by worker processes. In this example, the max and total are relatively high compared to both real time and CPU time, indicating that a lot of reserved CPU time went unused. This can indicate that the workflow is overestimating its required cores, that small jobs are running in the same resource reservations as large jobs via chaining, or that the workflow is having to wait around for slow disk I/O.

    •   The Memory column shows the peak memory usage of each worker process and its child processes.

    •   The Disk column shows the disk usage in each worker. This is polled at the end of each job that is run by the worker, so it may not always reflect the actual peak disk usage.

  Job Breakdown

Finally, there is the breakdown of resource usage by jobs. This starts with a table summarizing the counts of jobs that ran on each worker:

            Job
             Worker Jobs  |     min    med    ave    max
                          |       1      1 1.3333      2

In this example, most of the workers ran one job each, but one worker managed to run two jobs, via chaining. (Jobs will chain when a job has only one dependant job, which in turn depends on only that first job, and the second job needs no more resources than the first job did.)

Next, we have statistics for resource usage over all jobs together:

            Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
                n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
                4 |     0.33   10.83    8.10   10.85   32.38 |     0.33   10.43   13.46   41.70   53.82 |     0.01    1.68    2.78    9.02   11.10 |  177168Ki 179488Ki 178916Ki 179696Ki 715664Ki |      0Ki     4Ki    18Ki    64Ki    72Ki

And finally, for each kind of job (as determined by the job's displayName), we have statistics summarizing the resources used by the instances of that kind of job:

            multithreadedJob
               Total Cores: 4.0
               Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
                   n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
                   1 |    10.85   10.85   10.85   10.85   10.85 |    41.70   41.70   41.70   41.70   41.70 |     1.68    1.68    1.68    1.68    1.68 |  179488Ki 179488Ki 179488Ki 179488Ki 179488Ki |      4Ki     4Ki     4Ki     4Ki     4Ki
            efficientJob
               Total Cores: 1.0
               Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
                   n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
                   1 |    10.83   10.83   10.83   10.83   10.83 |    10.43   10.43   10.43   10.43   10.43 |     0.40    0.40    0.40    0.40    0.40 |  179312Ki 179312Ki 179312Ki 179312Ki 179312Ki |      4Ki     4Ki     4Ki     4Ki     4Ki
            inefficientJob
               Total Cores: 1.0
               Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
                   n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
                   1 |    10.38   10.38   10.38   10.38   10.38 |     1.36    1.36    1.36    1.36    1.36 |     9.02    9.02    9.02    9.02    9.02 |  179696Ki 179696Ki 179696Ki 179696Ki 179696Ki |     64Ki    64Ki    64Ki    64Ki    64Ki
            doNothing
               Total Cores: 1.0
               Count |                           Real Time (s)* |                        CPU Time (core·s) |                        CPU Wait (core·s) |                                    Memory (B) |                                 Disk (B)
                   n |      min    med*     ave     max   total |      min     med     ave     max   total |      min     med     ave     max   total |       min      med      ave      max    total |      min     med     ave     max   total
                   1 |     0.33    0.33    0.33    0.33    0.33 |     0.33    0.33    0.33    0.33    0.33 |     0.01    0.01    0.01    0.01    0.01 |  177168Ki 177168Ki 177168Ki 177168Ki 177168Ki |      0Ki     0Ki     0Ki     0Ki     0Ki

For each job, we first list its name, and then the total cores that it asked for, summed across all instances of it. Then we show a table of statistics.

Here the * marker in the table headers becomes relevant; it shows that jobs are being sorted by the median of the real time used. You can control this with the --sortCategory option.

The columns meanings are the same as for the workers:

    •   The Count column shows the number of jobs of each type that ran.

    •   The Real Time column shows satistics about the wall clock times that instances of the job type took. All the sub-column values are in seconds.

    •   The CPU Time column shows statistics about the CPU usage amounts of each job. Note that multithreadedJob managed to use CPU time at faster than one core second per second, because it reserved multiple cores and ran multiple threads.

    •   The CPU Wait column shows statistics about CPU time reserved for but not consumed by jobs. Note that inefficientJob used hardly any of the cores it requested for most of its real time.

    •   The Memory column shows the peak memory usage of each job.

    •   The Disk column shows the disk usage at the end of each job. It may not always reflect the actual peak disk usage.

  Example Cleanup

Once we're done looking at the stats, we can clean up the job store by running:

            toil clean file:my-jobstore

  Status Command

Continuing the example from the stats section above, if we ran our workflow with the command

            python3 tutorial_stats.py file:my-jobstore --stats

We could interrogate our jobstore with the status command, for example:

            toil status file:my-jobstore

If the run was successful, this would not return much valuable information, something like

            2018-01-11 19:31:29,739 - toil.lib.bioio - INFO - Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
            2018-01-11 19:31:29,740 - toil.utils.toilStatus - INFO - Parsed arguments
            2018-01-11 19:31:29,740 - toil.utils.toilStatus - INFO - Checking if we have files for Toil
            The root job of the job store is absent, the workflow completed successfully.

Otherwise, the toil status command will return something like the following:
        Of the 3 jobs considered, there are 1 completely failed jobs, 1 jobs with children, 2 jobs ready to run, 0 zombie jobs, 0 jobs with services, 0 services, and 0 jobs with log files currently in FileJobStore(/Users/anovak/workspace/toil/tree).

The toil status command supports several useful flags, including --perJob to get per-job status information, --logs to print stored worker logs, and --failed to list all failed jobs in the workflow. For more information, run toil status --help.

One use case of toil status is with the --printStatus argument. Running toil status --printStatus file:my-jobstore at any point of the workflow's lifecycle can tell you the progress of the workflow. Note: This command will output all current running jobs but not any finished or failed jobs.

For example, after running workflow.py in another terminal:

            $ toil status --printStatus file:my-jobstore
            [2024-05-31T13:59:13-0700] [MainThread] [I] [toil.utils.toilStatus] Traversing the job graph gathering jobs. This may take a couple of minutes.
            Of the 2 jobs considered, there are 0 completely failed jobs, 1 jobs with children, 1 jobs ready to run, 0 zombie jobs, 0 jobs with services, 0 services, and 0 jobs with log files currently in FileJobStore(/path/to/my-jobstore).

            Message bus path:  /tmp/tmp9cnaq3bm
            Job ID kind-TimeWaster/instance-zvdsdkm_ with name TimeWaster is running on SingleMachineBatchSystem as ID 101349.
            Job ID kind-TimeWaster/instance-7clm8cv2 with name TimeWaster is running on SingleMachineBatchSystem as ID 101350.

At this moment in time, two jobs with the name "TimeWaster" is running on my local machine.

  Clean Command

If a Toil pipeline didn't finish successfully, or was run using --clean=always or --stats, the job store will exist until it is deleted. toil clean <jobStore> ensures that all artifacts associated with a job store are removed. This is particularly useful for deleting AWS job stores, which reserves an SDB domain as well as an S3 bucket.

The deletion of the job store can be modified by the --clean argument, and may be set to always, onError, never, or onSuccess (default).

Temporary directories where jobs are running can also be saved from deletion using the --cleanWorkDir, which has the same options as --clean. This option should only be run when debugging, as intermediate jobs will fill up disk space.

  Debug Job Command

If a Toil worklfow fails, and it wasn't run with --clean=always, the failing job will be waiting in the job store to be debugged. (With WDL or CWL workflows, you may have needed to manually set a --jobStore location you can find again.)

You can use toil debug-job on a job in the job store to run it on your local machine, to locally reproduce any error that may have happened during a remote workflow.

The toil debug-job command takes a job store, and the ID or a name of a job in it. If multiple jobs match a job name, and only one seems to have run out of retries and completely failed, it will run that one.

You can also pass the --printJobInfo flag to dump information about the job instead of running it.

  Kill Command

To kill all currently running jobs for a given jobstore, use the command

            toil kill file:my-jobstore

TOIL DEBUGGING

Toil has a number of tools to assist in debugging. Here we provide help in working through potential problems that a user might encounter in attempting to run a workflow.

  Failing Fast

If you have a workflow you are testing, you can pass --stopOnFirstFailure=True to have Toil stop the workflow as soon as any job is complely failed (i.e. runs out of retry attempts). If you don't want Toil to retry failing jobs, you can pass --retryCount=0.

  Reading the Log

Usually, at the end of a failed Toil worklfow, Toil will reproduce the job logs for the jobs that failed. You can look at the end of your workflow log and use the job logs to identify which jobs are failing and why.

  Using DEBUG Logging

When debugging a broken workflow, you can run the workflow with --logDebug, to set the log level to DEBUG.

When debug logging is on, the log from every Toil job is inserted in the main Toil log between these markers:

            =========>
                 Toil job log is here
            <=========

Normally, only the logs of failing jobs and the output of commands run from WDL are reproduced like this.

  Finding Failed Jobs in the Jobstore

The toil status command (Status Command <#cli-status>) can be used with the --failed option to list all failed jobs in a Toil job store.

You can also use it with the --logs option to retrieve per-job logs from the job store, for failed jobs that left them. These logs might be useful for diagnosing and fixing the problem.

  Running a Job Locally

If you have a failing job's ID or name, you can reproduce its failure on your local machine with toil debug-job. See Debug Job Command <#cli-debug-job>.

For example, say you have this WDL workflow in test.wdl. This workflow cannot succeed, due to the typo in the echo command:

            version 1.0
            workflow test {
              call hello
            }
            task hello {
              input {
              }
              command <<<
                set -e
                echoo "Hello"
              >>>
              output {
              }
            }

You could try to run it with:

            toil-wdl-runner --jobStore ./store test.wdl --retryCount 0

But it will fail.

If you want to reproduce the failure later, or on another machine, you can first find out what jobs failed with toil status:

            toil status --failed --noAggStats ./store

This will produce something like:

            [2024-03-14T17:45:15-0400] [MainThread] [I] [toil.utils.toilStatus] Traversing the job graph gathering jobs. This may take a couple of minutes.
            Failed jobs:
            'WDLTaskJob' test.hello.command kind-WDLTaskJob/instance-r9u6_dcs v6

And we can see a failed job with the display name test.hello.command, which describes the job's location in the WDL workflow as the command section of the hello task called from the test workflow. (If you are writing a Toil Python script, this is the job's displayName.) We can then run that job again locally by name with:

            toil debug-job ./store test.hello.command

If there were multiple failed jobs with that name (perhaps because of a WDL scatter), we would need to select one by Toil job ID instead:

            toil debug-job ./store kind-WDLTaskJob/instance-r9u6_dcs

And if we know there's only one failed WDL task, we can just tell Toil to rerun the failed WDLTaskJob by Python class name:

            toil debug-job ./store WDLTaskJob

Any of these will run the job (including any containers) on the local machine, where its execution can be observed live or monitored with a debugger.

  Fetching Job Inputs

The --retrieveTaskDirectory option to toil debug-job allows you to send the input files for a job to a directory, and then stop running the job. It works for CWL and WDL jobs, and for Python workflows that call toil.job.Job.files_downloaded_hook() <#toil.job.Job.files_downloaded_hook> after downloading their files. It will make the worker work in the specified directory, so the job's temporary directory will be at worker/job inside it. For WDL and CWL jobs that mount files into containers, there will also be an inside directory populated with symlinks to the files as they would be visible from the root of the container's filesystem.

For example, say you have a broken WDL workflow named example_alwaysfail_with_files.wdl, like this:

            version 1.0
            workflow test {
                call make_file as f1
                call make_file as f2
                call hello {
                    input:
                        name_file=f1.out,
                        unused_file=f2.out
                }
            }
            task make_file {
                input {
                }
                command <<<
                    echo "These are the contents" >test.txt
                >>>
                output {
                    File out = "test.txt"
                }
            }
            task hello {
                input {
                    File name_file
                    File? unused_file
                }
                command <<<
                    set -e
                    echoo "Hello" "$(cat ~{name_file})"
                >>>
                output {
                    File out = stdout()
                }
            }

You can try and fail to run it like this:

            toil-wdl-runner --jobStore ./store example_alwaysfail_with_files.wdl --retryCount 0

If you dump the files from the failing job:

            toil debug-job ./store WDLTaskJob --retrieveTaskDirectory dumpdir

You will end up with a directory tree that looks, accorfing to tree, something like this:

            dumpdir
            ├── inside
            │   └── mnt
            │       └── miniwdl_task_container
            │           └── work
            │               └── _miniwdl_inputs
            │                   ├── 0
            │                   │   └── test.txt -> ../../../../../../worker/job/2c6b3dc4-1d21-4abf-9937-db475e6a6bc2/test.txt
            │                   └── 1
            │                       └── test.txt -> ../../../../../../worker/job/e3d724e1-e6cc-4165-97f1-6f62ab0fb1ef/test.txt
            └── worker
                └── job
                    ├── 2c6b3dc4-1d21-4abf-9937-db475e6a6bc2
                    │   └── test.txt
                    ├── e3d724e1-e6cc-4165-97f1-6f62ab0fb1ef
                    │   └── test.txt
                    ├── tmpr2j5yaic
                    ├── tmpxqr9__y4
                    └── work

            15 directories, 4 files

You can see where Toil downloaded the input files for the job to the worker's temporary directory, and how they would be mounted into the container.

  Interactively Investigating Running Jobs

Say you have a broken WDL workflow that can't complete. Whenever you run tutorial_debugging_hangs.wdl, it hangs:

            version 1.1

            workflow TutorialDebugging {

                input {
                    Array[String] messages = ["Uh-oh!", "Oh dear", "Oops"]
                }

                scatter(message in messages) {

                    call WhaleSay {
                        input:
                            message = message
                    }

                    call CountLines {
                        input:
                            to_count = WhaleSay.result
                    }
                }

                Array[File] to_compress = flatten([CountLines.result, WhaleSay.result])

                call CompressFiles {
                    input:
                        files = to_compress
                }

                output {
                    File compressed = CompressFiles.result
                }

            }

            # Draw ASCII art
            task WhaleSay {
                input {
                    String message
                }

                command <<<
                    cowsay "~{message}"
                >>>

                output {
                    File result = stdout()
                }

                runtime {
                    container: "docker/whalesay"
                }
            }

            # Count the lines in a file
            task CountLines {
                input {
                    File to_count
                }

                command <<<
                    wc -l ~{to_count}
                >>>

                output {
                    File result = stdout()
                }

                runtime {
                    container: ["ubuntu:latest", "https://gcr.io/standard-images/ubuntu:latest"]
                }
            }

            # Compress files into a ZIP
            task CompressFiles {
                input {
                    Array[File] files
                }

                command <<<
                set -e
                cat >script.py <<'EOF'
                import sys
                from zipfile import ZipFile
                import os

                # Interpret command line arguments
                to_compress = list(reversed(sys.argv[1:]))

                with ZipFile("compressed.zip", "w") as z:
                    while to_compress != []:
                        # Grab the file to add off the end of the list
                        input_filename = to_compress[-1]
                        # Now we need to write this to the zip file.
                        # What internal filename should we use?
                        basename = os.path.basename(input_filename)
                        disambiguation_number = 0
                        while True:
                            target_filename = str(disambiguation_number) + basename
                            try:
                                z.getinfo(target_filename)
                            except KeyError:
                                # Filename is free
                                break
                            # Otherwise try another name
                            disambiguation_number += 1
                        # Now we can actually make the compressed file
                        with z.open(target_filename, 'w') as out_stream:
                            with open(input_filename) as in_stream:
                                for line in in_stream:
                                    # Prefix each line of text with the original input file
                                    # it came from.
                                    # Also remember to encode the text as the zip file
                                    # stream is in binary mode.
                                    out_stream.write(f"{basename}: {line}".encode("utf-8"))
                EOF
                python script.py ~{sep(" ", files)}
                >>>

                output {
                    File result = "compressed.zip"
                }

                runtime {
                    container: "python:3.11"
                }
            }

You can try to run it like this, using Docker containers. Pretend this was actually a run on a large cluster:

            $ toil-wdl-runner --jobStore ./store tutorial_debugging_hangs.wdl --container docker

If you run this, it will hang at the TutorialDebugging.CompressFiles.command step:

            [2024-06-18T12:12:49-0400] [MainThread] [I] [toil.leader] Issued job 'WDLTaskJob' TutorialDebugging.CompressFiles.command kind-WDLTaskJob/instance-y0ga_907 v1 with job batch system ID: 16 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False

            Workflow Progress  94%|██████████▎| 15/16 (0 failures) [00:36<00:02, 0.42 jobs/s]

Say you want to find out why it is stuck. First, you need to kill the workflow. Open a new shell in the same directory and run:

            # toil kill ./store

You can also hit Control+C in its terminal window and wait for it to stop.

Then, you need to use toil debug-job to run the stuck job on your local machine:

            $ toil debug-job ./store TutorialDebugging.CompressFiles.command

This produces some more informative logging messages, showing that the Docker container is managing to start up, but that it stays running indefinitely, with a repeating message:

            [2024-06-18T12:18:00-0400] [MainThread] [N] [MiniWDLContainers] docker task running :: service: "lhui2bdzmzmg", task: "sg371eb2yk", node: "zyu9drdp6a", message: "started"
            [2024-06-18T12:18:01-0400] [MainThread] [D] [MiniWDLContainers] docker task status :: Timestamp: "2024-06-18T16:17:58.545272049Z", State: "running", Message: "started", ContainerStatus: {"ContainerID": "b7210b346637210b49e7b6353dd24108bc3632bbf2ce7479829d450df6ee453a", "PID": 36510, "ExitCode": 0}, PortStatus: {}
            [2024-06-18T12:18:03-0400] [MainThread] [D] [MiniWDLContainers] docker task status :: Timestamp: "2024-06-18T16:17:58.545272049Z", State: "running", Message: "started", ContainerStatus: {"ContainerID": "b7210b346637210b49e7b6353dd24108bc3632bbf2ce7479829d450df6ee453a", "PID": 36510, "ExitCode": 0}, PortStatus: {}
            [2024-06-18T12:18:04-0400] [MainThread] [D] [MiniWDLContainers] docker task status :: Timestamp: "2024-06-18T16:17:58.545272049Z", State: "running", Message: "started", ContainerStatus: {"ContainerID": "b7210b346637210b49e7b6353dd24108bc3632bbf2ce7479829d450df6ee453a", "PID": 36510, "ExitCode": 0}, PortStatus: {}
            ...

This also gives you the Docker container ID of the running container, b7210b346637210b49e7b6353dd24108bc3632bbf2ce7479829d450df6ee453a. You can use that to get a shell inside the running container:

            $ docker exec -ti b7210b346637210b49e7b6353dd24108bc3632bbf2ce7479829d450df6ee453a bash
            root@b7210b346637:/mnt/miniwdl_task_container/work#

Your shell is already in the working directory of the task, so we can inspect the files there to get an idea of how far the task has gotten. Has it managed to create script.py? Has the script managed to create compressed.zip? Let's check:

            # ls -lah
            total 6.1M
            drwxrwxr-x 6 root root  192 Jun 18 16:17 .
            drwxr-xr-x 3 root root 4.0K Jun 18 16:17 ..
            drwxr-xr-x 3 root root   96 Jun 18 16:17 .toil_wdl_runtime
            drwxrwxr-x 8 root root  256 Jun 18 16:17 _miniwdl_inputs
            -rw-r--r-- 1 root root 6.0M Jun 18 16:23 compressed.zip
            -rw-r--r-- 1 root root 1.3K Jun 18 16:17 script.py

So we can see that the script exists, and the zip file also exists. So maybe the script is still running? We can check with ps, but we need the -x option to include processes not under the current shell. We can also include the -u option to get statistics:

            # ps -xu
            USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
            root         1  0.0  0.0   2316   808 ?        Ss   16:17   0:00 /bin/sh -c /bin/
            root         7  0.0  0.0   4208  3056 ?        S    16:17   0:00 /bin/bash ../com
            root         8  0.1  0.0   4208  1924 ?        S    16:17   0:00 /bin/bash ../com
            root        20 95.0  0.4  41096 36428 ?        R    16:17   7:09 python script.py
            root       645  0.0  0.0   4472  3492 pts/0    Ss   16:21   0:00 bash
            root      1379  0.0  0.0   2636   764 ?        S    16:25   0:00 sleep 1
            root      1380  0.0  0.0   8584  3912 pts/0    R+   16:25   0:00 ps -xu

Here we can see that python is indeed running, and it is using 95% of a CPU core. So we can surmise that Python is probably stuck spinning around in an infinite loop. Let's look at our files again:

            # ls -lah
            total 8.1M
            drwxrwxr-x 6 root root  192 Jun 18 16:17 .
            drwxr-xr-x 3 root root 4.0K Jun 18 16:17 ..
            drwxr-xr-x 3 root root   96 Jun 18 16:17 .toil_wdl_runtime
            drwxrwxr-x 8 root root  256 Jun 18 16:17 _miniwdl_inputs
            -rw-r--r-- 1 root root 7.6M Jun 18  2024 compressed.zip
            -rw-r--r-- 1 root root 1.3K Jun 18 16:17 script.py

Note that, while we've been investigating, our compressed.zip file has grown from 6.0M to 7.6M. So we now know that, not only is the Python script stuck in a loop, it is also writing to the ZIP file inside that loop.

Let's inspect the inputs:

            # ls -lah _miniwdl_inputs/*
            _miniwdl_inputs/0:
            total 4.0K
            drwxrwxr-x 3 root root  96 Jun 18 16:17 .
            drwxrwxr-x 8 root root 256 Jun 18 16:17 ..
            -rw-r--r-- 1 root root  65 Jun 18 16:15 stdout.txt

            _miniwdl_inputs/1:
            total 4.0K
            drwxrwxr-x 3 root root  96 Jun 18 16:17 .
            drwxrwxr-x 8 root root 256 Jun 18 16:17 ..
            -rw-r--r-- 1 root root  65 Jun 18 16:15 stdout.txt

            _miniwdl_inputs/2:
            total 4.0K
            drwxrwxr-x 3 root root  96 Jun 18 16:17 .
            drwxrwxr-x 8 root root 256 Jun 18 16:17 ..
            -rw-r--r-- 1 root root  65 Jun 18 16:15 stdout.txt

            _miniwdl_inputs/3:
            total 4.0K
            drwxrwxr-x 3 root root  96 Jun 18 16:17 .
            drwxrwxr-x 8 root root 256 Jun 18 16:17 ..
            -rw-r--r-- 1 root root 384 Jun 18 16:15 stdout.txt

            _miniwdl_inputs/4:
            total 4.0K
            drwxrwxr-x 3 root root  96 Jun 18 16:17 .
            drwxrwxr-x 8 root root 256 Jun 18 16:17 ..
            -rw-r--r-- 1 root root 387 Jun 18 16:15 stdout.txt

            _miniwdl_inputs/5:
            total 4.0K
            drwxrwxr-x 3 root root  96 Jun 18 16:17 .
            drwxrwxr-x 8 root root 256 Jun 18 16:17 ..
            -rw-r--r-- 1 root root 378 Jun 18 16:15 stdout.txt

There are the files that are meant to be being compressed into that ZIP file. But, hang on, there are only six of these files, and none of them is over 400 bytes in size. How did we get a multi-megabyte ZIP file? The script must be putting more data than we expected into the ZIP file it is writing.

Taking what we know, we can now inspect the Python script again and see if we can find a way in which it could get stuck in an infinite loop, writing much more data to the ZIP than is actually in the input files. We can also inspect it for WDL variable substitutions (there aren't any). Let's look at it with line numbers using the nl tool, numbering even blank lines with -b a:

            # nl -b a script.py
             1  import sys
             2  from zipfile import ZipFile
             3  import os
             4
             5  # Interpret command line arguments
             6  to_compress = list(reversed(sys.argv[1:]))
             7
             8  with ZipFile("compressed.zip", "w") as z:
             9      while to_compress != []:
            10          # Grab the file to add off the end of the list
            11          input_filename = to_compress[-1]
            12          # Now we need to write this to the zip file.
            13          # What internal filename should we use?
            14          basename = os.path.basename(input_filename)
            15          disambiguation_number = 0
            16          while True:
            17              target_filename = str(disambiguation_number) + basename
            18              try:
            19                  z.getinfo(target_filename)
            20              except KeyError:
            21                  # Filename is free
            22                  break
            23              # Otherwise try another name
            24              disambiguation_number += 1
            25          # Now we can actually make the compressed file
            26          with z.open(target_filename, 'w') as out_stream:
            27              with open(input_filename) as in_stream:
            28                  for line in in_stream:
            29                      # Prefix each line of text with the original input file
            30                      # it came from.
            31                      # Also remember to encode the text as the zip file
            32                      # stream is in binary mode.
            33                      out_stream.write(f"{basename}: {line}".encode("utf-8"))

We have three loops here: while to_compress != [] on line 9, while True on line 16, and for line in in_stream on line 28.

The while True loop is immediately suspicious, but none of the code inside it writes to the ZIP file, so we know we can't be stuck in there.

The for line in in_stream loop contains the only call that writes data to the ZIP, so we must be spending time inside it, but it is constrained to loop over a single file at a time, so it can't be the infinite loop we're looking for.

So then we must be infinitely looping at while to_compress != [], and indeed we can see that to_compress is never modified, so it can never become [].

So now we have a theory as to what the problem is, and we can exit out of our shell in the container, and stop toil debug-job with Control+C. Then we can make the following change to our workflow, adding code to the script to actually pop the handled files off the end of the list:

            --- tutorial_debugging_works.wdl	2024-06-18 12:03:32
            +++ tutorial_debugging_hangs.wdl	2024-06-18 12:03:53
            @@ -112,9 +112,6 @@
                                     # Also remember to encode the text as the zip file
                                     # stream is in binary mode.
                                     out_stream.write(f"{basename}: {line}".encode("utf-8"))
            -            # Even though we got distracted by zip file manipulation, remember
            -            # to pop off the file we just did.
            -            to_compress.pop()
                 EOF
                 python script.py ~{sep(" ", files)}
                 >>>

If we apply that change and produce a new file, tutorial_debugging_works.wdl, we can clean up from the old failed run and run a new one:

            $ toil clean ./store
            $ toil-wdl-runner --jobStore ./store tutorial_debugging_works.wdl --container docker

This will produce a successful log, ending with something like:

            [2024-06-18T12:42:20-0400] [MainThread] [I] [toil.leader] Finished toil run successfully.

            Workflow Progress 100%|███████████| 17/17 (0 failures) [00:24<00:00, 0.72 jobs/s]
            {"TutorialDebugging.compressed": "/Users/anovak/workspace/toil/src/toil/test/docs/scripts/wdl-out-u7fkgqbe/f5e16468-0cf6-4776-a5c1-d93d993c4db2/compressed.zip"}
            [2024-06-18T12:42:20-0400] [MainThread] [I] [toil.common] Successfully deleted the job store: FileJobStore(/Users/anovak/workspace/toil/src/toil/test/docs/scripts/store)

Note the line to standard output giving us the path on disk where the TutorialDebugging.compressed output from the workflow is. If you look at that ZIP file, you can see it contains the expected files, such as 3stdout.txt, which should contain this suitably prefixed dismayed whale:

            stdout.txt:  ________
            stdout.txt: < Uh-oh! >
            stdout.txt:  --------
            stdout.txt:     \
            stdout.txt:      \
            stdout.txt:       \
            stdout.txt:                     ##        .
            stdout.txt:               ## ## ##       ==
            stdout.txt:            ## ## ## ##      ===
            stdout.txt:        /""""""""""""""""___/ ===
            stdout.txt:   ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
            stdout.txt:        \______ o          __/
            stdout.txt:         \    \        __/
            stdout.txt:           \____\______/

When we're done inspecting the output, and satisfied that the workflow now works, we might want to clean up all the auto-generated WDL output directories from the successful and failed run(s):

            $ rm -Rf wdl-out-*

  Introspecting the Job Store

Note: Currently these features are only implemented for use locally (single machine) with the fileJobStore.

To view what files currently reside in the jobstore, run the following command:

            $ toil debug-file file:path-to-jobstore-directory \
                  --listFilesInJobStore

When run from the commandline, this should generate a file containing the contents of the job store (in addition to displaying a series of log messages to the terminal). This file is named "jobstore_files.txt" by default and will be generated in the current working directory.

If one wishes to copy any of these files to a local directory, one can run for example:

            $ toil debug-file file:path-to-jobstore \
                  --fetch overview.txt *.bam *.fastq \
                  --localFilePath=/home/user/localpath

To fetch overview.txt, and all .bam and .fastq files. This can be used to recover previously used input and output files for debugging or reuse in other workflows, or use in general debugging to ensure that certain outputs were imported into the jobStore.

  Stats and Status

See Stats Command <#cli-stats> and Status Command <#cli-status> for more about gathering statistics about job success, runtime, and resource usage from workflows.

  Using a Python debugger

If you execute a workflow using the --debugWorker flag, or if you use toil debug-job, Toil will run the job in the process you started from the command line. This means you can either use pdb <https://docs.python.org/3/library/pdb.html>, or an IDE that supports debugging Python <https://wiki.python.org/moin/PythonDebuggingTools#IDEs_with_Debug_Capabilities> to interact with the Python process as it runs your job. Note that the --debugWorker flag will only work with the single_machine batch system (the default), and not any of the custom job schedulers.

RUNNING IN THE CLOUD

Toil supports Amazon Web Services (AWS) and Google Compute Engine (GCE) in the cloud and has autoscaling capabilities that can adapt to the size of your workflow, whether your workflow requires 10 instances or 20,000.

Toil does this by creating a virtual cluster running Kubernetes <https://kubernetes.io/>. Kubernetes requires a leader node to coordinate the workflow, and worker nodes to execute the various tasks within the workflow. As the workflow runs, Kubernetes will "autoscale", creating and terminating workers as needed to meet the demands of the workflow. Historically, Toil has spun up clusters with Apache Mesos <https://mesos.apache.org/gettingstarted/>, but it is no longer recommended.

Once a user is familiar with the basics of running Toil locally (specifying a jobStore <#jobstoreoverview>, and how to write a workflow), they can move on to the guides below to learn how to translate these workflows into cloud ready workflows.

  Managing a Cluster of Virtual Machines (Provisioning)

Toil can launch and manage a cluster of virtual machines to run using the provisioner to run a workflow distributed over several nodes. The provisioner also has the ability to automatically scale up or down the size of the cluster to handle dynamic changes in computational demand (autoscaling). Currently we have working provisioners with AWS and GCE (Azure support has been deprecated).

Toil uses Kubernetes <https://kubernetes.io/> as the Batch System <#batchsystemoverview>.

See here for instructions for Running in AWS <#runningaws>.

See here for instructions for Running in Google Compute Engine (GCE) <#runninggce>.

Toil offers a suite of commands for using the provisioners to manage clusters.

  Toil Cluster Utilities

In addition to the generic Toil Utilities <#utils>, there are several utilities used for starting and managing a Toil cluster using the AWS or GCE provisioners. They are installed via the [aws] or [google] extra. For installation details see Toil Provisioner <#installprovisioner>.

The toil cluster subcommands are:
        destroy-cluster --- For autoscaling. Terminates the specified cluster and associated resources.

        launch-cluster --- For autoscaling. This is used to launch a toil leader instance with the specified provisioner.

        rsync-cluster --- For autoscaling. Used to transfer files to a cluster launched with toil launch-cluster.

        ssh-cluster --- SSHs into the toil appliance container running on the leader of the cluster.

For information on a specific utility, run it with the --help option:

            toil launch-cluster --help

The cluster utilities can be used for Running in Google Compute Engine (GCE) <#runninggce> and Running in AWS <#runningaws>.

Tip:
        By default, all of the cluster utilities expect to be running on AWS. To run with Google you will need to specify the --provisioner gce option for each utility.

Note:
        Boto must be configured <http://boto3.readthedocs.io/en/latest/guide/quickstart.html#configuration> with AWS credentials before using cluster utilities.

        Running in Google Compute Engine (GCE) <#runninggce> contains instructions for

  Launch-Cluster Command

Running toil launch-cluster starts up a leader for a cluster. Workers can be added to the initial cluster by specifying the -w option. An example would be

            $ toil launch-cluster my-cluster \
                  --leaderNodeType t2.small -z us-west-2a \
                  --keyPairName your-AWS-key-pair-name \
                  --nodeTypes m3.large,t2.micro -w 1,4

Options are listed below. These can also be displayed by running

            $ toil launch-cluster --help

launch-cluster's main positional argument is the clusterName. This is simply the name of your cluster. If it does not exist yet, Toil will create it for you.

Launch-Cluster Options

            --help
                -h also accepted. Displays this help menu.

            --tempDirRoot TEMPDIRROOT
                Path to the temporary directory where all temp files are created, by default uses the current working directory as the base.

            --version
                Display version.

            --provisioner CLOUDPROVIDER
                -p CLOUDPROVIDER also accepted. The provisioner for cluster auto-scaling. Both AWS and GCE are currently supported.

            --zone ZONE
                -z ZONE also accepted. The availability zone of the leader. This parameter can also be set via the TOIL_AWS_ZONE or TOIL_GCE_ZONE environment variables, or by the ec2_region_name parameter in your .boto file if using AWS, or derived from the instance metadata if using this utility on an existing EC2 instance.

            --leaderNodeType LEADERNODETYPE
                Non-preemptable node type to use for the cluster leader.

            --keyPairName KEYPAIRNAME
                The name of the AWS or ssh key pair to include on the instance.

            --owner OWNER
                The owner tag for all instances. If not given, the value in TOIL_OWNER_TAG will be used, or else the value of --keyPairName.

            --boto BOTOPATH
                The path to the boto credentials directory. This is transferred to all nodes in order to access the AWS jobStore from non-AWS instances.

            --tag KEYVALUE
                KEYVALUE is specified as KEY=VALUE. -t KEY=VALUE also accepted. Tags are added to the AWS cluster for this node and all of its children. Tags are of the form: -t key1=value1 --tag key2=value2. Multiple tags are allowed and each tag needs its own flag. By default the cluster is tagged with: { "Name": clusterName, "Owner": IAM username }.

            --vpcSubnet VPCSUBNET
                VPC subnet ID to launch cluster leader in. Uses default subnet if not specified. This subnet needs to have auto assign IPs turned on.

            --nodeTypes NODETYPES
                Comma-separated list of node types to create while launching the leader. The syntax for each node type depends on the provisioner used. For the AWS provisioner this is the name of an EC2 instance type followed by a colon and the price in dollars to bid for a spot instance, for example 'c3.8xlarge:0.42'. Must also provide the --workers argument to specify how many workers of each node type to create.

            --workers WORKERS
                -w WORKERS also accepted. Comma-separated list of the number of workers of each node type to launch alongside the leader when the cluster is created. This can be useful if running toil without auto-scaling but with need of more hardware support.

            --leaderStorage LEADERSTORAGE
                Specify the size (in gigabytes) of the root volume for the leader instance. This is an EBS volume.

            --nodeStorage NODESTORAGE
                Specify the size (in gigabytes) of the root volume for any worker instances created when using the -w flag. This is an EBS volume.

            --nodeStorageOverrides NODESTORAGEOVERRIDES
                Comma-separated list of nodeType:nodeStorage that are used to override the default value from --nodeStorage for the specified nodeType(s). This is useful for heterogeneous jobs where some tasks require much more disk than others.

            --allowFuse BOOL
                Whether to allow FUSE mounts for faster runtimes with Singularity. Note: This will result in the Toil container running as privileged. For Kubernetes, pods will be asked to run as privileged. If this is not allowed, Singularity containers will use sandbox directories instead.

Logging Options

            --logOff
                Same as --logCritical.

            --logCritical
                Turn on logging at level CRITICAL and above. (default is INFO)

            --logError
                Turn on logging at level ERROR and above. (default is INFO)

            --logWarning
                Turn on logging at level WARNING and above. (default is INFO)

            --logInfo
                Turn on logging at level INFO and above. (default is INFO)

            --logDebug
                Turn on logging at level DEBUG and above. (default is INFO)

            --logDebug
                Turn on logging at level TRACE and above. (default is INFO)

            --logLevel LOGLEVEL
                Log at given level (may be either OFF (or CRITICAL), ERROR, WARN (or WARNING), INFO, DEBUG, or TRACE). (default is INFO)

            --logFile LOGFILE
                File to log in.

            --rotatingLogging
                Turn on rotating logging, which prevents log files getting too big.

  Ssh-Cluster Command

Toil provides the ability to ssh into the leader of the cluster. This can be done as follows:

            $ toil ssh-cluster CLUSTER-NAME-HERE

This will open a shell on the Toil leader and is used to start an Running a Workflow with Autoscaling <#autoscaling> run. Issues with docker prevent using screen and tmux when sshing the cluster (The shell doesn't know that it is a TTY which prevents it from allocating a new screen session). This can be worked around via

            $ script
            $ screen

Simply running screen within script will get things working properly again.

Finally, you can execute remote commands with the following syntax:

            $ toil ssh-cluster CLUSTER-NAME-HERE remoteCommand

It is not advised that you run your Toil workflow using remote execution like this unless a tool like nohup <https://linux.die.net/man/1/nohup> is used to ensure the process does not die if the SSH connection is interrupted.

For an example usage, see Running a Workflow with Autoscaling <#autoscaling>.

  Rsync-Cluster Command

The most frequent use case for the rsync-cluster utility is deploying your workflow code to the Toil leader. Note that the syntax is the same as traditional rsync <https://linux.die.net/man/1/rsync> with the exception of the hostname before the colon. This is not needed in toil rsync-cluster since the hostname is automatically determined by Toil.

Here is an example of its usage:

            $ toil rsync-cluster CLUSTER-NAME-HERE \
               ~/localFile :/remoteDestination

  Destroy-Cluster Command

The destroy-cluster command is the advised way to get rid of any Toil cluster launched using the Launch-Cluster Command command. It ensures that all attached nodes, volumes, security groups, etc. are deleted. If a node or cluster is shut down using Amazon's online portal residual resources may still be in use in the background. To delete a cluster run

            $ toil destroy-cluster CLUSTER-NAME-HERE

  Storage (Toil jobStore)

Toil can make use of cloud storage such as AWS or Google buckets to take care of storage needs.

This is useful when running Toil in single machine mode on any cloud platform since it allows you to make use of their integrated storage systems.

For an overview of the job store see Job Store <#jobstoreoverview>.

For instructions configuring a particular job store see:

    •   AWS Job Store <#awsjobstore>

    •   Google Job Store <#googlejobstore>

CLOUD PLATFORMS

  Running on Kubernetes

Kubernetes <https://kubernetes.io/> is a very popular container orchestration tool that has become a de facto cross-cloud-provider API for accessing cloud resources. Major cloud providers like Amazon <https://aws.amazon.com/kubernetes/>, Microsoft <https://azure.microsoft.com/en-us/overview/kubernetes-getting-started/>, Kubernetes owner Google <https://cloud.google.com/kubernetes-engine/>, and DigitalOcean <https://www.digitalocean.com/products/kubernetes/> have invested heavily in making Kubernetes work well on their platforms, by writing their own deployment documentation and developing provider-managed Kubernetes-based products. Using minikube <https://github.com/kubernetes/minikube>, Kubernetes can even be run on a single machine.

Toil supports running Toil workflows against a Kubernetes cluster, either in the cloud or deployed on user-owned hardware.

  Preparing your Kubernetes environment

    1.  Get a Kubernetes cluster

        To run Toil workflows on Kubernetes, you need to have a Kubernetes cluster set up. This will not be covered here, but there are many options available, and which one you choose will depend on which cloud ecosystem if any you use already, and on pricing. If you are just following along with the documentation, use minikube on your local machine.

        Alternatively, Toil can set up a Kubernetes cluster for you with the Toil provisioner <#installprovisioner>. Follow this <#provisioningwithkubernetes> guide to get started with a Toil-managed Kubernetes cluster on AWS.

        Note that currently the only way to run a Toil workflow on Kubernetes is to use the AWS Job Store, so your Kubernetes workflow will currently have to store its data in Amazon's cloud regardless of where you run it. This can result in significant egress charges from Amazon if you run it outside of Amazon.

        Kubernetes Cluster Providers:

        •   Your own institution

        •   Amazon EKS <https://aws.amazon.com/eks/>

        •   Microsoft Azure AKS <https://docs.microsoft.com/en-us/azure/aks/>

        •   Google GKE <https://cloud.google.com/kubernetes-engine/>

        •   DigitalOcean Kubernetes <https://www.digitalocean.com/docs/kubernetes/>

        •   minikube <https://kubernetes.io/docs/tasks/tools/install-minikube/>

    2.  Get a Kubernetes context on your local machine

        There are two main ways to run Toil workflows on Kubernetes. You can either run the Toil leader on a machine outside the cluster, with jobs submitted to and run on the cluster, or you can submit the Toil leader itself as a job and have it run inside the cluster. Either way, you will need to configure your own machine to be able to submit jobs to the Kubernetes cluster. Generally, this involves creating and populating a file named .kube/config in your user's home directory, and specifying the cluster to connect to, the certificate and token information needed for mutual authentication, and the Kubernetes namespace within which to work. However, Kubernetes configuration can also be picked up from other files in the .kube directory, environment variables, and the enclosing host when running inside a Kubernetes-managed container.

        You will have to do different things here depending on where you got your Kubernetes cluster:

        •   Configuring for Amazon EKS <https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html>

        •   Configuring for Microsoft Azure AKS <https://docs.microsoft.com/en-us/cli/azure/aks?view=azure-cli-latest#az-aks-get-credentials>

        •   Configuring for Google GKE <https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl>

        •   Configuring for DigitalOcean Kubernetes Clusters <https://www.digitalocean.com/docs/kubernetes/how-to/connect-to-cluster/>

        •   Configuring for minikube <https://kubernetes.io/docs/setup/learning-environment/minikube/#kubectl>

    Toil's internal Kubernetes configuration logic mirrors that of the kubectl command. Toil workflows will use the current kubectl context to launch their Kubernetes jobs.

    3.  If running the Toil leader in the cluster, get a service account

        If you are going to run your workflow's leader within the Kubernetes cluster (see Option 1: Running the Leader Inside Kubernetes), you will need a service account in your chosen Kubernetes namespace. Most namespaces should have a service account named default which should work fine. If your cluster requires you to use a different service account, you will need to obtain its name and use it when launching the Kubernetes job containing the Toil leader.

    4.  Set up appropriate permissions

        Your local Kubernetes context and/or the service account you are using to run the leader in the cluster will need to have certain permissions in order to run the workflow. Toil needs to be able to interact with jobs and pods in the cluster, and to retrieve pod logs. You as a user may need permission to set up an AWS credentials secret, if one is not already available. Additionally, it is very useful for you as a user to have permission to interact with nodes, and to shell into pods.

        The appropriate permissions may already be available to you and your service account by default, especially in managed or ease-of-use-optimized setups such as EKS or minikube.

        However, if the appropriate permissions are not already available, you or your cluster administrator will have to grant them manually. The following Role (toil-user) and ClusterRole (node-reader), to be applied with kubectl apply -f filename.yaml, should grant sufficient permissions to run Toil workflows when bound to your account and the service account used by Toil workflows. Be sure to replace YOUR_NAMESPACE_HERE with the namespace you are running your workflows in

                apiVersion: rbac.authorization.k8s.io/v1
                kind: Role
                metadata:
                  namespace: YOUR_NAMESPACE_HERE
                  name: toil-user
                rules:
                - apiGroups: ["*"]
                  resources: ["*"]
                  verbs: ["explain", "get", "watch", "list", "describe", "logs", "attach", "exec", "port-forward", "proxy", "cp", "auth"]
                - apiGroups: ["batch"]
                  resources: ["*"]
                  verbs: ["get", "watch", "list", "create", "run", "set", "delete"]
                - apiGroups: [""]
                  resources: ["secrets", "pods", "pods/attach", "podtemplates", "configmaps", "events", "services"]
                  verbs: ["patch", "get", "update", "watch", "list", "create", "run", "set", "delete", "exec"]
                - apiGroups: [""]
                  resources: ["pods", "pods/log"]
                  verbs: ["get", "list"]
                - apiGroups: [""]
                  resources: ["pods/exec"]
                  verbs: ["create"]

                apiVersion: rbac.authorization.k8s.io/v1
                kind: ClusterRole
                metadata:
                  name: node-reader
                rules:
                - apiGroups: [""]
                  resources: ["nodes"]
                  verbs: ["get", "list", "describe"]
                - apiGroups: [""]
                  resources: ["namespaces"]
                  verbs: ["get", "list", "describe"]
                - apiGroups: ["metrics.k8s.io"]
                  resources: ["*"]
                  verbs: ["*"]

    To bind a user or service account to the Role or ClusterRole and actually grant the permissions, you will need a RoleBinding and a ClusterRoleBinding, respectively. Make sure to fill in the namespace, username, and service account name, and add more user stanzas if your cluster is to support multiple Toil users.

                apiVersion: rbac.authorization.k8s.io/v1
                kind: RoleBinding
                metadata:
                  name: toil-developer-member
                  namespace: toil
                subjects:
                - kind: User
                  name: YOUR_KUBERNETES_USERNAME_HERE
                  apiGroup: rbac.authorization.k8s.io
                - kind: ServiceAccount
                  name: YOUR_SERVICE_ACCOUNT_NAME_HERE
                  namespace: YOUR_NAMESPACE_HERE
                roleRef:
                  kind: Role
                  name: toil-user
                  apiGroup: rbac.authorization.k8s.io

                apiVersion: rbac.authorization.k8s.io/v1
                kind: ClusterRoleBinding
                metadata:
                  name: read-nodes
                subjects:
                - kind: User
                  name: YOUR_KUBERNETES_USERNAME_HERE
                  apiGroup: rbac.authorization.k8s.io
                - kind: ServiceAccount
                  name: YOUR_SERVICE_ACCOUNT_NAME_HERE
                  namespace: YOUR_NAMESPACE_HERE
                roleRef:
                  kind: ClusterRole
                  name: node-reader
                  apiGroup: rbac.authorization.k8s.io

  AWS Job Store for Kubernetes

Currently, the only job store, which is what Toil uses to exchange data between jobs, that works with jobs running on Kubernetes is the AWS Job Store. This requires that the Toil leader and Kubernetes jobs be able to connect to and use Amazon S3 and Amazon SimpleDB. It also requires that you have an Amazon Web Services account.

    1.  Get access to AWS S3 and SimpleDB

        In your AWS account, you need to create an AWS access key. First go to the IAM dashboard; for "us-west1", the link would be:

                https://console.aws.amazon.com/iam/home?region=us-west-1#/home

    Then create an access key, and save the Access Key ID and the Secret Key. As documented in the AWS documentation <https://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html>:

        1.  On the IAM Dashboard page, choose your account name in the navigation bar, and then choose My Security Credentials.

        2.  Expand the Access keys (access key ID and secret access key) section.

        3.  Choose Create New Access Key. Then choose Download Key File to save the access key ID and secret access key to a file on your computer. After you close the dialog box, you can't retrieve this secret access key again.

    Make sure that, if your AWS infrastructure requires your user to authenticate with a multi-factor authentication (MFA) token, you obtain a second secret key and access key that don't have this requirement. The secret key and access key used to populate the Kubernetes secret that allows the jobs to contact the job store need to be usable without human intervention.

    2.  Configure AWS access from the local machine

        This only really needs to happen if you run the leader on the local machine. But we need the files in place to fill in the secret in the next step. Run:

                $ aws configure

    Then when prompted, enter your secret key and access key. This should create a file ~/.aws/credentials that looks like this:

                [default]
                aws_access_key_id =  BLAH
                aws_secret_access_key =  blahblahblah

    3.  Create a Kubernetes secret to give jobs access to AWS
        Go into the directory where the credentials file is:

                    $ cd ~/.aws

        Then, create a Kubernetes secret that contains it. We'll call it aws-credentials:

                    $ kubectl create secret generic aws-credentials --from-file credentials

  Configuring Toil for your Kubernetes environment

To configure your workflow to run on Kubernetes, you will have to configure several environment variables, in addition to passing the --batchSystem kubernetes option. Doing the research to figure out what values to give these variables may require talking to your cluster provider.

    1.  TOIL_AWS_SECRET_NAME is the most important, and must be set to the secret that contains your AWS credentials file, if your cluster nodes don't otherwise have access to S3 and SimpleDB (such as through IAM roles). This is required for the AWS job store to work, which is currently the only job store that can be used on Kubernetes. In this example we are using aws-credentials.

    2.  TOIL_KUBERNETES_HOST_PATH can be set to allow Toil jobs on the same physical host to share a cache. It should be set to a path on the host where the shared cache should be stored. It will be mounted as /var/lib/toil, or at TOIL_WORKDIR if specified, inside the container. This path must already exist on the host, and must have as much free space as your Kubernetes node offers to jobs. In this example, we are using /data/scratch. To actually make use of caching, make sure not to use --disableCaching.

    3.  TOIL_KUBERNETES_OWNER should be set to the username of the user running the Toil workflow. The jobs that Toil creates will include this username, so they can be more easily recognized, and cleaned up by the user if anything happens to the Toil leader. In this example we are using demo-user.

    4.  TOIL_KUBERNETES_PRIVILEGED can be set to True or False. When true, this allows pods to run as privileged, enabling FUSE mounts for Singularity for faster runtimes. If this is not set to true, Singularity will extract images to sandbox directories. This is unset/False by default except in Toil-managed clusters.

Note that Docker containers cannot be run inside of unprivileged Kubernetes pods (which are themselves containers). The Docker daemon does not (yet) support this. Other tools, such as Singularity in its user-namespace mode, are able to run containers from within containers. If using Singularity to run containerized tools, and you want downloaded container images to persist between Toil jobs, some setup may be required:

On non-Toil managed clusters: You will also want to set TOIL_KUBERNETES_HOST_PATH, and make sure that Singularity is downloading its containers under the Toil work directory (/var/lib/toil by default) by setting SINGULARITY_CACHEDIR.

On Toil-managed clusters: On clusters created with the launch-cluster command, no setup is required. TOIL_KUBERNETES_HOST_PATH is already set to /var/lib/toil. SINGULARITY_CACHEDIR is set to /var/lib/toil/singularity which is a shared location; however, you may need to implement Singularity locking as shown below or change the Singularity cache location to somewhere else.

If using toil-wdl-runner, all the necessary locking for Singularity is already in place and no work should be necessary. Else, for both Toil managed and non-Toil managed clusters, you will need to make sure that no two jobs try to download the same container at the same time; Singularity has no synchronization or locking around its cache, but the cache is also not safe for simultaneous access by multiple Singularity invocations. Some Toil workflows use their own custom workaround logic for this problem; for example, see this section in toil-wdl-runner <https://github.com/DataBiosphere/toil/blob/1821782b9ec40431e8afa36030fb8ee5df943003/src/toil/wdl/wdltoil.py#L1675-L1679>.

  Running workflows

To run the workflow, you will need to run the Toil leader process somewhere. It can either be run inside Kubernetes as a Kubernetes job, or outside Kubernetes as a normal command.

  Option 1: Running the Leader Inside Kubernetes

Once you have determined a set of environment variable values for your workflow run, write a YAML file that defines a Kubernetes job to run your workflow with that configuration. Some configuration items (such as your username, and the name of your AWS credentials secret) need to be written into the YAML so that they can be used from the leader as well.

Note that the leader pod will need your workflow, its other dependencies, and Toil all installed. An easy way to get Toil installed is to start with the Toil appliance image for the version of Toil you want to use. In this example, we use quay.io/ucsc_cgl/toil:5.5.0.

Here's an example YAML file to run a test workflow:

            apiVersion: batch/v1
            kind: Job
            metadata:
              # It is good practice to include your username in your job name.
              # Also specify it in TOIL_KUBERNETES_OWNER
              name: demo-user-toil-test
            # Do not try and rerun the leader job if it fails

            spec:
             backoffLimit: 0
             template:
               spec:
                 # Do not restart the pod when the job fails, but keep it around so the
                 # log can be retrieved
                 restartPolicy: Never
                 volumes:
                 - name: aws-credentials-vol
                   secret:
                     # Make sure the AWS credentials are available as a volume.
                     # This should match TOIL_AWS_SECRET_NAME
                     secretName: aws-credentials
                 # You may need to replace this with a different service account name as
                 # appropriate for your cluster.
                 serviceAccountName: default
                 containers:
                 - name: main
                   image: quay.io/ucsc_cgl/toil:5.5.0
                   env:
                   # Specify your username for inclusion in job names
                   - name: TOIL_KUBERNETES_OWNER
                     value: demo-user
                   # Specify where to find the AWS credentials to access the job store with
                   - name: TOIL_AWS_SECRET_NAME
                     value: aws-credentials
                   # Specify where per-host caches should be stored, on the Kubernetes hosts.
                   # Needs to be set for Toil's caching to be efficient.
                   - name: TOIL_KUBERNETES_HOST_PATH
                     value: /data/scratch
                   volumeMounts:
                   # Mount the AWS credentials volume
                   - mountPath: /root/.aws
                     name: aws-credentials-vol
                   resources:
                     # Make sure to set these resource limits to values large enough
                     # to accommodate the work your workflow does in the leader
                     # process, but small enough to fit on your cluster.
                     #
                     # Since no request values are specified, the limits are also used
                     # for the requests.
                     limits:
                       cpu: 2
                       memory: "4Gi"
                       ephemeral-storage: "10Gi"
                   command:
                   - /bin/bash
                   - -c
                   - |
                     # This Bash script will set up Toil and the workflow to run, and run them.
                     set -e
                     # We make sure to create a work directory; Toil can't hot-deploy a
                     # Python file from the root of the filesystem, which is where we start.
                     mkdir /tmp/work
                     cd /tmp/work
                     # We make a virtual environment to allow workflow dependencies to be
                     # hot-deployed.
                     #
                     # We don't really make use of it in this example, but for workflows
                     # that depend on PyPI packages we will need this.
                     #
                     # We use --system-site-packages so that the Toil installed in the
                     # appliance image is still available.
                     virtualenv --python python3 --system-site-packages venv
                     . venv/bin/activate
                     # Now we install the workflow. Here we're using a demo workflow
                     # from Toil itself.
                     wget https://raw.githubusercontent.com/DataBiosphere/toil/releases/4.1.0/src/toil/test/docs/scripts/tutorial_helloworld.py
                     # Now we run the workflow. We make sure to use the Kubernetes batch
                     # system and an AWS job store, and we set some generally useful
                     # logging options. We also make sure to enable caching.
                     python3 tutorial_helloworld.py \
                         aws:us-west-2:demouser-toil-test-jobstore \
                         --batchSystem kubernetes \
                         --realTimeLogging \
                         --logInfo

You can save this YAML as leader.yaml, and then run it on your Kubernetes installation with:

            $ kubectl apply -f leader.yaml

To monitor the progress of the leader job, you will want to read its logs. If you are using a Kubernetes dashboard such as k9s <https://github.com/derailed/k9s>, you can simply find the pod created for the job in the dashboard, and view its logs there. If not, you will need to locate the pod by hand.

  Monitoring and Debugging Kubernetes Jobs and Pods

The following techniques are most useful for looking at the pod which holds the Toil leader, but they can also be applied to individual Toil jobs on Kubernetes, even when the leader is outside the cluster.

Kubernetes names pods for jobs by appending a short random string to the name of the job. You can find the name of the pod for your job by doing:

            $ kubectl get pods | grep demo-user-toil-test
            demo-user-toil-test-g5496                                         1/1     Running     0          2m

Assuming you have set TOIL_KUBERNETES_OWNER correctly, you should be able to find all of your workflow's pods by searching for your username:

            $ kubectl get pods | grep demo-user

If the status of a pod is anything other than Pending, you will be able to view its logs with:

            $ kubectl logs demo-user-toil-test-g5496

This will dump the pod's logs from the beginning to now and terminate. To follow along with the logs from a running pod, add the -f option:

            $ kubectl logs -f demo-user-toil-test-g5496

A status of ImagePullBackoff suggests that you have requested to use an image that is not available. Check the image section of your YAML if you are looking at a leader, or the value of TOIL_APPLIANCE_SELF if you are delaying with a worker job. You also might want to check your Kubernetes node's Internet connectivity and DNS function; in Kubernetes, DNS depends on system-level pods which can be terminated or evicted in cases of resource oversubscription, just like user workloads.

If your pod seems to be stuck Pending, ContainerCreating, you can get information on what is wrong with it by using kubectl describe pod:

            $ kubectl describe pod demo-user-toil-test-g5496

Pay particular attention to the Events: section at the end of the output. An indication that a job is too big for the available nodes on your cluster, or that your cluster is too busy for your jobs, is FailedScheduling events:

            Type     Reason            Age                  From               Message
            ----     ------            ----                 ----               -------
            Warning  FailedScheduling  13s (x79 over 100m)  default-scheduler  0/4 nodes are available: 1 Insufficient cpu, 1 Insufficient ephemeral-storage, 4 Insufficient memory.

If a pod is running but seems to be behaving erratically, or seems stuck, you can shell into it and look around:

            $ kubectl exec -ti demo-user-toil-test-g5496 /bin/bash

One common cause of stuck pods is attempting to use more memory than allowed by Kubernetes (or by the Toil job's memory resource requirement), but in a way that does not trigger the Linux OOM killer to terminate the pod's processes. In these cases, the pod can remain stuck at nearly 100% memory usage more or less indefinitely, and attempting to shell into the pod (which needs to start a process within the pod, using some of its memory) will fail. In these cases, the recommended solution is to kill the offending pod and increase its (or its Toil job's) memory requirement, or reduce its memory needs by adapting user code.

  When Things Go Wrong

The Toil Kubernetes batch system includes cleanup code to terminate worker jobs when the leader shuts down. However, if the leader pod is removed by Kubernetes, is forcibly killed or otherwise suffers a sudden existence failure, it can go away while its worker jobs live on. It is not recommended to restart a workflow in this state, as jobs from the previous invocation will remain running and will be trying to modify the job store concurrently with jobs from the new invocation.

To clean up dangling jobs, you can use the following snippet:

            $ kubectl get jobs | grep demo-user | cut -f1 -d' ' | xargs -n10 kubectl delete job

This will delete all jobs with demo-user's username in their names, in batches of 10. You can also use the UUID that Toil assigns to a particular workflow invocation in the filter, to clean up only the jobs pertaining to that workflow invocation.

  Option 2: Running the Leader Outside Kubernetes

If you don't want to run your Toil leader inside Kubernetes, you can run it locally instead. This can be useful when developing a workflow; files can be hot-deployed from your local machine directly to Kubernetes. However, your local machine will have to have (ideally role-assumption- and MFA-free) access to AWS, and access to Kubernetes. Real time logging will not work unless your local machine is able to listen for incoming UDP packets on arbitrary ports on the address it uses to contact the IPv4 Internet; Toil does no NAT traversal or detection.

Note that if you set TOIL_WORKDIR when running your workflow like this, it will need to be a directory that exists both on the host and in the Toil appliance.

Here is an example of running our test workflow leader locally, outside of Kubernetes:

            $ export TOIL_KUBERNETES_OWNER=demo-user  # This defaults to your local username if not set
            $ export TOIL_AWS_SECRET_NAME=aws-credentials
            $ export TOIL_KUBERNETES_HOST_PATH=/data/scratch
            $ virtualenv --python python3 --system-site-packages venv
            $ . venv/bin/activate
            $ wget https://raw.githubusercontent.com/DataBiosphere/toil/releases/4.1.0/src/toil/test/docs/scripts/tutorial_helloworld.py
            $ python3 tutorial_helloworld.py \
                  aws:us-west-2:demouser-toil-test-jobstore \
                  --batchSystem kubernetes \
                  --realTimeLogging \
                  --logInfo

  Running CWL Workflows

Running CWL workflows on Kubernetes can be challenging, because executing CWL can require toil-cwl-runner to orchestrate containers of its own, within a Kubernetes job running in the Toil appliance container.

Normally, running a CWL workflow should Just Work, as long as the workflow's Docker containers are able to be executed with Singularity, your Kubernetes cluster does not impose extra capability-based confinement (i.e. SELinux, AppArmor) that interferes with Singularity's use of user-mode namespaces, and you make sure to configure Toil so that its workers know where to store their data within the Kubernetes pods (which would be done for you if using a Toil-managed cluster). For example, you should be able to run a CWL workflow like this:

            $ export TOIL_KUBERNETES_OWNER=demo-user  # This defaults to your local username if not set
            $ export TOIL_AWS_SECRET_NAME=aws-credentials
            $ export TOIL_KUBERNETES_HOST_PATH=/data/scratch
            $ virtualenv --python python3 --system-site-packages venv
            $ . venv/bin/activate
            $ pip install toil[kubernetes,cwl]==5.8.0
            $ toil-cwl-runner  \
                 --jobStore  aws:us-west-2:demouser-toil-test-jobstore \
                 --batchSystem kubernetes \
                 --realTimeLogging \
                 --logInfo \
                 --disableCaching \
                 path/to/cwl/workflow \
                 path/to/cwl/input/object

Additional cwltool options that your workflow might require, such as --no-match-user, can be passed to toil-cwl-runner, which inherits most cwltool options.

  AppArmor and Singularity

Kubernetes clusters based on Ubuntu hosts often will have AppArmor enabled on the host. AppArmor is a capability-based security enhancement system that integrates with the Linux kernel to enforce lists of things which programs may or may not do, called profiles. For example, an AppArmor profile could be applied to a web server process to stop it from using the mount() system call to manipulate the filesystem, because it has no business doing that under normal circumstances but might attempt to do it if compromised by hackers.

Kubernetes clusters also often use Docker as the backing container runtime, to run pod containers. When AppArmor is enabled, Docker will load an AppArmor profile and apply it to all of its containers by default, with the ability for the profile to be overridden on a per-container basis. This profile unfortunately prevents some of the mount() system calls that Singularity uses to set up user-mode containers from working inside the pod, even though these calls would be allowed for an unprivileged user under normal circumstances.

On the UCSC Kubernetes cluster, we configure our Ubuntu hosts with an alternative default AppArmor profile for Docker containers <https://github.com/adamnovak/gi-kubernetes-autoscaling-config/blob/e1350ac9ad17d94b5073b20db3c75620957926e3/kubenode.ubuntu.cloud-config.yaml#L27-L67> which allows these calls. Other solutions include turning off AppArmor on the host, configuring Kubernetes with a container runtime other than Docker, or using Kubernetes's AppArmor integration <https://kubernetes.io/docs/tutorials/security/apparmor/> to apply a more permissive profile or the unconfined profile to pods that Toil launches.

Toil does not yet have a way to apply a container.apparmor.security.beta.kubernetes.io/runner-container: unconfined annotation to its pods, as described in the Kubernetes AppArmor documentation <https://kubernetes.io/docs/tutorials/security/apparmor/#securing-a-pod>. This feature is tracked in issue #4331 <https://github.com/DataBiosphere/toil/issues/4331>.

  Running in AWS

Toil jobs can be run on a variety of cloud platforms. Of these, Amazon Web Services (AWS) is currently the best-supported solution. Toil provides the Toil Cluster Utilities <#clusterutils> to conveniently create AWS clusters, connect to the leader of the cluster, and then launch a workflow. The leader handles distributing the jobs over the worker nodes and autoscaling to optimize costs.

The Running a Workflow with Autoscaling section details how to create a cluster and run a workflow that will dynamically scale depending on the workflow's needs.

The Static Provisioning section explains how a static cluster (one that won't automatically change in size) can be created and provisioned (grown, shrunk, destroyed, etc.).

  Preparing your AWS environment

To use Amazon Web Services (AWS) to run Toil or to just use S3 to host the files during the computation of a workflow, first set up and configure an account with AWS:

    1.  If necessary, create and activate an AWS account <https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/>

    2.  Next, generate a key pair for AWS with the command (do NOT generate your key pair with the Amazon browser):

                $ ssh-keygen -t rsa

    3.  This should prompt you to save your key. Please save it in

                ~/.ssh/id_rsa

    4.  Now move this to where your OS can see it as an authorized key:

                $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

    5.  Next, you'll need to add your key to the ssh-agent:

                $ eval `ssh-agent -s`
                $ ssh-add

    If your key has a passphrase, you will be prompted to enter it here once.

    6.  You'll also need to chmod your private key (good practice but also enforced by AWS):

                $ chmod 400 id_rsa

    7.  Now you'll need to add the key to AWS via the browser. For example, on us-west1, this address would accessible at:

                https://us-west-1.console.aws.amazon.com/ec2/v2/home?region=us-west-1#KeyPairs:sort=keyName

    8.  Now click on the "Import Key Pair" button to add your key:
            [image: Adding an Amazon Key Pair] [image]
             <https://us-west-1.console.aws.amazon.com/ec2/v2/home?region=us-west-1#KeyPairs:sort=keyName>.UNINDENT

        9.  Next, you need to create an AWS access key. First go to the IAM dashboard, again; for "us-west1", the example link would be here:

                    https://console.aws.amazon.com/iam/home?region=us-west-1#/home

        10. The directions (transcribed from: <https://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html> ) are now:

                    1.  On the IAM Dashboard page, choose your account name in the navigation bar, and then choose My Security Credentials.

                    2.  Expand the Access keys (access key ID and secret access key) section.

                    3.  Choose Create New Access Key. Then choose Download Key File to save the access key ID and secret access key to a file on your computer. After you close the dialog box, you can't retrieve this secret access key again.

        11. Now you should have a newly generated "AWS Access Key ID" and "AWS Secret Access Key". We can now install the AWS CLI and make sure that it has the proper credentials:

                    $ pip install awscli --upgrade --user

        12. Now configure your AWS credentials with:

                    $ aws configure

        13. Add your "AWS Access Key ID" and "AWS Secret Access Key" from earlier and your region and output format:

                    " AWS Access Key ID [****************Q65Q]: "
                    " AWS Secret Access Key [****************G0ys]: "
                    " Default region name [us-west-1]: "
                    " Default output format [json]: "

        This will create the files ~/.aws/config and ~/.aws/credentials.

        14. If not done already, install toil (example uses version 5.12.0, but we recommend the latest release):

                    $ virtualenv venv
                    $ source venv/bin/activate
                    $ pip install toil[all]==5.12.0

        15. Now that toil is installed and you are running a virtualenv, an example of launching a toil leader node would be the following (again, note that we set TOIL_APPLIANCE_SELF to toil version 5.3.0 in this example, but please set the version to the installed version that you are using if you're using a different version):

                    $ toil launch-cluster <cluster-name> \
                          --clusterType kubernetes \
                          --leaderNodeType t2.medium \
                          --nodeTypes t2.medium -w 1 \
                          --zone us-west-1a \
                          --keyPairName id_rsa

    To further break down each of these commands:
            toil launch-cluster --- Base command in toil to launch a cluster.

            <cluster-name> --- Just choose a name for your cluster.

            --clusterType kubernetes --- Specify the type of cluster to coordinate and execute your workflow. Kubernetes is the recommended option.

            --leaderNodeType t2.medium --- Specify the leader node type. Make a t2.medium (2CPU; 4Gb RAM; $0.0464/Hour). List of available AWS instances: <https://aws.amazon.com/ec2/pricing/on-demand/>

            --nodeTypes t2.medium -w 1 --- Specify the worker node type and the number of worker nodes to launch. The Kubernetes cluster requires at least 1 worker node.

            --zone us-west-1a --- Specify the AWS zone you want to launch the instance in. Must have the same prefix as the zone in your awscli credentials (which, in the example of this tutorial is: "us-west-1").

            --keyPairName id_rsa --- The name of your key pair, which should be "id_rsa" if you've followed this tutorial.

    Note:
            You can set the TOIL_AWS_TAGS environment variable to a JSON object to specify arbitrary tags for AWS resources. For example, if you export TOIL_AWS_TAGS='{"project-name": "variant-calling"}' in your shell before using Toil, AWS resources created by Toil will be tagged with a project-name tag with the value variant-calling.

            You can also set the TOIL_APPLIANCE_SELF environment variable to one of the Toil project's Docker images <https://quay.io/repository/ucsc_cgl/toil?tag=latest&tab=tags>, if you would like to launch a cluster using a different version of Toil than the one you have installed.

  AWS Job Store

Using the AWS job store is straightforward after you've finished Preparing your AWS environment; all you need to do is specify the prefix for the job store name.

To run the sort example sort example <#sortexample> with the AWS job store you would type

            $ python3 sort.py aws:us-west-2:my-aws-sort-jobstore

  Toil Provisioner

The Toil provisioner is the component responsible for creating resources in Amazon's cloud. It is included in Toil alongside the [aws] extra and allows us to spin up a cluster.

Getting started with the provisioner is simple:

    1.  Make sure you have Toil installed with the AWS extras. For detailed instructions see Installing Toil with Extra Features <#extras>.

    2.  You will need an AWS account and you will need to save your AWS credentials on your local machine. For help setting up an AWS account see here <http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html>. For setting up your AWS credentials follow instructions here <http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-config-files>.

The Toil provisioner makes heavy use of the Toil Appliance, a Docker image that bundles Toil and all its requirements (e.g. Kubernetes). This makes deployment simple across platforms, and you can even simulate a cluster locally (see Developing with Docker <#appliance-dev> for details).

        Choosing Toil Appliance Image

            When using the Toil provisioner, the appliance image will be automatically chosen based on the pip-installed version of Toil on your system. That choice can be overridden by setting the environment variables TOIL_DOCKER_REGISTRY and TOIL_DOCKER_NAME or TOIL_APPLIANCE_SELF. See Environment Variables <#envars> for more information on these variables. If you are developing with autoscaling and want to test and build your own appliance have a look at Developing with Docker <#appliance-dev>.

For information on using the Toil Provisioner have a look at Running a Workflow with Autoscaling.

  Details about Launching a Cluster in AWS

Using the provisioner to launch a Toil leader instance is simple using the launch-cluster command. For example, to launch a Kubernetes cluster named "my-cluster" with a t2.medium leader in the us-west-2a zone, run

            (venv) $ toil launch-cluster my-cluster \
                         --clusterType kubernetes \
                         --leaderNodeType t2.medium \
                         --nodeTypes t2.medium -w 1 \
                         --zone us-west-2a \
                         --keyPairName <AWS-key-pair-name>

The cluster name is used to uniquely identify your cluster and will be used to populate the instance's Name tag. Also, the Toil provisioner will automatically tag your cluster with an Owner tag that corresponds to your keypair name to facilitate cost tracking. In addition, the ToilNodeType tag can be used to filter "leader" vs. "worker" nodes in your cluster.

The leaderNodeType is an EC2 instance type <https://aws.amazon.com/ec2/instance-types/>. This only affects the leader node.

The --zone parameter specifies which EC2 availability zone to launch the cluster in. Alternatively, you can specify this option via the TOIL_AWS_ZONE environment variable. Note: the zone is different from an EC2 region. A region corresponds to a geographical area like us-west-2 (Oregon), and availability zones are partitions of this area like us-west-2a.

By default, Toil creates an IAM role for each cluster with sufficient permissions to perform cluster operations (e.g. full S3, EC2, and SDB access). If the default permissions are not sufficient for your use case (e.g. if you need access to ECR), you may create a custom IAM role with all necessary permissions and set the --awsEc2ProfileArn parameter when launching the cluster. Note that your custom role must at least have these permissions <#minawspermissions> in order for the Toil cluster to function properly.

In addition, Toil creates a new security group with the same name as the cluster name with default rules (e.g. opens port 22 for SSH access). If you require additional security groups, you may use the --awsEc2ExtraSecurityGroupId parameter when launching the cluster. Note: Do not use the same name as the cluster name for the extra security groups as any security group matching the cluster name will be deleted once the cluster is destroyed.

For more information on options try:

            (venv) $ toil launch-cluster --help

  Static Provisioning

Toil can be used to manage a cluster in the cloud by using the Toil Cluster Utilities <#clusterutils>. The cluster utilities also make it easy to run a toil workflow directly on this cluster. We call this static provisioning because the size of the cluster does not change. This is in contrast with Running a Workflow with Autoscaling.

To launch worker nodes alongside the leader we use the -w option:

            (venv) $ toil launch-cluster my-cluster \
                         --clusterType kubernetes \
                         --leaderNodeType t2.small -z us-west-2a \
                         --keyPairName <AWS-key-pair-name> \
                         --nodeTypes m3.large,t2.micro -w 1,4 \
                         --zone us-west-2a

This will spin up a leader node of type t2.small with five additional workers --- one m3.large instance and four t2.micro.

Currently static provisioning is only possible during the cluster's creation. The ability to add new nodes and remove existing nodes via the native provisioner is in development. Of course the cluster can always be deleted with the Destroy-Cluster Command <#destroycluster> utility.

  Uploading Workflows

Now that our cluster is launched, we use the Rsync-Cluster Command <#rsynccluster> utility to copy the workflow to the leader. For a simple workflow in a single file this might look like

            (venv) $ toil rsync-cluster -z us-west-2a my-cluster toil-workflow.py :/

Note:
        If your toil workflow has dependencies have a look at the Auto-Deployment <#autodeploying> section for a detailed explanation on how to include them.

  Running a Workflow with Autoscaling

Toil can create an autoscaling Kubernetes cluster for you using the AWS provisioner. Autoscaling is a feature of running Toil in a cloud whereby additional cloud instances are launched as needed to run the workflow.

Note:
        Make sure you've done the AWS setup in Preparing your AWS environment.

To set up a Kubernetes cluster, simply use the --clusterType=kubernetes command line option to toil launch-cluster. To make it autoscale, specify a range of possible node counts for a node type (such as -w 1-4). The cluster will automatically add and remove nodes, within that range, depending on how many seem to be needed to run the jobs submitted to the cluster.

For example, to launch a Toil cluster with a Kubernetes scheduler, run:

            (venv) $ toil launch-cluster <cluster-name> \
                    --provisioner=aws \
                    --clusterType kubernetes \
                    --zone us-west-2a \
                    --keyPairName <AWS-key-pair-name> \
                    --leaderNodeType t2.medium \
                    --leaderStorage 50 \
                    --nodeTypes t2.medium -w 1-4 \
                    --nodeStorage 20 \
                    --logDebug

Behind the scenes, Toil installs kubeadm and configures the kubelet on the Toil leader and all worker nodes. This Toil cluster can then schedule jobs using Kubernetes.

Note:
        You should set at least one worker node, otherwise Kubernetes would not be able to schedule any jobs. It is also normal for this step to take a while.

As a demonstration, we will use sort.py again, but run it on a Toil cluster with Kubernetes. First, download this file and put it to the current working directory.

We then need to copy over the workflow file and SSH into the cluster:

            (venv) $ toil rsync-cluster -z us-west-2a <cluster-name> sort.py :/root
            (venv) $ toil ssh-cluster -z us-west-2a <cluster-name>

Remember to replace <cluster-name> with your actual cluster name, and feel free to use your own cluster configuration and/or workflow files. For more information on this step, see the corresponding section of the Static Provisioning tutorial.

Important:
        Some important caveats about starting a toil run through an ssh session are explained in the Ssh-Cluster Command <#sshcluster> section.

Now that we are inside the cluster, a Kubernetes environment should already be configured and running. To verify this, simply run:

            $ kubectl get nodes

You should see a leader node with the Ready status. Depending on the number of worker nodes you set to create upfront, you should also see them displayed here.

Additionally, you can also verify that the metrics server is running:

            $ kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"

If there is a JSON response (similar to the output below), and you are not seeing any errors, that means the metrics server is set up and running, and you are good to start running workflows.

            {"kind":"NodeMetricsList","apiVersion":"metrics.k8s.io/v1beta1", ...}

Note:
        It'll take a while for all nodes to get set up and running, so you might not be able to see all nodes running at first. You can start running workflows already, but Toil might complain until the necessary resources are set up and running.

Now we can run the workflow:

            $ python3 sort.py \
                    --batchSystem kubernetes \
                    aws:<region>:<job-store-name>

Make sure to replace <region> and <job-store-name>. It is required to use a cloud-accessible job store like AWS or Google when using the Kubernetes batch system.

The sort workflow should start running on the Kubernetes cluster set up by Toil. This workflow would take a while to execute, so you could put the job in the background and monitor the Kubernetes cluster using kubectl. For example, you can check out the pods that are running:

            $ kubectl get pods

You should see an output like:

            NAME                                                      READY   STATUS              RESTARTS   AGE
            root-toil-a864e1b0-2e1f-48db-953c-038e5ad293c7-11-4cwdl   0/1     ContainerCreating   0          85s
            root-toil-a864e1b0-2e1f-48db-953c-038e5ad293c7-14-5dqtk   0/1     Completed           0          18s
            root-toil-a864e1b0-2e1f-48db-953c-038e5ad293c7-7-gkwc9    0/1     ContainerCreating   0          107s
            root-toil-a864e1b0-2e1f-48db-953c-038e5ad293c7-9-t7vsb    1/1     Running             0          96s

If a pod failed for whatever reason or if you want to make sure a pod isn't stuck, you can use kubectl describe pod <pod-name> or kubectl logs <pod-name> to inspect the pod.

If everything is successful, you should be able to see an output file from the sort workflow:

            $ head sortedFile.txt

You can now run your own workflows!

  Preemptibility

Toil can run on a heterogeneous cluster of both preemptible and non-preemptible nodes. Being a preemptible node simply means that the node may be shut down at any time, while jobs are running. These jobs can then be restarted later somewhere else.

A node type can be specified as preemptible by adding a spot bid <https://aws.amazon.com/ec2/spot/pricing/> in dollars, after a colon, to its entry in the list of node types provided with the --nodeTypes flag. If spot instance prices rise above your bid, the preemptible nodes will be shut down.

For example, this cluster will have both preemptible and non-preemptible nodes:

            (venv) $ toil launch-cluster <cluster-name> \
                    --provisioner=aws \
                    --clusterType kubernetes \
                    --zone us-west-2a \
                    --keyPairName <AWS-key-pair-name> \
                    --leaderNodeType t2.medium \
                    --leaderStorage 50 \
                    --nodeTypes t2.medium -w 1-4 \
                    --nodeTypes t2.large:0.20 -w 1-4 \
                    --nodeStorage 20 \
                    --logDebug

Individual jobs can explicitly specify whether they should be run on preemptible nodes via the boolean preemptible resource requirement in Toil's Python API. In CWL, this is exposed as a hint <https://doc.arvados.org/user/cwl/cwl-extensions.html> UsePreemptible in the http://arvados.org/cwl# namespace (usually imported as arv). In WDL, this is exposed as a runtime attribute <https://cromwell.readthedocs.io/en/stable/RuntimeAttributes/#preemptible> preemptible as recognized by Cromwell. Toil's Kubernetes batch system will prefer to schedule preemptible jobs on preemptible nodes.

If a job is not specified to be preemptible, the job will not run on preemptible nodes even if preemptible nodes are available, unless the workflow is run with the --defaultPreemptible flag. The --defaultPreemptible flag will allow jobs without an explicit preemptible requirement to run on preemptible machines. For example:

            $ python3 /root/sort.py aws:us-west-2:<my-jobstore-name> \
                  --batchSystem kubernetes \
                  --defaultPreemptible

        Specify Preemptibility Carefully

            Ensure that your choices for --nodeTypes and --maxNodes <> make sense for your workflow and won't cause it to hang. You should make sure the provisioner is able to create nodes large enough to run the largest job in the workflow, and that non-preemptible node types are allowed if there are non-preemptible jobs in the workflow.

  Using MinIO and S3-Compatible object stores

Toil can be configured to access files stored in an S3-compatible object store <https://en.wikipedia.org/wiki/Amazon_S3#S3_API_and_competing_services> such as MinIO <https://min.io/>. The following environment variables can be used to configure the S3 connection used:

    •   TOIL_S3_HOST: the IP address or hostname to use for connecting to S3

    •   TOIL_S3_PORT: the port number to use for connecting to S3, if needed

    •   TOIL_S3_USE_SSL: enable or disable the usage of SSL for connecting to S3 (True by default)

Examples:

            TOIL_S3_HOST=127.0.0.1
            TOIL_S3_PORT=9010
            TOIL_S3_USE_SSL=False

  In-Workflow Autoscaling with Mesos

Instead of the normal Kubernetes-based autoscaling, you can also use Toil's old Mesos-based autoscaling method, where the scaling logic runs inside the Toil workflow. With this approach, a Toil cluster can only run one workflow at a time. This method also does not work on the ARM architecture.

In this mode, the --preemptibleCompensation flag can be used to handle cases where preemptible nodes may not be available but are required for your workflow. With this flag enabled, the autoscaler will attempt to compensate for a shortage of preemptible nodes of a certain type by creating non-preemptible nodes of that type, if non-preemptible nodes of that type were specified in --nodeTypes.

Note:
        This approach is deprecated, because the Mesos project is no longer publishing up-to-date builds.

    1.  Download sort.py

    2.  Launch a Mesos leader node in AWS using the Launch-Cluster Command <#launchcluster> command, without using any ranges of node counts:

                (venv) $ toil launch-cluster <cluster-name> \
                             --clusterType mesos \
                             --keyPairName <AWS-key-pair-name> \
                             --leaderNodeType t2.medium \
                             --zone us-west-2a

    3.  Copy the sort.py workflow up to the leader node:

                (venv) $ toil rsync-cluster -z us-west-2a <cluster-name> sort.py :/root

    4.  Login to the leader node:

                (venv) $ toil ssh-cluster -z us-west-2a <cluster-name>

    5.  Run the workflow with in-workflow autoscaling, specifying a provisioner and node types and counts as workflow arguments:

                $ python3 /root/sort.py aws:us-west-2:<my-jobstore-name> \
                      --provisioner aws \
                      --nodeTypes c3.large \
                      --maxNodes 2 \
                      --batchSystem mesos

Note:
        In this example, the autoscaling Toil code creates up to two instances of type c3.large and launches Mesos agent containers inside them. The containers are then available to run jobs defined by the sort.py workflow. Toil also creates a bucket in S3 called aws:us-west-2:autoscaling-sort-jobstore to store intermediate job results. The Toil autoscaler can also provision multiple different node types, which is useful for workflows that have jobs with varying resource requirements. For example, one could execute the workflow with --nodeTypes c3.large,r3.xlarge --maxNodes 5,1, which would allow the provisioner to create up to five c3.large nodes and one r3.xlarge node for memory-intensive jobs. In this situation, the autoscaler would avoid creating the more expensive r3.xlarge node until needed, running most jobs on the c3.large nodes.

    1.  View the generated file to sort:

                $ head fileToSort.txt

    2.  View the sorted file:

                $ head sortedFile.txt

  Dashboard

Toil provides a dashboard for viewing the RAM and CPU usage of each node, the number of issued jobs of each type, the number of failed jobs, and the size of the jobs queue. To launch this dashboard for a Toil workflow, pass the --metrics flag on the workflow's command line. The dashboard can then be viewed in your browser at localhost:3000 while connected to the leader node through toil ssh-cluster:

To change the default port number, you can use the --grafana_port argument:

            (venv) $ toil ssh-cluster -z us-west-2a --grafana_port 8000 <cluster-name>

On AWS, the dashboard keeps track of every node in the cluster to monitor CPU and RAM usage, but it can also be used while running a workflow on a single machine. The dashboard uses Grafana as the front end for displaying real-time plots, and Prometheus for tracking metrics exported by toil: [image]

In order to use the dashboard for a non-released toil version, you will have to build the containers locally with make docker, since the prometheus, grafana, and mtail containers used in the dashboard are tied to a specific toil version.

  Running in Google Compute Engine (GCE)

Toil supports a provisioner with Google, and a Google Job Store. To get started, follow instructions for Preparing your Google environment.

  Preparing your Google environment

Toil supports using the Google Cloud Platform <https://cloud.google.com/storage/>. Setting this up is easy!

    1.  Make sure that the google extra (Installing Toil with Extra Features <#extras>) is installed

    2.  Follow Google's Instructions <https://cloud.google.com/docs/authentication/getting-started> to download credentials and set the GOOGLE_APPLICATION_CREDENTIALS environment variable

    3.  Create a new ssh key with the proper format. To create a new ssh key run the command

                $ ssh-keygen -t rsa -f ~/.ssh/id_rsa -C [USERNAME]

    where [USERNAME] is something like jane@example.com. Make sure to leave your password blank.

    Warning:
            This command could overwrite an old ssh key you may be using. If you have an existing ssh key you would like to use, it will need to be called id_rsa and it needs to have no password set.

    Make sure only you can read the SSH keys:

                $ chmod 400 ~/.ssh/id_rsa ~/.ssh/id_rsa.pub

    4.  Add your newly formatted public key to Google. To do this, log into your Google Cloud account and go to metadata <https://console.cloud.google.com/compute/metadata> section under the Compute tab. [image]

        Near the top of the screen click on 'SSH Keys', then edit, add item, and paste the key. Then save: [image]

For more details look at Google's instructions for adding SSH keys <https://cloud.google.com/compute/docs/instances/adding-removing-ssh-keys>.

  Google Job Store

To use the Google Job Store you will need to set the GOOGLE_APPLICATION_CREDENTIALS environment variable by following Google's instructions <https://cloud.google.com/docs/authentication/getting-started>.

Then to run the sort example with the Google job store you would type

            $ python3 sort.py google:my-project-id:my-google-sort-jobstore

  Running a Workflow with Autoscaling

Warning:
        Google Autoscaling is in beta!

The steps to run a GCE workflow are similar to those of AWS (Running a Workflow with Autoscaling <#autoscaling>), except you will need to explicitly specify the --provisioner gce option which otherwise defaults to aws.

    1.  Download sort.py

    2.  Launch the leader node in GCE using the Launch-Cluster Command <#launchcluster> command:

                (venv) $ toil launch-cluster <CLUSTER-NAME> \
                             --provisioner gce \
                             --leaderNodeType n1-standard-1 \
                             --keyPairName <SSH-KEYNAME> \
                             --zone us-west1-a

    Where <SSH-KEYNAME> is the first part of [USERNAME] used when setting up your ssh key. For example if [USERNAME] was <jane@example.com>, <SSH-KEYNAME> should be jane.

    The --keyPairName option is for an SSH key that was added to the Google account. If your ssh key [USERNAME] was jane@example.com, then your key pair name will be just jane.

    3.  Upload the sort example and ssh into the leader:

                (venv) $ toil rsync-cluster --provisioner gce <CLUSTER-NAME> sort.py :/root
                (venv) $ toil ssh-cluster --provisioner gce <CLUSTER-NAME>

    4.  Run the workflow:

                $ python3 /root/sort.py  google:<PROJECT-ID>:<JOBSTORE-NAME> \
                      --provisioner gce \
                      --batchSystem mesos \
                      --nodeTypes n1-standard-2 \
                      --maxNodes 2

    5.  Clean up:

                $ exit  # this exits the ssh from the leader node
                (venv) $ toil destroy-cluster --provisioner gce <CLUSTER-NAME>

HPC ENVIRONMENTS

Toil is a flexible framework that can be leveraged in a variety of environments, including high-performance computing (HPC) environments. Toil provides support for a number of batch systems, including Grid Engine <http://www.univa.com/oracle>, Slurm <https://www.schedmd.com/>, Torque <http://www.adaptivecomputing.com/products/open-source/torque/> and LSF <https://en.wikipedia.org/wiki/Platform_LSF>, which are popular schedulers used in these environments. Toil also supports HTCondor <https://research.cs.wisc.edu/htcondor/>, which is a popular scheduler for high-throughput computing (HTC). To use one of these batch systems specify the --batchSystem argument to the workflow.

Due to the cost and complexity of maintaining support for these schedulers we currently consider all but Slurm to be "community supported", that is the core development team does not regularly test or develop support for these systems. However, there are members of the Toil community currently deploying Toil in a wide variety of HPC environments and we welcome external contributions.

Developing the support of a new or existing batch system involves extending the abstract batch system class toil.batchSystems.abstractBatchSystem.AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>.

  Running on Slurm

When running Toil workflows on Slurm, you usually want to run the workflow itself from the head node. Toil will take care of running all the required sbatch commands for you. You probably do not want to submit the Toil workflow as a Slurm job with sbatch (although you can if you have a large number of workflows to run). You also probably do not want to manually allocate resources with sallocate.

To run a Toil workflow on Slurm, include --batchSystem slurm in your command line arguments. Generally Slurm clusters have shared filesystems, meaning the file job store would be appropriate. You want to make sure to use a job store location that is shared across your Slurm cluster. Additionally, you will likely want to provide another shared directory with the --batchLogsDir option, to allow the Slurm job logs to be retrieved by Toil in case something goes wrong with a job.

For example, to run the sort example sort example <#sortexample> on Slurm, assuming you are currently in a shared directory, you would type, on the cluster head node:

            $ mkdir -p logs
            $ python3 sort.py ./store --batchSystem slurm --batchLogsDir ./logs

By default, this does not include any time limit or particular Slurm partition. If your Slurm cluster requires time limits, add the --slurmTime option to set the time limit to use for jobs.

If you do specify a time limit, a partition will be automatically selected that can accommodate jobs of that duration, and a partition will be automatically selected for jobs that need GPUs. To use a particular partition, use the --slurmPartition argument. If you are running GPU jobs and they need to go to a different partition, use the --slurmGPUPartition argument. For example, to run a WDL workflow from Dockstore <#runwdl> using GPUs on Slurm, with a time limit of 4 hours per job and partitions manually specified, you would run:

            $ toil-wdl-runner '#workflow/github.com/vgteam/vg_wdl/GiraffeDeepVariantFromGAF:gbz' \
              https://raw.githubusercontent.com/vgteam/vg_wdl/refs/heads/gbz/params/giraffe_and_deepvariant_gaf.json \
              --jobStore ./store --batchSystem slurm --batchLogsDir ./logs \
              --slurmTime 4:00:00 --slurmPartition medium --slurmGPUPartition gpu

Any additional Slurm sbatch arguments your cluster needs that aren't directly supported by Toil can be passed via the Toil --slurmArgs option, or the TOIL_SLURM_ARGS environment variable. (There is special handling for some Slurm options Toil needs to interact with, like --time or --partition, if they are passed this way.)

  Slurm Tips

    1.  If using Toil workflows that run containers with Singularity on Slurm (such as WDL workflows), you will want to make sure that Singularity caching, and Toil's MiniWDL caching, use a shared directory across your cluster nodes. By default, Toil will configure Singularity to cache per-workflow and per-node, but in Slurm a shared filesystem is almost always available. Assuming your home directory is shared, to set this up, you can:

                $ echo 'export SINGULARITY_CACHEDIR="${HOME}/.singularity/cache"' >>~/.bashrc
                $ echo 'export MINIWDL__SINGULARITY__IMAGE_CACHE="${HOME}/.cache/miniwdl"' >>~/.bashrc

    Then make sure to log out and back in again for the setting to take effect.

    2.  If your home directory is not shared across the cluster nodes, make sure that you have installed Toil in such a way that it is in your PATH on the cluster nodes.

    3.  Slurm sandboxing and resource limitation does not apply to Docker containers, because there is no relationship between the sandbox cgroup that your Toil job runs in and the sandbox cgroup that the Docker daemon creates to run the Docker container your job requested to run. If you want your Toil jobs' containers to actually be inside their Slurm job resource allocations, you should make sure to run containers with Singularity or another user-mode or daemon-less containerization system.

    4.  Slurm can sometimes report that a job has finished before that job's changes to the cluster's shared filesystem are visible to other nodes or to the head node. Toil tries to anticipate and compensate for this situation, but there is no amount of waiting or retrying that Toil could do to guarantee correct behavior in theory in these situations; the shared filesystem could in theory be days or months behind. In practice, the delay is usually no more than a few seconds, and Toil can handle it. But if you are seeing odd behavior from Toil related to files not existing when they should or still existing when they shouldn't, your problem could be that your cluster's filesystem is unusually slow to reach consistency across nodes.

    5.  If you see warnings about XDG_RUNTIME_DIR, your Slurm cluster might not be managing XDG login sessions correctly for Slurm jobs. Toil can work around this, but as a result of the workaround it might have trouble finding an appropriate "coordination directory" where it can store state files local to each Slurm node. If you are seeing unusual behavior like Toil jobs on one node waiting for operations on a different node, you can try giving Toil a path to a per-node, writable directory with the --coordinationDir option, to tell it where to put those files explicitly.

    6.  With a shared filesystem, Toil's caching system is not necessarily going to help your workflow. Try running and timing test workflows with --caching true and with --caching false, to determine whether it is worth it for your workload to copy files from the shared filesystem to local storage on each node.

    7.  If running CWL workflows on Slurm, with a shared filesystem, you can try the --bypass-file-store option to toil-cwl-runner. It may speed up your workflow, but you may also need to make sure to change Toil's work directory to a shared directory provided with the --workDir option in order for it to work properly across machines.

  Standard Output/Error from Batch System Jobs

Standard output and error from batch system jobs (except for the Mesos batch system) are redirected to files in the toil-<workflowID> directory created within the temporary directory specified by the --workDir option; see Commandline Options <#optionsref>. Each file is named as follows: toil_job_<Toil job ID>_batch_<name of batch system>_<job ID from batch system>_<file description>.log, where <file description> is std_output for standard output, and std_error for standard error. HTCondor will also write job event log files with <file description> = job_events.

If capturing standard output and error is desired, --workDir will generally need to be on a shared file system; otherwise if these are written to local temporary directories on each node (e.g. /tmp) Toil will not be able to retrieve them. Alternatively, the --noStdOutErr option forces Toil to discard all standard output and error from batch system jobs.

WORKFLOW EXECUTION SERVICE (WES)

The GA4GH Workflow Execution Service (WES) is a standardized API for submitting and monitoring workflows. Toil has experimental support for setting up a WES server and executing CWL, WDL, and Toil workflows using the WES API. More information about the WES API specification can be found here <https://ga4gh.github.io/workflow-execution-service-schemas/docs/>.

To get started with the Toil WES server, make sure that the server extra (Installing Toil with Extra Features <#extras>) is installed.

  Preparing your WES environment

The WES server requires Celery <https://docs.celeryproject.org/en/stable/getting-started/introduction.html> to distribute and execute workflows. To set up Celery:

    1.  Start RabbitMQ, which is the broker between the WES server and Celery workers:

                docker run -d --name wes-rabbitmq -p 5672:5672 rabbitmq:3.9.5

    2.  Start Celery workers:

                celery -A toil.server.celery_app worker --loglevel=INFO

  Starting a WES server

To start a WES server on the default port 8080, run the Toil command:

            $ toil server

The WES API will be hosted on the following URL:

            http://localhost:8080/ga4gh/wes/v1

To use another port, e.g.: 3000, you can specify the --port argument:

            $ toil server --port 3000

There are many other command line options. Help information can be found by using this command:

            $ toil server --help

Below is a detailed summary of all server-specific options:

    --debug
        Enable debug mode.

    --bypass_celery
        Skip sending workflows to Celery and just run them under the server. For testing.

    --host HOST
        The host interface that the Toil server binds on. (default: "127.0.0.1").

    --port PORT
        The port that the Toil server listens on. (default: 8080).

    --swagger_ui
        If True, the swagger UI will be enabled and hosted on the {api_base_path}/ui endpoint. (default: False)

    --cors
        Enable Cross Origin Resource Sharing (CORS). This should only be turned on if the server is intended to be used by a website or domain. (default: False).

    --cors_origins CORS_ORIGIN
        Ignored if --cors is False. This sets the allowed origins for CORS. For details about CORS and its security risks, see the GA4GH docs on CORS <https://w3id.org/ga4gh/product-approval-support/cors>. (default: "*").

    --workers WORKERS, -w WORKERS
        Ignored if --debug is True. The number of worker processes launched by the WSGI server. (default: 2).

    --work_dir WORK_DIR
        The directory where workflows should be stored. This directory should be empty or only contain previous workflows. (default: './workflows').

    --state_store STATE_STORE
        The local path or S3 URL where workflow state metadata should be stored. (default: in --work_dir)

    --opt OPT, -o OPT
        Specify the default parameters to be sent to the workflow engine for each run. Options taking arguments must use = syntax. Accepts multiple values. Example: --opt=--logLevel=CRITICAL --opt=--workDir=/tmp.

    --dest_bucket_base DEST_BUCKET_BASE
        Direct CWL workflows to save output files to dynamically generated unique paths under the given URL. Supports AWS S3.

    --wes_dialect DIALECT
        Restrict WES responses to a dialect compatible with clients that do not fully implement the WES standard. (default: 'standard')

  Running the Server with docker-compose

Instead of manually setting up the server components (toil server, RabbitMQ, and Celery), you can use the following docker-compose.yml file to orchestrate and link them together.

Make sure to change the credentials for basic authentication by updating the traefik.http.middlewares.auth.basicauth.users label. The passwords can be generated with tools like htpasswd like this <https://doc.traefik.io/traefik/v2.0/middlewares/basicauth/#configuration-examples>. (Note that single $ signs need to be replaced with $$ in the yaml file).

When running on a different host other than localhost, make sure to change the Host to your tartget host in the traefik.http.routers.wes.rule and traefik.http.routers.wespublic.rule labels.

You can also change /tmp/toil-workflows if you want Toil workflows to live somewhere else, and create the directory before starting the server.

In order to run workflows that require Docker, the docker.sock socket must be mounted as volume for Celery. Additionally, the TOIL_WORKDIR directory (defaults to: /var/lib/toil) and /var/lib/cwl (if running CWL workflows with DockerRequirement) should exist on the host and also be mounted as volumes.

Also make sure to run it behind a firewall; it opens up the Toil server on port 8080 to anyone who connects.

            # docker-compose.yml
            version: "3.8"

            services:
              rabbitmq:
                image: rabbitmq:3.9.5
                hostname: rabbitmq
              celery:
                image: ${TOIL_APPLIANCE_SELF}
                volumes:
                  - /var/run/docker.sock:/var/run/docker.sock
                  - /var/lib/docker:/var/lib/docker
                  - /var/lib/toil:/var/lib/toil
                  - /var/lib/cwl:/var/lib/cwl
                  - /tmp/toil-workflows:/tmp/toil-workflows
                command: celery --broker=amqp://guest:guest@rabbitmq:5672// -A toil.server.celery_app worker --loglevel=INFO
                depends_on:
                  - rabbitmq
              wes-server:
                image: ${TOIL_APPLIANCE_SELF}
                volumes:
                  - /tmp/toil-workflows:/tmp/toil-workflows
                environment:
                  - TOIL_WES_BROKER_URL=amqp://guest:guest@rabbitmq:5672//
                command: toil server --host 0.0.0.0 --port 8000 --work_dir /tmp/toil-workflows
                expose:
                  - 8000
                labels:
                  - "traefik.enable=true"
                  - "traefik.http.routers.wes.rule=Host(`localhost`)"
                  - "traefik.http.routers.wes.entrypoints=web"
                  - "traefik.http.routers.wes.middlewares=auth"
                  - "traefik.http.middlewares.auth.basicauth.users=test:$$2y$$12$$ci.4U63YX83CwkyUrjqxAucnmi2xXOIlEF6T/KdP9824f1Rf1iyNG"
                  - "traefik.http.routers.wespublic.rule=Host(`localhost`) && Path(`/ga4gh/wes/v1/service-info`)"
                depends_on:
                  - rabbitmq
                  - celery
              traefik:
                image: traefik:v2.2
                command:
                  - "--providers.docker"
                  - "--providers.docker.exposedbydefault=false"
                  - "--entrypoints.web.address=:8080"
                ports:
                  - "8080:8080"
                volumes:
                  - /var/run/docker.sock:/var/run/docker.sock

Further customization can also be made as needed. For example, if you have a domain, you can set up HTTPS with Let's Encrypt <https://doc.traefik.io/traefik/user-guides/docker-compose/acme-http/>.

Once everything is configured, simply run docker-compose up to start the containers. Run docker-compose down to stop and remove all containers.

Note:
        docker-compose is not installed on the Toil appliance by default. See the following section to set up the WES server on a Toil cluster.

  Running on a Toil cluster

To run the server on a Toil leader instance on EC2:

    1.  Launch a Toil cluster with the toil launch-cluster command with the AWS provisioner

    2.  SSH into your cluster with the --sshOption=-L8080:localhost:8080 option to forward port 8080

    3.  Install Docker Compose by running the following commands from the Docker docs <https://docs.docker.com/compose/install/#install-compose>:

                curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
                chmod +x /usr/local/bin/docker-compose

                # check installation
                docker-compose --version

    or, install a different version of Docker Compose by changing "1.29.2" to another version.

    4.  Copy the docker-compose.yml file from (Running the Server with docker-compose) to an empty directory, and modify the configuration as needed.

    5.  Now, run docker-compose up -d to start the WES server in detach mode on the Toil appliance.

    6.  To stop the server, run docker-compose down.

  WES API Endpoints

As defined by the GA4GH WES API specification, the following endpoints with base path ga4gh/wes/v1/ are supported by Toil:
GET /service-info | Get information about the Workflow Execution Service.
GET /runs | List the workflow runs.
POST /runs | Run a workflow. This endpoint creates a new workflow run and returns a run_id to monitor its progress.
GET /runs/{run_id} | Get detailed info about a workflow run.
POST /runs/{run_id}/cancel | Cancel a running workflow.
GET /runs/{run_id}/status | Get the status (overall state) of a workflow run.

When running the WES server with the docker-compose setup above, most endpoints (except GET /service-info) will be protected with basic authentication. Make sure to set the Authorization header with the correct credentials when submitting or retrieving a workflow.

  Submitting a Workflow

Now that the WES API is up and running, we can submit and monitor workflows remotely using the WES API endpoints. A workflow can be submitted for execution using the POST /runs endpoint.

As a quick example, we can submit the example CWL workflow from Running a basic CWL workflow <#cwlquickstart> to our WES API:

            # example.cwl
            cwlVersion: v1.0
            class: CommandLineTool
            baseCommand: echo
            stdout: output.txt
            inputs:
             message:
               type: string
               inputBinding:
                 position: 1
            outputs:
             output:
               type: stdout

using cURL:

            $ curl --location --request POST 'http://localhost:8080/ga4gh/wes/v1/runs' \
                --user test:test \
                --form 'workflow_url="example.cwl"' \
                --form 'workflow_type="cwl"' \
                --form 'workflow_type_version="v1.0"' \
                --form 'workflow_params="{\"message\": \"Hello world!\"}"' \
                --form 'workflow_attachment=@"./toil_test_files/example.cwl"'
            {
              "run_id": "4deb8beb24894e9eb7c74b0f010305d1"
            }

Note that the --user argument is used to attach the basic authentication credentials along with the request. Make sure to change test:test to the username and password you configured for your WES server. Alternatively, you can also set the Authorization header manually as "Authorization: Basic base64_encoded_auth".

If the workflow is submitted successfully, a JSON object containing a run_id will be returned. The run_id is a unique identifier of your requested workflow, which can be used to monitor or cancel the run.

There are a few required parameters that have to be set for all workflow submissions, which are the following:
workflow_url | The URL of the workflow to run. This can refer to a file from workflow_attachment.
workflow_type | The type of workflow language. Toil currently supports one of the following: "CWL", "WDL", or "py". To run a Toil Python workflow, set this to "py".
workflow_type_version | The version of the workflow language. Supported versions can be found by accessing the GET /service-info endpoint of your WES server.
workflow_params | A JSON object that specifies the inputs of the workflow.

Additionally, the following optional parameters are also available:
workflow_attachment | A list of files associated with the workflow run.
workflow_engine_parameters | A JSON key-value map of workflow engine parameters to send to the runner. Example: {"--logLevel": "INFO", "--workDir": "/tmp/"}
tags | A JSON key-value map of metadata associated with the workflow.

For more details about these parameters, refer to the Run Workflow section <https://ga4gh.github.io/workflow-execution-service-schemas/docs/#operation/RunWorkflow> in the WES API spec.

  Upload multiple files

Looking at the body of the request of the previous example, note that the workflow_url is a relative URL that refers to the example.cwl file uploaded from the local path ./toil_test_files/example.cwl.

To specify the file name (or subdirectory) of the remote destination file, set the filename field in the Content-Disposition header. You could also upload more than one file by providing the workflow_attachment parameter multiple times with different files.

This can be shown by the following example:

            $ curl --location --request POST 'http://localhost:8080/ga4gh/wes/v1/runs' \
                --user test:test \
                --form 'workflow_url="example.cwl"' \
                --form 'workflow_type="cwl"' \
                --form 'workflow_type_version="v1.0"' \
                --form 'workflow_params="{\"message\": \"Hello world!\"}"' \
                --form 'workflow_attachment=@"./toil_test_files/example.cwl"' \
                --form 'workflow_attachment=@"./toil_test_files/2.fasta";filename=inputs/test.fasta' \
                --form 'workflow_attachment=@"./toil_test_files/2.fastq";filename=inputs/test.fastq'

On the server, the execution directory would have the following structure from the above request:

            execution/
            ├── example.cwl
            ├── inputs
            │     ├── test.fasta
            |     └── test.fastq
            └── wes_inputs.json

  Specify Toil options

To pass Toil-specific parameters to the workflow, you can include the workflow_engine_parameters parameter along with your request.

For example, to set the logging level to INFO, and change the working directory of the workflow, simply include the following as workflow_engine_parameters:

            {"--logLevel": "INFO", "--workDir": "/tmp/"}

These options would be appended at the end of existing parameters during command construction, which would override the default parameters if provided. (Default parameters that can be passed multiple times would not be overridden).

  Monitoring a Workflow

With the run_id returned when submitting the workflow, we can check the status or get the full logs of the workflow run.

  Checking the state

The GET /runs/{run_id}/status endpoint can be used to get a simple result with the overall state of your run:

            $ curl --user test:test http://localhost:8080/ga4gh/wes/v1/runs/4deb8beb24894e9eb7c74b0f010305d1/status
            {
              "run_id": "4deb8beb24894e9eb7c74b0f010305d1",
              "state": "RUNNING"
            }

The possible states here are: QUEUED, INITIALIZING, RUNNING, COMPLETE, EXECUTOR_ERROR, SYSTEM_ERROR, CANCELING, and CANCELED.

  Getting the full logs

To get the detailed information about a workflow run, use the GET /runs/{run_id} endpoint:

            $ curl --user test:test http://localhost:8080/ga4gh/wes/v1/runs/4deb8beb24894e9eb7c74b0f010305d1
            {
              "run_id": "4deb8beb24894e9eb7c74b0f010305d1",
              "request": {
                "workflow_attachment": [
                  "example.cwl"
                ],
                "workflow_url": "example.cwl",
                "workflow_type": "cwl",
                "workflow_type_version": "v1.0",
                "workflow_params": {
                  "message": "Hello world!"
                }
              },
              "state": "RUNNING",
              "run_log": {
                "cmd": [
                  "toil-cwl-runner --outdir=/home/toil/workflows/4deb8beb24894e9eb7c74b0f010305d1/outputs --jobStore=file:/home/toil/workflows/4deb8beb24894e9eb7c74b0f010305d1/toil_job_store /home/toil/workflows/4deb8beb24894e9eb7c74b0f010305d1/execution/example.cwl /home/workflows/4deb8beb24894e9eb7c74b0f010305d1/execution/wes_inputs.json"
                ],
                "start_time": "2021-08-30T17:35:50Z",
                "end_time": null,
                "stdout": null,
                "stderr": null,
                "exit_code": null
              },
              "task_logs": [],
              "outputs": {}
            }

  Canceling a run

To cancel a workflow run, use the POST /runs/{run_id}/cancel endpoint:

            $ curl --location --request POST 'http://localhost:8080/ga4gh/wes/v1/runs/4deb8beb24894e9eb7c74b0f010305d1/cancel' \
                  --user test:test
            {
              "run_id": "4deb8beb24894e9eb7c74b0f010305d1"
            }

DEVELOPING A PYTHON WORKFLOW

This tutorial walks through the features of Toil necessary for developing a workflow using the Toil Python API.

  Scripting Quick Start

To begin, consider this short Toil Python workflow which illustrates defining a workflow:

            import os

            from toil.common import Toil
            from toil.job import Job
            from toil.lib.io import mkdtemp

            def helloWorld(message):
                return f"Hello, world!, here's a message: {message}"

            if __name__ == "__main__":
                jobstore: str = mkdtemp("tutorial_quickstart")
                os.rmdir(jobstore)
                options = Job.Runner.getDefaultOptions(jobstore)
                options.logLevel = "OFF"
                options.clean = "always"

                hello_job = Job.wrapFn(helloWorld, "Woot")

                with Toil(options) as toil:
                    print(toil.start(hello_job))  # prints "Hello, world!, ..."

The workflow consists of a single job. The resource requirements for that job are (optionally) specified by keyword arguments (memory, cores, disk). The workflow is run using toil.job.Job.Runner.getDefaultOptions() <#toil.job.Job.Runner.getDefaultOptions>. Below we explain the components of this code in detail.

  Job Basics

The atomic unit of work in a Toil workflow is a Job <#toil.job.Job>. User code extends this base class, or uses helper methods like toil.job.Job.addChildJobFn() <#toil.job.Job.addChildJobFn>, to define units of work. For example, here is a more long-winded class-based version of the job in the quick start example:

            from toil.job import Job

            class HelloWorld(Job):
                def __init__(self, message):
                    Job.__init__(self,  memory="2G", cores=2, disk="3G")
                    self.message = message

                def run(self, fileStore):
                    return f"Hello, world! Here's a message: {self.message}"

In the example a class, HelloWorld, is defined. The constructor requests 2 gigabytes of memory, 2 cores and 3 gigabytes of local disk to complete the work.

The toil.job.Job.run() <#toil.job.Job.run> method is the function the user overrides to get work done. Here it just returns a message.

It is also possible to log a message using toil.job.Job.log() <#toil.job.Job.log>, which will be registered in the log output of the leader process of the workflow:

            ...
                def run(self, fileStore):
                    self.log(f"Hello, world! Here's a message: {self.message}")

  Invoking a Workflow

We can add to the previous example to turn it into a complete workflow by adding the necessary function calls to create an instance of HelloWorld and to run this as a workflow containing a single job. For example:

            import os

            from toil.common import Toil
            from toil.job import Job
            from toil.lib.io import mkdtemp

            class HelloWorld(Job):
                def __init__(self, message):
                    Job.__init__(self)
                    self.message = message

                def run(self, fileStore):
                    return f"Hello, world!, here's a message: {self.message}"

            if __name__ == "__main__":
                jobstore: str = mkdtemp("tutorial_invokeworkflow")
                os.rmdir(jobstore)
                options = Job.Runner.getDefaultOptions(jobstore)
                options.logLevel = "OFF"
                options.clean = "always"

                hello_job = HelloWorld("Woot")

                with Toil(options) as toil:
                    print(toil.start(hello_job))

Note:
        Do not include a . in the name of your python script (besides .py at the end). This is to allow toil to import the types and functions defined in your file while starting a new process.

This uses the toil.common.Toil <#toil.common.Toil> class, which is used to run and resume Toil workflows. It is used as a context manager and allows for preliminary setup, such as staging of files into the job store on the leader node. An instance of the class is initialized by specifying an options object. The actual workflow is then invoked by calling the toil.common.Toil.start() <#toil.common.Toil.start> method, passing the root job of the workflow, or, if a workflow is being restarted, toil.common.Toil.restart() <#toil.common.Toil.restart> should be used. Note that the context manager should have explicit if else branches addressing restart and non restart cases. The boolean value for these if else blocks is toil.options.restart.

For example:

            import os

            from toil.common import Toil
            from toil.job import Job
            from toil.lib.io import mkdtemp

            class HelloWorld(Job):
                def __init__(self, message):
                    Job.__init__(self)
                    self.message = message

                def run(self, fileStore):
                    return f"Hello, world!, I have a message: {self.message}"

            if __name__ == "__main__":
                jobstore: str = mkdtemp("tutorial_invokeworkflow2")
                os.rmdir(jobstore)
                options = Job.Runner.getDefaultOptions(jobstore)
                options.logLevel = "INFO"
                options.clean = "always"

                with Toil(options) as toil:
                    if not toil.options.restart:
                        job = HelloWorld("Woot!")
                        output = toil.start(job)
                    else:
                        output = toil.restart()
                print(output)

The call to toil.job.Job.Runner.getDefaultOptions() <#toil.job.Job.Runner.getDefaultOptions> creates a set of default options for the workflow. The only argument is a description of how to store the workflow's state in what we call a job-store. Here the job-store is contained in a directory within the current working directory called "toilWorkflowRun". Alternatively this string can encode other ways to store the necessary state, e.g. an S3 bucket object store location. By default the job-store is deleted if the workflow completes successfully.

The workflow is executed in the final line, which creates an instance of HelloWorld and runs it as a workflow. Note all Toil workflows start from a single starting job, referred to as the root job. The return value of the root job is returned as the result of the completed workflow (see promises below to see how this is a useful feature!).

  Specifying Commandline Arguments

To allow command line control of the options we can use the toil.job.Job.Runner.getDefaultArgumentParser() <#toil.job.Job.Runner.getDefaultArgumentParser> method to create a configargparse.ArgParser object which can be used to parse command line options for a Toil Python workflow. For example:

            from toil.common import Toil
            from toil.job import Job

            class HelloWorld(Job):
                def __init__(self, message):
                    Job.__init__(self)
                    self.message = message

                def run(self, fileStore):
                    return "Hello, world!, here's a message: %s" % self.message

            if __name__ == "__main__":
                parser = Job.Runner.getDefaultArgumentParser()
                options = parser.parse_args()
                options.logLevel = "OFF"
                options.clean = "always"

                hello_job = HelloWorld("Woot")

                with Toil(options) as toil:
                    print(toil.start(hello_job))

This creates a fully fledged Toil Python workflow with all the options Toil exposes as command line arguments. Running this program with --help will print the full list of options.

Alternatively, any existing argparse.ArgumentParser </usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser> object can have Toil command line options added to it with the toil.job.Job.Runner.addToilOptions() <#toil.job.Job.Runner.addToilOptions> method. However, if it is not also a configargparse.ArgParser, it will not read the Toil config file or the environment variables that Toil specifies as alternatives to command-line options, which may lead to unexpected behavior.

  Resuming a Workflow

In the event that a workflow fails, either because of programmatic error within the jobs being run, or because of node failure, the workflow can be resumed. Workflows can only not be reliably resumed if the job-store itself becomes corrupt.

Critical to resumption is that jobs can be rerun, even if they have apparently completed successfully. Put succinctly, a user defined job should not corrupt its input arguments. That way, regardless of node, network or leader failure the job can be restarted and the workflow resumed.

To resume a workflow specify the "restart" option in the options object passed to toil.common.Toil.start() <#toil.common.Toil.start>. If node failures are expected it can also be useful to use the integer "retryCount" option, which will attempt to rerun a job retryCount number of times before marking it fully failed.

In the common scenario that a small subset of jobs fail (including retry attempts) within a workflow Toil will continue to run other jobs until it can do no more, at which point toil.common.Toil.start() <#toil.common.Toil.start> will raise a toil.exceptions.FailedJobsException <#toil.exceptions.FailedJobsException> exception. Typically at this point the user can decide to fix the script and resume the workflow or delete the job-store manually and rerun the complete workflow.

  Functions and Job Functions

Defining jobs by creating class definitions generally involves the boilerplate of creating a constructor. To avoid this the classes toil.job.FunctionWrappingJob <#toil.job.FunctionWrappingJob> and toil.job.JobFunctionWrappingTarget allow functions to be directly converted to jobs. For example, the quick start example (repeated here):

            import os

            from toil.common import Toil
            from toil.job import Job
            from toil.lib.io import mkdtemp

            def helloWorld(message):
                return f"Hello, world!, here's a message: {message}"

            if __name__ == "__main__":
                jobstore: str = mkdtemp("tutorial_quickstart")
                os.rmdir(jobstore)
                options = Job.Runner.getDefaultOptions(jobstore)
                options.logLevel = "OFF"
                options.clean = "always"

                hello_job = Job.wrapFn(helloWorld, "Woot")

                with Toil(options) as toil:
                    print(toil.start(hello_job))  # prints "Hello, world!, ..."

Is equivalent to the previous example, but using a function to define the job.

The function call:

            Job.wrapFn(helloWorld, "Woot")

Creates the instance of the toil.job.FunctionWrappingTarget that wraps the function.

The keyword arguments memory, cores and disk allow resource requirements to be specified as before. Even if they are not included as keyword arguments within a function header they can be passed as arguments when wrapping a function as a job and will be used to specify resource requirements.

We can also use the function wrapping syntax to a job function, a function whose first argument is a reference to the wrapping job. Just like a self argument in a class, this allows access to the methods of the wrapping job, see toil.job.JobFunctionWrappingTarget. For example:

            import os

            from toil.common import Toil
            from toil.job import Job
            from toil.lib.io import mkdtemp

            def helloWorld(job, message):
                job.log(f"Hello world, I have a message: {message}")

            if __name__ == "__main__":
                jobstore: str = mkdtemp("tutorial_jobfunctions")
                os.rmdir(jobstore)
                options = Job.Runner.getDefaultOptions(jobstore)
                options.logLevel = "INFO"
                options.clean = "always"

                hello_job = Job.wrapJobFn(helloWorld, "Woot!")

                with Toil(options) as toil:
                    toil.start(hello_job)

Here helloWorld() is a job function. It uses the toil.job.Job.log() <#toil.job.Job.log> to log a message that will be printed to the output console. Here the only subtle difference to note is the line:

            hello_job = Job.wrapJobFn(helloWorld, "Woot")

Which uses the function toil.job.Job.wrapJobFn() <#toil.job.Job.wrapJobFn> to wrap the job function instead of toil.job.Job.wrapFn() <#toil.job.Job.wrapFn> which wraps a vanilla function.

  Workflows with Multiple Jobs

A parent job can have child jobs and follow-on jobs. These relationships are specified by methods of the job class, e.g. toil.job.Job.addChild() <#toil.job.Job.addChild> and toil.job.Job.addFollowOn() <#toil.job.Job.addFollowOn>.

Considering a set of jobs the nodes in a job graph and the child and follow-on relationships the directed edges of the graph, we say that a job B that is on a directed path of child/follow-on edges from a job A in the job graph is a successor of A, similarly A is a predecessor of B.

A parent job's child jobs are run directly after the parent job has completed, and in parallel. The follow-on jobs of a job are run after its child jobs and their successors have completed. They are also run in parallel. Follow-ons allow the easy specification of cleanup tasks that happen after a set of parallel child tasks. The following shows a simple example that uses the earlier helloWorld() job function:

            from toil.common import Toil
            from toil.job import Job

            def helloWorld(job, message):
                job.log(f"Hello world, I have a message: {message}")

            if __name__ == "__main__":
                parser = Job.Runner.getDefaultArgumentParser()
                options = parser.parse_args()
                options.logLevel = "INFO"
                options.clean = "always"

                j1 = Job.wrapJobFn(helloWorld, "first")
                j2 = Job.wrapJobFn(helloWorld, "second or third")
                j3 = Job.wrapJobFn(helloWorld, "second or third")
                j4 = Job.wrapJobFn(helloWorld, "last")

                j1.addChild(j2)
                j1.addChild(j3)
                j1.addFollowOn(j4)

                with Toil(options) as toil:
                    toil.start(j1)

In the example four jobs are created, first j1 is run, then j2 and j3 are run in parallel as children of j1, finally j4 is run as a follow-on of j1.

There are multiple short hand functions to achieve the same workflow, for example:

            from toil.common import Toil
            from toil.job import Job

            def helloWorld(job, message):
                job.log(f"Hello world, I have a message: {message}")

            if __name__ == "__main__":
                parser = Job.Runner.getDefaultArgumentParser()
                options = parser.parse_args()
                options.logLevel = "INFO"
                options.clean = "always"

                j1 = Job.wrapJobFn(helloWorld, "first")
                j2 = j1.addChildJobFn(helloWorld, "second or third")
                j3 = j1.addChildJobFn(helloWorld, "second or third")
                j4 = j1.addFollowOnJobFn(helloWorld, "last")

                with Toil(options) as toil:
                    toil.start(j1)

Equivalently defines the workflow, where the functions toil.job.Job.addChildJobFn() <#toil.job.Job.addChildJobFn> and toil.job.Job.addFollowOnJobFn() <#toil.job.Job.addFollowOnJobFn> are used to create job functions as children or follow-ons of an earlier job.

Jobs graphs are not limited to trees, and can express arbitrary directed acyclic graphs. For a precise definition of legal graphs see toil.job.Job.checkJobGraphForDeadlocks() <#toil.job.Job.checkJobGraphForDeadlocks>. The previous example could be specified as a DAG as follows:

            from toil.common import Toil
            from toil.job import Job

            def helloWorld(job, message):
                job.log(f"Hello world, I have a message: {message}")

            if __name__ == "__main__":
                parser = Job.Runner.getDefaultArgumentParser()
                options = parser.parse_args()
                options.logLevel = "INFO"
                options.clean = "always"

                j1 = Job.wrapJobFn(helloWorld, "first")
                j2 = j1.addChildJobFn(helloWorld, "second or third")
                j3 = j1.addChildJobFn(helloWorld, "second or third")
                j4 = j2.addChildJobFn(helloWorld, "last")
                j3.addChild(j4)

                with Toil(options) as toil:
                    toil.start(j1)

Note the use of an extra child edge to make j4 a child of both j2 and j3.

  Dynamic Job Creation

The previous examples show a workflow being defined outside of a job. However, Toil also allows jobs to be created dynamically within jobs. For example:

            import os

            from toil.common import Toil
            from toil.job import Job
            from toil.lib.io import mkdtemp

            def binaryStringFn(job, depth, message=""):
                if depth > 0:
                    job.addChildJobFn(binaryStringFn, depth - 1, message + "0")
                    job.addChildJobFn(binaryStringFn, depth - 1, message + "1")
                else:
                    job.log(f"Binary string: {message}")

            if __name__ == "__main__":
                jobstore: str = mkdtemp("tutorial_dynamic")
                os.rmdir(jobstore)
                options = Job.Runner.getDefaultOptions(jobstore)
                options.logLevel = "INFO"
                options.clean = "always"

                with Toil(options) as toil:
                    toil.start(Job.wrapJobFn(binaryStringFn, depth=5))

The job function binaryStringFn logs all possible binary strings of length n (here n=5), creating a total of 2^(n+2) - 1 jobs dynamically and recursively. Static and dynamic creation of jobs can be mixed in a Toil workflow, with jobs defined within a job or job function being created at run time.

  Promises

The previous example of dynamic job creation shows variables from a parent job being passed to a child job. Such forward variable passing is naturally specified by recursive invocation of successor jobs within parent jobs. This can also be achieved statically by passing around references to the return variables of jobs. In Toil this is achieved with promises, as illustrated in the following example:

            import os

            from toil.common import Toil
            from toil.job import Job
            from toil.lib.io import mkdtemp

            def fn(job, i):
                job.log("i is: %s" % i, level=100)
                return i + 1

            if __name__ == "__main__":
                jobstore: str = mkdtemp("tutorial_promises")
                os.rmdir(jobstore)
                options = Job.Runner.getDefaultOptions(jobstore)
                options.logLevel = "INFO"
                options.clean = "always"

                j1 = Job.wrapJobFn(fn, 1)
                j2 = j1.addChildJobFn(fn, j1.rv())
                j3 = j1.addFollowOnJobFn(fn, j2.rv())

                with Toil(options) as toil:
                    toil.start(j1)

Running this workflow results in three log messages from the jobs: i is 1 from j1, i is 2 from j2 and i is 3 from j3.

The return value from the first job is promised to the second job by the call to toil.job.Job.rv() <#toil.job.Job.rv> in the following line:

            j2 = j1.addChildFn(fn, j1.rv())

The value of j1.rv() is a promise, rather than the actual return value of the function, because j1 for the given input has at that point not been evaluated. A promise (toil.job.Promise <#toil.job.Promise>) is essentially a pointer to for the return value that is replaced by the actual return value once it has been evaluated. Therefore, when j2 is run the promise becomes 2.

Promises also support indexing of return values:

            def parent(job):
                indexable = Job.wrapJobFn(fn)
                job.addChild(indexable)
                job.addFollowOnFn(raiseWrap, indexable.rv(2))

            def raiseWrap(arg):
                raise RuntimeError(arg) # raises "2"

            def fn(job):
                return (0, 1, 2, 3)

Promises can be quite useful. For example, we can combine dynamic job creation with promises to achieve a job creation process that mimics the functional patterns possible in many programming languages:

            import os

            from toil.common import Toil
            from toil.job import Job
            from toil.lib.io import mkdtemp

            def binaryStrings(job, depth, message=""):
                if depth > 0:
                    s = [
                        job.addChildJobFn(binaryStrings, depth - 1, message + "0").rv(),
                        job.addChildJobFn(binaryStrings, depth - 1, message + "1").rv(),
                    ]
                    return job.addFollowOnFn(merge, s).rv()
                return [message]

            def merge(strings):
                return strings[0] + strings[1]

            if __name__ == "__main__":
                jobstore: str = mkdtemp("tutorial_promises2")
                os.rmdir(jobstore)
                options = Job.Runner.getDefaultOptions(jobstore)
                options.loglevel = "OFF"
                options.clean = "always"

                with Toil(options) as toil:
                    print(toil.start(Job.wrapJobFn(binaryStrings, depth=5)))

The return value l of the workflow is a list of all binary strings of length 10, computed recursively. Although a toy example, it demonstrates how closely Toil workflows can mimic typical programming patterns.

  Promised Requirements

Promised requirements are a special case of Promises that allow a job's return value to be used as another job's resource requirements.

This is useful when, for example, a job's storage requirement is determined by a file staged to the job store by an earlier job:

            import os

            from toil.common import Toil
            from toil.job import Job, PromisedRequirement
            from toil.lib.io import mkdtemp

            def parentJob(job):
                downloadJob = Job.wrapJobFn(
                    stageFn,
                    "file://" + os.path.realpath(__file__),
                    cores=0.1,
                    memory="32M",
                    disk="1M",
                )
                job.addChild(downloadJob)

                analysis = Job.wrapJobFn(
                    analysisJob,
                    fileStoreID=downloadJob.rv(0),
                    disk=PromisedRequirement(downloadJob.rv(1)),
                )
                job.addFollowOn(analysis)

            def stageFn(job, url):
                importedFile = job.fileStore.import_file(url)
                return importedFile, importedFile.size

            def analysisJob(job, fileStoreID):
                # now do some analysis on the file
                pass

            if __name__ == "__main__":
                jobstore: str = mkdtemp("tutorial_requirements")
                os.rmdir(jobstore)
                options = Job.Runner.getDefaultOptions(jobstore)
                options.logLevel = "INFO"
                options.clean = "always"

                with Toil(options) as toil:
                    toil.start(Job.wrapJobFn(parentJob))

Note that this also makes use of the size attribute of the FileID object. This promised requirements mechanism can also be used in combination with an aggregator for multiple jobs' output values:

            def parentJob(job):
                aggregator = []
                for fileNum in range(0, 10):
                    downloadJob = Job.wrapJobFn(stageFn, "file://" + os.path.realpath(__file__), cores=0.1, memory='32M', disk='1M')
                    job.addChild(downloadJob)
                    aggregator.append(downloadJob)

                analysis = Job.wrapJobFn(analysisJob,
                                         fileStoreID=downloadJob.rv(0),
                                         disk=PromisedRequirement(lambda xs: sum(xs), [j.rv(1) for j in aggregator]))
                job.addFollowOn(analysis)

        Limitations

            Just like regular promises, the return value must be determined prior to scheduling any job that depends on the return value. In our example above, notice how the dependent jobs were follow ons to the parent while promising jobs are children of the parent. This ordering ensures that all promises are properly fulfilled.

  FileID

The toil.fileStore.FileID class is a small wrapper around Python's builtin string class. It is used to represent a file's ID in the file store, and has a size attribute that is the file's size in bytes. This object is returned by importFile and writeGlobalFile.

  Managing files within a workflow

It is frequently the case that a workflow will want to create files, both persistent and temporary, during its run. The toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore> class is used by jobs to manage these files in a manner that guarantees cleanup and resumption on failure.

The toil.job.Job.run() <#toil.job.Job.run> method has a file store instance as an argument. The following example shows how this can be used to create temporary files that persist for the length of the job, be placed in a specified local disk of the node and that will be cleaned up, regardless of failure, when the job finishes:

            import os

            from toil.common import Toil
            from toil.job import Job
            from toil.lib.io import mkdtemp

            class LocalFileStoreJob(Job):
                def run(self, fileStore):
                    # self.tempDir will always contain the name of a directory within the allocated disk space reserved for the job
                    scratchDir = self.tempDir

                    # Similarly create a temporary file.
                    scratchFile = fileStore.getLocalTempFile()

            if __name__ == "__main__":
                jobstore: str = mkdtemp("tutorial_managing")
                os.rmdir(jobstore)
                options = Job.Runner.getDefaultOptions(jobstore)
                options.logLevel = "INFO"
                options.clean = "always"

                # Create an instance of FooJob which will have at least 2 gigabytes of storage space.
                j = LocalFileStoreJob(disk="2G")

                # Run the workflow
                with Toil(options) as toil:
                    toil.start(j)

Job functions can also access the file store for the job. The equivalent of the LocalFileStoreJob class is

            def localFileStoreJobFn(job):
                scratchDir = job.tempDir
                scratchFile = job.fileStore.getLocalTempFile()

Note that the fileStore attribute is accessed as an attribute of the job argument.

In addition to temporary files that exist for the duration of a job, the file store allows the creation of files in a global store, which persists during the workflow and are globally accessible (hence the name) between jobs. For example:

            import os

            from toil.common import Toil
            from toil.job import Job
            from toil.lib.io import mkdtemp

            def globalFileStoreJobFn(job):
                job.log(
                    "The following example exercises all the methods provided "
                    "by the toil.fileStores.abstractFileStore.AbstractFileStore class"
                )

                # Create a local temporary file.
                scratchFile = job.fileStore.getLocalTempFile()

                # Write something in the scratch file.
                with open(scratchFile, "w") as fH:
                    fH.write("What a tangled web we weave")

                # Write a copy of the file into the file-store; fileID is the key that can be used to retrieve the file.
                # This write is asynchronous by default
                fileID = job.fileStore.writeGlobalFile(scratchFile)

                # Write another file using a stream; fileID2 is the
                # key for this second file.
                with job.fileStore.writeGlobalFileStream(cleanup=True) as (fH, fileID2):
                    fH.write(b"Out brief candle")

                # Now read the first file; scratchFile2 is a local copy of the file that is read-only by default.
                scratchFile2 = job.fileStore.readGlobalFile(fileID)

                # Read the second file to a desired location: scratchFile3.
                scratchFile3 = os.path.join(job.tempDir, "foo.txt")
                job.fileStore.readGlobalFile(fileID2, userPath=scratchFile3)

                # Read the second file again using a stream.
                with job.fileStore.readGlobalFileStream(fileID2) as fH:
                    print(fH.read())  # This prints "Out brief candle"

                # Delete the first file from the global file-store.
                job.fileStore.deleteGlobalFile(fileID)

                # It is unnecessary to delete the file keyed by fileID2 because we used the cleanup flag,
                # which removes the file after this job and all its successors have run (if the file still exists)

            if __name__ == "__main__":
                jobstore: str = mkdtemp("tutorial_managing2")
                os.rmdir(jobstore)
                options = Job.Runner.getDefaultOptions(jobstore)
                options.logLevel = "INFO"
                options.clean = "always"

                with Toil(options) as toil:
                    toil.start(Job.wrapJobFn(globalFileStoreJobFn))

The example demonstrates the global read, write and delete functionality of the file-store, using both local copies of the files and streams to read and write the files. It covers all the methods provided by the file store interface.

What is obvious is that the file-store provides no functionality to update an existing "global" file, meaning that files are, barring deletion, immutable. Also worth noting is that there is no file system hierarchy for files in the global file store. These limitations allow us to fairly easily support different object stores and to use caching to limit the amount of network file transfer between jobs.

  Staging of Files into the Job Store

External files can be imported into or exported out of the job store prior to running a workflow when the toil.common.Toil <#toil.common.Toil> context manager is used on the leader. The context manager provides methods toil.common.Toil.importFile() <#toil.common.Toil.importFile>, and toil.common.Toil.exportFile() <#toil.common.Toil.exportFile> for this purpose. The destination and source locations of such files are described with URLs passed to the two methods. Local files can be imported and exported as relative paths, and should be relative to the directory where the toil workflow is initially run from.

Using absolute paths and appropriate schema where possible (prefixing with "<file://>" or "s3:/" for example), make imports and exports less ambiguous and is recommended.

A list of the currently supported URLs can be found at toil.jobStores.abstractJobStore.AbstractJobStore.importFile() <#toil.jobStores.abstractJobStore.AbstractJobStore.importFile>. To import an external file into the job store as a shared file, pass the optional sharedFileName parameter to that method.

If a workflow fails for any reason an imported file acts as any other file in the job store. If the workflow was configured such that it not be cleaned up on a failed run, the file will persist in the job store and needs not be staged again when the workflow is resumed.

Example:

            import os

            from toil.common import Toil
            from toil.job import Job
            from toil.lib.io import mkdtemp

            class HelloWorld(Job):
                def __init__(self, id):
                    Job.__init__(self)
                    self.inputFileID = id

                def run(self, fileStore):
                    with fileStore.readGlobalFileStream(self.inputFileID, encoding="utf-8") as fi:
                        with fileStore.writeGlobalFileStream(encoding="utf-8") as (
                            fo,
                            outputFileID,
                        ):
                            fo.write(fi.read() + "World!")
                    return outputFileID

            if __name__ == "__main__":
                jobstore: str = mkdtemp("tutorial_staging")
                tmp: str = mkdtemp("tutorial_staging_tmp")
                os.rmdir(jobstore)
                options = Job.Runner.getDefaultOptions(jobstore)
                options.logLevel = "INFO"
                options.clean = "always"

                with Toil(options) as toil:
                    if not toil.options.restart:
                        # Prepare an input file
                        path = os.path.join(tmp, "in.txt")
                        with open(path, "w") as f:
                            f.write("Hello,\n")
                        # In a real workflow, you would obtain an input file path from the
                        # user.

                        # Stage it into the Toil job store.
                        #
                        # Note: this may create a symlink depending on the value of the
                        # --linkImports command line option, in which case the original
                        # input file needs to still exist if the workflow is restarted.
                        inputFileID = toil.importFile(f"file://{path}")

                        # Run the workflow
                        outputFileID = toil.start(HelloWorld(inputFileID))
                    else:
                        outputFileID = toil.restart()

                    toil.exportFile(
                        outputFileID,
                        "file://" + os.path.join(tmp, "out.txt"),
                    )

  Using Docker Containers in Toil

Docker containers are commonly used with Toil. The combination of Toil and Docker allows for pipelines to be fully portable between any platform that has both Toil and Docker installed. Docker eliminates the need for the user to do any other tool installation or environment setup.

In order to use Docker containers with Toil, Docker must be installed on all workers of the cluster. Instructions for installing Docker can be found on the Docker <https://docs.docker.com/engine/getstarted/step_one/> website.

When using Toil-based autoscaling, Docker will be automatically set up on the cluster's worker nodes, so no additional installation steps are necessary. Further information on using Toil-based autoscaling can be found in the Running a Workflow with Autoscaling <#autoscaling> documentation.

In order to use docker containers in a Toil workflow, the container can be built locally or downloaded in real time from an online docker repository like Quay <quay.io>. If the container is not in a repository, the container's layers must be accessible on each node of the cluster.

When invoking docker containers from within a Toil workflow, it is strongly recommended that you use dockerCall(), a toil job function provided in toil.lib.docker. dockerCall leverages docker's own python API, and provides container cleanup on job failure. When docker containers are run without this feature, failed jobs can result in resource leaks. Docker's API can be found at docker-py <https://docker-py.readthedocs.io/en/stable/>.

In order to use dockerCall, your installation of Docker must be set up to run without sudo. Instructions for setting this up can be found here <https://docs.docker.com/engine/installation/linux/ubuntulinux/#/create-a-docker-group>.

An example of a basic dockerCall is below:

            dockerCall(job=job,
                        tool='quay.io/ucsc_cgl/bwa',
                        workDir=job.tempDir,
                        parameters=['index', '/data/reference.fa'])

Note the assumption that reference.fa file is located in /data. This is Toil's standard convention as a mount location to reduce boilerplate when calling dockerCall. Users can choose their own mount locations by supplying a volumes kwarg to dockerCall, such as: volumes={working_dir: {'bind': '/data', 'mode': 'rw'}}, where working_dir is an absolute path on the user's filesystem.

dockerCall can also be added to workflows like any other job function:

            import os

            from toil.common import Toil
            from toil.job import Job
            from toil.lib.docker import apiDockerCall
            from toil.lib.io import mkdtemp

            align = Job.wrapJobFn(
                apiDockerCall, image="ubuntu", working_dir=os.getcwd(), parameters=["ls", "-lha"]
            )

            if __name__ == "__main__":
                jobstore: str = mkdtemp("tutorial_docker")
                os.rmdir(jobstore)
                options = Job.Runner.getDefaultOptions(jobstore)
                options.logLevel = "INFO"
                options.clean = "always"

                with Toil(options) as toil:
                    toil.start(align)

cgl-docker-lib <https://github.com/BD2KGenomics/cgl-docker-lib/blob/master/README.md> contains dockerCall-compatible Dockerized tools that are commonly used in bioinformatics analysis.

The documentation provides guidelines for developing your own Docker containers that can be used with Toil and dockerCall. In order for a container to be compatible with dockerCall, it must have an ENTRYPOINT set to a wrapper script, as described in cgl-docker-lib containerization standards. This can be set by passing in the optional keyword argument, 'entrypoint'. Example:
        entrypoint=["/bin/bash","-c"]

dockerCall supports currently the 75 keyword arguments found in the python Docker API <https://docker-py.readthedocs.io/en/stable/containers.html>, under the 'run' command.

  Services

It is sometimes desirable to run services, such as a database or server, concurrently with a workflow. The toil.job.Job.Service <#toil.job.Job.Service> class provides a simple mechanism for spawning such a service within a Toil workflow, allowing precise specification of the start and end time of the service, and providing start and end methods to use for initialization and cleanup. The following simple, conceptual example illustrates how services work:

            import os

            from toil.common import Toil
            from toil.job import Job
            from toil.lib.io import mkdtemp

            class DemoService(Job.Service):
                def start(self, fileStore):
                    # Start up a database/service here
                    # Return a value that enables another process to connect to the database
                    return "loginCredentials"

                def check(self):
                    # A function that if it returns False causes the service to quit
                    # If it raises an exception the service is killed and an error is reported
                    return True

                def stop(self, fileStore):
                    # Cleanup the database here
                    pass

            j = Job()
            s = DemoService()
            loginCredentialsPromise = j.addService(s)

            def dbFn(loginCredentials):
                # Use the login credentials returned from the service's start method to connect to the service
                pass

            j.addChildFn(dbFn, loginCredentialsPromise)

            if __name__ == "__main__":
                jobstore: str = mkdtemp("tutorial_services")
                os.rmdir(jobstore)
                options = Job.Runner.getDefaultOptions(jobstore)
                options.logLevel = "INFO"
                options.clean = "always"

                with Toil(options) as toil:
                    toil.start(j)

In this example the DemoService starts a database in the start method, returning an object from the start method indicating how a client job would access the database. The service's stop method cleans up the database, while the service's check method is polled periodically to check the service is alive.

A DemoService instance is added as a service of the root job j, with resource requirements specified. The return value from toil.job.Job.addService() <#toil.job.Job.addService> is a promise to the return value of the service's start method. When the promised is fulfilled it will represent how to connect to the database. The promise is passed to a child job of j, which uses it to make a database connection. The services of a job are started before any of its successors have been run and stopped after all the successors of the job have completed successfully.

Multiple services can be created per job, all run in parallel. Additionally, services can define sub-services using toil.job.Job.Service.addChild(). This allows complex networks of services to be created, e.g. Apache Spark clusters, within a workflow.

  Checkpoints

Services complicate resuming a workflow after failure, because they can create complex dependencies between jobs. For example, consider a service that provides a database that multiple jobs update. If the database service fails and loses state, it is not clear that just restarting the service will allow the workflow to be resumed, because jobs that created that state may have already finished. To get around this problem Toil supports checkpoint jobs, specified as the boolean keyword argument checkpoint to a job or wrapped function, e.g.:

            j = Job(checkpoint=True)

A checkpoint job is rerun if one or more of its successors fails its retry attempts, until it itself has exhausted its retry attempts. Upon restarting a checkpoint job all its existing successors are first deleted, and then the job is rerun to define new successors. By checkpointing a job that defines a service, upon failure of the service the database and the jobs that access the service can be redefined and rerun.

To make the implementation of checkpoint jobs simple, a job can only be a checkpoint if when first defined it has no successors, i.e. it can only define successors within its run method.

  Encapsulation

Let A be a root job potentially with children and follow-ons. Without an encapsulated job the simplest way to specify a job B which runs after A and all its successors is to create a parent of A, call it Ap, and then make B a follow-on of Ap. e.g.:

            import os

            from toil.common import Toil
            from toil.job import Job
            from toil.lib.io import mkdtemp

            if __name__ == "__main__":
                # A is a job with children and follow-ons, for example:
                A = Job()
                A.addChild(Job())
                A.addFollowOn(Job())

                # B is a job which needs to run after A and its successors
                B = Job()

                # The way to do this without encapsulation is to make a parent of A, Ap, and make B a follow-on of Ap.
                Ap = Job()
                Ap.addChild(A)
                Ap.addFollowOn(B)

                jobstore: str = mkdtemp("tutorial_encapsulations")
                os.rmdir(jobstore)
                options = Job.Runner.getDefaultOptions(jobstore)
                options.logLevel = "INFO"
                options.clean = "always"

                with Toil(options) as toil:
                    print(toil.start(Ap))

An encapsulated job E(A) of A saves making Ap, instead we can write:

            import os

            from toil.common import Toil
            from toil.job import Job
            from toil.lib.io import mkdtemp

            if __name__ == "__main__":
                # A
                A = Job()
                A.addChild(Job())
                A.addFollowOn(Job())

                # Encapsulate A
                A = A.encapsulate()

                # B is a job which needs to run after A and its successors
                B = Job()

                # With encapsulation A and its successor subgraph appear to be a single job, hence:
                A.addChild(B)

                jobstore: str = mkdtemp("tutorial_encapsulations2")
                os.rmdir(jobstore)
                options = Job.Runner.getDefaultOptions(jobstore)
                options.logLevel = "INFO"
                options.clean = "always"

                with Toil(options) as toil:
                    print(toil.start(A))

Note the call to toil.job.Job.encapsulate() <#toil.job.Job.encapsulate> creates the toil.job.Job.EncapsulatedJob.

  Depending on Toil

If you are packing your workflow(s) as a pip-installable distribution on PyPI, you might be tempted to declare Toil as a dependency in your setup.py, via the install_requires keyword argument to setup(). Unfortunately, this does not work, for two reasons: For one, Toil uses Setuptools' extra mechanism to manage its own optional dependencies. If you explicitly declared a dependency on Toil, you would have to hard-code a particular combination of extras (or no extras at all), robbing the user of the choice what Toil extras to install. Secondly, and more importantly, declaring a dependency on Toil would only lead to Toil being installed on the leader node of a cluster, but not the worker nodes. Auto-deployment does not work here because Toil cannot auto-deploy itself, the classic "Which came first, chicken or egg?" problem.

In other words, you shouldn't explicitly depend on Toil. Document the dependency instead (as in "This workflow needs Toil version X.Y.Z to be installed") and optionally add a version check to your setup.py. Refer to the check_version() function in the toil-lib project's setup.py <https://github.com/BD2KGenomics/toil-lib/blob/master/setup.py> for an example. Alternatively, you can also just depend on toil-lib and you'll get that check for free.

If your workflow depends on a dependency of Toil, consider not making that dependency explicit either. If you do, you risk a version conflict between your project and Toil. The pip utility may silently ignore that conflict, breaking either Toil or your workflow. It is safest to simply assume that Toil installs that dependency for you. The only downside is that you are locked into the exact version of that dependency that Toil declares. But such is life with Python, which, unlike Java, has no means of dependencies belonging to different software components within the same process, and whose favored software distribution utility is incapable <https://github.com/pypa/pip/issues/988> of properly resolving overlapping dependencies and detecting conflicts.

  Best Practices for Dockerizing Toil Workflows

Computational Genomics Lab <https://cgl.genomics.ucsc.edu/>'s Dockstore <https://dockstore.org/docs> based production system provides workflow authors a way to run Dockerized versions of their pipeline in an automated, scalable fashion. To be compatible with this system of a workflow should meet the following requirements. In addition to the Docker container, a common workflow language descriptor file <https://dockstore.org/docs/getting-started-with-cwl> is needed. For inputs:

    •   Only command line arguments should be used for configuring the workflow. If the workflow relies on a configuration file, like Toil-RNAseq <https://github.com/BD2KGenomics/toil-rnaseq> or ProTECT <https://github.com/BD2KGenomics/protect>, a wrapper script inside the Docker container can be used to parse the CLI and generate the necessary configuration file.

    •   All inputs to the pipeline should be explicitly enumerated rather than implicit. For example, don't rely on one FASTQ read's path to discover the location of its pair. This is necessary since all inputs are mapped to their own isolated directories when the Docker is called via Dockstore.

    •   All inputs must be documented in the CWL descriptor file. Examples of this file can be seen in both Toil-RNAseq <https://github.com/BD2KGenomics/toil-rnaseq> and ProTECT <https://github.com/BD2KGenomics/protect>.

For outputs:

    •   All outputs should be written to a local path rather than S3.

    •   Take care to package outputs in a local and user-friendly way. For example, don't tar up all output if there are specific files that will care to see individually.

    •   All output file names should be deterministic and predictable. For example, don't prepend the name of an output file with PASS/FAIL depending on the outcome of the pipeline.

    •   All outputs must be documented in the CWL descriptor file. Examples of this file can be seen in both Toil-RNAseq <https://github.com/BD2KGenomics/toil-rnaseq> and ProTECT <https://github.com/BD2KGenomics/protect>.

TOIL CLASS API

The Toil class configures and starts a Toil run.

    class toil.common.Toil(options, workflow_name=None, trs_spec=None)
        A context manager that represents a Toil workflow.

        Specifically the batch system, job store, and its configuration.

        Parameters

            •   options (Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)

            •   workflow_name (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>)

            •   trs_spec (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>)

        __init__(options, workflow_name=None, trs_spec=None)
            Initialize a Toil object from the given options.

            Note that this is very light-weight and that the bulk of the work is done when the context is entered.

            Parameters

                •   options (Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>) -- command line options specified by the user

                •   workflow_name (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- A human-readable name (probably a filename, URL, or TRS specifier) for the workflow being run. Used for Toil history storage.

                •   trs_spec (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- A TRS id:version string for the workflow being run, if any. Used for Toil history storage and publishing workflow execution metrics to Dockstore.

            Return type
                None

        start(rootJob)
            Invoke a Toil workflow with the given job as the root for an initial run.

            This method must be called in the body of a with Toil(...) as toil: statement. This method should not be called more than once for a workflow that has not finished.

            Parameters
                rootJob (Job <#toil.job.Job>) -- The root job of the workflow

            Return type
                Any </usr/share/doc/python3/html/library/typing.html#typing.Any>

            Returns
                The root job's return value

        restart()
            Restarts a workflow that has been interrupted.

            Return type
                Any </usr/share/doc/python3/html/library/typing.html#typing.Any>

            Returns
                The root job's return value

        classmethod getJobStore(locator)
            Create an instance of the concrete job store implementation that matches the given locator.

            Parameters

                •   locator (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The location of the job store to be represent by the instance

                •   locator

            Return type
                AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>

            Returns
                an instance of a concrete subclass of AbstractJobStore

        static parseLocator(locator)
            Parse a job store locator to a type string and the data needed for that implementation to connect to it.

            Does not validate the set of possible job store types.

            Raises
                RuntimeError </usr/share/doc/python3/html/library/exceptions.html#RuntimeError> -- if the locator is not in the appropriate syntax.

            Parameters
                locator (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        classmethod canonical_locator(locator)
            Turn a job store locator into one that will work from any directory and always includes the explicit type of job store.

            Parameters
                locator (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        static createBatchSystem(config)
            Create an instance of the batch system specified in the given config.

            Parameters
                config (Config <#toil.common.Config>) -- the current configuration

            Return type
                AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>

            Returns
                an instance of a concrete subclass of AbstractBatchSystem

        import_file(src_uri, shared_file_name=None, symlink=True, check_existence=True)
            Import the file at the given URL into the job store.

            By default, returns None if the file does not exist.

            Parameters

                •   check_existence (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If true, raise FileNotFoundError if the file does not exist. If false, return None when the file does not exist.

                •   src_uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   shared_file_name (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None)

                •   symlink (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                FileID <#toil.fileStores.FileID> | None </usr/share/doc/python3/html/library/constants.html#None>

        See toil.jobStores.abstractJobStore.AbstractJobStore.importFile() <#toil.jobStores.abstractJobStore.AbstractJobStore.importFile> for a full description

            Parameters

                •   src_uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   shared_file_name (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>)

                •   symlink (bool </usr/share/doc/python3/html/library/functions.html#bool>)

                •   check_existence (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                FileID <#toil.fileStores.FileID> | None

        export_file(file_id, dst_uri)
            Export file to destination pointed at by the destination URL.

            See toil.jobStores.abstractJobStore.AbstractJobStore.exportFile() <#toil.jobStores.abstractJobStore.AbstractJobStore.exportFile> for a full description

            Parameters

                •   file_id (FileID <#toil.fileStores.FileID>)

                •   dst_uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        static normalize_uri(uri, check_existence=False, dir_path=None)
            Given a URI, if it has no scheme, make it a properly quoted file: URI.

            Parameters

                •   check_existence (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If set, raise FileNotFoundError if a URI points to a local file that does not exist.

                •   dir_path (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- If specified, interpret relative paths relative to the given directory path instead of the current one.

                •   uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        static getToilWorkDir(configWorkDir=None)
            Return a path to a writable directory under which per-workflow directories exist.

            This directory is always required to exist on a machine, even if the Toil worker has not run yet. If your workers and leader have different temp directories, you may need to set TOIL_WORKDIR.

            Parameters
                configWorkDir (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- Value passed to the program using the --workDir flag

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

            Returns
                Path to the Toil work directory, constant across all machines

        classmethod get_toil_coordination_dir(config_work_dir, config_coordination_dir)
            Return a path to a writable directory, which will be in memory if convenient. Ought to be used for file locking and coordination.

            Parameters

                •   config_work_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- Value passed to the program using the --workDir flag

                •   config_coordination_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- Value passed to the program using the --coordinationDir flag

                •   workflow_id -- Used if a tmpdir_prefix exists to create full directory paths unique per workflow

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

            Returns
                Path to the Toil coordination directory. Ought to be on a POSIX filesystem that allows directories containing open files to be deleted.

        static get_workflow_path_component(workflow_id)
            Get a safe filesystem path component for a workflow.

            Will be consistent for all processes on a given machine, and different for all processes on different machines.

            Parameters
                workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The ID of the current Toil workflow.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        classmethod getLocalWorkflowDir(workflowID, configWorkDir=None)
            Return the directory where worker directories and the cache will be located for this workflow on this machine.

            Parameters

                •   configWorkDir (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- Value passed to the program using the --workDir flag

                •   workflowID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

            Returns
                Path to the local workflow directory on this machine

        classmethod get_local_workflow_coordination_dir(workflow_id, config_work_dir, config_coordination_dir)
            Return the directory where coordination files should be located for this workflow on this machine. These include internal Toil databases and lock files for the machine.

            If an in-memory filesystem is available, it is used. Otherwise, the local workflow directory, which may be on a shared network filesystem, is used.

            Parameters

                •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Unique ID of the current workflow.

                •   config_work_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- Value used for the work directory in the current Toil Config.

                •   config_coordination_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- Value used for the coordination directory in the current Toil Config.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

            Returns
                Path to the local workflow coordination directory on this machine.

JOB STORE API

The job store interface is an abstraction layer that that hides the specific details of file storage, for example standard file systems, S3, etc. The AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore> API is implemented to support a give file store, e.g. S3. Implement this API to support a new file store.

    class toil.jobStores.abstractJobStore.AbstractJobStore(locator)
        Represents the physical storage for the jobs and files in a Toil workflow.

        JobStores are responsible for storing toil.job.JobDescription <#toil.job.JobDescription> (which relate jobs to each other) and files.

        Actual toil.job.Job <#toil.job.Job> objects are stored in files, referenced by JobDescriptions. All the non-file CRUD methods the JobStore provides deal in JobDescriptions and not full, executable Jobs.

        To actually get ahold of a toil.job.Job <#toil.job.Job>, use toil.job.Job.loadJob() <#toil.job.Job.loadJob> with a JobStore and the relevant JobDescription.

        Parameters
            locator (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        __init__(locator)
            Create an instance of the job store.

            The instance will not be fully functional until either initialize() <#toil.jobStores.abstractJobStore.AbstractJobStore.initialize> or resume() <#toil.jobStores.abstractJobStore.AbstractJobStore.resume> is invoked. Note that the destroy() <#toil.jobStores.abstractJobStore.AbstractJobStore.destroy> method may be invoked on the object with or without prior invocation of either of these two methods.

            Takes and stores the locator string for the job store, which will be accessible via self.locator.

            Parameters
                locator (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        initialize(config)
            Initialize this job store.

            Create the physical storage for this job store, allocate a workflow ID and persist the given Toil configuration to the store.

            Parameters
                config (Config <#toil.common.Config>) -- the Toil configuration to initialize this job store with. The given configuration will be updated with the newly allocated workflow ID.

            Raises
                JobStoreExistsException <#toil.jobStores.abstractJobStore.JobStoreExistsException> -- if the physical storage for this job store already exists

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        write_config()
            Persists the value of the AbstractJobStore.config <#toil.jobStores.abstractJobStore.AbstractJobStore.config> attribute to the job store, so that it can be retrieved later by other instances of this class.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        resume()
            Connect this instance to the physical storage it represents and load the Toil configuration into the AbstractJobStore.config <#toil.jobStores.abstractJobStore.AbstractJobStore.config> attribute.

            Raises
                NoSuchJobStoreException <#toil.jobStores.abstractJobStore.NoSuchJobStoreException> -- if the physical storage for this job store doesn't exist

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        property config: Config <#toil.common.Config>
            Return the Toil configuration associated with this job store.

            Raises
                AttributeError </usr/share/doc/python3/html/library/exceptions.html#AttributeError> -- if the config has not yet been assigned (i.e. during resume() <#toil.jobStores.abstractJobStore.AbstractJobStore.resume>).

            Return type
                toil.common.Config <#toil.common.Config>

        property locator: str </usr/share/doc/python3/html/library/stdtypes.html#str>
            Get the locator that defines the job store, which can be used to connect to it.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        setRootJob(rootJobStoreID)
            Set the root job of the workflow backed by this job store.

            Parameters
                rootJobStoreID (FileID <#toil.fileStores.FileID>)

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        set_root_job(job_id)
            Set the root job of the workflow backed by this job store.

            Parameters
                job_id (FileID <#toil.fileStores.FileID>) -- The ID of the job to set as root

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        load_root_job()
            Loads the JobDescription for the root job in the current job store.

            Raises
                toil.job.JobException <#toil.job.JobException> -- If no root job is set or if the root job doesn't exist in this job store

            Return type
                JobDescription <#toil.job.JobDescription>

            Returns
                The root job.

        create_root_job(job_description)
            Create the given JobDescription and set it as the root job in this job store.

            Parameters
                job_description (JobDescription <#toil.job.JobDescription>) -- JobDescription to save and make the root job.

            Return type
                JobDescription <#toil.job.JobDescription>

        get_root_job_return_value()
            Parse the return value from the root job.

            Raises an exception if the root job hasn't fulfilled its promise yet.

            Return type
                Any </usr/share/doc/python3/html/library/typing.html#typing.Any>

        import_file(src_uri, shared_file_name=None, hardlink=False, symlink=True)
            Imports the file at the given URL into job store. The ID of the newly imported file is returned. If the name of a shared file name is provided, the file will be imported as such and None is returned. If an executable file on the local filesystem is uploaded, its executability will be preserved when it is downloaded.

            Currently supported schemes are:

                    •

                        's3' for objects in Amazon S3
                            e.g. s3://bucket/key

                    •

                        'file' for local files
                            e.g. <file:///local/file/path>

                    •

                        'http'
                            e.g. <http://someurl.com/path>

                    •

                        'gs'
                            e.g. gs://bucket/file

        Raises FileNotFoundError if the file does not exist.

            Parameters

                •   src_uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- URL that points to a file or object in the storage mechanism of a supported URL scheme e.g. a blob in an AWS s3 bucket. It must be a file, not a directory or prefix.

                •   shared_file_name (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- Optional name to assign to the imported file within the job store

                •   src_uri

                •   shared_file_name

                •   hardlink (bool </usr/share/doc/python3/html/library/functions.html#bool>)

                •   symlink (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Returns
                The jobStoreFileID of the imported file or None if shared_file_name was given

            Return type
                toil.fileStores.FileID <#toil.fileStores.FileID> or None

        export_file(file_id, dst_uri)
            Exports file to destination pointed at by the destination URL. The exported file will be executable if and only if it was originally uploaded from an executable file on the local filesystem.

            Refer to AbstractJobStore.import_file() <#toil.jobStores.abstractJobStore.AbstractJobStore.import_file> documentation for currently supported URL schemes.

            Note that the helper method _exportFile is used to read from the source and write to destination. To implement any optimizations that circumvent this, the _exportFile method should be overridden by subclasses of AbstractJobStore.

            Parameters

                •   file_id (FileID <#toil.fileStores.FileID>) -- The id of the file in the job store that should be exported.

                •   dst_uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- URL that points to a file or object in the storage mechanism of a supported URL scheme e.g. a blob in an AWS s3 bucket. May also be a local path.

                •   file_id

                •   dst_uri

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        abstractmethod destroy()
            The inverse of initialize() <#toil.jobStores.abstractJobStore.AbstractJobStore.initialize>, this method deletes the physical storage represented by this instance. While not being atomic, this method is at least idempotent, as a means to counteract potential issues with eventual consistency exhibited by the underlying storage mechanisms. This means that if the method fails (raises an exception), it may (and should be) invoked again. If the underlying storage mechanism is eventually consistent, even a successful invocation is not an ironclad guarantee that the physical storage vanished completely and immediately. A successful invocation only guarantees that the deletion will eventually happen. It is therefore recommended to not immediately reuse the same job store location for a new Toil workflow.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        get_env()
            Returns a dictionary of environment variables that this job store requires to be set in order to function properly on a worker.

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>,str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        clean(jobCache=None)
            Function to cleanup the state of a job store after a restart.

            Fixes jobs that might have been partially updated. Resets the try counts and removes jobs that are not successors of the current root job.

            Parameters
                jobCache (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str> | TemporaryID <#toil.job.TemporaryID>, JobDescription <#toil.job.JobDescription>] | None </usr/share/doc/python3/html/library/constants.html#None>) -- if a value it must be a dict from job ID keys to JobDescription object values. Jobs will be loaded from the cache (which can be downloaded from the job store in a batch) instead of piecemeal when recursed into.

            Return type
                JobDescription <#toil.job.JobDescription>

        abstractmethod assign_job_id(job_description)
            Get a new jobStoreID to be used by the described job, and assigns it to the JobDescription.

            Files associated with the assigned ID will be accepted even if the JobDescription has never been created or updated.

            Parameters

                •   job_description (JobDescription <#toil.job.JobDescription>) -- The JobDescription to give an ID to

                •   job_description

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        abstractmethod create_job(job_description)
            Writes the given JobDescription to the job store. The job must have an ID assigned already.

            Must call jobDescription.pre_update_hook()

            Returns
                The JobDescription passed.

            Return type
                toil.job.JobDescription <#toil.job.JobDescription>

            Parameters
                job_description (JobDescription <#toil.job.JobDescription>)

        abstractmethod job_exists(job_id)
            Indicates whether a description of the job with the specified jobStoreID exists in the job store

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

            Parameters
                job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        abstractmethod get_public_url(file_name)
            Returns a publicly accessible URL to the given file in the job store. The returned URL may expire as early as 1h after its been returned. Throw an exception if the file does not exist.

            Parameters

                •   file_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the jobStoreFileID of the file to generate a URL for

                •   file_name

            Raises
                NoSuchFileException <#toil.jobStores.abstractJobStore.NoSuchFileException> -- if the specified file does not exist in this job store

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        abstractmethod get_shared_public_url(shared_file_name)
            Differs from getPublicUrl() <#toil.jobStores.abstractJobStore.AbstractJobStore.getPublicUrl> in that this method is for generating URLs for shared files written by writeSharedFileStream() <#toil.jobStores.abstractJobStore.AbstractJobStore.writeSharedFileStream>.

            Returns a publicly accessible URL to the given file in the job store. The returned URL starts with 'http:', 'https:' or 'file:'. The returned URL may expire as early as 1h after its been returned. Throw an exception if the file does not exist.

            Parameters

                •   shared_file_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The name of the shared file to generate a publically accessible url for.

                •   shared_file_name

            Raises
                NoSuchFileException <#toil.jobStores.abstractJobStore.NoSuchFileException> -- raised if the specified file does not exist in the store

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        abstractmethod load_job(job_id)
            Loads the description of the job referenced by the given ID, assigns it the job store's config, and returns it.

            May declare the job to have failed (see toil.job.JobDescription.setupJobAfterFailure() <#toil.job.JobDescription.setupJobAfterFailure>) if there is evidence of a failed update attempt.

            Parameters
                job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the ID of the job to load

            Raises
                NoSuchJobException <#toil.jobStores.abstractJobStore.NoSuchJobException> -- if there is no job with the given ID

            Return type
                JobDescription <#toil.job.JobDescription>

        batch()
            If supported by the batch system, calls to create() with this context manager active will be performed in a batch after the context manager is released.

            Return type
                Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[None </usr/share/doc/python3/html/library/constants.html#None>]

        abstractmethod update_job(job_description)
            Persists changes to the state of the given JobDescription in this store atomically.

            Must call jobDescription.pre_update_hook()

            Parameters

                •   job (toil.job.JobDescription <#toil.job.JobDescription>) -- the job to write to this job store

                •   job_description (JobDescription <#toil.job.JobDescription>)

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        abstractmethod delete_job(job_id)
            Removes the JobDescription from the store atomically. You may not then subsequently call load(), write(), update(), etc. with the same jobStoreID or any JobDescription bearing it.

            This operation is idempotent, i.e. deleting a job twice or deleting a non-existent job will succeed silently.

            Parameters

                •   job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the ID of the job to delete from this job store

                •   job_id

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        jobs()
            Best effort attempt to return iterator on JobDescriptions for all jobs in the store. The iterator may not return all jobs and may also contain orphaned jobs that have already finished successfully and should not be rerun. To guarantee you get any and all jobs that can be run instead construct a more expensive ToilState object

            Returns
                Returns iterator on jobs in the store. The iterator may or may not contain all jobs and may contain invalid jobs

            Return type
                Iterator[toil.job.jobDescription]

        abstractmethod write_file(local_path, job_id=None, cleanup=False)
            Takes a file (as a path) and places it in this job store. Returns an ID that can be used to retrieve the file at a later time. The file is written in a atomic manner. It will not appear in the jobStore until the write has successfully completed.

            Parameters

                •   local_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the path to the local file that will be uploaded to the job store. The last path component (basename of the file) will remain associated with the file in the file store, if supported, so that the file can be searched for by name or name glob.

                •   job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the id of a job, or None. If specified, the may be associated with that job in a job-store-specific way. This may influence the returned ID.

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing.

            Raises

                •   ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

                •   NoSuchJobException <#toil.jobStores.abstractJobStore.NoSuchJobException> -- if the job specified via jobStoreID does not exist

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        FIXME: some implementations may not raise this

            Returns
                an ID referencing the newly created file and can be used to read the file in the future.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

            Parameters

                •   local_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>)

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        abstractmethod write_file_stream(job_id=None, cleanup=False, basename=None, encoding=None, errors=None)
            Similar to writeFile, but returns a context manager yielding a tuple of 1) a file handle which can be written to and 2) the ID of the resulting file in the job store. The yielded file handle does not need to and should not be closed explicitly. The file is written in a atomic manner. It will not appear in the jobStore until the write has successfully completed.

            Parameters

                •   job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the id of a job, or None. If specified, the may be associated with that job in a job-store-specific way. This may influence the returned ID.

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing.

                •   basename (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- If supported by the implementation, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected.

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

            Raises

                •   ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

                •   NoSuchJobException <#toil.jobStores.abstractJobStore.NoSuchJobException> -- if the job specified via jobStoreID does not exist

            Return type
                Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[IO </usr/share/doc/python3/html/library/typing.html#typing.IO>[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>], str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

        FIXME: some implementations may not raise this

            Returns
                a context manager yielding a file handle which can be written to and an ID that references the newly created file and can be used to read the file in the future.

            Return type
                Iterator[Tuple[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>], str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

            Parameters

                •   job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>)

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>)

                •   basename (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>)

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>)

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>)

        abstractmethod get_empty_file_store_id(job_id=None, cleanup=False, basename=None)
            Creates an empty file in the job store and returns its ID. Call to fileExists(getEmptyFileStoreID(jobStoreID)) will return True.

            Parameters

                •   job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- the id of a job, or None. If specified, the may be associated with that job in a job-store-specific way. This may influence the returned ID.

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing.

                •   basename (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- If supported by the implementation, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected.

            Returns
                a jobStoreFileID that references the newly created file and can be used to reference the file in the future.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        abstractmethod read_file(file_id, local_path, symlink=False)
            Copies or hard links the file referenced by jobStoreFileID to the given local file path. The version will be consistent with the last copy of the file written/updated. If the file in the job store is later modified via updateFile or updateFileStream, it is implementation-defined whether those writes will be visible at localFilePath. The file is copied in an atomic manner. It will not appear in the local file system until the copy has completed.

            The file at the given local path may not be modified after this method returns!

            Note! Implementations of readFile need to respect/provide the executable attribute on FileIDs.

            Parameters

                •   file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- ID of the file to be copied

                •   local_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the local path indicating where to place the contents of the given file in the job store

                •   symlink (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- whether the reader can tolerate a symlink. If set to true, the job store may create a symlink instead of a full copy of the file or a hard link.

                •   file_id

                •   local_path

                •   symlink

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        abstractmethod read_file_stream(file_id, encoding=None, errors=None)
            Similar to readFile, but returns a context manager yielding a file handle which can be read from. The yielded file handle does not need to and should not be closed explicitly.

            Parameters

                •   file_id (FileID <#toil.fileStores.FileID> | str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- ID of the file to get a readable file handle for

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

                •   file_id

                •   encoding

                •   errors

            Returns
                a context manager yielding a file handle which can be read from

            Return type
                Iterator[Union[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>], IO[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]]

        abstractmethod delete_file(file_id)
            Deletes the file with the given ID from this job store. This operation is idempotent, i.e. deleting a file twice or deleting a non-existent file will succeed silently.

            Parameters

                •   file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- ID of the file to delete

                •   file_id

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        fileExists(jobStoreFileID)
            Determine whether a file exists in this job store.

            Parameters
                jobStoreFileID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        abstractmethod file_exists(file_id)
            Determine whether a file exists in this job store.

            Parameters
                file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an ID referencing the file to be checked

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        getFileSize(jobStoreFileID)
            Get the size of the given file in bytes.

            Parameters
                jobStoreFileID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        abstractmethod get_file_size(file_id)
            Get the size of the given file in bytes, or 0 if it does not exist when queried.

            Note that job stores which encrypt files might return overestimates of file sizes, since the encrypted file may have been padded to the nearest block, augmented with an initialization vector, etc.

            Parameters

                •   file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an ID referencing the file to be checked

                •   file_id

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        updateFile(jobStoreFileID, localFilePath)
            Replaces the existing version of a file in the job store.

            Parameters

                •   jobStoreFileID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   localFilePath (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        abstractmethod update_file(file_id, local_path)
            Replaces the existing version of a file in the job store.

            Throws an exception if the file does not exist.

            Parameters

                •   file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the ID of the file in the job store to be updated

                •   local_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the local path to a file that will overwrite the current version in the job store

            Raises

                •   ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

                •   NoSuchFileException <#toil.jobStores.abstractJobStore.NoSuchFileException> -- if the specified file does not exist

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        abstractmethod update_file_stream(file_id, encoding=None, errors=None)
            Replaces the existing version of a file in the job store. Similar to writeFile, but returns a context manager yielding a file handle which can be written to. The yielded file handle does not need to and should not be closed explicitly.

            Parameters

                •   file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the ID of the file in the job store to be updated

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

                •   file_id

                •   encoding

                •   errors

            Raises

                •   ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

                •   NoSuchFileException <#toil.jobStores.abstractJobStore.NoSuchFileException> -- if the specified file does not exist

            Return type
                Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[IO </usr/share/doc/python3/html/library/typing.html#typing.IO>[Any </usr/share/doc/python3/html/library/typing.html#typing.Any>]]

        abstractmethod write_shared_file_stream(shared_file_name, encrypted=None, encoding=None, errors=None)
            Returns a context manager yielding a writable file handle to the global file referenced by the given name. File will be created in an atomic manner.

            Parameters

                •   shared_file_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- A file name matching AbstractJobStore.fileNameRegex, unique within this job store

                •   encrypted (bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None>) -- True if the file must be encrypted, None if it may be encrypted or False if it must be stored in the clear.

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

                •   shared_file_name

                •   encrypted

                •   encoding

                •   errors

            Raises
                ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

            Returns
                a context manager yielding a writable file handle

            Return type
                Iterator[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]]

        abstractmethod read_shared_file_stream(shared_file_name, encoding=None, errors=None)
            Returns a context manager yielding a readable file handle to the global file referenced by the given name.

            Parameters

                •   shared_file_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- A file name matching AbstractJobStore.fileNameRegex, unique within this job store

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

                •   shared_file_name

                •   encoding

                •   errors

            Return type
                Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[IO </usr/share/doc/python3/html/library/typing.html#typing.IO>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]] | Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[IO </usr/share/doc/python3/html/library/typing.html#typing.IO>[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]]

            Returns
                a context manager yielding a readable file handle

        abstractmethod write_logs(msg)
            Stores a message as a log in the jobstore.

            Parameters

                •   msg (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the string to be written

                •   msg

            Raises
                ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        abstractmethod read_logs(callback, read_all=False)
            Reads logs accumulated by the write_logs() method. For each log this method calls the given callback function with the message as an argument (rather than returning logs directly, this method must be supplied with a callback which will process log messages).

            Only unread logs will be read unless the read_all parameter is set.

            Parameters

                •   callback (Callable </usr/share/doc/python3/html/library/typing.html#typing.Callable>[... </usr/share/doc/python3/html/library/constants.html#Ellipsis>, Any </usr/share/doc/python3/html/library/typing.html#typing.Any>]) -- a function to be applied to each of the stats file handles found

                •   read_all (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- a boolean indicating whether to read the already processed stats files in addition to the unread stats files

                •   callback

                •   read_all

            Raises
                ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

            Returns
                the number of stats files processed

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        write_leader_pid()
            Write the pid of this process to a file in the job store.

            Overwriting the current contents of pid.log is a feature, not a bug of this method. Other methods will rely on always having the most current pid available. So far there is no reason to store any old pids.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        read_leader_pid()
            Read the pid of the leader process to a file in the job store.

            Raises
                NoSuchFileException <#toil.jobStores.abstractJobStore.NoSuchFileException> -- If the PID file doesn't exist.

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        write_leader_node_id()
            Write the leader node id to the job store. This should only be called by the leader.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        read_leader_node_id()
            Read the leader node id stored in the job store.

            Raises
                NoSuchFileException <#toil.jobStores.abstractJobStore.NoSuchFileException> -- If the node ID file doesn't exist.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        write_kill_flag(kill=False)
            Write a file inside the job store that serves as a kill flag.

            The initialized file contains the characters "NO". This should only be changed when the user runs the "toil kill" command.

            Changing this file to a "YES" triggers a kill of the leader process. The workers are expected to be cleaned up by the leader.

            Parameters
                kill (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        read_kill_flag()
            Read the kill flag from the job store, and return True if the leader has been killed. False otherwise.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        default_caching()
            Jobstore's preference as to whether it likes caching or doesn't care about it. Some jobstores benefit from caching, however on some local configurations it can be flaky.

            see <https://github.com/DataBiosphere/toil/issues/4218>

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

TOIL JOB API

Functions to wrap jobs and return values (promises).

  FunctionWrappingJob

The subclass of Job for wrapping user functions.

    class toil.job.FunctionWrappingJob(userFunction, *args, **kwargs)
        Job used to wrap a function. In its run method the wrapped function is called.

        Parameters

            •   userFunction (Callable </usr/share/doc/python3/html/library/typing.html#typing.Callable>[... </usr/share/doc/python3/html/library/constants.html#Ellipsis>, Any </usr/share/doc/python3/html/library/typing.html#typing.Any>])

            •   args (Any </usr/share/doc/python3/html/library/typing.html#typing.Any>)

            •   kwargs (Any </usr/share/doc/python3/html/library/typing.html#typing.Any>)

        __init__(userFunction, *args, **kwargs)

            Parameters

                •   userFunction (callable) -- The function to wrap. It will be called with *args and **kwargs as arguments.

                •   args (Any </usr/share/doc/python3/html/library/typing.html#typing.Any>)

                •   kwargs (Any </usr/share/doc/python3/html/library/typing.html#typing.Any>)

            Return type
                None

        The keywords memory, cores, disk, accelerators`, ``preemptible and checkpoint are reserved keyword arguments that if specified will be used to determine the resources required for the job, as toil.job.Job.__init__(). If they are keyword arguments to the function they will be extracted from the function definition, but may be overridden by the user (as you would expect).

            Parameters

                •   userFunction (Callable </usr/share/doc/python3/html/library/typing.html#typing.Callable>[... </usr/share/doc/python3/html/library/constants.html#Ellipsis>, Any </usr/share/doc/python3/html/library/typing.html#typing.Any>])

                •   args (Any </usr/share/doc/python3/html/library/typing.html#typing.Any>)

                •   kwargs (Any </usr/share/doc/python3/html/library/typing.html#typing.Any>)

            Return type
                None

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore (AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>) -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Return type
                Any </usr/share/doc/python3/html/library/typing.html#typing.Any>

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>.

  JobFunctionWrappingJob

The subclass of FunctionWrappingJob for wrapping user job functions.

    class toil.job.JobFunctionWrappingJob(userFunction, *args, **kwargs)
        A job function is a function whose first argument is a Job <#toil.job.Job> instance that is the wrapping job for the function. This can be used to add successor jobs for the function and perform all the functions the Job <#toil.job.Job> class provides.

        To enable the job function to get access to the toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore> instance (see toil.job.Job.run() <#toil.job.Job.run>), it is made a variable of the wrapping job called fileStore.

        To specify a job's resource requirements the following default keyword arguments can be specified:

                •   memory

                •   disk

                •   cores

                •   accelerators

                •   preemptible

    For example to wrap a function into a job we would call:

                Job.wrapJobFn(myJob, memory='100k', disk='1M', cores=0.1)

        Parameters

            •   userFunction (Callable </usr/share/doc/python3/html/library/typing.html#typing.Callable>[... </usr/share/doc/python3/html/library/constants.html#Ellipsis>, Any </usr/share/doc/python3/html/library/typing.html#typing.Any>])

            •   args (Any </usr/share/doc/python3/html/library/typing.html#typing.Any>)

            •   kwargs (Any </usr/share/doc/python3/html/library/typing.html#typing.Any>)

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore (AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>) -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Return type
                Any </usr/share/doc/python3/html/library/typing.html#typing.Any>

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>.

  EncapsulatedJob

The subclass of Job for encapsulating a job, allowing a subgraph of jobs to be treated as a single job.

    class toil.job.EncapsulatedJob(job, unitName=None)
        A convenience Job class used to make a job subgraph appear to be a single job.

        Let A be the root job of a job subgraph and B be another job we'd like to run after A and all its successors have completed, for this use encapsulate:

                #  Job A and subgraph, Job B
                A, B = A(), B()
                Aprime = A.encapsulate()
                Aprime.addChild(B)
                #  B will run after A and all its successors have completed, A and its subgraph of
                # successors in effect appear to be just one job.

    If the job being encapsulated has predecessors (e.g. is not the root job), then the encapsulated job will inherit these predecessors. If predecessors are added to the job being encapsulated after the encapsulated job is created then the encapsulating job will NOT inherit these predecessors automatically. Care should be exercised to ensure the encapsulated job has the proper set of predecessors.

    The return value of an encapsulated job (as accessed by the toil.job.Job.rv() <#toil.job.Job.rv> function) is the return value of the root job, e.g. A().encapsulate().rv() and A().rv() will resolve to the same value after A or A.encapsulate() has been run.

        Parameters

            •   job (Job <#toil.job.Job> | None </usr/share/doc/python3/html/library/constants.html#None>)

            •   unitName (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>)

        __init__(job, unitName=None)

            Parameters

                •   job (Job <#toil.job.Job> | None </usr/share/doc/python3/html/library/constants.html#None>) -- the job to encapsulate.

                •   unitName (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- human-readable name to identify this job instance.

                •   job

                •   unitName

            Return type
                None

        addChild(childJob)
            Add a childJob to be run as child of this job.

            Child jobs will be run directly after this job's toil.job.Job.run() <#toil.job.Job.run> method has completed.

            Return type
                Job <#toil.job.Job>

            Returns
                childJob: for call chaining

            Parameters
                childJob (Job <#toil.job.Job>)

        addService(service, parentService=None)
            Add a service.

            The toil.job.Job.Service.start() <#toil.job.Job.Service.start> method of the service will be called after the run method has completed but before any successors are run. The service's toil.job.Job.Service.stop() <#toil.job.Job.Service.stop> method will be called once the successors of the job have been run.

            Services allow things like databases and servers to be started and accessed by jobs in a workflow.

            Raises
                toil.job.JobException <#toil.job.JobException> -- If service has already been made the child of a job or another service.

            Parameters

                •   service -- Service to add.

                •   parentService -- Service that will be started before 'service' is started. Allows trees of services to be established. parentService must be a service of this job.

            Returns
                a promise that will be replaced with the return value from toil.job.Job.Service.start() <#toil.job.Job.Service.start> of service in any successor of the job.

        addFollowOn(followOnJob)
            Add a follow-on job.

            Follow-on jobs will be run after the child jobs and their successors have been run.

            Return type
                Job <#toil.job.Job>

            Returns
                followOnJob for call chaining

            Parameters
                followOnJob (Job <#toil.job.Job>)

        rv(*path)
            Create a promise (toil.job.Promise <#toil.job.Promise>).

            The "promise" representing a return value of the job's run method, or, in case of a function-wrapping job, the wrapped function's return value.

            Parameters
                path ((Any)) -- Optional path for selecting a component of the promised return value. If absent or empty, the entire return value will be used. Otherwise, the first element of the path is used to select an individual item of the return value. For that to work, the return value must be a list, dictionary or of any other type implementing the __getitem__() magic method. If the selected item is yet another composite value, the second element of the path can be used to select an item from it, and so on. For example, if the return value is [6,{'a':42}], .rv(0) would select 6 , rv(1) would select {'a':3} while rv(1,'a') would select 3. To select a slice from a return value that is slicable, e.g. tuple or list, the path element should be a slice object. For example, assuming that the return value is [6, 7, 8, 9] then .rv(slice(1, 3)) would select [7, 8]. Note that slicing really only makes sense at the end of path.

            Return type
                Promise <#toil.job.Promise>

            Returns
                A promise representing the return value of this jobs toil.job.Job.run() <#toil.job.Job.run> method.

        prepareForPromiseRegistration(jobStore)
            Set up to allow this job's promises to register themselves.

            Prepare this job (the promisor) so that its promises can register themselves with it, when the jobs they are promised to (promisees) are serialized.

            The promissee holds the reference to the promise (usually as part of the job arguments) and when it is being pickled, so will the promises it refers to. Pickling a promise triggers it to be registered with the promissor.

  Promise

The class used to reference return values of jobs/services not yet run/started.

    class toil.job.Promise(job, path)
        References a return value from a method as a promise before the method itself is run.

        References a return value from a toil.job.Job.run() <#toil.job.Job.run> or toil.job.Job.Service.start() <#toil.job.Job.Service.start> method as a promise before the method itself is run.

        Let T be a job. Instances of Promise <#toil.job.Promise> (termed a promise) are returned by T.rv(), which is used to reference the return value of T's run function. When the promise is passed to the constructor (or as an argument to a wrapped function) of a different, successor job the promise will be replaced by the actual referenced return value. This mechanism allows a return values from one job's run method to be input argument to job before the former job's run function has been executed.

        Parameters

            •   job (Job <#toil.job.Job>)

            •   path (Any </usr/share/doc/python3/html/library/typing.html#typing.Any>)

        Return type
            Promise <#toil.job.Promise>

        filesToDelete = {}
            A set of IDs of files containing promised values when we know we won't need them anymore

        __init__(job, path)
            Initialize this promise.

            Parameters

                •   job (Job <#toil.job.Job>) -- the job whose return value this promise references

                •   path (Any </usr/share/doc/python3/html/library/typing.html#typing.Any>) -- see Job.rv() <#toil.job.Job.rv>

                •   job

    class toil.job.PromisedRequirement(valueOrCallable, *args)
        Class for dynamically allocating job function resource requirements.

        (involving toil.job.Promise <#toil.job.Promise> instances.)

        Use when resource requirements depend on the return value of a parent function. PromisedRequirements can be modified by passing a function that takes the Promise <#toil.job.Promise> as input.

        For example, let f, g, and h be functions. Then a Toil workflow can be defined as follows:: A = Job.wrapFn(f) B = A.addChildFn(g, cores=PromisedRequirement(A.rv()) C = B.addChildFn(h, cores=PromisedRequirement(lambda x: 2*x, B.rv()))

        Parameters

            •   valueOrCallable (Any </usr/share/doc/python3/html/library/typing.html#typing.Any>)

            •   args (Any </usr/share/doc/python3/html/library/typing.html#typing.Any>)

        __init__(valueOrCallable, *args)
            Initialize this Promised Requirement.

            Parameters

                •   valueOrCallable (Any </usr/share/doc/python3/html/library/typing.html#typing.Any>) -- A single Promise instance or a function that takes args as input parameters.

                •   args (int </usr/share/doc/python3/html/library/functions.html#int> or .Promise) -- variable length argument list

            Return type
                None

        getValue()
            Return PromisedRequirement value.

            Return type
                Any </usr/share/doc/python3/html/library/typing.html#typing.Any>

        static convertPromises(kwargs)
            Return True if reserved resource keyword is a Promise or PromisedRequirement instance.

            Converts Promise instance to PromisedRequirement.

            Parameters
                kwargs (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any </usr/share/doc/python3/html/library/typing.html#typing.Any>]) -- function keyword arguments

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

JOB METHODS API

Jobs are the units of work in Toil which are composed into workflows.

    class toil.job.Job(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, preemptable=None, unitName='', checkpoint=False, displayName='', descriptionClass=None, local=None, files=None)
        Class represents a unit of work in toil.

        Parameters

            •   memory (str </usr/share/doc/python3/html/library/stdtypes.html#str> | int </usr/share/doc/python3/html/library/functions.html#int> | None </usr/share/doc/python3/html/library/constants.html#None>)

            •   cores (str </usr/share/doc/python3/html/library/stdtypes.html#str> | int </usr/share/doc/python3/html/library/functions.html#int> | float </usr/share/doc/python3/html/library/functions.html#float> | None </usr/share/doc/python3/html/library/constants.html#None>)

            •   disk (str </usr/share/doc/python3/html/library/stdtypes.html#str> | int </usr/share/doc/python3/html/library/functions.html#int> | None </usr/share/doc/python3/html/library/constants.html#None>)

            •   accelerators (str </usr/share/doc/python3/html/library/stdtypes.html#str> | int </usr/share/doc/python3/html/library/functions.html#int> | Mapping </usr/share/doc/python3/html/library/typing.html#typing.Mapping>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any </usr/share/doc/python3/html/library/typing.html#typing.Any>] | AcceleratorRequirement <#toil.job.AcceleratorRequirement> | Sequence </usr/share/doc/python3/html/library/typing.html#typing.Sequence>[str </usr/share/doc/python3/html/library/stdtypes.html#str> | int </usr/share/doc/python3/html/library/functions.html#int> | Mapping </usr/share/doc/python3/html/library/typing.html#typing.Mapping>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any </usr/share/doc/python3/html/library/typing.html#typing.Any>] | AcceleratorRequirement <#toil.job.AcceleratorRequirement>] | None </usr/share/doc/python3/html/library/constants.html#None>)

            •   preemptible (str </usr/share/doc/python3/html/library/stdtypes.html#str> | int </usr/share/doc/python3/html/library/functions.html#int> | bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None>)

            •   preemptable (str </usr/share/doc/python3/html/library/stdtypes.html#str> | int </usr/share/doc/python3/html/library/functions.html#int> | bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None>)

            •   unitName (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>)

            •   checkpoint (bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None>)

            •   displayName (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>)

            •   descriptionClass (type </usr/share/doc/python3/html/library/functions.html#type> | None </usr/share/doc/python3/html/library/constants.html#None>)

            •   local (bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None>)

            •   files (set </usr/share/doc/python3/html/library/stdtypes.html#set>[FileID <#toil.fileStores.FileID>] | None </usr/share/doc/python3/html/library/constants.html#None>)

        __init__(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, preemptable=None, unitName='', checkpoint=False, displayName='', descriptionClass=None, local=None, files=None)
            Job initializer.

            This method must be called by any overriding constructor.

            Parameters

                •   memory (int </usr/share/doc/python3/html/library/functions.html#int> or string convertible by toil.lib.conversions.human2bytes to an int) -- the maximum number of bytes of memory the job will require to run.

                •   cores (float </usr/share/doc/python3/html/library/functions.html#float>, int </usr/share/doc/python3/html/library/functions.html#int>, or string convertible by toil.lib.conversions.human2bytes to an int) -- the number of CPU cores required.

                •   disk (int </usr/share/doc/python3/html/library/functions.html#int> or string convertible by toil.lib.conversions.human2bytes to an int) -- the amount of local disk space required by the job, expressed in bytes.

                •   accelerators (int </usr/share/doc/python3/html/library/functions.html#int>, string, dict </usr/share/doc/python3/html/library/stdtypes.html#dict>, or list </usr/share/doc/python3/html/library/stdtypes.html#list> of those. Strings and dicts must be parseable by parse_accelerator.) -- the computational accelerators required by the job. If a string, can be a string of a number, or a string specifying a model, brand, or API (with optional colon-delimited count).

                •   preemptible (bool </usr/share/doc/python3/html/library/functions.html#bool>, int in {0, 1}, or string in {'false', 'true'} in any case) -- if the job can be run on a preemptible node.

                •   preemptable (str </usr/share/doc/python3/html/library/stdtypes.html#str> | int </usr/share/doc/python3/html/library/functions.html#int> | bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None>) -- legacy preemptible parameter, for backwards compatibility with workflows not using the preemptible keyword

                •   unitName (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Human-readable name for this instance of the job.

                •   checkpoint (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- if any of this job's successor jobs completely fails, exhausting all their retries, remove any successor jobs and rerun this job to restart the subtree. Job must be a leaf vertex in the job graph when initially defined, see toil.job.Job.checkNewCheckpointsAreCutVertices().

                •   displayName (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Human-readable job type display name.

                •   descriptionClass (class) -- Override for the JobDescription class used to describe the job.

                •   local (bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None>) -- if the job can be run on the leader.

                •   files (set </usr/share/doc/python3/html/library/stdtypes.html#set>[FileID <#toil.fileStores.FileID>] | None </usr/share/doc/python3/html/library/constants.html#None>) -- Set of Files that the job will want to use.

            Return type
                None

        check_initialized()
            Ensure that Job.__init__() has been called by any subclass __init__().

            This uses the fact that the self._description instance variable should always be set after __init__().

            If __init__() has not been called, raise an error.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        property jobStoreID: str </usr/share/doc/python3/html/library/stdtypes.html#str> | TemporaryID <#toil.job.TemporaryID>
            Get the ID of this Job.

            Return type
                Union[str </usr/share/doc/python3/html/library/stdtypes.html#str>, TemporaryID <#toil.job.TemporaryID>]

        property description: JobDescription <#toil.job.JobDescription>
            Expose the JobDescription that describes this job.

            Return type
                JobDescription <#toil.job.JobDescription>

        property disk: int </usr/share/doc/python3/html/library/functions.html#int>
            The maximum number of bytes of disk the job will require to run.

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        property memory: int </usr/share/doc/python3/html/library/functions.html#int>
            The maximum number of bytes of memory the job will require to run.

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        property cores: int </usr/share/doc/python3/html/library/functions.html#int> | float </usr/share/doc/python3/html/library/functions.html#float>
            The number of CPU cores required.

            Return type
                Union[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>]

        property accelerators: list </usr/share/doc/python3/html/library/stdtypes.html#list>[AcceleratorRequirement <#toil.job.AcceleratorRequirement>]
            Any accelerators, such as GPUs, that are needed.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[AcceleratorRequirement <#toil.job.AcceleratorRequirement>]

        property preemptible: bool </usr/share/doc/python3/html/library/functions.html#bool>
            Whether the job can be run on a preemptible node.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        property checkpoint: bool </usr/share/doc/python3/html/library/functions.html#bool>
            Determine if the job is a checkpoint job or not.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        assignConfig(config)
            Assign the given config object.

            It will be used by various actions implemented inside the Job class.

            Parameters
                config (Config <#toil.common.Config>) -- Config object to query

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore (AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>) -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Return type
                Any </usr/share/doc/python3/html/library/typing.html#typing.Any>

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>.

        addChild(childJob)
            Add a childJob to be run as child of this job.

            Child jobs will be run directly after this job's toil.job.Job.run() <#toil.job.Job.run> method has completed.

            Return type
                Job <#toil.job.Job>

            Returns
                childJob: for call chaining

            Parameters
                childJob (Job <#toil.job.Job>)

        hasChild(childJob)
            Check if childJob is already a child of this job.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

            Returns
                True if childJob is a child of the job, else False.

            Parameters
                childJob (Job <#toil.job.Job>)

        addFollowOn(followOnJob)
            Add a follow-on job.

            Follow-on jobs will be run after the child jobs and their successors have been run.

            Return type
                Job <#toil.job.Job>

            Returns
                followOnJob for call chaining

            Parameters
                followOnJob (Job <#toil.job.Job>)

        hasPredecessor(job)
            Check if a given job is already a predecessor of this job.

            Parameters
                job (Job <#toil.job.Job>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        hasFollowOn(followOnJob)
            Check if given job is already a follow-on of this job.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

            Returns
                True if the followOnJob is a follow-on of this job, else False.

            Parameters
                followOnJob (Job <#toil.job.Job>)

        addService(service, parentService=None)
            Add a service.

            The toil.job.Job.Service.start() <#toil.job.Job.Service.start> method of the service will be called after the run method has completed but before any successors are run. The service's toil.job.Job.Service.stop() <#toil.job.Job.Service.stop> method will be called once the successors of the job have been run.

            Services allow things like databases and servers to be started and accessed by jobs in a workflow.

            Raises
                toil.job.JobException <#toil.job.JobException> -- If service has already been made the child of a job or another service.

            Parameters

                •   service (Service <#toil.job.Job.Service>) -- Service to add.

                •   parentService (Service <#toil.job.Job.Service> | None </usr/share/doc/python3/html/library/constants.html#None>) -- Service that will be started before 'service' is started. Allows trees of services to be established. parentService must be a service of this job.

            Return type
                Promise <#toil.job.Promise>

            Returns
                a promise that will be replaced with the return value from toil.job.Job.Service.start() <#toil.job.Job.Service.start> of service in any successor of the job.

        hasService(service)
            Return True if the given Service is a service of this job, and False otherwise.

            Parameters
                service (Service <#toil.job.Job.Service>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        addChildFn(fn, *args, **kwargs)
            Add a function as a child job.

            Parameters
                fn (Callable </usr/share/doc/python3/html/library/typing.html#typing.Callable>) -- Function to be run as a child job with *args and **kwargs as arguments to this function. See toil.job.FunctionWrappingJob for reserved keyword arguments used to specify resource requirements.

            Return type
                FunctionWrappingJob <#toil.job.FunctionWrappingJob>

            Returns
                The new child job that wraps fn.

        addFollowOnFn(fn, *args, **kwargs)
            Add a function as a follow-on job.

            Parameters
                fn (Callable </usr/share/doc/python3/html/library/typing.html#typing.Callable>) -- Function to be run as a follow-on job with *args and **kwargs as arguments to this function. See toil.job.FunctionWrappingJob for reserved keyword arguments used to specify resource requirements.

            Return type
                FunctionWrappingJob <#toil.job.FunctionWrappingJob>

            Returns
                The new follow-on job that wraps fn.

        addChildJobFn(fn, *args, **kwargs)
            Add a job function as a child job.

            See toil.job.JobFunctionWrappingJob <#toil.job.JobFunctionWrappingJob> for a definition of a job function.

            Parameters
                fn (Callable </usr/share/doc/python3/html/library/typing.html#typing.Callable>) -- Job function to be run as a child job with *args and **kwargs as arguments to this function. See toil.job.JobFunctionWrappingJob for reserved keyword arguments used to specify resource requirements.

            Return type
                FunctionWrappingJob <#toil.job.FunctionWrappingJob>

            Returns
                The new child job that wraps fn.

        addFollowOnJobFn(fn, *args, **kwargs)
            Add a follow-on job function.

            See toil.job.JobFunctionWrappingJob <#toil.job.JobFunctionWrappingJob> for a definition of a job function.

            Parameters
                fn (Callable </usr/share/doc/python3/html/library/typing.html#typing.Callable>) -- Job function to be run as a follow-on job with *args and **kwargs as arguments to this function. See toil.job.JobFunctionWrappingJob for reserved keyword arguments used to specify resource requirements.

            Return type
                FunctionWrappingJob <#toil.job.FunctionWrappingJob>

            Returns
                The new follow-on job that wraps fn.

        property tempDir: str </usr/share/doc/python3/html/library/stdtypes.html#str>
            Shortcut to calling job.fileStore.getLocalTempDir().

            Temp dir is created on first call and will be returned for first and future calls :return: Path to tempDir. See job.fileStore.getLocalTempDir

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        log(text, level=20)
            Log using fileStore.log_to_leader().

            Parameters
                text (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        static wrapFn(fn, *args, **kwargs)
            Makes a Job out of a function.

            Convenience function for constructor of toil.job.FunctionWrappingJob <#toil.job.FunctionWrappingJob>.

            Parameters
                fn -- Function to be run with *args and **kwargs as arguments. See toil.job.JobFunctionWrappingJob for reserved keyword arguments used to specify resource requirements.

            Return type
                FunctionWrappingJob <#toil.job.FunctionWrappingJob>

            Returns
                The new function that wraps fn.

        static wrapJobFn(fn, *args, **kwargs)
            Makes a Job out of a job function.

            Convenience function for constructor of toil.job.JobFunctionWrappingJob <#toil.job.JobFunctionWrappingJob>.

            Parameters
                fn -- Job function to be run with *args and **kwargs as arguments. See toil.job.JobFunctionWrappingJob for reserved keyword arguments used to specify resource requirements.

            Return type
                JobFunctionWrappingJob <#toil.job.JobFunctionWrappingJob>

            Returns
                The new job function that wraps fn.

        encapsulate(name=None)
            Encapsulates the job, see toil.job.EncapsulatedJob <#toil.job.EncapsulatedJob>. Convenience function for constructor of toil.job.EncapsulatedJob <#toil.job.EncapsulatedJob>.

            Parameters
                name (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- Human-readable name for the encapsulated job.

            Return type
                EncapsulatedJob <#toil.job.EncapsulatedJob>

            Returns
                an encapsulated version of this job.

        rv(*path)
            Create a promise (toil.job.Promise <#toil.job.Promise>).

            The "promise" representing a return value of the job's run method, or, in case of a function-wrapping job, the wrapped function's return value.

            Parameters
                path ((Any)) -- Optional path for selecting a component of the promised return value. If absent or empty, the entire return value will be used. Otherwise, the first element of the path is used to select an individual item of the return value. For that to work, the return value must be a list, dictionary or of any other type implementing the __getitem__() magic method. If the selected item is yet another composite value, the second element of the path can be used to select an item from it, and so on. For example, if the return value is [6,{'a':42}], .rv(0) would select 6 , rv(1) would select {'a':3} while rv(1,'a') would select 3. To select a slice from a return value that is slicable, e.g. tuple or list, the path element should be a slice object. For example, assuming that the return value is [6, 7, 8, 9] then .rv(slice(1, 3)) would select [7, 8]. Note that slicing really only makes sense at the end of path.

            Return type
                Promise <#toil.job.Promise>

            Returns
                A promise representing the return value of this jobs toil.job.Job.run() <#toil.job.Job.run> method.

        prepareForPromiseRegistration(jobStore)
            Set up to allow this job's promises to register themselves.

            Prepare this job (the promisor) so that its promises can register themselves with it, when the jobs they are promised to (promisees) are serialized.

            The promissee holds the reference to the promise (usually as part of the job arguments) and when it is being pickled, so will the promises it refers to. Pickling a promise triggers it to be registered with the promissor.

            Parameters
                jobStore (AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        checkJobGraphForDeadlocks()
            Ensures that a graph of Jobs (that hasn't yet been saved to the JobStore) doesn't contain any pathological relationships between jobs that would result in deadlocks if we tried to run the jobs.

            See toil.job.Job.checkJobGraphConnected() <#toil.job.Job.checkJobGraphConnected>, toil.job.Job.checkJobGraphAcyclic() and toil.job.Job.checkNewCheckpointsAreLeafVertices() <#toil.job.Job.checkNewCheckpointsAreLeafVertices> for more info.

            Raises
                toil.job.JobGraphDeadlockException <#toil.job.JobGraphDeadlockException> -- if the job graph is cyclic, contains multiple roots or contains checkpoint jobs that are not leaf vertices when defined (see toil.job.Job.checkNewCheckpointsAreLeaves()).

        getRootJobs()
            Return the set of root job objects that contain this job.

            A root job is a job with no predecessors (i.e. which are not children, follow-ons, or services).

            Only deals with jobs created here, rather than loaded from the job store.

            Return type
                set </usr/share/doc/python3/html/library/stdtypes.html#set>[Job <#toil.job.Job>]

        checkJobGraphConnected()

            Raises
                toil.job.JobGraphDeadlockException <#toil.job.JobGraphDeadlockException> -- if toil.job.Job.getRootJobs() <#toil.job.Job.getRootJobs> does not contain exactly one root job.

            Return type
                None

        As execution always starts from one root job, having multiple root jobs will cause a deadlock to occur.

        Only deals with jobs created here, rather than loaded from the job store.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        checkJobGraphAcylic()

            Raises
                toil.job.JobGraphDeadlockException <#toil.job.JobGraphDeadlockException> -- if the connected component of jobs containing this job contains any cycles of child/followOn dependencies in the augmented job graph (see below). Such cycles are not allowed in valid job graphs.

            Return type
                None

        A follow-on edge (A, B) between two jobs A and B is equivalent to adding a child edge to B from (1) A, (2) from each child of A, and (3) from the successors of each child of A. We call each such edge an edge an "implied" edge. The augmented job graph is a job graph including all the implied edges.

        For a job graph G = (V, E) the algorithm is O(|V|^2). It is O(|V| + |E|) for a graph with no follow-ons. The former follow-on case could be improved!

        Only deals with jobs created here, rather than loaded from the job store.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        checkNewCheckpointsAreLeafVertices()
            A checkpoint job is a job that is restarted if either it fails, or if any of its successors completely fails, exhausting their retries.

            A job is a leaf it is has no successors.

            A checkpoint job must be a leaf when initially added to the job graph. When its run method is invoked it can then create direct successors. This restriction is made to simplify implementation.

            Only works on connected components of jobs not yet added to the JobStore.

            Raises
                toil.job.JobGraphDeadlockException <#toil.job.JobGraphDeadlockException> -- if there exists a job being added to the graph for which checkpoint=True and which is not a leaf.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        defer(function, *args, **kwargs)
            Register a deferred function, i.e. a callable that will be invoked after the current attempt at running this job concludes. A job attempt is said to conclude when the job function (or the toil.job.Job.run() <#toil.job.Job.run> method for class-based jobs) returns, raises an exception or after the process running it terminates abnormally. A deferred function will be called on the node that attempted to run the job, even if a subsequent attempt is made on another node. A deferred function should be idempotent because it may be called multiple times on the same node or even in the same process. More than one deferred function may be registered per job attempt by calling this method repeatedly with different arguments. If the same function is registered twice with the same or different arguments, it will be called twice per job attempt.

            Examples for deferred functions are ones that handle cleanup of resources external to Toil, like Docker containers, files outside the work directory, etc.

            Parameters

                •   function (callable) -- The function to be called after this job concludes.

                •   args (list </usr/share/doc/python3/html/library/stdtypes.html#list>) -- The arguments to the function

                •   kwargs (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>) -- The keyword arguments to the function

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        getTopologicalOrderingOfJobs()

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[Job <#toil.job.Job>]

            Returns
                a list of jobs such that for all pairs of indices i, j for which i < j, the job at index i can be run before the job at index j.

        Only considers jobs in this job's subgraph that are newly added, not loaded from the job store.

        Ignores service jobs.

        saveBody(jobStore)
            Save the execution data for just this job to the JobStore, and fill in the JobDescription with the information needed to retrieve it.

            The Job's JobDescription must have already had a real jobStoreID assigned to it.

            Does not save the JobDescription.

            Parameters
                jobStore (AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>) -- The job store to save the job body into.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        saveAsRootJob(jobStore)
            Save this job to the given jobStore as the root job of the workflow.

            Return type
                JobDescription <#toil.job.JobDescription>

            Returns
                the JobDescription describing this job.

            Parameters
                jobStore (AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

        classmethod loadJob(job_store, job_description)
            Retrieves a toil.job.Job <#toil.job.Job> instance from a JobStore

            Parameters

                •   job_store (AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>) -- The job store.

                •   job_description (JobDescription <#toil.job.JobDescription>) -- the JobDescription of the job to retrieve.

            Return type
                Job <#toil.job.Job>

            Returns
                The job referenced by the JobDescription.

        set_debug_flag(flag)
            Enable the given debug option on the job.

            Parameters
                flag (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        has_debug_flag(flag)
            Return true if the given debug flag is set.

            Parameters
                flag (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        files_downloaded_hook(host_and_job_paths=None)
            Function that subclasses can call when they have downloaded their input files.

            Will abort the job if the "download_only" debug flag is set.

            Can be hinted a list of file path pairs outside and inside the job container, in which case the container environment can be reconstructed.

            Parameters
                host_and_job_paths (list </usr/share/doc/python3/html/library/stdtypes.html#list>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]] | None </usr/share/doc/python3/html/library/constants.html#None>)

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

  JobDescription

The class used to store all the information that the Toil Leader ever needs to know about a Job.

    class toil.job.JobDescription(requirements, jobName, unitName='', displayName='', local=None, files=None)
        Stores all the information that the Toil Leader ever needs to know about a Job.

        This includes:

            •   Resource requirements.

            •   Which jobs are children or follow-ons or predecessors of this job.

            •   A reference to the Job object in the job store.

    Can be obtained from an actual (i.e. executable) Job object, and can be used to obtain the Job object from the JobStore.

    Never contains other Jobs or JobDescriptions: all reference is by ID.

    Subclassed into variants for checkpoint jobs and service jobs that have their specific parameters.

        Parameters

            •   requirements (Mapping </usr/share/doc/python3/html/library/typing.html#typing.Mapping>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, int </usr/share/doc/python3/html/library/functions.html#int> | str </usr/share/doc/python3/html/library/stdtypes.html#str> | float </usr/share/doc/python3/html/library/functions.html#float> | bool </usr/share/doc/python3/html/library/functions.html#bool> | list </usr/share/doc/python3/html/library/stdtypes.html#list>])

            •   jobName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   unitName (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>)

            •   displayName (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>)

            •   local (bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None>)

            •   files (set </usr/share/doc/python3/html/library/stdtypes.html#set>[FileID <#toil.fileStores.FileID>] | None </usr/share/doc/python3/html/library/constants.html#None>)

        __init__(requirements, jobName, unitName='', displayName='', local=None, files=None)
            Create a new JobDescription.

            Parameters

                •   requirements (Mapping </usr/share/doc/python3/html/library/typing.html#typing.Mapping>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, int </usr/share/doc/python3/html/library/functions.html#int> | str </usr/share/doc/python3/html/library/stdtypes.html#str> | float </usr/share/doc/python3/html/library/functions.html#float> | bool </usr/share/doc/python3/html/library/functions.html#bool> | list </usr/share/doc/python3/html/library/stdtypes.html#list>]) -- Dict from string to number, string, or bool describing the resource requirements of the job. 'cores', 'memory', 'disk', and 'preemptible' fields, if set, are parsed and broken out into properties. If unset, the relevant property will be unspecified, and will be pulled from the assigned Config object if queried (see toil.job.Requirer.assignConfig() <#toil.job.Requirer.assignConfig>).

                •   jobName (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Name of the kind of job this is. May be used in job store IDs and logging. Also used to let the cluster scaler learn a model for how long the job will take. Ought to be the job class's name if no real user-defined name is available.

                •   unitName (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- Name of this instance of this kind of job. May appear with jobName in logging.

                •   displayName (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- A human-readable name to identify this particular job instance. Ought to be the job class's name if no real user-defined name is available.

                •   local (bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None>) -- If True, the job is meant to use minimal resources but is sensitive to execution latency, and so should be executed by the leader.

                •   files (set </usr/share/doc/python3/html/library/stdtypes.html#set>[FileID <#toil.fileStores.FileID>] | None </usr/share/doc/python3/html/library/constants.html#None>) -- Set of FileID objects that the job plans to use.

            Return type
                None

        get_names()
            Get the names and ID of this job as a named tuple.

            Return type
                Names <#toil.bus.Names>

        get_chain()
            Get all the jobs that executed in this job's chain, in order.

            For each job, produces a named tuple with its various names and its original job store ID. The jobs in the chain are in execution order.

            If the job hasn't run yet or it didn't chain, produces a one-item list.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[Names <#toil.bus.Names>]

        serviceHostIDsInBatches()
            Find all batches of service host job IDs that can be started at the same time.

            (in the order they need to start in)

            Return type
                Iterator </usr/share/doc/python3/html/library/typing.html#typing.Iterator>[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

        successorsAndServiceHosts()
            Get an iterator over all child, follow-on, and service job IDs.

            Return type
                Iterator </usr/share/doc/python3/html/library/typing.html#typing.Iterator>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        allSuccessors()
            Get an iterator over all child, follow-on, and chained, inherited successor job IDs.

            Follow-ons will come before children.

            Return type
                Iterator </usr/share/doc/python3/html/library/typing.html#typing.Iterator>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        successors_by_phase()
            Get an iterator over all child/follow-on/chained inherited successor job IDs, along with their phase number on the stack.

            Phases execute higher numbers to lower numbers.

            Return type
                Iterator </usr/share/doc/python3/html/library/typing.html#typing.Iterator>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[int </usr/share/doc/python3/html/library/functions.html#int>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

        property services
            Get a collection of the IDs of service host jobs for this job, in arbitrary order.

            Will be empty if the job has no unfinished services.

        has_body()
            Returns True if we have a job body associated, and False otherwise.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        attach_body(file_store_id, user_script)
            Attach a job body to this JobDescription.

            Takes the file store ID that the body is stored at, and the required user script module.

            The file store ID can also be "firstJob" for the root job, stored as a shared file instead.

            Parameters

                •   file_store_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   user_script (ModuleDescriptor <#toil.resource.ModuleDescriptor>)

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        detach_body()
            Drop the body reference from a JobDescription.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        get_body()
            Get the information needed to load the job body.

            Return type
                tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, ModuleDescriptor <#toil.resource.ModuleDescriptor>]

            Returns
                a file store ID (or magic shared file name "firstJob") and a user script module.

        Fails if no body is attached; check has_body() first.

        nextSuccessors()
            Return the collection of job IDs for the successors of this job that are ready to run.

            If those jobs have multiple predecessor relationships, they may still be blocked on other jobs.

            Returns None when at the final phase (all successors done), and an empty collection if there are more phases but they can't be entered yet (e.g. because we are waiting for the job itself to run).

            Return type
                set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None </usr/share/doc/python3/html/library/constants.html#None>

        filterSuccessors(predicate)
            Keep only successor jobs for which the given predicate function approves.

            The predicate function is called with the job's ID.

            Treats all other successors as complete and forgets them.

            Parameters
                predicate (Callable </usr/share/doc/python3/html/library/typing.html#typing.Callable>[[str </usr/share/doc/python3/html/library/stdtypes.html#str>], bool </usr/share/doc/python3/html/library/functions.html#bool>])

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        filterServiceHosts(predicate)
            Keep only services for which the given predicate approves.

            The predicate function is called with the service host job's ID.

            Treats all other services as complete and forgets them.

            Parameters
                predicate (Callable </usr/share/doc/python3/html/library/typing.html#typing.Callable>[[str </usr/share/doc/python3/html/library/stdtypes.html#str>], bool </usr/share/doc/python3/html/library/functions.html#bool>])

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        clear_nonexistent_dependents(job_store)
            Remove all references to child, follow-on, and associated service jobs that do not exist.

            That is to say, all those that have been completed and removed.

            Parameters
                job_store (AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        clear_dependents()
            Remove all references to successor and service jobs.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        is_subtree_done()
            Check if the subtree is done.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

            Returns
                True if the job appears to be done, and all related child, follow-on, and service jobs appear to be finished and removed.

        replace(other)
            Take on the ID of another JobDescription, retaining our own state and type.

            When updated in the JobStore, we will save over the other JobDescription.

            Useful for chaining jobs: the chained-to job can replace the parent job.

            Merges cleanup state and successors other than this job from the job being replaced into this one.

            Parameters
                other (JobDescription <#toil.job.JobDescription>) -- Job description to replace.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        assert_is_not_newer_than(other)
            Make sure this JobDescription is not newer than a prospective new version of the JobDescription.

            Parameters
                other (JobDescription <#toil.job.JobDescription>)

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        is_updated_by(other)
            Return True if the passed JobDescription is a distinct, newer version of this one.

            Parameters
                other (JobDescription <#toil.job.JobDescription>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        addChild(childID)
            Make the job with the given ID a child of the described job.

            Parameters
                childID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        addFollowOn(followOnID)
            Make the job with the given ID a follow-on of the described job.

            Parameters
                followOnID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        addServiceHostJob(serviceID, parentServiceID=None)
            Make the ServiceHostJob with the given ID a service of the described job.

            If a parent ServiceHostJob ID is given, that parent service will be started first, and must have already been added.

        hasChild(childID)
            Return True if the job with the given ID is a child of the described job.

            Parameters
                childID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        hasFollowOn(followOnID)
            Test if the job with the given ID is a follow-on of the described job.

            Parameters
                followOnID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        hasServiceHostJob(serviceID)
            Test if the ServiceHostJob is a service of the described job.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        renameReferences(renames)
            Apply the given dict of ID renames to all references to jobs.

            Does not modify our own ID or those of finished predecessors. IDs not present in the renames dict are left as-is.

            Parameters
                renames (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[TemporaryID <#toil.job.TemporaryID>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Rename operations to apply.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        addPredecessor()
            Notify the JobDescription that a predecessor has been added to its Job.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        onRegistration(jobStore)
            Perform setup work that requires the JobStore.

            Called by the Job saving logic when this JobDescription meets the JobStore and has its ID assigned.

            Overridden to perform setup work (like hooking up flag files for service jobs) that requires the JobStore.

            Parameters
                jobStore (AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>) -- The job store we are being placed into

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        setupJobAfterFailure(exit_status=None, exit_reason=None)
            Configure job after a failure.

            Reduce the remainingTryCount if greater than zero and set the memory to be at least as big as the default memory (in case of exhaustion of memory, which is common).

            Requires a configuration to have been assigned (see toil.job.Requirer.assignConfig() <#toil.job.Requirer.assignConfig>).

            Parameters

                •   exit_status (int </usr/share/doc/python3/html/library/functions.html#int> | None </usr/share/doc/python3/html/library/constants.html#None>) -- The exit code from the job.

                •   exit_reason (BatchJobExitReason <#toil.batchSystems.abstractBatchSystem.BatchJobExitReason> | None </usr/share/doc/python3/html/library/constants.html#None>) -- The reason the job stopped, if available from the batch system.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        getLogFileHandle(jobStore)
            Create a context manager that yields a file handle to the log file.

            Assumes logJobStoreFileID is set.

        property remainingTryCount
            Get the number of tries remaining.

            The try count set on the JobDescription, or the default based on the retry count from the config if none is set.

        clearRemainingTryCount()
            Clear remainingTryCount and set it back to its default value.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

            Returns
                True if a modification to the JobDescription was made, and False otherwise.

        reserve_versions(count)
            Reserve a job version number for later, for journaling asynchronously.

            Parameters
                count (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        pre_update_hook()
            Run before pickling and saving a created or updated version of this job.

            Called by the job store.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

JOB.RUNNER API

The Runner contains the methods needed to configure and start a Toil run.

    class Job.Runner
        Used to setup and run Toil workflow.

        static getDefaultArgumentParser(jobstore_as_flag=False)
            Get argument parser with added toil workflow options.

            This is the Right Way to get an argument parser in a Toil Python workflow.

            Parameters
                jobstore_as_flag (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- make the job store option a --jobStore flag instead of a required jobStore positional argument.

            Return type
                ArgumentParser

            Returns
                The argument parser used by a toil workflow with added Toil options.

        static getDefaultOptions(jobStore=None, jobstore_as_flag=False)
            Get default options for a toil workflow.

            Parameters

                •   jobStore (str </usr/share/doc/python3/html/library/stdtypes.html#str> | PathLike </usr/share/doc/python3/html/library/os.html#os.PathLike>[str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None </usr/share/doc/python3/html/library/constants.html#None>) -- A string describing the jobStore for the workflow.

                •   jobstore_as_flag (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- make the job store option a --jobStore flag instead of a required jobStore positional argument.

            Return type
                Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>

            Returns
                The options used by a toil workflow.

        static addToilOptions(parser, jobstore_as_flag=False)
            Adds the default toil options to an optparse </usr/share/doc/python3/html/library/optparse.html#module-optparse> or argparse </usr/share/doc/python3/html/library/argparse.html#module-argparse> parser object.

            Consider using getDefaultArgumentParser() <#toil.job.Job.Runner.getDefaultArgumentParser> instead, which will produce a parser of the correct class to use Toil's config file and environment variables. If ther parser passed here is just an argparse.ArgumentParser </usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser> and not a configargparse.ArgParser, the Toil config file and environment variables will not be respected.

            Parameters

                •   parser (OptionParser </usr/share/doc/python3/html/library/optparse.html#optparse.OptionParser> | ArgumentParser </usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser>) -- Options object to add toil options to.

                •   jobstore_as_flag (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- make the job store option a --jobStore flag instead of a required jobStore positional argument.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        static startToil(job, options)
            Run the toil workflow using the given options.

            Deprecated by toil.common.Toil.start.

            (see Job.Runner.getDefaultOptions and Job.Runner.addToilOptions) starting with this job. :type job: Job <#toil.job.Job> :param job: root job of the workflow :raises: toil.exceptions.FailedJobsException if at the end of function there remain failed jobs. :rtype: Any </usr/share/doc/python3/html/library/typing.html#typing.Any> :return: The return value of the root job's run function.

            Parameters
                job (Job <#toil.job.Job>)

            Return type
                Any </usr/share/doc/python3/html/library/typing.html#typing.Any>

JOB.FILESTORE API

The AbstractFileStore is an abstraction of a Toil run's shared storage.

    class toil.fileStores.abstractFileStore.AbstractFileStore(jobStore, jobDesc, file_store_dir, waitForPreviousCommit)
        Interface used to allow user code run by Toil to read and write files.

        Also provides the interface to other Toil facilities used by user code, including:

                •   normal (non-real-time) logging

                •   finding the correct temporary directory for scratch work

                •   importing and exporting files into and out of the workflow

    Stores user files in the jobStore, but keeps them separate from actual jobs.

    May implement caching.

    Passed as argument to the toil.job.Job.run() <#toil.job.Job.run> method.

    Access to files is only permitted inside the context manager provided by toil.fileStores.abstractFileStore.AbstractFileStore.open() <#toil.fileStores.abstractFileStore.AbstractFileStore.open>.

    Also responsible for committing completed jobs back to the job store with an update operation, and allowing that commit operation to be waited for.

        Parameters

            •   jobStore (AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

            •   jobDesc (JobDescription <#toil.job.JobDescription>)

            •   file_store_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   waitForPreviousCommit (Callable </usr/share/doc/python3/html/library/typing.html#typing.Callable>[[], Any </usr/share/doc/python3/html/library/typing.html#typing.Any>])

        __init__(jobStore, jobDesc, file_store_dir, waitForPreviousCommit)
            Create a new file store object.

            Parameters

                •   jobStore (AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>) -- the job store in use for the current Toil run.

                •   jobDesc (JobDescription <#toil.job.JobDescription>) -- the JobDescription object for the currently running job.

                •   file_store_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the per-worker local temporary directory where the file store should store local files. Per-job directories will be created under here by the file store.

                •   waitForPreviousCommit (Callable </usr/share/doc/python3/html/library/typing.html#typing.Callable>[[], Any </usr/share/doc/python3/html/library/typing.html#typing.Any>]) -- the waitForCommit method of the previous job's file store, when jobs are running in sequence on the same worker. Used to prevent this file store's startCommit and the previous job's startCommit methods from running at the same time and racing. If they did race, it might be possible for the later job to be fully marked as completed in the job store before the eralier job was.

            Return type
                None

        static createFileStore(jobStore, jobDesc, file_store_dir, waitForPreviousCommit, caching)
            Create a concreate FileStore.

            Parameters

                •   jobStore (AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

                •   jobDesc (JobDescription <#toil.job.JobDescription>)

                •   file_store_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   waitForPreviousCommit (Callable </usr/share/doc/python3/html/library/typing.html#typing.Callable>[[], Any </usr/share/doc/python3/html/library/typing.html#typing.Any>])

                •   caching (bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None>)

            Return type
                NonCachingFileStore <#toil.fileStores.nonCachingFileStore.NonCachingFileStore> | CachingFileStore <#toil.fileStores.cachingFileStore.CachingFileStore>

        static shutdownFileStore(workflowID, config_work_dir, config_coordination_dir)
            Carry out any necessary filestore-specific cleanup.

            This is a destructive operation and it is important to ensure that there are no other running processes on the system that are modifying or using the file store for this workflow.

            This is the intended to be the last call to the file store in a Toil run, called by the batch system cleanup function upon batch system shutdown.

            Parameters

                •   workflowID (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The workflow ID for this invocation of the workflow

                •   config_work_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- The path to the work directory in the Toil Config.

                •   config_coordination_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- The path to the coordination directory in the Toil Config.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        open(job)
            Create the context manager around tasks prior and after a job has been run.

            File operations are only permitted inside the context manager.

            Implementations must only yield from within with super().open(job):.

            Parameters
                job (Job <#toil.job.Job>) -- The job instance of the toil job to run.

            Return type
                Generator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator>[None </usr/share/doc/python3/html/library/constants.html#None>, None </usr/share/doc/python3/html/library/constants.html#None>, None </usr/share/doc/python3/html/library/constants.html#None>]

        get_disk_usage()
            Get the number of bytes of disk used by the last job run under open().

            Disk usage is measured at the end of the job. TODO: Sample periodically and record peak usage.

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int> | None </usr/share/doc/python3/html/library/constants.html#None>

        getLocalTempDir()
            Get a new local temporary directory in which to write files.

            The directory will only persist for the duration of the job.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

            Returns
                The absolute path to a new local temporary directory. This directory will exist for the duration of the job only, and is guaranteed to be deleted once the job terminates, removing all files it contains recursively.

        getLocalTempFile(suffix=None, prefix=None)
            Get a new local temporary file that will persist for the duration of the job.

            Parameters

                •   suffix (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- If not None, the file name will end with this string. Otherwise, default value ".tmp" will be used

                •   prefix (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- If not None, the file name will start with this string. Otherwise, default value "tmp" will be used

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

            Returns
                The absolute path to a local temporary file. This file will exist for the duration of the job only, and is guaranteed to be deleted once the job terminates.

        getLocalTempFileName(suffix=None, prefix=None)
            Get a valid name for a new local file. Don't actually create a file at the path.

            Parameters

                •   suffix (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- If not None, the file name will end with this string. Otherwise, default value ".tmp" will be used

                •   prefix (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- If not None, the file name will start with this string. Otherwise, default value "tmp" will be used

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

            Returns
                Path to valid file

        abstractmethod writeGlobalFile(localFileName, cleanup=False)
            Upload a file (as a path) to the job store.

            If the file is in a FileStore-managed temporary directory (i.e. from toil.fileStores.abstractFileStore.AbstractFileStore.getLocalTempDir() <#toil.fileStores.abstractFileStore.AbstractFileStore.getLocalTempDir>), it will become a local copy of the file, eligible for deletion by toil.fileStores.abstractFileStore.AbstractFileStore.deleteLocalFile() <#toil.fileStores.abstractFileStore.AbstractFileStore.deleteLocalFile>.

            If an executable file on the local filesystem is uploaded, its executability will be preserved when it is downloaded again.

            Parameters

                •   localFileName (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The path to the local file to upload. The last path component (basename of the file) will remain associated with the file in the file store, if supported by the backing JobStore, so that the file can be searched for by name or name glob.

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- if True then the copy of the global file will be deleted once the job and all its successors have completed running. If not the global file must be deleted manually.

            Return type
                FileID <#toil.fileStores.FileID>

            Returns
                an ID that can be used to retrieve the file.

        writeGlobalFileStream(cleanup=False, basename=None, encoding=None, errors=None)
            Similar to writeGlobalFile, but allows the writing of a stream to the job store. The yielded file handle does not need to and should not be closed explicitly.

            Parameters

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- The name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- Specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- is as in toil.fileStores.abstractFileStore.AbstractFileStore.writeGlobalFile() <#toil.fileStores.abstractFileStore.AbstractFileStore.writeGlobalFile>.

                •   basename (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- If supported by the backing JobStore, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected.

            Return type
                Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[WriteWatchingStream <#toil.lib.io.WriteWatchingStream>, FileID <#toil.fileStores.FileID>]]

            Returns
                A context manager yielding a tuple of 1) a file handle which can be written to and 2) the toil.fileStores.FileID of the resulting file in the job store.

        logAccess(fileStoreID, destination=None)
            Record that the given file was read by the job.

            (to be announced if the job fails)

            If destination is not None, it gives the path that the file was downloaded to. Otherwise, assumes that the file was streamed.

            Must be called by readGlobalFile() <#toil.fileStores.abstractFileStore.AbstractFileStore.readGlobalFile> and readGlobalFileStream() <#toil.fileStores.abstractFileStore.AbstractFileStore.readGlobalFileStream> implementations.

            Parameters

                •   fileStoreID (FileID <#toil.fileStores.FileID> | str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   destination (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>)

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        abstractmethod readGlobalFile(fileStoreID, userPath=None, cache=True, mutable=False, symlink=False)
            Make the file associated with fileStoreID available locally.

            If mutable is True, then a copy of the file will be created locally so that the original is not modified and does not change the file for other jobs. If mutable is False, then a link can be created to the file, saving disk resources. The file that is downloaded will be executable if and only if it was originally uploaded from an executable file on the local filesystem.

            If a user path is specified, it is used as the destination. If a user path isn't specified, the file is stored in the local temp directory with an encoded name.

            The destination file must not be deleted by the user; it can only be deleted through deleteLocalFile.

            Implementations must call logAccess() <#toil.fileStores.abstractFileStore.AbstractFileStore.logAccess> to report the download.

            Parameters

                •   fileStoreID (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- job store id for the file

                •   userPath (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- a path to the name of file to which the global file will be copied or hard-linked (see below).

                •   cache (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Described in toil.fileStores.CachingFileStore.readGlobalFile()

                •   mutable (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Described in toil.fileStores.CachingFileStore.readGlobalFile()

                •   symlink (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- True if caller can accept symlink, False if caller can only accept a normal file or hardlink

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

            Returns
                An absolute path to a local, temporary copy of the file keyed by fileStoreID.

        abstractmethod readGlobalFileStream(fileStoreID, encoding=None, errors=None)
            Read a stream from the job store; similar to readGlobalFile.

            The yielded file handle does not need to and should not be closed explicitly.

            Parameters

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

                •   fileStoreID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                ContextManager </usr/share/doc/python3/html/library/typing.html#typing.ContextManager>[IO </usr/share/doc/python3/html/library/typing.html#typing.IO>[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>] | IO </usr/share/doc/python3/html/library/typing.html#typing.IO>[str </usr/share/doc/python3/html/library/stdtypes.html#str>], bool </usr/share/doc/python3/html/library/functions.html#bool> | None]

        Implementations must call logAccess() <#toil.fileStores.abstractFileStore.AbstractFileStore.logAccess> to report the download.

            Return type
                ContextManager </usr/share/doc/python3/html/library/typing.html#typing.ContextManager>[IO </usr/share/doc/python3/html/library/typing.html#typing.IO>[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>] | IO </usr/share/doc/python3/html/library/typing.html#typing.IO>[str </usr/share/doc/python3/html/library/stdtypes.html#str>], bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None>]

            Returns
                a context manager yielding a file handle which can be read from.

            Parameters

                •   fileStoreID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None)

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None)

        getGlobalFileSize(fileStoreID)
            Get the size of the file pointed to by the given ID, in bytes.

            If a FileID or something else with a non-None 'size' field, gets that.

            Otherwise, asks the job store to poll the file's size.

            Note that the job store may overestimate the file's size, for example if it is encrypted and had to be augmented with an IV or other encryption framing.

            Parameters
                fileStoreID (FileID <#toil.fileStores.FileID> | str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- File ID for the file

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

            Returns
                File's size in bytes, as stored in the job store

        abstractmethod deleteLocalFile(fileStoreID)
            Delete local copies of files associated with the provided job store ID.

            Raises an OSError with an errno of errno.ENOENT if no such local copies exist. Thus, cannot be called multiple times in succession.

            The files deleted are all those previously read from this file ID via readGlobalFile by the current job into the job's file-store-provided temp directory, plus the file that was written to create the given file ID, if it was written by the current job from the job's file-store-provided temp directory.

            Parameters
                fileStoreID (FileID <#toil.fileStores.FileID> | str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- File Store ID of the file to be deleted.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        abstractmethod deleteGlobalFile(fileStoreID)
            Delete local files and then permanently deletes them from the job store.

            To ensure that the job can be restarted if necessary, the delete will not happen until after the job's run method has completed.

            Parameters
                fileStoreID (FileID <#toil.fileStores.FileID> | str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the File Store ID of the file to be deleted.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        log_to_leader(text, level=20)
            Send a logging message to the leader. The message will also be logged by the worker at the same level.

            Does not depend on the commit system, so this is safe to use during an ansynchronous commit, or without a commit afterward.

            Parameters

                •   text (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The string to log.

                •   level (int </usr/share/doc/python3/html/library/functions.html#int>) -- The logging level.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        log_user_stream(name, stream)
            Send a stream of UTF-8 text to the leader as a named log stream.

            Useful for things like the error logs of Docker containers. The leader will show it to the user or organize it appropriately for user-level log information.

            Parameters

                •   name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- A hierarchical, .-delimited string.

                •   stream (IO </usr/share/doc/python3/html/library/typing.html#typing.IO>[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]) -- A stream of encoded text. Encoding errors will be tolerated.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        abstractmethod startCommit(jobState=False)
            Update the status of the job on the disk.

            May bump the version number of the job.

            May start an asynchronous process. Call waitForCommit() to wait on that process. You must waitForCommit() before committing any further updates to the job. During the asynchronous process, it is safe to modify the job; modifications after this call will not be committed until the next call.

            Parameters
                jobState (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If True, commit the state of the FileStore's job, and file deletes. Otherwise, commit only file creates/updates.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        abstractmethod waitForCommit()
            Blocks while startCommit is running.

            This function is called by this job's successor to ensure that it does not begin modifying the job store until after this job has finished doing so.

            Might be called when startCommit is never called on a particular instance, in which case it does not block.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

            Returns
                Always returns True

        abstractmethod classmethod shutdown(shutdown_info)
            Shutdown the filestore on this node.

            This is intended to be called on batch system shutdown.

            Parameters
                shutdown_info (Any </usr/share/doc/python3/html/library/typing.html#typing.Any>) -- The implementation-specific shutdown information, for shutting down the file store and removing all its state and all job local temp directories from the node.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

    class toil.fileStores.FileID(fileStoreID, size, executable=False)
        A small wrapper around Python's builtin string class.

        It is used to represent a file's ID in the file store, and has a size attribute that is the file's size in bytes. This object is returned by importFile and writeGlobalFile.

        Calls into the file store can use bare strings; size will be queried from the job store if unavailable in the ID.

        Parameters

            •   fileStoreID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   size (int </usr/share/doc/python3/html/library/functions.html#int>)

            •   executable (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            •   args (Any </usr/share/doc/python3/html/library/typing.html#typing.Any>)

            •   kwargs (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any </usr/share/doc/python3/html/library/typing.html#typing.Any>])

        Return type
            FileID <#toil.fileStores.FileID>

        __init__(fileStoreID, size, executable=False)

            Parameters

                •   fileStoreID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   size (int </usr/share/doc/python3/html/library/functions.html#int>)

                •   executable (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        pack()
            Pack the FileID into a string so it can be passed through external code.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        classmethod unpack(packedFileStoreID)
            Unpack the result of pack() into a FileID object.

            Parameters
                packedFileStoreID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                FileID <#toil.fileStores.FileID>

BATCH SYSTEM API

The batch system interface is used by Toil to abstract over different ways of running batches of jobs, for example on Slurm clusters, Kubernetes clusters, or a single node. The toil.batchSystems.abstractBatchSystem.AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem> API is implemented to run jobs using a given job management system.

  Batch System Environment Variables

Environmental variables allow passing of scheduler specific parameters.

For SLURM there are two environment variables - the first applies to all jobs, while the second defined the partition to use for parallel jobs:

            export TOIL_SLURM_ARGS="-t 1:00:00 -q fatq"
            export TOIL_SLURM_PE='multicore'

For TORQUE there are two environment variables - one for everything but the resource requirements, and another - for resources requirements (without the -l prefix):

            export TOIL_TORQUE_ARGS="-q fatq"
            export TOIL_TORQUE_REQS="walltime=1:00:00"

For GridEngine (SGE, UGE), there is an additional environmental variable to define the parallel environment <http://www.softpanorama.org/HPC/Grid_engine/parallel_environment.shtml#Important_details> for running multicore jobs:

            export TOIL_GRIDENGINE_PE='smp'
            export TOIL_GRIDENGINE_ARGS='-q batch.q'

For HTCondor, additional parameters can be included in the submit file passed to condor_submit:

            export TOIL_HTCONDOR_PARAMS='requirements = TARGET.has_sse4_2 == true; accounting_group = test'

The environment variable is parsed as a semicolon-separated string of parameter = value pairs.

  Batch System API

    class toil.batchSystems.abstractBatchSystem.AbstractBatchSystem
        An abstract base class to represent the interface the batch system must provide to Toil.

        abstractmethod classmethod supportsAutoDeployment()
            Whether this batch system supports auto-deployment of the user script itself.

            If it does, the setUserScript() <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem.setUserScript> can be invoked to set the resource object representing the user script.

            Note to implementors: If your implementation returns True here, it should also override

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        abstractmethod classmethod supportsWorkerCleanup()
            Whether this batch system supports worker cleanup.

            Indicates whether this batch system invokes BatchSystemSupport.workerCleanup() <#toil.batchSystems.abstractBatchSystem.BatchSystemSupport.workerCleanup> after the last job for a particular workflow invocation finishes. Note that the term worker refers to an entire node, not just a worker process. A worker process may run more than one job sequentially, and more than one concurrent worker process may exist on a worker node, for the same workflow. The batch system is said to shut down after the last worker process terminates.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        setUserScript(userScript)
            Set the user script for this workflow.

            This method must be called before the first job is issued to this batch system, and only if supportsAutoDeployment() <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem.supportsAutoDeployment> returns True, otherwise it will raise an exception.

            Parameters
                userScript (Resource <#toil.resource.Resource>) -- the resource object representing the user script or module and the modules it depends on.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        set_message_bus(message_bus)
            Give the batch system an opportunity to connect directly to the message bus, so that it can send informational messages about the jobs it is running to other Toil components.

            Parameters
                message_bus (MessageBus <#toil.bus.MessageBus>)

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        abstractmethod issueBatchJob(command, job_desc, job_environment=None)
            Issues a job with the specified command to the batch system and returns a unique job ID number.

            Parameters

                •   command (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the command to execute somewhere to run the Toil worker process

                •   job_desc (JobDescription <#toil.job.JobDescription>) -- the JobDescription for the job being run

                •   job_environment (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None </usr/share/doc/python3/html/library/constants.html#None>) -- a collection of job-specific environment variables to be set on the worker.

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

            Returns
                a unique job ID number that can be used to reference the newly issued job

        abstractmethod killBatchJobs(jobIDs)
            Kills the given job IDs. After returning, the killed jobs will not appear in the results of getRunningBatchJobIDs. The killed job will not be returned from getUpdatedBatchJob.

            Parameters
                jobIDs (list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>]) -- list of IDs of jobs to kill

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        abstractmethod getIssuedBatchJobIDs()
            Gets all currently issued jobs

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>]

            Returns
                A list of jobs (as job ID numbers) currently issued (may be running, or may be waiting to be run). Despite the result being a list, the ordering should not be depended upon.

        abstractmethod getRunningBatchJobIDs()
            Gets a map of jobs as job ID numbers that are currently running (not just waiting) and how long they have been running, in seconds.

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>]

            Returns
                dictionary with currently running job ID number keys and how many seconds they have been running as the value

        abstractmethod getUpdatedBatchJob(maxWait)
            Returns information about job that has updated its status (i.e. ceased running, either successfully or with an error). Each such job will be returned exactly once.

            Does not return info for jobs killed by killBatchJobs, although they may cause None to be returned earlier than maxWait.

            Parameters
                maxWait (int </usr/share/doc/python3/html/library/functions.html#int>) -- the number of seconds to block, waiting for a result

            Return type
                UpdatedBatchJobInfo <#toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo> | None </usr/share/doc/python3/html/library/constants.html#None>

            Returns
                If a result is available, returns UpdatedBatchJobInfo. Otherwise it returns None. wallTime is the number of seconds (a strictly positive float) in wall-clock time the job ran for, or None if this batch system does not support tracking wall time.

        getSchedulingStatusMessage()
            Get a log message fragment for the user about anything that might be going wrong in the batch system, if available.

            If no useful message is available, return None.

            This can be used to report what resource is the limiting factor when scheduling jobs, for example. If the leader thinks the workflow is stuck, the message can be displayed to the user to help them diagnose why it might be stuck.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

            Returns
                User-directed message about scheduling state.

        abstractmethod shutdown()
            Called at the completion of a toil invocation. Should cleanly terminate all worker threads.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        setEnv(name, value=None)
            Set an environment variable for the worker process before it is launched.

            The worker process will typically inherit the environment of the machine it is running on but this method makes it possible to override specific variables in that inherited environment before the worker is launched. Note that this mechanism is different to the one used by the worker internally to set up the environment of a job. A call to this method affects all jobs issued after this method returns. Note to implementors: This means that you would typically need to copy the variables before enqueuing a job.

            If no value is provided it will be looked up from the current environment.

            Parameters

                •   name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   value (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>)

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        classmethod add_options(parser)
            If this batch system provides any command line options, add them to the given parser.

            Parameters
                parser (ArgumentParser </usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser> | _ArgumentGroup)

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        classmethod setOptions(setOption)
            Process command line or configuration options relevant to this batch system.

            Parameters
                setOption (OptionSetter <#toil.batchSystems.options.OptionSetter>) -- A function with signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) returning nothing, used to update run configuration as a side effect.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        getWorkerContexts()
            Get a list of picklable context manager objects to wrap worker work in, in order.

            Can be used to ask the Toil worker to do things in-process (such as configuring environment variables, hot-deploying user scripts, or cleaning up a node) that would otherwise require a wrapping "executor" process.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[ContextManager </usr/share/doc/python3/html/library/typing.html#typing.ContextManager>[Any </usr/share/doc/python3/html/library/typing.html#typing.Any>, bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None>]]

JOB.SERVICE API

The Service class allows databases and servers to be spawned within a Toil workflow.

    class Job.Service(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, unitName='')
        Abstract class used to define the interface to a service.

        Should be subclassed by the user to define services.

        Is not executed as a job; runs within a ServiceHostJob.

        Parameters

            •   memory (str </usr/share/doc/python3/html/library/stdtypes.html#str> | int </usr/share/doc/python3/html/library/functions.html#int> | None </usr/share/doc/python3/html/library/constants.html#None>)

            •   cores (str </usr/share/doc/python3/html/library/stdtypes.html#str> | int </usr/share/doc/python3/html/library/functions.html#int> | float </usr/share/doc/python3/html/library/functions.html#float> | None </usr/share/doc/python3/html/library/constants.html#None>)

            •   disk (str </usr/share/doc/python3/html/library/stdtypes.html#str> | int </usr/share/doc/python3/html/library/functions.html#int> | None </usr/share/doc/python3/html/library/constants.html#None>)

            •   accelerators (str </usr/share/doc/python3/html/library/stdtypes.html#str> | int </usr/share/doc/python3/html/library/functions.html#int> | Mapping </usr/share/doc/python3/html/library/typing.html#typing.Mapping>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any </usr/share/doc/python3/html/library/typing.html#typing.Any>] | AcceleratorRequirement <#toil.job.AcceleratorRequirement> | Sequence </usr/share/doc/python3/html/library/typing.html#typing.Sequence>[str </usr/share/doc/python3/html/library/stdtypes.html#str> | int </usr/share/doc/python3/html/library/functions.html#int> | Mapping </usr/share/doc/python3/html/library/typing.html#typing.Mapping>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any </usr/share/doc/python3/html/library/typing.html#typing.Any>] | AcceleratorRequirement <#toil.job.AcceleratorRequirement>] | None </usr/share/doc/python3/html/library/constants.html#None>)

            •   preemptible (str </usr/share/doc/python3/html/library/stdtypes.html#str> | int </usr/share/doc/python3/html/library/functions.html#int> | bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None>)

            •   unitName (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>)

        __init__(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, unitName='')
            Memory, core and disk requirements are specified identically to as in toil.job.Job.__init__().

            Parameters

                •   memory (str </usr/share/doc/python3/html/library/stdtypes.html#str> | int </usr/share/doc/python3/html/library/functions.html#int> | None </usr/share/doc/python3/html/library/constants.html#None>)

                •   cores (str </usr/share/doc/python3/html/library/stdtypes.html#str> | int </usr/share/doc/python3/html/library/functions.html#int> | float </usr/share/doc/python3/html/library/functions.html#float> | None </usr/share/doc/python3/html/library/constants.html#None>)

                •   disk (str </usr/share/doc/python3/html/library/stdtypes.html#str> | int </usr/share/doc/python3/html/library/functions.html#int> | None </usr/share/doc/python3/html/library/constants.html#None>)

                •   accelerators (str </usr/share/doc/python3/html/library/stdtypes.html#str> | int </usr/share/doc/python3/html/library/functions.html#int> | Mapping </usr/share/doc/python3/html/library/typing.html#typing.Mapping>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any </usr/share/doc/python3/html/library/typing.html#typing.Any>] | AcceleratorRequirement <#toil.job.AcceleratorRequirement> | Sequence </usr/share/doc/python3/html/library/typing.html#typing.Sequence>[str </usr/share/doc/python3/html/library/stdtypes.html#str> | int </usr/share/doc/python3/html/library/functions.html#int> | Mapping </usr/share/doc/python3/html/library/typing.html#typing.Mapping>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any </usr/share/doc/python3/html/library/typing.html#typing.Any>] | AcceleratorRequirement <#toil.job.AcceleratorRequirement>] | None </usr/share/doc/python3/html/library/constants.html#None>)

                •   preemptible (str </usr/share/doc/python3/html/library/stdtypes.html#str> | int </usr/share/doc/python3/html/library/functions.html#int> | bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None>)

                •   unitName (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>)

            Return type
                None

        abstractmethod start(job)
            Start the service.

            Parameters
                job (ServiceHostJob <#toil.job.ServiceHostJob>) -- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files.

            Return type
                Any </usr/share/doc/python3/html/library/typing.html#typing.Any>

            Returns
                An object describing how to access the service. The object must be pickleable and will be used by jobs to access the service (see toil.job.Job.addService() <#toil.job.Job.addService>).

        abstractmethod stop(job)
            Stops the service. Function can block until complete.

            Parameters
                job (ServiceHostJob <#toil.job.ServiceHostJob>) -- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files.

            Return type
                None </usr/share/doc/python3/html/library/constants.html#None>

        check()
            Checks the service is still running.

            Raises
                exceptions.RuntimeError -- If the service failed, this will cause the service job to be labeled failed.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

            Returns
                True if the service is still running, else False. If False then the service job will be terminated, and considered a success. Important point: if the service job exits due to a failure, it should raise a RuntimeError, not return False!

EXCEPTIONS API

Toil specific exceptions.

    exception toil.job.JobException(message)
        General job exception.

        Parameters
            message (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            None

        __init__(message)

            Parameters
                message (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

    exception toil.job.JobGraphDeadlockException(string)
        An exception raised in the event that a workflow contains an unresolvable dependency, such as a cycle. See toil.job.Job.checkJobGraphForDeadlocks() <#toil.job.Job.checkJobGraphForDeadlocks>.

        __init__(string)

    exception toil.jobStores.abstractJobStore.ConcurrentFileModificationException(jobStoreFileID)
        Indicates that the file was attempted to be modified by multiple processes at once.

        Parameters
            jobStoreFileID (FileID <#toil.fileStores.FileID> | str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        __init__(jobStoreFileID)

            Parameters
                jobStoreFileID (FileID <#toil.fileStores.FileID> | str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the ID of the file that was modified by multiple workers or processes concurrently

    exception toil.jobStores.abstractJobStore.JobStoreExistsException(locator, prefix)
        Indicates that the specified job store already exists.

        Parameters

            •   locator (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   prefix (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        __init__(locator, prefix)

            Parameters

                •   locator (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The location of the job store

                •   locator

                •   prefix (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

    exception toil.jobStores.abstractJobStore.NoSuchFileException(jobStoreFileID, customName=None, *extra)
        Indicates that the specified file does not exist.

        Parameters

            •   jobStoreFileID (FileID <#toil.fileStores.FileID> | str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   customName (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>)

            •   extra (Any </usr/share/doc/python3/html/library/typing.html#typing.Any>)

        __init__(jobStoreFileID, customName=None, *extra)

            Parameters

                •   jobStoreFileID (FileID <#toil.fileStores.FileID> | str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the ID of the file that was mistakenly assumed to exist

                •   customName (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>) -- optionally, an alternate name for the nonexistent file

                •   extra (Any </usr/share/doc/python3/html/library/typing.html#typing.Any>) -- optional extra information to add to the error message

                •   extra

    exception toil.jobStores.abstractJobStore.NoSuchJobException(jobStoreID)
        Indicates that the specified job does not exist.

        Parameters
            jobStoreID (FileID <#toil.fileStores.FileID> | str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        __init__(jobStoreID)

            Parameters

                •   jobStoreID (FileID <#toil.fileStores.FileID> | str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the jobStoreID that was mistakenly assumed to exist

                •   jobStoreID

    exception toil.jobStores.abstractJobStore.NoSuchJobStoreException(locator, prefix)
        Indicates that the specified job store does not exist.

        Parameters

            •   locator (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   prefix (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        __init__(locator, prefix)

            Parameters

                •   locator (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The location of the job store

                •   prefix (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The type of job store

                •   locator

                •   prefix

RUNNING TESTS

Test make targets, invoked as $ make <target>, subject to which environment variables are set (see Running Integration Tests).
TARGET | DESCRIPTION
test | Invokes all tests.
integration_test | Invokes only the integration tests.
test_offline | Skips building the Docker appliance and only invokes tests that have no docker dependencies.
integration_test_local | Makes integration tests easier to debug locally by running the integration tests serially and doesn't redirect output. This makes it appears on the terminal as expected.

Before running tests for the first time, initialize your virtual environment following the steps in Installing Plugins <#buildfromsource>.

Run all tests (including slow tests):

            $ make test

Run only quick tests (as of Jul 25, 2018, this was ~ 20 minutes):

            $ export TOIL_TEST_QUICK=True; make test

Run an individual test with:

            $ make test tests=src/toil/test/sort/sortTest.py::SortTest::testSort

The default value for tests is "src" which includes all tests in the src/ subdirectory of the project root. Tests that require a particular feature will be skipped implicitly. If you want to explicitly skip tests that depend on a currently installed feature, use

            $ make test tests="-m 'not aws' src"

This will run only the tests that don't depend on the aws extra, even if that extra is currently installed. Note the distinction between the terms feature and extra. Every extra is a feature but there are features that are not extras, such as the gridengine feature. To skip tests involving both the gridengine feature and the aws extra, use the following:

            $ make test tests="-m 'not aws and not gridengine' src"

  Running Tests with pytest

Often it is simpler to use pytest directly, instead of calling the make wrapper. This usually works as expected, but some tests need some manual preparation. To run a specific test with pytest, use the following:

            python3 -m pytest src/toil/test/sort/sortTest.py::SortTest::testSort

For more information, see the pytest documentation <https://docs.pytest.org/en/latest/>.

  Running Integration Tests

These tests are generally only run using in our CI workflow due to their resource requirements and cost. However, they can be made available for local testing:

            •   Running tests that make use of Docker (e.g. autoscaling tests and Docker tests) require an appliance image to be hosted. First, make sure you have gone through the set up found in Using Docker with Quay. Then to build and host the appliance image run the make target push_docker.

                        $ make push_docker

            •   Running integration tests require activation via an environment variable as well as exporting information relevant to the desired tests. Enable the integration tests:

                        $ export TOIL_TEST_INTEGRATIVE=True

            •   Finally, set the environment variables for keyname and desired zone:

                        $ export TOIL_X_KEYNAME=[Your Keyname]
                        $ export TOIL_X_ZONE=[Desired Zone]

            Where X is one of our currently supported cloud providers (GCE, AWS).

            •   See the above sections for guidance on running tests.

  Test Environment Variables
TOIL_TEST_TEMP | An absolute path to a directory where Toil tests will write their temporary files. Defaults to the system's standard temporary directory <https://docs.python.org/2/library/tempfile.html#tempfile.gettempdir>.
TOIL_TEST_INTEGRATIVE | If True, this allows the integration tests to run. Only valid when running the tests from the source directory via make test or make test_parallel.
TOIL_AWS_KEYNAME | An AWS keyname (see Preparing your AWS environment <#prepareaws>), which is required to run the AWS tests.
TOIL_GOOGLE_PROJECTID | A Google Cloud account projectID (see Running in Google Compute Engine (GCE) <#runninggce>), which is required to to run the Google Cloud tests.
TOIL_TEST_QUICK | If True, long running tests are skipped.

        Partial install and failing tests

            Some tests may fail with an ImportError if the required extras are not installed. Install Toil with all of the extras do prevent such errors.

  Using Docker with Quay

Docker <https://www.docker.com/products/docker> is needed for some of the tests. Follow the appropriate installation instructions for your system on their website to get started.

When running make test you might still get the following error:

            $ make test
            Please set TOIL_DOCKER_REGISTRY, e.g. to quay.io/USER.

To solve, make an account with Quay <https://quay.io/> and specify it like so:

            $ TOIL_DOCKER_REGISTRY=quay.io/USER make test

where USER is your Quay username.

For convenience you may want to add this variable to your bashrc by running

            $ echo 'export TOIL_DOCKER_REGISTRY=quay.io/USER' >> $HOME/.bashrc

  Running Mesos Tests

If you're running Toil's Mesos tests, be sure to create the virtualenv with --system-site-packages to include the Mesos Python bindings. Verify this by activating the virtualenv and running pip list | grep mesos. On macOS, this may come up empty. To fix it, run the following:

            for i in /usr/local/lib/python2.7/site-packages/*mesos*; do ln -snf $i venv/lib/python2.7/site-packages/; done

DEVELOPING WITH DOCKER

To develop on features reliant on the Toil Appliance (the docker image toil uses for AWS autoscaling), you should consider setting up a personal registry on Quay <https://quay.io/> or Docker Hub <https://hub.docker.com/>. Because the Toil Appliance images are tagged with the Git commit they are based on and because only commits on our master branch trigger an appliance build on Quay, as soon as a developer makes a commit or dirties the working copy they will no longer be able to rely on Toil to automatically detect the proper Toil Appliance image. Instead, developers wishing to test any appliance changes in autoscaling should build and push their own appliance image to a personal Docker registry. This is described in the next section.

  Making Your Own Toil Docker Image

Note! Toil checks if the docker image specified by TOIL_APPLIANCE_SELF exists prior to launching by using the docker v2 schema. This should be valid for any major docker repository, but there is an option to override this if desired using the option: --forceDockerAppliance.

Here is a general workflow (similar instructions apply when using Docker Hub):

    1.  Make some changes to the provisioner of your local version of Toil

    2.  Go to the location where you installed the Toil source code and run

                $ make docker

    to automatically build a docker image that can now be uploaded to your personal Quay <https://quay.io/> account. On Docker Desktop, containerd may have to be enabled <https://docs.docker.com/desktop/containerd/#enable-the-containerd-image-store>. If you have not installed Toil source code yet see Installing Plugins <#buildfromsource>.

    3.  If it's not already you will need Docker installed and need to log into Quay <https://docs.quay.io/solution/getting-started.html>. Also you will want to make sure that your Quay account is public.

    4.  Set the environment variable TOIL_DOCKER_REGISTRY to your Quay account. If you find yourself doing this often you may want to add

                export TOIL_DOCKER_REGISTRY=quay.io/<MY_QUAY_USERNAME>

    to your .bashrc or equivalent.

    5.  Now you can run

                $ make push_docker

    which will upload the docker image to your Quay account. Take note of the image's tag for the next step.

    6.  Finally you will need to tell Toil from where to pull the Appliance image you've created (it uses the Toil release you have installed by default). To do this set the environment variable TOIL_APPLIANCE_SELF to the url of your image. For more info see Environment Variables <#envars>.

    7.  Now you can launch your cluster! For more information see Running a Workflow with Autoscaling <#autoscaling>.

  Running a Cluster Locally

The Toil Appliance container can also be useful as a test environment since it can simulate a Toil cluster locally. An important caveat for this is autoscaling, since autoscaling will only work on an EC2 instance and cannot (at this time) be run on a local machine.

To spin up a local cluster, start by using the following Docker run command to launch a Toil leader container:

            docker run \
                --entrypoint=mesos-master \
                --net=host \
                -d \
                --name=leader \
                --volume=/home/jobStoreParentDir:/jobStoreParentDir \
                quay.io/ucsc_cgl/toil:3.6.0 \
                --registry=in_memory \
                --ip=127.0.0.1 \
                --port=5050 \
                --allocation_interval=500ms

A couple notes on this command: the -d flag tells Docker to run in daemon mode so the container will run in the background. To verify that the container is running you can run docker ps to see all containers. If you want to run your own container rather than the official UCSC container you can simply replace the quay.io/ucsc_cgl/toil:3.6.0 parameter with your own container name.

Also note that we are not mounting the job store directory itself, but rather the location where the job store will be written. Due to complications with running Docker on MacOS, I recommend only mounting directories within your home directory. The next command will launch the Toil worker container with similar parameters:

            docker run \
                --entrypoint=mesos-slave \
                --net=host \
                -d \
                --name=worker \
                --volume=/home/jobStoreParentDir:/jobStoreParentDir \
                quay.io/ucsc_cgl/toil:3.6.0 \
                --work_dir=/var/lib/mesos \
                --master=127.0.0.1:5050 \
                --ip=127.0.0.1 \
                —-attributes=preemptable:False \
                --resources=cpus:2

Note here that we are specifying 2 CPUs and a non-preemptable worker. We can easily change either or both of these in a logical way. To change the number of cores we can change the 2 to whatever number you like, and to change the worker to be preemptable we change preemptable:False to preemptable:True. Also note that the same volume is mounted into the worker. This is needed since both the leader and worker write and read from the job store. Now that your cluster is running, you can run

            docker exec -it leader bash

to get a shell in your leader 'node'. You can also replace the leader parameter with worker to get shell access in your worker.

        Docker-in-Docker issues

            If you want to run Docker inside this Docker cluster (Dockerized tools, perhaps), you should also mount in the Docker socket via -v /var/run/docker.sock:/var/run/docker.sock. This will give the Docker client inside the Toil Appliance access to the Docker engine on the host. Client/engine version mismatches have been known to cause issues, so we recommend using Docker version 1.12.3 on the host to be compatible with the Docker client installed in the Appliance. Finally, be careful where you write files inside the Toil Appliance - 'child' Docker containers launched in the Appliance will actually be siblings to the Appliance since the Docker engine is located on the host. This means that the 'child' container can only mount in files from the Appliance if the files are located in a directory that was originally mounted into the Appliance from the host - that way the files are accessible to the sibling container. Note: if Docker can't find the file/directory on the host it will silently fail and mount in an empty directory.

        Enabling FUSE

            When running toil-wdl-runner with Singularity, Singularity will decompress images to sandbox directories by default. This can take time if a workflow has lots of images. To avoid this, access to FUSE can be given to the Docker container at startup. There are 2 main ways to do this. Either run all the Docker containers in privileged mode:

                    docker run \
                        -d \
                        --name=toil_leader \
                        --privileged \
                        quay.io/ucsc_cgl/toil:6.2.0

        Or pass through the /dev/fuse device node into the container:

                    docker run \
                        -d \
                        --name=toil_leader \
                        --device=/dev/fuse \
                        quay.io/ucsc_cgl/toil:6.2.0

        toil-wdl-runner will handle the logic from there.

MAINTAINER'S GUIDELINES

In general, as developers and maintainers of the code, we adhere to the following guidelines:

    •   We strive to never break the build on master. All development should be done on branches, in either the main Toil repository or in developers' forks.

    •   Pull requests should be used for any and all changes (except truly trivial ones).

    •   Pull requests should be in response to issues. If you find yourself making a pull request without an issue, you should create the issue first.

  Naming Conventions

    •   Commit messages should be great <https://chris.beams.io/posts/git-commit/#seven-rules>. Most importantly, they must:

        •   Have a short subject line. If in need of more space, drop down two lines and write a body to explain what is changing and why it has to change.

        •   Write the subject line as a command: Destroy all humans, not All humans destroyed.

        •   Reference the issue being fixed in a Github-parseable format, such as (resolves #1234) at the end of the subject line, or This will fix #1234. somewhere in the body. If no single commit on its own fixes the issue, the cross-reference must appear in the pull request title or body instead.

    •   Branches in the main Toil repository must start with issues/, followed by the issue number (or numbers, separated by a dash), followed by a short, lowercase, hyphenated description of the change. (There can be many open pull requests with their associated branches at any given point in time and this convention ensures that we can easily identify branches.)

        Say there is an issue numbered #123 titled Foo does not work. The branch name would be issues/123-fix-foo and the title of the commit would be Fix foo in case of bar (resolves #123).

  Pull Requests

    •   All pull requests must be reviewed by a person other than the request's author. Review the PR by following the Reviewing Pull Requests <#reviewingprs> checklist.

    •   Modified pull requests must be re-reviewed before merging. Note that Github does not enforce this!

    •   Merge pull requests by following the Merging Pull Requests <#mergingprs> checklist.

    •   When merging a pull request, make sure to update the Draft Changelog <https://github.com/DataBiosphere/toil/wiki/Draft-Changelog> on the Github wiki, which we will use to produce the changelog for the next release. The PR template tells you to do this, so don't forget. New entries should go at the bottom.

    •   Pull requests will not be merged unless CI tests pass. Gitlab tests are only run on code in the main Toil repository on some branch, so it is the responsibility of the approving reviewer to make sure that pull requests from outside repositories are copied to branches in the main repository. This can be accomplished with (from a Toil clone):

                ./contrib/admin/test-pr theirusername their-branch issues/123-fix-description-here

    This must be repeated every time the PR submitter updates their PR, after checking to see that the update is not malicious.

    If there is no issue corresponding to the PR, after which the branch can be named, the reviewer of the PR should first create the issue.

    Developers who have push access to the main Toil repository are encouraged to make their pull requests from within the repository, to avoid this step.

    •   Prefer using "Squash and marge" when merging pull requests to master especially when the PR contains a "single unit" of work (i.e. if one were to rewrite the PR from scratch with all the fixes included, they would have one commit for the entire PR). This makes the commit history on master more readable and easier to debug in case of a breakage.

        When squashing a PR from multiple authors, please add Co-authored-by <https://github.blog/2018-01-29-commit-together-with-co-authors/> to give credit to all contributing authors.

        See Issue #2816 <https://github.com/DataBiosphere/toil/issues/2816> for more details.

  Publishing a Release

These are the steps to take to publish a Toil release:

    •   Determine the release version X.Y.Z. This should follow semantic versioning <https://semver.org/>; if user-workflow-breaking changes are made, X should be incremented, and Y and Z should be zero. If non-breaking changes are made but new functionality is added, X should remain the same as the last release, Y should be incremented, and Z should be zero. If only patches are released, X and Y should be the same as the last release and Z should be incremented.

    •   If it does not exist already, create a release branch in the Toil repo named X.Y.x, where x is a literal lower-case "x". For patch releases, find the existing branch and make sure it is up to date with the patch commits that are to be released. They may be cherry-picked over <https://trunkbaseddevelopment.com/branch-for-release/> from master.

    •   On the release branch, edit version_template.py in the root of the repository. Find the line that looks like this (slightly different for patch releases):

                baseVersion = 'X.Y.0a1'

    Make it look like this instead:

                baseVersion = 'X.Y.Z'

    Commit your change to the branch.

    •   Tag the current state of the release branch as releases/X.Y.Z.

    •   Make the Github release here <https://github.com/DataBiosphere/toil/releases/new>, referencing that tag. For a non-patch release, fill in the description with the changelog from the wiki page <https://github.com/DataBiosphere/toil/wiki/Draft-Changelog>, which you should clear. For a patch release, just describe the patch.

    •   For a non-patch release, set up the main branch so that development builds will declare themselves to be alpha versions of what the next release will probably be. Edit version_template.py in the root of the repository on the main branch to set baseVersion like this:

                baseVersion = 'X.Y+1.0a1'

    Make sure to replace X and Y+1 with actual numbers.

  Using Git Hooks

In the contrib/hooks directory, there are two scripts, mypy-after-commit.py and mypy-before-push.py, that can be set up as Git hooks to make sure you don't accidentally push commits that would immediately fail type-checking. These are supposed to eliminate the need to run make mypy constantly. You can install them into your Git working copy like this

            ln -rs ./contrib/hooks/mypy-after-commit.py .git/hooks/post-commit
            ln -rs ./contrib/hooks/mypy-before-push.py .git/hooks/pre-push

After you make a commit, the post-commit script will start type-checking it, and if it takes too long re-launch the process in the background. When you push, the pre-push script will see if the commit you are pushing type-checked successfully, and if it hasn't been type-checked but is currently checked out, it will be type-checked. If type-checking fails, the push will be aborted.

Type-checking will only be performed if you are in a Toil development virtual environment. If you aren't, the scripts won't do anything.

To bypass or override pre-push hook, if it is wrong or if you need to push something that doesn't typecheck, you can git push --no-verify. If the scripts get confused about whether a commit actually typechecks, you can clear out the type-checking result cache, which is in /var/run/user/<your UID>/.mypy_toil_result_cache on Linux and in .mypy_toil_result_cache in the Toil repo on Mac.

To uninstall the scripts, delete .git/hooks/post-commit and .git/hooks/pre-push.

  Adding Retries to a Function

See toil.lib.retry <https://github.com/DataBiosphere/toil/blob/master/src/toil/lib/retry.py> .

retry() can be used to decorate any function based on the list of errors one wishes to retry on.

This list of errors can contain normal Exception objects, and/or RetryCondition objects wrapping Exceptions to include additional conditions.

For example, retrying on a one Exception (HTTPError):

            from requests import get
            from requests.exceptions import HTTPError

            @retry(errors=[HTTPError])
            def update_my_wallpaper():
                return get('https://www.deviantart.com/')

Or:

            from requests import get
            from requests.exceptions import HTTPError

            @retry(errors=[HTTPError, ValueError])
            def update_my_wallpaper():
                return get('https://www.deviantart.com/')

The examples above will retry for the default interval on any errors specified the "errors=" arg list.

To retry on specifically 500/502/503/504 errors, you could specify an ErrorCondition object instead, for example:

            from requests import get
            from requests.exceptions import HTTPError

            @retry(errors=[
                ErrorCondition(
                           error=HTTPError,
                           error_codes=[500, 502, 503, 504]
                       )])
            def update_my_wallpaper():
                return requests.get('https://www.deviantart.com/')

To retry on specifically errors containing the phrase "NotFound":

            from requests import get
            from requests.exceptions import HTTPError

            @retry(errors=[
                ErrorCondition(
                    error=HTTPError,
                    error_message_must_include="NotFound"
                )])
            def update_my_wallpaper():
                return requests.get('https://www.deviantart.com/')

To retry on all HTTPError errors EXCEPT an HTTPError containing the phrase "NotFound":

            from requests import get
            from requests.exceptions import HTTPError

            @retry(errors=[
                HTTPError,
                ErrorCondition(
                           error=HTTPError,
                           error_message_must_include="NotFound",
                           retry_on_this_condition=False
                       )])
            def update_my_wallpaper():
                return requests.get('https://www.deviantart.com/')

To retry on boto3's specific status errors, an example of the implementation is:

            import boto3
            from botocore.exceptions import ClientError

            @retry(errors=[
                ErrorCondition(
                           error=ClientError,
                           boto_error_codes=["BucketNotFound"]
                       )])
            def boto_bucket(bucket_name):
                boto_session = boto3.session.Session()
                s3_resource = boto_session.resource('s3')
                return s3_resource.Bucket(bucket_name)

Any combination of these will also work, provided the codes are matched to the correct exceptions. A ValueError will not return a 404, for example.

The retry function as a decorator should make retrying functions easier and clearer. It also encourages smaller independent functions, as opposed to lumping many different things that may need to be retried on different conditions in the same function.

The ErrorCondition object tries to take some of the heavy lifting of writing specific retry conditions and boil it down to an API that covers all common use-cases without the user having to write any new bespoke functions.

Use-cases covered currently:

    1.  Retrying on a normal error, like a KeyError.

    2.  Retrying on HTTP error codes (use ErrorCondition).

    3.  Retrying on boto's specific status errors, like "BucketNotFound" (use ErrorCondition).

    4.  Retrying when an error message contains a certain phrase (use ErrorCondition).

    5.  Explicitly NOT retrying on a condition (use ErrorCondition).

If new functionality is needed, it's currently best practice in Toil to add functionality to the ErrorCondition itself rather than making a new custom retry method.

PULL REQUEST CHECKLISTS

This document contains checklists for dealing with PRs. More general PR information is available at Pull Requests <#prs>.

  Reviewing Pull Requests

This checklist is to be kept in sync with the checklist in the pull request template.

When reviewing a PR, do the following:

    •

        Make sure it is coming from issues/XXXX-fix-the-thing in the Toil repo, or from an external repo.

            •
                 If it is coming from an external repo, make sure to pull it in for CI with:

                        contrib/admin/test-pr otheruser theirbranchname issues/XXXX-fix-the-thing

            •
                 If there is no associated issue, create one <https://github.com/DataBiosphere/toil/issues/new>.

    •

        Read through the code changes. Make sure that it doesn't have:

            •
                 Addition of trailing whitespace.

            •
                 New variable or member names in camelCase that want to be in snake_case.

            •
                 New functions without type hints <https://docs.python.org/3/library/typing.html>.

            •
                 New functions or classes without informative docstrings.

            •
                 Changes to semantics not reflected in the relevant docstrings.

            •
                 New or changed command line options for Toil workflows that are not reflected in docs/running/cliOptions.rst

            •
                 New features without tests.

    •
         Comment on the lines of code where problems exist with a review comment. You can shift-click the line numbers in the diff to select multiple lines.

    •
         Finish the review with an overall description of your opinion.

  Merging Pull Requests

This checklist is to be kept in sync with the checklist in the pull request template.

When merging a PR, do the following:

    •
         Make sure the PR passed tests, including the Gitlab tests, for the most recent commit in its branch.

    •
         Make sure the PR has been reviewed. If not, review it. If it has been reviewed and any requested changes seem to have been addressed, proceed.

    •

        Merge with the Github "Squash and merge" feature.

            •

                If there are multiple authors' commits, add Co-authored-by <https://github.blog/2018-01-29-commit-together-with-co-authors/> to give credit to all contributing authors.

    •
         Copy its recommended changelog entry to the Draft Changelog <https://github.com/DataBiosphere/toil/wiki/Draft-Changelog>.

    •
         Append the issue number in parentheses to the changelog entry.

TOIL ARCHITECTURE

The following diagram layouts out the software architecture of Toil.
        [image: Toil's architecture is composed of the leader, the job store, the worker processes, the batch system, the node provisioner, and the stats and logging monitor.] [image] Figure 1: The basic components of Toil's architecture..UNINDENT

        These components are described below:

            •

                the leader:
                    The leader is responsible for deciding which jobs should be run. To do this it traverses the job graph. Currently this is a single threaded process, but we make aggressive steps to prevent it becoming a bottleneck (see Read-only Leader described below).

            •

                the job-store:
                    Handles all files shared between the components. Files in the job-store are the means by which the state of the workflow is maintained. Each job is backed by a file in the job store, and atomic updates to this state are used to ensure the workflow can always be resumed upon failure. The job-store can also store all user files, allowing them to be shared between jobs. The job-store is defined by the AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore> class. Multiple implementations of this class allow Toil to support different back-end file stores, e.g.: S3, network file systems, Google file store, etc.

            •

                workers:
                    The workers are temporary processes responsible for running jobs, one at a time per worker. Each worker process is invoked with a job argument that it is responsible for running. The worker monitors this job and reports back success or failure to the leader by editing the job's state in the file-store. If the job defines successor jobs the worker may choose to immediately run them (see Job Chaining below).

            •

                the batch-system:
                    Responsible for scheduling the jobs given to it by the leader, running a worker command for each job. The batch-system is defined by the AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem> class. Toil uses multiple existing batch systems to schedule jobs, including Apache Mesos, GridEngine and a multi-process single node implementation that allows workflows to be run without any of these frameworks. Toil can therefore fairly easily be made to run a workflow using an existing cluster.

            •

                the node provisioner:
                    Creates worker nodes in which the batch system schedules workers. It is defined by the AbstractProvisioner <#toil.provisioners.abstractProvisioner.AbstractProvisioner> class.

            •

                the statistics and logging monitor:
                    Monitors logging and statistics produced by the workers and reports them. Uses the job-store to gather this information.

  Jobs and JobDescriptions

As noted in Job Basics <#jobbasics>, a job is the atomic unit of work in a Toil workflow. Workflows extend the Job <#toil.job.Job> class to define units of work. These jobs are pickled and stored in the job-store by the leader, and are retrieved and un-pickled by the worker when they are scheduled to run.

During scheduling, Toil does not work with the actual Job objects. Instead, JobDescription <#toil.job.JobDescription> objects are used to store all the information that the Toil Leader ever needs to know about the Job. This includes requirements information, dependency information, body object to run, worker command to issue, etc.

Internally, the JobDescription object is referenced by its jobStoreID, which is often not human readable. However, the Job and JobDescription objects contain several human-readable names that are useful for logging and identification:
jobName | Name of the kind of job this is. This may be used in job store IDs and logging. Also used to let the cluster scaler learn a model for how long the job will take. Defaults to the job class's name if no real user-defined name is available. For a FunctionWrappingJob <#toil.job.FunctionWrappingJob>, the jobName is replaced by the wrapped function's name. For a CWL workflow, the jobName is the class name of the internal job that is running the CWL workflow, such as "CWLJob".
unitName | Name of this instance of this kind of job. If set by the user, it will appear with the jobName in logging. For a CWL workflow, the unitName is the dotted path from the workflow down to the task being run, including numbers for scatter steps.
displayName | A human-readable name to identify this particular job instance. Used as an identifier of the job class in the stats report. Defaults to the job class's name if no real user-defined name is available. For CWL workflows, this includes the jobName and the unitName.

  Statistics and Logging

Toil's statistics and logging system is implemented in a joint class StatsAndLogging <#toil.statsAndLogging.StatsAndLogging>. The class can be instantiated and run as a thread on the leader, where it polls for new log files in the job store with the read_logs() <#toil.jobStores.abstractJobStore.AbstractJobStore.read_logs> method. These are JSON files, which contain structured data. Structured log messages from user Python code, stored under workers.logs_to_leader, from the file store's log_to_leader() <#toil.fileStores.abstractFileStore.AbstractFileStore.log_to_leader> method, will be logged at the appropriate level. The text output that the worker captured for all its chained jobs, in logs.messages, will be logged at debug level in the worker's output. If --writeLogs or --writeLogsGzip is provided, the received worker logs will also be stored by the StatsAndLogging thread into per-job files inside the job store, using writeLogFiles() <#toil.statsAndLogging.StatsAndLogging.writeLogFiles>.

Note that the worker only fills this in if running with debug logging on, or if --writeLogsFromAllJobs is set. Otherwise, logs from successful jobs are not persisted. Logs from failed jobs are persisted differently; they are written to the file store, and the log file is made available through toil.job.JobDescription.getLogFileHandle() <#toil.job.JobDescription.getLogFileHandle>. The leader thread retrieves these logs and calls back into StatsAndLogging <#toil.statsAndLogging.StatsAndLogging> to print or locally save them as appropriate.

The CWL and WDL interpreters use log_user_stream() <#toil.fileStores.abstractFileStore.AbstractFileStore.log_user_stream> to inject CWL and WDL task-level logs into the stats and logging logging system. The full text of those logs gets stored in the JSON stats files, and when the StatsAndLogging thread sees them it reports and saves them, similarly to how it treats Toil job logs.

To ship the statistics and the non-failed-job logs around, the job store has a logs mailbox system: the write_logs() <#toil.jobStores.abstractJobStore.AbstractJobStore.write_logs> method deposits a string, and the read_logs() <#toil.jobStores.abstractJobStore.AbstractJobStore.read_logs> method on the leader passes the strings to a callback. It tracks a concept of new and old, based on whether the string has been read already by anyone, and one can read only the new values, or all values observed. The stats and logging system uses this to pass around structured JSON holding both log data and worker-measured stats, and expects the StatsAndLogging <#toil.statsAndLogging.StatsAndLogging> thread to be the only live reader.

  Optimizations

Toil implements lots of optimizations designed for scalability. Here we detail some of the key optimizations.

  Read-only leader

The leader process is currently implemented as a single thread. Most of the leader's tasks revolve around processing the state of jobs, each stored as a file within the job-store. To minimise the load on this thread, each worker does as much work as possible to manage the state of the job it is running. As a result, with a couple of minor exceptions, the leader process never needs to write or update the state of a job within the job-store. For example, when a job is complete and has no further successors the responsible worker deletes the job from the job-store, marking it complete. The leader then only has to check for the existence of the file when it receives a signal from the batch-system to know that the job is complete. This off-loading of state management is orthogonal to future parallelization of the leader.

  Job chaining

The scheduling of successor jobs is partially managed by the worker, reducing the number of individual jobs the leader needs to process. Currently this is very simple: if the there is a single next successor job to run and its resources fit within the resources of the current job and closely match the resources of the current job then the job is run immediately on the worker without returning to the leader. Further extensions of this strategy are possible, but for many workflows which define a series of serial successors (e.g. map sequencing reads, post-process mapped reads, etc.) this pattern is very effective at reducing leader workload.

  Preemptable node support

Critical to running at large-scale is dealing with intermittent node failures. Toil is therefore designed to always be resumable providing the job-store does not become corrupt. This robustness allows Toil to run on preemptible nodes, which are only available when others are not willing to pay more to use them. Designing workflows that divide into many short individual jobs that can use preemptable nodes allows for workflows to be efficiently scheduled and executed.

  Caching

Running bioinformatic pipelines often require the passing of large datasets between jobs. Toil caches the results from jobs such that child jobs running on the same node can directly use the same file objects, thereby eliminating the need for an intermediary transfer to the job store. Caching also reduces the burden on the local disks, because multiple jobs can share a single file. The resulting drop in I/O allows pipelines to run faster, and, by the sharing of files, allows users to run more jobs in parallel by reducing overall disk requirements.

To demonstrate the efficiency of caching, we ran an experimental internal pipeline on 3 samples from the TCGA Lung Squamous Carcinoma (LUSC) dataset. The pipeline takes the tumor and normal exome fastqs, and the tumor rna fastq and input, and predicts MHC presented neoepitopes in the patient that are potential targets for T-cell based immunotherapies. The pipeline was run individually on the samples on c3.8xlarge machines on AWS (60GB RAM,600GB SSD storage, 32 cores). The pipeline aligns the data to hg19-based references, predicts MHC haplotypes using PHLAT, calls mutations using 2 callers (MuTect and RADIA) and annotates them using SnpEff, then predicts MHC:peptide binding using the IEDB suite of tools before running an in-house rank boosting algorithm on the final calls.

To optimize time taken, The pipeline is written such that mutations are called on a per-chromosome basis from the whole-exome bams and are merged into a complete vcf. Running mutect in parallel on whole exome bams requires each mutect job to download the complete Tumor and Normal Bams to their working directories -- An operation that quickly fills the disk and limits the parallelizability of jobs. The workflow was run in Toil, with and without caching, and Figure 2 shows that the workflow finishes faster in the cached case while using less disk on average than the uncached run. We believe that benefits of caching arising from file transfers will be much higher on magnetic disk-based storage systems as compared to the SSD systems we tested this on.
        [image: Graph outlining the efficiency gain from caching.] [image] Figure 2: Efficiency gain from caching. The lower half of each plot describes the disk used by the pipeline recorded every 10 minutes over the duration of the pipeline, and the upper half shows the corresponding stage of the pipeline that is being processed. Since jobs requesting the same file shared the same inode, the effective load on the disk is considerably lower than in the uncached case where every job downloads a personal copy of every file it needs. We see that in all cases, the uncached run uses almost 300-400GB more that the cached run in the resource heavy mutation calling step. We also see a benefit in terms of wall time for each stage since we eliminate the time taken for file transfers..UNINDENT

  Toil support for Common Workflow Language

The CWL document and input document are loaded using the 'cwltool.load_tool' module. This performs normalization and URI expansion (for example, relative file references are turned into absolute file URIs), validates the document against the CWL schema, initializes Python objects corresponding to major document elements (command line tools, workflows, workflow steps), and performs static type checking that sources and sinks have compatible types.

Input files referenced by the CWL document and input document are imported into the Toil file store. CWL documents may use any URI scheme supported by Toil file store, including local files and object storage.

The 'location' field of File references are updated to reflect the import token returned by the Toil file store.

For directory inputs, the directory listing is stored in Directory object. Each individual files is imported into Toil file store.

An initial workflow Job is created from the toplevel CWL document. Then, control passes to the Toil engine which schedules the initial workflow job to run.

When the toplevel workflow job runs, it traverses the CWL workflow and creates a toil job for each step. The dependency graph is expressed by making downstream jobs children of upstream jobs, and initializing the child jobs with an input object containing the promises of output from upstream jobs.

Because Toil jobs have a single output, but CWL permits steps to have multiple output parameters that may feed into multiple other steps, the input to a CWLJob is expressed with an "indirect dictionary". This is a dictionary of input parameters, where each entry value is a tuple of a promise and a promise key. When the job runs, the indirect dictionary is turned into a concrete input object by resolving each promise into its actual value (which is always a dict), and then looking up the promise key to get the actual value for the the input parameter.

If a workflow step specifies a scatter, then a scatter job is created and connected into the workflow graph as described above. When the scatter step runs, it creates child jobs for each parameterizations of the scatter. A gather job is added as a follow-on to gather the outputs into arrays.

When running a command line tool, it first creates output and temporary directories under the Toil local temp dir. It runs the command line tool using the single_job_executor from CWLTool, providing a Toil-specific constructor for filesystem access, and overriding the default PathMapper to use ToilPathMapper.

The ToilPathMapper keeps track of a file's symbolic identifier (the Toil FileID), its local path on the host (the value returned by readGlobalFile) and the the location of the file inside the Docker container.

After executing single_job_executor from CWLTool, it gets back the output object and status. If the underlying job failed, raise an exception. Files from the output object are added to the file store using writeGlobalFile and the 'location' field of File references are updated to reflect the token returned by the Toil file store.

When the workflow completes, it returns an indirect dictionary linking to the outputs of the job steps that contribute to the final output. This is the value returned by toil.start() or toil.restart(). This is resolved to get the final output object. The files in this object are exported from the file store to 'outdir' on the host file system, and the 'location' field of File references are updated to reflect the final exported location of the output files.

MINIMUM AWS IAM PERMISSIONS

Toil requires at least the following permissions in an IAM role to operate on a cluster. These are added by default when launching a cluster. However, ensure that they are present if creating a custom IAM role when launching a cluster <#launchawsclusterdetails> with the --awsEc2ProfileArn parameter.

            {
                "Version": "2012-10-17",
                "Statement": [
                    {
                        "Effect": "Allow",
                        "Action": [
                            "ec2:*",
                            "s3:*",
                            "iam:PassRole"
                        ],
                        "Resource": "*"
                    }
                ]
            }

AUTO-DEPLOYMENT

If you want to run a Toil Python workflow in a distributed environment, on multiple worker machines, either in the cloud or on a bare-metal cluster, the Python code needs to be made available to those other machines. If the workflow's main module imports other modules, those modules also need to be made available on the workers. Toil can automatically do that for you, with a little help on your part. We call this feature auto-deployment of a workflow.

Let's first examine various scenarios of auto-deploying a workflow, which, as we'll see shortly cannot be auto-deployed. Lastly, we'll deal with the issue of declaring Toil as a dependency <#depending-on-toil> of a workflow that is packaged as a setuptools distribution.

Toil can be easily deployed to a remote host. First, assuming you've followed our Preparing your AWS environment <#prepareaws> section to install Toil and use it to create a remote leader node on (in this example) AWS, you can now log into this into using Ssh-Cluster Command <#sshcluster> and once on the remote host, create and activate a virtualenv (noting to make sure to use the --system-site-packages option!):

            $ virtualenv --system-site-packages venv
            $ . venv/bin/activate

Note the --system-site-packages option, which ensures that globally-installed packages are accessible inside the virtualenv. Do not (re)install Toil after this! The --system-site-packages option has already transferred Toil and the dependencies from your local installation of Toil for you.

From here, you can install a project and its dependencies:

            $ tree
            .
            ├── util
            │   ├── __init__.py
            │   └── sort
            │       ├── __init__.py
            │       └── quick.py
            └── workflow
                ├── __init__.py
                └── main.py

            3 directories, 5 files
            $ pip install matplotlib
            $ cp -R workflow util venv/lib/python3.9/site-packages

Ideally, your project would have a setup.py file (see setuptools <http://setuptools.readthedocs.io/en/latest/index.html>) which streamlines the installation process:

            $ tree
            .
            ├── util
            │   ├── __init__.py
            │   └── sort
            │       ├── __init__.py
            │       └── quick.py
            ├── workflow
            │   ├── __init__.py
            │   └── main.py
            └── setup.py

            3 directories, 6 files
            $ pip install .

Or, if your project has been published to PyPI:

            $ pip install my-project

In each case, we have created a virtualenv with the --system-site-packages flag in the venv subdirectory then installed the matplotlib distribution from PyPI along with the two packages that our project consists of. (Again, both Python and Toil are assumed to be present on the leader and all worker nodes.)

We can now run our workflow:

            $ python3 main.py --batchSystem=kubernetes …

Important:
        If workflow's external dependencies contain native code (i.e. are not pure Python) then they must be manually installed on each worker.

Warning:
        Neither python3 setup.py develop nor pip install -e . can be used in this process as, instead of copying the source files, they create .egg-link files that Toil can't auto-deploy. Similarly, python3 setup.py install doesn't work either as it installs the project as a Python .egg which is also not currently supported by Toil (though it could be <https://github.com/BD2KGenomics/toil/issues/1367> in the future).

        Also note that using the --single-version-externally-managed flag with setup.py will prevent the installation of your package as an .egg. It will also disable the automatic installation of your project's dependencies.

  Auto Deployment with Sibling Python Files

This scenario applies if a Python workflow imports files that are its siblings:

            $ cd my_project
            $ ls
            userScript.py utilities.py
            $ ./userScript.py --batchSystem=kubernetes …

Here userScript.py imports additional functionality from utilities.py. Toil detects that userScript.py has sibling Python files and copies them to the workers, alongside the main Python file. Note that sibling Python files will be auto-deployed regardless of whether they are actually imported by the workflow: all .py files residing in the same directory as the main workflow Python file will automatically be auto-deployed.

This structure is a suitable method of organizing the source code of reasonably complicated workflows.

  Auto-Deploying a Package Hierarchy

Recall that in Python, a package <https://docs.python.org/2/tutorial/modules.html#packages> is a directory containing one or more .py files, one of which must be called __init__.py, and optionally other packages. For more involved workflows that contain a significant amount of code, this is the recommended way of organizing the source code. Because we use a package hierarchy, the main workflow file is actually a Python module. It is merely one of the modules in the package hierarchy. We need to inform Toil that we want to use a package hierarchy by invoking Python's -m option. This enables Toil to identify the entire set of modules belonging to the workflow and copy all of them to each worker. Note that while using the -m option is optional in the scenarios above, it is mandatory in this one.

The following shell session illustrates this:

            $ cd my_project
            $ tree
            .
            ├── utils
            │   ├── __init__.py
            │   └── sort
            │       ├── __init__.py
            │       └── quick.py
            └── workflow
                ├── __init__.py
                └── main.py

            3 directories, 5 files
            $ python3 -m workflow.main --batchSystem=kubernetes …

Here the workflow entry point module main.py does not reside in the current directory, but is part of a package called util, in a subdirectory of the current directory. Additional functionality is in a separate module called util.sort.quick which corresponds to util/sort/quick.py. Because we invoke the workflow via python3 -m workflow.main, Toil can determine the root directory of the hierarchy–my_project in this case–and copy all Python modules underneath it to each worker. The -m option is documented here <https://docs.python.org/2/using/cmdline.html#cmdoption-m>

When -m is passed, Python adds the current working directory to sys.path, the list of root directories to be considered when resolving a module name like workflow.main. Without that added convenience we'd have to run the workflow as PYTHONPATH="$PWD" python3 -m workflow.main. This also means that Toil can detect the root directory of the invoked module's package hierarchy even if it isn't the current working directory. In other words we could do this:

            $ cd my_project
            $ export PYTHONPATH="$PWD"
            $ cd /some/other/dir
            $ python3 -m workflow.main --batchSystem=kubernetes …

Also note that the root directory itself must not be package, i.e. must not contain an __init__.py.

  Relying on Shared Filesystems

Bare-metal clusters typically mount a shared file system like NFS on each node. If every node has that file system mounted at the same path, you can place your project on that shared filesystem and run your Python workflow from there. Additionally, you can clone the Toil source tree into a directory on that shared file system and you won't even need to install Toil on every worker. Be sure to add both your project directory and the Toil clone to PYTHONPATH. Toil replicates PYTHONPATH from the leader to every worker.

        Using a shared filesystem

            Toil currently only supports a tempdir set to a local, non-shared directory.

  Toil Appliance

The term Toil Appliance refers to the Ubuntu-based Docker image that Toil uses for the machines in Toil-manages clusters, and for executing jobs on Kubernetes. It's easily deployed, only needs Docker, and allows a consistent environment on all Toil clusters. To specify a different image, see the Toil Environment Variables <#envars> section. For more information on the Toil Appliance, see the Running in AWS <#runningaws> section.

ENVIRONMENT VARIABLES

There are several environment variables that affect the way Toil runs.
TOIL_CHECK_ENV | A flag that determines whether Toil will try to refer back to a Python virtual environment in which it is installed when composing commands that may be run on other hosts. If set to True, if Toil is installed in the current virtual environment, it will use absolute paths to its own executables (and the virtual environment must thus be available on at the same path on all nodes). Otherwise, Toil internal commands such as _toil_worker will be resolved according to the PATH on the node where they are executed. This setting can be useful in a shared HPC environment, where users may have their own Toil installations in virtual environments.
TOIL_WORKDIR | An absolute path to a directory where Toil will write its temporary files. This directory must exist on each worker node and may be set to a different value on each worker. The --workDir command line option overrides this. When using the Toil docker container, such as on Kubernetes, this defaults to /var/lib/toil. When using Toil autoscaling with Mesos, this is somewhere inside the Mesos sandbox. In all other cases, the system's standard temporary directory <https://docs.python.org/3/library/tempfile.html#tempfile.gettempdir> is used.
TOIL_WORKDIR_OVERRIDE | An absolute path to a directory where Toil will write its temporary files. This overrides TOIL_WORKDIR and the --workDir command line option.
TOIL_COORDINATION_DIR | An absolute path to a directory where Toil will write its lock files. This directory must exist on each worker node and may be set to a different value on each worker. The --coordinationDir command line option overrides this.
TOIL_COORDINATION_DIR_OVERRIDE | An absolute path to a directory where Toil will write its lock files. This overrides TOIL_COORDINATION_DIR and the --coordinationDir command line option.
TOIL_BATCH_LOGS_DIR | A directory to save batch system logs into, where the leader can access them. The --batchLogsDir option overrides this. Only works for grid engine batch systems such as gridengine, htcondor, torque, slurm, and lsf.
TOIL_HISTORY | Whether to enable the Toil history database. Defaults to true.
TOIL_JOB_HISTORY | Whether to enable per-job history in the Toil history database. Defaults to false.
TOIL_KUBERNETES_HOST_PATH | A path on Kubernetes hosts that will be mounted as the Toil work directory in the workers, to allow for shared caching. Will be created if it doesn't already exist.
TOIL_KUBERNETES_OWNER | A name prefix for easy identification of Kubernetes jobs. If not set, Toil will use the current user name.
TOIL_KUBERNETES_SERVICE_ACCOUNT | A service account name to apply when creating Kubernetes pods.
TOIL_KUBERNETES_POD_TIMEOUT | Seconds to wait for a scheduled Kubernetes pod to start running.
TOIL_KUBERNETES_POD_SECURITY_CONTEXT | Path to a YAML defining a pod security context to apply to all pods.
TOIL_KUBERNETES_SECURITY_CONTEXT | Path to a YAML defining a security context to apply to all containers.
KUBE_WATCH_ENABLED | A boolean variable that allows for users to utilize kubernetes watch stream feature instead of polling for running jobs. Default value is set to False.
TOIL_APPLIANCE_SELF | The fully qualified reference for the Toil Appliance you wish to use, in the form REPO/IMAGE:TAG. quay.io/ucsc_cgl/toil:3.6.0 and cket/toil:3.5.0 are both examples of valid options. Note that since Docker defaults to Dockerhub repos, only quay.io repos need to specify their registry.
TOIL_DOCKER_REGISTRY | The URL of the registry of the Toil Appliance image you wish to use. Docker will use Dockerhub by default, but the quay.io registry is also very popular and easily specifiable by setting this option to quay.io.
TOIL_DOCKER_NAME | The name of the Toil Appliance image you wish to use. Generally this is simply toil but this option is provided to override this, since the image can be built with arbitrary names.
TOIL_AWS_SECRET_NAME | For the Kubernetes batch system, the name of a Kubernetes secret which contains a credentials file granting access to AWS resources. Will be mounted as ~/.aws inside Kubernetes-managed Toil containers. Enables the AWSJobStore to be used with the Kubernetes batch system, if the credentials allow access to S3 and SimpleDB.
TOIL_AWS_ZONE | Zone to use when using AWS. Also determines region. Overrides TOIL_AWS_REGION.
TOIL_AWS_REGION | Region to use when using AWS.
TOIL_AWS_AMI | ID of the AMI to use in node provisioning. If in doubt, don't set this variable.
TOIL_AWS_NODE_DEBUG | Determines whether to preserve nodes that have failed health checks. If set to True, nodes that fail EC2 health checks won't immediately be terminated so they can be examined and the cause of failure determined. If any EC2 nodes are left behind in this manner, the security group will also be left behind by necessity as it cannot be deleted until all associated nodes have been terminated.
TOIL_AWS_BATCH_QUEUE | Name or ARN of an AWS Batch Queue to use with the AWS Batch batch system.
TOIL_AWS_BATCH_JOB_ROLE_ARN | ARN of an IAM role to run AWS Batch jobs as with the AWS Batch batch system. If the jobs are not run with an IAM role or on machines that have access to S3 and SimpleDB, the AWS job store will not be usable.
TOIL_GOOGLE_PROJECTID | The Google project ID to use when generating Google job store names for tests or CWL workflows.
TOIL_SLURM_ALLOCATE_MEM | Whether to allocate memory in Slurm with --mem. True by default.
TOIL_SLURM_ARGS | Arguments for sbatch for the slurm batch system. Do not pass CPU or memory specifications here. Instead, define resource requirements for the job. There is no default value for this variable. If neither --export nor --export-file is in the argument list, --export=ALL will be provided.
TOIL_SLURM_PARTITION | Partition to send Slurm jobs to.
TOIL_SLURM_GPU_PARTITION | Partition to send Slurm jobs to if they ask for GPUs.
TOIL_SLURM_PE | Name of the slurm partition to use for parallel jobs. Useful for Slurm clusters that do not offer a partition accepting both single-core and multi-core jobs. There is no default value for this variable.
TOIL_SLURM_TIME | Slurm job time limit, in [DD-]HH:MM:SS format. For example, 2-07:15:30 for 2 days, 7 hours, 15 minutes and 30 seconds, or 4:00:00 for 4 hours.
TOIL_GRIDENGINE_ARGS | Arguments for qsub for the gridengine batch system. Do not pass CPU or memory specifications here. Instead, define resource requirements for the job. There is no default value for this variable.
TOIL_GRIDENGINE_PE | Parallel environment arguments for qsub and for the gridengine batch system. There is no default value for this variable.
TOIL_TORQUE_ARGS | Arguments for qsub for the Torque batch system. Do not pass CPU or memory specifications here. Instead, define extra parameters for the job such as queue. Example: -q medium Use TOIL_TORQUE_REQS to pass extra values for the -l resource requirements parameter. There is no default value for this variable.
TOIL_TORQUE_REQS | Arguments for the resource requirements for Torque batch system. Do not pass CPU or memory specifications here. Instead, define extra resource requirements as a string that goes after the -l argument to qsub. Example: walltime=2:00:00,file=50gb There is no default value for this variable.
TOIL_LSF_ARGS | Additional arguments for the LSF's bsub command. Instead, define extra parameters for the job such as queue. Example: -q medium. There is no default value for this variable.
TOIL_HTCONDOR_PARAMS | Additional parameters to include in the HTCondor submit file passed to condor_submit. Do not pass CPU or memory specifications here. Instead define extra parameters which may be required by HTCondor. This variable is parsed as a semicolon-separated string of parameter = value pairs. Example: requirements = TARGET.has_sse4_2 == true; accounting_group = test. There is no default value for this variable.
TOIL_CUSTOM_DOCKER_INIT_COMMAND | Any custom bash command to run in the Toil docker container prior to running the Toil services. Can be used for any custom initialization in the worker and/or primary nodes such as private docker docker authentication. Example for AWS ECR: pip install awscli && eval $(aws ecr get-login --no-include-email --region us-east-1).
TOIL_CUSTOM_INIT_COMMAND | Any custom bash command to run prior to starting the Toil appliance. Can be used for any custom initialization in the worker and/or primary nodes such as private docker authentication for the Toil appliance itself (i.e. from TOIL_APPLIANCE_SELF).
TOIL_S3_HOST | the IP address or hostname to use for connecting to S3. Example: TOIL_S3_HOST=127.0.0.1
TOIL_S3_PORT | a port number to use for connecting to S3. Example: TOIL_S3_PORT=9001
TOIL_S3_USE_SSL | enable or disable the usage of SSL for connecting to S3 (True by default). Example: TOIL_S3_USE_SSL=False
TOIL_FTP_USER | The FTP username to override all FTP logins with Example: TOIL_FTP_USER=ftp_user
TOIL_FTP_PASSWORD | The FTP password to override all FTP logins with Example: TOIL_FTP_PASSWORD=ftp_password
TOIL_WES_BROKER_URL | An optional broker URL to use to communicate between the WES server and Celery task queue. If unset, amqp://guest:guest@localhost:5672// is used.
TOIL_WES_JOB_STORE_TYPE | Type of job store to use by default for workflows run via the WES server. Can be file, aws, or google.
TOIL_OWNER_TAG | This will tag cloud resources with a tag reading: "Owner: $TOIL_OWNER_TAG". This is used internally at UCSC to stop a bot we have that terminates untagged resources.
TOIL_AWS_PROFILE | The name of an AWS profile to run TOIL with.
TOIL_AWS_TAGS | This will tag cloud resources with any arbitrary tags given in a JSON format. These are overwritten in favor of CLI options when using launch cluster. For information on valid AWS tags, see AWS Tags <https://docs.aws.amazon.com/general/latest/gr/aws_tagging.html>.
TOIL_TRS_ROOT | Override for the Tool Registry Service endpoint or Dockstore deployment to talk to for workflow lookups and metrics publication. Allows changing the workflow lookup and metrics publication destination to something else, such as https://qa.dockstore.org, for debugging.
TOIL_DOCKSTORE_TOKEN | When publishing workflow metrics, include this Dockstore authentication token instead of the default public Toil one.
SINGULARITY_DOCKER_HUB_MIRROR | An http or https URL for the Singularity wrapper in the Toil Docker container to use as a mirror for Docker Hub.
OMP_NUM_THREADS | The number of cores set for OpenMP applications in the workers. If not set, Toil will use the number of job threads.
GUNICORN_CMD_ARGS | Specify additional Gunicorn configurations for the Toil WES server. See Gunicorn settings <https://docs.gunicorn.org/en/stable/settings.html#settings>.

API REFERENCE

This page contains auto-generated API reference documentation [1].

  toil

  Submodules

  toil.batchSystems

  Submodules

  toil.batchSystems.abstractBatchSystem

  Attributes
logger |
EXIT_STATUS_UNAVAILABLE_VALUE |

  Exceptions
InsufficientSystemResources | Common base class for all non-exit exceptions.
AcquisitionTimeoutException | To be raised when a resource request times out.

  Classes
BatchJobExitReason | Enum where members are also (and must be) ints
UpdatedBatchJobInfo |
WorkerCleanupInfo |
AbstractBatchSystem | An abstract base class to represent the interface the batch system must provide to Toil.
BatchSystemSupport | Partial implementation of AbstractBatchSystem, support methods.
NodeInfo | The coresUsed attribute is a floating point value between 0 (all cores idle) and 1 (all cores
AbstractScalableBatchSystem | A batch system that supports a variable number of worker nodes.
ResourcePool | Represents an integral amount of a resource (such as memory bytes).
ResourceSet | Represents a collection of distinct resources (such as accelerators).

  Module Contents

    toil.batchSystems.abstractBatchSystem.logger

    toil.batchSystems.abstractBatchSystem.EXIT_STATUS_UNAVAILABLE_VALUE = 255

    class toil.batchSystems.abstractBatchSystem.BatchJobExitReason
        Bases: enum.IntEnum </usr/share/doc/python3/html/library/enum.html#enum.IntEnum>

        Enum where members are also (and must be) ints

        FINISHED = 1
            Successfully finished.

        FAILED = 2
            Job finished, but failed.

        LOST = 3
            Preemptable failure (job's executing host went away).

        KILLED = 4
            Job killed before finishing.

        ERROR = 5
            Internal error.

        MEMLIMIT = 6
            Job hit batch system imposed memory limit.

        MISSING = 7
            Job disappeared from the scheduler without actually stopping, so Toil killed it.

        MAXJOBDURATION = 8
            Job ran longer than --maxJobDuration, so Toil killed it.

        PARTITION = 9
            Job was not able to talk to the leader via the job store, so Toil declared it failed.

        classmethod to_string(value)
            Convert to human-readable string.

            Given an int that may be or may be equal to a value from the enum, produce the string value of its matching enum entry, or a stringified int.

            Parameters
                value (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

    class toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo

        jobID: int </usr/share/doc/python3/html/library/functions.html#int>
            The Toil batch system ID of the job.

        exitStatus: int </usr/share/doc/python3/html/library/functions.html#int> = 255
            The exit status (integer value) of the job. 0 implies successful.

            EXIT_STATUS_UNAVAILABLE_VALUE is used when the exit status is not available (e.g. job is lost, or otherwise died but actual exit code was not reported).

        exitReason: BatchJobExitReason <#toil.batchSystems.abstractBatchSystem.BatchJobExitReason> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        wallTime: float </usr/share/doc/python3/html/library/functions.html#float> | int </usr/share/doc/python3/html/library/functions.html#int> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        backing_id: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None
            The identifier for the job in the backing scheduler, if available.

    class toil.batchSystems.abstractBatchSystem.WorkerCleanupInfo
        Bases: NamedTuple

        work_dir: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>
            Work directory path (where the cache would go) if specified by user

        coordination_dir: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>
            Coordination directory path (where lock files would go) if specified by user

        workflow_id: str </usr/share/doc/python3/html/library/stdtypes.html#str>
            Used to identify files specific to this workflow

        clean_work_dir: str </usr/share/doc/python3/html/library/stdtypes.html#str>
            When to clean up the work and coordination directories for a job ('always', 'onSuccess', 'onError', 'never')

    class toil.batchSystems.abstractBatchSystem.AbstractBatchSystem
        Bases: abc.ABC </usr/share/doc/python3/html/library/abc.html#abc.ABC>

        An abstract base class to represent the interface the batch system must provide to Toil.

        classmethod supportsAutoDeployment()

            Abstractmethod

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        Whether this batch system supports auto-deployment of the user script itself.

        If it does, the setUserScript() can be invoked to set the resource object representing the user script.

        Note to implementors: If your implementation returns True here, it should also override

        classmethod supportsWorkerCleanup()

            Abstractmethod

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        Whether this batch system supports worker cleanup.

        Indicates whether this batch system invokes BatchSystemSupport.workerCleanup() after the last job for a particular workflow invocation finishes. Note that the term worker refers to an entire node, not just a worker process. A worker process may run more than one job sequentially, and more than one concurrent worker process may exist on a worker node, for the same workflow. The batch system is said to shut down after the last worker process terminates.

        abstractmethod setUserScript(userScript)
            Set the user script for this workflow.

            This method must be called before the first job is issued to this batch system, and only if supportsAutoDeployment() returns True, otherwise it will raise an exception.

            Parameters
                userScript (toil.resource.Resource <#toil.resource.Resource>) -- the resource object representing the user script or module and the modules it depends on.

            Return type
                None

        set_message_bus(message_bus)
            Give the batch system an opportunity to connect directly to the message bus, so that it can send informational messages about the jobs it is running to other Toil components.

            Parameters
                message_bus (toil.bus.MessageBus <#toil.bus.MessageBus>)

            Return type
                None

        abstractmethod issueBatchJob(command, job_desc, job_environment=None)
            Issues a job with the specified command to the batch system and returns a unique job ID number.

            Parameters

                •   command (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the command to execute somewhere to run the Toil worker process

                •   job_desc (toil.job.JobDescription <#toil.job.JobDescription>) -- the JobDescription for the job being run

                •   job_environment (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) -- a collection of job-specific environment variables to be set on the worker.

            Returns
                a unique job ID number that can be used to reference the newly issued job

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        abstractmethod killBatchJobs(jobIDs)
            Kills the given job IDs. After returning, the killed jobs will not appear in the results of getRunningBatchJobIDs. The killed job will not be returned from getUpdatedBatchJob.

            Parameters
                jobIDs (list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>]) -- list of IDs of jobs to kill

            Return type
                None

        abstractmethod getIssuedBatchJobIDs()
            Gets all currently issued jobs

            Returns
                A list of jobs (as job ID numbers) currently issued (may be running, or may be waiting to be run). Despite the result being a list, the ordering should not be depended upon.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>]

        abstractmethod getRunningBatchJobIDs()
            Gets a map of jobs as job ID numbers that are currently running (not just waiting) and how long they have been running, in seconds.

            Returns
                dictionary with currently running job ID number keys and how many seconds they have been running as the value

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>]

        abstractmethod getUpdatedBatchJob(maxWait)
            Returns information about job that has updated its status (i.e. ceased running, either successfully or with an error). Each such job will be returned exactly once.

            Does not return info for jobs killed by killBatchJobs, although they may cause None to be returned earlier than maxWait.

            Parameters
                maxWait (int </usr/share/doc/python3/html/library/functions.html#int>) -- the number of seconds to block, waiting for a result

            Returns
                If a result is available, returns UpdatedBatchJobInfo. Otherwise it returns None. wallTime is the number of seconds (a strictly positive float) in wall-clock time the job ran for, or None if this batch system does not support tracking wall time.

            Return type
                Optional[UpdatedBatchJobInfo <#toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo>]

        getSchedulingStatusMessage()
            Get a log message fragment for the user about anything that might be going wrong in the batch system, if available.

            If no useful message is available, return None.

            This can be used to report what resource is the limiting factor when scheduling jobs, for example. If the leader thinks the workflow is stuck, the message can be displayed to the user to help them diagnose why it might be stuck.

            Returns
                User-directed message about scheduling state.

            Return type
                Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        abstractmethod shutdown()
            Called at the completion of a toil invocation. Should cleanly terminate all worker threads.

            Return type
                None

        abstractmethod setEnv(name, value=None)
            Set an environment variable for the worker process before it is launched.

            The worker process will typically inherit the environment of the machine it is running on but this method makes it possible to override specific variables in that inherited environment before the worker is launched. Note that this mechanism is different to the one used by the worker internally to set up the environment of a job. A call to this method affects all jobs issued after this method returns. Note to implementors: This means that you would typically need to copy the variables before enqueuing a job.

            If no value is provided it will be looked up from the current environment.

            Parameters

                •   name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   value (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

        classmethod add_options(parser)
            If this batch system provides any command line options, add them to the given parser.

            Parameters
                parser (Union[argparse.ArgumentParser </usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser>, argparse._ArgumentGroup])

            Return type
                None

        classmethod setOptions(setOption)
            Process command line or configuration options relevant to this batch system.

            Parameters
                setOption (toil.batchSystems.options.OptionSetter <#toil.batchSystems.options.OptionSetter>) -- A function with signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) returning nothing, used to update run configuration as a side effect.

            Return type
                None

        getWorkerContexts()
            Get a list of picklable context manager objects to wrap worker work in, in order.

            Can be used to ask the Toil worker to do things in-process (such as configuring environment variables, hot-deploying user scripts, or cleaning up a node) that would otherwise require a wrapping "executor" process.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[ContextManager[Any]]

    class toil.batchSystems.abstractBatchSystem.BatchSystemSupport(config, maxCores, maxMemory, maxDisk)
        Bases: AbstractBatchSystem

        Partial implementation of AbstractBatchSystem, support methods.

        Parameters

            •   config (toil.common.Config <#toil.common.Config>)

            •   maxCores (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxMemory (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxDisk (int </usr/share/doc/python3/html/library/functions.html#int>)

        config

        maxCores

        maxMemory

        maxDisk

        environment: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        check_resource_request(requirer)
            Check resource request is not greater than that available or allowed.

            Parameters

                •   requirer (toil.job.Requirer <#toil.job.Requirer>) -- Object whose requirements are being checked

                •   job_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Name of the job being checked, for generating a useful error report.

                •   detail (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Batch-system-specific message to include in the error.

            Raises
                InsufficientSystemResources <#toil.batchSystems.abstractBatchSystem.InsufficientSystemResources> -- raised when a resource is requested in an amount greater than allowed

            Return type
                None

        setEnv(name, value=None)
            Set an environment variable for the worker process before it is launched. The worker process will typically inherit the environment of the machine it is running on but this method makes it possible to override specific variables in that inherited environment before the worker is launched. Note that this mechanism is different to the one used by the worker internally to set up the environment of a job. A call to this method affects all jobs issued after this method returns. Note to implementors: This means that you would typically need to copy the variables before enqueuing a job.

            If no value is provided it will be looked up from the current environment.

            Parameters

                •   name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the environment variable to be set on the worker.

                •   value (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- if given, the environment variable given by name will be set to this value. If None, the variable's current value will be used as the value on the worker

            Raises
                RuntimeError </usr/share/doc/python3/html/library/exceptions.html#RuntimeError> -- if value is None and the name cannot be found in the environment

            Return type
                None

        set_message_bus(message_bus)
            Give the batch system an opportunity to connect directly to the message bus, so that it can send informational messages about the jobs it is running to other Toil components.

            Parameters
                message_bus (toil.bus.MessageBus <#toil.bus.MessageBus>)

            Return type
                None

        get_batch_logs_dir()
            Get the directory where the backing batch system should save its logs.

            Only really makes sense if the backing batch system actually saves logs to a filesystem; Kubernetes for example does not. Ought to be a directory shared between the leader and the workers, if the backing batch system writes logs onto the worker's view of the filesystem, like many HPC schedulers do.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        format_std_out_err_path(toil_job_id, cluster_job_id, std)
            Format path for batch system standard output/error and other files generated by the batch system itself.

            Files will be written to the batch logs directory (--batchLogsDir, defaulting to the Toil work directory) with names containing both the Toil and batch system job IDs, for ease of debugging job failures.

            Param
                int toil_job_id : The unique id that Toil gives a job.

            Param
                cluster_job_id : What the cluster, for example, GridEngine, uses as its internal job id.

            Param
                string std : The provenance of the stream (for example: 'err' for 'stderr' or 'out' for 'stdout')

            Return type
                string : Formatted filename; however if self.config.noStdOutErr is true, returns '/dev/null' or equivalent.

            Parameters

                •   toil_job_id (int </usr/share/doc/python3/html/library/functions.html#int>)

                •   cluster_job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   std (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        format_std_out_err_glob(toil_job_id)
            Get a glob string that will match all file paths generated by format_std_out_err_path for a job.

            Parameters
                toil_job_id (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        static workerCleanup(info)
            Cleans up the worker node on batch system shutdown.

            Also see supportsWorkerCleanup().

            Parameters
                info (WorkerCleanupInfo <#toil.batchSystems.abstractBatchSystem.WorkerCleanupInfo>) -- A named tuple consisting of all the relevant information for cleaning up the worker.

            Return type
                None

    class toil.batchSystems.abstractBatchSystem.NodeInfo(coresUsed, memoryUsed, coresTotal, memoryTotal, requestedCores, requestedMemory, workers)
        The coresUsed attribute is a floating point value between 0 (all cores idle) and 1 (all cores busy), reflecting the CPU load of the node.

        The memoryUsed attribute is a floating point value between 0 (no memory used) and 1 (all memory used), reflecting the memory pressure on the node.

        The coresTotal and memoryTotal attributes are the node's resources, not just the used resources

        The requestedCores and requestedMemory attributes are all the resources that Toil Jobs have reserved on the node, regardless of whether the resources are actually being used by the Jobs.

        The workers attribute is an integer reflecting the number of workers currently active workers on the node.

        Parameters

            •   coresUsed (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   memoryUsed (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   coresTotal (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   memoryTotal (int </usr/share/doc/python3/html/library/functions.html#int>)

            •   requestedCores (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   requestedMemory (int </usr/share/doc/python3/html/library/functions.html#int>)

            •   workers (int </usr/share/doc/python3/html/library/functions.html#int>)

        coresUsed

        memoryUsed

        coresTotal

        memoryTotal

        requestedCores

        requestedMemory

        workers

    class toil.batchSystems.abstractBatchSystem.AbstractScalableBatchSystem
        Bases: AbstractBatchSystem

        A batch system that supports a variable number of worker nodes.

        Used by toil.provisioners.clusterScaler.ClusterScaler <#toil.provisioners.clusterScaler.ClusterScaler> to scale the number of worker nodes in the cluster up or down depending on overall load.

        abstractmethod getNodes(preemptible=None, timeout=600)
            Returns a dictionary mapping node identifiers of preemptible or non-preemptible nodes to NodeInfo objects, one for each node.

            Parameters

                •   preemptible (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>]) -- If True (False) only (non-)preemptible nodes will be returned. If None, all nodes will be returned.

                •   timeout (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, NodeInfo <#toil.batchSystems.abstractBatchSystem.NodeInfo>]

        abstractmethod nodeInUse(nodeIP)
            Can be used to determine if a worker node is running any tasks. If the node is doesn't exist, this function should simply return False.

            Parameters
                nodeIP (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The worker nodes private IP address

            Returns
                True if the worker node has been issued any tasks, else False

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        abstractmethod ignoreNode(nodeAddress)
            Stop sending jobs to this node. Used in autoscaling when the autoscaler is ready to terminate a node, but jobs are still running. This allows the node to be terminated after the current jobs have finished.

            Parameters
                nodeAddress (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- IP address of node to ignore.

            Return type
                None

        abstractmethod unignoreNode(nodeAddress)
            Stop ignoring this address, presumably after a node with this address has been terminated. This allows for the possibility of a new node having the same address as a terminated one.

            Parameters
                nodeAddress (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

    exception toil.batchSystems.abstractBatchSystem.InsufficientSystemResources(requirer, resource, available=None, batch_system=None, source=None, details=[])
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Common base class for all non-exit exceptions.

        Parameters

            •   requirer (toil.job.Requirer <#toil.job.Requirer>)

            •   resource (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   available (Optional[toil.job.ParsedRequirement])

            •   batch_system (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   source (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   details (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        job_name: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = ''

        resource

        requested

        available = None

        batch_system = None

        source = None

        details = []

        __str__()
            Explain the exception.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

    exception toil.batchSystems.abstractBatchSystem.AcquisitionTimeoutException(resource, requested, available)
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        To be raised when a resource request times out.

        Parameters

            •   resource (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   requested (Union[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>, set </usr/share/doc/python3/html/library/stdtypes.html#set>[int </usr/share/doc/python3/html/library/functions.html#int>]])

            •   available (Union[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>, set </usr/share/doc/python3/html/library/stdtypes.html#set>[int </usr/share/doc/python3/html/library/functions.html#int>]])

        requested

        available

        resource

    class toil.batchSystems.abstractBatchSystem.ResourcePool(initial_value, resource_type, timeout=5)
        Represents an integral amount of a resource (such as memory bytes). Amounts can be acquired immediately or with a timeout, and released. Provides a context manager to do something with an amount of resource acquired.

        Parameters

            •   initial_value (int </usr/share/doc/python3/html/library/functions.html#int>)

            •   resource_type (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   timeout (float </usr/share/doc/python3/html/library/functions.html#float>)

        condition

        value

        resource_type

        timeout = 5

        acquireNow(amount)
            Reserve the given amount of the given resource. Returns True if successful and False if this is not possible immediately.

            Parameters
                amount (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        acquire(amount)
            Reserve the given amount of the given resource. Raises AcquisitionTimeoutException if this is not possible in under self.timeout time.

            Parameters
                amount (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                None

        release(amount)

            Parameters
                amount (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                None

        __str__()

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        __repr__()

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        acquisitionOf(amount)

            Parameters
                amount (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                collections.abc.Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[None]

    class toil.batchSystems.abstractBatchSystem.ResourceSet(initial_value, resource_type, timeout=5)
        Represents a collection of distinct resources (such as accelerators). Subsets can be acquired immediately or with a timeout, and released. Provides a context manager to do something with a set of of resources acquired.

        Parameters

            •   initial_value (set </usr/share/doc/python3/html/library/stdtypes.html#set>[int </usr/share/doc/python3/html/library/functions.html#int>])

            •   resource_type (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   timeout (float </usr/share/doc/python3/html/library/functions.html#float>)

        condition

        value

        resource_type

        timeout = 5

        acquireNow(subset)
            Reserve the given amount of the given resource. Returns True if successful and False if this is not possible immediately.

            Parameters
                subset (set </usr/share/doc/python3/html/library/stdtypes.html#set>[int </usr/share/doc/python3/html/library/functions.html#int>])

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        acquire(subset)
            Reserve the given amount of the given resource. Raises AcquisitionTimeoutException if this is not possible in under self.timeout time.

            Parameters
                subset (set </usr/share/doc/python3/html/library/stdtypes.html#set>[int </usr/share/doc/python3/html/library/functions.html#int>])

            Return type
                None

        release(subset)

            Parameters
                subset (set </usr/share/doc/python3/html/library/stdtypes.html#set>[int </usr/share/doc/python3/html/library/functions.html#int>])

            Return type
                None

        get_free_snapshot()
            Get a snapshot of what items are free right now. May be stale as soon as you get it, but you will need some kind of hint to try and do an acquire.

            Return type
                set </usr/share/doc/python3/html/library/stdtypes.html#set>[int </usr/share/doc/python3/html/library/functions.html#int>]

        __str__()

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        __repr__()

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        acquisitionOf(subset)

            Parameters
                subset (set </usr/share/doc/python3/html/library/stdtypes.html#set>[int </usr/share/doc/python3/html/library/functions.html#int>])

            Return type
                collections.abc.Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[None]

  toil.batchSystems.abstractGridEngineBatchSystem

  Attributes
logger |
JobTuple |

  Exceptions
ExceededRetryAttempts | Common base class for all non-exit exceptions.

  Classes
AbstractGridEngineBatchSystem | A partial implementation of BatchSystemSupport for batch systems run on a

  Module Contents

    toil.batchSystems.abstractGridEngineBatchSystem.logger

    toil.batchSystems.abstractGridEngineBatchSystem.JobTuple

    exception toil.batchSystems.abstractGridEngineBatchSystem.ExceededRetryAttempts
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Common base class for all non-exit exceptions.

    class toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem(config, maxCores, maxMemory, maxDisk)
        Bases: toil.batchSystems.cleanup_support.BatchSystemCleanupSupport <#toil.batchSystems.cleanup_support.BatchSystemCleanupSupport>

        A partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster. By default auto-deployment is not implemented.

        Parameters

            •   config (toil.common.Config <#toil.common.Config>)

            •   maxCores (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxMemory (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxDisk (float </usr/share/doc/python3/html/library/functions.html#float>)

        exception GridEngineThreadException
            Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

            Common base class for all non-exit exceptions.

        class GridEngineThread(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss)
            Bases: threading.Thread </usr/share/doc/python3/html/library/threading.html#threading.Thread>

            A class that represents a thread of control.

            This class can be safely subclassed in a limited fashion. There are two ways to specify the activity: by passing a callable object to the constructor, or by overriding the run() method in a subclass.

            Parameters

                •   newJobsQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)

                •   updatedJobsQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)

                •   killQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)

                •   killedJobsQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)

                •   boss (AbstractGridEngineBatchSystem <#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem>)

            boss

            newJobsQueue

            updatedJobsQueue

            killQueue

            killedJobsQueue

            waitingJobs: list </usr/share/doc/python3/html/library/stdtypes.html#list>[JobTuple] = []

            runningJobs

            runningJobsLock

            batchJobIDs: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[int </usr/share/doc/python3/html/library/functions.html#int>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

            exception = None

            getBatchSystemID(jobID)
                Get batch system-specific job ID

                Note: for the moment this is the only consistent way to cleanly get the batch system job ID

                Parameters
                    jobID (int </usr/share/doc/python3/html/library/functions.html#int>) -- Toil BatchSystem numerical job ID

                Return type
                    str </usr/share/doc/python3/html/library/stdtypes.html#str>

            forgetJob(jobID)
                Remove jobID passed

                Parameters
                    jobID (int </usr/share/doc/python3/html/library/functions.html#int>) -- toil job ID

                Return type
                    None

            createJobs(newJob)
                Create a new job with the given attributes.

                Implementation-specific; called by GridEngineThread.run()

                Parameters
                    newJob (JobTuple)

                Return type
                    bool </usr/share/doc/python3/html/library/functions.html#bool>

            killJobs()
                Kill any running jobs within thread

            checkOnJobs()
                Check and update status of all running jobs.

                Respects statePollingWait and will return cached results if not within time period to talk with the scheduler.

            run()
                Run any new jobs

            coalesce_job_exit_codes(batch_job_id_list)
                Returns exit codes and possibly exit reasons for a list of jobs, or None if they are running.

                Called by GridEngineThread.checkOnJobs().

                The default implementation falls back on self.getJobExitCode and polls each job individually

                Parameters
                    batch_job_id_list (string) -- List of batch system job ID

                Return type
                    list </usr/share/doc/python3/html/library/stdtypes.html#list>[Union[int </usr/share/doc/python3/html/library/functions.html#int>, tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[int </usr/share/doc/python3/html/library/functions.html#int>, Optional[toil.batchSystems.abstractBatchSystem.BatchJobExitReason <#toil.batchSystems.abstractBatchSystem.BatchJobExitReason>]], None]]

            abstractmethod prepareSubmission(cpu, memory, jobID, command, jobName, job_environment=None, gpus=None)
                Preparation in putting together a command-line string for submitting to batch system (via submitJob().)

                Param
                    int cpu

                Param
                    int memory

                Param
                    int jobID: Toil job ID

                Param
                    string subLine: the command line string to be called

                Param
                    string jobName: the name of the Toil job, to provide metadata to batch systems if desired

                Param
                    dict job_environment: the environment variables to be set on the worker

                Return type
                    List[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

                Parameters

                    •   cpu (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   memory (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   jobID (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   command (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                    •   jobName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                    •   job_environment (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

                    •   gpus (Optional[int </usr/share/doc/python3/html/library/functions.html#int>])

            abstractmethod submitJob(subLine)
                Wrapper routine for submitting the actual command-line call, then processing the output to get the batch system job ID

                Param
                    string subLine: the literal command line string to be called

                Return type
                    string: batch system job ID, which will be stored internally

            abstractmethod getRunningJobIDs()
                Get a list of running job IDs. Implementation-specific; called by boss AbstractGridEngineBatchSystem implementation via AbstractGridEngineBatchSystem.getRunningBatchJobIDs()

                Return type
                    list </usr/share/doc/python3/html/library/stdtypes.html#list>

            abstractmethod killJob(jobID)
                Kill specific job with the Toil job ID. Implementation-specific; called by GridEngineThread.killJobs()

                Parameters
                    jobID (string) -- Toil job ID

            abstractmethod getJobExitCode(batchJobID)
                Returns job exit code and possibly an instance of abstractBatchSystem.BatchJobExitReason.

                Returns None if the job is still running.

                If the job is not running but the exit code is not available, it will be EXIT_STATUS_UNAVAILABLE_VALUE. Implementation-specific; called by GridEngineThread.checkOnJobs().

                The exit code will only be 0 if the job affirmatively succeeded.

                Parameters
                    batchjobID (string) -- batch system job ID

                Return type
                    Union[int </usr/share/doc/python3/html/library/functions.html#int>, tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[int </usr/share/doc/python3/html/library/functions.html#int>, Optional[toil.batchSystems.abstractBatchSystem.BatchJobExitReason <#toil.batchSystems.abstractBatchSystem.BatchJobExitReason>]], None]

        config

        currentJobs

        newJobsQueue

        updatedJobsQueue

        killQueue

        killedJobsQueue

        background_thread

        classmethod supportsAutoDeployment()
            Whether this batch system supports auto-deployment of the user script itself.

            If it does, the setUserScript() can be invoked to set the resource object representing the user script.

            Note to implementors: If your implementation returns True here, it should also override

        count_needed_gpus(job_desc)
            Count the number of cluster-allocateable GPUs we want to allocate for the given job.

            Parameters
                job_desc (toil.job.JobDescription <#toil.job.JobDescription>)

        issueBatchJob(command, job_desc, job_environment=None)
            Issues a job with the specified command to the batch system and returns a unique job ID number.

            Parameters

                •   command (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the command to execute somewhere to run the Toil worker process

                •   job_desc (toil.job.JobDescription <#toil.job.JobDescription>) -- the JobDescription for the job being run

                •   job_environment (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) -- a collection of job-specific environment variables to be set on the worker.

            Returns
                a unique job ID number that can be used to reference the newly issued job

        killBatchJobs(jobIDs)
            Kills the given jobs, represented as Job ids, then checks they are dead by checking they are not in the list of issued jobs.

        getIssuedBatchJobIDs()
            Gets the list of issued jobs

        getRunningBatchJobIDs()
            Retrieve running job IDs from local and batch scheduler.

            Respects statePollingWait and will return cached results if not within time period to talk with the scheduler.

        getUpdatedBatchJob(maxWait)
            Returns information about job that has updated its status (i.e. ceased running, either successfully or with an error). Each such job will be returned exactly once.

            Does not return info for jobs killed by killBatchJobs, although they may cause None to be returned earlier than maxWait.

            Parameters
                maxWait -- the number of seconds to block, waiting for a result

            Returns
                If a result is available, returns UpdatedBatchJobInfo. Otherwise it returns None. wallTime is the number of seconds (a strictly positive float) in wall-clock time the job ran for, or None if this batch system does not support tracking wall time.

        shutdown()
            Signals thread to shutdown (via sentinel) then cleanly joins the thread

            Return type
                None

        setEnv(name, value=None)
            Set an environment variable for the worker process before it is launched. The worker process will typically inherit the environment of the machine it is running on but this method makes it possible to override specific variables in that inherited environment before the worker is launched. Note that this mechanism is different to the one used by the worker internally to set up the environment of a job. A call to this method affects all jobs issued after this method returns. Note to implementors: This means that you would typically need to copy the variables before enqueuing a job.

            If no value is provided it will be looked up from the current environment.

            Parameters

                •   name -- the environment variable to be set on the worker.

                •   value -- if given, the environment variable given by name will be set to this value. If None, the variable's current value will be used as the value on the worker

            Raises
                RuntimeError </usr/share/doc/python3/html/library/exceptions.html#RuntimeError> -- if value is None and the name cannot be found in the environment

        classmethod getWaitDuration()

        sleepSeconds(sleeptime=1)
            Helper function to drop on all state-querying functions to avoid over-querying.

        with_retries(operation, *args, **kwargs)
            Call operation with args and kwargs. If one of the calls to a command fails, sleep and try again.

  toil.batchSystems.awsBatch

Batch system for running Toil workflows on AWS Batch.

Useful with the AWS job store.

AWS Batch has no means for scheduling based on disk usage, so the backing machines need to have "enough" disk and other constraints need to guarantee that disk does not fill.

Assumes that an AWS Batch Queue name or ARN is already provided.

Handles creating and destroying a JobDefinition for the workflow run.

Additional containers should be launched with Singularity, not Docker.

  Attributes
logger |
STATE_TO_EXIT_REASON |
MAX_POLL_COUNT |
MIN_REQUESTABLE_MIB |
MIN_REQUESTABLE_CORES |

  Classes
AWSBatchBatchSystem | Adds cleanup support when the last running job leaves a node, for batch

  Module Contents

    toil.batchSystems.awsBatch.logger

    toil.batchSystems.awsBatch.STATE_TO_EXIT_REASON: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.batchSystems.abstractBatchSystem.BatchJobExitReason <#toil.batchSystems.abstractBatchSystem.BatchJobExitReason>]

    toil.batchSystems.awsBatch.MAX_POLL_COUNT = 100

    toil.batchSystems.awsBatch.MIN_REQUESTABLE_MIB = 4

    toil.batchSystems.awsBatch.MIN_REQUESTABLE_CORES = 1

    class toil.batchSystems.awsBatch.AWSBatchBatchSystem(config, maxCores, maxMemory, maxDisk)
        Bases: toil.batchSystems.cleanup_support.BatchSystemCleanupSupport <#toil.batchSystems.cleanup_support.BatchSystemCleanupSupport>

        Adds cleanup support when the last running job leaves a node, for batch systems that can't provide it using the backing scheduler.

        Parameters

            •   config (toil.common.Config <#toil.common.Config>)

            •   maxCores (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxMemory (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxDisk (int </usr/share/doc/python3/html/library/functions.html#int>)

        classmethod supportsAutoDeployment()
            Whether this batch system supports auto-deployment of the user script itself.

            If it does, the setUserScript() can be invoked to set the resource object representing the user script.

            Note to implementors: If your implementation returns True here, it should also override

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        region

        client

        queue

        job_role_arn

        owner_tag

        worker_work_dir

        user_script: toil.resource.Resource <#toil.resource.Resource> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        docker_image

        job_definition: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        bs_id_to_aws_id: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[int </usr/share/doc/python3/html/library/functions.html#int>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        aws_id_to_bs_id: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, int </usr/share/doc/python3/html/library/functions.html#int>]

        killed_job_aws_ids: set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        setUserScript(user_script)
            Set the user script for this workflow.

            This method must be called before the first job is issued to this batch system, and only if supportsAutoDeployment() returns True, otherwise it will raise an exception.

            Parameters

                •   userScript -- the resource object representing the user script or module and the modules it depends on.

                •   user_script (toil.resource.Resource <#toil.resource.Resource>)

            Return type
                None

        issueBatchJob(command, job_desc, job_environment=None)
            Issues a job with the specified command to the batch system and returns a unique job ID number.

            Parameters

                •   command (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the command to execute somewhere to run the Toil worker process

                •   job_desc (toil.job.JobDescription <#toil.job.JobDescription>) -- the JobDescription for the job being run

                •   job_environment (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) -- a collection of job-specific environment variables to be set on the worker.

            Returns
                a unique job ID number that can be used to reference the newly issued job

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        getUpdatedBatchJob(maxWait)
            Returns information about job that has updated its status (i.e. ceased running, either successfully or with an error). Each such job will be returned exactly once.

            Does not return info for jobs killed by killBatchJobs, although they may cause None to be returned earlier than maxWait.

            Parameters
                maxWait (int </usr/share/doc/python3/html/library/functions.html#int>) -- the number of seconds to block, waiting for a result

            Returns
                If a result is available, returns UpdatedBatchJobInfo. Otherwise it returns None. wallTime is the number of seconds (a strictly positive float) in wall-clock time the job ran for, or None if this batch system does not support tracking wall time.

            Return type
                Optional[toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo <#toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo>]

        shutdown()
            Called at the completion of a toil invocation. Should cleanly terminate all worker threads.

            Return type
                None

        getIssuedBatchJobIDs()
            Gets all currently issued jobs

            Returns
                A list of jobs (as job ID numbers) currently issued (may be running, or may be waiting to be run). Despite the result being a list, the ordering should not be depended upon.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>]

        getRunningBatchJobIDs()
            Gets a map of jobs as job ID numbers that are currently running (not just waiting) and how long they have been running, in seconds.

            Returns
                dictionary with currently running job ID number keys and how many seconds they have been running as the value

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>]

        killBatchJobs(job_ids)
            Kills the given job IDs. After returning, the killed jobs will not appear in the results of getRunningBatchJobIDs. The killed job will not be returned from getUpdatedBatchJob.

            Parameters

                •   jobIDs -- list of IDs of jobs to kill

                •   job_ids (list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>])

            Return type
                None

        classmethod add_options(parser)
            If this batch system provides any command line options, add them to the given parser.

            Parameters
                parser (Union[argparse.ArgumentParser </usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser>, argparse._ArgumentGroup])

            Return type
                None

        classmethod setOptions(setOption)
            Process command line or configuration options relevant to this batch system.

            Parameters
                setOption (toil.batchSystems.options.OptionSetter <#toil.batchSystems.options.OptionSetter>) -- A function with signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) returning nothing, used to update run configuration as a side effect.

            Return type
                None

  toil.batchSystems.cleanup_support

  Attributes
logger |

  Classes
BatchSystemCleanupSupport | Adds cleanup support when the last running job leaves a node, for batch
WorkerCleanupContext | Context manager used by BatchSystemCleanupSupport to implement

  Module Contents

    toil.batchSystems.cleanup_support.logger

    class toil.batchSystems.cleanup_support.BatchSystemCleanupSupport(config, maxCores, maxMemory, maxDisk)
        Bases: toil.batchSystems.local_support.BatchSystemLocalSupport <#toil.batchSystems.local_support.BatchSystemLocalSupport>

        Adds cleanup support when the last running job leaves a node, for batch systems that can't provide it using the backing scheduler.

        Parameters

            •   config (toil.common.Config <#toil.common.Config>)

            •   maxCores (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxMemory (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxDisk (int </usr/share/doc/python3/html/library/functions.html#int>)

        classmethod supportsWorkerCleanup()
            Whether this batch system supports worker cleanup.

            Indicates whether this batch system invokes BatchSystemSupport.workerCleanup() after the last job for a particular workflow invocation finishes. Note that the term worker refers to an entire node, not just a worker process. A worker process may run more than one job sequentially, and more than one concurrent worker process may exist on a worker node, for the same workflow. The batch system is said to shut down after the last worker process terminates.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        getWorkerContexts()
            Get a list of picklable context manager objects to wrap worker work in, in order.

            Can be used to ask the Toil worker to do things in-process (such as configuring environment variables, hot-deploying user scripts, or cleaning up a node) that would otherwise require a wrapping "executor" process.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[ContextManager[Any]]

    class toil.batchSystems.cleanup_support.WorkerCleanupContext(workerCleanupInfo)
        Context manager used by BatchSystemCleanupSupport to implement cleanup on a node after the last worker is done working.

        Gets wrapped around the worker's work.

        Parameters
            workerCleanupInfo (toil.batchSystems.abstractBatchSystem.WorkerCleanupInfo <#toil.batchSystems.abstractBatchSystem.WorkerCleanupInfo>)

        workerCleanupInfo

        __enter__()

            Return type
                None

        __exit__(type, value, traceback)

            Parameters

                •   type (Optional[WorkerCleanupContext.__exit__.type[BaseException </usr/share/doc/python3/html/library/exceptions.html#BaseException>]])

                •   value (Optional[BaseException </usr/share/doc/python3/html/library/exceptions.html#BaseException>])

                •   traceback (Optional[types.TracebackType </usr/share/doc/python3/html/library/types.html#types.TracebackType>])

            Return type
                None

  toil.batchSystems.contained_executor

Executor for running inside a container.

Useful for Kubernetes batch system and TES batch system plugin.

  Attributes
logger |

  Functions
pack_job(command[, user_script, environment]) | Create a command that runs the given command in an environment.
executor() | Main function of the _toil_contained_executor entrypoint.

  Module Contents

    toil.batchSystems.contained_executor.logger

    toil.batchSystems.contained_executor.pack_job(command, user_script=None, environment=None)
        Create a command that runs the given command in an environment.

        Parameters

            •   command (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Worker command to run to run the job.

            •   user_script (Optional[toil.resource.Resource <#toil.resource.Resource>]) -- User script that will be loaded before the job is run.

            •   environment (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) -- Environment variable dict that will be applied before the job is run.

        Returns
            Command to run the job, as an argument list that can be run inside the Toil appliance container.

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.batchSystems.contained_executor.executor()
        Main function of the _toil_contained_executor entrypoint.

        Runs inside the Toil container.

        Responsible for setting up the user script and running the command for the job (which may in turn invoke the Toil worker entrypoint).

        Return type
            None

  toil.batchSystems.gridengine

  Attributes
logger |

  Classes
GridEngineBatchSystem | A partial implementation of BatchSystemSupport for batch systems run on a

  Module Contents

    toil.batchSystems.gridengine.logger

    class toil.batchSystems.gridengine.GridEngineBatchSystem(config, maxCores, maxMemory, maxDisk)
        Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem <#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem>

        A partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster. By default auto-deployment is not implemented.

        Parameters

            •   config (toil.common.Config <#toil.common.Config>)

            •   maxCores (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxMemory (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxDisk (float </usr/share/doc/python3/html/library/functions.html#float>)

        class GridEngineThread(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss)
            Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.GridEngineThread <#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.GridEngineThread>

            Grid Engine-specific AbstractGridEngineWorker methods

            Parameters

                •   newJobsQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)

                •   updatedJobsQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)

                •   killQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)

                •   killedJobsQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)

                •   boss (AbstractGridEngineBatchSystem <#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem>)

            getRunningJobIDs()
                Get a list of running job IDs. Implementation-specific; called by boss AbstractGridEngineBatchSystem implementation via AbstractGridEngineBatchSystem.getRunningBatchJobIDs()

                Return type
                    list </usr/share/doc/python3/html/library/stdtypes.html#list>

            killJob(jobID)
                Kill specific job with the Toil job ID. Implementation-specific; called by GridEngineThread.killJobs()

                Parameters
                    jobID (string) -- Toil job ID

            prepareSubmission(cpu, memory, jobID, command, jobName, job_environment=None, gpus=None)
                Preparation in putting together a command-line string for submitting to batch system (via submitJob().)

                Param
                    int cpu

                Param
                    int memory

                Param
                    int jobID: Toil job ID

                Param
                    string subLine: the command line string to be called

                Param
                    string jobName: the name of the Toil job, to provide metadata to batch systems if desired

                Param
                    dict job_environment: the environment variables to be set on the worker

                Return type
                    List[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

                Parameters

                    •   cpu (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   memory (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   jobID (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   command (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                    •   jobName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                    •   job_environment (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

                    •   gpus (Optional[int </usr/share/doc/python3/html/library/functions.html#int>])

            submitJob(subLine)
                Wrapper routine for submitting the actual command-line call, then processing the output to get the batch system job ID

                Param
                    string subLine: the literal command line string to be called

                Return type
                    string: batch system job ID, which will be stored internally

            getJobExitCode(sgeJobID)
                Get job exist code, checking both qstat and qacct. Return None if still running. Higher level should retry on CalledProcessErrorStderr, for the case the job has finished and qacct result is stale.

            prepareQsub(cpu, mem, jobID, job_environment=None)

                Parameters

                    •   cpu (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   mem (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   jobID (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   job_environment (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

                Return type
                    list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        classmethod getWaitDuration()

  toil.batchSystems.htcondor

  Attributes
logger |
JobTuple |
schedd_lock |

  Classes
HTCondorBatchSystem | A partial implementation of BatchSystemSupport for batch systems run on a

  Module Contents

    toil.batchSystems.htcondor.logger

    toil.batchSystems.htcondor.JobTuple

    toil.batchSystems.htcondor.schedd_lock

    class toil.batchSystems.htcondor.HTCondorBatchSystem(config, maxCores, maxMemory, maxDisk)
        Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem <#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem>

        A partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster. By default auto-deployment is not implemented.

        Parameters

            •   config (toil.common.Config <#toil.common.Config>)

            •   maxCores (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxMemory (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxDisk (float </usr/share/doc/python3/html/library/functions.html#float>)

        class GridEngineThread(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss)
            Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.GridEngineThread <#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.GridEngineThread>

            A class that represents a thread of control.

            This class can be safely subclassed in a limited fashion. There are two ways to specify the activity: by passing a callable object to the constructor, or by overriding the run() method in a subclass.

            Parameters

                •   newJobsQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)

                •   updatedJobsQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)

                •   killQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)

                •   killedJobsQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)

                •   boss (AbstractGridEngineBatchSystem <#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem>)

            createJobs(newJob)
                Create a new job with the given attributes.

                Implementation-specific; called by GridEngineThread.run()

                Parameters
                    newJob (JobTuple)

                Return type
                    bool </usr/share/doc/python3/html/library/functions.html#bool>

            prepareSubmission(cpu, memory, disk, jobID, jobName, command, environment)
                Preparation in putting together a command-line string for submitting to batch system (via submitJob().)

                Param
                    int cpu

                Param
                    int memory

                Param
                    int jobID: Toil job ID

                Param
                    string subLine: the command line string to be called

                Param
                    string jobName: the name of the Toil job, to provide metadata to batch systems if desired

                Param
                    dict job_environment: the environment variables to be set on the worker

                Return type
                    List[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

                Parameters

                    •   cpu (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   memory (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   disk (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   jobID (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   jobName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                    •   command (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                    •   environment (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            submitJob(submitObj)
                Wrapper routine for submitting the actual command-line call, then processing the output to get the batch system job ID

                Param
                    string subLine: the literal command line string to be called

                Return type
                    string: batch system job ID, which will be stored internally

            getRunningJobIDs()
                Get a list of running job IDs. Implementation-specific; called by boss AbstractGridEngineBatchSystem implementation via AbstractGridEngineBatchSystem.getRunningBatchJobIDs()

                Return type
                    list </usr/share/doc/python3/html/library/stdtypes.html#list>

            killJob(jobID)
                Kill specific job with the Toil job ID. Implementation-specific; called by GridEngineThread.killJobs()

                Parameters
                    jobID (string) -- Toil job ID

            getJobExitCode(batchJobID)
                Returns job exit code and possibly an instance of abstractBatchSystem.BatchJobExitReason.

                Returns None if the job is still running.

                If the job is not running but the exit code is not available, it will be EXIT_STATUS_UNAVAILABLE_VALUE. Implementation-specific; called by GridEngineThread.checkOnJobs().

                The exit code will only be 0 if the job affirmatively succeeded.

                Parameters
                    batchjobID (string) -- batch system job ID

            connectSchedd()
                Connect to HTCondor Schedd and yield a Schedd object.

                You can only use it inside the context. Handles locking to make sure that only one thread is trying to do this at a time.

            duplicate_quotes(value)
                Escape a string by doubling up all single and double quotes.

                This is used for arguments we pass to htcondor that need to be inside both double and single quote enclosures.

                Parameters
                    value (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                Return type
                    str </usr/share/doc/python3/html/library/stdtypes.html#str>

            getEnvString(overrides)
                Build an environment string that a HTCondor Submit object can use.

                For examples of valid strings, see: <http://research.cs.wisc.edu/htcondor/manual/current/condor_submit.html#man-condor-submit-environment>

                Parameters
                    overrides (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                Return type
                    str </usr/share/doc/python3/html/library/stdtypes.html#str>

        issueBatchJob(command, jobNode, job_environment=None)
            Issues a job with the specified command to the batch system and returns a unique job ID number.

            Parameters

                •   command (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the command to execute somewhere to run the Toil worker process

                •   job_desc -- the JobDescription for the job being run

                •   job_environment (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) -- a collection of job-specific environment variables to be set on the worker.

            Returns
                a unique job ID number that can be used to reference the newly issued job

  toil.batchSystems.kubernetes

Batch system for running Toil workflows on Kubernetes.

Ony useful with network-based job stores, like AWSJobStore.

Within non-privileged Kubernetes containers, additional Docker containers cannot yet be launched. That functionality will need to wait for user-mode Docker

  Attributes
logger |
retryable_kubernetes_errors |
KeyValuesList |

  Classes
KubernetesBatchSystem | Adds cleanup support when the last running job leaves a node, for batch

  Functions
is_retryable_kubernetes_error(e) | A function that determines whether or not Toil should retry or stop given

  Module Contents

    toil.batchSystems.kubernetes.logger

    toil.batchSystems.kubernetes.retryable_kubernetes_errors: list </usr/share/doc/python3/html/library/stdtypes.html#list>[type </usr/share/doc/python3/html/library/functions.html#type>[Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>] | toil.lib.retry.ErrorCondition <#toil.lib.retry.ErrorCondition>]

    toil.batchSystems.kubernetes.is_retryable_kubernetes_error(e)
        A function that determines whether or not Toil should retry or stop given exceptions thrown by Kubernetes.

        Parameters
            e (Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>)

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.batchSystems.kubernetes.KeyValuesList

    class toil.batchSystems.kubernetes.KubernetesBatchSystem(config, maxCores, maxMemory, maxDisk)
        Bases: toil.batchSystems.cleanup_support.BatchSystemCleanupSupport <#toil.batchSystems.cleanup_support.BatchSystemCleanupSupport>

        Adds cleanup support when the last running job leaves a node, for batch systems that can't provide it using the backing scheduler.

        Parameters

            •   config (toil.common.Config <#toil.common.Config>)

            •   maxCores (int </usr/share/doc/python3/html/library/functions.html#int>)

            •   maxMemory (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxDisk (int </usr/share/doc/python3/html/library/functions.html#int>)

        classmethod supportsAutoDeployment()
            Whether this batch system supports auto-deployment of the user script itself.

            If it does, the setUserScript() can be invoked to set the resource object representing the user script.

            Note to implementors: If your implementation returns True here, it should also override

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        credential_time: datetime.datetime </usr/share/doc/python3/html/library/datetime.html#datetime.datetime> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        namespace: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        host_path: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        service_account: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        pod_timeout: float </usr/share/doc/python3/html/library/functions.html#float>

        unique_id

        job_prefix: str </usr/share/doc/python3/html/library/stdtypes.html#str> = 'Uninferable-toil-Instance of uuid.UUID-'

        finished_job_ttl: int </usr/share/doc/python3/html/library/functions.html#int> = 3600

        user_script: toil.resource.Resource <#toil.resource.Resource> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        docker_image: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        worker_work_dir: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        aws_secret_name: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        enable_watching: bool </usr/share/doc/python3/html/library/functions.html#bool>

        run_id: str </usr/share/doc/python3/html/library/stdtypes.html#str> = 'toil-Instance of uuid.UUID'

        resource_sources: list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.batchSystems.abstractBatchSystem.ResourcePool <#toil.batchSystems.abstractBatchSystem.ResourcePool>]

        schedulingThread: threading.Thread </usr/share/doc/python3/html/library/threading.html#threading.Thread>

        class DecoratorWrapper(to_wrap, decorator)
            Class to wrap an object so all its methods are decorated.

            Parameters

                •   to_wrap (Any)

                •   decorator (Callable[[Callable[P, Any]], Callable[P, Any]])

            P

            __getattr__(name)
                Get a member as if we are actually the wrapped object. If it looks callable, we will decorate it.

                Parameters
                    name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                Return type
                    Any

        ItemT

        CovItemT

        P

        R

        setUserScript(userScript)
            Set the user script for this workflow.

            This method must be called before the first job is issued to this batch system, and only if supportsAutoDeployment() returns True, otherwise it will raise an exception.

            Parameters
                userScript (toil.resource.Resource <#toil.resource.Resource>) -- the resource object representing the user script or module and the modules it depends on.

            Return type
                None

        class Placement
            Internal format for pod placement constraints and preferences.

            required_labels: KeyValuesList = []
                Labels which are required to be present (with these values).

            desired_labels: KeyValuesList = []
                Labels which are optional, but preferred to be present (with these values).

            prohibited_labels: KeyValuesList = []
                Labels which are not allowed to be present (with these values).

            tolerated_taints: KeyValuesList = []
                Taints which are allowed to be present (with these values).

            set_preemptible(preemptible)
                Add constraints for a job being preemptible or not.

                Preemptible jobs will be able to run on preemptible or non-preemptible nodes, and will prefer preemptible nodes if available.

                Non-preemptible jobs will not be allowed to run on nodes that are marked as preemptible.

                Understands the labeling scheme used by EKS, and the taint scheme used by GCE. The Toil-managed Kubernetes setup will mimic at least one of these.

                Parameters
                    preemptible (bool </usr/share/doc/python3/html/library/functions.html#bool>)

                Return type
                    None

            apply(pod_spec)
                Set affinity and/or tolerations fields on pod_spec, so that it runs on the right kind of nodes for the constraints we represent.

                Parameters
                    pod_spec (kubernetes.client.V1PodSpec)

                Return type
                    None

        class FakeResponse

            data: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        T

        issueBatchJob(command, job_desc, job_environment=None)
            Issues a job with the specified command to the batch system and returns a unique job ID number.

            Parameters

                •   command (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the command to execute somewhere to run the Toil worker process

                •   job_desc (toil.job.JobDescription <#toil.job.JobDescription>) -- the JobDescription for the job being run

                •   job_environment (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) -- a collection of job-specific environment variables to be set on the worker.

            Returns
                a unique job ID number that can be used to reference the newly issued job

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        getUpdatedBatchJob(maxWait)
            Returns information about job that has updated its status (i.e. ceased running, either successfully or with an error). Each such job will be returned exactly once.

            Does not return info for jobs killed by killBatchJobs, although they may cause None to be returned earlier than maxWait.

            Parameters
                maxWait (float </usr/share/doc/python3/html/library/functions.html#float>) -- the number of seconds to block, waiting for a result

            Returns
                If a result is available, returns UpdatedBatchJobInfo. Otherwise it returns None. wallTime is the number of seconds (a strictly positive float) in wall-clock time the job ran for, or None if this batch system does not support tracking wall time.

            Return type
                Optional[toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo <#toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo>]

        shutdown()
            Called at the completion of a toil invocation. Should cleanly terminate all worker threads.

            Return type
                None

        getIssuedBatchJobIDs()
            Gets all currently issued jobs

            Returns
                A list of jobs (as job ID numbers) currently issued (may be running, or may be waiting to be run). Despite the result being a list, the ordering should not be depended upon.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>]

        getRunningBatchJobIDs()
            Gets a map of jobs as job ID numbers that are currently running (not just waiting) and how long they have been running, in seconds.

            Returns
                dictionary with currently running job ID number keys and how many seconds they have been running as the value

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>]

        killBatchJobs(jobIDs)
            Kills the given job IDs. After returning, the killed jobs will not appear in the results of getRunningBatchJobIDs. The killed job will not be returned from getUpdatedBatchJob.

            Parameters
                jobIDs (list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>]) -- list of IDs of jobs to kill

            Return type
                None

        classmethod get_default_kubernetes_owner()
            Get the default Kubernetes-acceptable username string to tack onto jobs.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        class KubernetesConfig
            Bases: Protocol

            Type-enforcing protocol for Toil configs that have the extra Kubernetes batch system fields.

            TODO: Until MyPY lets protocols inherit form non-protocols, we will have to let the fact that this also has to be a Config just be manually enforced.

            kubernetes_host_path: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

            kubernetes_owner: str </usr/share/doc/python3/html/library/stdtypes.html#str>

            kubernetes_service_account: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

            kubernetes_pod_timeout: float </usr/share/doc/python3/html/library/functions.html#float>

        classmethod add_options(parser)
            If this batch system provides any command line options, add them to the given parser.

            Parameters
                parser (Union[argparse.ArgumentParser </usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser>, argparse._ArgumentGroup])

            Return type
                None

        OptionType

        classmethod setOptions(setOption)
            Process command line or configuration options relevant to this batch system.

            Parameters
                setOption (toil.batchSystems.options.OptionSetter <#toil.batchSystems.options.OptionSetter>) -- A function with signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) returning nothing, used to update run configuration as a side effect.

            Return type
                None

  toil.batchSystems.local_support

  Attributes
logger |

  Classes
BatchSystemLocalSupport | Adds a local queue for helper jobs, useful for CWL & others.

  Module Contents

    toil.batchSystems.local_support.logger

    class toil.batchSystems.local_support.BatchSystemLocalSupport(config, maxCores, maxMemory, maxDisk)
        Bases: toil.batchSystems.abstractBatchSystem.BatchSystemSupport <#toil.batchSystems.abstractBatchSystem.BatchSystemSupport>

        Adds a local queue for helper jobs, useful for CWL & others.

        Parameters

            •   config (toil.common.Config <#toil.common.Config>)

            •   maxCores (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxMemory (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxDisk (int </usr/share/doc/python3/html/library/functions.html#int>)

        localBatch: toil.batchSystems.singleMachine.SingleMachineBatchSystem <#toil.batchSystems.singleMachine.SingleMachineBatchSystem>

        handleLocalJob(command, jobDesc)
            To be called by issueBatchJob.

            Returns the jobID if the jobDesc has been submitted to the local queue, otherwise returns None

            Parameters

                •   command (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   jobDesc (toil.job.JobDescription <#toil.job.JobDescription>)

            Return type
                Optional[int </usr/share/doc/python3/html/library/functions.html#int>]

        killLocalJobs(jobIDs)
            Will kill all local jobs that match the provided jobIDs.

            To be called by killBatchJobs.

            Parameters
                jobIDs (list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>])

            Return type
                None

        getIssuedLocalJobIDs()
            To be called by getIssuedBatchJobIDs.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>]

        getRunningLocalJobIDs()
            To be called by getRunningBatchJobIDs().

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>]

        getUpdatedLocalJob(maxWait)
            To be called by getUpdatedBatchJob().

            Parameters
                maxWait (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                Optional[toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo <#toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo>]

        getNextJobID()
            Must be used to get job IDs so that the local and batch jobs do not conflict.

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        shutdownLocal()
            To be called from shutdown().

            Return type
                None

  toil.batchSystems.lsf

  Attributes
logger |

  Classes
LSFBatchSystem | A partial implementation of BatchSystemSupport for batch systems run on a

  Module Contents

    toil.batchSystems.lsf.logger

    class toil.batchSystems.lsf.LSFBatchSystem(config, maxCores, maxMemory, maxDisk)
        Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem <#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem>

        A partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster. By default auto-deployment is not implemented.

        Parameters

            •   config (toil.common.Config <#toil.common.Config>)

            •   maxCores (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxMemory (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxDisk (float </usr/share/doc/python3/html/library/functions.html#float>)

        class GridEngineThread(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss)
            Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.GridEngineThread <#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.GridEngineThread>

            LSF specific GridEngineThread methods.

            Parameters

                •   newJobsQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)

                •   updatedJobsQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)

                •   killQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)

                •   killedJobsQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)

                •   boss (AbstractGridEngineBatchSystem <#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem>)

            getRunningJobIDs()
                Get a list of running job IDs. Implementation-specific; called by boss AbstractGridEngineBatchSystem implementation via AbstractGridEngineBatchSystem.getRunningBatchJobIDs()

                Return type
                    list </usr/share/doc/python3/html/library/stdtypes.html#list>

            fallbackRunningJobIDs(currentjobs)

            killJob(jobID)
                Kill specific job with the Toil job ID. Implementation-specific; called by GridEngineThread.killJobs()

                Parameters
                    jobID (string) -- Toil job ID

            prepareSubmission(cpu, memory, jobID, command, jobName, job_environment=None, gpus=None)
                Preparation in putting together a command-line string for submitting to batch system (via submitJob().)

                Param
                    int cpu

                Param
                    int memory

                Param
                    int jobID: Toil job ID

                Param
                    string subLine: the command line string to be called

                Param
                    string jobName: the name of the Toil job, to provide metadata to batch systems if desired

                Param
                    dict job_environment: the environment variables to be set on the worker

                Return type
                    List[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

                Parameters

                    •   cpu (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   memory (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   jobID (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   command (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                    •   jobName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                    •   job_environment (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

                    •   gpus (Optional[int </usr/share/doc/python3/html/library/functions.html#int>])

            submitJob(subLine)
                Wrapper routine for submitting the actual command-line call, then processing the output to get the batch system job ID

                Param
                    string subLine: the literal command line string to be called

                Return type
                    string: batch system job ID, which will be stored internally

            coalesce_job_exit_codes(batch_job_id_list)
                Returns exit codes and possibly exit reasons for a list of jobs, or None if they are running.

                Called by GridEngineThread.checkOnJobs().

                The default implementation falls back on self.getJobExitCode and polls each job individually

                Parameters
                    batch_job_id_list (string) -- List of batch system job ID

                Return type
                    list </usr/share/doc/python3/html/library/stdtypes.html#list>

            getJobExitCode(lsfJobID)
                Returns job exit code and possibly an instance of abstractBatchSystem.BatchJobExitReason.

                Returns None if the job is still running.

                If the job is not running but the exit code is not available, it will be EXIT_STATUS_UNAVAILABLE_VALUE. Implementation-specific; called by GridEngineThread.checkOnJobs().

                The exit code will only be 0 if the job affirmatively succeeded.

                Parameters
                    batchjobID (string) -- batch system job ID

                Return type
                    Union[int </usr/share/doc/python3/html/library/functions.html#int>, tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[int </usr/share/doc/python3/html/library/functions.html#int>, Optional[toil.batchSystems.abstractBatchSystem.BatchJobExitReason <#toil.batchSystems.abstractBatchSystem.BatchJobExitReason>]], None]

            parse_bjobs_record(bjobs_record, job)
                Helper functions for getJobExitCode and to parse the bjobs status record

                Parameters

                    •   bjobs_record (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>)

                    •   job (int </usr/share/doc/python3/html/library/functions.html#int>)

                Return type
                    Union[int </usr/share/doc/python3/html/library/functions.html#int>, tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[int </usr/share/doc/python3/html/library/functions.html#int>, Optional[toil.batchSystems.abstractBatchSystem.BatchJobExitReason <#toil.batchSystems.abstractBatchSystem.BatchJobExitReason>]], None]

            getJobExitCodeBACCT(job)

                Return type
                    Union[int </usr/share/doc/python3/html/library/functions.html#int>, tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[int </usr/share/doc/python3/html/library/functions.html#int>, Optional[toil.batchSystems.abstractBatchSystem.BatchJobExitReason <#toil.batchSystems.abstractBatchSystem.BatchJobExitReason>]], None]

            fallbackGetJobExitCode(job)

                Return type
                    Union[int </usr/share/doc/python3/html/library/functions.html#int>, tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[int </usr/share/doc/python3/html/library/functions.html#int>, Optional[toil.batchSystems.abstractBatchSystem.BatchJobExitReason <#toil.batchSystems.abstractBatchSystem.BatchJobExitReason>]], None]

            prepareBsub(cpu, mem, jobID)
                Make a bsub commandline to execute.

                params:
                    cpu: number of cores needed mem: number of bytes of memory needed jobID: ID number of the job

                Parameters

                    •   cpu (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   mem (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   jobID (int </usr/share/doc/python3/html/library/functions.html#int>)

                Return type
                    list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

            parseBjobs(bjobs_output_str)
                Parse records from bjobs json type output

                Params bjobs_output_str
                    stdout of bjobs json type output

            parseMaxMem(jobID)
                Parse the maximum memory from job.

                Parameters
                    jobID -- ID number of the job

        getWaitDuration()
            We give LSF a second to catch its breath (in seconds)

  toil.batchSystems.lsfHelper

  Attributes
LSB_PARAMS_FILENAME |
LSF_CONF_FILENAME |
LSF_CONF_ENV |
DEFAULT_LSF_UNITS |
DEFAULT_RESOURCE_UNITS |
LSF_JSON_OUTPUT_MIN_VERSION |
logger |

  Functions
find(basedir, string) | walk basedir and return all files matching string
find_first_match(basedir, string) | return the first file that matches string starting from basedir
get_conf_file(filename, env) |
apply_conf_file(fn, conf_filename) |
per_core_reserve_from_stream(stream) |
get_lsf_units_from_stream(stream) |
tokenize_conf_stream(conf_handle) | convert the key=val pairs in a LSF config stream to tuples of tokens
apply_bparams(fn) | apply fn to each line of bparams, returning the result
apply_lsadmin(fn) | apply fn to each line of lsadmin, returning the result
get_lsf_units([resource]) | check if we can find LSF_UNITS_FOR_LIMITS in lsadmin and lsf.conf
parse_mem_and_cmd_from_output(output) | Use regex to find "MAX MEM" and "Command" inside of an output.
get_lsf_version() | Get current LSF version
check_lsf_json_output_supported() | Check if the current LSF system supports bjobs json output.
parse_memory(mem) | Parse memory parameter.
per_core_reservation() | returns True if the cluster is configured for reservations to be per core,

  Module Contents

    toil.batchSystems.lsfHelper.LSB_PARAMS_FILENAME = 'lsb.params'

    toil.batchSystems.lsfHelper.LSF_CONF_FILENAME = 'lsf.conf'

    toil.batchSystems.lsfHelper.LSF_CONF_ENV = ['LSF_CONFDIR', 'LSF_ENVDIR']

    toil.batchSystems.lsfHelper.DEFAULT_LSF_UNITS = 'KB'

    toil.batchSystems.lsfHelper.DEFAULT_RESOURCE_UNITS = 'MB'

    toil.batchSystems.lsfHelper.LSF_JSON_OUTPUT_MIN_VERSION = '10.1.0.2'

    toil.batchSystems.lsfHelper.logger

    toil.batchSystems.lsfHelper.find(basedir, string)
        walk basedir and return all files matching string

    toil.batchSystems.lsfHelper.find_first_match(basedir, string)
        return the first file that matches string starting from basedir

    toil.batchSystems.lsfHelper.get_conf_file(filename, env)

    toil.batchSystems.lsfHelper.apply_conf_file(fn, conf_filename)

    toil.batchSystems.lsfHelper.per_core_reserve_from_stream(stream)

    toil.batchSystems.lsfHelper.get_lsf_units_from_stream(stream)

    toil.batchSystems.lsfHelper.tokenize_conf_stream(conf_handle)
        convert the key=val pairs in a LSF config stream to tuples of tokens

    toil.batchSystems.lsfHelper.apply_bparams(fn)
        apply fn to each line of bparams, returning the result

    toil.batchSystems.lsfHelper.apply_lsadmin(fn)
        apply fn to each line of lsadmin, returning the result

    toil.batchSystems.lsfHelper.get_lsf_units(resource=False)
        check if we can find LSF_UNITS_FOR_LIMITS in lsadmin and lsf.conf files, preferring the value in bparams, then lsadmin, then the lsf.conf file

        Parameters
            resource (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.batchSystems.lsfHelper.parse_mem_and_cmd_from_output(output)
        Use regex to find "MAX MEM" and "Command" inside of an output.

        Parameters
            output (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

    toil.batchSystems.lsfHelper.get_lsf_version()
        Get current LSF version

    toil.batchSystems.lsfHelper.check_lsf_json_output_supported()
        Check if the current LSF system supports bjobs json output.

    toil.batchSystems.lsfHelper.parse_memory(mem)
        Parse memory parameter.

        Parameters
            mem (float </usr/share/doc/python3/html/library/functions.html#float>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.batchSystems.lsfHelper.per_core_reservation()
        returns True if the cluster is configured for reservations to be per core, False if it is per job

  toil.batchSystems.mesos

  Submodules

  toil.batchSystems.mesos.batchSystem

  Attributes
log |

  Classes
MesosBatchSystem | A Toil batch system implementation that uses Apache Mesos to distribute toil jobs as Mesos

  Module Contents

    toil.batchSystems.mesos.batchSystem.log

    class toil.batchSystems.mesos.batchSystem.MesosBatchSystem(config, maxCores, maxMemory, maxDisk)
        Bases: toil.batchSystems.local_support.BatchSystemLocalSupport <#toil.batchSystems.local_support.BatchSystemLocalSupport>, toil.batchSystems.abstractBatchSystem.AbstractScalableBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractScalableBatchSystem>, pymesos.Scheduler

        A Toil batch system implementation that uses Apache Mesos to distribute toil jobs as Mesos tasks over a cluster of agent nodes. A Mesos framework consists of a scheduler and an executor. This class acts as the scheduler and is typically run on the master node that also runs the Mesos master process with which the scheduler communicates via a driver component. The executor is implemented in a separate class. It is run on each agent node and communicates with the Mesos agent process via another driver object. The scheduler may also be run on a separate node from the master, which we then call somewhat ambiguously the driver node.

        Parameters

            •   config (toil.common.Config <#toil.common.Config>)

            •   maxCores (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxMemory (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxDisk (int </usr/share/doc/python3/html/library/functions.html#int>)

        classmethod supportsAutoDeployment()
            Whether this batch system supports auto-deployment of the user script itself.

            If it does, the setUserScript() can be invoked to set the resource object representing the user script.

            Note to implementors: If your implementation returns True here, it should also override

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        classmethod supportsWorkerCleanup()
            Whether this batch system supports worker cleanup.

            Indicates whether this batch system invokes BatchSystemSupport.workerCleanup() after the last job for a particular workflow invocation finishes. Note that the term worker refers to an entire node, not just a worker process. A worker process may run more than one job sequentially, and more than one concurrent worker process may exist on a worker node, for the same workflow. The batch system is said to shut down after the last worker process terminates.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        class ExecutorInfo(nodeAddress, agentId, nodeInfo, lastSeen)

            Parameters

                •   nodeAddress (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   agentId (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   nodeInfo (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   lastSeen (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            nodeAddress

            agentId

            nodeInfo

            lastSeen

        userScript = None

            Type
                toil.resource.Resource <#toil.resource.Resource>

        jobQueues

        mesos_endpoint

        mesos_name

        mesos_timeout = 60

        killedJobIds

        killJobIds

        intendedKill

        hostToJobIDs

        nodeFilter = []

        runningJobMap

        taskResources

        updatedJobsQueue

        driver = None

        frameworkId = None

        executors

        agentsByID

        nonPreemptibleNodes

        executor

        lastTimeOfferLogged = 0

        logPeriod = 30

        ignoredNodes

        setUserScript(userScript)
            Set the user script for this workflow.

            This method must be called before the first job is issued to this batch system, and only if supportsAutoDeployment() returns True, otherwise it will raise an exception.

            Parameters
                userScript (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the resource object representing the user script or module and the modules it depends on.

            Return type
                None

        ignoreNode(nodeAddress)
            Stop sending jobs to this node. Used in autoscaling when the autoscaler is ready to terminate a node, but jobs are still running. This allows the node to be terminated after the current jobs have finished.

            Parameters
                nodeAddress (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- IP address of node to ignore.

            Return type
                None

        unignoreNode(nodeAddress)
            Stop ignoring this address, presumably after a node with this address has been terminated. This allows for the possibility of a new node having the same address as a terminated one.

            Parameters
                nodeAddress (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        issueBatchJob(command, jobNode, job_environment=None)
            Issues the following command returning a unique jobID. Command is the string to run, memory is an int giving the number of bytes the job needs to run in and cores is the number of cpus needed for the job and error-file is the path of the file to place any std-err/std-out in.

            Parameters

                •   command (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   jobNode (toil.job.JobDescription <#toil.job.JobDescription>)

                •   job_environment (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        killBatchJobs(jobIDs)
            Kills the given job IDs. After returning, the killed jobs will not appear in the results of getRunningBatchJobIDs. The killed job will not be returned from getUpdatedBatchJob.

            Parameters
                jobIDs -- list of IDs of jobs to kill

        getIssuedBatchJobIDs()
            Gets all currently issued jobs

            Returns
                A list of jobs (as job ID numbers) currently issued (may be running, or may be waiting to be run). Despite the result being a list, the ordering should not be depended upon.

        getRunningBatchJobIDs()
            Gets a map of jobs as job ID numbers that are currently running (not just waiting) and how long they have been running, in seconds.

            Returns
                dictionary with currently running job ID number keys and how many seconds they have been running as the value

        getUpdatedBatchJob(maxWait)
            Returns information about job that has updated its status (i.e. ceased running, either successfully or with an error). Each such job will be returned exactly once.

            Does not return info for jobs killed by killBatchJobs, although they may cause None to be returned earlier than maxWait.

            Parameters
                maxWait -- the number of seconds to block, waiting for a result

            Returns
                If a result is available, returns UpdatedBatchJobInfo. Otherwise it returns None. wallTime is the number of seconds (a strictly positive float) in wall-clock time the job ran for, or None if this batch system does not support tracking wall time.

        nodeInUse(nodeIP)
            Can be used to determine if a worker node is running any tasks. If the node is doesn't exist, this function should simply return False.

            Parameters
                nodeIP (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The worker nodes private IP address

            Returns
                True if the worker node has been issued any tasks, else False

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        getWaitDuration()
            Gets the period of time to wait (floating point, in seconds) between checking for missing/overlong jobs.

        shutdown()
            Called at the completion of a toil invocation. Should cleanly terminate all worker threads.

            Return type
                None

        registered(driver, frameworkId, masterInfo)
            Invoked when the scheduler successfully registers with a Mesos master

        error(driver, message)
            Invoked when Mesos reports an unrecoverable error.

        resourceOffers(driver, offers)
            Invoked when resources have been offered to this framework.

        statusUpdate(driver, update)
            Invoked when the status of a task has changed (e.g., a agent is lost and so the task is lost, a task finishes and an executor sends a status update saying so, etc). Note that returning from this callback _acknowledges_ receipt of this status update! If for whatever reason the scheduler aborts during this callback (or the process exits) another status update will be delivered (note, however, that this is currently not true if the agent sending the status update is lost/fails during that time).

        frameworkMessage(driver, executorId, agentId, message)
            Invoked when an executor sends a message.

        getNodes(preemptible=None, timeout=None)

            Return all nodes that match:

                •   preemptible status (None includes all)

                •   timeout period (seen within the last # seconds, or None for all)

            Parameters

                •   preemptible (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>])

                •   timeout (Optional[int </usr/share/doc/python3/html/library/functions.html#int>])

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.batchSystems.abstractBatchSystem.NodeInfo <#toil.batchSystems.abstractBatchSystem.NodeInfo>]

        reregistered(driver, masterInfo)
            Invoked when the scheduler re-registers with a newly elected Mesos master.

        executorLost(driver, executorId, agentId, status)
            Invoked when an executor has exited/terminated abnormally.

        classmethod get_default_mesos_endpoint()
            Get the default IP/hostname and port that we will look for Mesos at.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        classmethod add_options(parser)
            If this batch system provides any command line options, add them to the given parser.

            Parameters
                parser (Union[argparse.ArgumentParser </usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser>, argparse._ArgumentGroup])

            Return type
                None

        classmethod setOptions(setOption)
            Process command line or configuration options relevant to this batch system.

            Parameters
                setOption (toil.batchSystems.options.OptionSetter <#toil.batchSystems.options.OptionSetter>) -- A function with signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) returning nothing, used to update run configuration as a side effect.

  toil.batchSystems.mesos.conftest

  Attributes
collect_ignore |

  Module Contents

    toil.batchSystems.mesos.conftest.collect_ignore = []

  toil.batchSystems.mesos.executor

  Attributes
log |

  Classes
MesosExecutor | Part of Toil's Mesos framework, runs on a Mesos agent. A Toil job is passed to it via the

  Functions
main() |

  Module Contents

    toil.batchSystems.mesos.executor.log

    class toil.batchSystems.mesos.executor.MesosExecutor
        Bases: pymesos.Executor

        Part of Toil's Mesos framework, runs on a Mesos agent. A Toil job is passed to it via the task.data field, and launched via call(toil.command).

        popenLock

        runningTasks

        workerCleanupInfo = None

        address = None

        id = None

        registered(driver, executorInfo, frameworkInfo, agentInfo)
            Invoked once the executor driver has been able to successfully connect with Mesos.

        reregistered(driver, agentInfo)
            Invoked when the executor re-registers with a restarted agent.

        disconnected(driver)
            Invoked when the executor becomes "disconnected" from the agent (e.g., the agent is being restarted due to an upgrade).

        killTask(driver, taskId)
            Kill parent task process and all its spawned children

        shutdown(driver)

        error(driver, message)
            Invoked when a fatal error has occurred with the executor and/or executor driver.

        launchTask(driver, task)
            Invoked by SchedulerDriver when a Mesos task should be launched by this executor

        frameworkMessage(driver, message)
            Invoked when a framework message has arrived for this executor.

    toil.batchSystems.mesos.executor.main()

  toil.batchSystems.mesos.test

  Attributes
log |

  Classes
MesosTestSupport | Mixin for test cases that need a running Mesos master and agent on the local host.

  Package Contents

    toil.batchSystems.mesos.test.log

    class toil.batchSystems.mesos.test.MesosTestSupport
        Mixin for test cases that need a running Mesos master and agent on the local host.

        wait_for_master()

        class MesosThread(numCores)
            Bases: toil.lib.threading.ExceptionalThread <#toil.lib.threading.ExceptionalThread>

            A thread whose join() method re-raises exceptions raised during run(). While join() is idempotent, the exception is only during the first invocation of join() that successfully joined the thread. If join() times out, no exception will be re reraised even though an exception might already have occurred in run().

            When subclassing this thread, override tryRun() instead of run().

                >>> def f():
                ...     assert 0
                >>> t = ExceptionalThread(target=f)
                >>> t.start()
                >>> t.join()
                Traceback (most recent call last):
                ...
                AssertionError

                >>> class MyThread(ExceptionalThread):
                ...     def tryRun( self ):
                ...         assert 0
                >>> t = MyThread()
                >>> t.start()
                >>> t.join()
                Traceback (most recent call last):
                ...
                AssertionError

            lock

            numCores

            abstractmethod mesosCommand()

            tryRun()

            findMesosBinary(names)

        class MesosMasterThread(numCores)
            Bases: MesosThread

            A thread whose join() method re-raises exceptions raised during run(). While join() is idempotent, the exception is only during the first invocation of join() that successfully joined the thread. If join() times out, no exception will be re reraised even though an exception might already have occurred in run().

            When subclassing this thread, override tryRun() instead of run().

                >>> def f():
                ...     assert 0
                >>> t = ExceptionalThread(target=f)
                >>> t.start()
                >>> t.join()
                Traceback (most recent call last):
                ...
                AssertionError

                >>> class MyThread(ExceptionalThread):
                ...     def tryRun( self ):
                ...         assert 0
                >>> t = MyThread()
                >>> t.start()
                >>> t.join()
                Traceback (most recent call last):
                ...
                AssertionError

            mesosCommand()

        class MesosAgentThread(numCores)
            Bases: MesosThread

            A thread whose join() method re-raises exceptions raised during run(). While join() is idempotent, the exception is only during the first invocation of join() that successfully joined the thread. If join() times out, no exception will be re reraised even though an exception might already have occurred in run().

            When subclassing this thread, override tryRun() instead of run().

                >>> def f():
                ...     assert 0
                >>> t = ExceptionalThread(target=f)
                >>> t.start()
                >>> t.join()
                Traceback (most recent call last):
                ...
                AssertionError

                >>> class MyThread(ExceptionalThread):
                ...     def tryRun( self ):
                ...         assert 0
                >>> t = MyThread()
                >>> t.start()
                >>> t.join()
                Traceback (most recent call last):
                ...
                AssertionError

            mesosCommand()

  Classes
TaskData |
JobQueue |
MesosShape | Represents a job or a node's "shape", in terms of the dimensions of memory, cores, disk and
ToilJob |

  Package Contents

    class toil.batchSystems.mesos.TaskData
        Bases: tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>

        startTime

        agentID

        agentIP

        executorID

        memory

        cores

    class toil.batchSystems.mesos.JobQueue

        queues

        sortedTypes = []

        jobLock

        insertJob(job, jobType)

        jobIDs()

        nextJobOfType(jobType)

        typeEmpty(jobType)

    class toil.batchSystems.mesos.MesosShape(wallTime, memory, cores, disk, preemptible)
        Bases: toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>

        Represents a job or a node's "shape", in terms of the dimensions of memory, cores, disk and wall-time allocation.

        The wallTime attribute stores the number of seconds of a node allocation, e.g. 3600 for AWS. FIXME: and for jobs?

        The memory and disk attributes store the number of bytes required by a job (or provided by a node) in RAM or on disk (SSD or HDD), respectively.

        Parameters

            •   wallTime (Union[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>])

            •   memory (int </usr/share/doc/python3/html/library/functions.html#int>)

            •   cores (Union[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>])

            •   disk (int </usr/share/doc/python3/html/library/functions.html#int>)

            •   preemptible (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        __gt__(other)
            Inverted. Returns True if self is less than other, else returns False.

            This is because jobTypes are sorted in decreasing order, and this was done to give expensive jobs priority.

    class toil.batchSystems.mesos.ToilJob
        Bases: tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>

        jobID

        name

        resources

        command

        userScript

        environment

        workerCleanupInfo

  toil.batchSystems.options

  Attributes
logger |

  Classes
OptionSetter | Protocol for the setOption function we get to let us set up CLI options for

  Functions
set_batchsystem_options(batch_system, set_option) | Call set_option for all the options for the given named batch system, or
add_all_batchsystem_options(parser) |

  Module Contents

    toil.batchSystems.options.logger

    class toil.batchSystems.options.OptionSetter
        Bases: Protocol

        Protocol for the setOption function we get to let us set up CLI options for each batch system.

        Actual functionality is defined in the Config class.

        OptionType

        __call__(option_name, parsing_function=None, check_function=None, default=None, env=None, old_names=None)

            Parameters

                •   option_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   parsing_function (Optional[Callable[[Any], OptionType]])

                •   check_function (Optional[Callable[[OptionType], Union[None, bool </usr/share/doc/python3/html/library/functions.html#bool>]]])

                •   default (Optional[OptionType])

                •   env (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

                •   old_names (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.batchSystems.options.set_batchsystem_options(batch_system, set_option)
        Call set_option for all the options for the given named batch system, or all batch systems if no name is provided.

        Parameters

            •   batch_system (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   set_option (OptionSetter <#toil.batchSystems.options.OptionSetter>)

        Return type
            None

    toil.batchSystems.options.add_all_batchsystem_options(parser)

        Parameters
            parser (Union[argparse.ArgumentParser </usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser>, argparse._ArgumentGroup])

        Return type
            None

  toil.batchSystems.registry

  Attributes
logger |
DEFAULT_BATCH_SYSTEM |

  Functions
add_batch_system_factory(key, class_factory) | Adds a batch system to the registry for workflow or plugin-supplied batch systems.
get_batch_systems() | Get the names of all the available batch systems.
get_batch_system(key) | Get a batch system class by name.
aws_batch_batch_system_factory() |
gridengine_batch_system_factory() |
lsf_batch_system_factory() |
single_machine_batch_system_factory() |
mesos_batch_system_factory() |
slurm_batch_system_factory() |
torque_batch_system_factory() |
htcondor_batch_system_factory() |
kubernetes_batch_system_factory() |

  Module Contents

    toil.batchSystems.registry.logger

    toil.batchSystems.registry.add_batch_system_factory(key, class_factory)
        Adds a batch system to the registry for workflow or plugin-supplied batch systems.

        Parameters

            •   class_factory (Callable[[], type </usr/share/doc/python3/html/library/functions.html#type>[toil.batchSystems.abstractBatchSystem.AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>]]) -- A function that returns a batch system class (NOT an instance), which implements toil.batchSystems.abstractBatchSystem.AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>.

            •   key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

    toil.batchSystems.registry.get_batch_systems()
        Get the names of all the available batch systems.

        Return type
            collections.abc.Sequence </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Sequence>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.batchSystems.registry.get_batch_system(key)
        Get a batch system class by name.

        Raises
            KeyError if the key is not the name of a batch system, and ImportError if the batch system's class cannot be loaded.

        Parameters
            key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            type </usr/share/doc/python3/html/library/functions.html#type>[toil.batchSystems.abstractBatchSystem.AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>]

    toil.batchSystems.registry.DEFAULT_BATCH_SYSTEM = 'single_machine'

    toil.batchSystems.registry.aws_batch_batch_system_factory()

    toil.batchSystems.registry.gridengine_batch_system_factory()

    toil.batchSystems.registry.lsf_batch_system_factory()

    toil.batchSystems.registry.single_machine_batch_system_factory()

    toil.batchSystems.registry.mesos_batch_system_factory()

    toil.batchSystems.registry.slurm_batch_system_factory()

    toil.batchSystems.registry.torque_batch_system_factory()

    toil.batchSystems.registry.htcondor_batch_system_factory()

    toil.batchSystems.registry.kubernetes_batch_system_factory()

  toil.batchSystems.singleMachine

  Attributes
logger |

  Classes
SingleMachineBatchSystem | The interface for running jobs on a single machine, runs all the jobs you
Info | Record for a running job.

  Module Contents

    toil.batchSystems.singleMachine.logger

    class toil.batchSystems.singleMachine.SingleMachineBatchSystem(config, maxCores, maxMemory, maxDisk, max_jobs=None)
        Bases: toil.batchSystems.abstractBatchSystem.BatchSystemSupport <#toil.batchSystems.abstractBatchSystem.BatchSystemSupport>

        The interface for running jobs on a single machine, runs all the jobs you give it as they come in, but in parallel.

        Uses a single "daddy" thread to manage a fleet of child processes.

        Communication with the daddy thread happens via two queues: one queue of jobs waiting to be run (the input queue), and one queue of jobs that are finished/stopped and need to be returned by getUpdatedBatchJob (the output queue).

        When the batch system is shut down, the daddy thread is stopped.

        If running in debug-worker mode, jobs are run immediately as they are sent to the batch system, in the sending thread, and the daddy thread is not run. But the queues are still used.

        Parameters

            •   config (toil.common.Config <#toil.common.Config>)

            •   maxCores (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxMemory (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxDisk (int </usr/share/doc/python3/html/library/functions.html#int>)

            •   max_jobs (Optional[int </usr/share/doc/python3/html/library/functions.html#int>])

        classmethod supportsAutoDeployment()
            Whether this batch system supports auto-deployment of the user script itself.

            If it does, the setUserScript() can be invoked to set the resource object representing the user script.

            Note to implementors: If your implementation returns True here, it should also override

        classmethod supportsWorkerCleanup()
            Whether this batch system supports worker cleanup.

            Indicates whether this batch system invokes BatchSystemSupport.workerCleanup() after the last job for a particular workflow invocation finishes. Note that the term worker refers to an entire node, not just a worker process. A worker process may run more than one job sequentially, and more than one concurrent worker process may exist on a worker node, for the same workflow. The batch system is said to shut down after the last worker process terminates.

        numCores

        minCores = 0.1
            The minimal fractional CPU. Tasks with a smaller core requirement will be rounded up to this value.

        physicalMemory

        config

        physicalDisk

        scale

        debugWorker

        jobIndex = 1

        jobIndexLock

        jobs: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[int </usr/share/doc/python3/html/library/functions.html#int>, toil.job.JobDescription <#toil.job.JobDescription>]

        inputQueue

        outputQueue

        runningJobs: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[int </usr/share/doc/python3/html/library/functions.html#int>, Info <#toil.batchSystems.singleMachine.Info>]

        children: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[int </usr/share/doc/python3/html/library/functions.html#int>, subprocess.Popen </usr/share/doc/python3/html/library/subprocess.html#subprocess.Popen>]

        childToJob: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[int </usr/share/doc/python3/html/library/functions.html#int>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        accelerator_identities

        resource_sources

        schedulingStatusMessage = None

        shuttingDown

        daddyThread = None

        daddyException: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        daddy()
            Be the "daddy" thread.

            Our job is to look at jobs from the input queue.

            If a job fits in the available resources, we allocate resources for it and kick off a child process.

            We also check on our children.

            When a child finishes, we reap it, release its resources, and put its information in the output queue.

        getSchedulingStatusMessage()
            Get a log message fragment for the user about anything that might be going wrong in the batch system, if available.

            If no useful message is available, return None.

            This can be used to report what resource is the limiting factor when scheduling jobs, for example. If the leader thinks the workflow is stuck, the message can be displayed to the user to help them diagnose why it might be stuck.

            Returns
                User-directed message about scheduling state.

        check_resource_request(requirer)
            Check resource request is not greater than that available or allowed.

            Parameters

                •   requirer (toil.job.Requirer <#toil.job.Requirer>) -- Object whose requirements are being checked

                •   job_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Name of the job being checked, for generating a useful error report.

                •   detail (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Batch-system-specific message to include in the error.

            Raises
                InsufficientSystemResources <#toil.batchSystems.abstractBatchSystem.InsufficientSystemResources> -- raised when a resource is requested in an amount greater than allowed

            Return type
                None

        issueBatchJob(command, job_desc, job_environment=None)
            Adds the command and resources to a queue to be run.

            Parameters

                •   command (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   job_desc (toil.job.JobDescription <#toil.job.JobDescription>)

                •   job_environment (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        killBatchJobs(jobIDs)
            Kills jobs by ID.

            Parameters
                jobIDs (list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>])

            Return type
                None

        getIssuedBatchJobIDs()
            Just returns all the jobs that have been run, but not yet returned as updated.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>]

        getRunningBatchJobIDs()
            Gets a map of jobs as job ID numbers that are currently running (not just waiting) and how long they have been running, in seconds.

            Returns
                dictionary with currently running job ID number keys and how many seconds they have been running as the value

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>]

        shutdown()
            Terminate cleanly and join daddy thread.

            Return type
                None

        getUpdatedBatchJob(maxWait)
            Returns a tuple of a no-longer-running job, the return value of its process, and its runtime, or None.

            Parameters
                maxWait (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                Optional[toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo <#toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo>]

        classmethod add_options(parser)
            If this batch system provides any command line options, add them to the given parser.

            Parameters
                parser (Union[argparse.ArgumentParser </usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser>, argparse._ArgumentGroup])

            Return type
                None

        classmethod setOptions(setOption)
            Process command line or configuration options relevant to this batch system.

            Parameters
                setOption (toil.batchSystems.options.OptionSetter <#toil.batchSystems.options.OptionSetter>) -- A function with signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) returning nothing, used to update run configuration as a side effect.

    class toil.batchSystems.singleMachine.Info(startTime, popen, resources, killIntended)
        Record for a running job.

        Stores the start time of the job, the Popen object representing its child (or None), the tuple of (coreFractions, memory, disk) it is using (or None), and whether the job is supposed to be being killed.

        time

        popen

        resources

        killIntended

  toil.batchSystems.slurm

  Attributes
logger |
TERMINAL_STATES |
NONTERMINAL_STATES |

  Classes
SlurmBatchSystem | A partial implementation of BatchSystemSupport for batch systems run on a

  Functions
parse_slurm_time(slurm_time) | Parse a Slurm-style time duration like 7-00:00:00 to a number of seconds.
option_detector(long[, short]) | Get a function that returns true if it sees the long or short
any_option_detector(options) | Get a function that returns true if it sees any of the long

  Module Contents

    toil.batchSystems.slurm.logger

    toil.batchSystems.slurm.TERMINAL_STATES: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.batchSystems.abstractBatchSystem.BatchJobExitReason <#toil.batchSystems.abstractBatchSystem.BatchJobExitReason>]

    toil.batchSystems.slurm.NONTERMINAL_STATES: set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.batchSystems.slurm.parse_slurm_time(slurm_time)
        Parse a Slurm-style time duration like 7-00:00:00 to a number of seconds.

        Raises ValueError if not parseable.

        Parameters
            slurm_time (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            int </usr/share/doc/python3/html/library/functions.html#int>

    toil.batchSystems.slurm.option_detector(long, short=None)
        Get a function that returns true if it sees the long or short option.

        Parameters

            •   long (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   short (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None)

        Return type
            Callable[[str </usr/share/doc/python3/html/library/stdtypes.html#str>], bool </usr/share/doc/python3/html/library/functions.html#bool>]

    toil.batchSystems.slurm.any_option_detector(options)
        Get a function that returns true if it sees any of the long options or long or short option pairs.

        Parameters
            options (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str> | tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

        Return type
            Callable[[str </usr/share/doc/python3/html/library/stdtypes.html#str>], bool </usr/share/doc/python3/html/library/functions.html#bool>]

    class toil.batchSystems.slurm.SlurmBatchSystem(config, maxCores, maxMemory, maxDisk)
        Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem <#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem>

        A partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster. By default auto-deployment is not implemented.

        Parameters

            •   config (toil.common.Config <#toil.common.Config>)

            •   maxCores (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxMemory (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxDisk (float </usr/share/doc/python3/html/library/functions.html#float>)

        class PartitionInfo
            Bases: NamedTuple

            partition_name: str </usr/share/doc/python3/html/library/stdtypes.html#str>

            gres: bool </usr/share/doc/python3/html/library/functions.html#bool>

            time_limit: float </usr/share/doc/python3/html/library/functions.html#float>

            priority: int </usr/share/doc/python3/html/library/functions.html#int>

            cpus: str </usr/share/doc/python3/html/library/stdtypes.html#str>

            memory: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        class PartitionSet
            Set of available partitions detected on the slurm batch system

            default_gpu_partition: SlurmBatchSystem <#toil.batchSystems.slurm.SlurmBatchSystem> | None </usr/share/doc/python3/html/library/constants.html#None>

            all_partitions: list </usr/share/doc/python3/html/library/stdtypes.html#list>[SlurmBatchSystem <#toil.batchSystems.slurm.SlurmBatchSystem>]

            gpu_partitions: set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

            get_partition(time_limit)
                Get the partition name to use for a job with the given time limit.

                Parameters
                    time_limit (float </usr/share/doc/python3/html/library/functions.html#float> | None) -- Time limit in seconds.

                Return type
                    str </usr/share/doc/python3/html/library/stdtypes.html#str> | None

        class GridEngineThread(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss)
            Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.GridEngineThread <#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.GridEngineThread>

            A class that represents a thread of control.

            This class can be safely subclassed in a limited fashion. There are two ways to specify the activity: by passing a callable object to the constructor, or by overriding the run() method in a subclass.

            Parameters

                •   newJobsQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)

                •   updatedJobsQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)

                •   killQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)

                •   killedJobsQueue (queue.Queue </usr/share/doc/python3/html/library/queue.html#queue.Queue>)

                •   boss (AbstractGridEngineBatchSystem <#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem>)

            boss: SlurmBatchSystem <#toil.batchSystems.slurm.SlurmBatchSystem>

            getRunningJobIDs()
                Get a list of running job IDs. Implementation-specific; called by boss AbstractGridEngineBatchSystem implementation via AbstractGridEngineBatchSystem.getRunningBatchJobIDs()

                Return type
                    list </usr/share/doc/python3/html/library/stdtypes.html#list>

            killJob(jobID)
                Kill specific job with the Toil job ID. Implementation-specific; called by GridEngineThread.killJobs()

                Parameters
                    jobID (string) -- Toil job ID

                Return type
                    None

            prepareSubmission(cpu, memory, jobID, command, jobName, job_environment=None, gpus=None)
                Preparation in putting together a command-line string for submitting to batch system (via submitJob().)

                Param
                    int cpu

                Param
                    int memory

                Param
                    int jobID: Toil job ID

                Param
                    string subLine: the command line string to be called

                Param
                    string jobName: the name of the Toil job, to provide metadata to batch systems if desired

                Param
                    dict job_environment: the environment variables to be set on the worker

                Return type
                    List[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

                Parameters

                    •   cpu (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   memory (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   jobID (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   command (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                    •   jobName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                    •   job_environment (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None)

                    •   gpus (int </usr/share/doc/python3/html/library/functions.html#int> | None)

            submitJob(subLine)
                Wrapper routine for submitting the actual command-line call, then processing the output to get the batch system job ID

                Param
                    string subLine: the literal command line string to be called

                Return type
                    string: batch system job ID, which will be stored internally

                Parameters
                    subLine (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            coalesce_job_exit_codes(batch_job_id_list)
                Collect all job exit codes in a single call.

                Parameters
                    batch_job_id_list (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- list of Job ID strings, where each string has the form <job>[.<task>].

                Returns
                    list of job exit codes or exit code, exit reason pairs associated with the list of job IDs.

                Raises

                    •   CalledProcessErrorStderr <#toil.lib.misc.CalledProcessErrorStderr> -- if communicating with Slurm went wrong.

                    •   OSError </usr/share/doc/python3/html/library/exceptions.html#OSError> -- if job details are not available because a Slurm command could not start.

                Return type
                    list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int> | tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[int </usr/share/doc/python3/html/library/functions.html#int>, toil.batchSystems.abstractBatchSystem.BatchJobExitReason <#toil.batchSystems.abstractBatchSystem.BatchJobExitReason> | None] | None]

            getJobExitCode(batchJobID)
                Get job exit code for given batch job ID. :param batchJobID: string of the form "<job>[.<task>]". :return: integer job exit code.

                Parameters
                    batchJobID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                Return type
                    int </usr/share/doc/python3/html/library/functions.html#int> | tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[int </usr/share/doc/python3/html/library/functions.html#int>, toil.batchSystems.abstractBatchSystem.BatchJobExitReason <#toil.batchSystems.abstractBatchSystem.BatchJobExitReason> | None] | None

            prepareSbatch(cpu, mem, jobID, jobName, job_environment, gpus)
                Returns the sbatch command line to run to queue the job.

                Parameters

                    •   cpu (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   mem (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   jobID (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   jobName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                    •   job_environment (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None)

                    •   gpus (int </usr/share/doc/python3/html/library/functions.html#int> | None)

                Return type
                    list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        partitions

        start_time

        issueBatchJob(command, job_desc, job_environment=None)
            Issues a job with the specified command to the batch system and returns a unique job ID number.

            Parameters

                •   command (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the command to execute somewhere to run the Toil worker process

                •   job_desc (toil.job.JobDescription <#toil.job.JobDescription>) -- the JobDescription for the job being run

                •   job_environment (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None) -- a collection of job-specific environment variables to be set on the worker.

            Returns
                a unique job ID number that can be used to reference the newly issued job

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        classmethod add_options(parser)
            If this batch system provides any command line options, add them to the given parser.

            Parameters
                parser (argparse.ArgumentParser </usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser> | argparse._ArgumentGroup)

            Return type
                None

        OptionType

        classmethod setOptions(setOption)
            Process command line or configuration options relevant to this batch system.

            Parameters
                setOption (toil.batchSystems.options.OptionSetter <#toil.batchSystems.options.OptionSetter>) -- A function with signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) returning nothing, used to update run configuration as a side effect.

            Return type
                None

  toil.batchSystems.torque

  Attributes
logger |

  Classes
TorqueBatchSystem | A partial implementation of BatchSystemSupport for batch systems run on a

  Module Contents

    toil.batchSystems.torque.logger

    class toil.batchSystems.torque.TorqueBatchSystem(config, maxCores, maxMemory, maxDisk)
        Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem <#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem>

        A partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster. By default auto-deployment is not implemented.

        Parameters

            •   config (toil.common.Config <#toil.common.Config>)

            •   maxCores (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxMemory (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxDisk (float </usr/share/doc/python3/html/library/functions.html#float>)

        class GridEngineThread(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss)
            Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.GridEngineThread <#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.GridEngineThread>

            A class that represents a thread of control.

            This class can be safely subclassed in a limited fashion. There are two ways to specify the activity: by passing a callable object to the constructor, or by overriding the run() method in a subclass.

            getRunningJobIDs()
                Get a list of running job IDs. Implementation-specific; called by boss AbstractGridEngineBatchSystem implementation via AbstractGridEngineBatchSystem.getRunningBatchJobIDs()

                Return type
                    list </usr/share/doc/python3/html/library/stdtypes.html#list>

            getUpdatedBatchJob(maxWait)

            killJob(jobID)
                Kill specific job with the Toil job ID. Implementation-specific; called by GridEngineThread.killJobs()

                Parameters
                    jobID (string) -- Toil job ID

            prepareSubmission(cpu, memory, jobID, command, jobName, job_environment=None, gpus=None)
                Preparation in putting together a command-line string for submitting to batch system (via submitJob().)

                Param
                    int cpu

                Param
                    int memory

                Param
                    int jobID: Toil job ID

                Param
                    string subLine: the command line string to be called

                Param
                    string jobName: the name of the Toil job, to provide metadata to batch systems if desired

                Param
                    dict job_environment: the environment variables to be set on the worker

                Return type
                    List[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

                Parameters

                    •   cpu (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   memory (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   jobID (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   command (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                    •   jobName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                    •   job_environment (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

                    •   gpus (Optional[int </usr/share/doc/python3/html/library/functions.html#int>])

            submitJob(subLine)
                Wrapper routine for submitting the actual command-line call, then processing the output to get the batch system job ID

                Param
                    string subLine: the literal command line string to be called

                Return type
                    string: batch system job ID, which will be stored internally

            getJobExitCode(torqueJobID)
                Returns job exit code and possibly an instance of abstractBatchSystem.BatchJobExitReason.

                Returns None if the job is still running.

                If the job is not running but the exit code is not available, it will be EXIT_STATUS_UNAVAILABLE_VALUE. Implementation-specific; called by GridEngineThread.checkOnJobs().

                The exit code will only be 0 if the job affirmatively succeeded.

                Parameters
                    batchjobID (string) -- batch system job ID

            prepareQsub(cpu, mem, jobID, job_environment)

                Parameters

                    •   cpu (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   mem (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   jobID (int </usr/share/doc/python3/html/library/functions.html#int>)

                    •   job_environment (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

                Return type
                    list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

            generateTorqueWrapper(command, jobID)
                A very simple script generator that just wraps the command given; for now this goes to default tempdir

  Exceptions
DeadlockException | Exception thrown by the Leader or BatchSystem when a deadlock is encountered due to insufficient

  Package Contents

    exception toil.batchSystems.DeadlockException(msg)
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Exception thrown by the Leader or BatchSystem when a deadlock is encountered due to insufficient resources to run the workflow

        msg = 'Deadlock encountered: Uninferable'

        __str__()
            Stringify the exception, including the message.

  toil.bus

Message types and message bus for leader component coordination.

Historically, the Toil Leader has been organized around functions calling other functions to "handle" different things happening. Over time, it has become very brittle: exactly the right handling functions need to be called in exactly the right order, or it gets confused and does the wrong thing.

The MessageBus is meant to let the leader avoid this by more loosely coupling its components together, by having them communicate by sending messages instead of by calling functions.

When events occur (like a job coming back from the batch system with a failed exit status), this will be translated into a message that will be sent to the bus. Then, all the leader components that need to react to this message in some way (by, say, decrementing the retry count) would listen for the relevant messages on the bus and react to them. If a new component needs to be added, it can be plugged into the message bus and receive and react to messages without interfering with existing components' ability to react to the same messages.

Eventually, the different aspects of the Leader could become separate objects.

By default, messages stay entirely within the Toil leader process, and are not persisted anywhere, not even in the JobStore.

The Message Bus also provides an extension point: its messages can be serialized to a file by the leader (see the --writeMessages option), and they can then be decoded using MessageBus.scan_bus_messages() (as is done in the Toil WES server backend). By replaying the messages and tracking their effects on job state, you can get an up-to-date view of the state of the jobs in a workflow. This includes information, such as whether jobs are issued or running, or what jobs have completely finished, which is not persisted in the JobStore.

The MessageBus instance for the leader process is owned by the Toil leader, but the BatchSystem has an opportunity to connect to it, and can send (or listen for) messages. Right now the BatchSystem deos not have to send or receive any messages; the Leader is responsible for polling it via the BatchSystem API and generating the events. But a BatchSystem implementation may send additional events (like JobAnnotationMessage).

Currently, the MessageBus is implemented using pypubsub, and so messages are always handled in a single Thread, the Toil leader's main loop thread. If other components send events, they will be shipped over to that thread inside the MessageBus. Communication between processes is allowed using MessageBus.connect_output_file() and MessageBus.scan_bus_messages().

  Attributes
logger |
MessageType |

  Classes
Names | Stores all the kinds of name a job can have.
JobIssuedMessage | Produced when a job is issued to run on the batch system.
JobUpdatedMessage | Produced when a job is "updated" and ready to have something happen to it.
JobCompletedMessage | Produced when a job is completed, whether successful or not.
JobFailedMessage | Produced when a job is completely failed, and will not be retried again.
JobMissingMessage | Produced when a job goes missing and should be in the batch system but isn't.
JobAnnotationMessage | Produced when extra information (such as an AWS Batch job ID from the
ExternalBatchIdMessage | Produced when using a batch system, links toil assigned batch ID to
QueueSizeMessage | Produced to describe the size of the queue of jobs issued but not yet
ClusterSizeMessage | Produced by the Toil-integrated autoscaler describe the number of
ClusterDesiredSizeMessage | Produced by the Toil-integrated autoscaler to describe the number of
MessageBus | Holds messages that should cause jobs to change their scheduling states.
MessageBusClient | Base class for clients (inboxes and outboxes) of a message bus. Handles
MessageInbox | A buffered connection to a message bus that lets us receive messages.
MessageOutbox | A connection to a message bus that lets us publish messages.
MessageBusConnection | A two-way connection to a message bus. Buffers incoming messages until you
JobStatus | Records the status of a job.

  Functions
get_job_kind(names) | Return an identifying string for the job.
message_to_bytes(message) | Convert a plain-old-data named tuple into a byte string.
bytes_to_message(message_type, data) | Convert bytes from message_to_bytes back to a message of the given type.
replay_message_bus(path) | Replay all the messages and work out what they mean for jobs.
gen_message_bus_path([tmpdir]) | Return a file path in tmp to store the message bus at.

  Module Contents

    toil.bus.logger

    class toil.bus.Names
        Bases: NamedTuple

        Stores all the kinds of name a job can have.

        job_name: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        unit_name: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        display_name: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        stats_name: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        job_store_id: str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.bus.get_job_kind(names)
        Return an identifying string for the job.

        The result may contain spaces.

        Returns: Either the unit name, job name, or display name, which identifies
            the kind of job it is to toil. Otherwise "Unknown Job" in case no identifier is available

        Parameters
            names (Names <#toil.bus.Names>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    class toil.bus.JobIssuedMessage
        Bases: NamedTuple

        Produced when a job is issued to run on the batch system.

        job_type: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        job_id: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        toil_batch_id: int </usr/share/doc/python3/html/library/functions.html#int>

    class toil.bus.JobUpdatedMessage
        Bases: NamedTuple

        Produced when a job is "updated" and ready to have something happen to it.

        job_id: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        result_status: int </usr/share/doc/python3/html/library/functions.html#int>

    class toil.bus.JobCompletedMessage
        Bases: NamedTuple

        Produced when a job is completed, whether successful or not.

        job_type: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        job_id: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        exit_code: int </usr/share/doc/python3/html/library/functions.html#int>

    class toil.bus.JobFailedMessage
        Bases: NamedTuple

        Produced when a job is completely failed, and will not be retried again.

        job_type: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        job_id: str </usr/share/doc/python3/html/library/stdtypes.html#str>

    class toil.bus.JobMissingMessage
        Bases: NamedTuple

        Produced when a job goes missing and should be in the batch system but isn't.

        job_id: str </usr/share/doc/python3/html/library/stdtypes.html#str>

    class toil.bus.JobAnnotationMessage
        Bases: NamedTuple

        Produced when extra information (such as an AWS Batch job ID from the AWSBatchBatchSystem) is available that goes with a job.

        job_id: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        annotation_name: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        annotation_value: str </usr/share/doc/python3/html/library/stdtypes.html#str>

    class toil.bus.ExternalBatchIdMessage
        Bases: NamedTuple

        Produced when using a batch system, links toil assigned batch ID to Batch system ID (Whatever's returned by local implementation, PID, batch ID, etc)

        toil_batch_id: int </usr/share/doc/python3/html/library/functions.html#int>

        external_batch_id: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        batch_system: str </usr/share/doc/python3/html/library/stdtypes.html#str>

    class toil.bus.QueueSizeMessage
        Bases: NamedTuple

        Produced to describe the size of the queue of jobs issued but not yet completed. Theoretically recoverable from other messages.

        queue_size: int </usr/share/doc/python3/html/library/functions.html#int>

    class toil.bus.ClusterSizeMessage
        Bases: NamedTuple

        Produced by the Toil-integrated autoscaler describe the number of instances of a certain type in a cluster.

        instance_type: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        current_size: int </usr/share/doc/python3/html/library/functions.html#int>

    class toil.bus.ClusterDesiredSizeMessage
        Bases: NamedTuple

        Produced by the Toil-integrated autoscaler to describe the number of instances of a certain type that it thinks will be needed.

        instance_type: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        desired_size: int </usr/share/doc/python3/html/library/functions.html#int>

    toil.bus.message_to_bytes(message)
        Convert a plain-old-data named tuple into a byte string.

        Parameters
            message (NamedTuple)

        Return type
            bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>

    toil.bus.MessageType

    toil.bus.bytes_to_message(message_type, data)
        Convert bytes from message_to_bytes back to a message of the given type.

        Parameters

            •   message_type (type </usr/share/doc/python3/html/library/functions.html#type>[MessageType])

            •   data (bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>)

        Return type
            MessageType

    class toil.bus.MessageBus
        Holds messages that should cause jobs to change their scheduling states. Messages are put in and buffered, and can be taken out and handled as batches when convenient.

        All messages are NamedTuple objects of various subtypes.

        Message order is guaranteed to be preserved within a type.

        publish(message)
            Put a message onto the bus. Can be called from any thread.

            Parameters
                message (Any)

            Return type
                None

        check()
            If we are in the owning thread, deliver any messages that are in the queue for us. Must be called every once in a while in the main thread, possibly through inbox objects.

            Return type
                None

        MessageType

        subscribe(message_type, handler)
            Register the given callable to be called when messages of the given type are sent. It will be called with messages sent after the subscription is created. Returns a subscription object; when the subscription object is GC'd the subscription will end.

            Parameters

                •   message_type (type </usr/share/doc/python3/html/library/functions.html#type>[MessageType])

                •   handler (Callable[[MessageType], Any])

            Return type
                pubsub.core.listener.Listener

        connect(wanted_types)
            Get a connection object that serves as an inbox for messages of the given types. Messages of those types will accumulate in the inbox until it is destroyed. You can check for them at any time.

            Parameters
                wanted_types (list </usr/share/doc/python3/html/library/stdtypes.html#list>[type </usr/share/doc/python3/html/library/functions.html#type>])

            Return type
                MessageBusConnection <#toil.bus.MessageBusConnection>

        outbox()
            Get a connection object that only allows sending messages.

            Return type
                MessageOutbox <#toil.bus.MessageOutbox>

        connect_output_file(file_path)
            Send copies of all messages to the given output file.

            Returns connection data which must be kept alive for the connection to persist. That data is opaque: the user is not supposed to look at it or touch it or do anything with it other than store it somewhere or delete it.

            Parameters
                file_path (toil.lib.misc.FileDescriptorOrPath)

            Return type
                Any

        classmethod scan_bus_messages(stream, message_types)
            Get an iterator over all messages in the given log stream of the given types, in order. Discard any trailing partial messages.

            Parameters

                •   stream (IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>])

                •   message_types (list </usr/share/doc/python3/html/library/stdtypes.html#list>[type </usr/share/doc/python3/html/library/functions.html#type>[NamedTuple]])

            Return type
                collections.abc.Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[Any]

    class toil.bus.MessageBusClient
        Base class for clients (inboxes and outboxes) of a message bus. Handles keeping a reference to the message bus.

    class toil.bus.MessageInbox
        Bases: MessageBusClient

        A buffered connection to a message bus that lets us receive messages. Buffers incoming messages until you are ready for them. Does not preserve ordering between messages of different types.

        count(message_type)
            Get the number of pending messages of the given type.

            Parameters
                message_type (type </usr/share/doc/python3/html/library/functions.html#type>)

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        empty()
            Return True if no messages are pending, and false otherwise.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        MessageType

        for_each(message_type)
            Loop over all messages currently pending of the given type. Each that is handled without raising an exception will be removed.

            Messages sent while this function is running will not be yielded by the current call.

            Parameters
                message_type (type </usr/share/doc/python3/html/library/functions.html#type>[MessageType])

            Return type
                collections.abc.Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[MessageType]

    class toil.bus.MessageOutbox
        Bases: MessageBusClient

        A connection to a message bus that lets us publish messages.

        publish(message)
            Publish the given message to the connected message bus.

            We have this so you don't need to store both the bus and your connection.

            Parameters
                message (Any)

            Return type
                None

    class toil.bus.MessageBusConnection
        Bases: MessageInbox, MessageOutbox

        A two-way connection to a message bus. Buffers incoming messages until you are ready for them, and lets you send messages.

    class toil.bus.JobStatus
        Records the status of a job.

        When exit_code is -1, this means the job is either not observed or currently running.

        job_store_id: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        name: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        exit_code: int </usr/share/doc/python3/html/library/functions.html#int>

        annotations: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        toil_batch_id: int </usr/share/doc/python3/html/library/functions.html#int>

        external_batch_id: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        batch_system: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        __repr__()

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        is_running()

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.bus.replay_message_bus(path)
        Replay all the messages and work out what they mean for jobs.

        We track the state and name of jobs here, by ID. We would use a list of two items but MyPy can't understand a list of items of multiple types, so we need to define a new class.

        Returns a dictionary from the job_id to a dataclass, JobStatus. A JobStatus contains information about a job which we have gathered from the message bus, including the job store id, name of the job the exit code, any associated annotations, the toil batch id the external batch id, and the batch system on which the job is running.

        Parameters
            path (toil.lib.misc.FileDescriptorOrPath)

        Return type
            dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, JobStatus <#toil.bus.JobStatus>]

    toil.bus.gen_message_bus_path(tmpdir=None)
        Return a file path in tmp to store the message bus at. Calling function is responsible for cleaning the generated file.

        The tmpdir argument will override the directory that the message bus will be made in. If not provided, the standard tempfile order will be used.

        Parameters
            tmpdir (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

  toil.common

  Attributes
UUID_LENGTH |
logger |

  Exceptions
ToilRestartException | Common base class for all non-exit exceptions.
ToilContextManagerException | Common base class for all non-exit exceptions.

  Classes
Config | Class to represent configuration operations for a toil workflow run.
Toil | A context manager that represents a Toil workflow.
ToilMetrics |

  Functions
get_default_config_path() | Get the default path where the Toil configuration file lives.
ensure_config(filepath) | If the config file at the filepath does not exist, create it.
generate_config(filepath) | Write a Toil config file to the given path.
update_config(filepath, key, new_value) | Set the given top-level key to the given value in the given YAML config
parser_with_common_options([provisioner_options, ...]) | Get a command-line option parser for a Toil subcommand.
addOptions(parser[, jobstore_as_flag, cwl, wdl]) | Add all Toil command line options to a parser.
getNodeID() | Return unique ID of the current node (host). The resulting string will be convertible to a uuid.UUID.
cacheDirName(workflowID) |
getDirSizeRecursively(dirPath) | This method will return the cumulative number of bytes occupied by the files
getFileSystemSize(dirPath) | Return the free space, and total size of the file system hosting dirPath.
safeUnpickleFromStream(stream) |

  Module Contents

    toil.common.UUID_LENGTH = 32

    toil.common.logger

    toil.common.get_default_config_path()
        Get the default path where the Toil configuration file lives.

        The file at the path will not necessarily exist.

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    class toil.common.Config
        Class to represent configuration operations for a toil workflow run.

        logFile: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        logRotating: bool </usr/share/doc/python3/html/library/functions.html#bool>

        cleanWorkDir: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        max_jobs: int </usr/share/doc/python3/html/library/functions.html#int>

        max_local_jobs: int </usr/share/doc/python3/html/library/functions.html#int>

        manualMemArgs: bool </usr/share/doc/python3/html/library/functions.html#bool>

        run_local_jobs_on_workers: bool </usr/share/doc/python3/html/library/functions.html#bool>

        coalesceStatusCalls: bool </usr/share/doc/python3/html/library/functions.html#bool>

        mesos_endpoint: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        mesos_framework_id: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        mesos_role: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        mesos_name: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        kubernetes_host_path: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        kubernetes_owner: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        kubernetes_service_account: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        kubernetes_pod_timeout: float </usr/share/doc/python3/html/library/functions.html#float>

        kubernetes_privileged: bool </usr/share/doc/python3/html/library/functions.html#bool>

        kubernetes_pod_security_context: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        kubernetes_security_context: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        tes_endpoint: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        tes_user: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        tes_password: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        tes_bearer_token: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        aws_batch_region: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        aws_batch_queue: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        aws_batch_job_role_arn: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        scale: float </usr/share/doc/python3/html/library/functions.html#float>

        batchSystem: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        batch_logs_dir: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>
            The backing scheduler will be instructed, if possible, to save logs to this directory, where the leader can read them.

        statePollingWait: float </usr/share/doc/python3/html/library/functions.html#float>

        state_polling_timeout: int </usr/share/doc/python3/html/library/functions.html#int>

        disableAutoDeployment: bool </usr/share/doc/python3/html/library/functions.html#bool>

        workflowID: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>
            This attribute uniquely identifies the job store and therefore the workflow. It is necessary in order to distinguish between two consecutive workflows for which self.jobStore is the same, e.g. when a job store name is reused after a previous run has finished successfully and its job store has been clean up.

        workflowAttemptNumber: int </usr/share/doc/python3/html/library/functions.html#int>

        jobStore: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        logLevel: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        colored_logs: bool </usr/share/doc/python3/html/library/functions.html#bool>

        workDir: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        coordination_dir: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        noStdOutErr: bool </usr/share/doc/python3/html/library/functions.html#bool>

        stats: bool </usr/share/doc/python3/html/library/functions.html#bool>

        clean: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        clusterStats: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        restart: bool </usr/share/doc/python3/html/library/functions.html#bool>

        caching: bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None>

        symlinkImports: bool </usr/share/doc/python3/html/library/functions.html#bool>

        moveOutputs: bool </usr/share/doc/python3/html/library/functions.html#bool>

        symlink_job_store_reads: bool </usr/share/doc/python3/html/library/functions.html#bool>

        provisioner: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        nodeTypes: list </usr/share/doc/python3/html/library/stdtypes.html#list>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>], float </usr/share/doc/python3/html/library/functions.html#float> | None </usr/share/doc/python3/html/library/constants.html#None>]]

        minNodes: list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>]

        maxNodes: list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>]

        targetTime: float </usr/share/doc/python3/html/library/functions.html#float>

        betaInertia: float </usr/share/doc/python3/html/library/functions.html#float>

        scaleInterval: int </usr/share/doc/python3/html/library/functions.html#int>

        preemptibleCompensation: float </usr/share/doc/python3/html/library/functions.html#float>

        nodeStorage: int </usr/share/doc/python3/html/library/functions.html#int>

        nodeStorageOverrides: list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        metrics: bool </usr/share/doc/python3/html/library/functions.html#bool>

        assume_zero_overhead: bool </usr/share/doc/python3/html/library/functions.html#bool>

        maxPreemptibleServiceJobs: int </usr/share/doc/python3/html/library/functions.html#int>

        maxServiceJobs: int </usr/share/doc/python3/html/library/functions.html#int>

        deadlockWait: float </usr/share/doc/python3/html/library/functions.html#float> | int </usr/share/doc/python3/html/library/functions.html#int>

        deadlockCheckInterval: float </usr/share/doc/python3/html/library/functions.html#float> | int </usr/share/doc/python3/html/library/functions.html#int>

        defaultMemory: int </usr/share/doc/python3/html/library/functions.html#int>

        defaultCores: float </usr/share/doc/python3/html/library/functions.html#float> | int </usr/share/doc/python3/html/library/functions.html#int>

        defaultDisk: int </usr/share/doc/python3/html/library/functions.html#int>

        defaultPreemptible: bool </usr/share/doc/python3/html/library/functions.html#bool>

        defaultAccelerators: list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.job.AcceleratorRequirement <#toil.job.AcceleratorRequirement>]

        maxCores: int </usr/share/doc/python3/html/library/functions.html#int>

        maxMemory: int </usr/share/doc/python3/html/library/functions.html#int>

        maxDisk: int </usr/share/doc/python3/html/library/functions.html#int>

        retryCount: int </usr/share/doc/python3/html/library/functions.html#int>

        stop_on_first_failure: bool </usr/share/doc/python3/html/library/functions.html#bool>

        enableUnlimitedPreemptibleRetries: bool </usr/share/doc/python3/html/library/functions.html#bool>

        doubleMem: bool </usr/share/doc/python3/html/library/functions.html#bool>

        maxJobDuration: int </usr/share/doc/python3/html/library/functions.html#int>

        rescueJobsFrequency: int </usr/share/doc/python3/html/library/functions.html#int>

        job_store_timeout: float </usr/share/doc/python3/html/library/functions.html#float>

        maxLogFileSize: int </usr/share/doc/python3/html/library/functions.html#int>

        writeLogs: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        writeLogsGzip: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        writeLogsFromAllJobs: bool </usr/share/doc/python3/html/library/functions.html#bool>

        write_messages: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        realTimeLogging: bool </usr/share/doc/python3/html/library/functions.html#bool>

        publish_workflow_metrics: Literal['all'] | Literal['current'] | Literal['no'] | None </usr/share/doc/python3/html/library/constants.html#None>

        environment: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        disableChaining: bool </usr/share/doc/python3/html/library/functions.html#bool>

        disableJobStoreChecksumVerification: bool </usr/share/doc/python3/html/library/functions.html#bool>

        sseKey: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        servicePollingInterval: int </usr/share/doc/python3/html/library/functions.html#int>

        useAsync: bool </usr/share/doc/python3/html/library/functions.html#bool>

        forceDockerAppliance: bool </usr/share/doc/python3/html/library/functions.html#bool>

        statusWait: int </usr/share/doc/python3/html/library/functions.html#int>

        disableProgress: bool </usr/share/doc/python3/html/library/functions.html#bool>

        readGlobalFileMutableByDefault: bool </usr/share/doc/python3/html/library/functions.html#bool>

        debugWorker: bool </usr/share/doc/python3/html/library/functions.html#bool>

        disableWorkerOutputCapture: bool </usr/share/doc/python3/html/library/functions.html#bool>

        badWorker: float </usr/share/doc/python3/html/library/functions.html#float>

        badWorkerFailInterval: float </usr/share/doc/python3/html/library/functions.html#float>

        kill_polling_interval: int </usr/share/doc/python3/html/library/functions.html#int>

        cwl: bool </usr/share/doc/python3/html/library/functions.html#bool>

        memory_is_product: bool </usr/share/doc/python3/html/library/functions.html#bool>

        set_from_default_config()

            Return type
                None

        prepare_start()
            After options are set, prepare for initial start of workflow.

            Return type
                None

        prepare_restart()
            Before restart options are set, prepare for a restart of a workflow. Set up any execution-specific parameters and clear out any stale ones.

            Return type
                None

        setOptions(options)
            Creates a config object from the options object.

            Parameters
                options (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)

            Return type
                None

        check_configuration_consistency()
            Old checks that cannot be fit into an action class for argparse

            Return type
                None

        __eq__(other)

            Parameters
                other (object </usr/share/doc/python3/html/library/functions.html#object>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        __hash__()

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

    toil.common.ensure_config(filepath)
        If the config file at the filepath does not exist, create it. The parent directory should be created prior to calling this.

        Raises an error if the config file cannot be created. Safe to run simultaneously in multiple processes. If this process runs this function, it will always see the config file existing with parseable contents, even if other processes are racing to create it.

        No process will see a new empty or partially-written config file. The caller should still check to make sure there isn't a preexisting empty file here.

        Parameters
            filepath (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- path to config file

        Returns
            None

        Return type
            None

    toil.common.generate_config(filepath)
        Write a Toil config file to the given path.

        Safe to run simultaneously in multiple processes. No process will see an empty or partially-written file at the given path.

        Set include to "cwl" or "wdl" to include cwl options and wdl options respectfully

        Parameters
            filepath (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            None

    toil.common.update_config(filepath, key, new_value)
        Set the given top-level key to the given value in the given YAML config file.

        Does not dramatically alter comments or formatting, and does not make a partially-written file visible.

        Parameters

            •   key (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Setting to set. Must be the command-line option name, not the destination variable name.

            •   filepath (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   new_value (Union[str </usr/share/doc/python3/html/library/stdtypes.html#str>, bool </usr/share/doc/python3/html/library/functions.html#bool>, int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>])

        Return type
            None

    toil.common.parser_with_common_options(provisioner_options=False, jobstore_option=True, prog=None, default_log_level=None)
        Get a command-line option parser for a Toil subcommand.

        The returned parser just has basic options (like version reporting and logging) used by all Toil subcommands.

        Toil Python workflows should use toil.job.Job.Runner.getDefaultArgumentParser() <#toil.job.Job.Runner.getDefaultArgumentParser> instead, which makes sure to add all the important options for actually running a workflow.

        Parameters

            •   provisioner_options (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            •   jobstore_option (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            •   prog (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   default_log_level (Optional[int </usr/share/doc/python3/html/library/functions.html#int>])

        Return type
            configargparse.ArgParser

    toil.common.addOptions(parser, jobstore_as_flag=False, cwl=False, wdl=False)
        Add all Toil command line options to a parser.

        Support for config files if using configargparse. This will also check and set up the default config file.

        Parameters

            •   jobstore_as_flag (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- make the job store option a --jobStore flag instead of a required jobStore positional argument.

            •   cwl (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Whether CWL options are expected. If so, CWL options won't be suppressed.

            •   wdl (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Whether WDL options are expected. If so, WDL options won't be suppressed.

            •   parser (argparse.ArgumentParser </usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser>)

        Return type
            None

    toil.common.getNodeID()
        Return unique ID of the current node (host). The resulting string will be convertible to a uuid.UUID.

        Tries several methods until success. The returned ID should be identical across calls from different processes on the same node at least until the next OS reboot.

        The last resort method is uuid.getnode() that in some rare OS configurations may return a random ID each time it is called. However, this method should never be reached on a Linux system, because reading from /proc/sys/kernel/random/boot_id will be tried prior to that. If uuid.getnode() is reached, it will be called twice, and exception raised if the values are not identical.

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    class toil.common.Toil(options, workflow_name=None, trs_spec=None)
        Bases: ContextManager[Toil]

        A context manager that represents a Toil workflow.

        Specifically the batch system, job store, and its configuration.

        Parameters

            •   options (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)

            •   workflow_name (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   trs_spec (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        config: Config <#toil.common.Config>

        options

        __enter__()
            Derive configuration from the command line options.

            Then load the job store and, on restart, consolidate the derived configuration with the one from the previous invocation of the workflow.

            Return type
                Toil <#toil.common.Toil>

        __exit__(exc_type, exc_val, exc_tb)
            Clean up after a workflow invocation.

            Depending on the configuration, delete the job store.

            Parameters

                •   exc_type (Optional[type </usr/share/doc/python3/html/library/functions.html#type>[BaseException </usr/share/doc/python3/html/library/exceptions.html#BaseException>]])

                •   exc_val (Optional[BaseException </usr/share/doc/python3/html/library/exceptions.html#BaseException>])

                •   exc_tb (Optional[types.TracebackType </usr/share/doc/python3/html/library/types.html#types.TracebackType>])

            Return type
                Literal[False]

        start(rootJob)
            Invoke a Toil workflow with the given job as the root for an initial run.

            This method must be called in the body of a with Toil(...) as toil: statement. This method should not be called more than once for a workflow that has not finished.

            Parameters
                rootJob (toil.job.Job <#toil.job.Job>) -- The root job of the workflow

            Returns
                The root job's return value

            Return type
                Any

        restart()
            Restarts a workflow that has been interrupted.

            Returns
                The root job's return value

            Return type
                Any

        JOB_STORE_TYPES = ['file', 'aws', 'google']

        classmethod getJobStore(locator)
            Create an instance of the concrete job store implementation that matches the given locator.

            Parameters
                locator (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The location of the job store to be represent by the instance

            Returns
                an instance of a concrete subclass of AbstractJobStore

            Return type
                toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>

        static parseLocator(locator)
            Parse a job store locator to a type string and the data needed for that implementation to connect to it.

            Does not validate the set of possible job store types.

            Raises
                RuntimeError </usr/share/doc/python3/html/library/exceptions.html#RuntimeError> -- if the locator is not in the appropriate syntax.

            Parameters
                locator (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        static buildLocator(name, rest)

            Parameters

                •   name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   rest (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        classmethod canonical_locator(locator)
            Turn a job store locator into one that will work from any directory and always includes the explicit type of job store.

            Parameters
                locator (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        classmethod resumeJobStore(locator)

            Parameters
                locator (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>

        static createBatchSystem(config)
            Create an instance of the batch system specified in the given config.

            Parameters
                config (Config <#toil.common.Config>) -- the current configuration

            Returns
                an instance of a concrete subclass of AbstractBatchSystem

            Return type
                toil.batchSystems.abstractBatchSystem.AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>

        url_exists(src_uri)

            Parameters
                src_uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        importFile(srcUrl: str </usr/share/doc/python3/html/library/stdtypes.html#str>, sharedFileName: str </usr/share/doc/python3/html/library/stdtypes.html#str>, symlink: bool </usr/share/doc/python3/html/library/functions.html#bool> = True) -> None </usr/share/doc/python3/html/library/constants.html#None>

        importFile(srcUrl: str </usr/share/doc/python3/html/library/stdtypes.html#str>, sharedFileName: None </usr/share/doc/python3/html/library/constants.html#None> = None, symlink: bool </usr/share/doc/python3/html/library/functions.html#bool> = True) -> toil.fileStores.FileID <#toil.fileStores.FileID>

        import_file(src_uri: str </usr/share/doc/python3/html/library/stdtypes.html#str>, shared_file_name: str </usr/share/doc/python3/html/library/stdtypes.html#str>, symlink: bool </usr/share/doc/python3/html/library/functions.html#bool> = True, check_existence: bool </usr/share/doc/python3/html/library/functions.html#bool> = True) -> None </usr/share/doc/python3/html/library/constants.html#None>

        import_file(src_uri: str </usr/share/doc/python3/html/library/stdtypes.html#str>, shared_file_name: None </usr/share/doc/python3/html/library/constants.html#None> = None, symlink: bool </usr/share/doc/python3/html/library/functions.html#bool> = True, check_existence: Literal[True] = True) -> toil.fileStores.FileID <#toil.fileStores.FileID>

        import_file(src_uri: str </usr/share/doc/python3/html/library/stdtypes.html#str>, shared_file_name: None </usr/share/doc/python3/html/library/constants.html#None> = None, symlink: bool </usr/share/doc/python3/html/library/functions.html#bool> = True, check_existence: bool </usr/share/doc/python3/html/library/functions.html#bool> = True) -> toil.fileStores.FileID <#toil.fileStores.FileID> | None </usr/share/doc/python3/html/library/constants.html#None>
            Import the file at the given URL into the job store.

            By default, returns None if the file does not exist.

            Parameters
                check_existence -- If true, raise FileNotFoundError if the file does not exist. If false, return None when the file does not exist.

        See toil.jobStores.abstractJobStore.AbstractJobStore.importFile() <#toil.jobStores.abstractJobStore.AbstractJobStore.importFile> for a full description

        exportFile(jobStoreFileID, dstUrl)

            Parameters

                •   jobStoreFileID (toil.fileStores.FileID <#toil.fileStores.FileID>)

                •   dstUrl (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        export_file(file_id, dst_uri)
            Export file to destination pointed at by the destination URL.

            See toil.jobStores.abstractJobStore.AbstractJobStore.exportFile() <#toil.jobStores.abstractJobStore.AbstractJobStore.exportFile> for a full description

            Parameters

                •   file_id (toil.fileStores.FileID <#toil.fileStores.FileID>)

                •   dst_uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        static normalize_uri(uri, check_existence=False, dir_path=None)
            Given a URI, if it has no scheme, make it a properly quoted file: URI.

            Parameters

                •   check_existence (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If set, raise FileNotFoundError if a URI points to a local file that does not exist.

                •   dir_path (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- If specified, interpret relative paths relative to the given directory path instead of the current one.

                •   uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        static getToilWorkDir(configWorkDir=None)
            Return a path to a writable directory under which per-workflow directories exist.

            This directory is always required to exist on a machine, even if the Toil worker has not run yet. If your workers and leader have different temp directories, you may need to set TOIL_WORKDIR.

            Parameters
                configWorkDir (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Value passed to the program using the --workDir flag

            Returns
                Path to the Toil work directory, constant across all machines

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        classmethod get_toil_coordination_dir(config_work_dir, config_coordination_dir)
            Return a path to a writable directory, which will be in memory if convenient. Ought to be used for file locking and coordination.

            Parameters

                •   config_work_dir (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Value passed to the program using the --workDir flag

                •   config_coordination_dir (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Value passed to the program using the --coordinationDir flag

                •   workflow_id -- Used if a tmpdir_prefix exists to create full directory paths unique per workflow

            Returns
                Path to the Toil coordination directory. Ought to be on a POSIX filesystem that allows directories containing open files to be deleted.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        static get_workflow_path_component(workflow_id)
            Get a safe filesystem path component for a workflow.

            Will be consistent for all processes on a given machine, and different for all processes on different machines.

            Parameters
                workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The ID of the current Toil workflow.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        classmethod getLocalWorkflowDir(workflowID, configWorkDir=None)
            Return the directory where worker directories and the cache will be located for this workflow on this machine.

            Parameters

                •   configWorkDir (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Value passed to the program using the --workDir flag

                •   workflowID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Returns
                Path to the local workflow directory on this machine

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        classmethod get_local_workflow_coordination_dir(workflow_id, config_work_dir, config_coordination_dir)
            Return the directory where coordination files should be located for this workflow on this machine. These include internal Toil databases and lock files for the machine.

            If an in-memory filesystem is available, it is used. Otherwise, the local workflow directory, which may be on a shared network filesystem, is used.

            Parameters

                •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Unique ID of the current workflow.

                •   config_work_dir (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Value used for the work directory in the current Toil Config.

                •   config_coordination_dir (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Value used for the coordination directory in the current Toil Config.

            Returns
                Path to the local workflow coordination directory on this machine.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

    exception toil.common.ToilRestartException(message)
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Common base class for all non-exit exceptions.

        Parameters
            message (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

    exception toil.common.ToilContextManagerException
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Common base class for all non-exit exceptions.

    class toil.common.ToilMetrics(bus, provisioner=None)

        Parameters

            •   bus (toil.bus.MessageBus <#toil.bus.MessageBus>)

            •   provisioner (Optional[toil.provisioners.abstractProvisioner.AbstractProvisioner <#toil.provisioners.abstractProvisioner.AbstractProvisioner>])

        mtailImage = 'quay.io/ucsc_cgl/toil-mtail:9.1.2-355b72ba0e425619c0367562caf2a337078dba65-py3.13'

        grafanaImage = 'quay.io/ucsc_cgl/toil-grafana:9.1.2-355b72ba0e425619c0367562caf2a337078dba65-py3.13'

        prometheusImage = 'quay.io/ucsc_cgl/toil-prometheus:9.1.2-355b72ba0e425619c0367562caf2a337078dba65-py3.13'

        nodeExporterProc: subprocess.Popen </usr/share/doc/python3/html/library/subprocess.html#subprocess.Popen>[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>] | None </usr/share/doc/python3/html/library/constants.html#None> = None

        startDashboard(clusterName, zone)

            Parameters

                •   clusterName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   zone (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        add_prometheus_data_source()

            Return type
                None

        log(message)

            Parameters
                message (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        logClusterSize(m)

            Parameters
                m (toil.bus.ClusterSizeMessage <#toil.bus.ClusterSizeMessage>)

            Return type
                None

        logClusterDesiredSize(m)

            Parameters
                m (toil.bus.ClusterDesiredSizeMessage <#toil.bus.ClusterDesiredSizeMessage>)

            Return type
                None

        logQueueSize(m)

            Parameters
                m (toil.bus.QueueSizeMessage <#toil.bus.QueueSizeMessage>)

            Return type
                None

        logMissingJob(m)

            Parameters
                m (toil.bus.JobMissingMessage <#toil.bus.JobMissingMessage>)

            Return type
                None

        logIssuedJob(m)

            Parameters
                m (toil.bus.JobIssuedMessage <#toil.bus.JobIssuedMessage>)

            Return type
                None

        logFailedJob(m)

            Parameters
                m (toil.bus.JobFailedMessage <#toil.bus.JobFailedMessage>)

            Return type
                None

        logCompletedJob(m)

            Parameters
                m (toil.bus.JobCompletedMessage <#toil.bus.JobCompletedMessage>)

            Return type
                None

        shutdown()

            Return type
                None

    toil.common.cacheDirName(workflowID)

        Returns
            Name of the cache directory.

        Parameters
            workflowID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.common.getDirSizeRecursively(dirPath)
        This method will return the cumulative number of bytes occupied by the files on disk in the directory and its subdirectories.

        If the method is unable to access a file or directory (due to insufficient permissions, or due to the file or directory having been removed while this function was attempting to traverse it), the error will be handled internally, and a (possibly 0) lower bound on the size of the directory will be returned.

        Parameters
            dirPath (toil.lib.misc.StrPath) -- A valid path to a directory or file.

        Returns
            Total size, in bytes, of the file or directory at dirPath.

        Return type
            int </usr/share/doc/python3/html/library/functions.html#int>

    toil.common.getFileSystemSize(dirPath)
        Return the free space, and total size of the file system hosting dirPath.

        Parameters
            dirPath (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- A valid path to a directory.

        Returns
            free space and total size of file system

        Return type
            tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[int </usr/share/doc/python3/html/library/functions.html#int>, int </usr/share/doc/python3/html/library/functions.html#int>]

    toil.common.safeUnpickleFromStream(stream)

        Parameters
            stream (IO[Any])

        Return type
            Any

  toil.cwl

  Submodules

  toil.cwl.conftest

  Attributes
collect_ignore |

  Module Contents

    toil.cwl.conftest.collect_ignore = []

  toil.cwl.cwltoil

Implemented support for Common Workflow Language (CWL) for Toil.

  Attributes
logger |
DEFAULT_TMPDIR |
DEFAULT_TMPDIR_PREFIX |
MISSING_FILE |
V |
ProcessType |
T |
usage_message |

  Exceptions
NoAvailableJobStoreException | Indicates that no job store name is available.

  Classes
UnresolvedDict | Tag to indicate a dict contains promises that must be resolved.
SkipNull | Internal sentinel object.
Conditional | Object holding conditional expression until we are ready to evaluate it.
ResolveSource | Apply linkMerge and pickValue operators to values coming into a port.
StepValueFrom | A workflow step input which has a valueFrom expression attached to it.
DefaultWithSource | A workflow step input that has both a source and a default value.
JustAValue | A simple value masquerading as a 'resolve'-able object.
ToilPathMapper | Keeps track of files in a Toil way.
ToilSingleJobExecutor | A SingleJobExecutor that does not assume it is at the top level of the workflow.
ToilTool | Mixin to hook Toil into a cwltool tool type.
ToilCommandLineTool | Subclass the cwltool command line tool to provide the custom ToilPathMapper.
ToilExpressionTool | Subclass the cwltool expression tool to provide the custom ToilPathMapper.
ToilFsAccess | Custom filesystem access class which handles toil filestore references.
VisitFunc |
CWLNamedJob | Base class for all CWL jobs that do user work, to give them useful names.
ResolveIndirect | Helper Job.
CWLJobWrapper | Wrap a CWL job that uses dynamic resources requirement.
CWLJob | Execute a CWL tool using cwltool.executors.SingleJobExecutor.
CWLScatter | Implement workflow scatter step.
CWLGather | Follows on to a scatter Job.
SelfJob | Fake job object to facilitate implementation of CWLWorkflow.run().
CWLWorkflow | Toil Job to convert a CWL workflow graph into a Toil job graph.
CWLInstallImportsJob | Class represents a unit of work in toil.
CWLImportWrapper | Job to organize importing files on workers instead of the leader. Responsible for extracting filenames and metadata,
CWLStartJob | Job responsible for starting the CWL workflow.

  Functions
cwltoil_was_removed() | Complain about deprecated entrypoint.
filter_skip_null(name, value) | Recursively filter out SkipNull objects from 'value'.
ensure_no_collisions(directory[, dir_description]) | Make sure no items in the given CWL Directory have the same name.
try_prepull(cwl_tool_uri, runtime_context, batchsystem) | Try to prepull all containers in a CWL workflow with Singularity or Docker.
resolve_dict_w_promises(dict_w_promises[, file_store]) | Resolve a dictionary of promises evaluate expressions to produce the actual values.
simplify_list(maybe_list) | Turn a length one list loaded by cwltool into a scalar.
toil_make_tool(toolpath_object, loadingContext) | Emit custom ToilCommandLineTools.
toil_get_file(file_store, index, existing, uri[, ...]) | Set up the given file or directory from the Toil jobstore at a file URI
convert_file_uri_to_toil_uri(applyFunc, index, ...) | Given a file URI, convert it to a toil file URI. Uses applyFunc to handle the conversion.
path_to_loc(obj) | Make a path into a location.
extract_file_uri_once(fileindex, existing, file_metadata) | Extract the filename from a CWL file record.
visit_files(func, fs_access, fileindex, existing, ...) | Prepare all files and directories.
upload_directory(directory_metadata, directory_contents) | Upload a Directory object.
extract_and_convert_file_to_toil_uri(convertfunc, ...) | Extract the file URI out of a file object and convert it to a Toil URI.
writeGlobalFileWrapper(file_store, fileuri) | Wrap writeGlobalFile to accept <file://> URIs.
remove_empty_listings(rec) |
toilStageFiles(toil, cwljob, outdir[, destBucket, ...]) | Copy input files out of the global file store and update location and path.
get_container_engine(runtime_context) |
makeRootJob(tool, jobobj, runtime_context, ...) | Create the Toil root Job object for the CWL tool. Is the same as makeJob() except this also handles import logic.
makeJob(tool, jobobj, runtime_context, parent_name, ...) | Create the correct Toil Job object for the CWL tool.
remove_pickle_problems(obj) | Doc_loader does not pickle correctly, causing Toil errors, remove from objects.
extract_workflow_inputs(options, ...) | Collect all the workflow input files to import later.
import_workflow_inputs(jobstore, options, ...[, log_level]) | Import all workflow inputs on the leader.
visitSteps(cmdline_tool, op) | Iterate over a CWL Process object, running the op on each tool description
rm_unprocessed_secondary_files(job_params) |
filtered_secondary_files(unfiltered_secondary_files) | Remove unprocessed secondary files.
scan_for_unsupported_requirements(tool[, ...]) | Scan the given CWL tool for any unsupported optional features.
determine_load_listing(tool) | Determine the directory.listing feature in CWL.
generate_default_job_store(batch_system_name, ...) | Choose a default job store appropriate to the requested batch system and
get_options(args) | Parse given args and properly add non-Toil arguments into the cwljob of the Namespace.
main([args, stdout]) | Run the main loop for toil-cwl-runner.
find_default_container(args, builder) | Find the default constructor by consulting a Toil.options object.

  Module Contents

    toil.cwl.cwltoil.logger

    toil.cwl.cwltoil.DEFAULT_TMPDIR

    toil.cwl.cwltoil.DEFAULT_TMPDIR_PREFIX

    toil.cwl.cwltoil.cwltoil_was_removed()
        Complain about deprecated entrypoint.

        Return type
            None

    class toil.cwl.cwltoil.UnresolvedDict
        Bases: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[Any, Any]

        Tag to indicate a dict contains promises that must be resolved.

    class toil.cwl.cwltoil.SkipNull
        Internal sentinel object.

        Indicates a null value produced by each port of a skipped conditional step. The CWL 1.2 specification calls for treating this the exactly the same as a null value.

    toil.cwl.cwltoil.filter_skip_null(name, value)
        Recursively filter out SkipNull objects from 'value'.

        Parameters

            •   name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Name of port producing this value. Only used when we find an unhandled null from a conditional step and we print out a warning. The name allows the user to better localize which step/port was responsible for the unhandled null.

            •   value (Any) -- port output value object

        Return type
            Any

    toil.cwl.cwltoil.ensure_no_collisions(directory, dir_description=None)
        Make sure no items in the given CWL Directory have the same name.

        If any do, raise a WorkflowException about a "File staging conflict".

        Does not recurse into subdirectories.

        Parameters

            •   directory (cwltool.utils.DirectoryType)

            •   dir_description (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            None

    toil.cwl.cwltoil.try_prepull(cwl_tool_uri, runtime_context, batchsystem)
        Try to prepull all containers in a CWL workflow with Singularity or Docker. This will not prepull the default container specified on the command line. :param cwl_tool_uri: CWL workflow URL. Fragments are accepted as well :param runtime_context: runtime context of cwltool :param batchsystem: type of Toil batchsystem :return:

        Parameters

            •   cwl_tool_uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   runtime_context (cwltool.context.RuntimeContext </usr/share/doc/cwltool/html/autoapi/cwltool/context/index.html#cwltool.context.RuntimeContext>)

            •   batchsystem (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            None

    class toil.cwl.cwltoil.Conditional(expression=None, outputs=None, requirements=None, container_engine='docker')
        Object holding conditional expression until we are ready to evaluate it.

        Evaluation occurs before the enclosing step's inputs are type-checked.

        Parameters

            •   expression (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   outputs (Union[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, cwltool.utils.CWLOutputType], None])

            •   requirements (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[cwltool.utils.CWLObjectType]])

            •   container_engine (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        expression = None

        outputs

        requirements = []

        container_engine = 'docker'

        is_false(job)
            Determine if expression evaluates to False given completed step inputs.

            Parameters
                job (cwltool.utils.CWLObjectType) -- job output object

            Returns
                bool

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        skipped_outputs()
            Generate a dict of SkipNull objects corresponding to the output structure.

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, SkipNull <#toil.cwl.cwltoil.SkipNull>]

    class toil.cwl.cwltoil.ResolveSource(name, input, source_key, promises)
        Apply linkMerge and pickValue operators to values coming into a port.

        Parameters

            •   name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   input (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, cwltool.utils.CWLObjectType])

            •   source_key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   promises (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.job.Job <#toil.job.Job>])

        promise_tuples: list </usr/share/doc/python3/html/library/stdtypes.html#list>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.job.Promise <#toil.job.Promise>]] | tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.job.Promise <#toil.job.Promise>]

        __repr__()
            Allow for debug printing.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        resolve()
            First apply linkMerge then pickValue if either present.

            Return type
                Any

        link_merge(values)
            Apply linkMerge operator to values object.

            Parameters
                values (cwltool.utils.CWLObjectType) -- result of step

            Return type
                Union[list </usr/share/doc/python3/html/library/stdtypes.html#list>[cwltool.utils.CWLOutputType], cwltool.utils.CWLOutputType]

        pick_value(values)
            Apply pickValue operator to values object.

            Parameters
                values (Union[list </usr/share/doc/python3/html/library/stdtypes.html#list>[Union[str </usr/share/doc/python3/html/library/stdtypes.html#str>, SkipNull <#toil.cwl.cwltoil.SkipNull>]], Any]) -- Intended to be a list, but other types will be returned without modification.

            Returns

            Return type
                Any

    class toil.cwl.cwltoil.StepValueFrom(expr, source, req, container_engine)
        A workflow step input which has a valueFrom expression attached to it.

        The valueFrom expression will be evaluated to produce the actual input object for the step.

        Parameters

            •   expr (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   source (Any)

            •   req (list </usr/share/doc/python3/html/library/stdtypes.html#list>[cwltool.utils.CWLObjectType])

            •   container_engine (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        expr

        source

        context = None

        req

        container_engine

        __repr__()
            Allow for debug printing.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        eval_prep(step_inputs, file_store)
            Resolve the contents of any file in a set of inputs.

            The inputs must be associated with the StepValueFrom object's self.source.

            Called when loadContents is specified.

            Parameters

                •   step_inputs (cwltool.utils.CWLObjectType) -- Workflow step inputs.

                •   file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>) -- A toil file store, needed to resolve toilfile:// paths.

            Return type
                None

        resolve()
            Resolve the promise in the valueFrom expression's context.

            Returns
                object that will serve as expression context

            Return type
                Any

        do_eval(inputs)
            Evaluate the valueFrom expression with the given input object.

            Parameters
                inputs (cwltool.utils.CWLObjectType)

            Returns
                object

            Return type
                Any

    class toil.cwl.cwltoil.DefaultWithSource(default, source)
        A workflow step input that has both a source and a default value.

        Parameters

            •   default (Any)

            •   source (Any)

        default

        source

        __repr__()
            Allow for debug printing.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        resolve()
            Determine the final input value when the time is right.

            (when the source can be resolved)

            Returns
                dict

            Return type
                Any

    class toil.cwl.cwltoil.JustAValue(val)
        A simple value masquerading as a 'resolve'-able object.

        Parameters
            val (Any)

        val

        __repr__()
            Allow for debug printing.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        resolve()
            Return the value.

            Return type
                Any

    toil.cwl.cwltoil.resolve_dict_w_promises(dict_w_promises, file_store=None)
        Resolve a dictionary of promises evaluate expressions to produce the actual values.

        Parameters

            •   dict_w_promises (Union[UnresolvedDict <#toil.cwl.cwltoil.UnresolvedDict>, cwltool.utils.CWLObjectType, dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Union[str </usr/share/doc/python3/html/library/stdtypes.html#str>, StepValueFrom <#toil.cwl.cwltoil.StepValueFrom>]]]) -- input dict for these values

            •   file_store (Optional[toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>])

        Returns
            dictionary of actual values

        Return type
            cwltool.utils.CWLObjectType

    toil.cwl.cwltoil.simplify_list(maybe_list)
        Turn a length one list loaded by cwltool into a scalar.

        Anything else is passed as-is, by reference.

        Parameters
            maybe_list (Any)

        Return type
            Any

    class toil.cwl.cwltoil.ToilPathMapper(referenced_files, basedir, stagedir, separateDirs=True, get_file=None, stage_listing=False, streaming_allowed=True)
        Bases: cwltool.pathmapper.PathMapper </usr/share/doc/cwltool/html/autoapi/cwltool/pathmapper/index.html#cwltool.pathmapper.PathMapper>

        Keeps track of files in a Toil way.

        Maps between the symbolic identifier of a file (the Toil FileID), its local path on the host (the value returned by readGlobalFile) and the location of the file inside the software container.

        Parameters

            •   referenced_files (list </usr/share/doc/python3/html/library/stdtypes.html#list>[cwltool.utils.CWLObjectType])

            •   basedir (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   stagedir (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   separateDirs (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            •   get_file (Union[Any, None])

            •   stage_listing (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            •   streaming_allowed (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        get_file = None

        stage_listing = False

        streaming_allowed = True

        visit(obj, stagedir, basedir, copy=False, staged=False)
            Iterate over a CWL object, resolving File and Directory path references.

            This is called on each File or Directory CWL object. The Files and Directories all have "location" fields. For the Files, these are from upload_file(), and for the Directories, these are from upload_directory() or cwltool internally. With upload_directory(), they and their children will be assigned locations based on listing the Directories using ToilFsAccess. With cwltool, locations will be set as absolute paths.

            Parameters

                •   obj (cwltool.utils.CWLObjectType) -- The CWL File or Directory to process

                •   stagedir (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The base path for target paths to be generated under, except when a File or Directory has an overriding parent directory in dirname

                •   basedir (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The directory from which relative paths should be resolved; used as the base directory for the StdFsAccess that generated the listing being processed.

                •   copy (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If set, use writable types for Files and Directories.

                •   staged (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Starts as True at the top of the recursion. Set to False when entering a directory that we can actually download, so we don't stage files and subdirectories separately from the directory as a whole. Controls the staged flag on generated mappings, and therefore whether files and directories are actually placed at their mapped-to target locations. If stage_listing is True, we will leave this True throughout and stage everything.

            Return type
                None

        Produces one MapperEnt for every unique location for a File or Directory. These MapperEnt objects are instructions to cwltool's stage_files function: <https://github.com/common-workflow-language/cwltool/blob/a3e3a5720f7b0131fa4f9c0b3f73b62a347278a6/cwltool/process.py#L254>

        The MapperEnt has fields:

        resolved: An absolute local path anywhere on the filesystem where the file/directory can be found, or the contents of a file to populate it with if type is CreateWritableFile or CreateFile. Or, a URI understood by the StdFsAccess in use (for example, toilfile:).

        target: An absolute path under stagedir that the file or directory will then be placed at by cwltool. Except if a File or Directory has a dirname field, giving its parent path, that is used instead.

        type: One of:
                File: cwltool will copy or link the file from resolved to target, if possible.

                CreateFile: cwltool will create the file at target, treating resolved as the contents.

                WritableFile: cwltool will copy the file from resolved to target, making it writable.

                CreateWritableFile: cwltool will create the file at target, treating resolved as the contents, and make it writable.

                Directory: cwltool will copy or link the directory from resolved to target, if possible. Otherwise, cwltool will make the directory at target if resolved starts with "_:". Otherwise it will do nothing.

                WritableDirectory: cwltool will copy the directory from resolved to target, if possible. Otherwise, cwltool will make the directory at target if resolved starts with "_:". Otherwise it will do nothing.

        staged: if set to False, cwltool will not make or copy anything for this entry

    class toil.cwl.cwltoil.ToilSingleJobExecutor
        Bases: cwltool.executors.SingleJobExecutor </usr/share/doc/cwltool/html/autoapi/cwltool/executors/index.html#cwltool.executors.SingleJobExecutor>

        A SingleJobExecutor that does not assume it is at the top level of the workflow.

        We need this because otherwise every job thinks it is top level and tries to discover secondary files, which may exist when they haven't actually been passed at the top level and thus aren't supposed to be visible.

        run_jobs(process, job_order_object, logger, runtime_context)
            run_jobs from SingleJobExecutor, but not in a top level runtime context.

            Parameters

                •   process (cwltool.process.Process </usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process>)

                •   job_order_object (cwltool.utils.CWLObjectType)

                •   logger (logging.Logger </usr/share/doc/python3/html/library/logging.html#logging.Logger>)

                •   runtime_context (cwltool.context.RuntimeContext </usr/share/doc/cwltool/html/autoapi/cwltool/context/index.html#cwltool.context.RuntimeContext>)

            Return type
                None

    class toil.cwl.cwltoil.ToilTool(*args, **kwargs)
        Mixin to hook Toil into a cwltool tool type.

        Parameters

            •   args (Any)

            •   kwargs (Any)

        connect_toil_job(job)
            Attach the Toil tool to the Toil job that is executing it. This allows it to use the Toil job to stop at certain points if debugging flags are set.

            Parameters
                job (toil.job.Job <#toil.job.Job>)

            Return type
                None

        make_path_mapper(reffiles, stagedir, runtimeContext, separateDirs)
            Create the appropriate PathMapper for the situation.

            Parameters

                •   reffiles (list </usr/share/doc/python3/html/library/stdtypes.html#list>[Any])

                •   stagedir (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   runtimeContext (cwltool.context.RuntimeContext </usr/share/doc/cwltool/html/autoapi/cwltool/context/index.html#cwltool.context.RuntimeContext>)

                •   separateDirs (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                cwltool.pathmapper.PathMapper </usr/share/doc/cwltool/html/autoapi/cwltool/pathmapper/index.html#cwltool.pathmapper.PathMapper>

        __str__()
            Return string representation of this tool type.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

    class toil.cwl.cwltoil.ToilCommandLineTool(*args, **kwargs)
        Bases: ToilTool, cwltool.command_line_tool.CommandLineTool </usr/share/doc/cwltool/html/autoapi/cwltool/command_line_tool/index.html#cwltool.command_line_tool.CommandLineTool>

        Subclass the cwltool command line tool to provide the custom ToilPathMapper.

        Parameters

            •   args (Any)

            •   kwargs (Any)

    class toil.cwl.cwltoil.ToilExpressionTool(*args, **kwargs)
        Bases: ToilTool, cwltool.command_line_tool.ExpressionTool </usr/share/doc/cwltool/html/autoapi/cwltool/command_line_tool/index.html#cwltool.command_line_tool.ExpressionTool>

        Subclass the cwltool expression tool to provide the custom ToilPathMapper.

        Parameters

            •   args (Any)

            •   kwargs (Any)

    toil.cwl.cwltoil.toil_make_tool(toolpath_object, loadingContext)
        Emit custom ToilCommandLineTools.

        This factory function is meant to be passed to cwltool.load_tool().

        Parameters

            •   toolpath_object (ruamel.yaml.comments.CommentedMap)

            •   loadingContext (cwltool.context.LoadingContext </usr/share/doc/cwltool/html/autoapi/cwltool/context/index.html#cwltool.context.LoadingContext>)

        Return type
            cwltool.process.Process </usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process>

    toil.cwl.cwltoil.MISSING_FILE = 'missing://'

    class toil.cwl.cwltoil.ToilFsAccess(basedir, file_store=None)
        Bases: cwltool.stdfsaccess.StdFsAccess </usr/share/doc/cwltool/html/autoapi/cwltool/stdfsaccess/index.html#cwltool.stdfsaccess.StdFsAccess>

        Custom filesystem access class which handles toil filestore references.

        Normal file paths will be resolved relative to basedir, but 'toilfile:' and 'toildir:' URIs will be fulfilled from the Toil file store.

        Also supports URLs supported by Toil job store implementations.

        Parameters

            •   basedir (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   file_store (Optional[toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>])

        file_store = None

        dir_to_download: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        glob(pattern)
            Return a possibly empty list of absolute URI paths that match pathname.

            Parameters
                pattern (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        open(fn, mode)

            Parameters

                •   fn (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   mode (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                IO[Any]

        exists(path)
            Test for file existence.

            Parameters
                path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        size(path)

            Parameters
                path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        isfile(fn)

            Parameters
                fn (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        isdir(fn)

            Parameters
                fn (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        listdir(fn)
            Return a list containing the absolute path URLs of the entries in the directory given by path.

            Parameters
                fn (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        join(path, *paths)
            Join one or more path segments intelligently.

            Parameters

                •   path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   paths (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        realpath(fn)

            Parameters
                fn (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.cwl.cwltoil.toil_get_file(file_store, index, existing, uri, streamable=False, streaming_allowed=True, pipe_threads=None)
        Set up the given file or directory from the Toil jobstore at a file URI where it can be accessed locally.

        Run as part of the tool setup, inside jobs on the workers. Also used as part of reorganizing files to get them uploaded at the end of a tool.

        Parameters

            •   file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>) -- The Toil file store to download from.

            •   index (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Maps from downloaded file path back to input Toil URI.

            •   existing (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Maps from URI to downloaded file path.

            •   uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The URI for the file to download.

            •   streamable (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If the file is has 'streamable' flag set

            •   streaming_allowed (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If streaming is allowed

            •   pipe_threads (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[threading.Thread </usr/share/doc/python3/html/library/threading.html#threading.Thread>, int </usr/share/doc/python3/html/library/functions.html#int>]]]) -- List of threads responsible for streaming the data and open file descriptors corresponding to those files. Caller is responsible to close the file descriptors (to break the pipes) and join the threads

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.cwl.cwltoil.convert_file_uri_to_toil_uri(applyFunc, index, existing, file_uri)
        Given a file URI, convert it to a toil file URI. Uses applyFunc to handle the conversion.

        Runs once on every unique file URI.

        'existing' is a set of files retrieved as inputs from toil_get_file. This ensures they are mapped back as the same name if passed through.

        Returns a toil uri path to the object.

        Parameters

            •   applyFunc (Callable[[str </usr/share/doc/python3/html/library/stdtypes.html#str>], toil.fileStores.FileID <#toil.fileStores.FileID>])

            •   index (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   existing (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   file_uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.cwl.cwltoil.path_to_loc(obj)
        Make a path into a location.

        (If a CWL object has a "path" and not a "location")

        Parameters
            obj (cwltool.utils.CWLObjectType)

        Return type
            None

    toil.cwl.cwltoil.extract_file_uri_once(fileindex, existing, file_metadata, mark_broken=False, skip_remote=False)
        Extract the filename from a CWL file record.

        This function matches the predefined function signature in visit_files, which ensures that this function is called on all files inside a CWL object.

        Ensures no duplicate files are returned according to fileindex. If a file has not been resolved already (and had <file://> prepended) then resolve symlinks. :param fileindex: Forward mapping of filename :param existing: Reverse mapping of filename. This function does not use this :param file_metadata: CWL file record :param mark_broken: Whether files should be marked as missing :param skip_remote: Whether to skip remote files :return:

        Parameters

            •   fileindex (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   existing (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   file_metadata (cwltool.utils.CWLObjectType)

            •   mark_broken (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            •   skip_remote (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.cwl.cwltoil.V

    class toil.cwl.cwltoil.VisitFunc
        Bases: Protocol[V]

        __call__(fileindex, existing, file_metadata, mark_broken, skip_remote)

            Parameters

                •   fileindex (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   existing (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   file_metadata (cwltool.utils.CWLObjectType)

                •   mark_broken (bool </usr/share/doc/python3/html/library/functions.html#bool>)

                •   skip_remote (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                V

    toil.cwl.cwltoil.visit_files(func, fs_access, fileindex, existing, cwl_object, mark_broken=False, skip_remote=False, bypass_file_store=False)
        Prepare all files and directories.

        Will be executed from the leader or worker in the context of the given CWL tool, order, or output object to be used on the workers. Make sure their sizes are set and import all the files.

        Recurses inside directories using the fs_access to find files to upload and subdirectory structure to encode, even if their listings are not set or not recursive.

        Preserves any listing fields.

        If a file cannot be found (like if it is an optional secondary file that doesn't exist), fails, unless mark_broken is set, in which case it applies a sentinel location.

        Also does some miscellaneous normalization.

        Parameters

            •   import_function -- The function used to upload a URI and get a Toil FileID for it.

            •   fs_access (cwltool.stdfsaccess.StdFsAccess </usr/share/doc/cwltool/html/autoapi/cwltool/stdfsaccess/index.html#cwltool.stdfsaccess.StdFsAccess>) -- the CWL FS access object we use to access the filesystem to find files to import. Needs to support the URI schemes used.

            •   fileindex (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Forward map to fill in from file URI to Toil storage location, used by write_file to deduplicate writes.

            •   existing (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Reverse map to fill in from Toil storage location to file URI. Not read from.

            •   cwl_object (Optional[cwltool.utils.CWLObjectType]) -- CWL tool (or workflow order) we are importing files for

            •   mark_broken (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If True, when files can't be imported because they e.g. don't exist, set their locations to MISSING_FILE rather than failing with an error.

            •   skp_remote -- If True, leave remote URIs in place instead of importing files.

            •   bypass_file_store (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If True, leave <file://> URIs in place instead of importing files and directories.

            •   log_level -- Log imported files at the given level.

            •   func (VisitFunc <#toil.cwl.cwltoil.VisitFunc>[V])

            •   skip_remote (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[V]

    toil.cwl.cwltoil.upload_directory(directory_metadata, directory_contents, mark_broken=False)
        Upload a Directory object.

        Ignores the listing (which may not be recursive and isn't safe or efficient to touch), and instead uses directory_contents, which is a recursive dict structure from filename to file URI or subdirectory contents dict.

        Makes sure the directory actually exists, and rewrites its location to be something we can use on another machine.

        If mark_broken is set, ignores missing directories and replaces them with directories containing the given (possibly empty) contents.

        We can't rely on the directory's listing as visible to the next tool as a complete recursive description of the files we will need to present to the tool, since some tools require it to be cleared or single-level but still expect to see its contents in the filesystem.

        Parameters

            •   directory_metadata (cwltool.utils.CWLObjectType)

            •   directory_contents (toil.lib.directory.DirectoryContents)

            •   mark_broken (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            None

    toil.cwl.cwltoil.extract_and_convert_file_to_toil_uri(convertfunc, fileindex, existing, file_metadata, mark_broken=False, skip_remote=False)
        Extract the file URI out of a file object and convert it to a Toil URI.

        Runs convertfunc on the file URI to handle conversion.

        Is used to handle importing files into the jobstore.

        If a file doesn't exist, fails with an error, unless mark_broken is set, in which case the missing file is given a special sentinel location.

        Unless skip_remote is set, also run on remote files and sets their locations to toil URIs as well.

        Parameters

            •   convertfunc (Callable[[str </usr/share/doc/python3/html/library/stdtypes.html#str>], toil.fileStores.FileID <#toil.fileStores.FileID>])

            •   fileindex (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   existing (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   file_metadata (cwltool.utils.CWLObjectType)

            •   mark_broken (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            •   skip_remote (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            None

    toil.cwl.cwltoil.writeGlobalFileWrapper(file_store, fileuri)
        Wrap writeGlobalFile to accept <file://> URIs.

        Parameters

            •   file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            •   fileuri (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            toil.fileStores.FileID <#toil.fileStores.FileID>

    toil.cwl.cwltoil.remove_empty_listings(rec)

        Parameters
            rec (cwltool.utils.CWLObjectType)

        Return type
            None

    class toil.cwl.cwltoil.CWLNamedJob(cores=1, memory='1GiB', disk='1MiB', accelerators=None, preemptible=None, tool_id=None, parent_name=None, subjob_name=None, local=None)
        Bases: toil.job.Job <#toil.job.Job>

        Base class for all CWL jobs that do user work, to give them useful names.

        Parameters

            •   cores (Union[float </usr/share/doc/python3/html/library/functions.html#float>, None])

            •   memory (Union[int </usr/share/doc/python3/html/library/functions.html#int>, str </usr/share/doc/python3/html/library/stdtypes.html#str>, None])

            •   disk (Union[int </usr/share/doc/python3/html/library/functions.html#int>, str </usr/share/doc/python3/html/library/stdtypes.html#str>, None])

            •   accelerators (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.job.AcceleratorRequirement <#toil.job.AcceleratorRequirement>]])

            •   preemptible (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>])

            •   tool_id (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   parent_name (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   subjob_name (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   local (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>])

    class toil.cwl.cwltoil.ResolveIndirect(cwljob, parent_name=None)
        Bases: CWLNamedJob

        Helper Job.

        Accepts an unresolved dict (containing promises) and produces a dictionary of actual values.

        Parameters

            •   cwljob (toil.job.Promised[cwltool.utils.CWLObjectType])

            •   parent_name (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        cwljob

        run(file_store)
            Evaluate the promises and return their values.

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                cwltool.utils.CWLObjectType

    toil.cwl.cwltoil.toilStageFiles(toil, cwljob, outdir, destBucket=None, log_level=logging.DEBUG)
        Copy input files out of the global file store and update location and path.

        Parameters

            •   destBucket (Union[str </usr/share/doc/python3/html/library/stdtypes.html#str>, None]) -- If set, export to this base URL instead of to the local filesystem.

            •   log_level (int </usr/share/doc/python3/html/library/functions.html#int>) -- Log each file transferred at the given level.

            •   toil (toil.common.Toil <#toil.common.Toil>)

            •   cwljob (Union[cwltool.utils.CWLObjectType, list </usr/share/doc/python3/html/library/stdtypes.html#list>[cwltool.utils.CWLObjectType]])

            •   outdir (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            None

    class toil.cwl.cwltoil.CWLJobWrapper(tool, cwljob, runtime_context, parent_name, conditional=None)
        Bases: CWLNamedJob

        Wrap a CWL job that uses dynamic resources requirement.

        When executed, this creates a new child job which has the correct resource requirement set.

        Parameters

            •   tool (cwltool.process.Process </usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process>)

            •   cwljob (cwltool.utils.CWLObjectType)

            •   runtime_context (cwltool.context.RuntimeContext </usr/share/doc/cwltool/html/autoapi/cwltool/context/index.html#cwltool.context.RuntimeContext>)

            •   parent_name (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   conditional (Union[Conditional <#toil.cwl.cwltoil.Conditional>, None])

        cwltool

        cwljob

        runtime_context

        conditional

        parent_name

        run(file_store)
            Create a child job with the correct resource requirements set.

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                Any

    class toil.cwl.cwltoil.CWLJob(tool, cwljob, runtime_context, parent_name=None, conditional=None)
        Bases: CWLNamedJob

        Execute a CWL tool using cwltool.executors.SingleJobExecutor.

        Parameters

            •   tool (cwltool.process.Process </usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process>)

            •   cwljob (cwltool.utils.CWLObjectType)

            •   runtime_context (cwltool.context.RuntimeContext </usr/share/doc/cwltool/html/autoapi/cwltool/context/index.html#cwltool.context.RuntimeContext>)

            •   parent_name (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   conditional (Union[Conditional <#toil.cwl.cwltoil.Conditional>, None])

        cwltool

        conditional

        cwljob

        runtime_context

        step_inputs

        workdir: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        required_env_vars(cwljob)
            Yield environment variables from EnvVarRequirement.

            Parameters
                cwljob (Any)

            Return type
                Iterator[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

        populate_env_vars(cwljob)
            Prepare environment variables necessary at runtime for the job.

            Env vars specified in the CWL "requirements" section should already be loaded in self.cwltool.requirements, however those specified with "EnvVarRequirement" take precedence and are only populated here. Therefore, this not only returns a dictionary with all evaluated "EnvVarRequirement" env vars, but checks self.cwltool.requirements for any env vars with the same name and replaces their value with that found in the "EnvVarRequirement" env var if it exists.

            Parameters
                cwljob (cwltool.utils.CWLObjectType)

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        run(file_store)
            Execute the CWL document.

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                Any

    toil.cwl.cwltoil.get_container_engine(runtime_context)

        Parameters
            runtime_context (cwltool.context.RuntimeContext </usr/share/doc/cwltool/html/autoapi/cwltool/context/index.html#cwltool.context.RuntimeContext>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.cwl.cwltoil.makeRootJob(tool, jobobj, runtime_context, initialized_job_order, options, toil)
        Create the Toil root Job object for the CWL tool. Is the same as makeJob() except this also handles import logic.

        Actually creates what might be a subgraph of two jobs. The second of which may be the follow on of the first. If only one job is created, it is returned twice.

        Returns

        Parameters

            •   tool (cwltool.process.Process </usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process>)

            •   jobobj (cwltool.utils.CWLObjectType)

            •   runtime_context (cwltool.context.RuntimeContext </usr/share/doc/cwltool/html/autoapi/cwltool/context/index.html#cwltool.context.RuntimeContext>)

            •   initialized_job_order (cwltool.utils.CWLObjectType)

            •   options (configargparse.Namespace)

            •   toil (toil.common.Toil <#toil.common.Toil>)

        Return type
            CWLNamedJob <#toil.cwl.cwltoil.CWLNamedJob>

    toil.cwl.cwltoil.makeJob(tool, jobobj, runtime_context, parent_name, conditional)
        Create the correct Toil Job object for the CWL tool.

        Actually creates what might be a subgraph of two jobs. The second of which may be the follow on of the first. If only one job is created, it is returned twice.

        Types: workflow, job, or job wrapper for dynamic resource requirements.

        Returns
            "wfjob, followOn" if the input tool is a workflow, and "job, job" otherwise

        Parameters

            •   tool (cwltool.process.Process </usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process>)

            •   jobobj (cwltool.utils.CWLObjectType)

            •   runtime_context (cwltool.context.RuntimeContext </usr/share/doc/cwltool/html/autoapi/cwltool/context/index.html#cwltool.context.RuntimeContext>)

            •   parent_name (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   conditional (Union[Conditional <#toil.cwl.cwltoil.Conditional>, None])

        Return type
            Union[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[CWLWorkflow <#toil.cwl.cwltoil.CWLWorkflow>, ResolveIndirect <#toil.cwl.cwltoil.ResolveIndirect>], tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[CWLJob <#toil.cwl.cwltoil.CWLJob>, CWLJob <#toil.cwl.cwltoil.CWLJob>], tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[CWLJobWrapper <#toil.cwl.cwltoil.CWLJobWrapper>, CWLJobWrapper <#toil.cwl.cwltoil.CWLJobWrapper>]]

    class toil.cwl.cwltoil.CWLScatter(step, cwljob, runtime_context, parent_name, conditional)
        Bases: toil.job.Job <#toil.job.Job>

        Implement workflow scatter step.

        When run, this creates a child job for each parameterization of the scatter.

        Parameters

            •   step (cwltool.workflow.WorkflowStep </usr/share/doc/cwltool/html/autoapi/cwltool/workflow/index.html#cwltool.workflow.WorkflowStep>)

            •   cwljob (cwltool.utils.CWLObjectType)

            •   runtime_context (cwltool.context.RuntimeContext </usr/share/doc/cwltool/html/autoapi/cwltool/context/index.html#cwltool.context.RuntimeContext>)

            •   parent_name (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   conditional (Union[Conditional <#toil.cwl.cwltoil.Conditional>, None])

        step

        cwljob

        runtime_context

        conditional

        parent_name

        flat_crossproduct_scatter(joborder, scatter_keys, outputs, postScatterEval)
            Cartesian product of the inputs, then flattened.

            Parameters

                •   joborder (cwltool.utils.CWLObjectType)

                •   scatter_keys (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   outputs (list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.job.Promised[cwltool.utils.CWLObjectType]])

                •   postScatterEval (Callable[[cwltool.utils.CWLObjectType], cwltool.utils.CWLObjectType])

            Return type
                None

        nested_crossproduct_scatter(joborder, scatter_keys, postScatterEval)
            Cartesian product of the inputs.

            Parameters

                •   joborder (cwltool.utils.CWLObjectType)

                •   scatter_keys (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   postScatterEval (Callable[[cwltool.utils.CWLObjectType], cwltool.utils.CWLObjectType])

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.job.Promised[cwltool.utils.CWLObjectType]]

        run(file_store)
            Generate the follow on scatter jobs.

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.job.Promised[cwltool.utils.CWLObjectType]]

    class toil.cwl.cwltoil.CWLGather(step, outputs)
        Bases: toil.job.Job <#toil.job.Job>

        Follows on to a scatter Job.

        This gathers the outputs of each job in the scatter into an array for each output parameter.

        Parameters

            •   step (cwltool.workflow.WorkflowStep </usr/share/doc/cwltool/html/autoapi/cwltool/workflow/index.html#cwltool.workflow.WorkflowStep>)

            •   outputs (toil.job.Promised[Union[cwltool.utils.CWLObjectType, list </usr/share/doc/python3/html/library/stdtypes.html#list>[cwltool.utils.CWLObjectType]]])

        step

        outputs

        static extract(obj, k)
            Extract the given key from the obj.

            If the object is a list, extract it from all members of the list.

            Parameters

                •   obj (Union[cwltool.utils.CWLObjectType, list </usr/share/doc/python3/html/library/stdtypes.html#list>[cwltool.utils.CWLObjectType]])

                •   k (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                Union[cwltool.utils.CWLOutputType, list </usr/share/doc/python3/html/library/stdtypes.html#list>[cwltool.utils.CWLObjectType]]

        run(file_store)
            Gather all the outputs of the scatter.

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]

    class toil.cwl.cwltoil.SelfJob(j, v)
        Bases: toil.job.Job <#toil.job.Job>

        Fake job object to facilitate implementation of CWLWorkflow.run().

        Parameters

            •   j (CWLWorkflow <#toil.cwl.cwltoil.CWLWorkflow>)

            •   v (cwltool.utils.CWLObjectType)

        j

        v

        rv(*path)
            Return our properties dictionary.

            Parameters
                path (Any)

            Return type
                Any

        addChild(c)
            Add a child to our workflow.

            Parameters
                c (toil.job.Job <#toil.job.Job>)

            Return type
                Any

        hasChild(c)
            Check if the given child is in our workflow.

            Parameters
                c (toil.job.Job <#toil.job.Job>)

            Return type
                Any

    toil.cwl.cwltoil.ProcessType

    toil.cwl.cwltoil.remove_pickle_problems(obj)
        Doc_loader does not pickle correctly, causing Toil errors, remove from objects.

        See github issue: <https://github.com/mypyc/mypyc/issues/804>

        Parameters
            obj (ProcessType)

        Return type
            ProcessType

    class toil.cwl.cwltoil.CWLWorkflow(cwlwf, cwljob, runtime_context, parent_name=None, conditional=None)
        Bases: CWLNamedJob

        Toil Job to convert a CWL workflow graph into a Toil job graph.

        The Toil job graph will include the appropriate dependencies.

        Parameters

            •   cwlwf (cwltool.workflow.Workflow </usr/share/doc/cwltool/html/autoapi/cwltool/workflow/index.html#cwltool.workflow.Workflow>)

            •   cwljob (cwltool.utils.CWLObjectType)

            •   runtime_context (cwltool.context.RuntimeContext </usr/share/doc/cwltool/html/autoapi/cwltool/context/index.html#cwltool.context.RuntimeContext>)

            •   parent_name (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   conditional (Union[Conditional <#toil.cwl.cwltoil.Conditional>, None])

        cwlwf

        cwljob

        runtime_context

        conditional

        run(file_store)
            Convert a CWL Workflow graph into a Toil job graph.

            Always runs on the leader, because the batch system knows to schedule it as a local job.

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                Union[UnresolvedDict <#toil.cwl.cwltoil.UnresolvedDict>, dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, SkipNull <#toil.cwl.cwltoil.SkipNull>]]

    class toil.cwl.cwltoil.CWLInstallImportsJob(initialized_job_order, tool, basedir, skip_remote, bypass_file_store, import_data, **kwargs)
        Bases: toil.job.Job <#toil.job.Job>

        Class represents a unit of work in toil.

        Parameters

            •   initialized_job_order (toil.job.Promised[cwltool.utils.CWLObjectType])

            •   tool (toil.job.Promised[cwltool.process.Process </usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process>])

            •   basedir (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   skip_remote (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            •   bypass_file_store (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            •   import_data (list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.job.Promised[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.fileStores.FileID <#toil.fileStores.FileID>]]])

            •   kwargs (Any)

        initialized_job_order

        tool

        basedir

        skip_remote

        bypass_file_store

        import_data

        static fill_in_files(initialized_job_order, tool, candidate_to_fileid, basedir, skip_remote, bypass_file_store)
            Given a mapping of filenames to Toil file IDs, replace the filename with the file IDs throughout the CWL object.

            Parameters

                •   initialized_job_order (cwltool.utils.CWLObjectType)

                •   tool (cwltool.process.Process </usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process>)

                •   candidate_to_fileid (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.fileStores.FileID <#toil.fileStores.FileID>])

                •   basedir (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   skip_remote (bool </usr/share/doc/python3/html/library/functions.html#bool>)

                •   bypass_file_store (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[cwltool.process.Process </usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process>, cwltool.utils.CWLObjectType]

        run(file_store)
            Convert the filenames in the workflow inputs into the URIs :return: Promise of transformed workflow inputs. A tuple of the job order and process

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                Tuple[cwltool.process.Process </usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process>, cwltool.utils.CWLObjectType]

    class toil.cwl.cwltoil.CWLImportWrapper(initialized_job_order, tool, runtime_context, file_to_data, imported_files, options)
        Bases: CWLNamedJob

        Job to organize importing files on workers instead of the leader. Responsible for extracting filenames and metadata, calling ImportsJob, applying imports to the job objects, and scheduling the start workflow job

        This class is only used when runImportsOnWorkers is enabled.

        Parameters

            •   initialized_job_order (cwltool.utils.CWLObjectType)

            •   tool (cwltool.process.Process </usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process>)

            •   runtime_context (cwltool.context.RuntimeContext </usr/share/doc/cwltool/html/autoapi/cwltool/context/index.html#cwltool.context.RuntimeContext>)

            •   file_to_data (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.job.FileMetadata <#toil.job.FileMetadata>])

            •   imported_files (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.fileStores.FileID <#toil.fileStores.FileID>])

            •   options (configargparse.Namespace)

        initialized_job_order

        tool

        runtime_context

        file_to_data

        imported_files

        options

        run(file_store)
            Override this function to perform work and dynamically create successor jobs.

            Parameters

                •   fileStore -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

                •   file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>.

            Return type
                Any

    class toil.cwl.cwltoil.CWLStartJob(tool, initialized_job_order, runtime_context, **kwargs)
        Bases: CWLNamedJob

        Job responsible for starting the CWL workflow.

        Takes in the workflow/tool and inputs after all files are imported and creates jobs to run those workflows.

        Parameters

            •   tool (toil.job.Promised[cwltool.process.Process </usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process>])

            •   initialized_job_order (toil.job.Promised[cwltool.utils.CWLObjectType])

            •   runtime_context (cwltool.context.RuntimeContext </usr/share/doc/cwltool/html/autoapi/cwltool/context/index.html#cwltool.context.RuntimeContext>)

            •   kwargs (Any)

        tool

        initialized_job_order

        runtime_context

        run(file_store)
            Override this function to perform work and dynamically create successor jobs.

            Parameters

                •   fileStore -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

                •   file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>.

            Return type
                Any

    toil.cwl.cwltoil.extract_workflow_inputs(options, initialized_job_order, tool)
        Collect all the workflow input files to import later. :param options: namespace :param initialized_job_order: cwl object :param tool: tool object :return:

        Parameters

            •   options (configargparse.Namespace)

            •   initialized_job_order (cwltool.utils.CWLObjectType)

            •   tool (cwltool.process.Process </usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process>)

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.cwl.cwltoil.import_workflow_inputs(jobstore, options, initialized_job_order, tool, log_level=logging.DEBUG)
        Import all workflow inputs on the leader.

        Ran when not importing on workers. :param jobstore: Toil jobstore :param options: Namespace :param initialized_job_order: CWL object :param tool: CWL tool :param log_level: log level :return:

        Parameters

            •   jobstore (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

            •   options (configargparse.Namespace)

            •   initialized_job_order (cwltool.utils.CWLObjectType)

            •   tool (cwltool.process.Process </usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process>)

            •   log_level (int </usr/share/doc/python3/html/library/functions.html#int>)

        Return type
            None

    toil.cwl.cwltoil.T

    toil.cwl.cwltoil.visitSteps(cmdline_tool, op)
        Iterate over a CWL Process object, running the op on each tool description CWL object.

        Parameters

            •   cmdline_tool (cwltool.process.Process </usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process>)

            •   op (Callable[[ruamel.yaml.comments.CommentedMap], list </usr/share/doc/python3/html/library/stdtypes.html#list>[T]])

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[T]

    toil.cwl.cwltoil.rm_unprocessed_secondary_files(job_params)

        Parameters
            job_params (Any)

        Return type
            None

    toil.cwl.cwltoil.filtered_secondary_files(unfiltered_secondary_files)
        Remove unprocessed secondary files.

        Interpolated strings and optional inputs in secondary files were added to CWL in version 1.1.

        The CWL libraries we call do successfully resolve the interpolated strings, but add the resolved fields to the list of unresolved fields so we remove them here after the fact.

        We keep secondary files with anything other than MISSING_FILE as their location. The 'required' logic seems to be handled deeper in cwltool.builder.Builder(), and correctly determines which files should be imported. Therefore we remove the files here and if this file is SUPPOSED to exist, it will still give the appropriate file does not exist error, but just a bit further down the track.

        Parameters
            unfiltered_secondary_files (cwltool.utils.CWLObjectType)

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[cwltool.utils.CWLObjectType]

    toil.cwl.cwltoil.scan_for_unsupported_requirements(tool, bypass_file_store=False)
        Scan the given CWL tool for any unsupported optional features.

        If it has them, raise an informative UnsupportedRequirement.

        Parameters

            •   tool (cwltool.process.Process </usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process>) -- The CWL tool to check for unsupported requirements.

            •   bypass_file_store (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- True if the Toil file store is not being used to transport files between nodes, and raw origin node <file://> URIs are exposed to tools instead.

        Return type
            None

    toil.cwl.cwltoil.determine_load_listing(tool)
        Determine the directory.listing feature in CWL.

        In CWL, any input directory can have a DIRECTORY_NAME.listing (where DIRECTORY_NAME is any variable name) set to one of the following three options:

        1.

            no_listing: DIRECTORY_NAME.listing will be undefined.
                e.g.
                    inputs.DIRECTORY_NAME.listing == unspecified

        2.

            shallow_listing: DIRECTORY_NAME.listing will return a list one level
                deep of DIRECTORY_NAME's contents. e.g.

                        inputs.DIRECTORY_NAME.listing == [items in directory]
                            inputs.DIRECTORY_NAME.listing[0].listing == undefined inputs.DIRECTORY_NAME.listing.length == # of items in directory

        3.

            deep_listing: DIRECTORY_NAME.listing will return a list of the entire
                contents of DIRECTORY_NAME. e.g.
                    inputs.DIRECTORY_NAME.listing == [items in directory] inputs.DIRECTORY_NAME.listing[0].listing == [items in subdirectory if it exists and is the first item listed] inputs.DIRECTORY_NAME.listing.length == # of items in directory

    See <https://www.commonwl.org/v1.1/CommandLineTool.html#LoadListingRequirement> and <https://www.commonwl.org/v1.1/CommandLineTool.html#LoadListingEnum>

    DIRECTORY_NAME.listing should be determined first from loadListing. If that's not specified, from LoadListingRequirement. Else, default to "no_listing" if unspecified.

        Parameters
            tool (cwltool.process.Process </usr/share/doc/cwltool/html/autoapi/cwltool/process/index.html#cwltool.process.Process>) -- ToilCommandLineTool

        Return str
            One of 'no_listing', 'shallow_listing', or 'deep_listing'.

        Return type
            Literal['no_listing', 'shallow_listing', 'deep_listing']

    exception toil.cwl.cwltoil.NoAvailableJobStoreException
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Indicates that no job store name is available.

    toil.cwl.cwltoil.generate_default_job_store(batch_system_name, provisioner_name, local_directory)
        Choose a default job store appropriate to the requested batch system and provisioner, and installed modules. Raises an error if no good default is available and the user must choose manually.

        Parameters

            •   batch_system_name (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Registry name of the batch system the user has requested, if any. If no name has been requested, should be None.

            •   provisioner_name (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Name of the provisioner the user has requested, if any. Recognized provisioners include 'aws' and 'gce'. None indicates that no provisioner is in use.

            •   local_directory (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Path to a nonexistent local directory suitable for use as a file job store.

        Return str
            Job store specifier for a usable job store.

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.cwl.cwltoil.usage_message = Multiline-String

                """

                """

    toil.cwl.cwltoil.get_options(args)
        Parse given args and properly add non-Toil arguments into the cwljob of the Namespace. :param args: List of args from command line :return: options namespace

        Parameters
            args (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            configargparse.Namespace

    toil.cwl.cwltoil.main(args=None, stdout=sys.stdout)
        Run the main loop for toil-cwl-runner.

        Parameters

            •   args (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

            •   stdout (TextIO)

        Return type
            int </usr/share/doc/python3/html/library/functions.html#int>

    toil.cwl.cwltoil.find_default_container(args, builder)
        Find the default constructor by consulting a Toil.options object.

        Parameters

            •   args (configargparse.Namespace)

            •   builder (cwltool.builder.Builder </usr/share/doc/cwltool/html/autoapi/cwltool/builder/index.html#cwltool.builder.Builder>)

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

  toil.cwl.utils

Utility functions used for Toil's CWL interpreter.

  Attributes
logger |
CWL_UNSUPPORTED_REQUIREMENT_EXIT_CODE |
CWL_UNSUPPORTED_REQUIREMENT_EXCEPTION |
DownReturnType |
UpReturnType |
DirectoryStructure |

  Exceptions
CWLUnsupportedException | Fallback exception.

  Functions
visit_top_cwl_class(rec, classes, op) | Apply the given operation to all top-level CWL objects with the given named CWL class.
visit_cwl_class_and_reduce(rec, classes, op_down, op_up) | Apply the given operations to all CWL objects with the given named CWL class.
get_from_structure(dir_dict, path) | Given a relative path, follow it in the given directory structure.
download_structure(file_store, index, existing, ...) | Download nested dictionary from the Toil file store to a local path.
trim_mounts_op_down(file_or_directory) | No-op function for mount-point trimming.
sniff_location(file_or_directory) | Get the local bare path for a CWL file or directory, or None.
trim_mounts_op_up(file_or_directory, op_down_ret, ...) | Remove subtrees of the CWL file or directory object tree that only have redundant stuff in them.
remove_redundant_mounts(cwljob) | Remove any redundant mount points from the listing. Modifies the CWL object in place.

  Module Contents

    toil.cwl.utils.logger

    toil.cwl.utils.CWL_UNSUPPORTED_REQUIREMENT_EXIT_CODE = 33

    exception toil.cwl.utils.CWLUnsupportedException
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Fallback exception.

    toil.cwl.utils.CWL_UNSUPPORTED_REQUIREMENT_EXCEPTION: type </usr/share/doc/python3/html/library/functions.html#type>[cwltool.errors.UnsupportedRequirement </usr/share/doc/cwltool/html/autoapi/cwltool/errors/index.html#cwltool.errors.UnsupportedRequirement>] | type </usr/share/doc/python3/html/library/functions.html#type>[CWLUnsupportedException <#toil.cwl.utils.CWLUnsupportedException>]

    toil.cwl.utils.visit_top_cwl_class(rec, classes, op)
        Apply the given operation to all top-level CWL objects with the given named CWL class.

        Like cwltool's visit_class but doesn't look inside any object visited.

        Parameters

            •   rec (Any)

            •   classes (collections.abc.Iterable </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterable>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   op (Callable[[Any], Any])

        Return type
            None

    toil.cwl.utils.DownReturnType

    toil.cwl.utils.UpReturnType

    toil.cwl.utils.visit_cwl_class_and_reduce(rec, classes, op_down, op_up)
        Apply the given operations to all CWL objects with the given named CWL class.

        Applies the down operation top-down, and the up operation bottom-up, and passes the down operation's result and a list of the up operation results for all child keys (flattening across lists and collapsing nodes of non-matching classes) to the up operation.

        Returns
            The flattened list of up operation results from all calls.

        Parameters

            •   rec (Any)

            •   classes (collections.abc.Iterable </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterable>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   op_down (Callable[[Any], DownReturnType])

            •   op_up (Callable[[Any, DownReturnType, list </usr/share/doc/python3/html/library/stdtypes.html#list>[UpReturnType]], UpReturnType])

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[UpReturnType]

    toil.cwl.utils.DirectoryStructure

    toil.cwl.utils.get_from_structure(dir_dict, path)
        Given a relative path, follow it in the given directory structure.

        Return the string URI for files, the directory dict for subdirectories, or None for nonexistent things.

        Parameters

            •   dir_dict (DirectoryStructure)

            •   path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            Union[str </usr/share/doc/python3/html/library/stdtypes.html#str>, DirectoryStructure, None]

    toil.cwl.utils.download_structure(file_store, index, existing, dir_dict, into_dir)
        Download nested dictionary from the Toil file store to a local path.

        Guaranteed to fill the structure with real files, and not symlinks out of it to elsewhere. File URIs may be toilfile: URIs or any other URI that Toil's job store system can read.

        Parameters

            •   file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>) -- The Toil file store to download from.

            •   index (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Maps from downloaded file path back to input URI.

            •   existing (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Maps from file_store_id URI to downloaded file path.

            •   dir_dict (DirectoryStructure) -- a dict from string to string (for files) or dict (for subdirectories) describing a directory structure.

            •   into_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The directory to download the top-level dict's files into.

        Return type
            None

    toil.cwl.utils.trim_mounts_op_down(file_or_directory)
        No-op function for mount-point trimming.

        Parameters
            file_or_directory (cwltool.utils.CWLObjectType)

        Return type
            None

    toil.cwl.utils.sniff_location(file_or_directory)
        Get the local bare path for a CWL file or directory, or None.

        Returns
            None if we don't have a local path or file URI

        Parameters
            file_or_directory (cwltool.utils.CWLObjectType)

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.cwl.utils.trim_mounts_op_up(file_or_directory, op_down_ret, child_results)
        Remove subtrees of the CWL file or directory object tree that only have redundant stuff in them.

        Nonredundant for something in a directory means its path or location is not within the parent directory or doesn't match its basename Nonredundant for something in a secondary file means its path or location is not adjacent to the primary file or doesn't match its basename

        If on a File: Returns True if anything in secondary files is nonredundant or has nonredundant children to this file, false otherwise If on a Directory: Returns True if anything in top level listing is nonredundant or has nonredundant children, otherwise false. If something in the listing is redundant and all children are redundant, then delete it :param file_or_directory: CWL file or CWL directory type :return: boolean

        Parameters

            •   file_or_directory (cwltool.utils.CWLObjectType)

            •   op_down_ret (None)

            •   child_results (list </usr/share/doc/python3/html/library/stdtypes.html#list>[bool </usr/share/doc/python3/html/library/functions.html#bool>])

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.cwl.utils.remove_redundant_mounts(cwljob)
        Remove any redundant mount points from the listing. Modifies the CWL object in place.

        Parameters
            cwljob (cwltool.utils.CWLObjectType)

        Return type
            None

  Functions
check_cwltool_version() | Check if the installed cwltool version matches Toil's expected version.

  Package Contents

    toil.cwl.check_cwltool_version()
        Check if the installed cwltool version matches Toil's expected version.

        A warning is printed to standard error if the versions differ. We do not assume that logging is set up already. Safe to call repeatedly; only one warning will be printed.

        Return type
            None

  toil.deferred

  Attributes
logger |

  Classes
DeferredFunction |
DeferredFunctionManager | Implements a deferred function system. Each Toil worker will have an

  Module Contents

    toil.deferred.logger

    class toil.deferred.DeferredFunction
        Bases: namedtuple('DeferredFunction', 'function args kwargs name module')

            >>> from collections import defaultdict
            >>> df = DeferredFunction.create(defaultdict, None, {'x':1}, y=2)
            >>> df
            DeferredFunction(defaultdict, ...)
            >>> df.invoke() == defaultdict(None, x=1, y=2)
            True

        classmethod create(function, *args, **kwargs)
            Capture the given callable and arguments as an instance of this class.

            Parameters

                •   function (callable) -- The deferred action to take in the form of a function

                •   args (tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>) -- Non-keyword arguments to the function

                •   kwargs (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>) -- Keyword arguments to the function

        invoke()
            Invoke the captured function with the captured arguments.

        __str__()
            Return str(self).

        __repr__
            Return repr(self).

    class toil.deferred.DeferredFunctionManager(stateDirBase)
        Implements a deferred function system. Each Toil worker will have an instance of this class. When a job is executed, it will happen inside a context manager from this class. If the job registers any "deferred" functions, they will be executed when the context manager is exited.

        If the Python process terminates before properly exiting the context manager and running the deferred functions, and some other worker process enters or exits the per-job context manager of this class at a later time, or when the DeferredFunctionManager is shut down on the worker, the earlier job's deferred functions will be picked up and run.

        Note that deferred function cleanup is on a best-effort basis, and deferred functions may end up getting executed multiple times.

        Internally, deferred functions are serialized into files in the given directory, which are locked by the owning process.

        If that process dies, other processes can detect that the files are able to be locked, and will take them over.

        Parameters
            stateDirBase (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        STATE_DIR_STEM = 'deferred'

        PREFIX = 'func'

        WIP_SUFFIX = '.tmp'

        stateDir

        stateFileName

        stateFileOut

        stateFileIn

        __del__()
            Clean up our state on disk. We assume that the deferred functions we manage have all been executed, and none are currently recorded.

        open()
            Yields a single-argument function that allows for deferred functions of type toil.DeferredFunction to be registered. We use this design so deferred functions can be registered only inside this context manager.

            Not thread safe.

        classmethod cleanupWorker(stateDirBase)
            Called by the batch system when it shuts down the node, after all workers are done, if the batch system supports worker cleanup. Checks once more for orphaned deferred functions and runs them.

            Parameters
                stateDirBase (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

  toil.exceptions

Neutral place for exceptions, to break import cycles.

  Attributes
logger |

  Exceptions
FailedJobsException | Common base class for all non-exit exceptions.

  Module Contents

    toil.exceptions.logger

    exception toil.exceptions.FailedJobsException(job_store, failed_jobs, exit_code=1)
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Common base class for all non-exit exceptions.

        Parameters

            •   job_store (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

            •   failed_jobs (list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.job.JobDescription <#toil.job.JobDescription>])

            •   exit_code (int </usr/share/doc/python3/html/library/functions.html#int>)

        msg

        exit_code = 1

        jobStoreLocator

        numberOfFailedJobs

        __str__()
            Stringify the exception, including the message.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

  toil.fileStores

  Submodules

  toil.fileStores.abstractFileStore

  Attributes
logger |

  Classes
AbstractFileStore | Interface used to allow user code run by Toil to read and write files.

  Module Contents

    toil.fileStores.abstractFileStore.logger

    class toil.fileStores.abstractFileStore.AbstractFileStore(jobStore, jobDesc, file_store_dir, waitForPreviousCommit)
        Bases: abc.ABC </usr/share/doc/python3/html/library/abc.html#abc.ABC>

        Interface used to allow user code run by Toil to read and write files.

        Also provides the interface to other Toil facilities used by user code, including:

                •   normal (non-real-time) logging

                •   finding the correct temporary directory for scratch work

                •   importing and exporting files into and out of the workflow

    Stores user files in the jobStore, but keeps them separate from actual jobs.

    May implement caching.

    Passed as argument to the toil.job.Job.run() <#toil.job.Job.run> method.

    Access to files is only permitted inside the context manager provided by toil.fileStores.abstractFileStore.AbstractFileStore.open().

    Also responsible for committing completed jobs back to the job store with an update operation, and allowing that commit operation to be waited for.

        Parameters

            •   jobStore (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

            •   jobDesc (toil.job.JobDescription <#toil.job.JobDescription>)

            •   file_store_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   waitForPreviousCommit (Callable[[], Any])

        jobStore

        jobDesc

        localTempDir: str </usr/share/doc/python3/html/library/stdtypes.html#str> = b'.'

        workflow_dir: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        coordination_dir: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        jobName: str </usr/share/doc/python3/html/library/stdtypes.html#str> = ''

        waitForPreviousCommit

        logging_messages: list </usr/share/doc/python3/html/library/stdtypes.html#list>[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, int </usr/share/doc/python3/html/library/functions.html#int> | str </usr/share/doc/python3/html/library/stdtypes.html#str>]] = []

        logging_user_streams: list </usr/share/doc/python3/html/library/stdtypes.html#list>[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]] = []

        filesToDelete: set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        static createFileStore(jobStore, jobDesc, file_store_dir, waitForPreviousCommit, caching)
            Create a concreate FileStore.

            Parameters

                •   jobStore (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

                •   jobDesc (toil.job.JobDescription <#toil.job.JobDescription>)

                •   file_store_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   waitForPreviousCommit (Callable[[], Any])

                •   caching (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>])

            Return type
                Union[toil.fileStores.nonCachingFileStore.NonCachingFileStore <#toil.fileStores.nonCachingFileStore.NonCachingFileStore>, toil.fileStores.cachingFileStore.CachingFileStore <#toil.fileStores.cachingFileStore.CachingFileStore>]

        static shutdownFileStore(workflowID, config_work_dir, config_coordination_dir)
            Carry out any necessary filestore-specific cleanup.

            This is a destructive operation and it is important to ensure that there are no other running processes on the system that are modifying or using the file store for this workflow.

            This is the intended to be the last call to the file store in a Toil run, called by the batch system cleanup function upon batch system shutdown.

            Parameters

                •   workflowID (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The workflow ID for this invocation of the workflow

                •   config_work_dir (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- The path to the work directory in the Toil Config.

                •   config_coordination_dir (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- The path to the coordination directory in the Toil Config.

            Return type
                None

        open(job)
            Create the context manager around tasks prior and after a job has been run.

            File operations are only permitted inside the context manager.

            Implementations must only yield from within with super().open(job):.

            Parameters
                job (toil.job.Job <#toil.job.Job>) -- The job instance of the toil job to run.

            Return type
                collections.abc.Generator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator>[None, None, None]

        get_disk_usage()
            Get the number of bytes of disk used by the last job run under open().

            Disk usage is measured at the end of the job. TODO: Sample periodically and record peak usage.

            Return type
                Optional[int </usr/share/doc/python3/html/library/functions.html#int>]

        getLocalTempDir()
            Get a new local temporary directory in which to write files.

            The directory will only persist for the duration of the job.

            Returns
                The absolute path to a new local temporary directory. This directory will exist for the duration of the job only, and is guaranteed to be deleted once the job terminates, removing all files it contains recursively.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        getLocalTempFile(suffix=None, prefix=None)
            Get a new local temporary file that will persist for the duration of the job.

            Parameters

                •   suffix (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- If not None, the file name will end with this string. Otherwise, default value ".tmp" will be used

                •   prefix (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- If not None, the file name will start with this string. Otherwise, default value "tmp" will be used

            Returns
                The absolute path to a local temporary file. This file will exist for the duration of the job only, and is guaranteed to be deleted once the job terminates.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        getLocalTempFileName(suffix=None, prefix=None)
            Get a valid name for a new local file. Don't actually create a file at the path.

            Parameters

                •   suffix (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- If not None, the file name will end with this string. Otherwise, default value ".tmp" will be used

                •   prefix (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- If not None, the file name will start with this string. Otherwise, default value "tmp" will be used

            Returns
                Path to valid file

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        abstractmethod writeGlobalFile(localFileName, cleanup=False)
            Upload a file (as a path) to the job store.

            If the file is in a FileStore-managed temporary directory (i.e. from toil.fileStores.abstractFileStore.AbstractFileStore.getLocalTempDir()), it will become a local copy of the file, eligible for deletion by toil.fileStores.abstractFileStore.AbstractFileStore.deleteLocalFile().

            If an executable file on the local filesystem is uploaded, its executability will be preserved when it is downloaded again.

            Parameters

                •   localFileName (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The path to the local file to upload. The last path component (basename of the file) will remain associated with the file in the file store, if supported by the backing JobStore, so that the file can be searched for by name or name glob.

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- if True then the copy of the global file will be deleted once the job and all its successors have completed running. If not the global file must be deleted manually.

            Returns
                an ID that can be used to retrieve the file.

            Return type
                toil.fileStores.FileID <#toil.fileStores.FileID>

        writeGlobalFileStream(cleanup=False, basename=None, encoding=None, errors=None)
            Similar to writeGlobalFile, but allows the writing of a stream to the job store. The yielded file handle does not need to and should not be closed explicitly.

            Parameters

                •   encoding (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- The name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode.

                •   errors (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- is as in toil.fileStores.abstractFileStore.AbstractFileStore.writeGlobalFile().

                •   basename (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- If supported by the backing JobStore, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected.

            Returns
                A context manager yielding a tuple of 1) a file handle which can be written to and 2) the toil.fileStores.FileID of the resulting file in the job store.

            Return type
                collections.abc.Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[toil.lib.io.WriteWatchingStream <#toil.lib.io.WriteWatchingStream>, toil.fileStores.FileID <#toil.fileStores.FileID>]]

        logAccess(fileStoreID, destination=None)
            Record that the given file was read by the job.

            (to be announced if the job fails)

            If destination is not None, it gives the path that the file was downloaded to. Otherwise, assumes that the file was streamed.

            Must be called by readGlobalFile() and readGlobalFileStream() implementations.

            Parameters

                •   fileStoreID (Union[toil.fileStores.FileID <#toil.fileStores.FileID>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   destination (Union[str </usr/share/doc/python3/html/library/stdtypes.html#str>, None])

            Return type
                None

        abstractmethod readGlobalFile(fileStoreID, userPath=None, cache=True, mutable=False, symlink=False)
            Make the file associated with fileStoreID available locally.

            If mutable is True, then a copy of the file will be created locally so that the original is not modified and does not change the file for other jobs. If mutable is False, then a link can be created to the file, saving disk resources. The file that is downloaded will be executable if and only if it was originally uploaded from an executable file on the local filesystem.

            If a user path is specified, it is used as the destination. If a user path isn't specified, the file is stored in the local temp directory with an encoded name.

            The destination file must not be deleted by the user; it can only be deleted through deleteLocalFile.

            Implementations must call logAccess() to report the download.

            Parameters

                •   fileStoreID (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- job store id for the file

                •   userPath (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- a path to the name of file to which the global file will be copied or hard-linked (see below).

                •   cache (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Described in toil.fileStores.CachingFileStore.readGlobalFile()

                •   mutable (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Described in toil.fileStores.CachingFileStore.readGlobalFile()

                •   symlink (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- True if caller can accept symlink, False if caller can only accept a normal file or hardlink

            Returns
                An absolute path to a local, temporary copy of the file keyed by fileStoreID.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        readGlobalFileStream(fileStoreID: str </usr/share/doc/python3/html/library/stdtypes.html#str>, encoding: Literal[None] = None, errors: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> ContextManager[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]]

        readGlobalFileStream(fileStoreID: str </usr/share/doc/python3/html/library/stdtypes.html#str>, encoding: str </usr/share/doc/python3/html/library/stdtypes.html#str>, errors: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> ContextManager[IO[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]
            Read a stream from the job store; similar to readGlobalFile.

            The yielded file handle does not need to and should not be closed explicitly.

            Parameters

                •   encoding -- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode.

                •   errors -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

        Implementations must call logAccess() to report the download.

            Returns
                a context manager yielding a file handle which can be read from.

        getGlobalFileSize(fileStoreID)
            Get the size of the file pointed to by the given ID, in bytes.

            If a FileID or something else with a non-None 'size' field, gets that.

            Otherwise, asks the job store to poll the file's size.

            Note that the job store may overestimate the file's size, for example if it is encrypted and had to be augmented with an IV or other encryption framing.

            Parameters
                fileStoreID (Union[toil.fileStores.FileID <#toil.fileStores.FileID>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- File ID for the file

            Returns
                File's size in bytes, as stored in the job store

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        abstractmethod deleteLocalFile(fileStoreID)
            Delete local copies of files associated with the provided job store ID.

            Raises an OSError with an errno of errno.ENOENT if no such local copies exist. Thus, cannot be called multiple times in succession.

            The files deleted are all those previously read from this file ID via readGlobalFile by the current job into the job's file-store-provided temp directory, plus the file that was written to create the given file ID, if it was written by the current job from the job's file-store-provided temp directory.

            Parameters
                fileStoreID (Union[toil.fileStores.FileID <#toil.fileStores.FileID>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- File Store ID of the file to be deleted.

            Return type
                None

        abstractmethod deleteGlobalFile(fileStoreID)
            Delete local files and then permanently deletes them from the job store.

            To ensure that the job can be restarted if necessary, the delete will not happen until after the job's run method has completed.

            Parameters
                fileStoreID (Union[toil.fileStores.FileID <#toil.fileStores.FileID>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- the File Store ID of the file to be deleted.

            Return type
                None

        importFile(srcUrl, sharedFileName=None)

            Parameters

                •   srcUrl (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   sharedFileName (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                Optional[toil.fileStores.FileID <#toil.fileStores.FileID>]

        import_file(src_uri, shared_file_name=None)

            Parameters

                •   src_uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   shared_file_name (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                Optional[toil.fileStores.FileID <#toil.fileStores.FileID>]

        exportFile(jobStoreFileID, dstUrl)

            Parameters

                •   jobStoreFileID (toil.fileStores.FileID <#toil.fileStores.FileID>)

                •   dstUrl (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        abstractmethod export_file(file_id, dst_uri)

            Parameters

                •   file_id (toil.fileStores.FileID <#toil.fileStores.FileID>)

                •   dst_uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        log_to_leader(text, level=logging.INFO)
            Send a logging message to the leader. The message will also be logged by the worker at the same level.

            Does not depend on the commit system, so this is safe to use during an ansynchronous commit, or without a commit afterward.

            Parameters

                •   text (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The string to log.

                •   level (int </usr/share/doc/python3/html/library/functions.html#int>) -- The logging level.

            Return type
                None

        logToMaster(text, level=logging.INFO)

            Parameters

                •   text (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   level (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                None

        log_user_stream(name, stream)
            Send a stream of UTF-8 text to the leader as a named log stream.

            Useful for things like the error logs of Docker containers. The leader will show it to the user or organize it appropriately for user-level log information.

            Parameters

                •   name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- A hierarchical, .-delimited string.

                •   stream (IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]) -- A stream of encoded text. Encoding errors will be tolerated.

            Return type
                None

        abstractmethod startCommit(jobState=False)
            Update the status of the job on the disk.

            May bump the version number of the job.

            May start an asynchronous process. Call waitForCommit() to wait on that process. You must waitForCommit() before committing any further updates to the job. During the asynchronous process, it is safe to modify the job; modifications after this call will not be committed until the next call.

            Parameters
                jobState (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If True, commit the state of the FileStore's job, and file deletes. Otherwise, commit only file creates/updates.

            Return type
                None

        abstractmethod waitForCommit()
            Blocks while startCommit is running.

            This function is called by this job's successor to ensure that it does not begin modifying the job store until after this job has finished doing so.

            Might be called when startCommit is never called on a particular instance, in which case it does not block.

            Returns
                Always returns True

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        classmethod shutdown(shutdown_info)

            Abstractmethod

            Parameters
                shutdown_info (Any)

            Return type
                None

        Shutdown the filestore on this node.

        This is intended to be called on batch system shutdown.

            Parameters
                shutdown_info (Any) -- The implementation-specific shutdown information, for shutting down the file store and removing all its state and all job local temp directories from the node.

            Return type
                None

  toil.fileStores.cachingFileStore

  Attributes
logger |
SQLITE_TIMEOUT_SECS |

  Exceptions
CacheError | Error Raised if the user attempts to add a non-local file to cache
CacheUnbalancedError | Raised if file store can't free enough space for caching
IllegalDeletionCacheError | Error raised if the caching code discovers a file that represents a
InvalidSourceCacheError | Error raised if the user attempts to add a non-local file to cache

  Classes
CachingFileStore | A cache-enabled file store.

  Module Contents

    toil.fileStores.cachingFileStore.logger

    toil.fileStores.cachingFileStore.SQLITE_TIMEOUT_SECS = 60.0

    exception toil.fileStores.cachingFileStore.CacheError(message)
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Error Raised if the user attempts to add a non-local file to cache

    exception toil.fileStores.cachingFileStore.CacheUnbalancedError
        Bases: CacheError

        Raised if file store can't free enough space for caching

        message = 'Unable unable to free enough space for caching. This error frequently arises due to jobs using...

    exception toil.fileStores.cachingFileStore.IllegalDeletionCacheError(deletedFile)
        Bases: CacheError

        Error raised if the caching code discovers a file that represents a reference to a cached file to have gone missing.

        This can be a big problem if a hard link is moved, because then the cache will be unable to evict the file it links to.

        Remember that files read with readGlobalFile may not be deleted by the user and need to be deleted with deleteLocalFile.

    exception toil.fileStores.cachingFileStore.InvalidSourceCacheError(message)
        Bases: CacheError

        Error raised if the user attempts to add a non-local file to cache

    class toil.fileStores.cachingFileStore.CachingFileStore(jobStore, jobDesc, file_store_dir, waitForPreviousCommit)
        Bases: toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>

        A cache-enabled file store.

        Provides files that are read out as symlinks or hard links into a cache directory for the node, if permitted by the workflow.

        Also attempts to write files back to the backing JobStore asynchronously, after quickly taking them into the cache. Writes are only required to finish when the job's actual state after running is committed back to the job store.

        Internaly, manages caching using a database. Each node has its own database, shared between all the workers on the node. The database contains several tables:

        files contains one entry for each file in the cache. Each entry knows the path to its data on disk. It also knows its global file ID, its state, and its owning worker PID. If the owning worker dies, another worker will pick it up. It also knows its size.

        File states are:

        •   "cached": happily stored in the cache. Reads can happen immediately. Owner is null. May be adopted and moved to state "deleting" by anyone, if it has no outstanding immutable references.

        •   "downloading": in the process of being saved to the cache by a non-null owner. Reads must wait for the state to become "cached". If the worker dies, goes to state "deleting", because we don't know if it was fully downloaded or if anyone still needs it. No references can be created to a "downloading" file except by the worker responsible for downloading it.

        •   "uploadable": stored in the cache and ready to be written to the job store by a non-null owner. Transitions to "uploading" when a (thread of) the owning worker process picks it up and begins uploading it, to free cache space or to commit a completed job. If the worker dies, goes to state "cached", because it may have outstanding immutable references from the dead-but-not-cleaned-up job that was going to write it.

        •   "uploading": stored in the cache and being written to the job store by a non-null owner. Transitions to "cached" when successfully uploaded. If the worker dies, goes to state "cached", because it may have outstanding immutable references from the dead-but-not-cleaned-up job that was writing it.

        •   "deleting": in the process of being removed from the cache by a non-null owner. Will eventually be removed from the database.

    refs contains one entry for each outstanding reference to a cached file (hard link, symlink, or full copy). The table name is refs instead of references because references is an SQL reserved word. It remembers what job ID has the reference, and the path the reference is at. References have three states:

        •   "immutable": represents a hardlink or symlink to a file in the cache. Dedicates the file's size in bytes of the job's disk requirement to the cache, to be used to cache this file or to keep around other files without references. May be upgraded to "copying" if the link can't actually be created.

        •   "copying": records that a file in the cache is in the process of being copied to a path. Will be upgraded to a mutable reference eventually.

        •   "mutable": records that a file from the cache was copied to a certain path. Exist only to support deleteLocalFile's API. Only files with only mutable references (or no references) are eligible for eviction.

    jobs contains one entry for each job currently running. It keeps track of the job's ID, the worker that is supposed to be running the job, the job's disk requirement, and the job's local temp dir path that will need to be cleaned up. When workers check for jobs whose workers have died, they null out the old worker, and grab ownership of and clean up jobs and their references until the null-worker jobs are gone.

    properties contains key, value pairs for tracking total space available, and whether caching is free for this run.

        Parameters

            •   jobStore (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

            •   jobDesc (toil.job.JobDescription <#toil.job.JobDescription>)

            •   file_store_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   waitForPreviousCommit (Callable[[], Any])

        forceNonFreeCaching = False

        forceDownloadDelay = None

        contentionBackoff = 15

        localCacheDir

        jobName: str </usr/share/doc/python3/html/library/stdtypes.html#str> = ''

        jobID

        jobDiskBytes: float </usr/share/doc/python3/html/library/functions.html#float> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        workflowAttemptNumber

        dbPath

        process_identity_lock

        commitThread = None

        as_process()
            Assume the process's identity to act on the caching database.

            Yields the process's name in the caching database, and holds onto a lock while your thread has it.

            Return type
                collections.abc.Generator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, None, None]

        property con: sqlite3.Connection </usr/share/doc/python3/html/library/sqlite3.html#sqlite3.Connection>
            Get the database connection to be used for the current thread.

            Return type
                sqlite3.Connection </usr/share/doc/python3/html/library/sqlite3.html#sqlite3.Connection>

        property cur: sqlite3.Cursor </usr/share/doc/python3/html/library/sqlite3.html#sqlite3.Cursor>
            Get the main cursor to be used for the current thread.

            Return type
                sqlite3.Cursor </usr/share/doc/python3/html/library/sqlite3.html#sqlite3.Cursor>

        getCacheLimit()
            Return the total number of bytes to which the cache is limited.

            If no limit is available, raises an error.

        getCacheUsed()
            Return the total number of bytes used in the cache.

            If no value is available, raises an error.

        getCacheExtraJobSpace()
            Return the total number of bytes of disk space requested by jobs running against this cache but not yet used.

            We can get into a situation where the jobs on the node take up all its space, but then they want to write to or read from the cache. So when that happens, we need to debit space from them somehow...

            If no value is available, raises an error.

        getCacheAvailable()
            Return the total number of free bytes available for caching, or, if negative, the total number of bytes of cached files that need to be evicted to free up enough space for all the currently scheduled jobs.

            If no value is available, raises an error.

        getSpaceUsableForJobs()
            Return the total number of bytes that are not taken up by job requirements, ignoring files and file usage. We can't ever run more jobs than we actually have room for, even with caching.

            If not retrievable, raises an error.

        getCacheUnusedJobRequirement()
            Return the total number of bytes of disk space requested by the current job and not used by files the job is using in the cache.

            Mutable references don't count, but immutable/uploading ones do.

            If no value is available, raises an error.

        adjustCacheLimit(newTotalBytes)
            Adjust the total cache size limit to the given number of bytes.

        fileIsCached(fileID)
            Return true if the given file is currently cached, and false otherwise.

            Note that this can't really be relied upon because a file may go cached -> deleting after you look at it. If you need to do something with the file you need to do it in a transaction.

        getFileReaderCount(fileID)
            Return the number of current outstanding reads of the given file.

            Counts mutable references too.

        cachingIsFree()
            Return true if files can be cached for free, without taking up space. Return false otherwise.

            This will be true when working with certain job stores in certain configurations, most notably the FileJobStore.

        open(job)
            This context manager decorated method allows cache-specific operations to be conducted before and after the execution of a job in worker.py

            Parameters
                job (toil.job.Job <#toil.job.Job>)

            Return type
                collections.abc.Generator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator>[None, None, None]

        writeGlobalFile(localFileName, cleanup=False)
            Creates a file in the jobstore and returns a FileID reference.

        readGlobalFile(fileStoreID, userPath=None, cache=True, mutable=False, symlink=False)
            Make the file associated with fileStoreID available locally.

            If mutable is True, then a copy of the file will be created locally so that the original is not modified and does not change the file for other jobs. If mutable is False, then a link can be created to the file, saving disk resources. The file that is downloaded will be executable if and only if it was originally uploaded from an executable file on the local filesystem.

            If a user path is specified, it is used as the destination. If a user path isn't specified, the file is stored in the local temp directory with an encoded name.

            The destination file must not be deleted by the user; it can only be deleted through deleteLocalFile.

            Implementations must call logAccess() to report the download.

            Parameters

                •   fileStoreID -- job store id for the file

                •   userPath -- a path to the name of file to which the global file will be copied or hard-linked (see below).

                •   cache -- Described in toil.fileStores.CachingFileStore.readGlobalFile()

                •   mutable -- Described in toil.fileStores.CachingFileStore.readGlobalFile()

                •   symlink -- True if caller can accept symlink, False if caller can only accept a normal file or hardlink

            Returns
                An absolute path to a local, temporary copy of the file keyed by fileStoreID.

        readGlobalFileStream(fileStoreID, encoding=None, errors=None)
            Read a stream from the job store; similar to readGlobalFile.

            The yielded file handle does not need to and should not be closed explicitly.

            Parameters

                •   encoding -- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode.

                •   errors -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

        Implementations must call logAccess() to report the download.

            Returns
                a context manager yielding a file handle which can be read from.

        deleteLocalFile(fileStoreID)
            Delete local copies of files associated with the provided job store ID.

            Raises an OSError with an errno of errno.ENOENT if no such local copies exist. Thus, cannot be called multiple times in succession.

            The files deleted are all those previously read from this file ID via readGlobalFile by the current job into the job's file-store-provided temp directory, plus the file that was written to create the given file ID, if it was written by the current job from the job's file-store-provided temp directory.

            Parameters
                fileStoreID -- File Store ID of the file to be deleted.

        deleteGlobalFile(fileStoreID)
            Delete local files and then permanently deletes them from the job store.

            To ensure that the job can be restarted if necessary, the delete will not happen until after the job's run method has completed.

            Parameters
                fileStoreID -- the File Store ID of the file to be deleted.

        exportFile(jobStoreFileID, dstUrl)

            Parameters

                •   jobStoreFileID (toil.fileStores.FileID <#toil.fileStores.FileID>)

                •   dstUrl (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        export_file(file_id, dst_uri)

            Parameters

                •   file_id (toil.fileStores.FileID <#toil.fileStores.FileID>)

                •   dst_uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        waitForCommit()
            Blocks while startCommit is running.

            This function is called by this job's successor to ensure that it does not begin modifying the job store until after this job has finished doing so.

            Might be called when startCommit is never called on a particular instance, in which case it does not block.

            Returns
                Always returns True

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        startCommit(jobState=False)
            Update the status of the job on the disk.

            May bump the version number of the job.

            May start an asynchronous process. Call waitForCommit() to wait on that process. You must waitForCommit() before committing any further updates to the job. During the asynchronous process, it is safe to modify the job; modifications after this call will not be committed until the next call.

            Parameters
                jobState -- If True, commit the state of the FileStore's job, and file deletes. Otherwise, commit only file creates/updates.

        startCommitThread(state_to_commit)
            Run in a thread to actually commit the current job.

            Parameters
                state_to_commit (Optional[toil.job.JobDescription <#toil.job.JobDescription>])

        classmethod shutdown(shutdown_info)

            Parameters
                shutdown_info (tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Tuple of the coordination directory (where the cache database is) and the cache directory (where the cached data is).

            Return type
                None

        Job local temp directories will be removed due to their appearance in the database.

        __del__()
            Cleanup function that is run when destroying the class instance that ensures that all the file writing threads exit.

  toil.fileStores.nonCachingFileStore

  Attributes
logger |

  Classes
NonCachingFileStore | Interface used to allow user code run by Toil to read and write files.

  Module Contents

    toil.fileStores.nonCachingFileStore.logger: logging.Logger </usr/share/doc/python3/html/library/logging.html#logging.Logger>

    class toil.fileStores.nonCachingFileStore.NonCachingFileStore(jobStore, jobDesc, file_store_dir, waitForPreviousCommit)
        Bases: toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>

        Interface used to allow user code run by Toil to read and write files.

        Also provides the interface to other Toil facilities used by user code, including:

                •   normal (non-real-time) logging

                •   finding the correct temporary directory for scratch work

                •   importing and exporting files into and out of the workflow

    Stores user files in the jobStore, but keeps them separate from actual jobs.

    May implement caching.

    Passed as argument to the toil.job.Job.run() <#toil.job.Job.run> method.

    Access to files is only permitted inside the context manager provided by toil.fileStores.abstractFileStore.AbstractFileStore.open() <#toil.fileStores.abstractFileStore.AbstractFileStore.open>.

    Also responsible for committing completed jobs back to the job store with an update operation, and allowing that commit operation to be waited for.

        Parameters

            •   jobStore (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

            •   jobDesc (toil.job.JobDescription <#toil.job.JobDescription>)

            •   file_store_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   waitForPreviousCommit (Callable[[], Any])

        jobStateFile: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        localFileMap: DefaultDict[str </usr/share/doc/python3/html/library/stdtypes.html#str>, list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

        static check_for_coordination_corruption(coordination_dir)
            Make sure the coordination directory hasn't been deleted unexpectedly.

            Slurm has been known to delete XDG_RUNTIME_DIR out from under processes it was promised to, so it is possible that in certain misconfigured environments the coordination directory and everything in it could go away unexpectedly. We are going to regularly make sure that the things we think should exist actually exist, and we are going to abort if they do not.

            Parameters
                coordination_dir (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

        check_for_state_corruption()
            Make sure state tracking information hasn't been deleted unexpectedly.

            Return type
                None

        open(job)
            Create the context manager around tasks prior and after a job has been run.

            File operations are only permitted inside the context manager.

            Implementations must only yield from within with super().open(job):.

            Parameters
                job (toil.job.Job <#toil.job.Job>) -- The job instance of the toil job to run.

            Return type
                collections.abc.Generator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator>[None, None, None]

        writeGlobalFile(localFileName, cleanup=False)
            Upload a file (as a path) to the job store.

            If the file is in a FileStore-managed temporary directory (i.e. from toil.fileStores.abstractFileStore.AbstractFileStore.getLocalTempDir() <#toil.fileStores.abstractFileStore.AbstractFileStore.getLocalTempDir>), it will become a local copy of the file, eligible for deletion by toil.fileStores.abstractFileStore.AbstractFileStore.deleteLocalFile() <#toil.fileStores.abstractFileStore.AbstractFileStore.deleteLocalFile>.

            If an executable file on the local filesystem is uploaded, its executability will be preserved when it is downloaded again.

            Parameters

                •   localFileName (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The path to the local file to upload. The last path component (basename of the file) will remain associated with the file in the file store, if supported by the backing JobStore, so that the file can be searched for by name or name glob.

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- if True then the copy of the global file will be deleted once the job and all its successors have completed running. If not the global file must be deleted manually.

            Returns
                an ID that can be used to retrieve the file.

            Return type
                toil.fileStores.FileID <#toil.fileStores.FileID>

        readGlobalFile(fileStoreID, userPath=None, cache=True, mutable=False, symlink=False)
            Make the file associated with fileStoreID available locally.

            If mutable is True, then a copy of the file will be created locally so that the original is not modified and does not change the file for other jobs. If mutable is False, then a link can be created to the file, saving disk resources. The file that is downloaded will be executable if and only if it was originally uploaded from an executable file on the local filesystem.

            If a user path is specified, it is used as the destination. If a user path isn't specified, the file is stored in the local temp directory with an encoded name.

            The destination file must not be deleted by the user; it can only be deleted through deleteLocalFile.

            Implementations must call logAccess() to report the download.

            Parameters

                •   fileStoreID (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- job store id for the file

                •   userPath (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- a path to the name of file to which the global file will be copied or hard-linked (see below).

                •   cache (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Described in toil.fileStores.CachingFileStore.readGlobalFile()

                •   mutable (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Described in toil.fileStores.CachingFileStore.readGlobalFile()

                •   symlink (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- True if caller can accept symlink, False if caller can only accept a normal file or hardlink

            Returns
                An absolute path to a local, temporary copy of the file keyed by fileStoreID.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        readGlobalFileStream(fileStoreID: str </usr/share/doc/python3/html/library/stdtypes.html#str>, encoding: Literal[None] = None, errors: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> ContextManager[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]]

        readGlobalFileStream(fileStoreID: str </usr/share/doc/python3/html/library/stdtypes.html#str>, encoding: str </usr/share/doc/python3/html/library/stdtypes.html#str>, errors: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> ContextManager[IO[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]
            Read a stream from the job store; similar to readGlobalFile.

            The yielded file handle does not need to and should not be closed explicitly.

            Parameters

                •   encoding -- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode.

                •   errors -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

        Implementations must call logAccess() to report the download.

            Returns
                a context manager yielding a file handle which can be read from.

        exportFile(jobStoreFileID, dstUrl)

            Parameters

                •   jobStoreFileID (toil.fileStores.FileID <#toil.fileStores.FileID>)

                •   dstUrl (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        export_file(file_id, dst_uri)

            Parameters

                •   file_id (toil.fileStores.FileID <#toil.fileStores.FileID>)

                •   dst_uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        deleteLocalFile(fileStoreID)
            Delete local copies of files associated with the provided job store ID.

            Raises an OSError with an errno of errno.ENOENT if no such local copies exist. Thus, cannot be called multiple times in succession.

            The files deleted are all those previously read from this file ID via readGlobalFile by the current job into the job's file-store-provided temp directory, plus the file that was written to create the given file ID, if it was written by the current job from the job's file-store-provided temp directory.

            Parameters
                fileStoreID (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- File Store ID of the file to be deleted.

            Return type
                None

        deleteGlobalFile(fileStoreID)
            Delete local files and then permanently deletes them from the job store.

            To ensure that the job can be restarted if necessary, the delete will not happen until after the job's run method has completed.

            Parameters
                fileStoreID (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the File Store ID of the file to be deleted.

            Return type
                None

        waitForCommit()
            Blocks while startCommit is running.

            This function is called by this job's successor to ensure that it does not begin modifying the job store until after this job has finished doing so.

            Might be called when startCommit is never called on a particular instance, in which case it does not block.

            Returns
                Always returns True

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        startCommit(jobState=False)
            Update the status of the job on the disk.

            May bump the version number of the job.

            May start an asynchronous process. Call waitForCommit() to wait on that process. You must waitForCommit() before committing any further updates to the job. During the asynchronous process, it is safe to modify the job; modifications after this call will not be committed until the next call.

            Parameters
                jobState (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If True, commit the state of the FileStore's job, and file deletes. Otherwise, commit only file creates/updates.

            Return type
                None

        __del__()
            Cleanup function that is run when destroying the class instance. Nothing to do since there are no async write events.

            Return type
                None

        classmethod shutdown(shutdown_info)

            Parameters
                shutdown_info (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The coordination directory.

            Return type
                None

  Classes
FileID | A small wrapper around Python's builtin string class.

  Package Contents

    class toil.fileStores.FileID(fileStoreID, size, executable=False)
        Bases: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        A small wrapper around Python's builtin string class.

        It is used to represent a file's ID in the file store, and has a size attribute that is the file's size in bytes. This object is returned by importFile and writeGlobalFile.

        Calls into the file store can use bare strings; size will be queried from the job store if unavailable in the ID.

        Parameters

            •   fileStoreID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   size (int </usr/share/doc/python3/html/library/functions.html#int>)

            •   executable (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        size

        executable = False

        pack()
            Pack the FileID into a string so it can be passed through external code.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        classmethod forPath(fileStoreID, filePath)

            Parameters

                •   fileStoreID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   filePath (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                FileID <#toil.fileStores.FileID>

        classmethod unpack(packedFileStoreID)
            Unpack the result of pack() into a FileID object.

            Parameters
                packedFileStoreID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                FileID <#toil.fileStores.FileID>

  toil.job

  Attributes
logger |
REQUIREMENT_NAMES |
ParsedRequirement |
ParseableIndivisibleResource |
ParseableDivisibleResource |
ParseableFlag |
ParseableAcceleratorRequirement |
ParseableRequirement |
T |
Promised |

  Exceptions
JobPromiseConstraintError | Error for job being asked to promise its return value, but it not available.
ConflictingPredecessorError | Common base class for all non-exit exceptions.
DebugStoppingPointReached | Raised when a job reaches a point at which it has been instructed to stop for debugging.
FilesDownloadedStoppingPointReached | Raised when a job stops because it was asked to download its files, and the files are downloaded.
JobException | General job exception.
JobGraphDeadlockException | An exception raised in the event that a workflow contains an unresolvable dependency, such as a cycle. See toil.job.Job.checkJobGraphForDeadlocks().

  Classes
TemporaryID | Placeholder for a unregistered job ID used by a JobDescription.
AcceleratorRequirement | Requirement for one or more computational accelerators, like a GPU or FPGA.
RequirementsDict | Typed storage for requirements for a job.
Requirer | Base class implementing the storage and presentation of requirements.
JobBodyReference | Reference from a job description to its body.
JobDescription | Stores all the information that the Toil Leader ever needs to know about a Job.
ServiceJobDescription | A description of a job that hosts a service.
CheckpointJobDescription | A description of a job that is a checkpoint.
Job | Class represents a unit of work in toil.
FunctionWrappingJob | Job used to wrap a function. In its run method the wrapped function is called.
JobFunctionWrappingJob | A job function is a function whose first argument is a Job
PromisedRequirementFunctionWrappingJob | Handles dynamic resource allocation using toil.job.Promise instances.
PromisedRequirementJobFunctionWrappingJob | Handles dynamic resource allocation for job functions.
EncapsulatedJob | A convenience Job class used to make a job subgraph appear to be a single job.
ServiceHostJob | Job that runs a service. Used internally by Toil. Users should subclass Service instead of using this.
FileMetadata | Metadata for a file.
CombineImportsJob | Combine the outputs of multiple WorkerImportsJob into one promise
WorkerImportJob | Job to do file imports on a worker instead of a leader. Assumes all local and cloud files are accessible.
ImportsJob | Job to organize and delegate files to individual WorkerImportJobs.
Promise | References a return value from a method as a promise before the method itself is run.
PromisedRequirement | Class for dynamically allocating job function resource requirements.
UnfulfilledPromiseSentinel | This should be overwritten by a proper promised value.

  Functions
parse_accelerator(spec) | Parse an AcceleratorRequirement specified by user code.
accelerator_satisfies(candidate, requirement[, ignore]) | Test if candidate partially satisfies the given requirement.
accelerators_fully_satisfy(candidates, requirement[, ...]) | Determine if a set of accelerators satisfy a requirement.
potential_absolute_uris(uri, path[, importer, ...]) | Get potential absolute URIs to check for an imported file.
get_file_sizes(filenames, file_source[, search_paths, ...]) | Resolve relative-URI files in the given environment and turn them into absolute normalized URIs. Returns a dictionary of the string values from the WDL file values
unwrap(p) | Function for ensuring you actually have a promised value, and not just a promise.
unwrap_all(p) | Function for ensuring you actually have a collection of promised values,

  Module Contents

    toil.job.logger

    exception toil.job.JobPromiseConstraintError(promisingJob, recipientJob=None)
        Bases: RuntimeError </usr/share/doc/python3/html/library/exceptions.html#RuntimeError>

        Error for job being asked to promise its return value, but it not available.

        (Due to the return value not yet been hit in the topological order of the job graph.)

        Parameters

            •   promisingJob (Job <#toil.job.Job>)

            •   recipientJob (Optional[Job <#toil.job.Job>])

        promisingJob

        recipientJob = None

    exception toil.job.ConflictingPredecessorError(predecessor, successor)
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Common base class for all non-exit exceptions.

        Parameters

            •   predecessor (Job <#toil.job.Job>)

            •   successor (Job <#toil.job.Job>)

    exception toil.job.DebugStoppingPointReached
        Bases: BaseException </usr/share/doc/python3/html/library/exceptions.html#BaseException>

        Raised when a job reaches a point at which it has been instructed to stop for debugging.

    exception toil.job.FilesDownloadedStoppingPointReached(message, host_and_job_paths=None)
        Bases: DebugStoppingPointReached

        Raised when a job stops because it was asked to download its files, and the files are downloaded.

        Parameters

            •   message (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   host_and_job_paths (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]]])

        host_and_job_paths = None

    class toil.job.TemporaryID
        Placeholder for a unregistered job ID used by a JobDescription.

        Needs to be held:

            •   By JobDescription objects to record normal relationships.

            •   By Jobs to key their connected-component registries and to record predecessor relationships to facilitate EncapsulatedJob adding itself as a child.

            •   By Services to tie back to their hosting jobs, so the service tree can be built up from Service objects.

        __str__()

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        __repr__()

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        __hash__()

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        __eq__(other)

            Parameters
                other (Any)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        __ne__(other)

            Parameters
                other (Any)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

    class toil.job.AcceleratorRequirement
        Bases: TypedDict

        Requirement for one or more computational accelerators, like a GPU or FPGA.

        count: int </usr/share/doc/python3/html/library/functions.html#int>
            How many of the accelerator are needed to run the job.

        kind: str </usr/share/doc/python3/html/library/stdtypes.html#str>
            What kind of accelerator is required. Can be "gpu". Other kinds defined in the future might be "fpga", etc.

        model: typing_extensions.NotRequired[str </usr/share/doc/python3/html/library/stdtypes.html#str>]
            What model of accelerator is needed. The exact set of values available depends on what the backing scheduler calls its accelerators; strings like "nvidia-tesla-k80" might be expected to work. If a specific model of accelerator is not required, this should be absent.

        brand: typing_extensions.NotRequired[str </usr/share/doc/python3/html/library/stdtypes.html#str>]
            What brand or manufacturer of accelerator is required. The exact set of values available depends on what the backing scheduler calls the brands of its accleerators; strings like "nvidia" or "amd" might be expected to work. If a specific brand of accelerator is not required (for example, because the job can use multiple brands of accelerator that support a given API) this should be absent.

        api: typing_extensions.NotRequired[str </usr/share/doc/python3/html/library/stdtypes.html#str>]
            What API is to be used to communicate with the accelerator. This can be "cuda". Other APIs supported in the future might be "rocm", "opencl", "metal", etc. If the job does not need a particular API to talk to the accelerator, this should be absent.

    toil.job.parse_accelerator(spec)
        Parse an AcceleratorRequirement specified by user code.

        Supports formats like:

            >>> parse_accelerator(8)
            {'count': 8, 'kind': 'gpu'}

            >>> parse_accelerator("1")
            {'count': 1, 'kind': 'gpu'}

            >>> parse_accelerator("nvidia-tesla-k80")
            {'count': 1, 'kind': 'gpu', 'model': 'nvidia-tesla-k80', 'brand': 'nvidia'}

            >>> parse_accelerator("nvidia-tesla-k80:2")
            {'count': 2, 'kind': 'gpu', 'model': 'nvidia-tesla-k80', 'brand': 'nvidia'}

            >>> parse_accelerator("gpu")
            {'count': 1, 'kind': 'gpu'}

            >>> parse_accelerator("cuda:1")
            {'count': 1, 'kind': 'gpu', 'api': 'cuda', 'brand': 'nvidia'}

            >>> parse_accelerator({"kind": "gpu"})
            {'count': 1, 'kind': 'gpu'}

            >>> parse_accelerator({"brand": "nvidia", "count": 5})
            {'count': 5, 'kind': 'gpu', 'brand': 'nvidia'}

        Assumes that if not specified, we are talking about GPUs, and about one of them. Knows that "gpu" is a kind, and "cuda" is an API, and "nvidia" is a brand.

        Raises

            •   ValueError </usr/share/doc/python3/html/library/exceptions.html#ValueError> -- if it gets something it can't parse

            •   TypeError </usr/share/doc/python3/html/library/exceptions.html#TypeError> -- if it gets something it can't parse because it's the wrong type.

        Parameters
            spec (Union[int </usr/share/doc/python3/html/library/functions.html#int>, str </usr/share/doc/python3/html/library/stdtypes.html#str>, dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Union[str </usr/share/doc/python3/html/library/stdtypes.html#str>, int </usr/share/doc/python3/html/library/functions.html#int>]]])

        Return type
            AcceleratorRequirement <#toil.job.AcceleratorRequirement>

    toil.job.accelerator_satisfies(candidate, requirement, ignore=[])
        Test if candidate partially satisfies the given requirement.

        Returns
            True if the given candidate at least partially satisfies the given requirement (i.e. check all fields other than count).

        Parameters

            •   candidate (AcceleratorRequirement <#toil.job.AcceleratorRequirement>)

            •   requirement (AcceleratorRequirement <#toil.job.AcceleratorRequirement>)

            •   ignore (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.job.accelerators_fully_satisfy(candidates, requirement, ignore=[])
        Determine if a set of accelerators satisfy a requirement.

        Ignores fields specified in ignore.

        Returns
            True if the requirement AcceleratorRequirement is fully satisfied by the ones in the list, taken together (i.e. check all fields including count).

        Parameters

            •   candidates (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[AcceleratorRequirement <#toil.job.AcceleratorRequirement>]])

            •   requirement (AcceleratorRequirement <#toil.job.AcceleratorRequirement>)

            •   ignore (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    class toil.job.RequirementsDict
        Bases: TypedDict

        Typed storage for requirements for a job.

        Where requirement values are of different types depending on the requirement.

        cores: typing_extensions.NotRequired[int </usr/share/doc/python3/html/library/functions.html#int> | float </usr/share/doc/python3/html/library/functions.html#float>]

        memory: typing_extensions.NotRequired[int </usr/share/doc/python3/html/library/functions.html#int>]

        disk: typing_extensions.NotRequired[int </usr/share/doc/python3/html/library/functions.html#int>]

        accelerators: typing_extensions.NotRequired[list </usr/share/doc/python3/html/library/stdtypes.html#list>[AcceleratorRequirement <#toil.job.AcceleratorRequirement>]]

        preemptible: typing_extensions.NotRequired[bool </usr/share/doc/python3/html/library/functions.html#bool>]

    toil.job.REQUIREMENT_NAMES = ['disk', 'memory', 'cores', 'accelerators', 'preemptible']

    toil.job.ParsedRequirement

    toil.job.ParseableIndivisibleResource

    toil.job.ParseableDivisibleResource

    toil.job.ParseableFlag

    toil.job.ParseableAcceleratorRequirement

    toil.job.ParseableRequirement

    class toil.job.Requirer(requirements)
        Base class implementing the storage and presentation of requirements.

        Has cores, memory, disk, and preemptability as properties.

        Parameters
            requirements (Mapping[str </usr/share/doc/python3/html/library/stdtypes.html#str>, ParseableRequirement])

        assignConfig(config)
            Assign the given config object to be used to provide default values.

            Must be called exactly once on a loaded JobDescription before any requirements are queried.

            Parameters
                config (toil.common.Config <#toil.common.Config>) -- Config object to query

            Return type
                None

        __getstate__()
            Return the dict to use as the instance's __dict__ when pickling.

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]

        __copy__()
            Return a semantically-shallow copy of the object, for copy.copy().

            Return type
                Requirer <#toil.job.Requirer>

        __deepcopy__(memo)
            Return a semantically-deep copy of the object, for copy.deepcopy().

            Parameters
                memo (Any)

            Return type
                Requirer <#toil.job.Requirer>

        property requirements: RequirementsDict <#toil.job.RequirementsDict>
            Get dict containing all non-None, non-defaulted requirements.

            Return type
                RequirementsDict <#toil.job.RequirementsDict>

        property disk: int </usr/share/doc/python3/html/library/functions.html#int>
            Get the maximum number of bytes of disk required.

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        property memory: int </usr/share/doc/python3/html/library/functions.html#int>
            Get the maximum number of bytes of memory required.

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        property cores: int </usr/share/doc/python3/html/library/functions.html#int> | float </usr/share/doc/python3/html/library/functions.html#float>
            Get the number of CPU cores required.

            Return type
                Union[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>]

        property preemptible: bool </usr/share/doc/python3/html/library/functions.html#bool>
            Whether a preemptible node is permitted, or a nonpreemptible one is required.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        preemptable(val)

            Parameters
                val (ParseableFlag)

            Return type
                None

        property accelerators: list </usr/share/doc/python3/html/library/stdtypes.html#list>[AcceleratorRequirement <#toil.job.AcceleratorRequirement>]
            Any accelerators, such as GPUs, that are needed.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[AcceleratorRequirement <#toil.job.AcceleratorRequirement>]

        scale(requirement, factor)
            Return a copy of this object with the given requirement scaled up or down.

            Only works on requirements where that makes sense.

            Parameters

                •   requirement (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   factor (float </usr/share/doc/python3/html/library/functions.html#float>)

            Return type
                Requirer <#toil.job.Requirer>

        requirements_string()
            Get a nice human-readable string of our requirements.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

    class toil.job.JobBodyReference
        Bases: NamedTuple

        Reference from a job description to its body.

        file_store_id: str </usr/share/doc/python3/html/library/stdtypes.html#str>
            File ID (or special shared file name for the root job) of the job's body.

        module_string: str </usr/share/doc/python3/html/library/stdtypes.html#str>
            Stringified description of the module needed to load the body.

    class toil.job.JobDescription(requirements, jobName, unitName='', displayName='', local=None, files=None)
        Bases: Requirer

        Stores all the information that the Toil Leader ever needs to know about a Job.

        This includes:

            •   Resource requirements.

            •   Which jobs are children or follow-ons or predecessors of this job.

            •   A reference to the Job object in the job store.

    Can be obtained from an actual (i.e. executable) Job object, and can be used to obtain the Job object from the JobStore.

    Never contains other Jobs or JobDescriptions: all reference is by ID.

    Subclassed into variants for checkpoint jobs and service jobs that have their specific parameters.

        Parameters

            •   requirements (Mapping[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Union[int </usr/share/doc/python3/html/library/functions.html#int>, str </usr/share/doc/python3/html/library/stdtypes.html#str>, float </usr/share/doc/python3/html/library/functions.html#float>, bool </usr/share/doc/python3/html/library/functions.html#bool>, list </usr/share/doc/python3/html/library/stdtypes.html#list>]])

            •   jobName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   unitName (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   displayName (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   local (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>])

            •   files (Optional[set </usr/share/doc/python3/html/library/stdtypes.html#set>[toil.fileStores.FileID <#toil.fileStores.FileID>]])

        local: bool </usr/share/doc/python3/html/library/functions.html#bool> = False

        jobName

        unitName = ''

        displayName = ''

        jobStoreID: str </usr/share/doc/python3/html/library/stdtypes.html#str> | TemporaryID <#toil.job.TemporaryID>

        filesToDelete = []

        predecessorNumber = 0

        predecessorsFinished

        childIDs: set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        followOnIDs: set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        successor_phases: list </usr/share/doc/python3/html/library/stdtypes.html#list>[set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

        serviceTree

        logJobStoreFileID = None

        files_to_use

        get_names()
            Get the names and ID of this job as a named tuple.

            Return type
                toil.bus.Names <#toil.bus.Names>

        get_chain()
            Get all the jobs that executed in this job's chain, in order.

            For each job, produces a named tuple with its various names and its original job store ID. The jobs in the chain are in execution order.

            If the job hasn't run yet or it didn't chain, produces a one-item list.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.bus.Names <#toil.bus.Names>]

        serviceHostIDsInBatches()
            Find all batches of service host job IDs that can be started at the same time.

            (in the order they need to start in)

            Return type
                Iterator[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

        successorsAndServiceHosts()
            Get an iterator over all child, follow-on, and service job IDs.

            Return type
                Iterator[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        allSuccessors()
            Get an iterator over all child, follow-on, and chained, inherited successor job IDs.

            Follow-ons will come before children.

            Return type
                Iterator[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        successors_by_phase()
            Get an iterator over all child/follow-on/chained inherited successor job IDs, along with their phase number on the stack.

            Phases execute higher numbers to lower numbers.

            Return type
                Iterator[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[int </usr/share/doc/python3/html/library/functions.html#int>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

        property services
            Get a collection of the IDs of service host jobs for this job, in arbitrary order.

            Will be empty if the job has no unfinished services.

        has_body()
            Returns True if we have a job body associated, and False otherwise.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        attach_body(file_store_id, user_script)
            Attach a job body to this JobDescription.

            Takes the file store ID that the body is stored at, and the required user script module.

            The file store ID can also be "firstJob" for the root job, stored as a shared file instead.

            Parameters

                •   file_store_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   user_script (toil.resource.ModuleDescriptor <#toil.resource.ModuleDescriptor>)

            Return type
                None

        detach_body()
            Drop the body reference from a JobDescription.

            Return type
                None

        get_body()
            Get the information needed to load the job body.

            Returns
                a file store ID (or magic shared file name "firstJob") and a user script module.

            Return type
                tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.resource.ModuleDescriptor <#toil.resource.ModuleDescriptor>]

        Fails if no body is attached; check has_body() first.

        nextSuccessors()
            Return the collection of job IDs for the successors of this job that are ready to run.

            If those jobs have multiple predecessor relationships, they may still be blocked on other jobs.

            Returns None when at the final phase (all successors done), and an empty collection if there are more phases but they can't be entered yet (e.g. because we are waiting for the job itself to run).

            Return type
                Optional[set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

        filterSuccessors(predicate)
            Keep only successor jobs for which the given predicate function approves.

            The predicate function is called with the job's ID.

            Treats all other successors as complete and forgets them.

            Parameters
                predicate (Callable[[str </usr/share/doc/python3/html/library/stdtypes.html#str>], bool </usr/share/doc/python3/html/library/functions.html#bool>])

            Return type
                None

        filterServiceHosts(predicate)
            Keep only services for which the given predicate approves.

            The predicate function is called with the service host job's ID.

            Treats all other services as complete and forgets them.

            Parameters
                predicate (Callable[[str </usr/share/doc/python3/html/library/stdtypes.html#str>], bool </usr/share/doc/python3/html/library/functions.html#bool>])

            Return type
                None

        clear_nonexistent_dependents(job_store)
            Remove all references to child, follow-on, and associated service jobs that do not exist.

            That is to say, all those that have been completed and removed.

            Parameters
                job_store (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

            Return type
                None

        clear_dependents()
            Remove all references to successor and service jobs.

            Return type
                None

        is_subtree_done()
            Check if the subtree is done.

            Returns
                True if the job appears to be done, and all related child, follow-on, and service jobs appear to be finished and removed.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        replace(other)
            Take on the ID of another JobDescription, retaining our own state and type.

            When updated in the JobStore, we will save over the other JobDescription.

            Useful for chaining jobs: the chained-to job can replace the parent job.

            Merges cleanup state and successors other than this job from the job being replaced into this one.

            Parameters
                other (JobDescription <#toil.job.JobDescription>) -- Job description to replace.

            Return type
                None

        assert_is_not_newer_than(other)
            Make sure this JobDescription is not newer than a prospective new version of the JobDescription.

            Parameters
                other (JobDescription <#toil.job.JobDescription>)

            Return type
                None

        is_updated_by(other)
            Return True if the passed JobDescription is a distinct, newer version of this one.

            Parameters
                other (JobDescription <#toil.job.JobDescription>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        addChild(childID)
            Make the job with the given ID a child of the described job.

            Parameters
                childID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        addFollowOn(followOnID)
            Make the job with the given ID a follow-on of the described job.

            Parameters
                followOnID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        addServiceHostJob(serviceID, parentServiceID=None)
            Make the ServiceHostJob with the given ID a service of the described job.

            If a parent ServiceHostJob ID is given, that parent service will be started first, and must have already been added.

        hasChild(childID)
            Return True if the job with the given ID is a child of the described job.

            Parameters
                childID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        hasFollowOn(followOnID)
            Test if the job with the given ID is a follow-on of the described job.

            Parameters
                followOnID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        hasServiceHostJob(serviceID)
            Test if the ServiceHostJob is a service of the described job.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        renameReferences(renames)
            Apply the given dict of ID renames to all references to jobs.

            Does not modify our own ID or those of finished predecessors. IDs not present in the renames dict are left as-is.

            Parameters
                renames (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[TemporaryID <#toil.job.TemporaryID>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Rename operations to apply.

            Return type
                None

        addPredecessor()
            Notify the JobDescription that a predecessor has been added to its Job.

            Return type
                None

        onRegistration(jobStore)
            Perform setup work that requires the JobStore.

            Called by the Job saving logic when this JobDescription meets the JobStore and has its ID assigned.

            Overridden to perform setup work (like hooking up flag files for service jobs) that requires the JobStore.

            Parameters
                jobStore (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>) -- The job store we are being placed into

            Return type
                None

        setupJobAfterFailure(exit_status=None, exit_reason=None)
            Configure job after a failure.

            Reduce the remainingTryCount if greater than zero and set the memory to be at least as big as the default memory (in case of exhaustion of memory, which is common).

            Requires a configuration to have been assigned (see toil.job.Requirer.assignConfig()).

            Parameters

                •   exit_status (Optional[int </usr/share/doc/python3/html/library/functions.html#int>]) -- The exit code from the job.

                •   exit_reason (Optional[toil.batchSystems.abstractBatchSystem.BatchJobExitReason <#toil.batchSystems.abstractBatchSystem.BatchJobExitReason>]) -- The reason the job stopped, if available from the batch system.

            Return type
                None

        getLogFileHandle(jobStore)
            Create a context manager that yields a file handle to the log file.

            Assumes logJobStoreFileID is set.

        property remainingTryCount
            Get the number of tries remaining.

            The try count set on the JobDescription, or the default based on the retry count from the config if none is set.

        clearRemainingTryCount()
            Clear remainingTryCount and set it back to its default value.

            Returns
                True if a modification to the JobDescription was made, and False otherwise.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        __str__()
            Produce a useful logging string identifying this job.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        __repr__()

        reserve_versions(count)
            Reserve a job version number for later, for journaling asynchronously.

            Parameters
                count (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                None

        pre_update_hook()
            Run before pickling and saving a created or updated version of this job.

            Called by the job store.

            Return type
                None

    class toil.job.ServiceJobDescription(*args, **kwargs)
        Bases: JobDescription

        A description of a job that hosts a service.

        terminateJobStoreID: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        startJobStoreID: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        errorJobStoreID: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        onRegistration(jobStore)
            Setup flag files.

            When a ServiceJobDescription first meets the JobStore, it needs to set up its flag files.

    class toil.job.CheckpointJobDescription(*args, **kwargs)
        Bases: JobDescription

        A description of a job that is a checkpoint.

        checkpoint: JobBodyReference <#toil.job.JobBodyReference> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        checkpointFilesToDelete = []

        set_checkpoint()
            Save a body checkpoint into self.checkpoint

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        restore_checkpoint()
            Restore the body checkpoint from self.checkpoint

            Return type
                None

        restartCheckpoint(jobStore)
            Restart a checkpoint after the total failure of jobs in its subtree.

            Writes the changes to the jobStore immediately. All the checkpoint's successors will be deleted, but its try count will not be decreased.

            Returns a list with the IDs of any successors deleted.

            Parameters
                jobStore (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    class toil.job.Job(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, preemptable=None, unitName='', checkpoint=False, displayName='', descriptionClass=None, local=None, files=None)
        Class represents a unit of work in toil.

        Parameters

            •   memory (Optional[ParseableIndivisibleResource])

            •   cores (Optional[ParseableDivisibleResource])

            •   disk (Optional[ParseableIndivisibleResource])

            •   accelerators (Optional[ParseableAcceleratorRequirement])

            •   preemptible (Optional[ParseableFlag])

            •   preemptable (Optional[ParseableFlag])

            •   unitName (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   checkpoint (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>])

            •   displayName (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   descriptionClass (Optional[type </usr/share/doc/python3/html/library/functions.html#type>])

            •   local (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>])

            •   files (Optional[set </usr/share/doc/python3/html/library/stdtypes.html#set>[toil.fileStores.FileID <#toil.fileStores.FileID>]])

        userModule: toil.resource.ModuleDescriptor <#toil.resource.ModuleDescriptor>

        __str__()
            Produce a useful logging string to identify this Job and distinguish it from its JobDescription.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        check_initialized()
            Ensure that Job.__init__() has been called by any subclass __init__().

            This uses the fact that the self._description instance variable should always be set after __init__().

            If __init__() has not been called, raise an error.

            Return type
                None

        property jobStoreID: str </usr/share/doc/python3/html/library/stdtypes.html#str> | TemporaryID <#toil.job.TemporaryID>
            Get the ID of this Job.

            Return type
                Union[str </usr/share/doc/python3/html/library/stdtypes.html#str>, TemporaryID <#toil.job.TemporaryID>]

        property description: JobDescription <#toil.job.JobDescription>
            Expose the JobDescription that describes this job.

            Return type
                JobDescription <#toil.job.JobDescription>

        property disk: int </usr/share/doc/python3/html/library/functions.html#int>
            The maximum number of bytes of disk the job will require to run.

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        property memory: int </usr/share/doc/python3/html/library/functions.html#int>
            The maximum number of bytes of memory the job will require to run.

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        property cores: int </usr/share/doc/python3/html/library/functions.html#int> | float </usr/share/doc/python3/html/library/functions.html#float>
            The number of CPU cores required.

            Return type
                Union[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>]

        property accelerators: list </usr/share/doc/python3/html/library/stdtypes.html#list>[AcceleratorRequirement <#toil.job.AcceleratorRequirement>]
            Any accelerators, such as GPUs, that are needed.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[AcceleratorRequirement <#toil.job.AcceleratorRequirement>]

        property preemptible: bool </usr/share/doc/python3/html/library/functions.html#bool>
            Whether the job can be run on a preemptible node.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        preemptable()

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        property checkpoint: bool </usr/share/doc/python3/html/library/functions.html#bool>
            Determine if the job is a checkpoint job or not.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        property files_to_use: set </usr/share/doc/python3/html/library/stdtypes.html#set>[toil.fileStores.FileID <#toil.fileStores.FileID>]

            Return type
                set </usr/share/doc/python3/html/library/stdtypes.html#set>[toil.fileStores.FileID <#toil.fileStores.FileID>]

        add_to_files_to_use(val)

            Parameters
                val (toil.fileStores.FileID <#toil.fileStores.FileID>)

            Return type
                None

        remove_from_files_to_use(val)

            Parameters
                val (toil.fileStores.FileID <#toil.fileStores.FileID>)

            Return type
                None

        assignConfig(config)
            Assign the given config object.

            It will be used by various actions implemented inside the Job class.

            Parameters
                config (toil.common.Config <#toil.common.Config>) -- Config object to query

            Return type
                None

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>) -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv().

            Return type
                Any

        addChild(childJob)
            Add a childJob to be run as child of this job.

            Child jobs will be run directly after this job's toil.job.Job.run() method has completed.

            Returns
                childJob: for call chaining

            Parameters
                childJob (Job <#toil.job.Job>)

            Return type
                Job <#toil.job.Job>

        hasChild(childJob)
            Check if childJob is already a child of this job.

            Returns
                True if childJob is a child of the job, else False.

            Parameters
                childJob (Job <#toil.job.Job>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        addFollowOn(followOnJob)
            Add a follow-on job.

            Follow-on jobs will be run after the child jobs and their successors have been run.

            Returns
                followOnJob for call chaining

            Parameters
                followOnJob (Job <#toil.job.Job>)

            Return type
                Job <#toil.job.Job>

        hasPredecessor(job)
            Check if a given job is already a predecessor of this job.

            Parameters
                job (Job <#toil.job.Job>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        hasFollowOn(followOnJob)
            Check if given job is already a follow-on of this job.

            Returns
                True if the followOnJob is a follow-on of this job, else False.

            Parameters
                followOnJob (Job <#toil.job.Job>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        addService(service, parentService=None)
            Add a service.

            The toil.job.Job.Service.start() method of the service will be called after the run method has completed but before any successors are run. The service's toil.job.Job.Service.stop() method will be called once the successors of the job have been run.

            Services allow things like databases and servers to be started and accessed by jobs in a workflow.

            Raises
                toil.job.JobException <#toil.job.JobException> -- If service has already been made the child of a job or another service.

            Parameters

                •   service (Job <#toil.job.Job>) -- Service to add.

                •   parentService (Optional[Job <#toil.job.Job>]) -- Service that will be started before 'service' is started. Allows trees of services to be established. parentService must be a service of this job.

            Returns
                a promise that will be replaced with the return value from toil.job.Job.Service.start() of service in any successor of the job.

            Return type
                Promise <#toil.job.Promise>

        hasService(service)
            Return True if the given Service is a service of this job, and False otherwise.

            Parameters
                service (Job <#toil.job.Job>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        addChildFn(fn, *args, **kwargs)
            Add a function as a child job.

            Parameters
                fn (Callable) -- Function to be run as a child job with *args and **kwargs as arguments to this function. See toil.job.FunctionWrappingJob for reserved keyword arguments used to specify resource requirements.

            Returns
                The new child job that wraps fn.

            Return type
                FunctionWrappingJob <#toil.job.FunctionWrappingJob>

        addFollowOnFn(fn, *args, **kwargs)
            Add a function as a follow-on job.

            Parameters
                fn (Callable) -- Function to be run as a follow-on job with *args and **kwargs as arguments to this function. See toil.job.FunctionWrappingJob for reserved keyword arguments used to specify resource requirements.

            Returns
                The new follow-on job that wraps fn.

            Return type
                FunctionWrappingJob <#toil.job.FunctionWrappingJob>

        addChildJobFn(fn, *args, **kwargs)
            Add a job function as a child job.

            See toil.job.JobFunctionWrappingJob for a definition of a job function.

            Parameters
                fn (Callable) -- Job function to be run as a child job with *args and **kwargs as arguments to this function. See toil.job.JobFunctionWrappingJob for reserved keyword arguments used to specify resource requirements.

            Returns
                The new child job that wraps fn.

            Return type
                FunctionWrappingJob <#toil.job.FunctionWrappingJob>

        addFollowOnJobFn(fn, *args, **kwargs)
            Add a follow-on job function.

            See toil.job.JobFunctionWrappingJob for a definition of a job function.

            Parameters
                fn (Callable) -- Job function to be run as a follow-on job with *args and **kwargs as arguments to this function. See toil.job.JobFunctionWrappingJob for reserved keyword arguments used to specify resource requirements.

            Returns
                The new follow-on job that wraps fn.

            Return type
                FunctionWrappingJob <#toil.job.FunctionWrappingJob>

        property tempDir: str </usr/share/doc/python3/html/library/stdtypes.html#str>
            Shortcut to calling job.fileStore.getLocalTempDir().

            Temp dir is created on first call and will be returned for first and future calls :return: Path to tempDir. See job.fileStore.getLocalTempDir

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        log(text, level=logging.INFO)
            Log using fileStore.log_to_leader().

            Parameters
                text (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        static wrapFn(fn, *args, **kwargs)
            Makes a Job out of a function.

            Convenience function for constructor of toil.job.FunctionWrappingJob.

            Parameters
                fn -- Function to be run with *args and **kwargs as arguments. See toil.job.JobFunctionWrappingJob for reserved keyword arguments used to specify resource requirements.

            Returns
                The new function that wraps fn.

            Return type
                FunctionWrappingJob <#toil.job.FunctionWrappingJob>

        static wrapJobFn(fn, *args, **kwargs)
            Makes a Job out of a job function.

            Convenience function for constructor of toil.job.JobFunctionWrappingJob.

            Parameters
                fn -- Job function to be run with *args and **kwargs as arguments. See toil.job.JobFunctionWrappingJob for reserved keyword arguments used to specify resource requirements.

            Returns
                The new job function that wraps fn.

            Return type
                JobFunctionWrappingJob <#toil.job.JobFunctionWrappingJob>

        encapsulate(name=None)
            Encapsulates the job, see toil.job.EncapsulatedJob. Convenience function for constructor of toil.job.EncapsulatedJob.

            Parameters
                name (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Human-readable name for the encapsulated job.

            Returns
                an encapsulated version of this job.

            Return type
                EncapsulatedJob <#toil.job.EncapsulatedJob>

        rv(*path)
            Create a promise (toil.job.Promise).

            The "promise" representing a return value of the job's run method, or, in case of a function-wrapping job, the wrapped function's return value.

            Parameters
                path ((Any)) -- Optional path for selecting a component of the promised return value. If absent or empty, the entire return value will be used. Otherwise, the first element of the path is used to select an individual item of the return value. For that to work, the return value must be a list, dictionary or of any other type implementing the __getitem__() magic method. If the selected item is yet another composite value, the second element of the path can be used to select an item from it, and so on. For example, if the return value is [6,{'a':42}], .rv(0) would select 6 , rv(1) would select {'a':3} while rv(1,'a') would select 3. To select a slice from a return value that is slicable, e.g. tuple or list, the path element should be a slice object. For example, assuming that the return value is [6, 7, 8, 9] then .rv(slice(1, 3)) would select [7, 8]. Note that slicing really only makes sense at the end of path.

            Returns
                A promise representing the return value of this jobs toil.job.Job.run() method.

            Return type
                Promise <#toil.job.Promise>

        registerPromise(path)

        prepareForPromiseRegistration(jobStore)
            Set up to allow this job's promises to register themselves.

            Prepare this job (the promisor) so that its promises can register themselves with it, when the jobs they are promised to (promisees) are serialized.

            The promissee holds the reference to the promise (usually as part of the job arguments) and when it is being pickled, so will the promises it refers to. Pickling a promise triggers it to be registered with the promissor.

            Parameters
                jobStore (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

            Return type
                None

        checkJobGraphForDeadlocks()
            Ensures that a graph of Jobs (that hasn't yet been saved to the JobStore) doesn't contain any pathological relationships between jobs that would result in deadlocks if we tried to run the jobs.

            See toil.job.Job.checkJobGraphConnected(), toil.job.Job.checkJobGraphAcyclic() and toil.job.Job.checkNewCheckpointsAreLeafVertices() for more info.

            Raises
                toil.job.JobGraphDeadlockException <#toil.job.JobGraphDeadlockException> -- if the job graph is cyclic, contains multiple roots or contains checkpoint jobs that are not leaf vertices when defined (see toil.job.Job.checkNewCheckpointsAreLeaves()).

        getRootJobs()
            Return the set of root job objects that contain this job.

            A root job is a job with no predecessors (i.e. which are not children, follow-ons, or services).

            Only deals with jobs created here, rather than loaded from the job store.

            Return type
                set </usr/share/doc/python3/html/library/stdtypes.html#set>[Job <#toil.job.Job>]

        checkJobGraphConnected()

            Raises
                toil.job.JobGraphDeadlockException <#toil.job.JobGraphDeadlockException> -- if toil.job.Job.getRootJobs() does not contain exactly one root job.

            Return type
                None

        As execution always starts from one root job, having multiple root jobs will cause a deadlock to occur.

        Only deals with jobs created here, rather than loaded from the job store.

        checkJobGraphAcylic()

            Raises
                toil.job.JobGraphDeadlockException <#toil.job.JobGraphDeadlockException> -- if the connected component of jobs containing this job contains any cycles of child/followOn dependencies in the augmented job graph (see below). Such cycles are not allowed in valid job graphs.

            Return type
                None

        A follow-on edge (A, B) between two jobs A and B is equivalent to adding a child edge to B from (1) A, (2) from each child of A, and (3) from the successors of each child of A. We call each such edge an edge an "implied" edge. The augmented job graph is a job graph including all the implied edges.

        For a job graph G = (V, E) the algorithm is O(|V|^2). It is O(|V| + |E|) for a graph with no follow-ons. The former follow-on case could be improved!

        Only deals with jobs created here, rather than loaded from the job store.

        checkNewCheckpointsAreLeafVertices()
            A checkpoint job is a job that is restarted if either it fails, or if any of its successors completely fails, exhausting their retries.

            A job is a leaf it is has no successors.

            A checkpoint job must be a leaf when initially added to the job graph. When its run method is invoked it can then create direct successors. This restriction is made to simplify implementation.

            Only works on connected components of jobs not yet added to the JobStore.

            Raises
                toil.job.JobGraphDeadlockException <#toil.job.JobGraphDeadlockException> -- if there exists a job being added to the graph for which checkpoint=True and which is not a leaf.

            Return type
                None

        defer(function, *args, **kwargs)
            Register a deferred function, i.e. a callable that will be invoked after the current attempt at running this job concludes. A job attempt is said to conclude when the job function (or the toil.job.Job.run() method for class-based jobs) returns, raises an exception or after the process running it terminates abnormally. A deferred function will be called on the node that attempted to run the job, even if a subsequent attempt is made on another node. A deferred function should be idempotent because it may be called multiple times on the same node or even in the same process. More than one deferred function may be registered per job attempt by calling this method repeatedly with different arguments. If the same function is registered twice with the same or different arguments, it will be called twice per job attempt.

            Examples for deferred functions are ones that handle cleanup of resources external to Toil, like Docker containers, files outside the work directory, etc.

            Parameters

                •   function (callable) -- The function to be called after this job concludes.

                •   args (list </usr/share/doc/python3/html/library/stdtypes.html#list>) -- The arguments to the function

                •   kwargs (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>) -- The keyword arguments to the function

            Return type
                None

        class Runner
            Used to setup and run Toil workflow.

            static getDefaultArgumentParser(jobstore_as_flag=False)
                Get argument parser with added toil workflow options.

                This is the Right Way to get an argument parser in a Toil Python workflow.

                Parameters
                    jobstore_as_flag (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- make the job store option a --jobStore flag instead of a required jobStore positional argument.

                Returns
                    The argument parser used by a toil workflow with added Toil options.

                Return type
                    configargparse.ArgParser

            static getDefaultOptions(jobStore=None, jobstore_as_flag=False)
                Get default options for a toil workflow.

                Parameters

                    •   jobStore (Optional[toil.lib.misc.StrPath]) -- A string describing the jobStore for the workflow.

                    •   jobstore_as_flag (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- make the job store option a --jobStore flag instead of a required jobStore positional argument.

                Returns
                    The options used by a toil workflow.

                Return type
                    argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>

            static addToilOptions(parser, jobstore_as_flag=False)
                Adds the default toil options to an optparse </usr/share/doc/python3/html/library/optparse.html#module-optparse> or argparse </usr/share/doc/python3/html/library/argparse.html#module-argparse> parser object.

                Consider using getDefaultArgumentParser() instead, which will produce a parser of the correct class to use Toil's config file and environment variables. If ther parser passed here is just an argparse.ArgumentParser </usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser> and not a configargparse.ArgParser, the Toil config file and environment variables will not be respected.

                Parameters

                    •   parser (Union[optparse.OptionParser </usr/share/doc/python3/html/library/optparse.html#optparse.OptionParser>, argparse.ArgumentParser </usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser>]) -- Options object to add toil options to.

                    •   jobstore_as_flag (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- make the job store option a --jobStore flag instead of a required jobStore positional argument.

                Return type
                    None

            static startToil(job, options)
                Run the toil workflow using the given options.

                Deprecated by toil.common.Toil.start.

                (see Job.Runner.getDefaultOptions and Job.Runner.addToilOptions) starting with this job. :param job: root job of the workflow :raises: toil.exceptions.FailedJobsException if at the end of function there remain failed jobs. :return: The return value of the root job's run function.

                Parameters
                    job (Job <#toil.job.Job>)

                Return type
                    Any

        class Service(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, unitName='')
            Bases: Requirer

            Abstract class used to define the interface to a service.

            Should be subclassed by the user to define services.

            Is not executed as a job; runs within a ServiceHostJob.

            Parameters

                •   memory (Optional[ParseableIndivisibleResource])

                •   cores (Optional[ParseableDivisibleResource])

                •   disk (Optional[ParseableIndivisibleResource])

                •   accelerators (Optional[ParseableAcceleratorRequirement])

                •   preemptible (Optional[ParseableFlag])

                •   unitName (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            unitName = ''

            jobName = 'Service'

            hostID = None

            abstractmethod start(job)
                Start the service.

                Parameters
                    job (ServiceHostJob <#toil.job.ServiceHostJob>) -- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files.

                Returns
                    An object describing how to access the service. The object must be pickleable and will be used by jobs to access the service (see toil.job.Job.addService()).

                Return type
                    Any

            abstractmethod stop(job)
                Stops the service. Function can block until complete.

                Parameters
                    job (ServiceHostJob <#toil.job.ServiceHostJob>) -- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files.

                Return type
                    None

            check()
                Checks the service is still running.

                Raises
                    exceptions.RuntimeError -- If the service failed, this will cause the service job to be labeled failed.

                Returns
                    True if the service is still running, else False. If False then the service job will be terminated, and considered a success. Important point: if the service job exits due to a failure, it should raise a RuntimeError, not return False!

                Return type
                    bool </usr/share/doc/python3/html/library/functions.html#bool>

        getUserScript()

            Return type
                toil.resource.ModuleDescriptor <#toil.resource.ModuleDescriptor>

        getTopologicalOrderingOfJobs()

            Returns
                a list of jobs such that for all pairs of indices i, j for which i < j, the job at index i can be run before the job at index j.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[Job <#toil.job.Job>]

        Only considers jobs in this job's subgraph that are newly added, not loaded from the job store.

        Ignores service jobs.

        saveBody(jobStore)
            Save the execution data for just this job to the JobStore, and fill in the JobDescription with the information needed to retrieve it.

            The Job's JobDescription must have already had a real jobStoreID assigned to it.

            Does not save the JobDescription.

            Parameters
                jobStore (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>) -- The job store to save the job body into.

            Return type
                None

        saveAsRootJob(jobStore)
            Save this job to the given jobStore as the root job of the workflow.

            Returns
                the JobDescription describing this job.

            Parameters
                jobStore (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

            Return type
                JobDescription <#toil.job.JobDescription>

        classmethod loadJob(job_store, job_description)
            Retrieves a toil.job.Job instance from a JobStore

            Parameters

                •   job_store (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>) -- The job store.

                •   job_description (JobDescription <#toil.job.JobDescription>) -- the JobDescription of the job to retrieve.

            Returns
                The job referenced by the JobDescription.

            Return type
                Job <#toil.job.Job>

        set_debug_flag(flag)
            Enable the given debug option on the job.

            Parameters
                flag (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        has_debug_flag(flag)
            Return true if the given debug flag is set.

            Parameters
                flag (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        files_downloaded_hook(host_and_job_paths=None)
            Function that subclasses can call when they have downloaded their input files.

            Will abort the job if the "download_only" debug flag is set.

            Can be hinted a list of file path pairs outside and inside the job container, in which case the container environment can be reconstructed.

            Parameters
                host_and_job_paths (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]]])

            Return type
                None

    exception toil.job.JobException(message)
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        General job exception.

        Parameters
            message (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

    exception toil.job.JobGraphDeadlockException(string)
        Bases: JobException

        An exception raised in the event that a workflow contains an unresolvable dependency, such as a cycle. See toil.job.Job.checkJobGraphForDeadlocks().

    class toil.job.FunctionWrappingJob(userFunction, *args, **kwargs)
        Bases: Job

        Job used to wrap a function. In its run method the wrapped function is called.

        Parameters

            •   userFunction (Callable[[Ellipsis], Any])

            •   args (Any)

            •   kwargs (Any)

        userFunctionModule

        userFunctionName = ''

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>) -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv().

            Return type
                Any

        getUserScript()

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

    class toil.job.JobFunctionWrappingJob(userFunction, *args, **kwargs)
        Bases: FunctionWrappingJob

        A job function is a function whose first argument is a Job instance that is the wrapping job for the function. This can be used to add successor jobs for the function and perform all the functions the Job class provides.

        To enable the job function to get access to the toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore> instance (see toil.job.Job.run()), it is made a variable of the wrapping job called fileStore.

        To specify a job's resource requirements the following default keyword arguments can be specified:

                •   memory

                •   disk

                •   cores

                •   accelerators

                •   preemptible

    For example to wrap a function into a job we would call:

                Job.wrapJobFn(myJob, memory='100k', disk='1M', cores=0.1)

        Parameters

            •   userFunction (Callable[[Ellipsis], Any])

            •   args (Any)

            •   kwargs (Any)

        property fileStore: toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>

            Return type
                toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>) -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv().

            Return type
                Any

    class toil.job.PromisedRequirementFunctionWrappingJob(userFunction, *args, **kwargs)
        Bases: FunctionWrappingJob

        Handles dynamic resource allocation using toil.job.Promise instances. Spawns child function using parent function parameters and fulfilled promised resource requirements.

        classmethod create(userFunction, *args, **kwargs)
            Creates an encapsulated Toil job function with unfulfilled promised resource requirements. After the promises are fulfilled, a child job function is created using updated resource values. The subgraph is encapsulated to ensure that this child job function is run before other children in the workflow. Otherwise, a different child may try to use an unresolved promise return value from the parent.

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv().

        evaluatePromisedRequirements()

    class toil.job.PromisedRequirementJobFunctionWrappingJob(userFunction, *args, **kwargs)
        Bases: PromisedRequirementFunctionWrappingJob

        Handles dynamic resource allocation for job functions. See toil.job.JobFunctionWrappingJob

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv().

    class toil.job.EncapsulatedJob(job, unitName=None)
        Bases: Job

        A convenience Job class used to make a job subgraph appear to be a single job.

        Let A be the root job of a job subgraph and B be another job we'd like to run after A and all its successors have completed, for this use encapsulate:

                #  Job A and subgraph, Job B
                A, B = A(), B()
                Aprime = A.encapsulate()
                Aprime.addChild(B)
                #  B will run after A and all its successors have completed, A and its subgraph of
                # successors in effect appear to be just one job.

    If the job being encapsulated has predecessors (e.g. is not the root job), then the encapsulated job will inherit these predecessors. If predecessors are added to the job being encapsulated after the encapsulated job is created then the encapsulating job will NOT inherit these predecessors automatically. Care should be exercised to ensure the encapsulated job has the proper set of predecessors.

    The return value of an encapsulated job (as accessed by the toil.job.Job.rv() function) is the return value of the root job, e.g. A().encapsulate().rv() and A().rv() will resolve to the same value after A or A.encapsulate() has been run.

        Parameters

            •   job (Optional[Job <#toil.job.Job>])

            •   unitName (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        addChild(childJob)
            Add a childJob to be run as child of this job.

            Child jobs will be run directly after this job's toil.job.Job.run() method has completed.

            Returns
                childJob: for call chaining

            Parameters
                childJob (Job <#toil.job.Job>)

            Return type
                Job <#toil.job.Job>

        addService(service, parentService=None)
            Add a service.

            The toil.job.Job.Service.start() method of the service will be called after the run method has completed but before any successors are run. The service's toil.job.Job.Service.stop() method will be called once the successors of the job have been run.

            Services allow things like databases and servers to be started and accessed by jobs in a workflow.

            Raises
                toil.job.JobException <#toil.job.JobException> -- If service has already been made the child of a job or another service.

            Parameters

                •   service -- Service to add.

                •   parentService -- Service that will be started before 'service' is started. Allows trees of services to be established. parentService must be a service of this job.

            Returns
                a promise that will be replaced with the return value from toil.job.Job.Service.start() of service in any successor of the job.

        addFollowOn(followOnJob)
            Add a follow-on job.

            Follow-on jobs will be run after the child jobs and their successors have been run.

            Returns
                followOnJob for call chaining

            Parameters
                followOnJob (Job <#toil.job.Job>)

            Return type
                Job <#toil.job.Job>

        rv(*path)
            Create a promise (toil.job.Promise).

            The "promise" representing a return value of the job's run method, or, in case of a function-wrapping job, the wrapped function's return value.

            Parameters
                path ((Any)) -- Optional path for selecting a component of the promised return value. If absent or empty, the entire return value will be used. Otherwise, the first element of the path is used to select an individual item of the return value. For that to work, the return value must be a list, dictionary or of any other type implementing the __getitem__() magic method. If the selected item is yet another composite value, the second element of the path can be used to select an item from it, and so on. For example, if the return value is [6,{'a':42}], .rv(0) would select 6 , rv(1) would select {'a':3} while rv(1,'a') would select 3. To select a slice from a return value that is slicable, e.g. tuple or list, the path element should be a slice object. For example, assuming that the return value is [6, 7, 8, 9] then .rv(slice(1, 3)) would select [7, 8]. Note that slicing really only makes sense at the end of path.

            Returns
                A promise representing the return value of this jobs toil.job.Job.run() method.

            Return type
                Promise <#toil.job.Promise>

        prepareForPromiseRegistration(jobStore)
            Set up to allow this job's promises to register themselves.

            Prepare this job (the promisor) so that its promises can register themselves with it, when the jobs they are promised to (promisees) are serialized.

            The promissee holds the reference to the promise (usually as part of the job arguments) and when it is being pickled, so will the promises it refers to. Pickling a promise triggers it to be registered with the promissor.

        __reduce__()
            Called during pickling to define the pickled representation of the job.

            We don't want to pickle our internal references to the job we encapsulate, so we elide them here. When actually run, we're just a no-op job that can maybe chain.

        getUserScript()

    class toil.job.ServiceHostJob(service)
        Bases: Job

        Job that runs a service. Used internally by Toil. Users should subclass Service instead of using this.

        serviceModule

        service

        pickledService = None

        property fileStore
            Return the file store, which the Service may need.

        addChild(child)
            Add a childJob to be run as child of this job.

            Child jobs will be run directly after this job's toil.job.Job.run() method has completed.

            Returns
                childJob: for call chaining

        addFollowOn(followOn)
            Add a follow-on job.

            Follow-on jobs will be run after the child jobs and their successors have been run.

            Returns
                followOnJob for call chaining

        addService(service, parentService=None)
            Add a service.

            The toil.job.Job.Service.start() method of the service will be called after the run method has completed but before any successors are run. The service's toil.job.Job.Service.stop() method will be called once the successors of the job have been run.

            Services allow things like databases and servers to be started and accessed by jobs in a workflow.

            Raises
                toil.job.JobException <#toil.job.JobException> -- If service has already been made the child of a job or another service.

            Parameters

                •   service -- Service to add.

                •   parentService -- Service that will be started before 'service' is started. Allows trees of services to be established. parentService must be a service of this job.

            Returns
                a promise that will be replaced with the return value from toil.job.Job.Service.start() of service in any successor of the job.

        saveBody(jobStore)
            Serialize the service itself before saving the host job's body.

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv().

        getUserScript()

    class toil.job.FileMetadata
        Bases: NamedTuple

        Metadata for a file. source is the URL to grab the file from parent_dir is parent directory of the source size is the size of the file. Is none if the filesize cannot be retrieved.

        source: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        parent_dir: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        size: int </usr/share/doc/python3/html/library/functions.html#int> | None </usr/share/doc/python3/html/library/constants.html#None>

    toil.job.potential_absolute_uris(uri, path, importer=None, execution_dir=None)
        Get potential absolute URIs to check for an imported file.

        Given a URI or bare path, yield in turn all the URIs, with schemes, where we should actually try to find it, given that we want to search under/against the given paths or URIs, the current directory, and the given importing WDL document if any.

        Parameters

            •   uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   path (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   importer (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   execution_dir (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            Iterator[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.job.get_file_sizes(filenames, file_source, search_paths=None, include_remote_files=True, execution_dir=None)
        Resolve relative-URI files in the given environment and turn them into absolute normalized URIs. Returns a dictionary of the string values from the WDL file values to a tuple of the normalized URI, parent directory ID, and size of the file. The size of the file may be None, which means unknown size.

        Parameters

            •   filenames (List[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- list of filenames to evaluate on

            •   file_source (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>) -- Context to search for files with

            •   task_path -- Dotted WDL name of the user-level code doing the importing (probably the workflow name).

            •   search_paths (Optional[List[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) -- If set, try resolving input location relative to the URLs or directories in this list.

            •   include_remote_files (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If set, import files from remote locations. Else leave them as URI references.

            •   execution_dir (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            Dict[str </usr/share/doc/python3/html/library/stdtypes.html#str>, FileMetadata <#toil.job.FileMetadata>]

    class toil.job.CombineImportsJob(d, **kwargs)
        Bases: Job

        Combine the outputs of multiple WorkerImportsJob into one promise

        Parameters
            d (Sequence[Promised[Dict[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.fileStores.FileID <#toil.fileStores.FileID>]]])

        run(file_store)
            Merge the dicts

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                Promised[Dict[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.fileStores.FileID <#toil.fileStores.FileID>]]

    class toil.job.WorkerImportJob(filenames, local=False, **kwargs)
        Bases: Job

        Job to do file imports on a worker instead of a leader. Assumes all local and cloud files are accessible.

        For the CWL/WDL runners, this class is only used when runImportsOnWorkers is enabled.

        Parameters

            •   filenames (List[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   local (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            •   kwargs (Any)

        filenames

        static import_files(files, file_source)
            Import a list of files into the jobstore. Returns a mapping of the filename to the associated FileIDs

            When stream is true but the import is not streamable, the worker will run out of disk space and run a new import job with enough disk space instead. :param files: list of files to import :param file_source: AbstractJobStore :return: Dictionary mapping filenames to associated jobstore FileID

            Parameters

                •   files (List[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   file_source (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

            Return type
                Dict[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.fileStores.FileID <#toil.fileStores.FileID>]

        run(file_store)
            Import the workflow inputs and then create and run the workflow. :return: Promise of workflow outputs

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                Promised[Dict[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.fileStores.FileID <#toil.fileStores.FileID>]]

    class toil.job.ImportsJob(file_to_data, max_batch_size, import_worker_disk, **kwargs)
        Bases: Job

        Job to organize and delegate files to individual WorkerImportJobs.

        For the CWL/WDL runners, this is only used when runImportsOnWorkers is enabled

        Parameters

            •   file_to_data (Dict[str </usr/share/doc/python3/html/library/stdtypes.html#str>, FileMetadata <#toil.job.FileMetadata>])

            •   max_batch_size (ParseableIndivisibleResource)

            •   import_worker_disk (ParseableIndivisibleResource)

            •   kwargs (Any)

        run(file_store)
            Import the workflow inputs and then create and run the workflow. :return: Tuple of a mapping from the candidate uri to the file id and a mapping of the source filenames to its metadata. The candidate uri is a field in the file metadata

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                Tuple[Promised[Dict[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.fileStores.FileID <#toil.fileStores.FileID>]], Dict[str </usr/share/doc/python3/html/library/stdtypes.html#str>, FileMetadata <#toil.job.FileMetadata>]]

    class toil.job.Promise(job, path)
        References a return value from a method as a promise before the method itself is run.

        References a return value from a toil.job.Job.run() or toil.job.Job.Service.start() method as a promise before the method itself is run.

        Let T be a job. Instances of Promise (termed a promise) are returned by T.rv(), which is used to reference the return value of T's run function. When the promise is passed to the constructor (or as an argument to a wrapped function) of a different, successor job the promise will be replaced by the actual referenced return value. This mechanism allows a return values from one job's run method to be input argument to job before the former job's run function has been executed.

        Parameters

            •   job (Job <#toil.job.Job>)

            •   path (Any)

        filesToDelete
            A set of IDs of files containing promised values when we know we won't need them anymore

        job

        path

        __reduce__()
            Return the Promise class and construction arguments.

            Called during pickling when a promise (an instance of this class) is about to be be pickled. Returns the Promise class and construction arguments that will be evaluated during unpickling, namely the job store coordinates of a file that will hold the promised return value. By the time the promise is about to be unpickled, that file should be populated.

    toil.job.T

    toil.job.Promised

    toil.job.unwrap(p)
        Function for ensuring you actually have a promised value, and not just a promise. Mostly useful for satisfying type-checking.

        The "unwrap" terminology is borrowed from Rust.

        Parameters
            p (Promised[T])

        Return type
            T

    toil.job.unwrap_all(p)
        Function for ensuring you actually have a collection of promised values, and not any remaining promises. Mostly useful for satisfying type-checking.

        The "unwrap" terminology is borrowed from Rust.

        Parameters
            p (Sequence[Promised[T]])

        Return type
            Sequence[T]

    class toil.job.PromisedRequirement(valueOrCallable, *args)
        Class for dynamically allocating job function resource requirements.

        (involving toil.job.Promise instances.)

        Use when resource requirements depend on the return value of a parent function. PromisedRequirements can be modified by passing a function that takes the Promise as input.

        For example, let f, g, and h be functions. Then a Toil workflow can be defined as follows:: A = Job.wrapFn(f) B = A.addChildFn(g, cores=PromisedRequirement(A.rv()) C = B.addChildFn(h, cores=PromisedRequirement(lambda x: 2*x, B.rv()))

        Parameters

            •   valueOrCallable (Any)

            •   args (Any)

        getValue()
            Return PromisedRequirement value.

            Return type
                Any

        static convertPromises(kwargs)
            Return True if reserved resource keyword is a Promise or PromisedRequirement instance.

            Converts Promise instance to PromisedRequirement.

            Parameters
                kwargs (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]) -- function keyword arguments

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

    class toil.job.UnfulfilledPromiseSentinel(fulfillingJobName, file_id, unpickled)
        This should be overwritten by a proper promised value.

        Throws an exception when unpickled.

        Parameters

            •   fulfillingJobName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   unpickled (Any)

        fulfillingJobName

        file_id

        static __setstate__(stateDict)
            Only called when unpickling.

            This won't be unpickled unless the promise wasn't resolved, so we throw an exception.

            Parameters
                stateDict (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any])

            Return type
                None

  toil.jobStores

  Submodules

  toil.jobStores.abstractJobStore

  Attributes
logger |

  Exceptions
ProxyConnectionError | Dummy class.
LocatorException | Base exception class for all locator exceptions.
InvalidImportExportUrlException | Common base class for all non-exit exceptions.
NoSuchJobException | Indicates that the specified job does not exist.
ConcurrentFileModificationException | Indicates that the file was attempted to be modified by multiple processes at once.
NoSuchFileException | Indicates that the specified file does not exist.
NoSuchJobStoreException | Indicates that the specified job store does not exist.
JobStoreExistsException | Indicates that the specified job store already exists.

  Classes
AbstractJobStore | Represents the physical storage for the jobs and files in a Toil workflow.
JobStoreSupport | A mostly fake JobStore to access URLs not really associated with real job

  Module Contents

    toil.jobStores.abstractJobStore.logger

    exception toil.jobStores.abstractJobStore.ProxyConnectionError
        Bases: BaseException </usr/share/doc/python3/html/library/exceptions.html#BaseException>

        Dummy class.

    exception toil.jobStores.abstractJobStore.LocatorException(error_msg, locator, prefix=None)
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Base exception class for all locator exceptions. For example, job store/aws bucket exceptions where they already exist

        Parameters

            •   error_msg (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   locator (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   prefix (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

    exception toil.jobStores.abstractJobStore.InvalidImportExportUrlException(url)
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Common base class for all non-exit exceptions.

        Parameters
            url (urllib.parse.ParseResult </usr/share/doc/python3/html/library/urllib.parse.html#urllib.parse.ParseResult>)

    exception toil.jobStores.abstractJobStore.NoSuchJobException(jobStoreID)
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Indicates that the specified job does not exist.

        Parameters
            jobStoreID (Union[toil.fileStores.FileID <#toil.fileStores.FileID>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

    exception toil.jobStores.abstractJobStore.ConcurrentFileModificationException(jobStoreFileID)
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Indicates that the file was attempted to be modified by multiple processes at once.

        Parameters
            jobStoreFileID (Union[toil.fileStores.FileID <#toil.fileStores.FileID>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

    exception toil.jobStores.abstractJobStore.NoSuchFileException(jobStoreFileID, customName=None, *extra)
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Indicates that the specified file does not exist.

        Parameters

            •   jobStoreFileID (Union[toil.fileStores.FileID <#toil.fileStores.FileID>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   customName (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   extra (Any)

    exception toil.jobStores.abstractJobStore.NoSuchJobStoreException(locator, prefix)
        Bases: LocatorException

        Indicates that the specified job store does not exist.

        Parameters

            •   locator (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   prefix (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

    exception toil.jobStores.abstractJobStore.JobStoreExistsException(locator, prefix)
        Bases: LocatorException

        Indicates that the specified job store already exists.

        Parameters

            •   locator (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   prefix (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

    class toil.jobStores.abstractJobStore.AbstractJobStore(locator)
        Bases: abc.ABC </usr/share/doc/python3/html/library/abc.html#abc.ABC>

        Represents the physical storage for the jobs and files in a Toil workflow.

        JobStores are responsible for storing toil.job.JobDescription <#toil.job.JobDescription> (which relate jobs to each other) and files.

        Actual toil.job.Job <#toil.job.Job> objects are stored in files, referenced by JobDescriptions. All the non-file CRUD methods the JobStore provides deal in JobDescriptions and not full, executable Jobs.

        To actually get ahold of a toil.job.Job <#toil.job.Job>, use toil.job.Job.loadJob() <#toil.job.Job.loadJob> with a JobStore and the relevant JobDescription.

        Parameters
            locator (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        initialize(config)
            Initialize this job store.

            Create the physical storage for this job store, allocate a workflow ID and persist the given Toil configuration to the store.

            Parameters
                config (toil.common.Config <#toil.common.Config>) -- the Toil configuration to initialize this job store with. The given configuration will be updated with the newly allocated workflow ID.

            Raises
                JobStoreExistsException <#toil.jobStores.abstractJobStore.JobStoreExistsException> -- if the physical storage for this job store already exists

            Return type
                None

        writeConfig()

            Return type
                None

        write_config()
            Persists the value of the AbstractJobStore.config attribute to the job store, so that it can be retrieved later by other instances of this class.

            Return type
                None

        resume()
            Connect this instance to the physical storage it represents and load the Toil configuration into the AbstractJobStore.config attribute.

            Raises
                NoSuchJobStoreException <#toil.jobStores.abstractJobStore.NoSuchJobStoreException> -- if the physical storage for this job store doesn't exist

            Return type
                None

        property config: toil.common.Config <#toil.common.Config>
            Return the Toil configuration associated with this job store.

            Raises
                AttributeError </usr/share/doc/python3/html/library/exceptions.html#AttributeError> -- if the config has not yet been assigned (i.e. during resume()).

            Return type
                toil.common.Config <#toil.common.Config>

        property locator: str </usr/share/doc/python3/html/library/stdtypes.html#str>
            Get the locator that defines the job store, which can be used to connect to it.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        rootJobStoreIDFileName = 'rootJobStoreID'

        setRootJob(rootJobStoreID)
            Set the root job of the workflow backed by this job store.

            Parameters
                rootJobStoreID (toil.fileStores.FileID <#toil.fileStores.FileID>)

            Return type
                None

        set_root_job(job_id)
            Set the root job of the workflow backed by this job store.

            Parameters
                job_id (toil.fileStores.FileID <#toil.fileStores.FileID>) -- The ID of the job to set as root

            Return type
                None

        loadRootJob()

            Return type
                toil.job.JobDescription <#toil.job.JobDescription>

        load_root_job()
            Loads the JobDescription for the root job in the current job store.

            Raises
                toil.job.JobException <#toil.job.JobException> -- If no root job is set or if the root job doesn't exist in this job store

            Returns
                The root job.

            Return type
                toil.job.JobDescription <#toil.job.JobDescription>

        createRootJob(desc)

            Parameters
                desc (toil.job.JobDescription <#toil.job.JobDescription>)

            Return type
                toil.job.JobDescription <#toil.job.JobDescription>

        create_root_job(job_description)
            Create the given JobDescription and set it as the root job in this job store.

            Parameters
                job_description (toil.job.JobDescription <#toil.job.JobDescription>) -- JobDescription to save and make the root job.

            Return type
                toil.job.JobDescription <#toil.job.JobDescription>

        getRootJobReturnValue()

            Return type
                Any

        get_root_job_return_value()
            Parse the return value from the root job.

            Raises an exception if the root job hasn't fulfilled its promise yet.

            Return type
                Any

        importFile(srcUrl: str </usr/share/doc/python3/html/library/stdtypes.html#str>, sharedFileName: str </usr/share/doc/python3/html/library/stdtypes.html#str>, hardlink: bool </usr/share/doc/python3/html/library/functions.html#bool> = False, symlink: bool </usr/share/doc/python3/html/library/functions.html#bool> = True) -> None </usr/share/doc/python3/html/library/constants.html#None>

        importFile(srcUrl: str </usr/share/doc/python3/html/library/stdtypes.html#str>, sharedFileName: None </usr/share/doc/python3/html/library/constants.html#None> = None, hardlink: bool </usr/share/doc/python3/html/library/functions.html#bool> = False, symlink: bool </usr/share/doc/python3/html/library/functions.html#bool> = True) -> toil.fileStores.FileID <#toil.fileStores.FileID>

        import_file(src_uri: str </usr/share/doc/python3/html/library/stdtypes.html#str>, shared_file_name: str </usr/share/doc/python3/html/library/stdtypes.html#str>, hardlink: bool </usr/share/doc/python3/html/library/functions.html#bool> = False, symlink: bool </usr/share/doc/python3/html/library/functions.html#bool> = True) -> None </usr/share/doc/python3/html/library/constants.html#None>

        import_file(src_uri: str </usr/share/doc/python3/html/library/stdtypes.html#str>, shared_file_name: None </usr/share/doc/python3/html/library/constants.html#None> = None, hardlink: bool </usr/share/doc/python3/html/library/functions.html#bool> = False, symlink: bool </usr/share/doc/python3/html/library/functions.html#bool> = True) -> toil.fileStores.FileID <#toil.fileStores.FileID>
            Imports the file at the given URL into job store. The ID of the newly imported file is returned. If the name of a shared file name is provided, the file will be imported as such and None is returned. If an executable file on the local filesystem is uploaded, its executability will be preserved when it is downloaded.

            Currently supported schemes are:

                    •

                        's3' for objects in Amazon S3
                            e.g. s3://bucket/key

                    •

                        'file' for local files
                            e.g. <file:///local/file/path>

                    •

                        'http'
                            e.g. <http://someurl.com/path>

                    •

                        'gs'
                            e.g. gs://bucket/file

        Raises FileNotFoundError if the file does not exist.

            Parameters

                •   src_uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- URL that points to a file or object in the storage mechanism of a supported URL scheme e.g. a blob in an AWS s3 bucket. It must be a file, not a directory or prefix.

                •   shared_file_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Optional name to assign to the imported file within the job store

            Returns
                The jobStoreFileID of the imported file or None if shared_file_name was given

            Return type
                toil.fileStores.FileID <#toil.fileStores.FileID> or None

        exportFile(jobStoreFileID, dstUrl)

            Parameters

                •   jobStoreFileID (toil.fileStores.FileID <#toil.fileStores.FileID>)

                •   dstUrl (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        export_file(file_id, dst_uri)
            Exports file to destination pointed at by the destination URL. The exported file will be executable if and only if it was originally uploaded from an executable file on the local filesystem.

            Refer to AbstractJobStore.import_file() documentation for currently supported URL schemes.

            Note that the helper method _exportFile is used to read from the source and write to destination. To implement any optimizations that circumvent this, the _exportFile method should be overridden by subclasses of AbstractJobStore.

            Parameters

                •   file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The id of the file in the job store that should be exported.

                •   dst_uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- URL that points to a file or object in the storage mechanism of a supported URL scheme e.g. a blob in an AWS s3 bucket. May also be a local path.

            Return type
                None

        abstractmethod destroy()
            The inverse of initialize(), this method deletes the physical storage represented by this instance. While not being atomic, this method is at least idempotent, as a means to counteract potential issues with eventual consistency exhibited by the underlying storage mechanisms. This means that if the method fails (raises an exception), it may (and should be) invoked again. If the underlying storage mechanism is eventually consistent, even a successful invocation is not an ironclad guarantee that the physical storage vanished completely and immediately. A successful invocation only guarantees that the deletion will eventually happen. It is therefore recommended to not immediately reuse the same job store location for a new Toil workflow.

            Return type
                None

        getEnv()

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        get_env()
            Returns a dictionary of environment variables that this job store requires to be set in order to function properly on a worker.

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>,str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        clean(jobCache=None)
            Function to cleanup the state of a job store after a restart.

            Fixes jobs that might have been partially updated. Resets the try counts and removes jobs that are not successors of the current root job.

            Parameters
                jobCache (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[Union[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.job.TemporaryID <#toil.job.TemporaryID>], toil.job.JobDescription <#toil.job.JobDescription>]]) -- if a value it must be a dict from job ID keys to JobDescription object values. Jobs will be loaded from the cache (which can be downloaded from the job store in a batch) instead of piecemeal when recursed into.

            Return type
                toil.job.JobDescription <#toil.job.JobDescription>

        assignID(jobDescription)

            Parameters
                jobDescription (toil.job.JobDescription <#toil.job.JobDescription>)

            Return type
                None

        abstractmethod assign_job_id(job_description)
            Get a new jobStoreID to be used by the described job, and assigns it to the JobDescription.

            Files associated with the assigned ID will be accepted even if the JobDescription has never been created or updated.

            Parameters
                job_description (toil.job.JobDescription <#toil.job.JobDescription>) -- The JobDescription to give an ID to

            Return type
                None

        create(jobDescription)

            Parameters
                jobDescription (toil.job.JobDescription <#toil.job.JobDescription>)

            Return type
                toil.job.JobDescription <#toil.job.JobDescription>

        abstractmethod create_job(job_description)
            Writes the given JobDescription to the job store. The job must have an ID assigned already.

            Must call jobDescription.pre_update_hook()

            Returns
                The JobDescription passed.

            Return type
                toil.job.JobDescription <#toil.job.JobDescription>

            Parameters
                job_description (toil.job.JobDescription <#toil.job.JobDescription>)

        exists(jobStoreID)

            Parameters
                jobStoreID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        abstractmethod job_exists(job_id)
            Indicates whether a description of the job with the specified jobStoreID exists in the job store

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

            Parameters
                job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        publicUrlExpiration

        getPublicUrl(fileName)

            Parameters
                fileName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        abstractmethod get_public_url(file_name)
            Returns a publicly accessible URL to the given file in the job store. The returned URL may expire as early as 1h after its been returned. Throw an exception if the file does not exist.

            Parameters
                file_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the jobStoreFileID of the file to generate a URL for

            Raises
                NoSuchFileException <#toil.jobStores.abstractJobStore.NoSuchFileException> -- if the specified file does not exist in this job store

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        getSharedPublicUrl(sharedFileName)

            Parameters
                sharedFileName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        abstractmethod get_shared_public_url(shared_file_name)
            Differs from getPublicUrl() in that this method is for generating URLs for shared files written by writeSharedFileStream().

            Returns a publicly accessible URL to the given file in the job store. The returned URL starts with 'http:', 'https:' or 'file:'. The returned URL may expire as early as 1h after its been returned. Throw an exception if the file does not exist.

            Parameters
                shared_file_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The name of the shared file to generate a publically accessible url for.

            Raises
                NoSuchFileException <#toil.jobStores.abstractJobStore.NoSuchFileException> -- raised if the specified file does not exist in the store

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        load(jobStoreID)

            Parameters
                jobStoreID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                toil.job.JobDescription <#toil.job.JobDescription>

        abstractmethod load_job(job_id)
            Loads the description of the job referenced by the given ID, assigns it the job store's config, and returns it.

            May declare the job to have failed (see toil.job.JobDescription.setupJobAfterFailure() <#toil.job.JobDescription.setupJobAfterFailure>) if there is evidence of a failed update attempt.

            Parameters
                job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the ID of the job to load

            Raises
                NoSuchJobException <#toil.jobStores.abstractJobStore.NoSuchJobException> -- if there is no job with the given ID

            Return type
                toil.job.JobDescription <#toil.job.JobDescription>

        update(jobDescription)

            Parameters
                jobDescription (toil.job.JobDescription <#toil.job.JobDescription>)

            Return type
                None

        batch()
            If supported by the batch system, calls to create() with this context manager active will be performed in a batch after the context manager is released.

            Return type
                collections.abc.Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[None]

        abstractmethod update_job(job_description)
            Persists changes to the state of the given JobDescription in this store atomically.

            Must call jobDescription.pre_update_hook()

            Parameters

                •   job (toil.job.JobDescription <#toil.job.JobDescription>) -- the job to write to this job store

                •   job_description (toil.job.JobDescription <#toil.job.JobDescription>)

            Return type
                None

        delete(jobStoreID)

            Parameters
                jobStoreID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        abstractmethod delete_job(job_id)
            Removes the JobDescription from the store atomically. You may not then subsequently call load(), write(), update(), etc. with the same jobStoreID or any JobDescription bearing it.

            This operation is idempotent, i.e. deleting a job twice or deleting a non-existent job will succeed silently.

            Parameters
                job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the ID of the job to delete from this job store

            Return type
                None

        abstractmethod jobs()
            Best effort attempt to return iterator on JobDescriptions for all jobs in the store. The iterator may not return all jobs and may also contain orphaned jobs that have already finished successfully and should not be rerun. To guarantee you get any and all jobs that can be run instead construct a more expensive ToilState object

            Returns
                Returns iterator on jobs in the store. The iterator may or may not contain all jobs and may contain invalid jobs

            Return type
                Iterator[toil.job.jobDescription]

        writeFile(localFilePath, jobStoreID=None, cleanup=False)

            Parameters

                •   localFilePath (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   jobStoreID (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        abstractmethod write_file(local_path, job_id=None, cleanup=False)
            Takes a file (as a path) and places it in this job store. Returns an ID that can be used to retrieve the file at a later time. The file is written in a atomic manner. It will not appear in the jobStore until the write has successfully completed.

            Parameters

                •   local_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the path to the local file that will be uploaded to the job store. The last path component (basename of the file) will remain associated with the file in the file store, if supported, so that the file can be searched for by name or name glob.

                •   job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the id of a job, or None. If specified, the may be associated with that job in a job-store-specific way. This may influence the returned ID.

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing.

            Raises

                •   ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

                •   NoSuchJobException <#toil.jobStores.abstractJobStore.NoSuchJobException> -- if the job specified via jobStoreID does not exist

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        FIXME: some implementations may not raise this

            Returns
                an ID referencing the newly created file and can be used to read the file in the future.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

            Parameters

                •   local_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   job_id (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        writeFileStream(jobStoreID=None, cleanup=False, basename=None, encoding=None, errors=None)

            Parameters

                •   jobStoreID (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>)

                •   basename (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   encoding (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   errors (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                ContextManager[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>], str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

        abstractmethod write_file_stream(job_id=None, cleanup=False, basename=None, encoding=None, errors=None)
            Similar to writeFile, but returns a context manager yielding a tuple of 1) a file handle which can be written to and 2) the ID of the resulting file in the job store. The yielded file handle does not need to and should not be closed explicitly. The file is written in a atomic manner. It will not appear in the jobStore until the write has successfully completed.

            Parameters

                •   job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the id of a job, or None. If specified, the may be associated with that job in a job-store-specific way. This may influence the returned ID.

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing.

                •   basename (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- If supported by the implementation, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected.

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

            Raises

                •   ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

                •   NoSuchJobException <#toil.jobStores.abstractJobStore.NoSuchJobException> -- if the job specified via jobStoreID does not exist

            Return type
                collections.abc.Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>], str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

        FIXME: some implementations may not raise this

            Returns
                a context manager yielding a file handle which can be written to and an ID that references the newly created file and can be used to read the file in the future.

            Return type
                Iterator[Tuple[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>], str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

            Parameters

                •   job_id (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>)

                •   basename (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   encoding (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   errors (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        getEmptyFileStoreID(jobStoreID=None, cleanup=False, basename=None)

            Parameters

                •   jobStoreID (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>)

                •   basename (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        abstractmethod get_empty_file_store_id(job_id=None, cleanup=False, basename=None)
            Creates an empty file in the job store and returns its ID. Call to fileExists(getEmptyFileStoreID(jobStoreID)) will return True.

            Parameters

                •   job_id (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- the id of a job, or None. If specified, the may be associated with that job in a job-store-specific way. This may influence the returned ID.

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing.

                •   basename (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- If supported by the implementation, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected.

            Returns
                a jobStoreFileID that references the newly created file and can be used to reference the file in the future.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        readFile(jobStoreFileID, localFilePath, symlink=False)

            Parameters

                •   jobStoreFileID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   localFilePath (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   symlink (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        abstractmethod read_file(file_id, local_path, symlink=False)
            Copies or hard links the file referenced by jobStoreFileID to the given local file path. The version will be consistent with the last copy of the file written/updated. If the file in the job store is later modified via updateFile or updateFileStream, it is implementation-defined whether those writes will be visible at localFilePath. The file is copied in an atomic manner. It will not appear in the local file system until the copy has completed.

            The file at the given local path may not be modified after this method returns!

            Note! Implementations of readFile need to respect/provide the executable attribute on FileIDs.

            Parameters

                •   file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- ID of the file to be copied

                •   local_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the local path indicating where to place the contents of the given file in the job store

                •   symlink (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- whether the reader can tolerate a symlink. If set to true, the job store may create a symlink instead of a full copy of the file or a hard link.

            Return type
                None

        readFileStream(jobStoreFileID, encoding=None, errors=None)

            Parameters

                •   jobStoreFileID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   encoding (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   errors (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                Union[ContextManager[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]], ContextManager[IO[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]]

        read_file_stream(file_id: toil.fileStores.FileID <#toil.fileStores.FileID> | str </usr/share/doc/python3/html/library/stdtypes.html#str>, encoding: Literal[None] = None, errors: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> ContextManager[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]]

        read_file_stream(file_id: toil.fileStores.FileID <#toil.fileStores.FileID> | str </usr/share/doc/python3/html/library/stdtypes.html#str>, encoding: str </usr/share/doc/python3/html/library/stdtypes.html#str>, errors: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> ContextManager[IO[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]
            Similar to readFile, but returns a context manager yielding a file handle which can be read from. The yielded file handle does not need to and should not be closed explicitly.

            Parameters

                •   file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- ID of the file to get a readable file handle for

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

            Returns
                a context manager yielding a file handle which can be read from

            Return type
                Iterator[Union[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>], IO[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]]

        deleteFile(jobStoreFileID)

            Parameters
                jobStoreFileID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        abstractmethod delete_file(file_id)
            Deletes the file with the given ID from this job store. This operation is idempotent, i.e. deleting a file twice or deleting a non-existent file will succeed silently.

            Parameters
                file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- ID of the file to delete

            Return type
                None

        fileExists(jobStoreFileID)
            Determine whether a file exists in this job store.

            Parameters
                jobStoreFileID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        abstractmethod file_exists(file_id)
            Determine whether a file exists in this job store.

            Parameters
                file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an ID referencing the file to be checked

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        getFileSize(jobStoreFileID)
            Get the size of the given file in bytes.

            Parameters
                jobStoreFileID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        abstractmethod get_file_size(file_id)
            Get the size of the given file in bytes, or 0 if it does not exist when queried.

            Note that job stores which encrypt files might return overestimates of file sizes, since the encrypted file may have been padded to the nearest block, augmented with an initialization vector, etc.

            Parameters
                file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an ID referencing the file to be checked

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        updateFile(jobStoreFileID, localFilePath)
            Replaces the existing version of a file in the job store.

            Parameters

                •   jobStoreFileID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   localFilePath (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        abstractmethod update_file(file_id, local_path)
            Replaces the existing version of a file in the job store.

            Throws an exception if the file does not exist.

            Parameters

                •   file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the ID of the file in the job store to be updated

                •   local_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the local path to a file that will overwrite the current version in the job store

            Raises

                •   ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

                •   NoSuchFileException <#toil.jobStores.abstractJobStore.NoSuchFileException> -- if the specified file does not exist

            Return type
                None

        updateFileStream(jobStoreFileID, encoding=None, errors=None)

            Parameters

                •   jobStoreFileID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   encoding (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   errors (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                ContextManager[IO[Any]]

        abstractmethod update_file_stream(file_id, encoding=None, errors=None)
            Replaces the existing version of a file in the job store. Similar to writeFile, but returns a context manager yielding a file handle which can be written to. The yielded file handle does not need to and should not be closed explicitly.

            Parameters

                •   file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the ID of the file in the job store to be updated

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

            Raises

                •   ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

                •   NoSuchFileException <#toil.jobStores.abstractJobStore.NoSuchFileException> -- if the specified file does not exist

            Return type
                collections.abc.Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[IO[Any]]

        sharedFileNameRegex

        writeSharedFileStream(sharedFileName, isProtected=None, encoding=None, errors=None)

            Parameters

                •   sharedFileName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   isProtected (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>])

                •   encoding (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   errors (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                ContextManager[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]]

        abstractmethod write_shared_file_stream(shared_file_name, encrypted=None, encoding=None, errors=None)
            Returns a context manager yielding a writable file handle to the global file referenced by the given name. File will be created in an atomic manner.

            Parameters

                •   shared_file_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- A file name matching AbstractJobStore.fileNameRegex, unique within this job store

                •   encrypted (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- True if the file must be encrypted, None if it may be encrypted or False if it must be stored in the clear.

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

            Raises
                ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

            Returns
                a context manager yielding a writable file handle

            Return type
                Iterator[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]]

        readSharedFileStream(sharedFileName, encoding=None, errors=None)

            Parameters

                •   sharedFileName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   encoding (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   errors (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                Union[ContextManager[IO[str </usr/share/doc/python3/html/library/stdtypes.html#str>]], ContextManager[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]]]

        abstractmethod read_shared_file_stream(shared_file_name: str </usr/share/doc/python3/html/library/stdtypes.html#str>, encoding: str </usr/share/doc/python3/html/library/stdtypes.html#str>, errors: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> collections.abc.Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[IO[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

        abstractmethod read_shared_file_stream(shared_file_name: str </usr/share/doc/python3/html/library/stdtypes.html#str>, encoding: Literal[None] = None, errors: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> collections.abc.Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]]
            Returns a context manager yielding a readable file handle to the global file referenced by the given name.

            Parameters

                •   shared_file_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- A file name matching AbstractJobStore.fileNameRegex, unique within this job store

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

            Returns
                a context manager yielding a readable file handle

        writeStatsAndLogging(statsAndLoggingString)

            Parameters
                statsAndLoggingString (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        abstractmethod write_logs(msg)
            Stores a message as a log in the jobstore.

            Parameters
                msg (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the string to be written

            Raises
                ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

            Return type
                None

        readStatsAndLogging(callback, readAll=False)

            Parameters

                •   callback (Callable[Ellipsis, Any])

                •   readAll (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        abstractmethod read_logs(callback, read_all=False)
            Reads logs accumulated by the write_logs() method. For each log this method calls the given callback function with the message as an argument (rather than returning logs directly, this method must be supplied with a callback which will process log messages).

            Only unread logs will be read unless the read_all parameter is set.

            Parameters

                •   callback (Callable) -- a function to be applied to each of the stats file handles found

                •   read_all (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- a boolean indicating whether to read the already processed stats files in addition to the unread stats files

            Raises
                ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

            Returns
                the number of stats files processed

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        write_leader_pid()
            Write the pid of this process to a file in the job store.

            Overwriting the current contents of pid.log is a feature, not a bug of this method. Other methods will rely on always having the most current pid available. So far there is no reason to store any old pids.

            Return type
                None

        read_leader_pid()
            Read the pid of the leader process to a file in the job store.

            Raises
                NoSuchFileException <#toil.jobStores.abstractJobStore.NoSuchFileException> -- If the PID file doesn't exist.

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        write_leader_node_id()
            Write the leader node id to the job store. This should only be called by the leader.

            Return type
                None

        read_leader_node_id()
            Read the leader node id stored in the job store.

            Raises
                NoSuchFileException <#toil.jobStores.abstractJobStore.NoSuchFileException> -- If the node ID file doesn't exist.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        write_kill_flag(kill=False)
            Write a file inside the job store that serves as a kill flag.

            The initialized file contains the characters "NO". This should only be changed when the user runs the "toil kill" command.

            Changing this file to a "YES" triggers a kill of the leader process. The workers are expected to be cleaned up by the leader.

            Parameters
                kill (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        read_kill_flag()
            Read the kill flag from the job store, and return True if the leader has been killed. False otherwise.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        default_caching()
            Jobstore's preference as to whether it likes caching or doesn't care about it. Some jobstores benefit from caching, however on some local configurations it can be flaky.

            see <https://github.com/DataBiosphere/toil/issues/4218>

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

    class toil.jobStores.abstractJobStore.JobStoreSupport(locator)
        Bases: AbstractJobStore, toil.lib.url.URLAccess <#toil.lib.url.URLAccess>

        A mostly fake JobStore to access URLs not really associated with real job stores.

        Parameters
            locator (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

  toil.jobStores.aws

  Submodules

  toil.jobStores.aws.jobStore

This file contains the AWS jobstore, which has its own docstring defining its use.

This docstring is about the organization of the file.

All direct AWS boto calls should live in toil.lib.aws, except for creating the session instance and the resource/client (which should only be made ONCE in the jobstore).

    Reasons for this

        •   DRY.

        •   All retries are on their individual boto functions, instead of here.

        •   Simple clear functions => simple clear unit tests (ideally).

Variables defining part size, parallelization, and other constants should live in toil.lib.aws.config.

  Attributes
DEFAULT_AWS_PART_SIZE |
logger |

  Classes
AWSJobStore | The AWS jobstore can be thought of as an AWS s3 bucket, with functions to

  Functions
parse_jobstore_identifier(jobstore_identifier) |

  Module Contents

    toil.jobStores.aws.jobStore.DEFAULT_AWS_PART_SIZE = 52428800

    toil.jobStores.aws.jobStore.logger

    class toil.jobStores.aws.jobStore.AWSJobStore(locator, partSize=DEFAULT_AWS_PART_SIZE)
        Bases: toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>, toil.lib.url.URLAccess <#toil.lib.url.URLAccess>

        The AWS jobstore can be thought of as an AWS s3 bucket, with functions to centralize, store, and track files for the workflow.

        The AWS jobstore stores 4 things:

        1.  Jobs: These are pickled as files, and contain the information necessary to run a job when unpickled. A job's file is deleted when finished, and its absence means it completed.

        2.  Files: The inputs and outputs of jobs. Each file is written in s3 with the file pattern: "files/{uuid4}/{original_filename}", where the file prefix "files/{uuid4}" should only point to one file.

        3.  Logs: The written log files of jobs that have run, plus the log file for the main Toil process.

        4.  Shared Files: Files with himan=-readable names, used by Toil itself or Python workflows. These include:

            •   environment.pickle (environment variables)

            •   config.pickle (user options)

            •   pid.log (process ID of the workflow; when it finishes, the workflow either succeeded/failed)

            •   userScript (hot deployment; this is the job module)

            •   rootJobReturnValue (workflow succeeded or not)

        NOTES

            •   The AWS jobstore does not use a database (directly, at least) currently. We can get away with this because:

                1.  AWS s3 has strong consistency.

                2.  s3's filter/query speed is pretty good.

            However, there may be reasons in the future to provide users with a database:

                •   s3 throttling has limits (3,500/5,000 requests (TODO: per second?); something like dynamodb supports 100,000+ requests).

                •   Access and filtering would be sped up, though how much faster this would be needs testing.

            ALSO NOTE: The caching filestore uses a local (per node) database with a very similar structure that maybe could be synced up with this.

            •   TODO: Etags are s3's native checksum, so use that for file integrity checking since it's free when fetching object headers from s3. Using an md5sum in addition to this would work well with the current filestore. WARNING: Etag values differ for the same file when the part size changes, so part size should always be Set In Stone, unless we hit s3's 10,000 part limit, and we need to account for that.

            •   This class fills in self.config only when initialized/restarted; it is None upon class instantiation. These are the options/config set by the user. When jobs are loaded/unpickled, they must re-incorporate this.

            •   The config.sseKey field is the single source of truth for bucket encryption status. The key is never stored inside this class; it is always read from the file referenced by the config when needed. Modifying the config at runtime will modify whether encryption is used. Note that files written without encryption (i.e. config.pickle) can't be read when encryption is enabled!

            •   TODO: In general, job stores should log the version of Toil they were initialized with and warn the user if restarting with a different version.

        Parameters

            •   locator (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   partSize (int </usr/share/doc/python3/html/library/functions.html#int>)

        s3_resource

        s3_client

        part_size = 52428800

        bucket = None

        job_key_prefix = 'jobs/'

        job_associations_key_prefix = 'job-associations/'

        content_key_prefix = 'files/'

        shared_key_prefix = ''

        logs_key_prefix = 'logs/'

        initialize(config)
            Called when starting a new jobstore with a non-existent bucket.

            Create bucket, raise if it already exists. Set options from config.

            Parameters
                config (toil.common.Config <#toil.common.Config>)

            Return type
                None

        resume()
            Called when reusing an old jobstore with an existing bucket.

            Raises
                NoSuchJobStoreException <#toil.jobStores.abstractJobStore.NoSuchJobStoreException> -- if the bucket doesn't exist.

            Return type
                None

        destroy()
            The inverse of initialize(), this method deletes the physical storage represented by this instance. While not being atomic, this method is at least idempotent, as a means to counteract potential issues with eventual consistency exhibited by the underlying storage mechanisms. This means that if the method fails (raises an exception), it may (and should be) invoked again. If the underlying storage mechanism is eventually consistent, even a successful invocation is not an ironclad guarantee that the physical storage vanished completely and immediately. A successful invocation only guarantees that the deletion will eventually happen. It is therefore recommended to not immediately reuse the same job store location for a new Toil workflow.

            Return type
                None

        is_in_bucket(identifier, prefix, bucket=None)
            Check if the key for the given identifier and prefix is in the bucket.

            Parameters

                •   identifier (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   prefix (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   bucket (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        write_to_bucket(identifier, prefix, data, bucket=None, encrypted=None)
            Write something directly to a bucket.

            Use for small files. Does not parallelize or use multipart.

            Parameters

                •   encrypted (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>]) -- Can be set to False to disable encryption.

                •   identifier (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   prefix (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   data (Optional[Union[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>, str </usr/share/doc/python3/html/library/stdtypes.html#str>, Dict[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]]])

                •   bucket (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

        read_from_bucket(identifier, prefix, bucket=None)
            Read something directly from a bucket.

            Use for small files. Does not parallelize or use multipart.

            Raises

                •   NoSuchJobException <#toil.jobStores.abstractJobStore.NoSuchJobException> -- if the prefix is the job prefix and the identifier is not found.

                •   NoSuchFileException <#toil.jobStores.abstractJobStore.NoSuchFileException> -- if the prefix is the content prefix and the identifier is not found.

                •   self.s3_client.exceptions.NoSuchKey -- in other cases where the identifier is not found.

            Parameters

                •   identifier (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   prefix (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   bucket (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>

        assign_job_id(jobDescription)
            Get a new jobStoreID to be used by the described job, and assigns it to the JobDescription.

            Files associated with the assigned ID will be accepted even if the JobDescription has never been created or updated.

            Parameters

                •   job_description (toil.job.JobDescription <#toil.job.JobDescription>) -- The JobDescription to give an ID to

                •   jobDescription (toil.job.JobDescription <#toil.job.JobDescription>)

            Return type
                None

        create_job(jobDescription)
            Pickle a jobDescription object and write it to the jobstore as a file.

            Responsible for calling toil.job.JobDescription.pre_update_hook() <#toil.job.JobDescription.pre_update_hook> on the job description.

            Parameters
                jobDescription (toil.job.JobDescription <#toil.job.JobDescription>)

            Return type
                toil.job.JobDescription <#toil.job.JobDescription>

        job_exists(job_id, check=False)
            Checks if the job_id is found in s3.

            Parameters

                •   check (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If True, raise an exception instead of returning false when a job does not exist.

                •   job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        jobs()
            Best effort attempt to return iterator on JobDescriptions for all jobs in the store. The iterator may not return all jobs and may also contain orphaned jobs that have already finished successfully and should not be rerun. To guarantee you get any and all jobs that can be run instead construct a more expensive ToilState object

            Returns
                Returns iterator on jobs in the store. The iterator may or may not contain all jobs and may contain invalid jobs

            Return type
                Iterator[toil.job.jobDescription]

        load_job(job_id)
            Use a job_id to get a job from the jobstore's s3 bucket, unpickle, and return it.

            Parameters
                job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                toil.job.JobDescription <#toil.job.JobDescription>

        update_job(jobDescription)
            Persists changes to the state of the given JobDescription in this store atomically.

            Must call jobDescription.pre_update_hook()

            Parameters

                •   job (toil.job.JobDescription <#toil.job.JobDescription>) -- the job to write to this job store

                •   jobDescription (toil.job.JobDescription <#toil.job.JobDescription>)

            Return type
                None

        delete_job(job_id)
            Removes the JobDescription from the store atomically. You may not then subsequently call load(), write(), update(), etc. with the same jobStoreID or any JobDescription bearing it.

            This operation is idempotent, i.e. deleting a job twice or deleting a non-existent job will succeed silently.

            Parameters
                job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the ID of the job to delete from this job store

            Return type
                None

        associate_job_with_file(job_id, file_id)

            Parameters

                •   job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        write_file(local_path, job_id=None, cleanup=False)
            Write a local file into the jobstore and return a file_id referencing it.

            Parameters

                •   job_id (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- If job_id AND cleanup are supplied, associate this file with that job. When the job is deleted, the file will be deleted as well.

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If job_id AND cleanup are supplied, associate this file with that job. When the job is deleted, the file will be deleted as well. TODO: we don't need cleanup; remove it and only use job_id

                •   local_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                toil.fileStores.FileID <#toil.fileStores.FileID>

        find_s3_key_from_file_id(file_id)
            This finds an s3 key for which file_id is the prefix, and which already exists.

            Parameters
                file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        write_file_stream(job_id=None, cleanup=False, basename=None, encoding=None, errors=None)
            Similar to writeFile, but returns a context manager yielding a tuple of 1) a file handle which can be written to and 2) the ID of the resulting file in the job store. The yielded file handle does not need to and should not be closed explicitly. The file is written in a atomic manner. It will not appear in the jobStore until the write has successfully completed.

            Parameters

                •   job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the id of a job, or None. If specified, the may be associated with that job in a job-store-specific way. This may influence the returned ID.

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing.

                •   basename (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- If supported by the implementation, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected.

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

            Raises

                •   ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

                •   NoSuchJobException <#toil.jobStores.abstractJobStore.NoSuchJobException> -- if the job specified via jobStoreID does not exist

            Return type
                Iterator[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>], str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

        FIXME: some implementations may not raise this

            Returns
                a context manager yielding a file handle which can be written to and an ID that references the newly created file and can be used to read the file in the future.

            Return type
                Iterator[Tuple[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>], str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

            Parameters

                •   job_id (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>)

                •   basename (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   encoding (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   errors (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        update_file_stream(file_id, encoding=None, errors=None)
            Replaces the existing version of a file in the job store. Similar to writeFile, but returns a context manager yielding a file handle which can be written to. The yielded file handle does not need to and should not be closed explicitly.

            Parameters

                •   file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the ID of the file in the job store to be updated

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

            Raises

                •   ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

                •   NoSuchFileException <#toil.jobStores.abstractJobStore.NoSuchFileException> -- if the specified file does not exist

            Return type
                Iterator[IO[Any]]

        write_shared_file_stream(shared_file_name, encrypted=None, encoding=None, errors=None)
            Returns a context manager yielding a writable file handle to the global file referenced by the given name. File will be created in an atomic manner.

            Parameters

                •   shared_file_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- A file name matching AbstractJobStore.fileNameRegex, unique within this job store

                •   encrypted (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- True if the file must be encrypted, None if it may be encrypted or False if it must be stored in the clear.

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

            Raises
                ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

            Returns
                a context manager yielding a writable file handle

            Return type
                Iterator[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]]

        update_file(file_id, local_path)
            Replaces the existing version of a file in the job store.

            Throws an exception if the file does not exist.

            Parameters

                •   file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the ID of the file in the job store to be updated

                •   local_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the local path to a file that will overwrite the current version in the job store

            Raises

                •   ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

                •   NoSuchFileException <#toil.jobStores.abstractJobStore.NoSuchFileException> -- if the specified file does not exist

            Return type
                None

        file_exists(file_id)
            Determine whether a file exists in this job store.

            Parameters
                file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an ID referencing the file to be checked

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        get_file_size(file_id)
            Do we need both get_file_size and _get_size???

            Parameters
                file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        read_file(file_id, local_path, symlink=False)
            Copies or hard links the file referenced by jobStoreFileID to the given local file path. The version will be consistent with the last copy of the file written/updated. If the file in the job store is later modified via updateFile or updateFileStream, it is implementation-defined whether those writes will be visible at localFilePath. The file is copied in an atomic manner. It will not appear in the local file system until the copy has completed.

            The file at the given local path may not be modified after this method returns!

            Note! Implementations of readFile need to respect/provide the executable attribute on FileIDs.

            Parameters

                •   file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- ID of the file to be copied

                •   local_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the local path indicating where to place the contents of the given file in the job store

                •   symlink (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- whether the reader can tolerate a symlink. If set to true, the job store may create a symlink instead of a full copy of the file or a hard link.

            Return type
                None

        read_file_stream(file_id, encoding=None, errors=None)
            Similar to readFile, but returns a context manager yielding a file handle which can be read from. The yielded file handle does not need to and should not be closed explicitly.

            Parameters

                •   file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- ID of the file to get a readable file handle for

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

            Returns
                a context manager yielding a file handle which can be read from

            Return type
                Iterator[Union[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>], IO[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]]

        read_shared_file_stream(shared_file_name: str </usr/share/doc/python3/html/library/stdtypes.html#str>, encoding: str </usr/share/doc/python3/html/library/stdtypes.html#str>, errors: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> Iterator[IO[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

        read_shared_file_stream(shared_file_name: str </usr/share/doc/python3/html/library/stdtypes.html#str>, encoding: Literal[None] = None, errors: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> Iterator[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]]
            Returns a context manager yielding a readable file handle to the global file referenced by the given name.

            Parameters

                •   shared_file_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- A file name matching AbstractJobStore.fileNameRegex, unique within this job store

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

            Returns
                a context manager yielding a readable file handle

        delete_file(file_id)
            Deletes the file with the given ID from this job store. This operation is idempotent, i.e. deleting a file twice or deleting a non-existent file will succeed silently.

            Parameters
                file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- ID of the file to delete

            Return type
                None

        get_public_url(file_id)
            Turn s3:// into http:// and put a public-read ACL on it.

            Parameters
                file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        get_shared_public_url(file_id)
            Turn s3:// into http:// and put a public-read ACL on it.

            Parameters
                file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        get_empty_file_store_id(job_id=None, cleanup=False, basename=None)
            Create an empty file in s3 and return a bare string file ID.

            Parameters

                •   job_id (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>)

                •   basename (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        write_logs(log_msg)
            Stores a message as a log in the jobstore.

            Parameters

                •   msg (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the string to be written

                •   log_msg (Union[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Raises
                ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

            Return type
                None

        read_logs(callback, read_all=False)
            This fetches all referenced logs in the database from s3 as readable objects and runs "callback()" on them.

            Parameters

                •   callback (Callable[Ellipsis, Any])

                •   read_all (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

    toil.jobStores.aws.jobStore.parse_jobstore_identifier(jobstore_identifier)

        Parameters
            jobstore_identifier (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            Tuple[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

  toil.jobStores.aws.utils

  Attributes
logger |
DIAL_SPECIFIC_REGION_CONFIG |

  Exceptions
ServerSideCopyProhibitedError | Raised when AWS refuses to perform a server-side copy between S3 keys, and

  Functions
fileSizeAndTime(localFilePath) |
uploadFromPath(localFilePath, resource, bucketName, fileID) | Uploads a file to s3, using multipart uploading if applicable
uploadFile(readable, resource, bucketName, fileID[, ...]) | Upload a readable object to s3, using multipart uploading if applicable.
copyKeyMultipart(resource, srcBucketName, srcKeyName, ...) | Copies a key from a source key to a destination key in multiple parts. Note that if the
retryable_ssl_error(e) |

  Module Contents

    toil.jobStores.aws.utils.logger

    toil.jobStores.aws.utils.DIAL_SPECIFIC_REGION_CONFIG

    toil.jobStores.aws.utils.fileSizeAndTime(localFilePath)

        Parameters
            localFilePath (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>]

    toil.jobStores.aws.utils.uploadFromPath(localFilePath, resource, bucketName, fileID, headerArgs=None, partSize=50 << 20)
        Uploads a file to s3, using multipart uploading if applicable

        Parameters

            •   localFilePath (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Path of the file to upload to s3

            •   resource (S3.Resource) -- boto3 resource

            •   bucketName (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- name of the bucket to upload to

            •   fileID (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the file to upload to

            •   headerArgs (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>) -- http headers to use when uploading - generally used for encryption purposes

            •   partSize (int </usr/share/doc/python3/html/library/functions.html#int>) -- max size of each part in the multipart upload, in bytes

        Returns
            version of the newly uploaded file

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.jobStores.aws.utils.uploadFile(readable, resource, bucketName, fileID, headerArgs=None, partSize=50 << 20)
        Upload a readable object to s3, using multipart uploading if applicable. :param readable: a readable stream or a file path to upload to s3 :param S3.Resource resource: boto3 resource :param str bucketName: name of the bucket to upload to :param str fileID: the name of the file to upload to :param dict headerArgs: http headers to use when uploading - generally used for encryption purposes :param int partSize: max size of each part in the multipart upload, in bytes :return: version of the newly uploaded file

        Parameters

            •   readable (IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>])

            •   resource (mypy_boto3_s3.S3ServiceResource)

            •   bucketName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   fileID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   headerArgs (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]])

            •   partSize (int </usr/share/doc/python3/html/library/functions.html#int>)

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    exception toil.jobStores.aws.utils.ServerSideCopyProhibitedError
        Bases: RuntimeError </usr/share/doc/python3/html/library/exceptions.html#RuntimeError>

        Raised when AWS refuses to perform a server-side copy between S3 keys, and insists that you pay to download and upload the data yourself instead.

    toil.jobStores.aws.utils.copyKeyMultipart(resource, srcBucketName, srcKeyName, srcKeyVersion, dstBucketName, dstKeyName, sseAlgorithm=None, sseKey=None, copySourceSseAlgorithm=None, copySourceSseKey=None)
        Copies a key from a source key to a destination key in multiple parts. Note that if the destination key exists it will be overwritten implicitly, and if it does not exist a new key will be created. If the destination bucket does not exist an error will be raised.

        This function will always do a fast, server-side copy, at least until/unless <<https://github.com/boto/boto3/issues/3270>> is fixed. In some situations, a fast, server-side copy is not actually possible. For example, when residing in an AWS VPC with an S3 VPC Endpoint configured, copying from a bucket in another region to a bucket in your own region cannot be performed server-side. This is because the VPC Endpoint S3 API servers refuse to perform server-side copies between regions, the source region's API servers refuse to initiate the copy and refer you to the destination bucket's region's API servers, and the VPC routing tables are configured to redirect all access to the current region's S3 API servers to the S3 Endpoint API servers instead.

        If a fast server-side copy is not actually possible, a ServerSideCopyProhibitedError will be raised.

        Parameters

            •   resource (mypy_boto3_s3.S3ServiceResource) -- boto3 resource

            •   srcBucketName (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The name of the bucket to be copied from.

            •   srcKeyName (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The name of the key to be copied from.

            •   srcKeyVersion (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The version of the key to be copied from.

            •   dstBucketName (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The name of the destination bucket for the copy.

            •   dstKeyName (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The name of the destination key that will be created or overwritten.

            •   sseAlgorithm (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Server-side encryption algorithm for the destination.

            •   sseKey (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Server-side encryption key for the destination.

            •   copySourceSseAlgorithm (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Server-side encryption algorithm for the source.

            •   copySourceSseKey (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Server-side encryption key for the source.

        Returns
            The version of the copied file (or None if versioning is not enabled for dstBucket).

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.jobStores.aws.utils.retryable_ssl_error(e)

        Parameters
            e (BaseException </usr/share/doc/python3/html/library/exceptions.html#BaseException>)

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

  toil.jobStores.conftest

  Attributes
collect_ignore |

  Module Contents

    toil.jobStores.conftest.collect_ignore = []

  toil.jobStores.fileJobStore

  Attributes
logger |

  Classes
FileJobStore | A job store that uses a directory on a locally attached file system. To be compatible with

  Module Contents

    toil.jobStores.fileJobStore.logger

    class toil.jobStores.fileJobStore.FileJobStore(path, fanOut=1000)
        Bases: toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>, toil.lib.url.URLAccess <#toil.lib.url.URLAccess>

        A job store that uses a directory on a locally attached file system. To be compatible with distributed batch systems, that file system must be shared by all worker nodes.

        Parameters

            •   path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   fanOut (int </usr/share/doc/python3/html/library/functions.html#int>)

        validDirs = 'abcdefghijklmnopqrstuvwxyz0123456789'

        validDirsSet

        JOB_DIR_PREFIX = 'instance-'

        JOB_NAME_DIR_PREFIX = 'kind-'

        BUFFER_SIZE = 10485760

        LOG_TEMP_SUFFIX = '.new'

        LOG_PREFIX = 'stats'

        default_caching()
            Jobstore's preference as to whether it likes caching or doesn't care about it. Some jobstores benefit from caching, however on some local configurations it can be flaky.

            see <https://github.com/DataBiosphere/toil/issues/4218>

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        jobStoreDir = b'.'

        jobsDir

        statsDir

        stats_inbox

        stats_archive

        filesDir

        jobFilesDir

        sharedFilesDir

        fanOut = 1000

        linkImports = None

        moveExports = None

        symlink_job_store_reads = None

        __repr__()

        initialize(config)
            Initialize this job store.

            Create the physical storage for this job store, allocate a workflow ID and persist the given Toil configuration to the store.

            Parameters
                config -- the Toil configuration to initialize this job store with. The given configuration will be updated with the newly allocated workflow ID.

            Raises
                JobStoreExistsException <#toil.jobStores.abstractJobStore.JobStoreExistsException> -- if the physical storage for this job store already exists

        resume()
            Connect this instance to the physical storage it represents and load the Toil configuration into the AbstractJobStore.config attribute.

            Raises
                NoSuchJobStoreException <#toil.jobStores.abstractJobStore.NoSuchJobStoreException> -- if the physical storage for this job store doesn't exist

        destroy()
            The inverse of initialize(), this method deletes the physical storage represented by this instance. While not being atomic, this method is at least idempotent, as a means to counteract potential issues with eventual consistency exhibited by the underlying storage mechanisms. This means that if the method fails (raises an exception), it may (and should be) invoked again. If the underlying storage mechanism is eventually consistent, even a successful invocation is not an ironclad guarantee that the physical storage vanished completely and immediately. A successful invocation only guarantees that the deletion will eventually happen. It is therefore recommended to not immediately reuse the same job store location for a new Toil workflow.

        assign_job_id(job_description)
            Get a new jobStoreID to be used by the described job, and assigns it to the JobDescription.

            Files associated with the assigned ID will be accepted even if the JobDescription has never been created or updated.

            Parameters
                job_description (toil.job.JobDescription <#toil.job.JobDescription>) -- The JobDescription to give an ID to

        create_job(job_description)
            Writes the given JobDescription to the job store. The job must have an ID assigned already.

            Must call jobDescription.pre_update_hook()

            Returns
                The JobDescription passed.

            Return type
                toil.job.JobDescription <#toil.job.JobDescription>

        batch()
            If supported by the batch system, calls to create() with this context manager active will be performed in a batch after the context manager is released.

        job_exists(job_id)
            Indicates whether a description of the job with the specified jobStoreID exists in the job store

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        get_public_url(jobStoreFileID)
            Returns a publicly accessible URL to the given file in the job store. The returned URL may expire as early as 1h after its been returned. Throw an exception if the file does not exist.

            Parameters
                file_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the jobStoreFileID of the file to generate a URL for

            Raises
                NoSuchFileException <#toil.jobStores.abstractJobStore.NoSuchFileException> -- if the specified file does not exist in this job store

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        get_shared_public_url(sharedFileName)
            Differs from getPublicUrl() in that this method is for generating URLs for shared files written by writeSharedFileStream().

            Returns a publicly accessible URL to the given file in the job store. The returned URL starts with 'http:', 'https:' or 'file:'. The returned URL may expire as early as 1h after its been returned. Throw an exception if the file does not exist.

            Parameters
                shared_file_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The name of the shared file to generate a publically accessible url for.

            Raises
                NoSuchFileException <#toil.jobStores.abstractJobStore.NoSuchFileException> -- raised if the specified file does not exist in the store

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        load_job(job_id)
            Loads the description of the job referenced by the given ID, assigns it the job store's config, and returns it.

            May declare the job to have failed (see toil.job.JobDescription.setupJobAfterFailure() <#toil.job.JobDescription.setupJobAfterFailure>) if there is evidence of a failed update attempt.

            Parameters
                job_id -- the ID of the job to load

            Raises
                NoSuchJobException <#toil.jobStores.abstractJobStore.NoSuchJobException> -- if there is no job with the given ID

        update_job(job)
            Persists changes to the state of the given JobDescription in this store atomically.

            Must call jobDescription.pre_update_hook()

            Parameters
                job (toil.job.JobDescription <#toil.job.JobDescription>) -- the job to write to this job store

        delete_job(job_id)
            Removes the JobDescription from the store atomically. You may not then subsequently call load(), write(), update(), etc. with the same jobStoreID or any JobDescription bearing it.

            This operation is idempotent, i.e. deleting a job twice or deleting a non-existent job will succeed silently.

            Parameters
                job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the ID of the job to delete from this job store

        jobs()
            Best effort attempt to return iterator on JobDescriptions for all jobs in the store. The iterator may not return all jobs and may also contain orphaned jobs that have already finished successfully and should not be rerun. To guarantee you get any and all jobs that can be run instead construct a more expensive ToilState object

            Returns
                Returns iterator on jobs in the store. The iterator may or may not contain all jobs and may contain invalid jobs

            Return type
                Iterator[toil.job.jobDescription]

        write_file(local_path, job_id=None, cleanup=False)
            Takes a file (as a path) and places it in this job store. Returns an ID that can be used to retrieve the file at a later time. The file is written in a atomic manner. It will not appear in the jobStore until the write has successfully completed.

            Parameters

                •   local_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the path to the local file that will be uploaded to the job store. The last path component (basename of the file) will remain associated with the file in the file store, if supported, so that the file can be searched for by name or name glob.

                •   job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the id of a job, or None. If specified, the may be associated with that job in a job-store-specific way. This may influence the returned ID.

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing.

            Raises

                •   ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

                •   NoSuchJobException <#toil.jobStores.abstractJobStore.NoSuchJobException> -- if the job specified via jobStoreID does not exist

        FIXME: some implementations may not raise this

            Returns
                an ID referencing the newly created file and can be used to read the file in the future.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        write_file_stream(job_id=None, cleanup=False, basename=None, encoding=None, errors=None)
            Similar to writeFile, but returns a context manager yielding a tuple of 1) a file handle which can be written to and 2) the ID of the resulting file in the job store. The yielded file handle does not need to and should not be closed explicitly. The file is written in a atomic manner. It will not appear in the jobStore until the write has successfully completed.

            Parameters

                •   job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the id of a job, or None. If specified, the may be associated with that job in a job-store-specific way. This may influence the returned ID.

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing.

                •   basename (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- If supported by the implementation, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected.

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

            Raises

                •   ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

                •   NoSuchJobException <#toil.jobStores.abstractJobStore.NoSuchJobException> -- if the job specified via jobStoreID does not exist

        FIXME: some implementations may not raise this

            Returns
                a context manager yielding a file handle which can be written to and an ID that references the newly created file and can be used to read the file in the future.

            Return type
                Iterator[Tuple[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>], str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

        get_empty_file_store_id(jobStoreID=None, cleanup=False, basename=None)
            Creates an empty file in the job store and returns its ID. Call to fileExists(getEmptyFileStoreID(jobStoreID)) will return True.

            Parameters

                •   job_id -- the id of a job, or None. If specified, the may be associated with that job in a job-store-specific way. This may influence the returned ID.

                •   cleanup -- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing.

                •   basename -- If supported by the implementation, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected.

            Returns
                a jobStoreFileID that references the newly created file and can be used to reference the file in the future.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        update_file(file_id, local_path)
            Replaces the existing version of a file in the job store.

            Throws an exception if the file does not exist.

            Parameters

                •   file_id -- the ID of the file in the job store to be updated

                •   local_path -- the local path to a file that will overwrite the current version in the job store

            Raises

                •   ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

                •   NoSuchFileException <#toil.jobStores.abstractJobStore.NoSuchFileException> -- if the specified file does not exist

        read_file(file_id, local_path, symlink=False)
            Copies or hard links the file referenced by jobStoreFileID to the given local file path. The version will be consistent with the last copy of the file written/updated. If the file in the job store is later modified via updateFile or updateFileStream, it is implementation-defined whether those writes will be visible at localFilePath. The file is copied in an atomic manner. It will not appear in the local file system until the copy has completed.

            The file at the given local path may not be modified after this method returns!

            Note! Implementations of readFile need to respect/provide the executable attribute on FileIDs.

            Parameters

                •   file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- ID of the file to be copied

                •   local_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the local path indicating where to place the contents of the given file in the job store

                •   symlink (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- whether the reader can tolerate a symlink. If set to true, the job store may create a symlink instead of a full copy of the file or a hard link.

            Return type
                None

        delete_file(file_id)
            Deletes the file with the given ID from this job store. This operation is idempotent, i.e. deleting a file twice or deleting a non-existent file will succeed silently.

            Parameters
                file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- ID of the file to delete

        file_exists(file_id)
            Determine whether a file exists in this job store.

            Parameters
                file_id -- an ID referencing the file to be checked

        get_file_size(file_id)
            Get the size of the given file in bytes, or 0 if it does not exist when queried.

            Note that job stores which encrypt files might return overestimates of file sizes, since the encrypted file may have been padded to the nearest block, augmented with an initialization vector, etc.

            Parameters
                file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an ID referencing the file to be checked

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        update_file_stream(file_id, encoding=None, errors=None)
            Replaces the existing version of a file in the job store. Similar to writeFile, but returns a context manager yielding a file handle which can be written to. The yielded file handle does not need to and should not be closed explicitly.

            Parameters

                •   file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the ID of the file in the job store to be updated

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

            Raises

                •   ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

                •   NoSuchFileException <#toil.jobStores.abstractJobStore.NoSuchFileException> -- if the specified file does not exist

        read_file_stream(file_id: str </usr/share/doc/python3/html/library/stdtypes.html#str> | toil.fileStores.FileID <#toil.fileStores.FileID>, encoding: Literal[None] = None, errors: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> collections.abc.Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]]

        read_file_stream(file_id: str </usr/share/doc/python3/html/library/stdtypes.html#str> | toil.fileStores.FileID <#toil.fileStores.FileID>, encoding: str </usr/share/doc/python3/html/library/stdtypes.html#str>, errors: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> collections.abc.Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[IO[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

        read_file_stream(file_id: str </usr/share/doc/python3/html/library/stdtypes.html#str> | toil.fileStores.FileID <#toil.fileStores.FileID>, encoding: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, errors: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> collections.abc.Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]] | collections.abc.Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[IO[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]
            Similar to readFile, but returns a context manager yielding a file handle which can be read from. The yielded file handle does not need to and should not be closed explicitly.

            Parameters

                •   file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- ID of the file to get a readable file handle for

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

            Returns
                a context manager yielding a file handle which can be read from

            Return type
                Iterator[Union[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>], IO[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]]

        write_shared_file_stream(shared_file_name, encrypted=None, encoding=None, errors=None)
            Returns a context manager yielding a writable file handle to the global file referenced by the given name. File will be created in an atomic manner.

            Parameters

                •   shared_file_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- A file name matching AbstractJobStore.fileNameRegex, unique within this job store

                •   encrypted (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- True if the file must be encrypted, None if it may be encrypted or False if it must be stored in the clear.

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

            Raises
                ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

            Returns
                a context manager yielding a writable file handle

            Return type
                Iterator[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]]

        read_shared_file_stream(shared_file_name: str </usr/share/doc/python3/html/library/stdtypes.html#str>, encoding: str </usr/share/doc/python3/html/library/stdtypes.html#str>, errors: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> collections.abc.Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[IO[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

        read_shared_file_stream(shared_file_name: str </usr/share/doc/python3/html/library/stdtypes.html#str>, encoding: Literal[None] = None, errors: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> collections.abc.Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]]
            Returns a context manager yielding a readable file handle to the global file referenced by the given name.

            Parameters

                •   shared_file_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- A file name matching AbstractJobStore.fileNameRegex, unique within this job store

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

            Returns
                a context manager yielding a readable file handle

        list_all_file_names(for_job=None)
            Get all the file names (not file IDs) of files stored in the job store.

            Used for debugging.

            Parameters
                for_job (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- If set, restrict the list to files for a particular job.

            Return type
                collections.abc.Iterable </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterable>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        write_logs(msg)
            Stores a message as a log in the jobstore.

            Parameters
                msg (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the string to be written

            Raises
                ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

        read_logs(callback, read_all=False)
            Reads logs accumulated by the write_logs() method. For each log this method calls the given callback function with the message as an argument (rather than returning logs directly, this method must be supplied with a callback which will process log messages).

            Only unread logs will be read unless the read_all parameter is set.

            Parameters

                •   callback (Callable) -- a function to be applied to each of the stats file handles found

                •   read_all (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- a boolean indicating whether to read the already processed stats files in addition to the unread stats files

            Raises
                ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

            Returns
                the number of stats files processed

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

  toil.jobStores.googleJobStore

  Attributes
log |
GOOGLE_STORAGE |
MAX_BATCH_SIZE |

  Classes
GoogleJobStore | Represents the physical storage for the jobs and files in a Toil workflow.

  Functions
google_retry_predicate(e) | necessary because under heavy load google may throw
google_retry(f) | This decorator retries the wrapped function if google throws any angry service
permission_error_reporter(url, notes) | Detect and usefully report permission errors.

  Module Contents

    toil.jobStores.googleJobStore.log

    toil.jobStores.googleJobStore.GOOGLE_STORAGE = 'gs'

    toil.jobStores.googleJobStore.MAX_BATCH_SIZE = 1000

    toil.jobStores.googleJobStore.google_retry_predicate(e)

        necessary because under heavy load google may throw
            TooManyRequests: 429 The project exceeded the rate limit for creating and deleting buckets.

    or numerous other server errors which need to be retried.

    toil.jobStores.googleJobStore.google_retry(f)
        This decorator retries the wrapped function if google throws any angry service errors.

        It should wrap any function that makes use of the Google Client API

    toil.jobStores.googleJobStore.permission_error_reporter(url, notes)
        Detect and usefully report permission errors.

        If we fall back to anonymous credentials, but they don't have permission for something, the Google Cloud Storage module will try to refresh them behind the scenes. Then it will complain:

                <class 'google.auth.exceptions.InvalidOperation'>: Anonymous credentials cannot be refreshed.

    We need to detect this and report that the real problem is that the user has not set up any credentials. When you try to make the client non-anonymously and don't have credentials set up, you get a nice error from Google:

                google.auth.exceptions.DefaultCredentialsError: Your default credentials were not found. To set up Application Default Credentials, see https://cloud.google.com/docs/authentication/external/set-up-adc for more information.

    But we swallow that when we fall back to anonymous access.

    So we take the URL and any notes from client setup here, and if something goes wrong that looks like a permission problem we complain with the notes attached.

    Also, if you don't have a project available, you can't use the Python API for storage with authentication at all. gsutil can do it but you can't. TODO: Fall back on command-line gsutil for authenticated reads???

        Parameters

            •   url (urllib.parse.ParseResult </usr/share/doc/python3/html/library/urllib.parse.html#urllib.parse.ParseResult>)

            •   notes (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            Iterator[None]

    class toil.jobStores.googleJobStore.GoogleJobStore(locator)
        Bases: toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>, toil.lib.url.URLAccess <#toil.lib.url.URLAccess>

        Represents the physical storage for the jobs and files in a Toil workflow.

        JobStores are responsible for storing toil.job.JobDescription <#toil.job.JobDescription> (which relate jobs to each other) and files.

        Actual toil.job.Job <#toil.job.Job> objects are stored in files, referenced by JobDescriptions. All the non-file CRUD methods the JobStore provides deal in JobDescriptions and not full, executable Jobs.

        To actually get ahold of a toil.job.Job <#toil.job.Job>, use toil.job.Job.loadJob() <#toil.job.Job.loadJob> with a JobStore and the relevant JobDescription.

        Parameters
            locator (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        nodeServiceAccountJson = '/root/service_account.json'

        projectID

        bucketName

        bucket = None

        statsBaseID = 'f16eef0c-b597-4b8b-9b0c-4d605b4f506c'

        statsReadPrefix = '_'

        readStatsBaseID = '_f16eef0c-b597-4b8b-9b0c-4d605b4f506c'

        sseKey = None

        classmethod create_client()
            Produce a client for Google Storage with the highest level of access we can get.

            Fall back to anonymous access if no project is available, unlike the Google Storage module's behavior.

            Warn if GOOGLE_APPLICATION_CREDENTIALS is set but not actually present.

            Returns
                the client, and any notes about why it might not have permissions.

            Return type
                tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[google.cloud.storage.Client, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        initialize(config=None)
            Initialize this job store.

            Create the physical storage for this job store, allocate a workflow ID and persist the given Toil configuration to the store.

            Parameters
                config -- the Toil configuration to initialize this job store with. The given configuration will be updated with the newly allocated workflow ID.

            Raises
                JobStoreExistsException <#toil.jobStores.abstractJobStore.JobStoreExistsException> -- if the physical storage for this job store already exists

        resume()
            Connect this instance to the physical storage it represents and load the Toil configuration into the AbstractJobStore.config attribute.

            Raises
                NoSuchJobStoreException <#toil.jobStores.abstractJobStore.NoSuchJobStoreException> -- if the physical storage for this job store doesn't exist

        destroy()
            The inverse of initialize(), this method deletes the physical storage represented by this instance. While not being atomic, this method is at least idempotent, as a means to counteract potential issues with eventual consistency exhibited by the underlying storage mechanisms. This means that if the method fails (raises an exception), it may (and should be) invoked again. If the underlying storage mechanism is eventually consistent, even a successful invocation is not an ironclad guarantee that the physical storage vanished completely and immediately. A successful invocation only guarantees that the deletion will eventually happen. It is therefore recommended to not immediately reuse the same job store location for a new Toil workflow.

        assign_job_id(job_description)
            Get a new jobStoreID to be used by the described job, and assigns it to the JobDescription.

            Files associated with the assigned ID will be accepted even if the JobDescription has never been created or updated.

            Parameters
                job_description (toil.job.JobDescription <#toil.job.JobDescription>) -- The JobDescription to give an ID to

        batch()
            If supported by the batch system, calls to create() with this context manager active will be performed in a batch after the context manager is released.

        create_job(job_description)
            Writes the given JobDescription to the job store. The job must have an ID assigned already.

            Must call jobDescription.pre_update_hook()

            Returns
                The JobDescription passed.

            Return type
                toil.job.JobDescription <#toil.job.JobDescription>

        job_exists(job_id)
            Indicates whether a description of the job with the specified jobStoreID exists in the job store

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        get_public_url(fileName)
            Returns a publicly accessible URL to the given file in the job store. The returned URL may expire as early as 1h after its been returned. Throw an exception if the file does not exist.

            Parameters
                file_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the jobStoreFileID of the file to generate a URL for

            Raises
                NoSuchFileException <#toil.jobStores.abstractJobStore.NoSuchFileException> -- if the specified file does not exist in this job store

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        get_shared_public_url(sharedFileName)
            Differs from getPublicUrl() in that this method is for generating URLs for shared files written by writeSharedFileStream().

            Returns a publicly accessible URL to the given file in the job store. The returned URL starts with 'http:', 'https:' or 'file:'. The returned URL may expire as early as 1h after its been returned. Throw an exception if the file does not exist.

            Parameters
                shared_file_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The name of the shared file to generate a publically accessible url for.

            Raises
                NoSuchFileException <#toil.jobStores.abstractJobStore.NoSuchFileException> -- raised if the specified file does not exist in the store

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        load_job(job_id)
            Loads the description of the job referenced by the given ID, assigns it the job store's config, and returns it.

            May declare the job to have failed (see toil.job.JobDescription.setupJobAfterFailure() <#toil.job.JobDescription.setupJobAfterFailure>) if there is evidence of a failed update attempt.

            Parameters
                job_id -- the ID of the job to load

            Raises
                NoSuchJobException <#toil.jobStores.abstractJobStore.NoSuchJobException> -- if there is no job with the given ID

        update_job(job)
            Persists changes to the state of the given JobDescription in this store atomically.

            Must call jobDescription.pre_update_hook()

            Parameters
                job (toil.job.JobDescription <#toil.job.JobDescription>) -- the job to write to this job store

        delete_job(job_id)
            Removes the JobDescription from the store atomically. You may not then subsequently call load(), write(), update(), etc. with the same jobStoreID or any JobDescription bearing it.

            This operation is idempotent, i.e. deleting a job twice or deleting a non-existent job will succeed silently.

            Parameters
                job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the ID of the job to delete from this job store

        get_env()
            Return a dict of environment variables to send out to the workers so they can load the job store.

        jobs()
            Best effort attempt to return iterator on JobDescriptions for all jobs in the store. The iterator may not return all jobs and may also contain orphaned jobs that have already finished successfully and should not be rerun. To guarantee you get any and all jobs that can be run instead construct a more expensive ToilState object

            Returns
                Returns iterator on jobs in the store. The iterator may or may not contain all jobs and may contain invalid jobs

            Return type
                Iterator[toil.job.jobDescription]

        write_file(local_path, job_id=None, cleanup=False)
            Takes a file (as a path) and places it in this job store. Returns an ID that can be used to retrieve the file at a later time. The file is written in a atomic manner. It will not appear in the jobStore until the write has successfully completed.

            Parameters

                •   local_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the path to the local file that will be uploaded to the job store. The last path component (basename of the file) will remain associated with the file in the file store, if supported, so that the file can be searched for by name or name glob.

                •   job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the id of a job, or None. If specified, the may be associated with that job in a job-store-specific way. This may influence the returned ID.

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing.

            Raises

                •   ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

                •   NoSuchJobException <#toil.jobStores.abstractJobStore.NoSuchJobException> -- if the job specified via jobStoreID does not exist

        FIXME: some implementations may not raise this

            Returns
                an ID referencing the newly created file and can be used to read the file in the future.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        write_file_stream(job_id=None, cleanup=False, basename=None, encoding=None, errors=None)
            Similar to writeFile, but returns a context manager yielding a tuple of 1) a file handle which can be written to and 2) the ID of the resulting file in the job store. The yielded file handle does not need to and should not be closed explicitly. The file is written in a atomic manner. It will not appear in the jobStore until the write has successfully completed.

            Parameters

                •   job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the id of a job, or None. If specified, the may be associated with that job in a job-store-specific way. This may influence the returned ID.

                •   cleanup (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing.

                •   basename (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- If supported by the implementation, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected.

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

            Raises

                •   ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

                •   NoSuchJobException <#toil.jobStores.abstractJobStore.NoSuchJobException> -- if the job specified via jobStoreID does not exist

        FIXME: some implementations may not raise this

            Returns
                a context manager yielding a file handle which can be written to and an ID that references the newly created file and can be used to read the file in the future.

            Return type
                Iterator[Tuple[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>], str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

        get_empty_file_store_id(job_id=None, cleanup=False, basename=None)
            Creates an empty file in the job store and returns its ID. Call to fileExists(getEmptyFileStoreID(jobStoreID)) will return True.

            Parameters

                •   job_id -- the id of a job, or None. If specified, the may be associated with that job in a job-store-specific way. This may influence the returned ID.

                •   cleanup -- Whether to attempt to delete the file when the job whose jobStoreID was given as jobStoreID is deleted with jobStore.delete(job). If jobStoreID was not given, does nothing.

                •   basename -- If supported by the implementation, use the given file basename so that when searching the job store with a query matching that basename, the file will be detected.

            Returns
                a jobStoreFileID that references the newly created file and can be used to reference the file in the future.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        read_file(file_id, local_path, symlink=False)
            Copies or hard links the file referenced by jobStoreFileID to the given local file path. The version will be consistent with the last copy of the file written/updated. If the file in the job store is later modified via updateFile or updateFileStream, it is implementation-defined whether those writes will be visible at localFilePath. The file is copied in an atomic manner. It will not appear in the local file system until the copy has completed.

            The file at the given local path may not be modified after this method returns!

            Note! Implementations of readFile need to respect/provide the executable attribute on FileIDs.

            Parameters

                •   file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- ID of the file to be copied

                •   local_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the local path indicating where to place the contents of the given file in the job store

                •   symlink (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- whether the reader can tolerate a symlink. If set to true, the job store may create a symlink instead of a full copy of the file or a hard link.

        read_file_stream(file_id, encoding=None, errors=None)
            Similar to readFile, but returns a context manager yielding a file handle which can be read from. The yielded file handle does not need to and should not be closed explicitly.

            Parameters

                •   file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- ID of the file to get a readable file handle for

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

            Returns
                a context manager yielding a file handle which can be read from

            Return type
                Iterator[Union[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>], IO[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]]

        delete_file(file_id)
            Deletes the file with the given ID from this job store. This operation is idempotent, i.e. deleting a file twice or deleting a non-existent file will succeed silently.

            Parameters
                file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- ID of the file to delete

        file_exists(file_id)
            Determine whether a file exists in this job store.

            Parameters
                file_id -- an ID referencing the file to be checked

        get_file_size(file_id)
            Get the size of the given file in bytes, or 0 if it does not exist when queried.

            Note that job stores which encrypt files might return overestimates of file sizes, since the encrypted file may have been padded to the nearest block, augmented with an initialization vector, etc.

            Parameters
                file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an ID referencing the file to be checked

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        update_file(file_id, local_path)
            Replaces the existing version of a file in the job store.

            Throws an exception if the file does not exist.

            Parameters

                •   file_id -- the ID of the file in the job store to be updated

                •   local_path -- the local path to a file that will overwrite the current version in the job store

            Raises

                •   ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

                •   NoSuchFileException <#toil.jobStores.abstractJobStore.NoSuchFileException> -- if the specified file does not exist

        update_file_stream(file_id, encoding=None, errors=None)
            Replaces the existing version of a file in the job store. Similar to writeFile, but returns a context manager yielding a file handle which can be written to. The yielded file handle does not need to and should not be closed explicitly.

            Parameters

                •   file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the ID of the file in the job store to be updated

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

            Raises

                •   ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

                •   NoSuchFileException <#toil.jobStores.abstractJobStore.NoSuchFileException> -- if the specified file does not exist

        write_shared_file_stream(shared_file_name, encrypted=True, encoding=None, errors=None)
            Returns a context manager yielding a writable file handle to the global file referenced by the given name. File will be created in an atomic manner.

            Parameters

                •   shared_file_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- A file name matching AbstractJobStore.fileNameRegex, unique within this job store

                •   encrypted (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- True if the file must be encrypted, None if it may be encrypted or False if it must be stored in the clear.

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the encoding used to encode the file. Encodings are the same as for encode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

            Raises
                ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

            Returns
                a context manager yielding a writable file handle

            Return type
                Iterator[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]]

        read_shared_file_stream(shared_file_name, isProtected=True, encoding=None, errors=None)
            Returns a context manager yielding a readable file handle to the global file referenced by the given name.

            Parameters

                •   shared_file_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- A file name matching AbstractJobStore.fileNameRegex, unique within this job store

                •   encoding (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the encoding used to decode the file. Encodings are the same as for decode(). Defaults to None which represents binary mode.

                •   errors (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- an optional string that specifies how encoding errors are to be handled. Errors are the same as for open(). Defaults to 'strict' when an encoding is specified.

            Returns
                a context manager yielding a readable file handle

        write_logs(msg)
            Stores a message as a log in the jobstore.

            Parameters
                msg (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the string to be written

            Raises
                ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

            Return type
                None

        read_logs(callback, read_all=False)
            Reads logs accumulated by the write_logs() method. For each log this method calls the given callback function with the message as an argument (rather than returning logs directly, this method must be supplied with a callback which will process log messages).

            Only unread logs will be read unless the read_all parameter is set.

            Parameters

                •   callback (Callable) -- a function to be applied to each of the stats file handles found

                •   read_all (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- a boolean indicating whether to read the already processed stats files in addition to the unread stats files

            Raises
                ConcurrentFileModificationException <#toil.jobStores.abstractJobStore.ConcurrentFileModificationException> -- if the file was modified concurrently during an invocation of this method

            Returns
                the number of stats files processed

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

  toil.jobStores.utils

  Attributes
log |

  Exceptions
JobStoreUnavailableException | Raised when a particular type of job store is requested but can't be used.

  Functions
generate_locator(job_store_type[, local_suggestion, ...]) | Generate a random locator for a job store of the given type. Raises an

  Module Contents

    toil.jobStores.utils.log

    exception toil.jobStores.utils.JobStoreUnavailableException
        Bases: RuntimeError </usr/share/doc/python3/html/library/exceptions.html#RuntimeError>

        Raised when a particular type of job store is requested but can't be used.

    toil.jobStores.utils.generate_locator(job_store_type, local_suggestion=None, decoration=None)
        Generate a random locator for a job store of the given type. Raises an JobStoreUnavailableException if that job store cannot be used.

        Parameters

            •   job_store_type (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Registry name of the job store to use.

            •   local_suggestion (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Path to a nonexistent local directory suitable for use as a file job store.

            •   decoration (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Extra string to add to the job store locator, if convenient.

        Return str
            Job store locator for a usable job store.

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

  toil.leader

The leader script (of the leader/worker pair) for running jobs.

  Attributes
logger |

  Classes
Leader | Represents the Toil leader.

  Module Contents

    toil.leader.logger

    class toil.leader.Leader(config, batchSystem, provisioner, jobStore, rootJob, jobCache=None)
        Represents the Toil leader.

        Responsible for determining what jobs are ready to be scheduled, by consulting the job store, and issuing them in the batch system.

        Parameters

            •   config (toil.common.Config <#toil.common.Config>)

            •   batchSystem (toil.batchSystems.abstractBatchSystem.AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>)

            •   provisioner (Optional[toil.provisioners.abstractProvisioner.AbstractProvisioner <#toil.provisioners.abstractProvisioner.AbstractProvisioner>])

            •   jobStore (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

            •   rootJob (toil.job.JobDescription <#toil.job.JobDescription>)

            •   jobCache (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[Union[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.job.TemporaryID <#toil.job.TemporaryID>], toil.job.JobDescription <#toil.job.JobDescription>]])

        config

        jobStore

        jobStoreLocator

        toilState

        batchSystem

        issued_jobs_by_batch_system_id: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[int </usr/share/doc/python3/html/library/functions.html#int>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        preemptibleJobsIssued = 0

        serviceJobsIssued = 0

        serviceJobsToBeIssued: list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>] = []

        preemptibleServiceJobsIssued = 0

        preemptibleServiceJobsToBeIssued: list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>] = []

        timeSinceJobsLastRescued = None

        reissueMissingJobs_missingHash: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[int </usr/share/doc/python3/html/library/functions.html#int>, int </usr/share/doc/python3/html/library/functions.html#int>]

        provisioner

        clusterScaler = None

        serviceManager

        statsAndLogging

        potentialDeadlockedJobs: set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        potentialDeadlockTime = 0

        toilMetrics: toil.common.ToilMetrics <#toil.common.ToilMetrics> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        debugJobNames = ('CWLJob', 'CWLWorkflow', 'CWLScatter', 'CWLGather', 'ResolveIndirect')

        deadlockThrottler

        statusThrottler

        kill_throttler

        progress_overall = None

        progress_failed = None

        GOOD_COLOR = (0, 60, 108)

        BAD_COLOR = (253, 199, 0)

        PROGRESS_BAR_FORMAT = '{desc}{desc_pad}{percentage:3.0f}%|{bar}| {count:{len_total}d}/{total:d} ({count_1:d} failures)...

        recommended_fail_exit_code = 1

        run()
            Run the leader process to issue and manage jobs.

            Raises
                toil.exceptions.FailedJobsException if failed jobs remain after running.

            Returns
                The return value of the root job's run function.

            Return type
                Any

        create_status_sentinel_file(fail)
            Create a file in the jobstore indicating failure or success.

            Parameters
                fail (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        innerLoop()
            Process jobs.

            This is the leader's main loop.

        checkForDeadlocks()
            Check if the system is deadlocked running service jobs.

        feed_deadlock_watchdog()
            Note that progress has been made and any pending deadlock checks should be reset.

            Return type
                None

        issueJob(jobNode)
            Add a job to the queue of jobs currently trying to run.

            Parameters
                jobNode (toil.job.JobDescription <#toil.job.JobDescription>)

            Return type
                None

        issueJobs(jobs)
            Add a list of jobs, each represented as a jobNode object.

        issueServiceJob(service_id)
            Issue a service job.

            Put it on a queue if the maximum number of service jobs to be scheduled has been reached.

            Parameters
                service_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        issueQueingServiceJobs()
            Issues any queuing service jobs up to the limit of the maximum allowed.

        getNumberOfJobsIssued(preemptible=None)
            Get number of jobs that have been added by issueJob(s) and not removed by removeJob.

            Parameters
                preemptible (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>]) -- If none, return all types of jobs. If true, return just the number of preemptible jobs. If false, return just the number of non-preemptible jobs.

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        removeJob(jobBatchSystemID)
            Remove a job from the system by batch system ID.

            Returns
                Job description as it was issued.

            Parameters
                jobBatchSystemID (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                toil.job.JobDescription <#toil.job.JobDescription>

        getJobs(preemptible=None)
            Get all issued jobs.

            Parameters
                preemptible (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>]) -- If specified, select only preemptible or only non-preemptible jobs.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.job.JobDescription <#toil.job.JobDescription>]

        killJobs(jobsToKill, exit_reason=BatchJobExitReason.KILLED)
            Kills the given set of jobs and then sends them for processing.

            Returns the jobs that, upon processing, were reissued.

            Parameters
                exit_reason (toil.batchSystems.abstractBatchSystem.BatchJobExitReason <#toil.batchSystems.abstractBatchSystem.BatchJobExitReason>)

        reissueOverLongJobs()
            Check each issued job.

            If a job is running for longer than desirable issue a kill instruction. Wait for the job to die then we pass the job to process_finished_job.

            Return type
                None

        reissueMissingJobs(killAfterNTimesMissing=3)
            Check all the current job ids are in the list of currently issued batch system jobs.

            If a job is missing, we mark it as so, if it is missing for a number of runs of this function (say 10).. then we try deleting the job (though its probably lost), we wait then we pass the job to process_finished_job.

        processRemovedJob(issuedJob, result_status)

        process_finished_job(batch_system_id, result_status, wall_time=None, exit_reason=None)
            Process finished jobs.

            Called when an attempt to run a job finishes, either successfully or otherwise.

            Takes the job out of the issued state, and then works out what to do about the fact that it succeeded or failed.

            Returns
                True if the job is going to run again, and False if the job is fully done or completely failed.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        process_finished_job_description(finished_job, result_status, wall_time=None, exit_reason=None, batch_system_id=None)
            Process a finished JobDescription based upon its success or failure.

            If wall-clock time is available, informs the cluster scaler about the job finishing.

            If the job failed and a batch system ID is available, checks for and reports batch system logs.

            Checks if it succeeded and was removed, or if it failed and needs to be set up after failure, and dispatches to the appropriate function.

            Returns
                True if the job is going to run again, and False if the job is fully done or completely failed.

            Parameters

                •   finished_job (toil.job.JobDescription <#toil.job.JobDescription>)

                •   result_status (int </usr/share/doc/python3/html/library/functions.html#int>)

                •   wall_time (Optional[float </usr/share/doc/python3/html/library/functions.html#float>])

                •   exit_reason (Optional[toil.batchSystems.abstractBatchSystem.BatchJobExitReason <#toil.batchSystems.abstractBatchSystem.BatchJobExitReason>])

                •   batch_system_id (Optional[int </usr/share/doc/python3/html/library/functions.html#int>])

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        getSuccessors(job_id, alreadySeenSuccessors)
            Get successors of the given job by walking the job graph recursively.

            Parameters

                •   alreadySeenSuccessors (set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- any successor seen here is ignored and not traversed.

                •   job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Returns
                The set of found successors. This set is added to alreadySeenSuccessors.

            Return type
                set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        processTotallyFailedJob(job_id)
            Process a totally failed job.

            Parameters
                job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

  toil.lib

  Submodules

  toil.lib.accelerators

Accelerator (i.e. GPU) utilities for Toil

  Functions
have_working_nvidia_smi() | Return True if the nvidia-smi binary, from nvidia's CUDA userspace
get_host_accelerator_numbers() | Work out what accelerator is what.
have_working_nvidia_docker_runtime() | Return True if Docker exists and can handle an "nvidia" runtime and the "--gpus" option.
count_nvidia_gpus() | Return the number of nvidia GPUs seen by nvidia-smi, or 0 if it is not working.
count_amd_gpus() | Return the number of amd GPUs seen by rocm-smi, or 0 if it is not working.
get_individual_local_accelerators() | Determine all the local accelerators available. Report each with count 1,
get_restrictive_environment_for_local_accelerators(...) | Get environment variables which can be applied to a process to restrict it

  Module Contents

    toil.lib.accelerators.have_working_nvidia_smi()
        Return True if the nvidia-smi binary, from nvidia's CUDA userspace utilities, is installed and can be run successfully.

        TODO: This isn't quite the same as the check that cwltool uses to decide if it can fulfill a CUDARequirement.

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.accelerators.get_host_accelerator_numbers()
        Work out what accelerator is what.

        For each accelerator visible to us, returns the host-side (for example, outside-of-Slurm-job) number for that accelerator. It is often the same as the apparent number.

        Can be used with Docker's --gpus='"device=#,#,#"' option to forward the right GPUs as seen from a Docker daemon.

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>]

    toil.lib.accelerators.have_working_nvidia_docker_runtime()
        Return True if Docker exists and can handle an "nvidia" runtime and the "--gpus" option.

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.accelerators.count_nvidia_gpus()
        Return the number of nvidia GPUs seen by nvidia-smi, or 0 if it is not working.

        Return type
            int </usr/share/doc/python3/html/library/functions.html#int>

    toil.lib.accelerators.count_amd_gpus()
        Return the number of amd GPUs seen by rocm-smi, or 0 if it is not working. :return:

        Return type
            int </usr/share/doc/python3/html/library/functions.html#int>

    toil.lib.accelerators.get_individual_local_accelerators()
        Determine all the local accelerators available. Report each with count 1, in the order of the number that can be used to assign them.

        TODO: How will numbers work with multiple types of accelerator? We need an accelerator assignment API.

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.job.AcceleratorRequirement <#toil.job.AcceleratorRequirement>]

    toil.lib.accelerators.get_restrictive_environment_for_local_accelerators(accelerator_numbers)
        Get environment variables which can be applied to a process to restrict it to using only the given accelerator numbers.

        The numbers are in the space of accelerators returned by get_individual_local_accelerators().

        Parameters
            accelerator_numbers (Union[set </usr/share/doc/python3/html/library/stdtypes.html#set>[int </usr/share/doc/python3/html/library/functions.html#int>], list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>]])

        Return type
            dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

  toil.lib.aws

  Submodules

  toil.lib.aws.ami

  Attributes
logger |

  Exceptions
ReleaseFeedUnavailableError | Raised when a Flatcar releases can't be located.

  Functions
get_flatcar_ami(ec2_client[, architecture]) | Retrieve the flatcar AMI image to use as the base for all Toil autoscaling instances.
flatcar_release_feed_ami(region[, architecture, source]) | Yield AMI IDs for the given architecture from the Flatcar release feed.
feed_flatcar_ami_release(ec2_client[, architecture, ...]) | Check a Flatcar release feed for the latest flatcar AMI.
aws_marketplace_flatcar_ami_search(ec2_client[, ...]) | Query AWS for all AMI names matching Flatcar-stable-* and return the most recent one.

  Module Contents

    toil.lib.aws.ami.logger

    exception toil.lib.aws.ami.ReleaseFeedUnavailableError
        Bases: RuntimeError </usr/share/doc/python3/html/library/exceptions.html#RuntimeError>

        Raised when a Flatcar releases can't be located.

    toil.lib.aws.ami.get_flatcar_ami(ec2_client, architecture='amd64')
        Retrieve the flatcar AMI image to use as the base for all Toil autoscaling instances.

        AMI must be available to the user on AWS (attempting to launch will return a 403 otherwise).

        Priority is:

            1.  User specified AMI via TOIL_AWS_AMI

            2.  Official AMI from stable.release.flatcar-linux.net

            3.  Search the AWS Marketplace

        Raises
            ReleaseFeedUnavailableError <#toil.lib.aws.ami.ReleaseFeedUnavailableError> -- if all of these sources fail.

        Parameters

            •   ec2_client (botocore.client.BaseClient) -- Boto3 EC2 Client

            •   architecture (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The architecture type for the new AWS machine. Can be either amd64 or arm64

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.aws.ami.flatcar_release_feed_ami(region, architecture='amd64', source='stable')
        Yield AMI IDs for the given architecture from the Flatcar release feed.

        Parameters

            •   source (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- can be set to a Flatcar release channel ('stable', 'beta', or 'alpha'), 'archive' to check the Internet Archive for a feed, and 'toil' to check if the Toil project has put up a feed.

            •   region (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   architecture (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    Retries if the release feed cannot be fetched. If the release feed has a permanent error, yields nothing. If some entries in the release feed are unparseable, yields the others.

    toil.lib.aws.ami.feed_flatcar_ami_release(ec2_client, architecture='amd64', source='stable')
        Check a Flatcar release feed for the latest flatcar AMI.

        Verify it's on AWS.

        Does not raise exceptions.

        Parameters

            •   ec2_client (botocore.client.BaseClient) -- Boto3 EC2 Client

            •   architecture (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The architecture type for the new AWS machine. Can be either amd64 or arm64

            •   source (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- can be set to a Flatcar release channel ('stable', 'beta', or 'alpha'), 'archive' to check the Internet Archive for a feed, and 'toil' to check if the Toil project has put up a feed.

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.lib.aws.ami.aws_marketplace_flatcar_ami_search(ec2_client, architecture='amd64')
        Query AWS for all AMI names matching Flatcar-stable-* and return the most recent one.

        Does not raise exceptions.

        Returns
            An AMI name, or None if no matching AMI was found or we could not talk to AWS.

        Parameters

            •   ec2_client (botocore.client.BaseClient)

            •   architecture (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

  toil.lib.aws.config

  Attributes
S3_PARALLELIZATION_FACTOR |
S3_PART_SIZE |
KiB |
MiB |
AWS_MIN_CHUNK_SIZE |
MULTIPART_THRESHOLD |
AWS_MAX_MULTIPART_COUNT |

  Functions
get_s3_multipart_chunk_size(filesize) | Returns the chunk size of the S3 multipart object, given a file's size in bytes.

  Module Contents

    toil.lib.aws.config.S3_PARALLELIZATION_FACTOR = 8

    toil.lib.aws.config.S3_PART_SIZE = 16777216

    toil.lib.aws.config.KiB = 1024

    toil.lib.aws.config.MiB = 1048576

    toil.lib.aws.config.AWS_MIN_CHUNK_SIZE = 67108864

    toil.lib.aws.config.MULTIPART_THRESHOLD = 67108865

    toil.lib.aws.config.AWS_MAX_MULTIPART_COUNT = 10000

    toil.lib.aws.config.get_s3_multipart_chunk_size(filesize)
        Returns the chunk size of the S3 multipart object, given a file's size in bytes.

        Parameters
            filesize (int </usr/share/doc/python3/html/library/functions.html#int>)

        Return type
            int </usr/share/doc/python3/html/library/functions.html#int>

  toil.lib.aws.iam

  Attributes
logger |
CLUSTER_LAUNCHING_PERMISSIONS |
AllowedActionCollection |

  Functions
delete_iam_instance_profile(instance_profile_name[, ...]) |
delete_iam_role(role_name[, region, quiet]) | Deletes an AWS IAM role. Any separate policies are detached from the role, and any inline policies are deleted.
create_iam_role(role_name, ...[, region]) | Creates an AWS IAM role. Any separate policies are detached from the role, and any inline policies are deleted.
init_action_collection() | Initialization of an action collection, an action collection contains allowed Actions and NotActions
add_to_action_collection(a, b) | Combines two action collections
policy_permissions_allow(given_permissions[, ...]) | Check whether given set of actions are a subset of another given set of actions, returns true if they are
permission_matches_any(perm, list_perms) | Takes a permission and checks whether it's contained within a list of given permissions
get_actions_from_policy_document(policy_doc) | Given a policy document, go through each statement and create an AllowedActionCollection representing the
allowed_actions_attached(iam, attached_policies) | Go through all attached policy documents and create an AllowedActionCollection representing granted permissions.
allowed_actions_roles(iam, policy_names, role_name) | Returns a dictionary containing a list of all aws actions allowed for a given role.
collect_policy_actions(policy_documents) | Collect all of the actions allowed by the given policy documents into one AllowedActionCollection.
allowed_actions_user(iam, policy_names, user_name) | Gets all allowed actions for a user given by user_name, returns a dictionary, keyed by resource,
allowed_actions_group(iam, policy_names, group_name) | Gets all allowed actions for a group given by group_name, returns a dictionary, keyed by resource,
get_policy_permissions(region) | Returns an action collection containing lists of all permission grant patterns keyed by resource
get_aws_account_num() | Returns AWS account num

  Module Contents

    toil.lib.aws.iam.logger

    toil.lib.aws.iam.CLUSTER_LAUNCHING_PERMISSIONS = ['iam:CreateRole', 'iam:CreateInstanceProfile', 'iam:TagInstanceProfile', 'iam:DeleteRole',...

    toil.lib.aws.iam.AllowedActionCollection

    toil.lib.aws.iam.delete_iam_instance_profile(instance_profile_name, region=None, quiet=True)

        Parameters

            •   instance_profile_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   region (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   quiet (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            None

    toil.lib.aws.iam.delete_iam_role(role_name, region=None, quiet=True)
        Deletes an AWS IAM role. Any separate policies are detached from the role, and any inline policies are deleted.

        Parameters

            •   role_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The name of the AWS IAM role.

            •   region (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- The AWS region that the role_name is in.

            •   quiet (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Whether or not to print/log information about the deletion to stdout.

        Return type
            None

    toil.lib.aws.iam.create_iam_role(role_name, assume_role_policy_document, policies, region=None)
        Creates an AWS IAM role. Any separate policies are detached from the role, and any inline policies are deleted.

        Parameters

            •   role_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The name of the AWS IAM role.

            •   region (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- The AWS region that the role_name is in.

            •   assume_role_policy_document (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Policies to create inline with the role.

            •   policies (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]) -- Global policies to attach to the role.

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.aws.iam.init_action_collection()
        Initialization of an action collection, an action collection contains allowed Actions and NotActions by resource, these are patterns containing wildcards, an Action explicitly allows a matched pattern, eg ec2:* will explicitly allow all ec2 permissions

        A NotAction will explicitly allow all actions that don't match a specific pattern eg iam:* allows all non iam actions

        Return type
            AllowedActionCollection

    toil.lib.aws.iam.add_to_action_collection(a, b)
        Combines two action collections

        Parameters

            •   a (AllowedActionCollection)

            •   b (AllowedActionCollection)

        Return type
            AllowedActionCollection

    toil.lib.aws.iam.policy_permissions_allow(given_permissions, required_permissions=[])
        Check whether given set of actions are a subset of another given set of actions, returns true if they are otherwise false and prints a warning.

        Parameters

            •   required_permissions (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Dictionary containing actions required, keyed by resource

            •   given_permissions (AllowedActionCollection) -- Set of actions that are granted to a user or role

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.aws.iam.permission_matches_any(perm, list_perms)
        Takes a permission and checks whether it's contained within a list of given permissions Returns True if it is otherwise False

        Parameters

            •   perm (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Permission to check in string form

            •   list_perms (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Permission list to check against

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.aws.iam.get_actions_from_policy_document(policy_doc)
        Given a policy document, go through each statement and create an AllowedActionCollection representing the permissions granted in the policy document.

        Parameters
            policy_doc (mypy_boto3_iam.type_defs.PolicyDocumentDictTypeDef) -- A policy document to examine

        Return type
            AllowedActionCollection

    toil.lib.aws.iam.allowed_actions_attached(iam, attached_policies)
        Go through all attached policy documents and create an AllowedActionCollection representing granted permissions.

        Parameters

            •   iam (mypy_boto3_iam.IAMClient) -- IAM client to use

            •   attached_policies (list </usr/share/doc/python3/html/library/stdtypes.html#list>[mypy_boto3_iam.type_defs.AttachedPolicyTypeDef]) -- Attached policies

        Return type
            AllowedActionCollection

    toil.lib.aws.iam.allowed_actions_roles(iam, policy_names, role_name)
        Returns a dictionary containing a list of all aws actions allowed for a given role. This dictionary is keyed by resource and gives a list of policies allowed on that resource.

        Parameters

            •   iam (mypy_boto3_iam.IAMClient) -- IAM client to use

            •   policy_names (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Name of policy document associated with a role

            •   role_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Name of role to get associated policies

        Return type
            AllowedActionCollection

    toil.lib.aws.iam.collect_policy_actions(policy_documents)
        Collect all of the actions allowed by the given policy documents into one AllowedActionCollection.

        Parameters
            policy_documents (list </usr/share/doc/python3/html/library/stdtypes.html#list>[Union[str </usr/share/doc/python3/html/library/stdtypes.html#str>, mypy_boto3_iam.type_defs.PolicyDocumentDictTypeDef]])

        Return type
            AllowedActionCollection

    toil.lib.aws.iam.allowed_actions_user(iam, policy_names, user_name)
        Gets all allowed actions for a user given by user_name, returns a dictionary, keyed by resource, with a list of permissions allowed for each given resource.

        Parameters

            •   iam (mypy_boto3_iam.IAMClient) -- IAM client to use

            •   policy_names (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Name of policy document associated with a user

            •   user_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Name of user to get associated policies

        Return type
            AllowedActionCollection

    toil.lib.aws.iam.allowed_actions_group(iam, policy_names, group_name)
        Gets all allowed actions for a group given by group_name, returns a dictionary, keyed by resource, with a list of permissions allowed for each given resource.

        Parameters

            •   iam (mypy_boto3_iam.IAMClient) -- IAM client to use

            •   policy_names (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Name of policy document associated with a user

            •   group_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Name of group to get associated policies

        Return type
            AllowedActionCollection

    toil.lib.aws.iam.get_policy_permissions(region)
        Returns an action collection containing lists of all permission grant patterns keyed by resource that they are allowed upon. Requires AWS credentials to be associated with a user or assumed role.

        Parameters

            •   zone -- AWS zone to connect to

            •   region (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            AllowedActionCollection

    toil.lib.aws.iam.get_aws_account_num()
        Returns AWS account num

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

  toil.lib.aws.s3

  Attributes
logger |
AWS_MAX_MULTIPART_COUNT |
AWS_MAX_CHUNK_SIZE |
AWS_MIN_CHUNK_SIZE |
DEFAULT_AWS_CHUNK_SIZE |

  Exceptions
AWSKeyNotFoundError | Common base class for all non-exit exceptions.
AWSKeyAlreadyExistsError | Common base class for all non-exit exceptions.
AWSBadEncryptionKeyError | Common base class for all non-exit exceptions.

  Classes
MultiPartPipe | An object-oriented wrapper for os.pipe. Clients should subclass it, implement

  Functions
create_s3_bucket(s3_resource, bucket_name, region[, ...]) | Create an AWS S3 bucket, using the given Boto3 S3 session, with the
delete_s3_bucket(s3_resource, bucket_name[, quiet]) | Delete the bucket with 'bucket_name'.
bucket_exists(s3_resource, bucket) |
head_s3_object(bucket, key, header[, region]) | Attempt to HEAD an s3 object and return its response.
list_multipart_uploads(bucket, region, prefix[, ...]) |
copy_s3_to_s3(s3_resource, src_bucket, src_key, ...[, ...]) |
copy_local_to_s3(s3_resource, local_file_path, ...[, ...]) |
copy_s3_to_local(s3_resource, local_file_path, ...[, ...]) |
parse_s3_uri(uri) |
list_s3_items(s3_resource, bucket, prefix[, startafter]) |
upload_to_s3(readable, s3_resource, bucket, key[, ...]) | Upload a readable object to s3, using multipart uploading if applicable.
download_stream(s3_resource, bucket, key[, ...]) | Context manager that gives out a download stream to download data.
download_fileobject(s3_resource, bucket, key, fileobj) |
s3_key_exists(s3_resource, bucket, key[, check, ...]) | Return True if the s3 obect exists, and False if not. Will error if encryption args are incorrect.
get_s3_object(s3_resource, bucket, key[, extra_args]) |
put_s3_object(s3_resource, bucket, key, body[, extra_args]) |
generate_presigned_url(s3_resource, bucket, key_name, ...) |
create_public_url(s3_resource, bucket, key) |
get_s3_bucket_region(s3_resource, bucket) |

  Module Contents

    toil.lib.aws.s3.logger

    toil.lib.aws.s3.AWS_MAX_MULTIPART_COUNT = 10000

    toil.lib.aws.s3.AWS_MAX_CHUNK_SIZE = 5000000000000

    toil.lib.aws.s3.AWS_MIN_CHUNK_SIZE = 5000000

    toil.lib.aws.s3.DEFAULT_AWS_CHUNK_SIZE = 134217728

    exception toil.lib.aws.s3.AWSKeyNotFoundError
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Common base class for all non-exit exceptions.

    exception toil.lib.aws.s3.AWSKeyAlreadyExistsError
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Common base class for all non-exit exceptions.

    exception toil.lib.aws.s3.AWSBadEncryptionKeyError
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Common base class for all non-exit exceptions.

    toil.lib.aws.s3.create_s3_bucket(s3_resource, bucket_name, region, tags=None, public=True)
        Create an AWS S3 bucket, using the given Boto3 S3 session, with the given name, in the given region.

        Supports the us-east-1 region, where bucket creation is special.

        ALL S3 bucket creation should use this function.

        Parameters

            •   s3_resource (mypy_boto3_s3.S3ServiceResource)

            •   bucket_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   region (Union[mypy_boto3_s3.literals.BucketLocationConstraintType, Literal['us-east-1']])

            •   tags (Optional[Dict[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

            •   public (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            mypy_boto3_s3.service_resource.Bucket

    toil.lib.aws.s3.delete_s3_bucket(s3_resource, bucket_name, quiet=True)
        Delete the bucket with 'bucket_name'.

        Note: 'quiet' is False when used for a clean up utility script (contrib/admin/cleanup_aws_resources.py)
            that prints progress rather than logging. Logging should be used for all other internal Toil usage.

        Parameters

            •   s3_resource (mypy_boto3_s3.S3ServiceResource)

            •   bucket_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   quiet (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            None

    toil.lib.aws.s3.bucket_exists(s3_resource, bucket)

        Parameters

            •   s3_resource (mypy_boto3_s3.S3ServiceResource)

            •   bucket (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            Union[bool </usr/share/doc/python3/html/library/functions.html#bool>, mypy_boto3_s3.service_resource.Bucket]

    toil.lib.aws.s3.head_s3_object(bucket, key, header, region=None)
        Attempt to HEAD an s3 object and return its response.

        Parameters

            •   bucket (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- AWS bucket name

            •   key (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- AWS Key name for the s3 object

            •   header (Dict[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]) -- Headers to include (mostly for encryption). See: <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/head_object.html>

            •   region (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Region that we want to look for the bucket in

        Return type
            mypy_boto3_s3.type_defs.HeadObjectOutputTypeDef

    toil.lib.aws.s3.list_multipart_uploads(bucket, region, prefix, max_uploads=1)

        Parameters

            •   bucket (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   region (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   prefix (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   max_uploads (int </usr/share/doc/python3/html/library/functions.html#int>)

        Return type
            mypy_boto3_s3.type_defs.ListMultipartUploadsOutputTypeDef

    toil.lib.aws.s3.copy_s3_to_s3(s3_resource, src_bucket, src_key, dst_bucket, dst_key, extra_args=None)

        Parameters

            •   s3_resource (mypy_boto3_s3.S3ServiceResource)

            •   src_bucket (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   src_key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   dst_bucket (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   dst_key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   extra_args (Optional[Dict[Any, Any]])

        Return type
            None

    toil.lib.aws.s3.copy_local_to_s3(s3_resource, local_file_path, dst_bucket, dst_key, extra_args=None)

        Parameters

            •   s3_resource (mypy_boto3_s3.S3ServiceResource)

            •   local_file_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   dst_bucket (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   dst_key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   extra_args (Optional[Dict[Any, Any]])

        Return type
            None

    toil.lib.aws.s3.copy_s3_to_local(s3_resource, local_file_path, src_bucket, src_key, extra_args=None)

        Parameters

            •   s3_resource (mypy_boto3_s3.S3ServiceResource)

            •   local_file_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   src_bucket (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   src_key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   extra_args (Optional[Dict[Any, Any]])

        Return type
            None

    class toil.lib.aws.s3.MultiPartPipe(part_size, s3_resource, bucket_name, file_id, encryption_args, encoding=None, errors=None)
        Bases: toil.lib.pipes.WritablePipe <#toil.lib.pipes.WritablePipe>

        An object-oriented wrapper for os.pipe. Clients should subclass it, implement readFrom() to consume the readable end of the pipe, then instantiate the class as a context manager to get the writable end. See the example below.

            >>> import sys, shutil, codecs
            >>> class MyPipe(WritablePipe):
            ...     def readFrom(self, readable):
            ...         shutil.copyfileobj(codecs.getreader('utf-8')(readable), sys.stdout)
            >>> with MyPipe() as writable:
            ...     _ = writable.write('Hello, world!\n'.encode('utf-8'))
            Hello, world!

        Each instance of this class creates a thread and invokes the readFrom method in that thread. The thread will be join()ed upon normal exit from the context manager, i.e. the body of the with statement. If an exception occurs, the thread will not be joined but a well-behaved readFrom() implementation will terminate shortly thereafter due to the pipe having been closed.

        Now, exceptions in the reader thread will be reraised in the main thread:

            >>> class MyPipe(WritablePipe):
            ...     def readFrom(self, readable):
            ...         raise RuntimeError('Hello, world!')
            >>> with MyPipe() as writable:
            ...     pass
            Traceback (most recent call last):
            ...
            RuntimeError: Hello, world!

        More complicated, less illustrative tests:

        Same as above, but proving that handles are closed:

            >>> x = os.dup(0); os.close(x)
            >>> class MyPipe(WritablePipe):
            ...     def readFrom(self, readable):
            ...         raise RuntimeError('Hello, world!')
            >>> with MyPipe() as writable:
            ...     pass
            Traceback (most recent call last):
            ...
            RuntimeError: Hello, world!
            >>> y = os.dup(0); os.close(y); x == y
            True

        Exceptions in the body of the with statement aren't masked, and handles are closed:

            >>> x = os.dup(0); os.close(x)
            >>> class MyPipe(WritablePipe):
            ...     def readFrom(self, readable):
            ...         pass
            >>> with MyPipe() as writable:
            ...     raise RuntimeError('Hello, world!')
            Traceback (most recent call last):
            ...
            RuntimeError: Hello, world!
            >>> y = os.dup(0); os.close(y); x == y
            True

        Parameters

            •   part_size (int </usr/share/doc/python3/html/library/functions.html#int>)

            •   s3_resource (mypy_boto3_s3.S3ServiceResource)

            •   bucket_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   encryption_args (Optional[Dict[Any, Any]])

            •   encoding (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   errors (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        encoding = None

        errors = None

        part_size

        s3_client

        bucket_name

        file_id

        encryption_args

        readFrom(readable)
            Implement this method to read data from the pipe. This method should support both binary and text mode output.

            Parameters
                readable (file) -- the file object representing the readable end of the pipe. Do not explicitly invoke the close() method of the object; that will be done automatically.

            Return type
                None

    toil.lib.aws.s3.parse_s3_uri(uri)

        Parameters
            uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            Tuple[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.lib.aws.s3.list_s3_items(s3_resource, bucket, prefix, startafter=None)

        Parameters

            •   s3_resource (mypy_boto3_s3.S3ServiceResource)

            •   bucket (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   prefix (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   startafter (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            Iterator[mypy_boto3_s3.type_defs.ObjectTypeDef]

    toil.lib.aws.s3.upload_to_s3(readable, s3_resource, bucket, key, extra_args=None)
        Upload a readable object to s3, using multipart uploading if applicable.

        Parameters

            •   readable (IO[Any]) -- a readable stream or a local file path to upload to s3

            •   resource (S3.Resource) -- boto3 resource

            •   bucket (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- name of the bucket to upload to

            •   key (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the file to upload to

            •   extra_args (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>) -- http headers to use when uploading - generally used for encryption purposes

            •   partSize (int </usr/share/doc/python3/html/library/functions.html#int>) -- max size of each part in the multipart upload, in bytes

            •   s3_resource (mypy_boto3_s3.S3ServiceResource)

        Returns
            version of the newly uploaded file

        Return type
            None

    toil.lib.aws.s3.download_stream(s3_resource, bucket, key, checksum_to_verify=None, extra_args=None, encoding=None, errors=None)
        Context manager that gives out a download stream to download data.

        Parameters

            •   s3_resource (mypy_boto3_s3.S3ServiceResource)

            •   bucket (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   checksum_to_verify (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   extra_args (Optional[Dict[Any, Any]])

            •   encoding (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   errors (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            Iterator[IO[Any]]

    toil.lib.aws.s3.download_fileobject(s3_resource, bucket, key, fileobj, extra_args=None)

        Parameters

            •   s3_resource (mypy_boto3_s3.S3ServiceResource)

            •   bucket (mypy_boto3_s3.service_resource.Bucket)

            •   key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   fileobj (io.BytesIO </usr/share/doc/python3/html/library/io.html#io.BytesIO>)

            •   extra_args (Optional[Dict[Any, Any]])

        Return type
            None

    toil.lib.aws.s3.s3_key_exists(s3_resource, bucket, key, check=False, extra_args=None)
        Return True if the s3 obect exists, and False if not. Will error if encryption args are incorrect.

        Parameters

            •   s3_resource (mypy_boto3_s3.S3ServiceResource)

            •   bucket (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   check (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            •   extra_args (Optional[Dict[Any, Any]])

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.aws.s3.get_s3_object(s3_resource, bucket, key, extra_args=None)

        Parameters

            •   s3_resource (mypy_boto3_s3.S3ServiceResource)

            •   bucket (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   extra_args (Optional[Dict[Any, Any]])

        Return type
            mypy_boto3_s3.type_defs.GetObjectOutputTypeDef

    toil.lib.aws.s3.put_s3_object(s3_resource, bucket, key, body, extra_args=None)

        Parameters

            •   s3_resource (mypy_boto3_s3.S3ServiceResource)

            •   bucket (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   body (Union[str </usr/share/doc/python3/html/library/stdtypes.html#str>, bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>])

            •   extra_args (Optional[Dict[Any, Any]])

        Return type
            mypy_boto3_s3.type_defs.PutObjectOutputTypeDef

    toil.lib.aws.s3.generate_presigned_url(s3_resource, bucket, key_name, expiration)

        Parameters

            •   s3_resource (mypy_boto3_s3.S3ServiceResource)

            •   bucket (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   key_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   expiration (int </usr/share/doc/python3/html/library/functions.html#int>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.aws.s3.create_public_url(s3_resource, bucket, key)

        Parameters

            •   s3_resource (mypy_boto3_s3.S3ServiceResource)

            •   bucket (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.aws.s3.get_s3_bucket_region(s3_resource, bucket)

        Parameters

            •   s3_resource (mypy_boto3_s3.S3ServiceResource)

            •   bucket (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

  toil.lib.aws.session

  Attributes
logger |
ANONYMOUS_CONFIG |

  Classes
AWSConnectionManager | Class that represents a connection to AWS. Caches Boto 3 and Boto 2 objects

  Functions
establish_boto3_session([region_name]) | Get a Boto 3 session usable by the current thread.
client(…) | Get a Boto 3 client for a particular AWS service, usable by the current thread.
resource(…) | Get a Boto 3 resource for a particular AWS service, usable by the current thread.

  Module Contents

    toil.lib.aws.session.logger

    toil.lib.aws.session.ANONYMOUS_CONFIG

    class toil.lib.aws.session.AWSConnectionManager
        Class that represents a connection to AWS. Caches Boto 3 and Boto 2 objects by region.

        Access to any kind of item goes through the particular method for the thing you want (session, resource, service, Boto2 Context), and then you pass the region you want to work in, and possibly the type of thing you want, as arguments.

        This class is intended to eventually enable multi-region clusters, where connections to multiple regions may need to be managed in the same provisioner.

        We also support None for a region, in which case no region will be passed to Boto/Boto3. The caller is responsible for implementing e.g. TOIL_AWS_REGION support.

        Since connection objects may not be thread safe (see <<https://boto3.amazonaws.com/v1/documentation/api/1.14.31/guide/session.html#multithreading-or-multiprocessing-with-sessions>>), one is created for each thread that calls the relevant lookup method.

        sessions_by_region: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, threading.local </usr/share/doc/python3/html/library/threading.html#threading.local>]

        resource_cache: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>], threading.local </usr/share/doc/python3/html/library/threading.html#threading.local>]

        client_cache: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>], threading.local </usr/share/doc/python3/html/library/threading.html#threading.local>]

        boto2_cache: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, str </usr/share/doc/python3/html/library/stdtypes.html#str>], threading.local </usr/share/doc/python3/html/library/threading.html#threading.local>]

        session(region)
            Get the Boto3 Session to use for the given region.

            Parameters
                region (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                boto3.session.Session

        resource(region: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, service_name: Literal['s3'], endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_s3.S3ServiceResource

        resource(region: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, service_name: Literal['iam'], endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_iam.IAMServiceResource

        resource(region: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, service_name: Literal['ec2'], endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_ec2.EC2ServiceResource
            Get the Boto3 Resource to use with the given service (like 'ec2') in the given region.

            Parameters
                endpoint_url -- AWS endpoint URL to use for the client. If not specified, a default is used.

        client(region: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, service_name: Literal['ec2'], endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_ec2.EC2Client

        client(region: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, service_name: Literal['iam'], endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_iam.IAMClient

        client(region: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, service_name: Literal['s3'], endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_s3.S3Client

        client(region: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, service_name: Literal['sts'], endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_sts.STSClient

        client(region: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, service_name: Literal['sdb'], endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_sdb.SimpleDBClient

        client(region: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>, service_name: Literal['autoscaling'], endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_autoscaling.AutoScalingClient
            Get the Boto3 Client to use with the given service (like 'ec2') in the given region.

            Parameters

                •   endpoint_url -- AWS endpoint URL to use for the client. If not specified, a default is used.

                •   config -- Custom configuration to use for the client.

    toil.lib.aws.session.establish_boto3_session(region_name=None)
        Get a Boto 3 session usable by the current thread.

        This function may not always establish a new session; it can be memoized.

        Parameters
            region_name (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            boto3.Session

    toil.lib.aws.session.client(service_name: Literal['ec2'], region_name: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_ec2.EC2Client

    toil.lib.aws.session.client(service_name: Literal['iam'], region_name: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_iam.IAMClient

    toil.lib.aws.session.client(service_name: Literal['s3'], region_name: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_s3.S3Client

    toil.lib.aws.session.client(service_name: Literal['sts'], region_name: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_sts.STSClient

    toil.lib.aws.session.client(service_name: Literal['sdb'], region_name: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_sdb.SimpleDBClient

    toil.lib.aws.session.client(service_name: Literal['autoscaling'], region_name: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_autoscaling.AutoScalingClient
        Get a Boto 3 client for a particular AWS service, usable by the current thread.

        Global alternative to AWSConnectionManager.

    toil.lib.aws.session.resource(service_name: Literal['s3'], region_name: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_s3.S3ServiceResource

    toil.lib.aws.session.resource(service_name: Literal['iam'], region_name: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_iam.IAMServiceResource

    toil.lib.aws.session.resource(service_name: Literal['ec2'], region_name: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, endpoint_url: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, config: botocore.client.Config | None </usr/share/doc/python3/html/library/constants.html#None> = None) -> mypy_boto3_ec2.EC2ServiceResource
        Get a Boto 3 resource for a particular AWS service, usable by the current thread.

        Global alternative to AWSConnectionManager.

  toil.lib.aws.utils

  Attributes
logger |
THROTTLED_ERROR_CODES |

  Exceptions
NoBucketLocationError | Error to represent that we could not get a location for a bucket.

  Functions
delete_sdb_domain(sdb_domain_name[, region, quiet]) |
connection_reset(e) | Return true if an error is a connection reset error.
connection_error(e) | Return True if an error represents a failure to make a network connection.
retryable_s3_errors(e) | Return true if this is an error from S3 that looks like we ought to retry our request.
retry_s3([delays, timeout, predicate]) | Retry iterator of context managers specifically for S3 operations.
delete_s3_bucket(s3_resource, bucket[, quiet]) | Delete the given S3 bucket.
create_s3_bucket(s3_resource, bucket_name, region) | Create an AWS S3 bucket, using the given Boto3 S3 session, with the
enable_public_objects(bucket_name) | Enable a bucket to contain objects which are public.
get_bucket_region(bucket_name[, endpoint_url, ...]) | Get the AWS region name associated with the given S3 bucket, or raise NoBucketLocationError.
get_bucket_region_if_available(bucket_name[, ...]) | Get the AWS region name associated with the given S3 bucket, or return None.
region_to_bucket_location(region) |
bucket_location_to_region(location) |
get_object_for_url(url[, existing, anonymous]) | Extracts a key (object) from a given parsed s3:// URL.
list_objects_for_url(url[, anonymous]) | Extracts a key (object) from a given parsed s3:// URL. The URL will be
flatten_tags(tags) | Convert tags from a key to value dict into a list of 'Key': xxx, 'Value': xxx dicts.
boto3_pager(requestor_callable, result_attribute_name, ...) | Yield all the results from calling the given Boto 3 method with the

  Module Contents

    toil.lib.aws.utils.logger

    toil.lib.aws.utils.THROTTLED_ERROR_CODES = ['Throttling', 'ThrottlingException', 'ThrottledException', 'RequestThrottledException',...

    toil.lib.aws.utils.delete_sdb_domain(sdb_domain_name, region=None, quiet=True)

        Parameters

            •   sdb_domain_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   region (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   quiet (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            None

    toil.lib.aws.utils.connection_reset(e)
        Return true if an error is a connection reset error.

        Parameters
            e (Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>)

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.aws.utils.connection_error(e)
        Return True if an error represents a failure to make a network connection.

        Parameters
            e (Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>)

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.aws.utils.retryable_s3_errors(e)
        Return true if this is an error from S3 that looks like we ought to retry our request.

        Parameters
            e (Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>)

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.aws.utils.retry_s3(delays=DEFAULT_DELAYS, timeout=DEFAULT_TIMEOUT, predicate=retryable_s3_errors)
        Retry iterator of context managers specifically for S3 operations.

        Parameters

            •   delays (collections.abc.Iterable </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterable>[float </usr/share/doc/python3/html/library/functions.html#float>])

            •   timeout (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   predicate (Callable[[Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>], bool </usr/share/doc/python3/html/library/functions.html#bool>])

        Return type
            collections.abc.Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[ContextManager[None]]

    toil.lib.aws.utils.delete_s3_bucket(s3_resource, bucket, quiet=True)
        Delete the given S3 bucket.

        Note that S3 bucket deletion is only eventually-consistent, and anyone can register any S3 bucket name. For both of these reasons, a bucket name might not be immediately (or ever) available for re-use after this function returns.

        Parameters

            •   s3_resource (mypy_boto3_s3.S3ServiceResource)

            •   bucket (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   quiet (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            None

    toil.lib.aws.utils.create_s3_bucket(s3_resource, bucket_name, region)
        Create an AWS S3 bucket, using the given Boto3 S3 session, with the given name, in the given region.

        Supports the us-east-1 region, where bucket creation is special.

        ALL S3 bucket creation should use this function.

        Parameters

            •   s3_resource (mypy_boto3_s3.S3ServiceResource)

            •   bucket_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   region (toil.lib.aws.AWSRegionName)

        Return type
            mypy_boto3_s3.service_resource.Bucket

    toil.lib.aws.utils.enable_public_objects(bucket_name)
        Enable a bucket to contain objects which are public.

        This adjusts the bucket's Public Access Block setting to not block all public access, and also adjusts the bucket's Object Ownership setting to a setting which enables object ACLs.

        Does not touch the account's Public Access Block setting, which can also interfere here. That is probably best left to the account administrator.

        This configuration used to be the default, and is what most of Toil's code is written to expect, but it was changed so that new buckets default to the more restrictive setting <<https://aws.amazon.com/about-aws/whats-new/2022/12/amazon-s3-automatically-enable-block-public-access-disable-access-control-lists-buckets-april-2023/>>, with the expectation that people would write IAM policies for the buckets to allow public access if needed. Toil expects to be able to make arbitrary objects in arbitrary places public, and naming them all in an IAM policy would be a very awkward way to do it. So we restore the old behavior.

        Parameters
            bucket_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            None

    exception toil.lib.aws.utils.NoBucketLocationError
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Error to represent that we could not get a location for a bucket.

    toil.lib.aws.utils.get_bucket_region(bucket_name, endpoint_url=None, only_strategies=None, anonymous=None)
        Get the AWS region name associated with the given S3 bucket, or raise NoBucketLocationError.

        Does not log at info level or above when this does not work; failures are expected in some contexts.

        Takes an optional S3 API URL override.

        Parameters

            •   only_strategies (Optional[set </usr/share/doc/python3/html/library/stdtypes.html#set>[int </usr/share/doc/python3/html/library/functions.html#int>]]) -- For testing, use only strategies with 1-based numbers in this set.

            •   bucket_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   endpoint_url (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   anonymous (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>])

        Raises
            NoBucketLocationError <#toil.lib.aws.utils.NoBucketLocationError> -- if the bucket's region cannot be determined (possibly due to lack of permissions).

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.aws.utils.get_bucket_region_if_available(bucket_name, endpoint_url=None, only_strategies=None, anonymous=None)
        Get the AWS region name associated with the given S3 bucket, or return None.

        Caches results, so may not return the location for a bucket that has been created but was previously observed to be nonexistent.

        Parameters

            •   only_strategies (Optional[set </usr/share/doc/python3/html/library/stdtypes.html#set>[int </usr/share/doc/python3/html/library/functions.html#int>]]) -- For testing, use only strategies with 1-based numbers in this set.

            •   bucket_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   endpoint_url (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   anonymous (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>])

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.lib.aws.utils.region_to_bucket_location(region)

        Parameters
            region (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.aws.utils.bucket_location_to_region(location)

        Parameters
            location (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.aws.utils.get_object_for_url(url, existing=None, anonymous=None)
        Extracts a key (object) from a given parsed s3:// URL.

        If existing is true and the object does not exist, raises FileNotFoundError.

        Parameters

            •   existing (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If True, key is expected to exist. If False, key is expected not to exists and it will be created. If None, the key will be created if it doesn't exist.

            •   url (urllib.parse.ParseResult </usr/share/doc/python3/html/library/urllib.parse.html#urllib.parse.ParseResult>)

            •   anonymous (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>])

        Raises

            •   FileNotFoundError </usr/share/doc/python3/html/library/exceptions.html#FileNotFoundError> -- when existing is True and the object does not exist.

            •   RuntimeError </usr/share/doc/python3/html/library/exceptions.html#RuntimeError> -- when existing is False but the object exists.

            •   PermissionError </usr/share/doc/python3/html/library/exceptions.html#PermissionError> -- when we are not authorized to look at the object.

        Return type
            mypy_boto3_s3.service_resource.Object

    toil.lib.aws.utils.list_objects_for_url(url, anonymous=None)
        Extracts a key (object) from a given parsed s3:// URL. The URL will be supplemented with a trailing slash if it is missing.

        Raises
            PermissionError </usr/share/doc/python3/html/library/exceptions.html#PermissionError> -- when we are not authorized to do the list operation.

        Parameters

            •   url (urllib.parse.ParseResult </usr/share/doc/python3/html/library/urllib.parse.html#urllib.parse.ParseResult>)

            •   anonymous (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>])

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.lib.aws.utils.flatten_tags(tags)
        Convert tags from a key to value dict into a list of 'Key': xxx, 'Value': xxx dicts.

        Parameters
            tags (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[Union[Literal['Key'], Literal['Value']], str]]

    toil.lib.aws.utils.boto3_pager(requestor_callable, result_attribute_name, **kwargs)
        Yield all the results from calling the given Boto 3 method with the given keyword arguments, paging through the results using the Marker or NextToken, and fetching out and looping over the list in the response with the given attribute name.

        Parameters

            •   requestor_callable (Callable[Ellipsis, Any])

            •   result_attribute_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   kwargs (Any)

        Return type
            collections.abc.Iterable </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterable>[Any]

  Attributes
AWSRegionName |
AWSServerErrors |
logger |

  Functions
get_current_aws_region() | Return the AWS region that the currently configured AWS zone (see
get_aws_zone_from_environment() | Get the AWS zone from TOIL_AWS_ZONE if set.
get_aws_zone_from_metadata() | Get the AWS zone from instance metadata, if on EC2 and the boto module is
get_aws_zone_from_boto() | Get the AWS zone from the Boto3 config file or from AWS_DEFAULT_REGION, if it is configured and the
get_aws_zone_from_environment_region() | Pick an AWS zone in the region defined by TOIL_AWS_REGION, if it is set.
get_current_aws_zone() | Get the currently configured or occupied AWS zone to use.
zone_to_region(zone) | Get a region (e.g. us-west-2) from a zone (e.g. us-west-1c).
running_on_ec2() | Return True if we are currently running on EC2, and false otherwise.
running_on_ecs() | Return True if we are currently running on Amazon ECS, and false otherwise.
build_tag_dict_from_env([environment]) |

  Package Contents

    toil.lib.aws.AWSRegionName

    toil.lib.aws.AWSServerErrors

    toil.lib.aws.logger

    toil.lib.aws.get_current_aws_region()
        Return the AWS region that the currently configured AWS zone (see get_current_aws_zone()) is in.

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.lib.aws.get_aws_zone_from_environment()
        Get the AWS zone from TOIL_AWS_ZONE if set.

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.lib.aws.get_aws_zone_from_metadata()
        Get the AWS zone from instance metadata, if on EC2 and the boto module is available. Otherwise, gets the AWS zone from ECS task metadata, if on ECS.

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.lib.aws.get_aws_zone_from_boto()
        Get the AWS zone from the Boto3 config file or from AWS_DEFAULT_REGION, if it is configured and the boto3 module is available.

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.lib.aws.get_aws_zone_from_environment_region()
        Pick an AWS zone in the region defined by TOIL_AWS_REGION, if it is set.

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.lib.aws.get_current_aws_zone()
        Get the currently configured or occupied AWS zone to use.

        Reports the TOIL_AWS_ZONE environment variable if set.

        Otherwise, if we have boto and are running on EC2, or if we are on ECS, reports the zone we are running in.

        Otherwise, if we have the TOIL_AWS_REGION variable set, chooses a zone in that region.

        Finally, if we have boto2, and a default region is configured in Boto 2, chooses a zone in that region.

        Returns 'us-east-1a' if no method can produce a zone to use.

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.lib.aws.zone_to_region(zone)
        Get a region (e.g. us-west-2) from a zone (e.g. us-west-1c).

        Parameters
            zone (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            AWSRegionName

    toil.lib.aws.running_on_ec2()
        Return True if we are currently running on EC2, and false otherwise.

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.aws.running_on_ecs()
        Return True if we are currently running on Amazon ECS, and false otherwise.

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.aws.build_tag_dict_from_env(environment=os.environ)

        Parameters
            environment (collections.abc.MutableMapping </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.MutableMapping>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

  toil.lib.bioio

  Functions
system(command) | A convenience wrapper around subprocess.check_call that logs the command before passing it
getLogLevelString([logger]) |
setLoggingFromOptions(options) |
getTempFile([suffix, rootDir]) |

  Module Contents

    toil.lib.bioio.system(command)
        A convenience wrapper around subprocess.check_call that logs the command before passing it on. The command can be either a string or a sequence of strings. If it is a string shell=True will be passed to subprocess.check_call. :type command: str | sequence[string]

        Parameters
            command (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            None

    toil.lib.bioio.getLogLevelString(logger=None)

    toil.lib.bioio.setLoggingFromOptions(options)

    toil.lib.bioio.getTempFile(suffix='', rootDir=None)

  toil.lib.checksum

  Attributes
logger |
hashers |

  Exceptions
ChecksumError | Raised when a download does not contain the correct data.

  Classes
Etag | A hasher for s3 etags.

  Functions
compute_checksum_for_file(local_file_path[, algorithm]) |
compute_checksum_for_content(fh[, algorithm]) | Note: Chunk size matters for s3 etags, and must be the same to get the same hash from the same object.

  Module Contents

    toil.lib.checksum.logger

    exception toil.lib.checksum.ChecksumError
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Raised when a download does not contain the correct data.

    class toil.lib.checksum.Etag(chunk_size)
        A hasher for s3 etags.

        Parameters
            chunk_size (int </usr/share/doc/python3/html/library/functions.html#int>)

        etag_bytes: int </usr/share/doc/python3/html/library/functions.html#int> = 0

        etag_parts: List[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>] = []

        etag_hasher: hashlib._Hash

        chunk_size: int </usr/share/doc/python3/html/library/functions.html#int>

        update(chunk)

            Parameters
                chunk (bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>)

            Return type
                None

        hexdigest()

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.checksum.hashers

    toil.lib.checksum.compute_checksum_for_file(local_file_path, algorithm='sha1')

        Parameters

            •   local_file_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   algorithm (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.checksum.compute_checksum_for_content(fh, algorithm='sha1')
        Note: Chunk size matters for s3 etags, and must be the same to get the same hash from the same object. Therefore this buffer is not modifiable throughout Toil.

        Parameters

            •   fh (Union[BinaryIO, io.BytesIO </usr/share/doc/python3/html/library/io.html#io.BytesIO>])

            •   algorithm (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

  toil.lib.compatibility

  Functions
deprecated(new_function_name) |
compat_bytes(s) |
compat_bytes_recursive(data) | Convert a tree of objects over bytes to objects over strings.

  Module Contents

    toil.lib.compatibility.deprecated(new_function_name)

        Parameters
            new_function_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            Callable[Ellipsis, Any]

    toil.lib.compatibility.compat_bytes(s)

        Parameters
            s (Union[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.compatibility.compat_bytes_recursive(data)
        Convert a tree of objects over bytes to objects over strings.

        Parameters
            data (Any)

        Return type
            Any

  toil.lib.conversions

Conversion utilities for mapping memory, disk, core declarations from strings to numbers and vice versa. Also contains general conversion functions

  Attributes
KIB |
MIB |
GIB |
TIB |
PIB |
EIB |
KB |
MB |
GB |
TB |
PB |
EB |
BINARY_PREFIXES |
DECIMAL_PREFIXES |
VALID_PREFIXES |

  Functions
bytes_in_unit([unit]) |
convert_units(num, src_unit[, dst_unit]) | Returns a float representing the converted input in dst_units.
parse_memory_string(string) | Given a string representation of some memory (i.e. '1024 Mib'), return the
human2bytes(string) | Given a string representation of some memory (i.e. '1024 Mib'), return the
bytes2human(n) | Return a binary value as a human readable string with units.
b_to_mib(n) | Convert a number from bytes to mibibytes.
mib_to_b(n) | Convert a number from mibibytes to bytes.
hms_duration_to_seconds(hms) | Parses a given time string in hours:minutes:seconds,
strtobool(val) | Make a human-readable string into a bool.
opt_strtobool(b) | Convert an optional string representation of bool to None or bool
modify_url(url, remove) | Given a valid URL string, split out the params, remove any offending

  Module Contents

    toil.lib.conversions.KIB = 1024

    toil.lib.conversions.MIB = 1048576

    toil.lib.conversions.GIB = 1073741824

    toil.lib.conversions.TIB = 1099511627776

    toil.lib.conversions.PIB = 1125899906842624

    toil.lib.conversions.EIB = 1152921504606846976

    toil.lib.conversions.KB = 1000

    toil.lib.conversions.MB = 1000000

    toil.lib.conversions.GB = 1000000000

    toil.lib.conversions.TB = 1000000000000

    toil.lib.conversions.PB = 1000000000000000

    toil.lib.conversions.EB = 1000000000000000000

    toil.lib.conversions.BINARY_PREFIXES = ['ki', 'mi', 'gi', 'ti', 'pi', 'ei', 'kib', 'mib', 'gib', 'tib', 'pib', 'eib']

    toil.lib.conversions.DECIMAL_PREFIXES = ['b', 'k', 'm', 'g', 't', 'p', 'e', 'kb', 'mb', 'gb', 'tb', 'pb', 'eb']

    toil.lib.conversions.VALID_PREFIXES = ['ki', 'mi', 'gi', 'ti', 'pi', 'ei', 'kib', 'mib', 'gib', 'tib', 'pib', 'eib', 'b', 'k', 'm',...

    toil.lib.conversions.bytes_in_unit(unit='B')

        Parameters
            unit (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            int </usr/share/doc/python3/html/library/functions.html#int>

    toil.lib.conversions.convert_units(num, src_unit, dst_unit='B')
        Returns a float representing the converted input in dst_units.

        Parameters

            •   num (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   src_unit (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   dst_unit (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            float </usr/share/doc/python3/html/library/functions.html#float>

    toil.lib.conversions.parse_memory_string(string)
        Given a string representation of some memory (i.e. '1024 Mib'), return the number and unit.

        Parameters
            string (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[float </usr/share/doc/python3/html/library/functions.html#float>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.lib.conversions.human2bytes(string)
        Given a string representation of some memory (i.e. '1024 Mib'), return the integer number of bytes.

        Parameters
            string (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            int </usr/share/doc/python3/html/library/functions.html#int>

    toil.lib.conversions.bytes2human(n)
        Return a binary value as a human readable string with units.

        Parameters
            n (SupportsInt)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.conversions.b_to_mib(n)
        Convert a number from bytes to mibibytes.

        Parameters
            n (Union[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>])

        Return type
            float </usr/share/doc/python3/html/library/functions.html#float>

    toil.lib.conversions.mib_to_b(n)
        Convert a number from mibibytes to bytes.

        Parameters
            n (Union[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>])

        Return type
            float </usr/share/doc/python3/html/library/functions.html#float>

    toil.lib.conversions.hms_duration_to_seconds(hms)
        Parses a given time string in hours:minutes:seconds, returns an equivalent total seconds value

        Parameters
            hms (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            float </usr/share/doc/python3/html/library/functions.html#float>

    toil.lib.conversions.strtobool(val)
        Make a human-readable string into a bool.

        Convert a string along the lines of "y", "1", "ON", "TrUe", or "Yes" to True, and the corresponding false-ish values to False.

        Parameters
            val (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.conversions.opt_strtobool(b)
        Convert an optional string representation of bool to None or bool

        Parameters
            b (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>]

    toil.lib.conversions.modify_url(url, remove)
        Given a valid URL string, split out the params, remove any offending params in 'remove', and return the cleaned URL.

        Parameters

            •   url (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   remove (List[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

  toil.lib.directory

  Attributes
TOIL_DIR_URI_SCHEME |
DirectoryContents |

  Functions
check_directory_dict_invariants(contents) | Make sure a directory structure dict makes sense. Throws an error
decode_directory(dir_path) | Decode a directory from a "toildir:" path to a directory (or a file in it).
encode_directory(contents[, name, source]) | Encode a directory from a "toildir:" path to a directory (or a file in it).
directory_item_exists(dir_path) | Checks that a URL to a Toil directory or thing in it actually exists.
get_directory_contents_item(contents, remaining_path) | Get a subdirectory or file from a decoded directory and remaining path.
get_directory_item(dir_path) | Get a subdirectory or file from a URL pointing to or into a toildir: directory.
directory_contents_items(contents) | Yield each file or directory under the given contents, including itself.
directory_items(dir_path) | Yield each file or directory under the given path, including itself.

  Module Contents

    toil.lib.directory.TOIL_DIR_URI_SCHEME = 'toildir:'

    toil.lib.directory.DirectoryContents

    toil.lib.directory.check_directory_dict_invariants(contents)
        Make sure a directory structure dict makes sense. Throws an error otherwise.

        Currently just checks to make sure no empty-string keys exist.

        Parameters
            contents (DirectoryContents)

        Return type
            None

    toil.lib.directory.decode_directory(dir_path)
        Decode a directory from a "toildir:" path to a directory (or a file in it).

        Returns
            the decoded directory dict, the remaining part of the path (which may be None), an identifier string for the directory (which is the stored name URI if one was provided), and the name URI and source task info.

        Parameters
            dir_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[DirectoryContents, Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>], str </usr/share/doc/python3/html/library/stdtypes.html#str>, Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>], Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

    toil.lib.directory.encode_directory(contents, name=None, source=None)
        Encode a directory from a "toildir:" path to a directory (or a file in it).

        Parameters

            •   contents (DirectoryContents) -- the directory dict, which is a dict from name to URI for a file or dict for a subdirectory.

            •   name (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- the path or URI the directory belongs at, including its basename. May not be empty if set.

            •   source (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- the name of a workflow component that uploaded the directory. May not be empty if set.

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.directory.directory_item_exists(dir_path)
        Checks that a URL to a Toil directory or thing in it actually exists.

        Assumes that all the pointed-to URLs exist; just checks tha tthe thing is actually in the encoded directory structure.

        Parameters
            dir_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.directory.get_directory_contents_item(contents, remaining_path)
        Get a subdirectory or file from a decoded directory and remaining path.

        Parameters

            •   contents (DirectoryContents)

            •   remaining_path (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            Union[DirectoryContents, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.lib.directory.get_directory_item(dir_path)
        Get a subdirectory or file from a URL pointing to or into a toildir: directory.

        Parameters
            dir_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            Union[DirectoryContents, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.lib.directory.directory_contents_items(contents)
        Yield each file or directory under the given contents, including itself.

        Yields parent items before children.

        Yields each item as a str path from the root (possibly empty), and either a
            str value for files or a None for directories.

    The path won't have trailing slashes.

        Parameters
            contents (DirectoryContents)

        Return type
            Iterator[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Union[str </usr/share/doc/python3/html/library/stdtypes.html#str>, None]]]

    toil.lib.directory.directory_items(dir_path)
        Yield each file or directory under the given path, including itself.

        Yields parent items before children.

        Yields each item as a str path from the root (possibly empty), and either a
            str value for files or a None for directories.

    The path won't have trailing slashes.

        Parameters
            dir_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            Iterator[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Union[str </usr/share/doc/python3/html/library/stdtypes.html#str>, None]]]

  toil.lib.docker

  Attributes
logger |
FORGO |
STOP |
RM |

  Functions
dockerCheckOutput(*args, **kwargs) |
dockerCall(*args, **kwargs) |
subprocessDockerCall(*args, **kwargs) |
apiDockerCall(…) | A toil wrapper for the python docker API.
dockerKill(container_name[, gentleKill, remove, timeout]) | Immediately kills a container. Equivalent to "docker kill":
dockerStop(container_name[, remove]) | Gracefully kills a container. Equivalent to "docker stop":
containerIsRunning(container_name[, timeout]) | Checks whether the container is running or not.
getContainerName(job) | Create a random string including the job name, and return it. Name will

  Module Contents

    toil.lib.docker.logger

    toil.lib.docker.FORGO = 0

    toil.lib.docker.STOP = 1

    toil.lib.docker.RM = 2

    toil.lib.docker.dockerCheckOutput(*args, **kwargs)

        Parameters

            •   args (Any)

            •   kwargs (Any)

        Return type
            NoReturn

    toil.lib.docker.dockerCall(*args, **kwargs)

        Parameters

            •   args (Any)

            •   kwargs (Any)

        Return type
            NoReturn

    toil.lib.docker.subprocessDockerCall(*args, **kwargs)

        Parameters

            •   args (Any)

            •   kwargs (Any)

        Return type
            NoReturn

    toil.lib.docker.apiDockerCall(job: toil.job.Job <#toil.job.Job>, image: str </usr/share/doc/python3/html/library/stdtypes.html#str>, parameters: list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>] | list </usr/share/doc/python3/html/library/stdtypes.html#list>[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]] | None </usr/share/doc/python3/html/library/constants.html#None> = None, deferParam: int </usr/share/doc/python3/html/library/functions.html#int> | None </usr/share/doc/python3/html/library/constants.html#None> = None, volumes: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]] | None </usr/share/doc/python3/html/library/constants.html#None> = None, working_dir: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, containerName: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, entrypoint: str </usr/share/doc/python3/html/library/stdtypes.html#str> | list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, detach: Literal[False] = False, log_config: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, auto_remove: bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None> = None, remove: bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None> = False, user: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, environment: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, stdout: bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None> = None, stderr: bool </usr/share/doc/python3/html/library/functions.html#bool> = False, stream: bool </usr/share/doc/python3/html/library/functions.html#bool> = False, demux: Literal[False] = False, streamfile: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, accelerators: list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, timeout: int </usr/share/doc/python3/html/library/functions.html#int> = 365 * 24 * 60 * 60, **kwargs: Any) -> str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.docker.apiDockerCall(job: toil.job.Job <#toil.job.Job>, image: str </usr/share/doc/python3/html/library/stdtypes.html#str>, parameters: list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>] | list </usr/share/doc/python3/html/library/stdtypes.html#list>[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]] | None </usr/share/doc/python3/html/library/constants.html#None> = None, deferParam: int </usr/share/doc/python3/html/library/functions.html#int> | None </usr/share/doc/python3/html/library/constants.html#None> = None, volumes: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]] | None </usr/share/doc/python3/html/library/constants.html#None> = None, working_dir: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, containerName: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, entrypoint: str </usr/share/doc/python3/html/library/stdtypes.html#str> | list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, detach: Literal[False] = False, log_config: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, auto_remove: bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None> = None, remove: bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None> = False, user: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, environment: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, stdout: bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None> = None, stderr: bool </usr/share/doc/python3/html/library/functions.html#bool> = False, stream: Literal[True] = True, demux: Literal[True] = True, streamfile: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, accelerators: list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, timeout: int </usr/share/doc/python3/html/library/functions.html#int> = 365 * 24 * 60 * 60, **kwargs: Any) -> collections.abc.Generator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

    toil.lib.docker.apiDockerCall(job: toil.job.Job <#toil.job.Job>, image: str </usr/share/doc/python3/html/library/stdtypes.html#str>, parameters: list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>] | list </usr/share/doc/python3/html/library/stdtypes.html#list>[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]] | None </usr/share/doc/python3/html/library/constants.html#None> = None, deferParam: int </usr/share/doc/python3/html/library/functions.html#int> | None </usr/share/doc/python3/html/library/constants.html#None> = None, volumes: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]] | None </usr/share/doc/python3/html/library/constants.html#None> = None, working_dir: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, containerName: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, entrypoint: str </usr/share/doc/python3/html/library/stdtypes.html#str> | list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, detach: Literal[False] = False, log_config: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, auto_remove: bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None> = None, remove: bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None> = False, user: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, environment: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, stdout: bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None> = None, stderr: bool </usr/share/doc/python3/html/library/functions.html#bool> = False, stream: Literal[False] = False, demux: Literal[True] = True, streamfile: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, accelerators: list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, timeout: int </usr/share/doc/python3/html/library/functions.html#int> = 365 * 24 * 60 * 60, **kwargs: Any) -> tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.lib.docker.apiDockerCall(job: toil.job.Job <#toil.job.Job>, image: str </usr/share/doc/python3/html/library/stdtypes.html#str>, parameters: list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>] | list </usr/share/doc/python3/html/library/stdtypes.html#list>[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]] | None </usr/share/doc/python3/html/library/constants.html#None> = None, deferParam: int </usr/share/doc/python3/html/library/functions.html#int> | None </usr/share/doc/python3/html/library/constants.html#None> = None, volumes: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]] | None </usr/share/doc/python3/html/library/constants.html#None> = None, working_dir: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, containerName: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, entrypoint: str </usr/share/doc/python3/html/library/stdtypes.html#str> | list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, detach: Literal[True] = True, log_config: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, auto_remove: bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None> = None, remove: bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None> = False, user: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, environment: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, stdout: bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None> = None, stderr: bool </usr/share/doc/python3/html/library/functions.html#bool> = False, stream: bool </usr/share/doc/python3/html/library/functions.html#bool> = False, demux: bool </usr/share/doc/python3/html/library/functions.html#bool> = False, streamfile: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, accelerators: list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, timeout: int </usr/share/doc/python3/html/library/functions.html#int> = 365 * 24 * 60 * 60, **kwargs: Any) -> docker.models.containers.Container

    toil.lib.docker.apiDockerCall(job: toil.job.Job <#toil.job.Job>, image: str </usr/share/doc/python3/html/library/stdtypes.html#str>, parameters: list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>] | list </usr/share/doc/python3/html/library/stdtypes.html#list>[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]] | None </usr/share/doc/python3/html/library/constants.html#None> = None, deferParam: int </usr/share/doc/python3/html/library/functions.html#int> | None </usr/share/doc/python3/html/library/constants.html#None> = None, volumes: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]] | None </usr/share/doc/python3/html/library/constants.html#None> = None, working_dir: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, containerName: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, entrypoint: str </usr/share/doc/python3/html/library/stdtypes.html#str> | list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, detach: bool </usr/share/doc/python3/html/library/functions.html#bool> = False, log_config: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, auto_remove: bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None> = None, remove: bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None> = False, user: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, environment: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, stdout: bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None> = None, stderr: bool </usr/share/doc/python3/html/library/functions.html#bool> = False, stream: bool </usr/share/doc/python3/html/library/functions.html#bool> = False, demux: bool </usr/share/doc/python3/html/library/functions.html#bool> = False, streamfile: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None, accelerators: list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>] | None </usr/share/doc/python3/html/library/constants.html#None> = None, timeout: int </usr/share/doc/python3/html/library/functions.html#int> = 365 * 24 * 60 * 60, **kwargs: Any) -> str </usr/share/doc/python3/html/library/stdtypes.html#str> | docker.models.containers.Container | collections.abc.Generator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]] | collections.abc.Generator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator>[str </usr/share/doc/python3/html/library/stdtypes.html#str>] | tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]
        A toil wrapper for the python docker API.

        Docker API Docs: <https://docker-py.readthedocs.io/en/stable/index.html> Docker API Code: <https://github.com/docker/docker-py>

        This implements docker's python API within toil so that calls are run as jobs, with the intention that failed/orphaned docker jobs be handled appropriately.

        Example of using dockerCall in toil to index a FASTA file with SAMtools:

                def toil_job(job):
                    working_dir = job.fileStore.getLocalTempDir()
                    path = job.fileStore.readGlobalFile(ref_id,
                                                      os.path.join(working_dir, 'ref.fasta')
                    parameters = ['faidx', path]
                    apiDockerCall(job,
                                  image='quay.io/ucgc_cgl/samtools:latest',
                                  working_dir=working_dir,
                                  parameters=parameters)

    Note that when run with detach=False, or with detach=True and stdout=True or stderr=True, this is a blocking call. When run with detach=True and without output capture, the container is started and returned without waiting for it to finish.

        Parameters

            •   job -- The Job instance for the calling function.

            •   image -- Name of the Docker image to be used. (e.g. 'quay.io/ucsc_cgl/samtools:latest')

            •   parameters -- A list of string elements. If there are multiple elements, these will be joined with spaces. This handling of multiple elements provides backwards compatibility with previous versions which called docker using subprocess.check_call(). If list of lists: list[list[str]], then treat as successive commands chained with pipe.

            •   deferParam -- Action to take on the container upon job completion. FORGO (0) leaves the container untouched and running. STOP (1) Sends SIGTERM, then SIGKILL if necessary to the container. RM (2) Immediately send SIGKILL to the container. This is the default behavior if deferParam is set to None.

            •   volumes -- A dictionary of volume locations to volume options.

            •   working_dir -- The working directory.

            •   containerName -- The name/ID of the container.

            •   entrypoint -- Prepends commands sent to the container. See: <https://docker-py.readthedocs.io/en/stable/containers.html>

            •   detach -- Run the container in detached mode. (equivalent to '-d')

            •   log_config -- Specify the logs to return from the container. See: <https://docker-py.readthedocs.io/en/stable/containers.html>

            •   remove -- Remove the container on exit or not.

            •   user -- The container will be run with the privileges of the user specified. Can be an actual name, such as 'root' or 'lifeisaboutfishtacos', or it can be the uid or gid of the user ('0' is root; '1000' is an example of a less privileged uid or gid), or a complement of the uid:gid (RECOMMENDED), such as '0:0' (root user : root group) or '1000:1000' (some other user : some other user group).

            •   environment -- Allows one to set environment variables inside of the container.

            •   stdout -- Return logs from STDOUT when detach=False (default: True). Block and capture stdout to a file when detach=True (default: False). Output capture defaults to output.log, and can be specified with the "streamfile" kwarg.

            •   stderr -- Return logs from STDERR when detach=False (default: False). Block and capture stderr to a file when detach=True (default: False). Output capture defaults to output.log, and can be specified with the "streamfile" kwarg.

            •   stream -- If True and detach=False, return a log generator instead of a string. Ignored if detach=True. (default: False).

            •   demux -- Similar to demux in container.exec_run(). If True and detach=False, returns a tuple of (stdout, stderr). If stream=True, returns a log generator with tuples of (stdout, stderr). Ignored if detach=True. (default: False).

            •   streamfile -- Collect container output to this file if detach=True and stderr and/or stdout are True. Defaults to "output.log".

            •   accelerators -- Toil accelerator numbers (usually GPUs) to forward to the container. These are interpreted in the current Python process's environment. See toil.lib.accelerators.get_individual_local_accelerators() for the menu of available accelerators.

            •   timeout -- Use the given timeout in seconds for interactions with the Docker daemon. Note that the underlying docker module is not always able to abort ongoing reads and writes in order to respect the timeout. Defaults to 1 year (i.e. wait essentially indefinitely).

            •   kwargs -- Additional keyword arguments supplied to the docker API's run command. The list is 75 keywords total, for examples and full documentation see: <https://docker-py.readthedocs.io/en/stable/containers.html>

        Returns
            Returns the standard output/standard error text, as requested, when detach=False. Returns the underlying docker.models.containers.Container object from the Docker API when detach=True.

    toil.lib.docker.dockerKill(container_name, gentleKill=False, remove=False, timeout=365 * 24 * 60 * 60)
        Immediately kills a container. Equivalent to "docker kill": <https://docs.docker.com/engine/reference/commandline/kill/>

        Parameters

            •   container_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Name of the container being killed.

            •   gentleKill (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If True, trigger a graceful shutdown.

            •   remove (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If True, remove the container after it exits.

            •   timeout (int </usr/share/doc/python3/html/library/functions.html#int>) -- Use the given timeout in seconds for interactions with the Docker daemon. Note that the underlying docker module is not always able to abort ongoing reads and writes in order to respect the timeout. Defaults to 1 year (i.e. wait essentially indefinitely).

        Return type
            None

    toil.lib.docker.dockerStop(container_name, remove=False)
        Gracefully kills a container. Equivalent to "docker stop": <https://docs.docker.com/engine/reference/commandline/stop/>

        Parameters

            •   container_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Name of the container being stopped.

            •   remove (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If True, remove the container after it exits.

        Return type
            None

    toil.lib.docker.containerIsRunning(container_name, timeout=365 * 24 * 60 * 60)
        Checks whether the container is running or not.

        Parameters

            •   container_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Name of the container being checked.

            •   timeout (int </usr/share/doc/python3/html/library/functions.html#int>) -- Use the given timeout in seconds for interactions with the Docker daemon. Note that the underlying docker module is not always able to abort ongoing reads and writes in order to respect the timeout. Defaults to 1 year (i.e. wait essentially indefinitely).

        Returns
            True if status is 'running', False if status is anything else, and None if the container does not exist.

        Return type
            Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>]

    toil.lib.docker.getContainerName(job)
        Create a random string including the job name, and return it. Name will match [a-zA-Z0-9][a-zA-Z0-9_.-].

        Parameters
            job (toil.job.Job <#toil.job.Job>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

  toil.lib.dockstore

Contains functions for integrating Toil with UCSC Dockstore, for reporting metrics.

For basic TRS functionality for fetching workflows, see trs.py.

  Attributes
logger |
DEFAULT_DOCKSTORE_TOKEN |
DOCKSTORE_TOKEN |
DOCKSTORE_PLATFORM |
ExecutionStatus |

  Classes
Cost | Representation of the cost of running something.
RunExecution | Dockstore metrics data for a workflow or task run.
TaskExecutions | Dockstore metrics data for all the tasks in a workflow.

  Functions
ensure_valid_id(execution_id) | Make sure the given execution ID is in Dockstore format and will be accepted by Dockstore.
pack_workflow_metrics(execution_id, start_time, ...[, ...]) | Pack up per-workflow metrics into a format that can be submitted to Dockstore.
pack_single_task_metrics(execution_id, start_time, ...) | Pack up metrics for a single task execution in a format that can be used in a Dockstore submission.
pack_workflow_task_set_metrics(execution_id, ...) | Pack up metrics for all the tasks in a workflow execution into a format that can be submitted to Dockstore.
send_metrics(trs_workflow_id, trs_version, ...) | Send packed workflow and/or task metrics to Dockstore.
get_metrics_url(trs_workflow_id, trs_version, execution_id) | Get the URL where a workflow metrics object (for a workflow, or for a set of tasks) can be fetched back from.

  Module Contents

    toil.lib.dockstore.logger

    toil.lib.dockstore.DEFAULT_DOCKSTORE_TOKEN = '2bff46294daddef6df185452b04db6143ea8a59f52ee3c325d3e1df418511b7d'

    toil.lib.dockstore.DOCKSTORE_TOKEN

    toil.lib.dockstore.DOCKSTORE_PLATFORM = 'TOIL'

    toil.lib.dockstore.ExecutionStatus

    class toil.lib.dockstore.Cost
        Bases: TypedDict

        Representation of the cost of running something.

        value: float </usr/share/doc/python3/html/library/functions.html#float>
            Cost in US Dollars.

    class toil.lib.dockstore.RunExecution
        Bases: TypedDict

        Dockstore metrics data for a workflow or task run.

        executionId: str </usr/share/doc/python3/html/library/stdtypes.html#str>
            Executor-generated unique execution ID for this workflow or task.

        dateExecuted: str </usr/share/doc/python3/html/library/stdtypes.html#str>
            ISO 8601 UTC timestamp when the execution happened.

        executionStatus: ExecutionStatus
            Did the execution work?

        executionTime: typing_extensions.NotRequired[str </usr/share/doc/python3/html/library/stdtypes.html#str>]
            Total time of the run in ISO 8601 duration format.

        memoryRequirementsGB: typing_extensions.NotRequired[float </usr/share/doc/python3/html/library/functions.html#float>]
            Memory required for the execution in gigabytes (not GiB).

        cpuRequirements: typing_extensions.NotRequired[int </usr/share/doc/python3/html/library/functions.html#int>]
            Number of CPUs required.

        cost: typing_extensions.NotRequired[Cost <#toil.lib.dockstore.Cost>]
            How much the execution cost to run.

        region: typing_extensions.NotRequired[str </usr/share/doc/python3/html/library/stdtypes.html#str>]
            The (cloud) region the workflow was executed in.

        additionalProperties: typing_extensions.NotRequired[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]]
            Any additional properties to send.

            Dockstore can take any JSON-able structured data.

    class toil.lib.dockstore.TaskExecutions
        Bases: TypedDict

        Dockstore metrics data for all the tasks in a workflow.

        executionId: str </usr/share/doc/python3/html/library/stdtypes.html#str>
            Executor-generated unique execution ID.

        dateExecuted: str </usr/share/doc/python3/html/library/stdtypes.html#str>
            ISO 8601 UTC timestamp when the execution happened.

        taskExecutions: list </usr/share/doc/python3/html/library/stdtypes.html#list>[RunExecution <#toil.lib.dockstore.RunExecution>]
            Individual executions of each task in the workflow.

        additionalProperties: typing_extensions.NotRequired[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]]
            Any additional properties to send.

            Dockstore can take any JSON-able structured data.

    toil.lib.dockstore.ensure_valid_id(execution_id)
        Make sure the given execution ID is in Dockstore format and will be accepted by Dockstore.

        Must be alphanumeric (with internal underscores allowed) and <100 characters long.

        Raises
            ValueError </usr/share/doc/python3/html/library/exceptions.html#ValueError> -- if the ID is not in the right format

        Parameters
            execution_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            None

    toil.lib.dockstore.pack_workflow_metrics(execution_id, start_time, runtime, succeeded, job_store_type=None, batch_system=None, caching=None, toil_version=None, python_version=None, platform_system=None, platform_machine=None)
        Pack up per-workflow metrics into a format that can be submitted to Dockstore.

        Parameters

            •   execution_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Unique ID for the workflow execution. Must be in Dockstore format.

            •   start_time (float </usr/share/doc/python3/html/library/functions.html#float>) -- Execution start time in seconds since the Unix epoch.

            •   rutime -- Execution duration in seconds.

            •   jobstore_type -- Kind of job store used, like "file" or "aws".

            •   batch_system (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Python class name implementing the batch system used.

            •   caching (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>]) -- Whether Toil filestore-level caching was used.

            •   toil_version (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Version of Toil used (without any Git hash).

            •   python_version (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Version of Python used.

            •   platform_system (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Operating system type (like "Darwin" or "Linux").

            •   platform_machine (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Machine architecture of the leader (like "AMD64").

            •   runtime (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   succeeded (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            •   job_store_type (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            RunExecution <#toil.lib.dockstore.RunExecution>

    toil.lib.dockstore.pack_single_task_metrics(execution_id, start_time, runtime, succeeded, job_name=None, cores=None, cpu_seconds=None, memory_bytes=None, disk_bytes=None)
        Pack up metrics for a single task execution in a format that can be used in a Dockstore submission.

        Parameters

            •   execution_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Unique ID for the workflow execution. Must be in Dockstore format.

            •   start_time (float </usr/share/doc/python3/html/library/functions.html#float>) -- Execution start time in seconds since the Unix epoch.

            •   rutime -- Execution duration in seconds.

            •   succeeded (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Whether the execution succeeded.

            •   job_name (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Name of the job within the workflow.

            •   cores (Optional[float </usr/share/doc/python3/html/library/functions.html#float>]) -- CPU cores allocated to the job.

            •   cpu_seconds (Optional[float </usr/share/doc/python3/html/library/functions.html#float>]) -- CPU seconds consumed by the job.

            •   memory_bytes (Optional[int </usr/share/doc/python3/html/library/functions.html#int>]) -- Memory consumed by the job in bytes.

            •   disk_bytes (Optional[int </usr/share/doc/python3/html/library/functions.html#int>]) -- Disk space consumed by the job in bytes.

            •   runtime (float </usr/share/doc/python3/html/library/functions.html#float>)

        Return type
            RunExecution <#toil.lib.dockstore.RunExecution>

    toil.lib.dockstore.pack_workflow_task_set_metrics(execution_id, start_time, tasks)
        Pack up metrics for all the tasks in a workflow execution into a format that can be submitted to Dockstore.

        Parameters

            •   execution_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Unique ID for the workflow execution. Must be in Dockstore format.

            •   start_time (float </usr/share/doc/python3/html/library/functions.html#float>) -- Execution start time for the overall workflow execution in seconds since the Unix epoch.

            •   tasks (list </usr/share/doc/python3/html/library/stdtypes.html#list>[RunExecution <#toil.lib.dockstore.RunExecution>]) -- Packed tasks from pack_single_task_metrics()

        Return type
            TaskExecutions <#toil.lib.dockstore.TaskExecutions>

    toil.lib.dockstore.send_metrics(trs_workflow_id, trs_version, workflow_runs, workflow_task_sets)
        Send packed workflow and/or task metrics to Dockstore.

        Parameters

            •   workflow_runs (list </usr/share/doc/python3/html/library/stdtypes.html#list>[RunExecution <#toil.lib.dockstore.RunExecution>]) -- list of packed metrics objects for each workflow.

            •   workflow_task_sets (list </usr/share/doc/python3/html/library/stdtypes.html#list>[TaskExecutions <#toil.lib.dockstore.TaskExecutions>]) -- list of packed metrics objects for the tasks in each workflow. Each workflow should have one entry containing all its tasks. Does not have to be the same order/set of workflows as workflow_runs.

            •   trs_workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   trs_version (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Raises
            requests.HTTPError -- if Dockstore does not accept the metrics.

        Return type
            None

    toil.lib.dockstore.get_metrics_url(trs_workflow_id, trs_version, execution_id)
        Get the URL where a workflow metrics object (for a workflow, or for a set of tasks) can be fetched back from.

        Parameters

            •   trs_workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   trs_version (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   execution_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

  toil.lib.ec2

  Attributes
a_short_time |
a_long_time |
logger |
INCONSISTENCY_ERRORS |
iam_client |

  Exceptions
UserError | Unspecified run-time error.
UnexpectedResourceState | Common base class for all non-exit exceptions.

  Functions
is_base64(value) | Return True if value is base64-decodeable, and False otherwise.
not_found(e) |
inconsistencies_detected(e) |
retry_ec2([t, retry_for, retry_while]) |
wait_transition(boto3_ec2, resource, from_states, to_state) | Wait until the specified EC2 resource (instance, image, volume, ...) transitions from any
wait_instances_running(boto3_ec2, instances) | Wait until no instance in the given iterable is 'pending'. Yield every instance that
wait_spot_requests_active(boto3_ec2, requests[, ...]) | Wait until no spot request in the given iterator is in the 'open' state or, optionally,
create_spot_instances(boto3_ec2, price, image_id, spec) | Create instances on the spot market.
create_ondemand_instances(boto3_ec2, image_id, spec[, ...]) | Requests the RunInstances EC2 API call but accounts for the race between recently created
increase_instance_hop_limit(boto3_ec2, boto_instance_list) | Increase the default HTTP hop limit, as we are running Toil and Kubernetes inside a Docker container, so the default
prune(bushy) | Prune entries in the given dict with false-y values.
wait_until_instance_profile_arn_exists(...) |
create_instances(ec2_resource, image_id, key_name, ...) | Replaces create_ondemand_instances. Uses boto3 and returns a list of Boto3 instance dicts.
create_launch_template(ec2_client, template_name, ...) | Creates a launch template with the given name for launching instances with the given parameters.
create_auto_scaling_group(autoscaling_client, ...[, ...]) | Create a new Auto Scaling Group with the given name (which is also its

  Module Contents

    toil.lib.ec2.a_short_time = 5

    toil.lib.ec2.a_long_time = 3600

    toil.lib.ec2.logger

    toil.lib.ec2.is_base64(value)
        Return True if value is base64-decodeable, and False otherwise.

        Parameters
            value (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    exception toil.lib.ec2.UserError(message=None, cause=None)
        Bases: RuntimeError </usr/share/doc/python3/html/library/exceptions.html#RuntimeError>

        Unspecified run-time error.

    toil.lib.ec2.not_found(e)

    toil.lib.ec2.inconsistencies_detected(e)

    toil.lib.ec2.INCONSISTENCY_ERRORS

    toil.lib.ec2.retry_ec2(t=a_short_time, retry_for=10 * a_short_time, retry_while=not_found)

    exception toil.lib.ec2.UnexpectedResourceState(resource, to_state, state)
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Common base class for all non-exit exceptions.

    toil.lib.ec2.wait_transition(boto3_ec2, resource, from_states, to_state, state_getter=lambda x: ...)
        Wait until the specified EC2 resource (instance, image, volume, ...) transitions from any of the given 'from' states to the specified 'to' state. If the instance is found in a state other that the to state or any of the from states, an exception will be thrown.

        Parameters

            •   resource (mypy_boto3_ec2.type_defs.InstanceTypeDef) -- the resource to monitor

            •   from_states (collections.abc.Iterable </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterable>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- a set of states that the resource is expected to be in before the transition occurs

            •   to_state (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the state of the resource when this method returns

            •   boto3_ec2 (mypy_boto3_ec2.client.EC2Client)

            •   state_getter (Callable[[mypy_boto3_ec2.type_defs.InstanceTypeDef], str </usr/share/doc/python3/html/library/stdtypes.html#str>])

    toil.lib.ec2.wait_instances_running(boto3_ec2, instances)
        Wait until no instance in the given iterable is 'pending'. Yield every instance that entered the running state as soon as it does.

        Parameters

            •   boto3_ec2 (mypy_boto3_ec2.client.EC2Client) -- the EC2 connection to use for making requests

            •   instances (collections.abc.Iterable </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterable>[mypy_boto3_ec2.type_defs.InstanceTypeDef]) -- the instances to wait on

        Return type
            collections.abc.Generator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator>[mypy_boto3_ec2.type_defs.InstanceTypeDef, None, None]

    toil.lib.ec2.wait_spot_requests_active(boto3_ec2, requests, timeout=None, tentative=False)
        Wait until no spot request in the given iterator is in the 'open' state or, optionally, a timeout occurs. Yield spot requests as soon as they leave the 'open' state.

        Parameters

            •   boto3_ec2 (mypy_boto3_ec2.client.EC2Client) -- ec2 client

            •   requests (collections.abc.Iterable </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterable>[mypy_boto3_ec2.type_defs.SpotInstanceRequestTypeDef]) -- The requests to wait on.

            •   timeout (float </usr/share/doc/python3/html/library/functions.html#float>) -- Maximum time in seconds to spend waiting or None to wait forever. If a timeout occurs, the remaining open requests will be cancelled.

            •   tentative (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- if True, give up on a spot request at the earliest indication of it not being fulfilled immediately

        Return type
            collections.abc.Iterable </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterable>[list </usr/share/doc/python3/html/library/stdtypes.html#list>[mypy_boto3_ec2.type_defs.SpotInstanceRequestTypeDef]]

    toil.lib.ec2.create_spot_instances(boto3_ec2, price, image_id, spec, num_instances=1, timeout=None, tentative=False, tags=None)
        Create instances on the spot market.

        The "UserData" field in "LaunchSpecification" in spec MUST ALREADY BE base64-encoded. It will NOT be automatically encoded.

        Parameters

            •   tags (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Dict from tag key to tag value of tags to apply to the request.

            •   boto3_ec2 (mypy_boto3_ec2.client.EC2Client)

            •   spec (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[Literal['LaunchSpecification'], dict[str, Any]])

        Return type
            collections.abc.Generator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator>[mypy_boto3_ec2.type_defs.DescribeInstancesResultTypeDef, None, None]

    toil.lib.ec2.create_ondemand_instances(boto3_ec2, image_id, spec, num_instances=1)
        Requests the RunInstances EC2 API call but accounts for the race between recently created instance profiles, IAM roles and an instance creation that refers to them.

        The "UserData" field in spec MUST NOT be base64 encoded; it will be base64-encoded by boto3 automatically. See <<https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2/client/run_instances.html>>.

        Replaced by create_instances.

        Parameters

            •   boto3_ec2 (mypy_boto3_ec2.client.EC2Client)

            •   image_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   spec (collections.abc.Mapping </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Mapping>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any])

            •   num_instances (int </usr/share/doc/python3/html/library/functions.html#int>)

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[mypy_boto3_ec2.type_defs.InstanceTypeDef]

    toil.lib.ec2.increase_instance_hop_limit(boto3_ec2, boto_instance_list)
        Increase the default HTTP hop limit, as we are running Toil and Kubernetes inside a Docker container, so the default hop limit of 1 will not be enough when grabbing metadata information with ec2_metadata

        Must be called after the instances are guaranteed to be running.

        Parameters

            •   boto_instance_list (list </usr/share/doc/python3/html/library/stdtypes.html#list>[mypy_boto3_ec2.type_defs.InstanceTypeDef]) -- List of boto instances to modify

            •   boto3_ec2 (mypy_boto3_ec2.client.EC2Client)

        Returns

        Return type
            None

    toil.lib.ec2.prune(bushy)
        Prune entries in the given dict with false-y values. Boto3 may not like None and instead wants no key.

        Parameters
            bushy (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>)

        Return type
            dict </usr/share/doc/python3/html/library/stdtypes.html#dict>

    toil.lib.ec2.iam_client

    toil.lib.ec2.wait_until_instance_profile_arn_exists(instance_profile_arn)

        Parameters
            instance_profile_arn (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

    toil.lib.ec2.create_instances(ec2_resource, image_id, key_name, instance_type, num_instances=1, security_group_ids=None, user_data=None, block_device_map=None, instance_profile_arn=None, placement_az=None, subnet_id=None, tags=None)
        Replaces create_ondemand_instances. Uses boto3 and returns a list of Boto3 instance dicts.

        See "create_instances" (returns a list of ec2.Instance objects):
            <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.ServiceResource.create_instances>

        Not to be confused with "run_instances" (same input args; returns a dictionary):
            <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Client.run_instances>

        Parameters

            •   user_data (Optional[Union[str </usr/share/doc/python3/html/library/stdtypes.html#str>, bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]]) -- non-base64-encoded user data to control instance startup.

            •   tags (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) -- if given, these tags are applied to the instances, and all volumes.

            •   ec2_resource (mypy_boto3_ec2.service_resource.EC2ServiceResource)

            •   image_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   key_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   instance_type (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   num_instances (int </usr/share/doc/python3/html/library/functions.html#int>)

            •   security_group_ids (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>])

            •   block_device_map (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>]])

            •   instance_profile_arn (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   placement_az (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   subnet_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[mypy_boto3_ec2.service_resource.Instance]

    toil.lib.ec2.create_launch_template(ec2_client, template_name, image_id, key_name, instance_type, security_group_ids=None, user_data=None, block_device_map=None, instance_profile_arn=None, placement_az=None, subnet_id=None, tags=None)
        Creates a launch template with the given name for launching instances with the given parameters.

        We only ever use the default version of any launch template.

        Internally calls <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html?highlight=create_launch_template#EC2.Client.create_launch_template>

        Parameters

            •   tags (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) -- Tags, if given, are applied to the template itself, all instances, and all volumes.

            •   user_data (Optional[Union[str </usr/share/doc/python3/html/library/stdtypes.html#str>, bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]]) -- non-base64-encoded user data to pass to the instances.

            •   ec2_client (mypy_boto3_ec2.client.EC2Client)

            •   template_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   image_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   key_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   instance_type (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   security_group_ids (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>])

            •   block_device_map (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>]])

            •   instance_profile_arn (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   placement_az (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   subnet_id (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Returns
            the ID of the launch template.

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.ec2.create_auto_scaling_group(autoscaling_client, asg_name, launch_template_ids, vpc_subnets, min_size, max_size, instance_types=None, spot_bid=None, spot_cheapest=False, tags=None)
        Create a new Auto Scaling Group with the given name (which is also its unique identifier).

        Parameters

            •   autoscaling_client (mypy_boto3_autoscaling.client.AutoScalingClient) -- Boto3 client for autoscaling.

            •   asg_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Unique name for the autoscaling group.

            •   launch_template_ids (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- ID of the launch template to make instances from, for each instance type.

            •   vpc_subnets (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- One or more subnet IDs to place instances in the group into. Determine the availability zone(s) instances will launch into.

            •   min_size (int </usr/share/doc/python3/html/library/functions.html#int>) -- Minimum number of instances to have in the group at all times.

            •   max_size (int </usr/share/doc/python3/html/library/functions.html#int>) -- Maximum number of instances to allow in the group at any time.

            •   instance_types (Optional[collections.abc.Iterable </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterable>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) -- Use a pool over the given instance types, instead of the type given in the launch template. For on-demand groups, this is a prioritized list. For spot groups, we let AWS balance according to spot_strategy. Must be 20 types or shorter.

            •   spot_bid (Optional[float </usr/share/doc/python3/html/library/functions.html#float>]) -- If set, the ASG will be a spot market ASG. Bid is in dollars per instance hour. All instance types in the group are bid on equivalently.

            •   spot_cheapest (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If true, use the cheapest spot instances available out of instance_types, instead of the spot instances that minimize eviction probability.

            •   tags (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) -- Tags to apply to the ASG only. Tags for the instances should be added to the launch template instead.

        Return type
            None

    The default version of the launch template is used.

  toil.lib.ec2nodes

  Attributes
logger |
manager |
dirname |
region_json_dirname |
EC2Regions |

  Classes
InstanceType |

  Functions
is_number(s) | Determines if a unicode string (that may include commas) is a number.
parse_storage(storage_info) | Parses EC2 JSON storage param string into a number.
parse_memory(mem_info) | Returns EC2 'memory' string as a float.
download_region_json(filename[, region]) | Downloads and writes the AWS Billing JSON to a file using the AWS pricing API.
reduce_region_json_size(filename) | Deletes information in the json file that we don't need, and rewrites it. This makes the file smaller.
updateStaticEC2Instances() | Generates a new python file of fetchable EC2 Instances by region with current prices and specs.

  Module Contents

    toil.lib.ec2nodes.logger

    toil.lib.ec2nodes.manager

    toil.lib.ec2nodes.dirname

    toil.lib.ec2nodes.region_json_dirname

    toil.lib.ec2nodes.EC2Regions

    class toil.lib.ec2nodes.InstanceType(name, cores, memory, disks, disk_capacity, architecture)

        Parameters

            •   name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   cores (int </usr/share/doc/python3/html/library/functions.html#int>)

            •   memory (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   disks (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   disk_capacity (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   architecture (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        __slots__ = ('name', 'cores', 'memory', 'disks', 'disk_capacity', 'architecture')

        name

        cores

        memory

        disks

        disk_capacity

        architecture

        __str__()

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        __eq__(other)

            Parameters
                other (object </usr/share/doc/python3/html/library/functions.html#object>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.ec2nodes.is_number(s)
        Determines if a unicode string (that may include commas) is a number.

        Parameters
            s (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Any unicode string.

        Returns
            True if s represents a number, False otherwise.

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.ec2nodes.parse_storage(storage_info)
        Parses EC2 JSON storage param string into a number.

        Examples:
            "2 x 160 SSD" "3 x 2000 HDD" "EBS only" "1 x 410" "8 x 1.9 NVMe SSD" "900 GB NVMe SSD"

        Parameters
            storage_info (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- EC2 JSON storage param string.

        Returns
            Two floats representing: (# of disks), and (disk_capacity in GiB of each disk).

        Return type
            Union[list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>], tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[Union[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>], float </usr/share/doc/python3/html/library/functions.html#float>]]

    toil.lib.ec2nodes.parse_memory(mem_info)
        Returns EC2 'memory' string as a float.

        Format should always be '#' GiB (example: '244 GiB' or '1,952 GiB'). Amazon loves to put commas in their numbers, so we have to accommodate that. If the syntax ever changes, this will raise.

        Parameters
            mem_info (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- EC2 JSON memory param string.

        Returns
            A float representing memory in GiB.

        Return type
            float </usr/share/doc/python3/html/library/functions.html#float>

    toil.lib.ec2nodes.download_region_json(filename, region='us-east-1')
        Downloads and writes the AWS Billing JSON to a file using the AWS pricing API.

        See: <https://aws.amazon.com/blogs/aws/new-aws-price-list-api/>

        Returns
            A dict of InstanceType objects, where the key is the string: aws instance name (example: 't2.micro'), and the value is an InstanceType object representing that aws instance name.

        Parameters

            •   filename (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   region (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            None

    toil.lib.ec2nodes.reduce_region_json_size(filename)
        Deletes information in the json file that we don't need, and rewrites it. This makes the file smaller.

        The reason being: we used to download the unified AWS Bulk API JSON, which eventually crept up to 5.6Gb, the loading of which could not be done on a 32Gb RAM machine. Now we download each region JSON individually (with AWS's new Query API), but even those may eventually one day grow ridiculously large, so we do what we can to keep the file sizes down (and thus also the amount loaded into memory) to keep this script working for longer.

        Parameters
            filename (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]]

    toil.lib.ec2nodes.updateStaticEC2Instances()
        Generates a new python file of fetchable EC2 Instances by region with current prices and specs.

        Takes a few (~3+) minutes to run (you'll need decent internet).

        Returns
            Nothing. Writes a new 'generatedEC2Lists.py' file.

        Return type
            None

  toil.lib.encryption

  Submodules

  toil.lib.encryption.conftest

  Attributes
collect_ignore |

  Module Contents

    toil.lib.encryption.conftest.collect_ignore = []

  toil.lib.exceptions

  Exceptions
UnimplementedURLException | Unspecified run-time error.

  Classes
panic | The Python idiom for reraising a primary exception fails when the except block raises a

  Functions
raise_(exc_type, exc_value, traceback) |

  Module Contents

    class toil.lib.exceptions.panic(log=None)
        The Python idiom for reraising a primary exception fails when the except block raises a secondary exception, e.g. while trying to cleanup. In that case the original exception is lost and the secondary exception is reraised. The solution seems to be to save the primary exception info as returned from sys.exc_info() and then reraise that.

        This is a contextmanager that should be used like this

        try:
            # do something that can fail

        except:

            with panic( log ):
                # do cleanup that can also fail

    If a logging logger is passed to panic(), any secondary Exception raised within the with block will be logged. Otherwise those exceptions are swallowed. At the end of the with block the primary exception will be reraised.

        Parameters
            log (Optional[logging.Logger </usr/share/doc/python3/html/library/logging.html#logging.Logger>])

        log = None

        exc_info = None

        __enter__()

            Return type
                None

        __exit__(*exc_info)

            Return type
                None

    toil.lib.exceptions.raise_(exc_type, exc_value, traceback)

        Parameters

            •   exc_type (Optional[type </usr/share/doc/python3/html/library/functions.html#type>[BaseException </usr/share/doc/python3/html/library/exceptions.html#BaseException>]])

            •   exc_value (Optional[BaseException </usr/share/doc/python3/html/library/exceptions.html#BaseException>])

            •   traceback (Optional[types.TracebackType </usr/share/doc/python3/html/library/types.html#types.TracebackType>])

        Return type
            None

    exception toil.lib.exceptions.UnimplementedURLException(url, operation)
        Bases: RuntimeError </usr/share/doc/python3/html/library/exceptions.html#RuntimeError>

        Unspecified run-time error.

        Parameters

            •   url (urllib.parse.ParseResult </usr/share/doc/python3/html/library/urllib.parse.html#urllib.parse.ParseResult>)

            •   operation (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

  toil.lib.expando

  Classes
Expando | Pass initial attributes to the constructor:
MagicExpando | Use MagicExpando for chained attribute access.

  Module Contents

    class toil.lib.expando.Expando(*args, **kwargs)
        Bases: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>

        Pass initial attributes to the constructor:

            >>> o = Expando(foo=42)
            >>> o.foo
            42

        Dynamically create new attributes:

            >>> o.bar = 'hi'
            >>> o.bar
            'hi'

        Expando is a dictionary:

            >>> isinstance(o,dict)
            True
            >>> o['foo']
            42

        Works great with JSON:

            >>> import json
            >>> s='{"foo":42}'
            >>> o = json.loads(s,object_hook=Expando)
            >>> o.foo
            42
            >>> o.bar = 'hi'
            >>> o.bar
            'hi'

        And since Expando is a dict, it serializes back to JSON just fine:

            >>> json.dumps(o, sort_keys=True)
            '{"bar": "hi", "foo": 42}'

        Attributes can be deleted, too:

            >>> o = Expando(foo=42)
            >>> o.foo
            42
            >>> del o.foo
            >>> o.foo
            Traceback (most recent call last):
            ...
            AttributeError: 'Expando' object has no attribute 'foo'
            >>> o['foo']
            Traceback (most recent call last):
            ...
            KeyError: 'foo'

            >>> del o.foo
            Traceback (most recent call last):
            ...
            AttributeError: 'Expando' object has no attribute 'foo'

        And copied:

            >>> o = Expando(foo=42)
            >>> p = o.copy()
            >>> isinstance(p,Expando)
            True
            >>> o == p
            True
            >>> o is p
            False

        Same with MagicExpando ...

            >>> o = MagicExpando()
            >>> o.foo.bar = 42
            >>> p = o.copy()
            >>> isinstance(p,MagicExpando)
            True
            >>> o == p
            True
            >>> o is p
            False

        ... but the copy is shallow:

            >>> o.foo is p.foo
            True

        __slots__ = None

        __dict__

        copy()
            Return a shallow copy of the dict.

    class toil.lib.expando.MagicExpando(*args, **kwargs)
        Bases: Expando

        Use MagicExpando for chained attribute access.

        The first time a missing attribute is accessed, it will be set to a new child MagicExpando.

            >>> o=MagicExpando()
            >>> o.foo = 42
            >>> o
            {'foo': 42}
            >>> o.bar.hello = 'hi'
            >>> o.bar
            {'hello': 'hi'}

        __getattribute__(name)
            Return getattr(self, name).

            Parameters
                name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

  toil.lib.ftp_utils

  Attributes
logger |

  Classes
FtpFsAccess | FTP access with upload.

  Module Contents

    toil.lib.ftp_utils.logger

    class toil.lib.ftp_utils.FtpFsAccess(cache=None)
        FTP access with upload.

        Taken and modified from <https://github.com/ohsu-comp-bio/cwl-tes/blob/03f0096f9fae8acd527687d3460a726e09190c3a/cwl_tes/ftp.py#L37-L251>

        Parameters
            cache (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[Any, ftplib.FTP </usr/share/doc/python3/html/library/ftplib.html#ftplib.FTP>]])

        cache

        netrc = None

        exists(fn)
            Check if a file/directory exists over an FTP server :param fn: FTP url :return: True or false depending on whether the object exists on the server

            Parameters
                fn (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        isfile(fn)
            Check if the FTP url points to a file :param fn: FTP url :return: True if url is file, else false

            Parameters
                fn (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        isdir(fn)
            Check if the FTP url points to a directory :param fn: FTP url :return: True if url is directory, else false

            Parameters
                fn (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        open(fn, mode)
            Open an FTP url.

            Only supports reading, no write support. :param fn: FTP url :param mode: Mode to open FTP url in :return:

            Parameters

                •   fn (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   mode (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]

        size(fn)
            Get the size of an FTP object :param fn: FTP url :return: Size of object

            Parameters
                fn (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                Optional[int </usr/share/doc/python3/html/library/functions.html#int>]

  toil.lib.generatedEC2Lists

  Attributes
E2Instances |
regionDict |
ec2InstancesByRegion |

  Module Contents

    toil.lib.generatedEC2Lists.E2Instances

    toil.lib.generatedEC2Lists.regionDict

    toil.lib.generatedEC2Lists.ec2InstancesByRegion

  toil.lib.history

Contains tools for tracking history.

  Attributes
logger |
RT |

  Exceptions
HistoryDatabaseSchemaTooNewError | Raised when we would write to the history database, but its schema is too

  Classes
WorkflowSummary | Data class holding summary information for a workflow.
WorkflowAttemptSummary | Data class holding summary information for a workflow attempt.
JobAttemptSummary | Data class holding summary information for a job attempt within a known
HistoryManager | Class responsible for managing the history of Toil runs.

  Functions
db_retry(function) | Decorate a function with the appropriate retries for accessing the database.

  Module Contents

    toil.lib.history.logger

    exception toil.lib.history.HistoryDatabaseSchemaTooNewError
        Bases: RuntimeError </usr/share/doc/python3/html/library/exceptions.html#RuntimeError>

        Raised when we would write to the history database, but its schema is too new for us to understand.

    class toil.lib.history.WorkflowSummary
        Data class holding summary information for a workflow.

        Represents all the attempts to execute one run of a workflow.

        id: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        name: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        job_store: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        total_attempts: int </usr/share/doc/python3/html/library/functions.html#int>

        total_job_attempts: int </usr/share/doc/python3/html/library/functions.html#int>

        succeeded: bool </usr/share/doc/python3/html/library/functions.html#bool>

        start_time: float </usr/share/doc/python3/html/library/functions.html#float> | None </usr/share/doc/python3/html/library/constants.html#None>
            Time when the first workflow attempt started, in seconds since epoch.

            None if there are no attempts recorded.

        runtime: float </usr/share/doc/python3/html/library/functions.html#float> | None </usr/share/doc/python3/html/library/constants.html#None>
            Time from the first workflow attempt's start to the last one's end, in seconds.

            None if there are no attempts recorded.

        trs_spec: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

    class toil.lib.history.WorkflowAttemptSummary
        Data class holding summary information for a workflow attempt.

        Helpfully includes the workflow metadata for Dockstore.

        workflow_id: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        attempt_number: int </usr/share/doc/python3/html/library/functions.html#int>

        succeeded: bool </usr/share/doc/python3/html/library/functions.html#bool>

        start_time: float </usr/share/doc/python3/html/library/functions.html#float>

        runtime: float </usr/share/doc/python3/html/library/functions.html#float>

        submitted_to_dockstore: bool </usr/share/doc/python3/html/library/functions.html#bool>

        batch_system: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        caching: bool </usr/share/doc/python3/html/library/functions.html#bool> | None </usr/share/doc/python3/html/library/constants.html#None>

        toil_version: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        python_version: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        platform_system: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        platform_machine: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        workflow_job_store: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        workflow_trs_spec: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

    class toil.lib.history.JobAttemptSummary
        Data class holding summary information for a job attempt within a known workflow attempt.

        id: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        job_name: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        succeeded: bool </usr/share/doc/python3/html/library/functions.html#bool>

        start_time: float </usr/share/doc/python3/html/library/functions.html#float>

        runtime: float </usr/share/doc/python3/html/library/functions.html#float>

        submitted_to_dockstore: bool </usr/share/doc/python3/html/library/functions.html#bool>

        cores: float </usr/share/doc/python3/html/library/functions.html#float> | None </usr/share/doc/python3/html/library/constants.html#None>

        cpu_seconds: float </usr/share/doc/python3/html/library/functions.html#float> | None </usr/share/doc/python3/html/library/constants.html#None>

        memory_bytes: int </usr/share/doc/python3/html/library/functions.html#int> | None </usr/share/doc/python3/html/library/constants.html#None>

        disk_bytes: int </usr/share/doc/python3/html/library/functions.html#int> | None </usr/share/doc/python3/html/library/constants.html#None>

    toil.lib.history.RT

    toil.lib.history.db_retry(function)
        Decorate a function with the appropriate retries for accessing the database.

        Parameters
            function (Callable[Ellipsis, RT])

        Return type
            Callable[Ellipsis, RT]

    class toil.lib.history.HistoryManager
        Class responsible for managing the history of Toil runs.

        classmethod enabled()
            Return True if history should be read from and written to the database.

            If False, no access at all shoulf be made to the database.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        classmethod enabled_job()
            Return True if job history should be read from and written to the database.

            Always returns False if enabled() returns False.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        database_path_override: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        classmethod database_path()
            Get the path at which the database we store history in lives.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        classmethod connection()
            Connect to the history database.

            Caller must not actually use the connection without using ensure_tables() to protect reads and updates.

            Must be called from inside a top-level method marked @db_retry.

            The connection will be in DEFERRED isolation_level, with autocommit off on Python versions that support it. In order to run any commands outside of a transaction use the no_transaction context manager.

            Return type
                sqlite3.Connection </usr/share/doc/python3/html/library/sqlite3.html#sqlite3.Connection>

        classmethod no_transaction(con)
            Temporarily disable the constant active transaction on the database connection, on Python versions where it exists.

            Commits the current transaction.

            Parameters
                con (sqlite3.Connection </usr/share/doc/python3/html/library/sqlite3.html#sqlite3.Connection>)

            Return type
                Iterator[None]

        classmethod ensure_tables(con, cur)
            Ensure that tables exist in the database and the schema is migrated to the current version.

            Leaves the cursor in a transaction where the schema version is known to be correct.

            Must be called from inside a top-level methodf marked @db_retry.

            Raises
                HistoryDatabaseSchemaTooNewError <#toil.lib.history.HistoryDatabaseSchemaTooNewError> -- If the schema is newer than the current version.

            Parameters

                •   con (sqlite3.Connection </usr/share/doc/python3/html/library/sqlite3.html#sqlite3.Connection>)

                •   cur (sqlite3.Cursor </usr/share/doc/python3/html/library/sqlite3.html#sqlite3.Cursor>)

            Return type
                None

        classmethod record_workflow_creation(workflow_id, job_store_spec)
            Record that a workflow is being run.

            Takes the Toil config's workflow ID and the location of the job store.

            Should only be called on the first attempt on a job store, not on a restart.

            A workflow may have multiple attempts to run it, some of which succeed and others of which fail. Probably only the last one should succeed.

            Parameters

                •   job_store_spec (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The job store specifier for the workflow. Should be canonical and always start with the type and a colon. If the job store is later moved by the user, the location will not be updated.

                •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        classmethod record_workflow_metadata(workflow_id, workflow_name, trs_spec=None)
            Associate a name and optionally a TRS ID and version with a workflow run.

            Parameters

                •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   workflow_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   trs_spec (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

        classmethod record_job_attempt(workflow_id, workflow_attempt_number, job_name, succeeded, start_time, runtime, cores=None, cpu_seconds=None, memory_bytes=None, disk_bytes=None)
            Record that a job ran in a workflow.

            Doesn't expect the provided information to uniquely identify the job attempt; assigns the job attempt its own unique ID.

            Thread safe.

            Parameters

                •   job_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- A human-readable name for the job. Not expected to be a job store ID or to necessarily uniquely identify the job within the workflow.

                •   start_time (float </usr/share/doc/python3/html/library/functions.html#float>) -- Job execution start time ins econds since epoch.

                •   runtime (float </usr/share/doc/python3/html/library/functions.html#float>) -- Job execution duration in seconds.

                •   cores (Optional[float </usr/share/doc/python3/html/library/functions.html#float>]) -- Number of CPU cores the job was scheduled on.

                •   cpu_seconds (Optional[float </usr/share/doc/python3/html/library/functions.html#float>]) -- CPU core-seconds actually consumed.

                •   memory_bytes (Optional[int </usr/share/doc/python3/html/library/functions.html#int>]) -- Peak observed job memory usage.

                •   disk_bytes (Optional[int </usr/share/doc/python3/html/library/functions.html#int>]) -- Observed job disk usage.

                •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   workflow_attempt_number (int </usr/share/doc/python3/html/library/functions.html#int>)

                •   succeeded (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        classmethod record_workflow_attempt(workflow_id, workflow_attempt_number, succeeded, start_time, runtime, batch_system=None, caching=None, toil_version=None, python_version=None, platform_system=None, platform_machine=None)
            Record a workflow attempt (start or restart) having finished or failed.

            Parameters

                •   batch_system (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- The Python type name of the batch system implementation used.

                •   caching (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>]) -- Whether Toil filestore-level caching was used.

                •   toil_version (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Version of Toil used to run the workflow.

                •   python_version (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Version of Python used to run the workflow.

                •   platform_system (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- OS ("Darwin", "Linux", etc.) used to run the workflow.

                •   platform_machine (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- CPU type ("AMD64", etc.) used to run the workflow leader.

                •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   workflow_attempt_number (int </usr/share/doc/python3/html/library/functions.html#int>)

                •   succeeded (bool </usr/share/doc/python3/html/library/functions.html#bool>)

                •   start_time (float </usr/share/doc/python3/html/library/functions.html#float>)

                •   runtime (float </usr/share/doc/python3/html/library/functions.html#float>)

            Return type
                None

        classmethod summarize_workflows()
            List all known workflows and their summary statistics.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[WorkflowSummary <#toil.lib.history.WorkflowSummary>]

        classmethod get_submittable_workflow_attempts(limit=sys.maxsize)
            List all workflow attempts not yet submitted to Dockstore.

            Parameters
                limit (int </usr/share/doc/python3/html/library/functions.html#int>) -- Get no more than this many.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[WorkflowAttemptSummary <#toil.lib.history.WorkflowAttemptSummary>]

        classmethod get_workflow_attempts_with_submittable_job_attempts(limit=sys.maxsize)
            Get all workflow attempts that have job attempts not yet submitted to Dockstore.

            The workflow attempts themselves will have finished and been recorded, and have TRS IDs.

            Parameters
                limit (int </usr/share/doc/python3/html/library/functions.html#int>) -- Get no more than this many.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[WorkflowAttemptSummary <#toil.lib.history.WorkflowAttemptSummary>]

        classmethod get_workflow_attempt(workflow_id, attempt_number)
            Get a single (not necessarily unsubmitted, not necessarily TRS-ID-having) workflow attempt summary, if present.

            Parameters

                •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   attempt_number (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                Optional[WorkflowAttemptSummary <#toil.lib.history.WorkflowAttemptSummary>]

        classmethod get_unsubmitted_job_attempts(workflow_id, attempt_number)
            List all job attempts in the given workflow attempt not yet submitted to Dockstore.

            Doesn't check to make sure the workflow has a TRS ID.

            Parameters

                •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   attempt_number (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[JobAttemptSummary <#toil.lib.history.JobAttemptSummary>]

        classmethod mark_workflow_attempt_submitted(workflow_id, attempt_number)
            Mark a workflow attempt as having been successfully submitted to Dockstore.

            Does not mark the workflow attempt's job attempts as submitted.

            Parameters

                •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   attempt_number (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                None

        classmethod mark_job_attempts_submitted(job_attempt_ids)
            Mark a collection of job attempts as submitted to Dockstore in a single transaction.

            Parameters
                job_attempt_ids (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

        classmethod count_workflows()
            Count workflows in the database.

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        classmethod count_workflow_attempts()
            Count workflow attempts in the database.

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        classmethod count_job_attempts()
            Count job attempts in the database.

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        classmethod get_fully_submitted_workflow_ids(limit=sys.maxsize)
            Get workflows that have a successful attempt and no unsubmitted attempts or job attempts.

            Parameters
                limit (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        classmethod get_oldest_workflow_ids(limit=sys.maxsize)
            Get workflows that are old.

            Parameters
                limit (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        classmethod delete_workflow(workflow_id)
            Delete a workflow and all its attempts and job attempts.

            Succeeds if the workflow does not exist.

            Parameters
                workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        classmethod get_database_byte_size()
            Get the total number of bytes used by the database.

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        classmethod compact_database()
            Shrink the database to remove unused space.

            Return type
                None

        classmethod enforce_byte_size_limit(limit=100 * 1024 * 1024)
            Shrink the database until it is smaller than the given limit, or until it is empty, by throwing away workflows.

            Throws data away in a sensible order, least important to most important.

            Parameters
                limit (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                None

        classmethod database_dump_lines()
            Yield lines from the database dump.

            For debugging tests.

            Return type
                Iterable[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

  toil.lib.history_submission

Contains logic for generating Toil usage history reports to send back to Toil HQ (via Dockstore), and for working out if the user wants to send them.

  Attributes
logger |
KeyType |
Decision |
DIALOG_TITLE |
DIALOG_TEXT |
DIALOG_OPTIONS |
DIALOG_TIMEOUT |

  Classes
Submission | Class holding a package of information to submit to Dockstore, and the

  Functions
workflow_execution_id(workflow_attempt) | Get the execution ID for a workflow attempt.
workflow_task_set_execution_id(workflow_attempt) | Get the execution ID for a workflow attempt's collection of tasks.
job_execution_id(job_attempt) | Get the execution ID for a job attempt.
get_parsed_trs_spec(workflow_attempt) | Get the TRS ID and version of the workflow, or raise an error.
create_history_submission([batch_size, desired_tasks]) | Make a package of data about recent workflow runs to send in.
create_current_submission(workflow_id, attempt_number) | Make a package of data about the current workflow attempt to send in.
dialog_tkinter(title, text, options, timeout) | Display a dialog with tkinter.
display_dialog_tkinter(title, text, options, timeout, ...) | Display a dialog with tkinter in the current process.
dialog_tui(title, text, options, timeout) | Display a dialog in the terminal.
ask_user_about_publishing_metrics() | Ask the user to set standing workflow submission consent.

  Module Contents

    toil.lib.history_submission.logger

    toil.lib.history_submission.workflow_execution_id(workflow_attempt)
        Get the execution ID for a workflow attempt.

        Result will follow Dockstore's rules.

        Deterministic.

        Parameters
            workflow_attempt (toil.lib.history.WorkflowAttemptSummary <#toil.lib.history.WorkflowAttemptSummary>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.history_submission.workflow_task_set_execution_id(workflow_attempt)
        Get the execution ID for a workflow attempt's collection of tasks.

        Result will follow Dockstore's rules.

        Deterministic.

        Parameters
            workflow_attempt (toil.lib.history.WorkflowAttemptSummary <#toil.lib.history.WorkflowAttemptSummary>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.history_submission.job_execution_id(job_attempt)
        Get the execution ID for a job attempt.

        Result will follow Dockstore's rules.

        Deterministic.

        Parameters
            job_attempt (toil.lib.history.JobAttemptSummary <#toil.lib.history.JobAttemptSummary>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.history_submission.get_parsed_trs_spec(workflow_attempt)
        Get the TRS ID and version of the workflow, or raise an error.

        Returns
            The TRS ID and the TRS version of the wrokflow run.

        Raises
            ValueError if the workflow does not have a TRS spec or if the spec does not contain a version.

        Parameters
            workflow_attempt (toil.lib.history.WorkflowAttemptSummary <#toil.lib.history.WorkflowAttemptSummary>)

        Return type
            tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    class toil.lib.history_submission.Submission
        Class holding a package of information to submit to Dockstore, and the information needed to mark the workflows/tasks as submitted if it is accepted.

        Acceptance is at the TRS id/version combination level within the Submission, since we need one request per TRS ID/version combination.

        data: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>], tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[list </usr/share/doc/python3/html/library/stdtypes.html#list>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[toil.lib.dockstore.RunExecution <#toil.lib.dockstore.RunExecution>, str </usr/share/doc/python3/html/library/stdtypes.html#str>, int </usr/share/doc/python3/html/library/functions.html#int>]], list </usr/share/doc/python3/html/library/stdtypes.html#list>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[toil.lib.dockstore.TaskExecutions <#toil.lib.dockstore.TaskExecutions>, list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]]]]

        add_workflow_attempt(workflow_attempt)
            Add a workflow attempt to the submission.

            May raise an exception if the workflow attempt is not well-formed.

            Parameters
                workflow_attempt (toil.lib.history.WorkflowAttemptSummary <#toil.lib.history.WorkflowAttemptSummary>)

            Return type
                None

        add_job_attempts(workflow_attempt, job_attempts)
            Add the job attempts for a workflow attempt to the submission.

            Parameters

                •   workflow_attempt (toil.lib.history.WorkflowAttemptSummary <#toil.lib.history.WorkflowAttemptSummary>)

                •   job_attempts (list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.lib.history.JobAttemptSummary <#toil.lib.history.JobAttemptSummary>])

            Return type
                None

        empty()
            Return True if there is nothing to actually submit.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        report()
            Compose a multi-line human-readable string report about what will be submitted.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        submit()
            Send in workflow and task information to Dockstore.

            Assumes the user has approved this already.

            If submission succeeds for a TRS id/version combination, records this in the history database.

            Handles errors internally. Will not raise if the submission doesn't go through.

            Returns
                False if there was any error, True if submission was accepted and recorded locally.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.history_submission.create_history_submission(batch_size=None, desired_tasks=None)
        Make a package of data about recent workflow runs to send in.

        Returns a Submission object that remembers how to update the history database to record when it is successfully submitted.

        Parameters

            •   batch_size (Optional[int </usr/share/doc/python3/html/library/functions.html#int>]) -- Number of workflows to try and submit in one request.

            •   desired_tasks (Optional[int </usr/share/doc/python3/html/library/functions.html#int>]) -- Number of tasks to try and put into a task submission batch. Use 0 to not submit any task information.

        Return type
            Submission <#toil.lib.history_submission.Submission>

    toil.lib.history_submission.create_current_submission(workflow_id, attempt_number)
        Make a package of data about the current workflow attempt to send in.

        Useful if the user wants to submit workflow metrics just this time.

        Parameters

            •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   attempt_number (int </usr/share/doc/python3/html/library/functions.html#int>)

        Return type
            Submission <#toil.lib.history_submission.Submission>

    toil.lib.history_submission.KeyType

    toil.lib.history_submission.dialog_tkinter(title, text, options, timeout)
        Display a dialog with tkinter.

        Dialog will have the given title, text, and options.

        Dialog will be displayed by a separate Python process to avoid a Mac dock icon sticking around as long as Toil runs.

        Parameters

            •   options (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[KeyType, str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Dict from machine-readable option key to button text.

            •   title (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   text (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   timeout (float </usr/share/doc/python3/html/library/functions.html#float>)

        Returns
            the key of the selected option, or None if the user declined to select an option.

        Raises
            an exception if the dialog cannot be displayed.

        Return type
            Optional[KeyType]

    toil.lib.history_submission.display_dialog_tkinter(title, text, options, timeout, result_queue)
        Display a dialog with tkinter in the current process.

        Dialog will have the given title, text, and options.

        Parameters

            •   options (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[KeyType, str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Dict from machine-readable option key to button text.

            •   title (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   text (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   timeout (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   result_queue (multiprocessing.Queue </usr/share/doc/python3/html/library/multiprocessing.html#multiprocessing.Queue>[Union[Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>, Optional[KeyType]]])

        Return type
            None

    Sends back either the key of the chosen option, or an exception, via the result queue.

    toil.lib.history_submission.dialog_tui(title, text, options, timeout)
        Display a dialog in the terminal.

        Dialog will have the given title, text, and options.

        Parameters

            •   options (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[KeyType, str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Dict from machine-readable option key to button text.

            •   title (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   text (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   timeout (float </usr/share/doc/python3/html/library/functions.html#float>)

        Returns
            the key of the selected option, or None if the user declined to select an option.

        Raises
            an exception if the dialog cannot be displayed.

        Return type
            Optional[KeyType]

    toil.lib.history_submission.Decision

    toil.lib.history_submission.DIALOG_TITLE = 'Publish Workflow Metrics on Dockstore.org?'

    toil.lib.history_submission.DIALOG_TEXT = ''

    toil.lib.history_submission.DIALOG_OPTIONS: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[Decision, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.lib.history_submission.DIALOG_TIMEOUT = 120

    toil.lib.history_submission.ask_user_about_publishing_metrics()
        Ask the user to set standing workflow submission consent.

        If the user makes a persistent decision (always or never), save it to the default Toil config file.

        Returns
            The user's decision about when to publish metrics.

        Return type
            Union[Literal['all'], Literal['current'], Literal['no']]

  toil.lib.humanize

  Attributes
logger |

  Functions
bytes2human(n) | Convert n bytes into a human readable string.
human2bytes(s) | Attempts to guess the string format based on default symbols

  Module Contents

    toil.lib.humanize.logger

    toil.lib.humanize.bytes2human(n)
        Convert n bytes into a human readable string.

        Parameters
            n (SupportsInt)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.humanize.human2bytes(s)
        Attempts to guess the string format based on default symbols set and return the corresponding bytes as an integer.

        When unable to recognize the format ValueError is raised.

        Parameters
            s (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            int </usr/share/doc/python3/html/library/functions.html#int>

  toil.lib.io

  Attributes
logger |
TOIL_URI_SCHEME |
STANDARD_SCHEMES |
REMOTE_SCHEMES |
ALL_SCHEMES |

  Classes
WriteWatchingStream | A stream wrapping class that calls any functions passed to onWrite() with the number of bytes written for every write.
ReadableFileObj | Protocol that is more specific than what file_digest takes as an argument.

  Functions
get_toil_home() | Get the Toil home directory for storing configuration and global state.
is_standard_url(filename) | Return True if the given URL is a non-Toil, non-file: URL.
is_remote_url(filename) | Decide if a filename is a known, non-file kind of URL
is_any_url(filename) | Decide if a string is a URI like <http://> or <file://>.
is_url_with_scheme(filename, schemes) | Return True if filename is a URL with any of the given schemes and False otherwise.
is_toil_url(filename) | Return True if a URL is a toilfile: or toildir: URL.
is_toil_file_url(filename) | Return True if a URL is a toilfile: URL.
is_toil_dir_url(filename) | Return True if a URL is a toildir: URL.
is_file_url(filename) | Return True if a URL is a file: URL.
is_directory_url(filename) | Return True if a URL points to a directory.
mkdtemp([suffix, prefix, dir]) | Make a temporary directory like tempfile.mkdtemp, but with relaxed permissions.
robust_rmtree(path) | Robustly tries to delete paths.
atomic_tmp_file(final_path) | Return a tmp file name to use with atomic_install. This will be in the
atomic_install(tmp_path, final_path) | atomic install of tmp_path as final_path
AtomicFileCreate(final_path[, keep]) | Context manager to create a temporary file. Entering returns path to
atomic_copy(src_path, dest_path[, executable]) | Copy a file using posix atomic creations semantics.
atomic_copyobj(src_fh, dest_path[, length, executable]) | Copy an open file using posix atomic creations semantics.
make_public_dir(in_directory[, suggested_name]) | Make a publicly-accessible directory in the given directory.
try_path(path[, min_size]) | Try to use the given path. Return it if it exists or can be made,
file_digest(f, alg_name) | Polyfilled hashlib.file_digest that works on Python <3.11.

  Module Contents

    toil.lib.io.logger

    toil.lib.io.get_toil_home()
        Get the Toil home directory for storing configuration and global state.

        Raises an error if it does not exist and cannot be created. Safe to run simultaneously in multiple processes.

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.io.TOIL_URI_SCHEME = 'toilfile:'

    toil.lib.io.STANDARD_SCHEMES = ['http:', 'https:', 's3:', 'gs:', 'ftp:']

    toil.lib.io.REMOTE_SCHEMES = ['http:', 'https:', 's3:', 'gs:', 'ftp:', 'toilfile:', 'toildir:']

    toil.lib.io.ALL_SCHEMES = ['http:', 'https:', 's3:', 'gs:', 'ftp:', 'toilfile:', 'toildir:', 'file:']

    toil.lib.io.is_standard_url(filename)
        Return True if the given URL is a non-Toil, non-file: URL.

        Parameters
            filename (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.io.is_remote_url(filename)
        Decide if a filename is a known, non-file kind of URL

        Parameters
            filename (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.io.is_any_url(filename)
        Decide if a string is a URI like <http://> or <file://>.

        Otherwise it might be a bare path.

        Parameters
            filename (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.io.is_url_with_scheme(filename, schemes)
        Return True if filename is a URL with any of the given schemes and False otherwise.

        Parameters

            •   filename (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   schemes (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.io.is_toil_url(filename)
        Return True if a URL is a toilfile: or toildir: URL.

        Parameters
            filename (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.io.is_toil_file_url(filename)
        Return True if a URL is a toilfile: URL.

        Parameters
            filename (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.io.is_toil_dir_url(filename)
        Return True if a URL is a toildir: URL.

        Note that this may point to either a directory or a leaf file.

        Parameters
            filename (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.io.is_file_url(filename)
        Return True if a URL is a file: URL.

        Will return False for bare paths.

        Parameters
            filename (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.io.is_directory_url(filename)
        Return True if a URL points to a directory.

        Covers toildir: URLs and deterrmines if they point to directories or leaf files. Also covers other supported remote URL schemes.

        Parameters
            filename (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.io.mkdtemp(suffix=None, prefix=None, dir=None)
        Make a temporary directory like tempfile.mkdtemp, but with relaxed permissions.

        The permissions on the directory will be 711 instead of 700, allowing the group and all other users to traverse the directory. This is necessary if the directory is on NFS and the Docker daemon would like to mount it or a file inside it into a container, because on NFS even the Docker daemon appears bound by the file permissions.

        See <<https://github.com/DataBiosphere/toil/issues/4644>>, and <<https://stackoverflow.com/a/67928880>> which talks about a similar problem but in the context of user namespaces.

        Parameters

            •   suffix (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   prefix (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   dir (Optional[toil.lib.misc.StrPath])

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.io.robust_rmtree(path)
        Robustly tries to delete paths.

        Continues silently if the path to be removed is already gone, or if it goes away while this function is executing.

        May raise an error if a path changes between file and directory while the function is executing, or if a permission error is encountered.

        Parameters
            path (Union[str </usr/share/doc/python3/html/library/stdtypes.html#str>, bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>])

        Return type
            None

    toil.lib.io.atomic_tmp_file(final_path)
        Return a tmp file name to use with atomic_install. This will be in the same directory as final_path. The temporary file will have the same extension as finalPath. It the final path is in /dev (/dev/null, /dev/stdout), it is returned unchanged and atomic_tmp_install will do nothing.

        Parameters
            final_path (toil.lib.misc.StrPath)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.io.atomic_install(tmp_path, final_path)
        atomic install of tmp_path as final_path

        Parameters

            •   tmp_path (toil.lib.misc.StrPath)

            •   final_path (toil.lib.misc.StrPath)

        Return type
            None

    toil.lib.io.AtomicFileCreate(final_path, keep=False)
        Context manager to create a temporary file. Entering returns path to the temporary file in the same directory as finalPath. If the code in context succeeds, the file renamed to its actual name. If an error occurs, the file is not installed and is removed unless keep is specified.

        Parameters

            •   final_path (toil.lib.misc.StrPath)

            •   keep (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            collections.abc.Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.lib.io.atomic_copy(src_path, dest_path, executable=None)
        Copy a file using posix atomic creations semantics.

        Parameters

            •   src_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   dest_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   executable (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>])

        Return type
            None

    toil.lib.io.atomic_copyobj(src_fh, dest_path, length=16384, executable=False)
        Copy an open file using posix atomic creations semantics.

        Parameters

            •   src_fh (io.BytesIO </usr/share/doc/python3/html/library/io.html#io.BytesIO>)

            •   dest_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   length (int </usr/share/doc/python3/html/library/functions.html#int>)

            •   executable (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            None

    toil.lib.io.make_public_dir(in_directory, suggested_name=None)
        Make a publicly-accessible directory in the given directory.

        Parameters

            •   suggested_name (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Use this directory name first if possible.

            •   in_directory (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    Try to make a random directory name with length 4 that doesn't exist, with the given prefix. Otherwise, try length 5, length 6, etc, up to a max of 32 (len of uuid4 with dashes replaced). This function's purpose is mostly to avoid having long file names when generating directories. If somehow this fails, which should be incredibly unlikely, default to a normal uuid4, which was our old default.

    toil.lib.io.try_path(path, min_size=100 * 1024 * 1024)
        Try to use the given path. Return it if it exists or can be made, and we can make things within it, or None otherwise.

        Parameters

            •   min_size (int </usr/share/doc/python3/html/library/functions.html#int>) -- Reject paths on filesystems smaller than this many bytes.

            •   path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    class toil.lib.io.WriteWatchingStream(backingStream)
        A stream wrapping class that calls any functions passed to onWrite() with the number of bytes written for every write.

        Not seekable.

        Parameters
            backingStream (IO[Any])

        backingStream

        writeListeners: list </usr/share/doc/python3/html/library/stdtypes.html#list>[Callable[[int </usr/share/doc/python3/html/library/functions.html#int>], None </usr/share/doc/python3/html/library/constants.html#None>]] = []

        onWrite(listener)
            Call the given listener with the number of bytes written on every write.

            Parameters
                listener (Callable[[int </usr/share/doc/python3/html/library/functions.html#int>], None])

            Return type
                None

        write(data)
            Write the given data to the file.

            Parameters
                data (bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>)

            Return type
                None

        writelines(datas)
            Write each string from the given iterable, without newlines.

            Parameters
                datas (collections.abc.Iterable </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterable>[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>])

            Return type
                None

        flush()
            Flush the backing stream.

            Return type
                None

        close()
            Close the backing stream.

            Return type
                None

    class toil.lib.io.ReadableFileObj
        Bases: Protocol

        Protocol that is more specific than what file_digest takes as an argument. Also guarantees a read() method. Would extend the protocol from Typeshed for hashlib but those are only declared for 3.11+.

        readinto(buf, /)

            Parameters
                buf (bytearray </usr/share/doc/python3/html/library/stdtypes.html#bytearray>)

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        readable()

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        read(number)

            Parameters
                number (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>

    toil.lib.io.file_digest(f, alg_name)
        Polyfilled hashlib.file_digest that works on Python <3.11.

        Parameters

            •   f (ReadableFileObj <#toil.lib.io.ReadableFileObj>)

            •   alg_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            hashlib._Hash

  toil.lib.memoize

  Attributes
memoize | Memoize a function result based on its parameters using this decorator.
MAT |
MRT |

  Functions
sync_memoize(f) | Like memoize, but guarantees that decorated function is only called once, even when multiple
parse_iso_utc(s) | Parses an RFC 3339 ISO 8601 time in the UTC timezone.
strict_bool(s) | Variant of bool() that only accepts two possible string values.

  Module Contents

    toil.lib.memoize.memoize
        Memoize a function result based on its parameters using this decorator.

        For example, this can be used in place of lazy initialization. If the decorating function is invoked by multiple threads, the decorated function may be called more than once with the same arguments.

    toil.lib.memoize.MAT

    toil.lib.memoize.MRT

    toil.lib.memoize.sync_memoize(f)
        Like memoize, but guarantees that decorated function is only called once, even when multiple threads are calling the decorating function with multiple parameters.

        Parameters
            f (Callable[[MAT], MRT])

        Return type
            Callable[[MAT], MRT]

    toil.lib.memoize.parse_iso_utc(s)
        Parses an RFC 3339 ISO 8601 time in the UTC timezone.

        Other timezones are not supported. Returns a timezone-aware UTC datetime object.

        Parameters
            s (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The ISO-formatted time

        Returns
            A timezone-aware UTC datetime object

        Raises
            ValueError </usr/share/doc/python3/html/library/exceptions.html#ValueError> -- if the string is not in the correct format or is not in the UTC timezone.

        Return type
            datetime.datetime </usr/share/doc/python3/html/library/datetime.html#datetime.datetime>

        >>> parse_iso_utc('2016-04-27T00:28:04.000Z')
        datetime.datetime(2016, 4, 27, 0, 28, 4, tzinfo=datetime.timezone.utc)
        >>> parse_iso_utc('2016-04-27T00:28:04Z')
        datetime.datetime(2016, 4, 27, 0, 28, 4, tzinfo=datetime.timezone.utc)
        >>> parse_iso_utc('2016-04-27T00:28:04X')
        Traceback (most recent call last):
        ...
        ValueError: Not a valid ISO datetime in UTC: 2016-04-27T00:28:04X

    toil.lib.memoize.strict_bool(s)
        Variant of bool() that only accepts two possible string values.

        Parameters
            s (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

  toil.lib.misc

  Attributes
logger |
StrPath |
FileDescriptorOrPath |

  Exceptions
CalledProcessErrorStderr | Version of CalledProcessError that include stderr in the error message if it is set

  Functions
get_public_ip() | Get the IP that this machine uses to contact the internet.
get_user_name() | Get the current user name, or a suitable substitute string if the user name
utc_now() | Return a datetime in the UTC timezone corresponding to right now.
unix_now_ms() | Return the current time in milliseconds since the Unix epoch.
unix_seconds_to_timestamp(timestamp) | Convert a time in seconds since the Unix epoch to an ISO 8601 string.
unix_seconds_to_local_time(timestamp) | Returns a local time corresponding to the given number of seconds since the Unix epoch.
seconds_to_duration(time_difference) | Convert a time difference in seconds to an ISO 8601 duration string.
slow_down(seconds) | Toil jobs that have completed are not allowed to have taken 0 seconds, but
printq(msg, quiet[, log]) | This is for functions used simultaneously in Toil proper and in the admin scripts.
truncExpBackoff() |
call_command(cmd, *args[, input, timeout, useCLocale, ...]) | Simplified calling of external commands.

  Module Contents

    toil.lib.misc.logger

    type toil.lib.misc.StrPath = str </usr/share/doc/python3/html/library/stdtypes.html#str> | os.PathLike </usr/share/doc/python3/html/library/os.html#os.PathLike>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    type toil.lib.misc.FileDescriptorOrPath = int </usr/share/doc/python3/html/library/functions.html#int> | bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes> | os.PathLike </usr/share/doc/python3/html/library/os.html#os.PathLike>[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>] | StrPath

    toil.lib.misc.get_public_ip()
        Get the IP that this machine uses to contact the internet.

        If behind a NAT, this will still be this computer's IP, and not the router's.

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.misc.get_user_name()
        Get the current user name, or a suitable substitute string if the user name is not available.

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.misc.utc_now()
        Return a datetime in the UTC timezone corresponding to right now.

        Return type
            datetime.datetime </usr/share/doc/python3/html/library/datetime.html#datetime.datetime>

    toil.lib.misc.unix_now_ms()
        Return the current time in milliseconds since the Unix epoch.

        Return type
            float </usr/share/doc/python3/html/library/functions.html#float>

    toil.lib.misc.unix_seconds_to_timestamp(timestamp)
        Convert a time in seconds since the Unix epoch to an ISO 8601 string.

        Parameters
            timestamp (float </usr/share/doc/python3/html/library/functions.html#float>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.misc.unix_seconds_to_local_time(timestamp)
        Returns a local time corresponding to the given number of seconds since the Unix epoch.

        Parameters
            timestamp (float </usr/share/doc/python3/html/library/functions.html#float>)

        Return type
            datetime.datetime </usr/share/doc/python3/html/library/datetime.html#datetime.datetime>

    toil.lib.misc.seconds_to_duration(time_difference)
        Convert a time difference in seconds to an ISO 8601 duration string.

        Parameters
            time_difference (float </usr/share/doc/python3/html/library/functions.html#float>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.misc.slow_down(seconds)
        Toil jobs that have completed are not allowed to have taken 0 seconds, but Kubernetes timestamps round things to the nearest second. It is possible in some batch systems for a pod to have identical start and end timestamps.

        This function takes a possibly 0 job length in seconds and enforces a minimum length to satisfy Toil.

        Parameters
            seconds (float </usr/share/doc/python3/html/library/functions.html#float>) -- Timestamp difference

        Returns
            seconds, or a small positive number if seconds is 0

        Return type
            float </usr/share/doc/python3/html/library/functions.html#float>

    toil.lib.misc.printq(msg, quiet, log=False)
        This is for functions used simultaneously in Toil proper and in the admin scripts.

        Our admin scripts "print" to stdout, while Toil proper uses logging. For a script that, for example, cleans up IAM, EC2, etc. cruft leftover after failed CI runs, we can call an AWS delete IAM role function, and this prints or logs progress (unless quiet is True), depending on whether the function is called in, say, the jobstore or a script.

        Parameters

            •   msg (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The string to print or log to stdout.

            •   quiet (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Silent output to stdout.

            •   log (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Use logging (else "print" to the screen).

        Return type
            None

    toil.lib.misc.truncExpBackoff()

        Return type
            collections.abc.Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[float </usr/share/doc/python3/html/library/functions.html#float>]

    exception toil.lib.misc.CalledProcessErrorStderr(returncode, cmd, output=None, stderr=None)
        Bases: subprocess.CalledProcessError </usr/share/doc/python3/html/library/subprocess.html#subprocess.CalledProcessError>

        Version of CalledProcessError that include stderr in the error message if it is set

        __str__()
            Return str(self).

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.misc.call_command(cmd, *args, input=None, timeout=None, useCLocale=True, env=None, quiet=False)
        Simplified calling of external commands.

        If the process fails, CalledProcessErrorStderr is raised.

        The captured stderr is always printed, regardless of if an exception occurs, so it can be logged.

        Always logs the command at debug log level.

        Parameters

            •   quiet (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>]) -- If True, do not log the command output. If False (the default), do log the command output at debug log level.

            •   useCLocale (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If True, C locale is forced, to prevent failures that can occur in some batch systems when using UTF-8 locale.

            •   cmd (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   args (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   input (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   timeout (Optional[float </usr/share/doc/python3/html/library/functions.html#float>])

            •   env (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

        Returns
            Command standard output, decoded as utf-8.

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

  toil.lib.objects

  Classes
InnerClass | Note that this is EXPERIMENTAL code.

  Module Contents

    class toil.lib.objects.InnerClass(inner_class)
        Note that this is EXPERIMENTAL code.

        A nested class (the inner class) decorated with this will have an additional attribute called 'outer' referencing the instance of the nesting class (the outer class) that was used to create the inner class. The outer instance does not need to be passed to the inner class's constructor, it will be set magically. Shamelessly stolen from

        <http://stackoverflow.com/questions/2278426/inner-classes-how-can-i-get-the-outer-class-object-at-construction-time#answer-2278595>.

        with names made more descriptive (I hope) and added caching of the BoundInner classes.

        Caveat: Within the inner class, self.__class__ will not be the inner class but a dynamically created subclass thereof. It's name will be the same as that of the inner class, but its __module__ will be different. There will be one such dynamic subclass per inner class and instance of outer class, if that outer class instance created any instances of inner the class.

            >>> class Outer(object):
            ...     def new_inner(self):
            ...         # self is an instance of the outer class
            ...         inner = self.Inner()
            ...         # the inner instance's 'outer' attribute is set to the outer instance
            ...         assert inner.outer is self
            ...         return inner
            ...     @InnerClass
            ...     class Inner(object):
            ...         def get_outer(self):
            ...             return self.outer
            ...         @classmethod
            ...         def new_inner(cls):
            ...             return cls()
            >>> o = Outer()
            >>> i = o.new_inner()
            >>> i
            <toil.lib.objects.Inner...> bound to <toil.lib.objects.Outer object at ...>

            >>> i.get_outer()
            <toil.lib.objects.Outer object at ...>

        Now with inheritance for both inner and outer:

            >>> class DerivedOuter(Outer):
            ...     def new_inner(self):
            ...         return self.DerivedInner()
            ...     @InnerClass
            ...     class DerivedInner(Outer.Inner):
            ...         def get_outer(self):
            ...             assert super( DerivedOuter.DerivedInner, self ).get_outer() == self.outer
            ...             return self.outer
            >>> derived_outer = DerivedOuter()
            >>> derived_inner = derived_outer.new_inner()
            >>> derived_inner
            <toil.lib.objects...> bound to <toil.lib.objects.DerivedOuter object at ...>

            >>> derived_inner.get_outer()
            <toil.lib.objects.DerivedOuter object at ...>

        Test a static references: >>> Outer.Inner # doctest: +ELLIPSIS <class 'toil.lib.objects...Inner'> >>> DerivedOuter.Inner # doctest: +ELLIPSIS <class 'toil.lib.objects...Inner'> >>> DerivedOuter.DerivedInner #doctest: +ELLIPSIS <class 'toil.lib.objects...DerivedInner'>

        Can't decorate top-level classes. Unfortunately, this is detected when the instance is created, not when the class is defined. >>> @InnerClass ... class Foo(object): ... pass >>> Foo() Traceback (most recent call last): ... RuntimeError: Inner classes must be nested in another class.

        All inner instances should refer to a single outer instance: >>> o = Outer() >>> o.new_inner().outer == o == o.new_inner().outer True

        All inner instances should be of the same class ... >>> o.new_inner().__class__ == o.new_inner().__class__ True

        ... but that class isn't the inner class ... >>> o.new_inner().__class__ != Outer.Inner True

        ... but a subclass of the inner class. >>> isinstance( o.new_inner(), Outer.Inner ) True

        Static and class methods, e.g. should work, too

            >>> o.Inner.new_inner().outer == o
            True

        inner_class

        __get__(instance, owner)

        __call__(**kwargs)

  toil.lib.pipes

  Attributes
log |

  Classes
WritablePipe | An object-oriented wrapper for os.pipe. Clients should subclass it, implement
ReadablePipe | An object-oriented wrapper for os.pipe. Clients should subclass it, implement
ReadableTransformingPipe | A pipe which is constructed around a readable stream, and which provides a
HashingPipe | Class which checksums all the data read through it. If it

  Module Contents

    toil.lib.pipes.log

    class toil.lib.pipes.WritablePipe(encoding=None, errors=None)
        Bases: abc.ABC </usr/share/doc/python3/html/library/abc.html#abc.ABC>

        An object-oriented wrapper for os.pipe. Clients should subclass it, implement readFrom() to consume the readable end of the pipe, then instantiate the class as a context manager to get the writable end. See the example below.

            >>> import sys, shutil, codecs
            >>> class MyPipe(WritablePipe):
            ...     def readFrom(self, readable):
            ...         shutil.copyfileobj(codecs.getreader('utf-8')(readable), sys.stdout)
            >>> with MyPipe() as writable:
            ...     _ = writable.write('Hello, world!\n'.encode('utf-8'))
            Hello, world!

        Each instance of this class creates a thread and invokes the readFrom method in that thread. The thread will be join()ed upon normal exit from the context manager, i.e. the body of the with statement. If an exception occurs, the thread will not be joined but a well-behaved readFrom() implementation will terminate shortly thereafter due to the pipe having been closed.

        Now, exceptions in the reader thread will be reraised in the main thread:

            >>> class MyPipe(WritablePipe):
            ...     def readFrom(self, readable):
            ...         raise RuntimeError('Hello, world!')
            >>> with MyPipe() as writable:
            ...     pass
            Traceback (most recent call last):
            ...
            RuntimeError: Hello, world!

        More complicated, less illustrative tests:

        Same as above, but proving that handles are closed:

            >>> x = os.dup(0); os.close(x)
            >>> class MyPipe(WritablePipe):
            ...     def readFrom(self, readable):
            ...         raise RuntimeError('Hello, world!')
            >>> with MyPipe() as writable:
            ...     pass
            Traceback (most recent call last):
            ...
            RuntimeError: Hello, world!
            >>> y = os.dup(0); os.close(y); x == y
            True

        Exceptions in the body of the with statement aren't masked, and handles are closed:

            >>> x = os.dup(0); os.close(x)
            >>> class MyPipe(WritablePipe):
            ...     def readFrom(self, readable):
            ...         pass
            >>> with MyPipe() as writable:
            ...     raise RuntimeError('Hello, world!')
            Traceback (most recent call last):
            ...
            RuntimeError: Hello, world!
            >>> y = os.dup(0); os.close(y); x == y
            True

        Parameters

            •   encoding (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   errors (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        encoding: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        errors: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        readable_fh: int </usr/share/doc/python3/html/library/functions.html#int> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        writable: IO[Any] | None </usr/share/doc/python3/html/library/constants.html#None> = None

        thread: toil.lib.threading.ExceptionalThread <#toil.lib.threading.ExceptionalThread> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        reader_done: bool </usr/share/doc/python3/html/library/functions.html#bool> = False

        __enter__()

            Return type
                IO[Any]

        __exit__(exc_type, exc_val, exc_tb)

            Parameters

                •   exc_type (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   exc_val (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   exc_tb (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

        abstractmethod readFrom(readable)
            Implement this method to read data from the pipe. This method should support both binary and text mode output.

            Parameters
                readable (file) -- the file object representing the readable end of the pipe. Do not explicitly invoke the close() method of the object; that will be done automatically.

            Return type
                None

    class toil.lib.pipes.ReadablePipe(encoding=None, errors=None)
        Bases: abc.ABC </usr/share/doc/python3/html/library/abc.html#abc.ABC>

        An object-oriented wrapper for os.pipe. Clients should subclass it, implement writeTo() to place data into the writable end of the pipe, then instantiate the class as a context manager to get the writable end. See the example below.

            >>> import sys, shutil, codecs
            >>> class MyPipe(ReadablePipe):
            ...     def writeTo(self, writable: IO[Any]) -> None:
            ...         writable.write('Hello, world!\n'.encode('utf-8'))
            >>> with MyPipe() as readable:
            ...     shutil.copyfileobj(codecs.getreader('utf-8')(readable), sys.stdout)
            Hello, world!

        Each instance of this class creates a thread and invokes the writeTo() method in that thread. The thread will be join()ed upon normal exit from the context manager, i.e. the body of the with statement. If an exception occurs, the thread will not be joined but a well-behaved writeTo() implementation will terminate shortly thereafter due to the pipe having been closed.

        Now, exceptions in the reader thread will be reraised in the main thread:

            >>> class MyPipe(ReadablePipe):
            ...     def writeTo(self, writable):
            ...         raise RuntimeError('Hello, world!')
            >>> with MyPipe() as readable:
            ...     pass
            Traceback (most recent call last):
            ...
            RuntimeError: Hello, world!

        More complicated, less illustrative tests:

        Same as above, but proving that handles are closed:

            >>> x = os.dup(0); os.close(x)
            >>> class MyPipe(ReadablePipe):
            ...     def writeTo(self, writable: IO[Any]) -> None:
            ...         raise RuntimeError('Hello, world!')
            >>> with MyPipe() as readable:
            ...     pass
            Traceback (most recent call last):
            ...
            RuntimeError: Hello, world!
            >>> y = os.dup(0); os.close(y); x == y
            True

        Exceptions in the body of the with statement aren't masked, and handles are closed:

            >>> x = os.dup(0); os.close(x)
            >>> class MyPipe(ReadablePipe):
            ...     def writeTo(self, writable):
            ...         pass
            >>> with MyPipe() as readable:
            ...     raise RuntimeError('Hello, world!')
            Traceback (most recent call last):
            ...
            RuntimeError: Hello, world!
            >>> y = os.dup(0); os.close(y); x == y
            True

        Parameters

            •   encoding (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   errors (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        abstractmethod writeTo(writable)
            Implement this method to write data from the pipe. This method should support both binary and text mode input.

            Parameters
                writable (file) -- the file object representing the writable end of the pipe. Do not explicitly invoke the close() method of the object, that will be done automatically.

            Return type
                None

        encoding: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        errors: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        writable_fh: int </usr/share/doc/python3/html/library/functions.html#int> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        readable: IO[Any] | None </usr/share/doc/python3/html/library/constants.html#None> = None

        thread: toil.lib.threading.ExceptionalThread <#toil.lib.threading.ExceptionalThread> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        __enter__()

            Return type
                IO[Any]

        __exit__(exc_type, exc_val, exc_tb)

            Parameters

                •   exc_type (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   exc_val (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   exc_tb (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

    class toil.lib.pipes.ReadableTransformingPipe(source, encoding=None, errors=None)
        Bases: ReadablePipe

        A pipe which is constructed around a readable stream, and which provides a context manager that gives a readable stream.

        Useful as a base class for pipes which have to transform or otherwise visit bytes that flow through them, instead of just consuming or producing data.

        Clients should subclass it and implement transform(), like so:

            >>> import sys, shutil, codecs
            >>> class MyPipe(ReadableTransformingPipe):
            ...     def transform(self, readable, writable):
            ...         writable.write(readable.read().decode('utf-8').upper().encode('utf-8'))
            >>> class SourcePipe(ReadablePipe):
            ...     def writeTo(self, writable):
            ...         writable.write('Hello, world!\n'.encode('utf-8'))
            >>> with SourcePipe() as source:
            ...     with MyPipe(source) as transformed:
            ...         shutil.copyfileobj(codecs.getreader('utf-8')(transformed), sys.stdout)
            HELLO, WORLD!

        The transform() method runs in its own thread, and should move data chunk by chunk instead of all at once. It should finish normally if it encounters either an EOF on the readable, or a BrokenPipeError </usr/share/doc/python3/html/library/exceptions.html#BrokenPipeError> on the writable. This means that it should make sure to actually catch a BrokenPipeError </usr/share/doc/python3/html/library/exceptions.html#BrokenPipeError> when writing.

        See also: toil.lib.misc.WriteWatchingStream.

        Parameters

            •   source (IO[Any])

            •   encoding (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   errors (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        source

        abstractmethod transform(readable, writable)
            Implement this method to ship data through the pipe.

            Parameters

                •   readable (file) -- the input stream file object to transform.

                •   writable (file) -- the file object representing the writable end of the pipe. Do not explicitly invoke the close() method of the object, that will be done automatically.

            Return type
                None

        writeTo(writable)
            Implement this method to write data from the pipe. This method should support both binary and text mode input.

            Parameters
                writable (file) -- the file object representing the writable end of the pipe. Do not explicitly invoke the close() method of the object, that will be done automatically.

            Return type
                None

    class toil.lib.pipes.HashingPipe(source, encoding=None, errors=None, checksum_to_verify=None)
        Bases: ReadableTransformingPipe

        Class which checksums all the data read through it. If it reaches EOF and the checksum isn't correct, raises ChecksumError.

        Assumes info actually has a checksum.

        Parameters

            •   source (IO[Any])

            •   encoding (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   errors (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   checksum_to_verify (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        checksum_to_verify = None

        transform(readable, writable)
            Implement this method to ship data through the pipe.

            Parameters

                •   readable (file) -- the input stream file object to transform.

                •   writable (file) -- the file object representing the writable end of the pipe. Do not explicitly invoke the close() method of the object, that will be done automatically.

            Return type
                None

  toil.lib.plugins

Generic plugin system for Toil plugins.

Plugins come in Python packages named:

            toil_{PLUGIN_TYPE}_{WHATEVER}

When looking for plugins, Toil will list all the Python packages with the right name prefix for the given type of plugin, and load them. The plugin modules then have an opportunity to import register_plugin() and register themselves.

  Attributes
PluginType |
plugin_types |

  Functions
register_plugin(plugin_type, plugin_name, ...) | Adds a plugin to the registry for the given type of plugin.
remove_plugin(plugin_type, plugin_name) | Removes a plugin from the registry for the given type of plugin.
get_plugin_names(plugin_type) | Get the names of all the available plugins of the given type.
get_plugin(plugin_type, plugin_name) | Get a plugin class factory function by name.

  Module Contents

    toil.lib.plugins.PluginType

    toil.lib.plugins.plugin_types: list </usr/share/doc/python3/html/library/stdtypes.html#list>[PluginType] = ['batch_system', 'url_access']

    toil.lib.plugins.register_plugin(plugin_type, plugin_name, plugin_being_registered)
        Adds a plugin to the registry for the given type of plugin.

        Parameters

            •   plugin_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- For batch systems, this is the string the user will use to select the batch system on the command line with --batchSystem. For URL access plugins, this is the URL scheme that the plugin implements.

            •   plugin_being_registered (Any) -- This is a function that, when called, imports and returns a plugin-provided class type. For batch systems, the resulting type must extend toil.batchSystems.abstractBatchSystem.AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>. For URL access plugins, it must extend toil.lib.url.URLAccess <#toil.lib.url.URLAccess>. Note that the function used here should return the class itself; it should not construct an instance of the class.

            •   plugin_type (PluginType)

        Return type
            None

    toil.lib.plugins.remove_plugin(plugin_type, plugin_name)
        Removes a plugin from the registry for the given type of plugin.

        Parameters

            •   plugin_type (PluginType)

            •   plugin_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            None

    toil.lib.plugins.get_plugin_names(plugin_type)
        Get the names of all the available plugins of the given type.

        Parameters
            plugin_type (PluginType)

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.lib.plugins.get_plugin(plugin_type, plugin_name)
        Get a plugin class factory function by name.

        Raises
            KeyError if plugin_name is not the name of a plugin of the given type.

        Parameters

            •   plugin_type (PluginType)

            •   plugin_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            Any

  toil.lib.resources

  Classes
ResourceMonitor | Global resource monitoring widget.

  Functions
glob(glob_pattern, directoryname) | Walks through a directory and its subdirectories looking for files matching

  Module Contents

    class toil.lib.resources.ResourceMonitor
        Global resource monitoring widget.

        Presents class methods to get the resource usage of this process and child processes, and other class methods to adjust the statistics so they can account for e.g. resources used inside containers, or other resource usage that should be billable to the current process.

        classmethod record_extra_memory(peak_ki)
            Become responsible for the given peak memory usage, in kibibytes.

            The memory will be treated as if it was used by a child process at the time our real child processes were also using their peak memory.

            Parameters
                peak_ki (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                None

        classmethod record_extra_cpu(seconds)
            Become responsible for the given CPU time.

            The CPU time will be treated as if it had been used by a child process.

            Parameters
                seconds (float </usr/share/doc/python3/html/library/functions.html#float>)

            Return type
                None

        classmethod get_total_cpu_time_and_memory_usage()
            Gives the total cpu time of itself and all its children, and the maximum RSS memory usage of itself and its single largest child (in kibibytes).

            Return type
                tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[float </usr/share/doc/python3/html/library/functions.html#float>, int </usr/share/doc/python3/html/library/functions.html#int>]

        classmethod get_total_cpu_time()
            Gives the total cpu time, including the children.

            Return type
                float </usr/share/doc/python3/html/library/functions.html#float>

    toil.lib.resources.glob(glob_pattern, directoryname)
        Walks through a directory and its subdirectories looking for files matching the glob_pattern and returns a list=[].

        Parameters

            •   directoryname (toil.lib.misc.StrPath) -- Any accessible folder name on the filesystem.

            •   glob_pattern (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- A string like *.txt, which would find all text files.

        Returns
            A list=[] of absolute filepaths matching the glob pattern.

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

  toil.lib.retry

This file holds the retry() decorator function and RetryCondition object.

retry() can be used to decorate any function based on the list of errors one wishes to retry on.

This list of errors can contain normal Exception objects, and/or RetryCondition objects wrapping Exceptions to include additional conditions.

For example, retrying on a one Exception (HTTPError):

            from toil.lib.web import web_session
            from requests.exceptions import HTTPError

            @retry(errors=[HTTPError])
            def update_my_wallpaper():
                return web_session.get('https://www.deviantart.com/')

Or:

            from toil.lib.web import web_session
            from requests.exceptions import HTTPError

            @retry(errors=[HTTPError, ValueError])
            def update_my_wallpaper():
                return web_session.get('https://www.deviantart.com/')

The examples above will retry for the default interval on any errors specified the "errors=" arg list.

To retry on specifically 500/502/503/504 errors, you could specify an ErrorCondition object instead, for example:

            from toil.lib.web import web_session
            from requests.exceptions import HTTPError

            @retry(errors=[
                ErrorCondition(
                           error=HTTPError,
                           error_codes=[500, 502, 503, 504]
                       )])
            def update_my_wallpaper():
                return web_session.get('https://www.deviantart.com/')

To retry on specifically errors containing the phrase "NotFound":

            from toil.lib.web import web_session
            from requests.exceptions import HTTPError

            @retry(errors=[
                ErrorCondition(
                    error=HTTPError,
                    error_message_must_include="NotFound"
                )])
            def update_my_wallpaper():
                return web_session.get('https://www.deviantart.com/')

To retry on all HTTPError errors EXCEPT an HTTPError containing the phrase "NotFound":

            from toil.lib.web import web_session
            from requests.exceptions import HTTPError

            @retry(errors=[
                HTTPError,
                ErrorCondition(
                           error=HTTPError,
                           error_message_must_include="NotFound",
                           retry_on_this_condition=False
                       )])
            def update_my_wallpaper():
                return web_session.get('https://www.deviantart.com/')

To retry on boto3's specific status errors, an example of the implementation is:

            import boto3
            from botocore.exceptions import ClientError

            @retry(errors=[
                ErrorCondition(
                           error=ClientError,
                           boto_error_codes=["BucketNotFound"]
                       )])
            def boto_bucket(bucket_name):
                boto_session = boto3.session.Session()
                s3_resource = boto_session.resource('s3')
                return s3_resource.Bucket(bucket_name)

Any combination of these will also work, provided the codes are matched to the correct exceptions. A ValueError will not return a 404, for example.

The retry function as a decorator should make retrying functions easier and clearer It also encourages smaller independent functions, as opposed to lumping many different things that may need to be retried on different conditions in the same function.

The ErrorCondition object tries to take some of the heavy lifting of writing specific retry conditions and boil it down to an API that covers all common use-cases without the user having to write any new bespoke functions.

Use-cases covered currently:

    1.  Retrying on a normal error, like a KeyError.

    2.  Retrying on HTTP error codes (use ErrorCondition).

    3.  Retrying on boto 3's specific status errors, like "BucketNotFound" (use ErrorCondition).

    4.  Retrying when an error message contains a certain phrase (use ErrorCondition).

    5.  Explicitly NOT retrying on a condition (use ErrorCondition).

If new functionality is needed, it's currently best practice in Toil to add functionality to the ErrorCondition itself rather than making a new custom retry method.

  Attributes
SUPPORTED_HTTP_ERRORS |
kubernetes |
botocore |
logger |
RT |
DEFAULT_DELAYS |
DEFAULT_TIMEOUT |
E |
retry_flaky_test |

  Classes
ErrorCondition | A wrapper describing an error condition.

  Functions
retry([intervals, infinite_retries, errors, ...]) | Retry a function if it fails with any Exception defined in "errors".
return_status_code(e) |
get_error_code(e) | Get the error code name from a Boto 2 or 3 error, or compatible types.
get_error_message(e) | Get the error message string from a Boto 2 or 3 error, or compatible types.
get_error_status(e) | Get the HTTP status code from a compatible source.
get_error_body(e) | Get the body from a Boto 2 or 3 error, or compatible types.
meets_error_message_condition(e, error_message) |
meets_error_code_condition(e, error_codes) | These are expected to be normal HTTP error codes, like 404 or 500.
meets_boto_error_code_condition(e, boto_error_codes) | These are expected to be AWS's custom error aliases, like 'BucketNotFound' or 'AccessDenied'.
error_meets_conditions(e, error_conditions) |
old_retry([delays, timeout, predicate]) | Deprecated.

  Module Contents

    toil.lib.retry.SUPPORTED_HTTP_ERRORS

    toil.lib.retry.kubernetes = None

    toil.lib.retry.botocore = None

    toil.lib.retry.logger

    class toil.lib.retry.ErrorCondition(error=None, error_codes=None, boto_error_codes=None, error_message_must_include=None, retry_on_this_condition=True)
        A wrapper describing an error condition.

        ErrorCondition events may be used to define errors in more detail to determine whether to retry.

        Parameters

            •   error (Optional[type </usr/share/doc/python3/html/library/functions.html#type>[BaseException </usr/share/doc/python3/html/library/exceptions.html#BaseException>]])

            •   error_codes (list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>])

            •   boto_error_codes (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   error_message_must_include (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   retry_on_this_condition (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        error = None

        error_codes = None

        boto_error_codes = None

        error_message_must_include = None

        retry_on_this_condition = True

    toil.lib.retry.RT

    toil.lib.retry.retry(intervals=None, infinite_retries=False, errors=None, log_message=None, prepare=None)
        Retry a function if it fails with any Exception defined in "errors".

        Does so every x seconds, where x is defined by a list of numbers (ints or floats) in "intervals". Also accepts ErrorCondition events for more detailed retry attempts.

        Parameters

            •   intervals (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>]) -- A list of times in seconds we keep retrying until returning failure. Defaults to retrying with the following exponential back-off before failing: 1s, 1s, 2s, 4s, 8s, 16s

            •   infinite_retries (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If this is True, reset the intervals when they run out. Defaults to: False.

            •   errors (Optional[collections.abc.Sequence </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Sequence>[Union[ErrorCondition <#toil.lib.retry.ErrorCondition>, type </usr/share/doc/python3/html/library/functions.html#type>[Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>]]]]) --

                A list of exceptions OR ErrorCondition objects to catch and retry on. ErrorCondition objects describe more detailed error event conditions than a plain error. An ErrorCondition specifies: - Exception (required) - Error codes that must match to be retried (optional; defaults to not checking) - A string that must be in the error message to be retried (optional; defaults to not checking) - A bool that can be set to False to always error on this condition.

                If not specified, this will default to a generic Exception.

            •   log_message (Optional[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[Callable, str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) -- Optional tuple of ("log/print function()", "message string") that will precede each attempt.

            •   prepare (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[Callable]]) -- Optional list of functions to call, with the function's arguments, between retries, to reset state.

        Returns
            The result of the wrapped function or raise.

        Return type
            Callable[[Callable[Ellipsis, RT]], Callable[Ellipsis, RT]]

    toil.lib.retry.return_status_code(e)

    toil.lib.retry.get_error_code(e)
        Get the error code name from a Boto 2 or 3 error, or compatible types.

        Returns empty string for other errors.

        Parameters
            e (Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.retry.get_error_message(e)
        Get the error message string from a Boto 2 or 3 error, or compatible types.

        Note that error message conditions also check more than this; this function does not fall back to the traceback for incompatible types.

        Parameters
            e (Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.retry.get_error_status(e)
        Get the HTTP status code from a compatible source.

        Such as a Boto 2 or 3 error, kubernetes.client.rest.ApiException, http.client.HTTPException, urllib3.exceptions.HTTPError, requests.exceptions.HTTPError, urllib.error.HTTPError, or compatible type

        Returns 0 from other errors.

        Parameters
            e (Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>)

        Return type
            int </usr/share/doc/python3/html/library/functions.html#int>

    toil.lib.retry.get_error_body(e)
        Get the body from a Boto 2 or 3 error, or compatible types.

        Returns the code and message if the error does not have a body.

        Parameters
            e (Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.retry.meets_error_message_condition(e, error_message)

        Parameters

            •   e (Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>)

            •   error_message (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

    toil.lib.retry.meets_error_code_condition(e, error_codes)
        These are expected to be normal HTTP error codes, like 404 or 500.

        Parameters

            •   e (Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>)

            •   error_codes (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>]])

    toil.lib.retry.meets_boto_error_code_condition(e, boto_error_codes)
        These are expected to be AWS's custom error aliases, like 'BucketNotFound' or 'AccessDenied'.

        Parameters

            •   e (Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>)

            •   boto_error_codes (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

    toil.lib.retry.error_meets_conditions(e, error_conditions)

    toil.lib.retry.DEFAULT_DELAYS = (0, 1, 1, 4, 16, 64)

    toil.lib.retry.DEFAULT_TIMEOUT = 300

    toil.lib.retry.E

    toil.lib.retry.old_retry(delays=DEFAULT_DELAYS, timeout=DEFAULT_TIMEOUT, predicate=lambda e: ...)
        Deprecated.

        Retry an operation while the failure matches a given predicate and until a given timeout expires, waiting a given amount of time in between attempts. This function is a generator that yields contextmanagers. See doctests below for example usage.

        Parameters

            •   delays (Iterable[float </usr/share/doc/python3/html/library/functions.html#float>]) -- an interable yielding the time in seconds to wait before each retried attempt, the last element of the iterable will be repeated.

            •   timeout (float </usr/share/doc/python3/html/library/functions.html#float>) -- a overall timeout that should not be exceeded for all attempts together. This is a best-effort mechanism only and it won't abort an ongoing attempt, even if the timeout expires during that attempt.

            •   predicate (Callable[[Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>],bool </usr/share/doc/python3/html/library/functions.html#bool>]) -- a unary callable returning True if another attempt should be made to recover from the given exception. The default value for this parameter will prevent any retries!

        Returns
            a generator yielding context managers, one per attempt

        Return type
            Iterator

    Retry for a limited amount of time:

        >>> true = lambda _:True
        >>> false = lambda _:False
        >>> i = 0
        >>> for attempt in old_retry( delays=[0], timeout=.1, predicate=true ):
        ...     with attempt:
        ...         i += 1
        ...         raise RuntimeError('foo')
        Traceback (most recent call last):
        ...
        RuntimeError: foo
        >>> i > 1
        True

    If timeout is 0, do exactly one attempt:

        >>> i = 0
        >>> for attempt in old_retry( timeout=0 ):
        ...     with attempt:
        ...         i += 1
        ...         raise RuntimeError( 'foo' )
        Traceback (most recent call last):
        ...
        RuntimeError: foo
        >>> i
        1

    Don't retry on success:

        >>> i = 0
        >>> for attempt in old_retry( delays=[0], timeout=.1, predicate=true ):
        ...     with attempt:
        ...         i += 1
        >>> i
        1

    Don't retry on unless predicate returns True:

        >>> i = 0
        >>> for attempt in old_retry( delays=[0], timeout=.1, predicate=false):
        ...     with attempt:
        ...         i += 1
        ...         raise RuntimeError( 'foo' )
        Traceback (most recent call last):
        ...
        RuntimeError: foo
        >>> i
        1

    toil.lib.retry.retry_flaky_test

  toil.lib.threading

  Attributes
logger |
current_process_name_lock |
current_process_name_for |

  Classes
ExceptionalThread | A thread whose join() method re-raises exceptions raised during run(). While join() is
LastProcessStandingArena | Class that lets a bunch of processes detect and elect a last process

  Functions
ensure_filesystem_lockable(path[, timeout, hint]) | Make sure that the filesystem used at the given path is one where locks are safe to use.
safe_lock(fd[, block, shared]) | Get an fcntl lock, while retrying on IO errors.
safe_unlock_and_close(fd) | Release an fcntl lock and close the file descriptor, while handling fcntl IO errors.
cpu_count() | Get the rounded-up integer number of whole CPUs available.
collect_process_name_garbage() | Delete all the process names that point to files that don't exist anymore
destroy_all_process_names() | Delete all our process name files because our process is going away.
get_process_name(base_dir) | Return the name of the current process. Like a PID but visible between
process_name_exists(base_dir, name) | Return true if the process named by the given name (from process_name) exists, and false otherwise.
global_mutex(base_dir, mutex) | Context manager that locks a mutex. The mutex is identified by the given

  Module Contents

    toil.lib.threading.logger

    toil.lib.threading.ensure_filesystem_lockable(path, timeout=30, hint=None)
        Make sure that the filesystem used at the given path is one where locks are safe to use.

        File locks are not safe to use on Ceph. See <<https://github.com/DataBiosphere/toil/issues/4972>>.

        Raises an exception if the filesystem is detected as one where using locks is known to trigger bugs in the filesystem implementation. Also raises an exception if the given path does not exist, or if attempting to determine the filesystem type takes more than the timeout in seconds.

        If the filesystem type cannot be determined, does nothing.

        Parameters

            •   hint (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Extra text to include in an error, if raised, telling the user how to change the offending path.

            •   path (toil.lib.misc.StrPath)

            •   timeout (float </usr/share/doc/python3/html/library/functions.html#float>)

        Return type
            None

    toil.lib.threading.safe_lock(fd, block=True, shared=False)
        Get an fcntl lock, while retrying on IO errors.

        Raises OSError with EACCES or EAGAIN when a nonblocking lock is not immediately available.

        Parameters

            •   fd (int </usr/share/doc/python3/html/library/functions.html#int>)

            •   block (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            •   shared (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            None

    toil.lib.threading.safe_unlock_and_close(fd)
        Release an fcntl lock and close the file descriptor, while handling fcntl IO errors.

        Parameters
            fd (int </usr/share/doc/python3/html/library/functions.html#int>)

        Return type
            None

    class toil.lib.threading.ExceptionalThread(group=None, target=None, name=None, args=(), kwargs=None, *, daemon=None, context=None)
        Bases: threading.Thread </usr/share/doc/python3/html/library/threading.html#threading.Thread>

        A thread whose join() method re-raises exceptions raised during run(). While join() is idempotent, the exception is only during the first invocation of join() that successfully joined the thread. If join() times out, no exception will be re reraised even though an exception might already have occurred in run().

        When subclassing this thread, override tryRun() instead of run().

            >>> def f():
            ...     assert 0
            >>> t = ExceptionalThread(target=f)
            >>> t.start()
            >>> t.join()
            Traceback (most recent call last):
            ...
            AssertionError

            >>> class MyThread(ExceptionalThread):
            ...     def tryRun( self ):
            ...         assert 0
            >>> t = MyThread()
            >>> t.start()
            >>> t.join()
            Traceback (most recent call last):
            ...
            AssertionError

        exc_info = None

        run()
            Method representing the thread's activity.

            You may override this method in a subclass. The standard run() method invokes the callable object passed to the object's constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.

            Return type
                None

        tryRun()

            Return type
                None

        join(*args, **kwargs)
            Wait until the thread terminates.

            This blocks the calling thread until the thread whose join() method is called terminates -- either normally or through an unhandled exception or until the optional timeout occurs.

            When the timeout argument is present and not None, it should be a floating-point number specifying a timeout for the operation in seconds (or fractions thereof). As join() always returns None, you must call is_alive() after join() to decide whether a timeout happened -- if the thread is still alive, the join() call timed out.

            When the timeout argument is not present or None, the operation will block until the thread terminates.

            A thread can be join()ed many times.

            join() raises a RuntimeError if an attempt is made to join the current thread as that would cause a deadlock. It is also an error to join() a thread before it has been started and attempts to do so raises the same exception.

            Parameters

                •   args (Optional[float </usr/share/doc/python3/html/library/functions.html#float>])

                •   kwargs (Optional[float </usr/share/doc/python3/html/library/functions.html#float>])

            Return type
                None

    toil.lib.threading.cpu_count()
        Get the rounded-up integer number of whole CPUs available.

        Counts hyperthreads as CPUs.

        Uses the system's actual CPU count, or the current v1 cgroup's quota per period, if the quota is set.

        Ignores the cgroup's cpu shares value, because it's extremely difficult to interpret. See <https://github.com/kubernetes/kubernetes/issues/81021>.

        Caches result for efficiency.

        Returns
            Integer count of available CPUs, minimum 1.

        Return type
            int </usr/share/doc/python3/html/library/functions.html#int>

    toil.lib.threading.current_process_name_lock

    toil.lib.threading.current_process_name_for: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.lib.threading.collect_process_name_garbage()
        Delete all the process names that point to files that don't exist anymore (because the work directory was temporary and got cleaned up). This is known to happen during the tests, which get their own temp directories.

        Caller must hold current_process_name_lock.

        Return type
            None

    toil.lib.threading.destroy_all_process_names()
        Delete all our process name files because our process is going away.

        We let all our FDs get closed by the process death.

        We assume there is nobody else using the system during exit to race with.

        Return type
            None

    toil.lib.threading.get_process_name(base_dir)
        Return the name of the current process. Like a PID but visible between containers on what to Toil appears to be a node.

        Parameters
            base_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Base directory to work in. Defines the shared namespace.

        Returns
            Process's assigned name

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.threading.process_name_exists(base_dir, name)
        Return true if the process named by the given name (from process_name) exists, and false otherwise.

        Can see across container boundaries using the given node workflow directory.

        Parameters

            •   base_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Base directory to work in. Defines the shared namespace.

            •   name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Process's name to poll

        Returns
            True if the named process is still alive, and False otherwise.

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.threading.global_mutex(base_dir, mutex)
        Context manager that locks a mutex. The mutex is identified by the given name, and scoped to the given directory. Works across all containers that have access to the given diectory. Mutexes held by dead processes are automatically released.

        Only works between processes, NOT between threads.

        Parameters

            •   base_dir (toil.lib.misc.StrPath) -- Base directory to work in. Defines the shared namespace.

            •   mutex (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Mutex to lock. Must be a permissible path component.

        Return type
            collections.abc.Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[None]

    class toil.lib.threading.LastProcessStandingArena(base_dir, name)
        Class that lets a bunch of processes detect and elect a last process standing.

        Process enter and leave (sometimes due to sudden existence failure). We guarantee that the last process to leave, if it leaves properly, will get a chance to do some cleanup. If new processes try to enter during the cleanup, they will be delayed until after the cleanup has happened and the previous "last" process has finished leaving.

        The user is responsible for making sure you always leave if you enter! Consider using a try/finally; this class is not a context manager.

        Parameters

            •   base_dir (toil.lib.misc.StrPath)

            •   name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        base_dir = ''

        mutex

        lockfileDir

        lockfileFD = None

        lockfileName = None

        enter()
            This process is entering the arena. If cleanup is in progress, blocks until it is finished.

            You may not enter the arena again before leaving it.

            Return type
                None

        leave()
            This process is leaving the arena. If this process happens to be the last process standing, yields something, with other processes blocked from joining the arena until the loop body completes and the process has finished leaving. Otherwise, does not yield anything.

            Should be used in a loop:

                    for _ in arena.leave():
                        # If we get here, we were the last process. Do the cleanup pass

            Return type
                collections.abc.Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[bool </usr/share/doc/python3/html/library/functions.html#bool>]

  toil.lib.throttle

  Classes
LocalThrottle | A thread-safe rate limiter that throttles each thread independently. Can be used as a
throttle | A context manager for ensuring that the execution of its body takes at least a given amount

  Module Contents

    class toil.lib.throttle.LocalThrottle(min_interval)
        A thread-safe rate limiter that throttles each thread independently. Can be used as a function or method decorator or as a simple object, via its .throttle() method.

        The use as a decorator is deprecated in favor of throttle().

        Parameters
            min_interval (int </usr/share/doc/python3/html/library/functions.html#int>)

        min_interval

        per_thread

        throttle(wait=True)
            If the wait parameter is True, this method returns True after suspending the current thread as necessary to ensure that no less than the configured minimum interval has passed since the last invocation of this method in the current thread returned True.

            If the wait parameter is False, this method immediatley returns True (if at least the configured minimum interval has passed since the last time this method returned True in the current thread) or False otherwise.

            Parameters
                wait (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        __call__(function)

    class toil.lib.throttle.throttle(min_interval)
        A context manager for ensuring that the execution of its body takes at least a given amount of time, sleeping if necessary. It is a simpler version of LocalThrottle if used as a decorator.

        Ensures that body takes at least the given amount of time.

            >>> start = time.time()
            >>> with throttle(1):
            ...     pass
            >>> 1 <= time.time() - start <= 1.1
            True

        Ditto when used as a decorator.

            >>> @throttle(1)
            ... def f():
            ...     pass
            >>> start = time.time()
            >>> f()
            >>> 1 <= time.time() - start <= 1.1
            True

        If the body takes longer by itself, don't throttle.

            >>> start = time.time()
            >>> with throttle(1):
            ...     time.sleep(2)
            >>> 2 <= time.time() - start <= 2.1
            True

        Ditto when used as a decorator.

            >>> @throttle(1)
            ... def f():
            ...     time.sleep(2)
            >>> start = time.time()
            >>> f()
            >>> 2 <= time.time() - start <= 2.1
            True

        If an exception occurs, don't throttle.

            >>> start = time.time()
            >>> try:
            ...     with throttle(1):
            ...         raise ValueError('foo')
            ... except ValueError:
            ...     end = time.time()
            ...     raise
            Traceback (most recent call last):
            ...
            ValueError: foo
            >>> 0 <= end - start <= 0.1
            True

        Ditto when used as a decorator.

            >>> @throttle(1)
            ... def f():
            ...     raise ValueError('foo')
            >>> start = time.time()
            >>> try:
            ...     f()
            ... except ValueError:
            ...     end = time.time()
            ...     raise
            Traceback (most recent call last):
            ...
            ValueError: foo
            >>> 0 <= end - start <= 0.1
            True

        Parameters
            min_interval (Union[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>])

        min_interval

        __enter__()

        __exit__(exc_type, exc_val, exc_tb)

        __call__(function)

  toil.lib.trs

Contains functions for integrating Toil with GA4GH Tool Registry Service servers, for fetching workflows.

  Attributes
logger |
TRS_ROOT |

  Functions
is_trs_workflow(workflow) | Returns True if a workflow string smells like TRS.
extract_trs_spec(workflow) | Parse a Dockstore workflow URL or TSR ID to a string that is definitely a TRS ID.
parse_trs_spec(trs_spec) | Parse a TRS ID to workflow and optional version.
compose_trs_spec(trs_workflow_id, trs_version) | Compose a TRS ID from a workflow ID and version ID.
find_workflow(workflow[, supported_languages]) | Given a Dockstore URL or TRS identifier, get the root WDL or CWL URL for the workflow, along with the TRS workflow ID and version.
fetch_workflow(trs_workflow_id, trs_version, language) | Returns a URL or local path to a workflow's primary descriptor file.
resolve_workflow(workflow[, supported_languages]) | Find the real workflow URL or filename from a command line argument.

  Module Contents

    toil.lib.trs.logger

    toil.lib.trs.TRS_ROOT = 'https://dockstore.org'

    toil.lib.trs.is_trs_workflow(workflow)
        Returns True if a workflow string smells like TRS.

        Detects Dockstore page URLs and strings that could be Dockstore TRS IDs.

        Parameters
            workflow (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.lib.trs.extract_trs_spec(workflow)
        Parse a Dockstore workflow URL or TSR ID to a string that is definitely a TRS ID.

        Parameters
            workflow (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.trs.parse_trs_spec(trs_spec)
        Parse a TRS ID to workflow and optional version.

        Parameters
            trs_spec (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

    toil.lib.trs.compose_trs_spec(trs_workflow_id, trs_version)
        Compose a TRS ID from a workflow ID and version ID.

        Parameters

            •   trs_workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   trs_version (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.trs.find_workflow(workflow, supported_languages=None)
        Given a Dockstore URL or TRS identifier, get the root WDL or CWL URL for the workflow, along with the TRS workflow ID and version.

        Accepts inputs like:

                •   <https://dockstore.org/workflows/github.com/dockstore-testing/md5sum-checker:master?tab=info>

                •   #workflow/github.com/dockstore-testing/md5sum-checker

    Assumes the input is actually one of the supported formats. See is_trs_workflow().

    TODO: Needs to handle multi-workflow files if Dockstore can.

        Raises

            •   FileNotFoundError </usr/share/doc/python3/html/library/exceptions.html#FileNotFoundError> -- if the workflow or version doesn't exist.

            •   ValueError </usr/share/doc/python3/html/library/exceptions.html#ValueError> -- if the version is not specified but cannot be automatically determined.

        Parameters

            •   workflow (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   supported_languages (Optional[set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

        Return type
            tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.lib.trs.fetch_workflow(trs_workflow_id, trs_version, language)
        Returns a URL or local path to a workflow's primary descriptor file.

        The file will be in context with its required files so it can actually run.

        Raises
            FileNotFoundError </usr/share/doc/python3/html/library/exceptions.html#FileNotFoundError> -- if the workflow or version doesn't exist.

        Parameters

            •   trs_workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   trs_version (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   language (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lib.trs.resolve_workflow(workflow, supported_languages=None)
        Find the real workflow URL or filename from a command line argument.

        Transform a workflow URL or path that might actually be a Dockstore page URL or TRS specifier to an actual URL or path to a workflow document, and optional TRS specifier.

        Accepts inputs like

                •   <https://dockstore.org/workflows/github.com/dockstore-testing/md5sum-checker:master?tab=info>

                •   #workflow/github.com/dockstore-testing/md5sum-checker

                •   ./local.cwl

                •   <https://example.com/~myuser/workflow/main.cwl>

        Raises
            FileNotFoundError </usr/share/doc/python3/html/library/exceptions.html#FileNotFoundError> -- if the workflow or version should be in Dockstore but doesn't seem to exist.

        Parameters

            •   workflow (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   supported_languages (Optional[set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

        Return type
            tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

  toil.lib.url

  Attributes
logger |

  Exceptions
ProxyConnectionError | Dummy class.

  Classes
URLAccess | Widget for accessing external storage (URLs).

  Functions
file_job_store_factory() |
google_job_store_factory() |
aws_job_store_factory() |
job_store_support_factory() |

  Module Contents

    exception toil.lib.url.ProxyConnectionError
        Bases: BaseException </usr/share/doc/python3/html/library/exceptions.html#BaseException>

        Dummy class.

    toil.lib.url.logger

    class toil.lib.url.URLAccess
        Widget for accessing external storage (URLs).

        classmethod url_exists(src_uri)
            Return True if the file at the given URI exists, and False otherwise.

            May raise an error if file existence cannot be determined.

            Parameters
                src_uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- URL that points to a file or object in the storage mechanism of a supported URL scheme e.g. a blob in an AWS s3 bucket.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        classmethod get_size(src_uri)
            Get the size in bytes of the file at the given URL, or None if it cannot be obtained.

            Parameters
                src_uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- URL that points to a file or object in the storage mechanism of a supported URL scheme e.g. a blob in an AWS s3 bucket.

            Return type
                Optional[int </usr/share/doc/python3/html/library/functions.html#int>]

        classmethod get_is_directory(src_uri)
            Return True if the thing at the given URL is a directory, and False if it is a file. The URL may or may not end in '/'.

            Parameters
                src_uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        classmethod list_url(src_uri)
            List the directory at the given URL. Returned path components can be joined with '/' onto the passed URL to form new URLs. Those that end in '/' correspond to directories. The provided URL may or may not end with '/'.

            Currently supported schemes are:

                    •

                        's3' for objects in Amazon S3
                            e.g. s3://bucket/prefix/

                    •

                        'file' for local files
                            e.g. <file:///local/dir/path/>

            Parameters
                src_uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- URL that points to a directory or prefix in the storage mechanism of a supported URL scheme e.g. a prefix in an AWS s3 bucket.

            Returns
                A list of URL components in the given directory, already URL-encoded.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        classmethod read_from_url(src_uri, writable)
            Read the given URL and write its content into the given writable stream.

            Raises FileNotFoundError if the URL doesn't exist.

            Returns
                The size of the file in bytes and whether the executable permission bit is set

            Parameters

                •   src_uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   writable (IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>])

            Return type
                tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[int </usr/share/doc/python3/html/library/functions.html#int>, bool </usr/share/doc/python3/html/library/functions.html#bool>]

        classmethod open_url(src_uri)
            Read from the given URI.

            Raises FileNotFoundError if the URL doesn't exist.

            Has a readable stream interface, unlike read_from_url() which takes a writable stream.

            Parameters
                src_uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]

    toil.lib.url.file_job_store_factory()

        Return type
            type </usr/share/doc/python3/html/library/functions.html#type>[URLAccess <#toil.lib.url.URLAccess>]

    toil.lib.url.google_job_store_factory()

        Return type
            type </usr/share/doc/python3/html/library/functions.html#type>[URLAccess <#toil.lib.url.URLAccess>]

    toil.lib.url.aws_job_store_factory()

        Return type
            type </usr/share/doc/python3/html/library/functions.html#type>[URLAccess <#toil.lib.url.URLAccess>]

    toil.lib.url.job_store_support_factory()

        Return type
            type </usr/share/doc/python3/html/library/functions.html#type>[URLAccess <#toil.lib.url.URLAccess>]

  toil.lib.web

Contains functions for making web requests with Toil.

All web requests should go through this module, to make sure they use the right user agent. >>> httpserver = getfixture("httpserver") >>> handler = httpserver.expect_request("/path").respond_with_json({}) >>> from toil.lib.web import web_session >>> web_session.get(httpserver.url_for("/path")) <Response [200]>

  Attributes
web_session |

  Module Contents

    toil.lib.web.web_session

  toil.options

  Submodules

  toil.options.common

  Attributes
logger |
defaultTargetTime |
SYS_MAX_SIZE |
JOBSTORE_HELP |

  Functions
parse_set_env(l) | Parse a list of strings of the form "NAME=VALUE" or just "NAME" into a dictionary.
parse_str_list(s) |
parse_int_list(s) |
iC(min_value[, max_value]) | Returns a function that checks if a given int is in the given half-open interval.
fC(minValue[, maxValue]) | Returns a function that checks if a given float is in the given half-open interval.
parse_accelerator_list(specs) | Parse a string description of one or more accelerator requirements.
parseBool(val) |
make_open_interval_action(min[, max]) | Returns an argparse action class to check if the input is within the given half-open interval.
parse_jobstore(jobstore_uri) | Turn the jobstore string into it's corresponding URI
add_base_toil_options(parser[, jobstore_as_flag, cwl]) | Add base Toil command line options to the parser.

  Module Contents

    toil.options.common.logger

    toil.options.common.defaultTargetTime = 1800

    toil.options.common.SYS_MAX_SIZE = 9223372036854775807

    toil.options.common.parse_set_env(l)
        Parse a list of strings of the form "NAME=VALUE" or just "NAME" into a dictionary.

        Strings of the latter from will result in dictionary entries whose value is None.

            >>> parse_set_env([])
            {}
            >>> parse_set_env(['a'])
            {'a': None}
            >>> parse_set_env(['a='])
            {'a': ''}
            >>> parse_set_env(['a=b'])
            {'a': 'b'}
            >>> parse_set_env(['a=a', 'a=b'])
            {'a': 'b'}
            >>> parse_set_env(['a=b', 'c=d'])
            {'a': 'b', 'c': 'd'}
            >>> parse_set_env(['a=b=c'])
            {'a': 'b=c'}
            >>> parse_set_env([''])
            Traceback (most recent call last):
            ...
            ValueError: Empty name
            >>> parse_set_env(['=1'])
            Traceback (most recent call last):
            ...
            ValueError: Empty name

        Parameters
            l (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

    toil.options.common.parse_str_list(s)

        Parameters
            s (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.options.common.parse_int_list(s)

        Parameters
            s (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>]

    toil.options.common.iC(min_value, max_value=None)
        Returns a function that checks if a given int is in the given half-open interval.

        Parameters

            •   min_value (int </usr/share/doc/python3/html/library/functions.html#int>)

            •   max_value (Optional[int </usr/share/doc/python3/html/library/functions.html#int>])

        Return type
            Callable[[int </usr/share/doc/python3/html/library/functions.html#int>], bool </usr/share/doc/python3/html/library/functions.html#bool>]

    toil.options.common.fC(minValue, maxValue=None)
        Returns a function that checks if a given float is in the given half-open interval.

        Parameters

            •   minValue (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxValue (Optional[float </usr/share/doc/python3/html/library/functions.html#float>])

        Return type
            Callable[[float </usr/share/doc/python3/html/library/functions.html#float>], bool </usr/share/doc/python3/html/library/functions.html#bool>]

    toil.options.common.parse_accelerator_list(specs)
        Parse a string description of one or more accelerator requirements.

        Parameters
            specs (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.job.AcceleratorRequirement <#toil.job.AcceleratorRequirement>]

    toil.options.common.parseBool(val)

        Parameters
            val (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.options.common.make_open_interval_action(min, max=None)
        Returns an argparse action class to check if the input is within the given half-open interval. ex: Provided value to argparse must be within the interval [min, max) Types of min and max must be the same (max may be None)

        Parameters

            •   min (Union[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>]) -- float/int

            •   max (Optional[Union[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>]]) -- optional float/int

        Returns
            argparse action class

        Return type
            type </usr/share/doc/python3/html/library/functions.html#type>[argparse.Action </usr/share/doc/python3/html/library/argparse.html#argparse.Action>]

    toil.options.common.parse_jobstore(jobstore_uri)
        Turn the jobstore string into it's corresponding URI ex: /path/to/jobstore -> <file:/path/to/jobstore>

        If the jobstore string already is a URI, return the jobstore: aws:/path/to/jobstore -> aws:/path/to/jobstore :param jobstore_uri: string of the jobstore :return: URI of the jobstore

        Parameters
            jobstore_uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.options.common.JOBSTORE_HELP = Multiline-String

                """The location of the job store for the workflow.  A job store holds persistent information about the jobs, stats, and files in a workflow. If the workflow is run with a distributed batch system, the job store must be accessible by all worker nodes. Depending on the desired job store implementation, the location should be formatted according to one of the following schemes:

                file:<path> where <path> points to a directory on the file system

                aws:<region>:<prefix> where <region> is the name of an AWS region like us-west-2 and <prefix> will be prepended to the names of any top-level AWS resources in use by job store, e.g. S3 buckets.

                 google:<project_id>:<prefix> TODO: explain

                For backwards compatibility, you may also specify ./foo (equivalent to file:./foo or just file:foo) or /bar (equivalent to file:/bar)."""

    toil.options.common.add_base_toil_options(parser, jobstore_as_flag=False, cwl=False)
        Add base Toil command line options to the parser. :param parser: Argument parser to add options to :param jobstore_as_flag: make the job store option a --jobStore flag instead of a required jobStore positional argument. :param cwl: whether CWL should be included or not

        Parameters

            •   parser (argparse.ArgumentParser </usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser>)

            •   jobstore_as_flag (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            •   cwl (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            None

  toil.options.cwl

  Functions
add_cwl_options(parser[, suppress]) | Add CWL options to the parser. This only adds nonpositional CWL arguments.

  Module Contents

    toil.options.cwl.add_cwl_options(parser, suppress=True)
        Add CWL options to the parser. This only adds nonpositional CWL arguments.

        Parameters

            •   parser (argparse.ArgumentParser </usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser>) -- Parser to add options to

            •   suppress (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Suppress help output

        Returns
            None

        Return type
            None

  toil.options.runner

  Functions
add_runner_options(parser[, cwl, wdl]) | Add to the WDL or CWL runners options that are shared or the same between runners

  Module Contents

    toil.options.runner.add_runner_options(parser, cwl=False, wdl=False)
        Add to the WDL or CWL runners options that are shared or the same between runners :param parser: parser to add arguments to :param cwl: bool :param wdl: bool :return: None

        Parameters

            •   parser (argparse.ArgumentParser </usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser>)

            •   cwl (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            •   wdl (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            None

  toil.options.wdl

  Functions
add_wdl_options(parser[, suppress]) | Add WDL options to a parser. This only adds nonpositional WDL arguments

  Module Contents

    toil.options.wdl.add_wdl_options(parser, suppress=True)
        Add WDL options to a parser. This only adds nonpositional WDL arguments :param parser: Parser to add options to :param suppress: Suppress help output :return: None

        Parameters

            •   parser (argparse.ArgumentParser </usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser>)

            •   suppress (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            None

  toil.provisioners

  Submodules

  toil.provisioners.abstractProvisioner

  Attributes
a_short_time |
logger |

  Exceptions
ManagedNodesNotSupportedException | Raised when attempting to add managed nodes (which autoscale up and down by

  Classes
Shape | Represents a job or a node's "shape", in terms of the dimensions of memory, cores, disk and
AbstractProvisioner | Interface for provisioning worker nodes to use in a Toil cluster.

  Module Contents

    toil.provisioners.abstractProvisioner.a_short_time = 5

    toil.provisioners.abstractProvisioner.logger

    exception toil.provisioners.abstractProvisioner.ManagedNodesNotSupportedException
        Bases: RuntimeError </usr/share/doc/python3/html/library/exceptions.html#RuntimeError>

        Raised when attempting to add managed nodes (which autoscale up and down by themselves, without the provisioner doing the work) to a provisioner that does not support them.

        Polling with this and try/except is the Right Way to check if managed nodes are available from a provisioner.

    class toil.provisioners.abstractProvisioner.Shape(wallTime, memory, cores, disk, preemptible)
        Represents a job or a node's "shape", in terms of the dimensions of memory, cores, disk and wall-time allocation.

        The wallTime attribute stores the number of seconds of a node allocation, e.g. 3600 for AWS. FIXME: and for jobs?

        The memory and disk attributes store the number of bytes required by a job (or provided by a node) in RAM or on disk (SSD or HDD), respectively.

        Parameters

            •   wallTime (Union[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>])

            •   memory (int </usr/share/doc/python3/html/library/functions.html#int>)

            •   cores (Union[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>])

            •   disk (int </usr/share/doc/python3/html/library/functions.html#int>)

            •   preemptible (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        wallTime

        memory

        cores

        disk

        preemptible

        __eq__(other)

            Parameters
                other (Any)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        greater_than(other)

            Parameters
                other (Any)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        __gt__(other)

            Parameters
                other (Any)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        __repr__()

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        __str__()

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        __hash__()

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

    class toil.provisioners.abstractProvisioner.AbstractProvisioner(clusterName=None, clusterType='mesos', zone=None, nodeStorage=50, nodeStorageOverrides=None, enable_fuse=False)
        Bases: abc.ABC </usr/share/doc/python3/html/library/abc.html#abc.ABC>

        Interface for provisioning worker nodes to use in a Toil cluster.

        Parameters

            •   clusterName (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   clusterType (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   zone (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   nodeStorage (int </usr/share/doc/python3/html/library/functions.html#int>)

            •   nodeStorageOverrides (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

            •   enable_fuse (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        LEADER_HOME_DIR = '/root/'

        cloud: str </usr/share/doc/python3/html/library/stdtypes.html#str> = None

        clusterName = None

        clusterType = 'mesos'

        enable_fuse = False

        abstractmethod supportedClusterTypes()
            Get all the cluster types that this provisioner implementation supports.

            Return type
                set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        abstractmethod createClusterSettings()
            Initialize class for a new cluster, to be deployed, when running outside the cloud.

        abstractmethod readClusterSettings()
            Initialize class from an existing cluster. This method assumes that the instance we are running on is the leader.

            Implementations must call _setLeaderWorkerAuthentication().

        setAutoscaledNodeTypes(nodeTypes)
            Set node types, shapes and spot bids for Toil-managed autoscaling. :param nodeTypes: A list of node types, as parsed with parse_node_types.

            Parameters
                nodeTypes (list </usr/share/doc/python3/html/library/stdtypes.html#list>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>], Optional[float </usr/share/doc/python3/html/library/functions.html#float>]]])

        hasAutoscaledNodeTypes()
            Check if node types have been configured on the provisioner (via setAutoscaledNodeTypes).

            Returns
                True if node types are configured for autoscaling, and false otherwise.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        getAutoscaledInstanceShapes()
            Get all the node shapes and their named instance types that the Toil autoscaler should manage.

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[Shape <#toil.provisioners.abstractProvisioner.Shape>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        static retryPredicate(e)
            Return true if the exception e should be retried by the cluster scaler. For example, should return true if the exception was due to exceeding an API rate limit. The error will be retried with exponential backoff.

            Parameters
                e -- exception raised during execution of setNodeCount

            Returns
                boolean indicating whether the exception e should be retried

        abstractmethod launchCluster(*args, **kwargs)
            Initialize a cluster and create a leader node.

            Implementations must call _setLeaderWorkerAuthentication() with the leader so that workers can be launched.

            Parameters

                •   leaderNodeType -- The leader instance.

                •   leaderStorage -- The amount of disk to allocate to the leader in gigabytes.

                •   owner -- Tag identifying the owner of the instances.

        abstractmethod addNodes(nodeTypes, numNodes, preemptible, spotBid=None)
            Used to add worker nodes to the cluster

            Parameters

                •   numNodes (int </usr/share/doc/python3/html/library/functions.html#int>) -- The number of nodes to add

                •   preemptible (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- whether or not the nodes will be preemptible

                •   spotBid (Optional[float </usr/share/doc/python3/html/library/functions.html#float>]) -- The bid for preemptible nodes if applicable (this can be set in config, also).

                •   nodeTypes (set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Returns
                number of nodes successfully added

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        addManagedNodes(nodeTypes, minNodes, maxNodes, preemptible, spotBid=None)
            Add a group of managed nodes of the given type, up to the given maximum. The nodes will automatically be launched and terminated depending on cluster load.

            Raises ManagedNodesNotSupportedException if the provisioner implementation or cluster configuration can't have managed nodes.

            Parameters

                •   minNodes -- The minimum number of nodes to scale to

                •   maxNodes -- The maximum number of nodes to scale to

                •   preemptible -- whether or not the nodes will be preemptible

                •   spotBid -- The bid for preemptible nodes if applicable (this can be set in config, also).

                •   nodeTypes (set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

        abstractmethod terminateNodes(nodes)
            Terminate the nodes represented by given Node objects

            Parameters
                nodes (list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.provisioners.node.Node <#toil.provisioners.node.Node>]) -- list of Node objects

            Return type
                None

        abstractmethod getLeader()

            Returns
                The leader node.

        abstractmethod getProvisionedWorkers(instance_type=None, preemptible=None)
            Gets all nodes, optionally of the given instance type or preemptability, from the provisioner. Includes both static and autoscaled nodes.

            Parameters

                •   preemptible (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>]) -- Boolean value to restrict to preemptible nodes or non-preemptible nodes

                •   instance_type (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Returns
                list of Node objects

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.provisioners.node.Node <#toil.provisioners.node.Node>]

        abstractmethod getNodeShape(instance_type, preemptible=False)
            The shape of a preemptible or non-preemptible node managed by this provisioner. The node shape defines key properties of a machine, such as its number of cores or the time between billing intervals.

            Parameters
                instance_type (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Instance type name to return the shape of.

            Return type
                Shape <#toil.provisioners.abstractProvisioner.Shape>

        abstractmethod destroyCluster()
            Terminates all nodes in the specified cluster and cleans up all resources associated with the cluster. :param clusterName: identifier of the cluster to terminate.

            Return type
                None

        class InstanceConfiguration
            Allows defining the initial setup for an instance and then turning it into an Ignition configuration for instance user data.

            files = []

            units = []

            sshPublicKeys = []

            addFile(path, filesystem='root', mode='0755', contents='', append=False)
                Make a file on the instance with the given filesystem, mode, and contents.

                See the storage.files section: <https://github.com/kinvolk/ignition/blob/flatcar-master/doc/configuration-v2_2.md>

                Parameters

                    •   path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                    •   filesystem (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                    •   mode (Union[str </usr/share/doc/python3/html/library/stdtypes.html#str>, int </usr/share/doc/python3/html/library/functions.html#int>])

                    •   contents (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                    •   append (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            addUnit(name, enabled=True, contents='')
                Make a systemd unit on the instance with the given name (including .service), and content. Units will be enabled by default.

                Unit logs can be investigated with:
                    systemctl status whatever.service

                or: journalctl -xe

                Parameters

                    •   name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                    •   enabled (bool </usr/share/doc/python3/html/library/functions.html#bool>)

                    •   contents (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            addSSHRSAKey(keyData)
                Authorize the given bare, encoded RSA key (without "ssh-rsa").

                Parameters
                    keyData (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            toIgnitionConfig()
                Return an Ignition configuration describing the desired config.

                Return type
                    str </usr/share/doc/python3/html/library/stdtypes.html#str>

        getBaseInstanceConfiguration()
            Get the base configuration for both leader and worker instances for all cluster types.

            Return type
                InstanceConfiguration <#toil.provisioners.abstractProvisioner.AbstractProvisioner.InstanceConfiguration>

        addVolumesService(config)
            Add a service to prepare and mount local scratch volumes.

            Parameters
                config (InstanceConfiguration <#toil.provisioners.abstractProvisioner.AbstractProvisioner.InstanceConfiguration>)

        addNodeExporterService(config)
            Add the node exporter service for Prometheus to an instance configuration.

            Parameters
                config (InstanceConfiguration <#toil.provisioners.abstractProvisioner.AbstractProvisioner.InstanceConfiguration>)

        toil_service_env_options()

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        add_toil_service(config, role, keyPath=None, preemptible=False)
            Add the Toil leader or worker service to an instance configuration.

            Will run Mesos master or agent as appropriate in Mesos clusters. For Kubernetes clusters, will just sleep to provide a place to shell into on the leader, and shouldn't run on the worker.

            Parameters

                •   role (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Should be 'leader' or 'worker'. Will not work for 'worker' until leader credentials have been collected.

                •   keyPath (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- path on the node to a server-side encryption key that will be added to the node after it starts. The service will wait until the key is present before starting.

                •   preemptible (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Whether a worker should identify itself as preemptible or not to the scheduler.

                •   config (InstanceConfiguration <#toil.provisioners.abstractProvisioner.AbstractProvisioner.InstanceConfiguration>)

        getKubernetesValues(architecture='amd64')
            Returns a dict of Kubernetes component versions and paths for formatting into Kubernetes-related templates.

            Parameters
                architecture (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        addKubernetesServices(config, architecture='amd64')
            Add installing Kubernetes and Kubeadm and setting up the Kubelet to run when configured to an instance configuration. The same process applies to leaders and workers.

            Parameters

                •   config (InstanceConfiguration <#toil.provisioners.abstractProvisioner.AbstractProvisioner.InstanceConfiguration>)

                •   architecture (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        abstractmethod getKubernetesAutoscalerSetupCommands(values)
            Return Bash commands that set up the Kubernetes cluster autoscaler for provisioning from the environment supported by this provisioner.

            Should only be implemented if Kubernetes clusters are supported.

            Parameters
                values (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Contains definitions of cluster variables, like AUTOSCALER_VERSION and CLUSTER_NAME.

            Returns
                Bash snippet

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        getKubernetesCloudProvider()
            Return the Kubernetes cloud provider (for example, 'aws'), to pass to the kubelets in a Kubernetes cluster provisioned using this provisioner.

            Defaults to None if not overridden, in which case no cloud provider integration will be used.

            Returns
                Cloud provider name, or None

            Return type
                Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        addKubernetesLeader(config)
            Add services to configure as a Kubernetes leader, if Kubernetes is already set to be installed.

            Parameters
                config (InstanceConfiguration <#toil.provisioners.abstractProvisioner.AbstractProvisioner.InstanceConfiguration>)

        addKubernetesWorker(config, authVars, preemptible=False)
            Add services to configure as a Kubernetes worker, if Kubernetes is already set to be installed.

            Authenticate back to the leader using the JOIN_TOKEN, JOIN_CERT_HASH, and JOIN_ENDPOINT set in the given authentication data dict.

            Parameters

                •   config (InstanceConfiguration <#toil.provisioners.abstractProvisioner.AbstractProvisioner.InstanceConfiguration>) -- The configuration to add services to

                •   authVars (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Dict with authentication info

                •   preemptible (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Whether the worker should be labeled as preemptible or not

  toil.provisioners.aws

  Submodules

  toil.provisioners.aws.awsProvisioner

  Attributes
logger |
F |

  Exceptions
InvalidClusterStateException | Common base class for all non-exit exceptions.

  Classes
AWSProvisioner | Interface for provisioning worker nodes to use in a Toil cluster.

  Functions
awsRetryPredicate(e) |
expectedShutdownErrors(e) | Matches errors that we expect to occur during shutdown, and which indicate
awsRetry(f) | This decorator retries the wrapped function if aws throws unexpected errors.
awsFilterImpairedNodes(nodes, boto3_ec2) |
collapse_tags(instance_tags) | Collapse tags from boto3 format to node format

  Module Contents

    toil.provisioners.aws.awsProvisioner.logger

    toil.provisioners.aws.awsProvisioner.awsRetryPredicate(e)

        Parameters
            e (Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>)

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.provisioners.aws.awsProvisioner.expectedShutdownErrors(e)
        Matches errors that we expect to occur during shutdown, and which indicate that we need to wait or try again.

        Should not match any errors which indicate that an operation is impossible or unnecessary (such as errors resulting from a thing not existing to be deleted).

        Parameters
            e (Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>)

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.provisioners.aws.awsProvisioner.F

    toil.provisioners.aws.awsProvisioner.awsRetry(f)
        This decorator retries the wrapped function if aws throws unexpected errors.

        It should wrap any function that makes use of boto

        Parameters
            f (Callable[Ellipsis, F])

        Return type
            Callable[Ellipsis, F]

    toil.provisioners.aws.awsProvisioner.awsFilterImpairedNodes(nodes, boto3_ec2)

        Parameters

            •   nodes (list </usr/share/doc/python3/html/library/stdtypes.html#list>[mypy_boto3_ec2.type_defs.InstanceTypeDef])

            •   boto3_ec2 (mypy_boto3_ec2.client.EC2Client)

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[mypy_boto3_ec2.type_defs.InstanceTypeDef]

    exception toil.provisioners.aws.awsProvisioner.InvalidClusterStateException
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Common base class for all non-exit exceptions.

    toil.provisioners.aws.awsProvisioner.collapse_tags(instance_tags)
        Collapse tags from boto3 format to node format :param instance_tags: tags as a list :return: Dict of tags

        Parameters
            instance_tags (list </usr/share/doc/python3/html/library/stdtypes.html#list>[mypy_boto3_ec2.type_defs.TagTypeDef])

        Return type
            dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    class toil.provisioners.aws.awsProvisioner.AWSProvisioner(clusterName, clusterType, zone, nodeStorage, nodeStorageOverrides, sseKey, enable_fuse)
        Bases: toil.provisioners.abstractProvisioner.AbstractProvisioner <#toil.provisioners.abstractProvisioner.AbstractProvisioner>

        Interface for provisioning worker nodes to use in a Toil cluster.

        Parameters

            •   clusterName (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None)

            •   clusterType (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None)

            •   zone (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None)

            •   nodeStorage (int </usr/share/doc/python3/html/library/functions.html#int>)

            •   nodeStorageOverrides (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None)

            •   sseKey (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None)

            •   enable_fuse (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        cloud = 'aws'

        aws

        s3_bucket_name = 'default-toil-cluster-name--internal'

        supportedClusterTypes()
            Get all the cluster types that this provisioner implementation supports.

            Return type
                set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        createClusterSettings()
            Create a new set of cluster settings for a cluster to be deployed into AWS.

            Return type
                None

        readClusterSettings()
            Reads the cluster settings from the instance metadata, which assumes the instance is the leader.

            Return type
                None

        launchCluster(leaderNodeType, leaderStorage, owner, keyName, botoPath, userTags, vpcSubnet, awsEc2ProfileArn, awsEc2ExtraSecurityGroupIds, **kwargs)
            Starts a single leader node and populates this class with the leader's metadata.

            Parameters

                •   leaderNodeType (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- An AWS instance type, like "t2.medium", for example.

                •   leaderStorage (int </usr/share/doc/python3/html/library/functions.html#int>) -- An integer number of gigabytes to provide the leader instance with.

                •   owner (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Resources will be tagged with this owner string.

                •   keyName (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The ssh key to use to access the leader node.

                •   botoPath (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The path to the boto credentials directory.

                •   userTags (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None) -- Optionally provided user tags to put on the cluster.

                •   vpcSubnet (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None) -- Optionally specify the VPC subnet for the leader.

                •   awsEc2ProfileArn (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None) -- Optionally provide the profile ARN.

                •   awsEc2ExtraSecurityGroupIds (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>] | None) -- Optionally provide additional security group IDs.

                •   kwargs (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any])

            Returns
                None

            Return type
                None

        toil_service_env_options()
            Set AWS tags in user docker container

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        getKubernetesAutoscalerSetupCommands(values)
            Get the Bash commands necessary to configure the Kubernetes Cluster Autoscaler for AWS.

            Parameters
                values (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        getKubernetesCloudProvider()
            Use the "aws" Kubernetes cloud provider when setting up Kubernetes.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str> | None

        getNodeShape(instance_type, preemptible=False)
            Get the Shape for the given instance type (e.g. 't2.medium').

            Parameters

                •   instance_type (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   preemptible (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>

        static retryPredicate(e)
            Return true if the exception e should be retried by the cluster scaler. For example, should return true if the exception was due to exceeding an API rate limit. The error will be retried with exponential backoff.

            Parameters
                e (Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>) -- exception raised during execution of setNodeCount

            Returns
                boolean indicating whether the exception e should be retried

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        destroyCluster()
            Terminate instances and delete the profile and security group.

            Return type
                None

        terminateNodes(nodes)
            Terminate the nodes represented by given Node objects

            Parameters
                nodes (list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.provisioners.node.Node <#toil.provisioners.node.Node>]) -- list of Node objects

            Return type
                None

        addNodes(nodeTypes, numNodes, preemptible, spotBid=None)
            Used to add worker nodes to the cluster

            Parameters

                •   numNodes (int </usr/share/doc/python3/html/library/functions.html#int>) -- The number of nodes to add

                •   preemptible (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- whether or not the nodes will be preemptible

                •   spotBid (float </usr/share/doc/python3/html/library/functions.html#float> | None) -- The bid for preemptible nodes if applicable (this can be set in config, also).

                •   nodeTypes (set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Returns
                number of nodes successfully added

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        addManagedNodes(nodeTypes, minNodes, maxNodes, preemptible, spotBid=None)
            Add a group of managed nodes of the given type, up to the given maximum. The nodes will automatically be launched and terminated depending on cluster load.

            Raises ManagedNodesNotSupportedException if the provisioner implementation or cluster configuration can't have managed nodes.

            Parameters

                •   minNodes (int </usr/share/doc/python3/html/library/functions.html#int>) -- The minimum number of nodes to scale to

                •   maxNodes (int </usr/share/doc/python3/html/library/functions.html#int>) -- The maximum number of nodes to scale to

                •   preemptible (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- whether or not the nodes will be preemptible

                •   spotBid (float </usr/share/doc/python3/html/library/functions.html#float> | None) -- The bid for preemptible nodes if applicable (this can be set in config, also).

                •   nodeTypes (set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

        getProvisionedWorkers(instance_type=None, preemptible=None)
            Gets all nodes, optionally of the given instance type or preemptability, from the provisioner. Includes both static and autoscaled nodes.

            Parameters

                •   preemptible (bool </usr/share/doc/python3/html/library/functions.html#bool> | None) -- Boolean value to restrict to preemptible nodes or non-preemptible nodes

                •   instance_type (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None)

            Returns
                list of Node objects

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.provisioners.node.Node <#toil.provisioners.node.Node>]

        getLeader(wait=False)
            Get the leader for the cluster as a Toil Node object.

            Parameters
                wait (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                toil.provisioners.node.Node <#toil.provisioners.node.Node>

        full_policy(resource)
            Produce a dict describing the JSON form of a full-access-granting AWS IAM policy for the service with the given name (e.g. 's3').

            Parameters
                resource (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]

        kubernetes_policy()
            Get the Kubernetes policy grants not provided by the full grants on EC2 and IAM. See <<https://github.com/DataBiosphere/toil/wiki/Manual-Autoscaling-Kubernetes-Setup#leader-policy>> and <<https://github.com/DataBiosphere/toil/wiki/Manual-Autoscaling-Kubernetes-Setup#worker-policy>>.

            These are mostly needed to support Kubernetes' AWS CloudProvider, and some are for the Kubernetes Cluster Autoscaler's AWS integration.

            Some of these are really only needed on the leader.

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]

  Attributes
logger |

  Classes
ZoneTuple |

  Functions
get_aws_zone_from_spot_market(spotBid, nodeType, ...) | If a spot bid, node type, and Boto2 EC2 connection are specified, picks a
get_best_aws_zone([spotBid, nodeType, boto3_ec2, ...]) | Get the right AWS zone to use.
choose_spot_zone(zones, bid, spot_history) | Returns the zone to put the spot request based on, in order of priority:
optimize_spot_bid(boto3_ec2, instance_type, spot_bid, ...) | Check whether the bid is in line with history and makes an effort to place

  Package Contents

    toil.provisioners.aws.logger

    class toil.provisioners.aws.ZoneTuple
        Bases: tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>

        name

        price_deviation

    toil.provisioners.aws.get_aws_zone_from_spot_market(spotBid, nodeType, boto3_ec2, zone_options)
        If a spot bid, node type, and Boto2 EC2 connection are specified, picks a zone where instances are easy to buy from the zones in the region of the Boto2 connection. These parameters must always be specified together, or not at all.

        In this case, zone_options can be used to restrict to a subset of the zones in the region.

        Parameters

            •   spotBid (Optional[float </usr/share/doc/python3/html/library/functions.html#float>])

            •   nodeType (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   boto3_ec2 (Optional[botocore.client.BaseClient])

            •   zone_options (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.provisioners.aws.get_best_aws_zone(spotBid=None, nodeType=None, boto3_ec2=None, zone_options=None)
        Get the right AWS zone to use.

        Reports the TOIL_AWS_ZONE environment variable if set.

        Otherwise, if we are running on EC2 or ECS, reports the zone we are running in.

        Otherwise, if a spot bid, node type, and Boto2 EC2 connection are specified, picks a zone where instances are easy to buy from the zones in the region of the Boto2 connection. These parameters must always be specified together, or not at all.

        In this case, zone_options can be used to restrict to a subset of the zones in the region.

        Otherwise, if we have the TOIL_AWS_REGION variable set, chooses a zone in that region.

        Finally, if a default region is configured in Boto 2, chooses a zone in that region.

        Returns None if no method can produce a zone to use.

        Parameters

            •   spotBid (Optional[float </usr/share/doc/python3/html/library/functions.html#float>])

            •   nodeType (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   boto3_ec2 (Optional[botocore.client.BaseClient])

            •   zone_options (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.provisioners.aws.choose_spot_zone(zones, bid, spot_history)
        Returns the zone to put the spot request based on, in order of priority:

                1.  zones with prices currently under the bid

                2.  zones with the most stable price

        Returns
            the name of the selected zone

        Parameters

            •   zones (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   bid (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   spot_history (list </usr/share/doc/python3/html/library/stdtypes.html#list>[mypy_boto3_ec2.type_defs.SpotPriceTypeDef])

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

        >>> zones = ['us-west-2a', 'us-west-2b']
        >>> FauxHistory = lambda p, z: {"SpotPrice": p, "AvailabilityZone": z}
        >>> spot_history = [FauxHistory("0.1", 'us-west-2a'),                         FauxHistory("0.2", 'us-west-2a'),                         FauxHistory("0.3", 'us-west-2b'),                         FauxHistory("0.6", 'us-west-2b')]
        >>> choose_spot_zone(zones, 0.15, spot_history)
        'us-west-2a'

        >>> spot_history=[FauxHistory("0.3", 'us-west-2a'),                       FauxHistory("0.2", 'us-west-2a'),                       FauxHistory("0.1", 'us-west-2b'),                       FauxHistory("0.6", 'us-west-2b')]
        >>> choose_spot_zone(zones, 0.15, spot_history)
        'us-west-2b'

        >>> spot_history=[FauxHistory("0.1", 'us-west-2a'),                       FauxHistory("0.7", 'us-west-2a'),                       FauxHistory("0.1", 'us-west-2b'),                       FauxHistory("0.6", 'us-west-2b')]
        >>> choose_spot_zone(zones, 0.15, spot_history)
        'us-west-2b'

    toil.provisioners.aws.optimize_spot_bid(boto3_ec2, instance_type, spot_bid, zone_options)
        Check whether the bid is in line with history and makes an effort to place the instance in a sensible zone.

        Parameters

            •   zone_options (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- The collection of allowed zones to consider, within the region associated with the Boto2 connection.

            •   boto3_ec2 (botocore.client.BaseClient)

            •   instance_type (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   spot_bid (float </usr/share/doc/python3/html/library/functions.html#float>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

  toil.provisioners.clusterScaler

  Attributes
logger |
EVICTION_THRESHOLD |
RESERVE_SMALL_LIMIT |
RESERVE_SMALL_AMOUNT |
RESERVE_BREAKPOINTS |
RESERVE_FRACTIONS |
OS_SIZE |
FailedConstraint |

  Exceptions
JobTooBigError | Raised in the scaler thread when a job cannot fit in any available node

  Classes
BinPackedFit | If jobShapes is a set of tasks with run requirements (mem/disk/cpu), and nodeShapes is a sorted
NodeReservation | The amount of resources that we expect to be available on a given node at each point in time.
ClusterScaler |
ScalerThread | A thread that automatically scales the number of either preemptible or non-preemptible worker
ClusterStats |

  Functions
adjustEndingReservationForJob(reservation, jobShape, ...) | Add a job to an ending reservation that ends at wallTime.
split(nodeShape, jobShape, wallTime) | Partition a node allocation into two to fit the job.
binPacking(nodeShapes, jobShapes, goalTime) | Using the given node shape bins, pack the given job shapes into nodes to

  Module Contents

    toil.provisioners.clusterScaler.logger

    toil.provisioners.clusterScaler.EVICTION_THRESHOLD

    toil.provisioners.clusterScaler.RESERVE_SMALL_LIMIT

    toil.provisioners.clusterScaler.RESERVE_SMALL_AMOUNT

    toil.provisioners.clusterScaler.RESERVE_BREAKPOINTS: list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int> | float </usr/share/doc/python3/html/library/functions.html#float>]

    toil.provisioners.clusterScaler.RESERVE_FRACTIONS = [0.25, 0.2, 0.1, 0.06, 0.02]

    toil.provisioners.clusterScaler.OS_SIZE

    toil.provisioners.clusterScaler.FailedConstraint

    class toil.provisioners.clusterScaler.BinPackedFit(nodeShapes, targetTime=defaultTargetTime)
        If jobShapes is a set of tasks with run requirements (mem/disk/cpu), and nodeShapes is a sorted list of available computers to run these jobs on, this function attempts to return a dictionary representing the minimum set of computerNode computers needed to run the tasks in jobShapes.

        Uses a first fit decreasing (FFD) bin packing like algorithm to calculate an approximate minimum number of nodes that will fit the given list of jobs. BinPackingFit assumes the ordered list, nodeShapes, is ordered for "node preference" outside of BinPackingFit beforehand. So when virtually "creating" nodes, the first node within nodeShapes that fits the job is the one that's added.

        Parameters

            •   nodeShapes (list </usr/share/doc/python3/html/library/stdtypes.html#list>) -- The properties of an atomic node allocation, in terms of wall-time, memory, cores, disk, and whether it is preemptible or not.

            •   targetTime (float </usr/share/doc/python3/html/library/functions.html#float>) -- The time before which all jobs should at least be started.

        Returns
            The minimum number of minimal node allocations estimated to be required to run all the jobs in jobShapes.

        nodeReservations: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>, list </usr/share/doc/python3/html/library/stdtypes.html#list>[NodeReservation <#toil.provisioners.clusterScaler.NodeReservation>]]

        nodeShapes

        targetTime = 1800

        binPack(jobShapes)
            Pack a list of jobShapes into the fewest nodes reasonable.

            Can be run multiple times.

            Returns any distinct Shapes that did not fit, mapping to reasons they did not fit.

            Parameters
                jobShapes (list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>])

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>, list </usr/share/doc/python3/html/library/stdtypes.html#list>[FailedConstraint]]

        addJobShape(jobShape)
            Add the job to the first node reservation in which it will fit. (This is the bin-packing aspect).

            Returns the job shape again, and a list of failed constraints, if it did not fit.

            Parameters
                jobShape (toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>)

            Return type
                Optional[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>, list </usr/share/doc/python3/html/library/stdtypes.html#list>[FailedConstraint]]]

        getRequiredNodes()
            Return a dict from node shape to number of nodes required to run the packed jobs.

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>, int </usr/share/doc/python3/html/library/functions.html#int>]

    class toil.provisioners.clusterScaler.NodeReservation(shape)
        The amount of resources that we expect to be available on a given node at each point in time.

        To represent the resources available in a reservation, we represent a reservation as a linked list of NodeReservations, each giving the resources free within a single timeslice.

        Parameters
            shape (toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>)

        shape

        nReservation: NodeReservation <#toil.provisioners.clusterScaler.NodeReservation> | None </usr/share/doc/python3/html/library/constants.html#None> = None

        __str__()

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        get_failed_constraints(job_shape)
            Check if a job shape's resource requirements will fit within this allocation.

            If the job does not fit, returns the failing constraints: the resources that can't be accomodated, and the limits that were hit.

            If the job does fit, returns an empty list.

            Must always agree with fits()! This codepath is slower and used for diagnosis.

            Parameters
                job_shape (toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>)

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[FailedConstraint]

        fits(jobShape)
            Check if a job shape's resource requirements will fit within this allocation.

            Parameters
                jobShape (toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        shapes()
            Get all time-slice shapes, in order, from this reservation on.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>]

        subtract(jobShape)
            Subtract the resources necessary to run a jobShape from the reservation.

            Parameters
                jobShape (toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>)

            Return type
                None

        attemptToAddJob(jobShape, nodeShape, targetTime)
            Attempt to pack a job into this reservation timeslice and/or the reservations after it.

            jobShape is the Shape of the job requirements, nodeShape is the Shape of the node this is a reservation for, and targetTime is the maximum time to wait before starting this job.

            Parameters

                •   jobShape (toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>)

                •   nodeShape (toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>)

                •   targetTime (float </usr/share/doc/python3/html/library/functions.html#float>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.provisioners.clusterScaler.adjustEndingReservationForJob(reservation, jobShape, wallTime)
        Add a job to an ending reservation that ends at wallTime.

        (splitting the reservation if the job doesn't fill the entire timeslice)

        Parameters

            •   reservation (NodeReservation <#toil.provisioners.clusterScaler.NodeReservation>)

            •   jobShape (toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>)

            •   wallTime (float </usr/share/doc/python3/html/library/functions.html#float>)

        Return type
            None

    toil.provisioners.clusterScaler.split(nodeShape, jobShape, wallTime)
        Partition a node allocation into two to fit the job.

        Returning the modified shape of the node and a new node reservation for the extra time that the job didn't fill.

        Parameters

            •   nodeShape (toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>)

            •   jobShape (toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>)

            •   wallTime (float </usr/share/doc/python3/html/library/functions.html#float>)

        Return type
            tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>, NodeReservation <#toil.provisioners.clusterScaler.NodeReservation>]

    toil.provisioners.clusterScaler.binPacking(nodeShapes, jobShapes, goalTime)
        Using the given node shape bins, pack the given job shapes into nodes to get them done in the given amount of time.

        Returns a dict saying how many of each node will be needed, a dict from job shapes that could not fit to reasons why.

        Parameters

            •   nodeShapes (list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>])

            •   jobShapes (list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>])

            •   goalTime (float </usr/share/doc/python3/html/library/functions.html#float>)

        Return type
            tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>, int </usr/share/doc/python3/html/library/functions.html#int>], dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>, list </usr/share/doc/python3/html/library/stdtypes.html#list>[FailedConstraint]]]

    class toil.provisioners.clusterScaler.ClusterScaler(provisioner, leader, config)

        Parameters

            •   provisioner (toil.provisioners.abstractProvisioner.AbstractProvisioner <#toil.provisioners.abstractProvisioner.AbstractProvisioner>)

            •   leader (toil.leader.Leader <#toil.leader.Leader>)

            •   config (toil.common.Config <#toil.common.Config>)

        provisioner

        leader

        config

        static: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[bool </usr/share/doc/python3/html/library/functions.html#bool>, dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.provisioners.node.Node <#toil.provisioners.node.Node>]]

        on_too_big: list </usr/share/doc/python3/html/library/stdtypes.html#list>[Callable[[toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>, list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>]], Any]] = []

        jobNameToAvgRuntime: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, float </usr/share/doc/python3/html/library/functions.html#float>]

        jobNameToNumCompleted: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, int </usr/share/doc/python3/html/library/functions.html#int>]

        totalAvgRuntime = 0.0

        totalJobsCompleted = 0

        targetTime: float </usr/share/doc/python3/html/library/functions.html#float>

        betaInertia

        nodeShapeToType

        instance_types

        nodeShapes

        ignoredNodes: set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        preemptibleNodeDeficit

        previousWeightedEstimate

        minNodes

        maxNodes

        node_shapes_after_overhead

        without_overhead

        getAverageRuntime(jobName, service=False)

            Parameters

                •   jobName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   service (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                float </usr/share/doc/python3/html/library/functions.html#float>

        addCompletedJob(job, wallTime)
            Adds the shape of a completed job to the queue, allowing the scalar to use the last N completed jobs in factoring how many nodes are required in the cluster. :param toil.job.JobDescription job: The description of the completed job :param int wallTime: The wall-time taken to complete the job in seconds.

            Parameters

                •   job (toil.job.JobDescription <#toil.job.JobDescription>)

                •   wallTime (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                None

        setStaticNodes(nodes, preemptible)
            Used to track statically provisioned nodes. This method must be called before any auto-scaled nodes are provisioned.

            These nodes are treated differently than auto-scaled nodes in that they should not be automatically terminated.

            Parameters

                •   nodes (list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.provisioners.node.Node <#toil.provisioners.node.Node>]) -- list of Node objects

                •   preemptible (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        getStaticNodes(preemptible)
            Returns nodes set in setStaticNodes().

            Parameters
                preemptible (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Returns
                Statically provisioned nodes.

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.provisioners.node.Node <#toil.provisioners.node.Node>]

        smoothEstimate(nodeShape, estimatedNodeCount)
            Smooth out fluctuations in the estimate for this node compared to previous runs.

            Returns an integer.

            Parameters

                •   nodeShape (toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>)

                •   estimatedNodeCount (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        getEstimatedNodeCounts(queuedJobShapes, currentNodeCounts)
            Given the resource requirements of queued jobs and the current size of the cluster.

            Returns a dict mapping from nodeShape to the number of nodes we want in the cluster right now, and a dict from job shapes that are too big to run on any node to reasons why.

            Parameters

                •   queuedJobShapes (list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>])

                •   currentNodeCounts (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>, int </usr/share/doc/python3/html/library/functions.html#int>])

            Return type
                tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>, int </usr/share/doc/python3/html/library/functions.html#int>], dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>, list </usr/share/doc/python3/html/library/stdtypes.html#list>[FailedConstraint]]]

        updateClusterSize(estimatedNodeCounts)
            Given the desired and current size of the cluster, attempts to launch/remove instances to get to the desired size.

            Also attempts to remove ignored nodes that were marked for graceful removal.

            Returns the new size of the cluster.

            Parameters
                estimatedNodeCounts (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>, int </usr/share/doc/python3/html/library/functions.html#int>])

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>, int </usr/share/doc/python3/html/library/functions.html#int>]

        setNodeCount(instance_type, numNodes, preemptible=False, force=False)
            Attempt to grow or shrink the number of preemptible or non-preemptible worker nodes in the cluster to the given value, or as close a value as possible, and, after performing the necessary additions or removals of worker nodes, return the resulting number of preemptible or non-preemptible nodes currently in the cluster.

            Parameters

                •   instance_type (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The instance type to add or remove.

                •   numNodes (int </usr/share/doc/python3/html/library/functions.html#int>) -- Desired size of the cluster

                •   preemptible (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- whether the added nodes will be preemptible, i.e. whether they may be removed spontaneously by the underlying platform at any time.

                •   force (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If False, the provisioner is allowed to deviate from the given number of nodes. For example, when downsizing a cluster, a provisioner might leave nodes running if they have active jobs running on them.

            Returns
                the number of worker nodes in the cluster after making the necessary adjustments. This value should be, but is not guaranteed to be, close or equal to the numNodes argument. It represents the closest possible approximation of the actual cluster size at the time this method returns.

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        filter_out_static_nodes(nodes, preemptible=False)

            Parameters

                •   nodes (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[toil.provisioners.node.Node <#toil.provisioners.node.Node>, toil.batchSystems.abstractBatchSystem.NodeInfo <#toil.batchSystems.abstractBatchSystem.NodeInfo>])

                •   preemptible (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[toil.provisioners.node.Node <#toil.provisioners.node.Node>, toil.batchSystems.abstractBatchSystem.NodeInfo <#toil.batchSystems.abstractBatchSystem.NodeInfo>]]

        getNodes(preemptible=None)
            Returns a dictionary mapping node identifiers of preemptible or non-preemptible nodes to NodeInfo objects, one for each node.

            This method is the definitive source on nodes in cluster, & is responsible for consolidating cluster state between the provisioner & batch system.

            Parameters
                preemptible (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If True (False) only (non-)preemptible nodes will be returned. If None, all nodes will be returned.

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[toil.provisioners.node.Node <#toil.provisioners.node.Node>, toil.batchSystems.abstractBatchSystem.NodeInfo <#toil.batchSystems.abstractBatchSystem.NodeInfo>]

        shutDown()

            Return type
                None

    exception toil.provisioners.clusterScaler.JobTooBigError(job=None, shape=None, constraints=None)
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Raised in the scaler thread when a job cannot fit in any available node type and is likely to lock up the workflow.

        Parameters

            •   job (Optional[toil.job.JobDescription <#toil.job.JobDescription>])

            •   shape (Optional[toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>])

            •   constraints (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[FailedConstraint]])

        job = None

        shape = None

        constraints

        msg = ''

        __str__()
            Stringify the exception, including the message.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

    class toil.provisioners.clusterScaler.ScalerThread(provisioner, leader, config, stop_on_exception=False)
        Bases: toil.lib.threading.ExceptionalThread <#toil.lib.threading.ExceptionalThread>

        A thread that automatically scales the number of either preemptible or non-preemptible worker nodes according to the resource requirements of the queued jobs.

        The scaling calculation is essentially as follows: start with 0 estimated worker nodes. For each queued job, check if we expect it can be scheduled into a worker node before a certain time (currently one hour). Otherwise, attempt to add a single new node of the smallest type that can fit that job.

        At each scaling decision point a comparison between the current, C, and newly estimated number of nodes is made. If the absolute difference is less than beta * C then no change is made, else the size of the cluster is adapted. The beta factor is an inertia parameter that prevents continual fluctuations in the number of nodes.

        Parameters

            •   provisioner (toil.provisioners.abstractProvisioner.AbstractProvisioner <#toil.provisioners.abstractProvisioner.AbstractProvisioner>)

            •   leader (toil.leader.Leader <#toil.leader.Leader>)

            •   config (toil.common.Config <#toil.common.Config>)

            •   stop_on_exception (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        scaler

        stop = False

        stop_on_exception = False

        stats = None

        check()
            Attempt to join any existing scaler threads that may have died or finished.

            This insures any exceptions raised in the threads are propagated in a timely fashion.

            Return type
                None

        shutdown()
            Shutdown the cluster.

            Return type
                None

        addCompletedJob(job, wallTime)

            Parameters

                •   job (toil.job.JobDescription <#toil.job.JobDescription>)

                •   wallTime (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                None

        tryRun()

            Return type
                None

    class toil.provisioners.clusterScaler.ClusterStats(path, batchSystem, clusterName)

        Parameters

            •   path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   batchSystem (toil.batchSystems.abstractBatchSystem.AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>)

            •   clusterName (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        stats: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, list </usr/share/doc/python3/html/library/stdtypes.html#list>[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]]]]

        statsThreads: list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.lib.threading.ExceptionalThread <#toil.lib.threading.ExceptionalThread>] = []

        statsPath

        stop = False

        clusterName

        batchSystem

        scalable

        shutDownStats()

            Return type
                None

        startStats(preemptible)

            Parameters
                preemptible (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        checkStats()

            Return type
                None

  toil.provisioners.gceProvisioner

  Attributes
logger |

  Classes
GCEProvisioner | Implements a Google Compute Engine Provisioner using libcloud.

  Module Contents

    toil.provisioners.gceProvisioner.logger

    class toil.provisioners.gceProvisioner.GCEProvisioner(clusterName, clusterType, zone, nodeStorage, nodeStorageOverrides, sseKey, enable_fuse)
        Bases: toil.provisioners.abstractProvisioner.AbstractProvisioner <#toil.provisioners.abstractProvisioner.AbstractProvisioner>

        Implements a Google Compute Engine Provisioner using libcloud.

        NODE_BOTO_PATH = '/root/.boto'

        SOURCE_IMAGE = b'projects/kinvolk-public/global/images/family/flatcar-stable'

        cloud = 'gce'

        supportedClusterTypes()
            Get all the cluster types that this provisioner implementation supports.

        createClusterSettings()
            Initialize class for a new cluster, to be deployed, when running outside the cloud.

        readClusterSettings()
            Read the cluster settings from the instance, which should be the leader. See <https://cloud.google.com/compute/docs/storing-retrieving-metadata> for details about reading the metadata.

        launchCluster(leaderNodeType, leaderStorage, owner, **kwargs)
            In addition to the parameters inherited from the abstractProvisioner, the Google launchCluster takes the following parameters: keyName: The key used to communicate with instances botoPath: Boto credentials for reading an AWS jobStore (optional). network: a network (optional) vpcSubnet: A subnet (optional). use_private_ip: even though a public ip exists, ignore it (optional)

        getNodeShape(instance_type, preemptible=False)
            The shape of a preemptible or non-preemptible node managed by this provisioner. The node shape defines key properties of a machine, such as its number of cores or the time between billing intervals.

            Parameters
                instance_type (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Instance type name to return the shape of.

            Return type
                toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>

        static retryPredicate(e)
            Not used by GCE

        destroyCluster()
            Try a few times to terminate all of the instances in the group.

            Return type
                None

        terminateNodes(nodes)
            Terminate the nodes represented by given Node objects

            Parameters
                nodes -- list of Node objects

        addNodes(nodeTypes, numNodes, preemptible, spotBid=None)
            Used to add worker nodes to the cluster

            Parameters

                •   numNodes -- The number of nodes to add

                •   preemptible -- whether or not the nodes will be preemptible

                •   spotBid -- The bid for preemptible nodes if applicable (this can be set in config, also).

                •   nodeTypes (set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Returns
                number of nodes successfully added

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        getProvisionedWorkers(instance_type=None, preemptible=None)
            Gets all nodes, optionally of the given instance type or preemptability, from the provisioner. Includes both static and autoscaled nodes.

            Parameters

                •   preemptible (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>]) -- Boolean value to restrict to preemptible nodes or non-preemptible nodes

                •   instance_type (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Returns
                list of Node objects

        getLeader()

            Returns
                The leader node.

        DEFAULT_TASK_COMPLETION_TIMEOUT = 180

        ex_create_multiple_nodes(base_name, size, image, number, location=None, ex_network='default', ex_subnetwork=None, ex_tags=None, ex_metadata=None, ignore_errors=True, use_existing_disk=True, poll_interval=2, external_ip='ephemeral', ex_disk_type='pd-standard', ex_disk_auto_delete=True, ex_service_accounts=None, timeout=DEFAULT_TASK_COMPLETION_TIMEOUT, description=None, ex_can_ip_forward=None, ex_disks_gce_struct=None, ex_nic_gce_struct=None, ex_on_host_maintenance=None, ex_automatic_restart=None, ex_image_family=None, ex_preemptible=None)
            Monkey patch to gce.py in libcloud to allow disk and images to be specified. Also changed name to a uuid below. The prefix 'wp' identifies preemptible nodes and 'wn' non-preemptible nodes.

  toil.provisioners.node

  Attributes
a_short_time |
logger |

  Classes
Node |

  Module Contents

    toil.provisioners.node.a_short_time = 5

    toil.provisioners.node.logger

    class toil.provisioners.node.Node(publicIP, privateIP, name, launchTime, nodeType, preemptible, tags=None, use_private_ip=None)

        Parameters

            •   publicIP (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   privateIP (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   launchTime (Union[datetime.datetime </usr/share/doc/python3/html/library/datetime.html#datetime.datetime>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   nodeType (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   preemptible (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            •   tags (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

            •   use_private_ip (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>])

        maxWaitTime = 420

        publicIP

        privateIP

        name

        nodeType

        preemptible

        tags = None

        __str__()

        __repr__()

        __hash__()

        remainingBillingInterval()
            Returns a floating point value between 0 and 1.0 representing how much time is left in the current billing cycle for the given instance. If the return value is .25, we are three quarters into the billing cycle, with one quarters remaining before we will be charged again for that instance.

            Assumes a billing cycle of one hour.

            Returns
                Float from 1.0 -> 0.0 representing fraction of pre-paid time remaining in cycle.

            Return type
                float </usr/share/doc/python3/html/library/functions.html#float>

            >>> node = Node("127.0.0.1", "127.0.0.1", "localhost",
            ...             datetime.datetime.utcnow(), None, False)
            >>> node.remainingBillingInterval() >= 0
            True
            >>> node.remainingBillingInterval() <= 1.0
            True
            >>> node.remainingBillingInterval() > 0.5
            True
            >>> interval1 = node.remainingBillingInterval()
            >>> time.sleep(1)
            >>> interval2 = node.remainingBillingInterval()
            >>> interval2 < interval1
            True

            >>> node = Node("127.0.0.1", "127.0.0.1", "localhost",
            ...             datetime.datetime.now(datetime.timezone.utc) -
            ...             datetime.timedelta(minutes=5), None, False)
            >>> node.remainingBillingInterval() < 0.99
            True
            >>> node.remainingBillingInterval() > 0.9
            True

        waitForNode(role, keyName='core')

            Parameters

                •   role (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   keyName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        copySshKeys(keyName)
            Copy authorized_keys file to the core user from the keyName user.

        injectFile(fromFile, toFile, role)
            rysnc a file to the container with the given role

        extractFile(fromFile, toFile, role)
            rysnc a file from the container with the given role

        sshAppliance(*args, **kwargs)

            Parameters

                •   args -- arguments to execute in the appliance

                •   kwargs -- tty=bool tells docker whether or not to create a TTY shell for interactive SSHing. The default value is False. Input=string is passed as input to the Popen call.

        sshInstance(*args, **kwargs)
            Run a command on the instance. Returns the binary output of the command.

        coreSSH(*args, **kwargs)
            If strict=False, strict host key checking will be temporarily disabled. This is provided as a convenience for internal/automated functions and ought to be set to True whenever feasible, or whenever the user is directly interacting with a resource (e.g. rsync-cluster or ssh-cluster). Assumed to be False by default.

            kwargs: input, tty, appliance, collectStdout, sshOptions, strict

            Parameters
                input (bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>) -- UTF-8 encoded input bytes to send to the command

        coreRsync(args, applianceName='toil_leader', **kwargs)

            Parameters

                •   args (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   applianceName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   kwargs (Any)

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

  Attributes
logger |

  Exceptions
NoSuchClusterException | Indicates that the specified cluster does not exist.
NoSuchZoneException | Indicates that a valid zone could not be found.
ClusterTypeNotSupportedException | Indicates that a provisioner does not support a given cluster type.
ClusterCombinationNotSupportedException | Indicates that a provisioner does not support making a given type of cluster with a given architecture.

  Functions
cluster_factory(provisioner[, clusterName, ...]) | Find and instantiate the appropriate provisioner instance to make clusters in the given cloud.
add_provisioner_options(parser) |
parse_node_types(node_type_specs) | Parse a specification for zero or more node types.
check_valid_node_types(provisioner, node_types) | Raises if an invalid nodeType is specified for aws or gce.

  Package Contents

    toil.provisioners.logger

    toil.provisioners.cluster_factory(provisioner, clusterName=None, clusterType='mesos', zone=None, nodeStorage=50, nodeStorageOverrides=None, sseKey=None, enable_fuse=False)
        Find and instantiate the appropriate provisioner instance to make clusters in the given cloud.

        Raises ClusterTypeNotSupportedException if the given provisioner does not implement clusters of the given type.

        Parameters

            •   provisioner (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The cloud type of the cluster.

            •   clusterName (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- The name of the cluster.

            •   clusterType (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The type of cluster: 'mesos' or 'kubernetes'.

            •   zone (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- The cloud zone

            •   nodeStorage (int </usr/share/doc/python3/html/library/functions.html#int>)

            •   nodeStorageOverrides (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

            •   sseKey (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   enable_fuse (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Returns
            A cluster object for the the cloud type.

        Return type
            Union[aws.awsProvisioner.AWSProvisioner <#toil.provisioners.aws.awsProvisioner.AWSProvisioner>, gceProvisioner.GCEProvisioner <#toil.provisioners.gceProvisioner.GCEProvisioner>]

    toil.provisioners.add_provisioner_options(parser)

        Parameters
            parser (argparse.ArgumentParser </usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser>)

        Return type
            None

    toil.provisioners.parse_node_types(node_type_specs)
        Parse a specification for zero or more node types.

        Takes a comma-separated list of node types. Each node type is a slash-separated list of at least one instance type name (like 'm5a.large' for AWS), and an optional bid in dollars after a colon.

        Raises ValueError if a node type cannot be parsed.

        Inputs should look something like this:

            >>> types = parse_node_types('c5.4xlarge/c5a.4xlarge:0.42,t2.large')
            >>> sorted(types[0][0]), types[0][1]
            (['c5.4xlarge', 'c5a.4xlarge'], 0.42)
            >>> sorted(types[1][0]), types[1][1]
            (['t2.large'], None)

        Parameters
            node_type_specs (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- A string defining node types

        Returns
            a list of node types, where each type is the set of instance types, and the float bid, or None.

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>], Optional[float </usr/share/doc/python3/html/library/functions.html#float>]]]

    toil.provisioners.check_valid_node_types(provisioner, node_types)
        Raises if an invalid nodeType is specified for aws or gce.

        Parameters

            •   provisioner (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- 'aws' or 'gce' to specify which cloud provisioner used.

            •   node_types (list </usr/share/doc/python3/html/library/stdtypes.html#list>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>], Optional[float </usr/share/doc/python3/html/library/functions.html#float>]]]) -- A list of node types. Example: [({'t2.micro'}, None), ({'t2.medium'}, 0.5)]

        Returns
            Nothing. Raises if any instance type in the node type isn't real.

    exception toil.provisioners.NoSuchClusterException(cluster_name)
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Indicates that the specified cluster does not exist.

        Parameters
            cluster_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

    exception toil.provisioners.NoSuchZoneException
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Indicates that a valid zone could not be found.

    exception toil.provisioners.ClusterTypeNotSupportedException(provisioner_class, cluster_type)
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Indicates that a provisioner does not support a given cluster type.

    exception toil.provisioners.ClusterCombinationNotSupportedException(provisioner_class, cluster_type, architecture, reason=None)
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Indicates that a provisioner does not support making a given type of cluster with a given architecture.

        Parameters

            •   provisioner_class (type </usr/share/doc/python3/html/library/functions.html#type>)

            •   cluster_type (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   architecture (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   reason (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

  toil.realtimeLogger

Implements a real-time UDP-based logging system that user scripts can use for debugging.

  Attributes
logger |

  Classes
LoggingDatagramHandler | Receive logging messages from the jobs and display them on the leader.
JSONDatagramHandler | Send logging records over UDP serialized as JSON.
RealtimeLoggerMetaclass | Metaclass for RealtimeLogger that lets add logging methods.
RealtimeLogger | Provide a logger that logs over UDP to the leader.

  Module Contents

    toil.realtimeLogger.logger

    class toil.realtimeLogger.LoggingDatagramHandler(request, client_address, server)
        Bases: socketserver.BaseRequestHandler </usr/share/doc/python3/html/library/socketserver.html#socketserver.BaseRequestHandler>

        Receive logging messages from the jobs and display them on the leader.

        Uses bare JSON message encoding.

        handle()
            Handle a single message. SocketServer takes care of splitting out the messages.

            Messages are JSON-encoded logging module records.

            Return type
                None

    class toil.realtimeLogger.JSONDatagramHandler(host, port)
        Bases: logging.handlers.DatagramHandler </usr/share/doc/python3/html/library/logging.handlers.html#logging.handlers.DatagramHandler>

        Send logging records over UDP serialized as JSON.

        They have to fit in a single UDP datagram, so don't try to log more than 64kb at once.

        makePickle(record)
            Actually, encode the record as bare JSON instead.

            Parameters
                record (logging.LogRecord </usr/share/doc/python3/html/library/logging.html#logging.LogRecord>)

            Return type
                bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>

    class toil.realtimeLogger.RealtimeLoggerMetaclass
        Bases: type </usr/share/doc/python3/html/library/functions.html#type>

        Metaclass for RealtimeLogger that lets add logging methods.

        Like RealtimeLogger.warning(), RealtimeLogger.info(), etc.

        __getattr__(name)
            Fallback to attributes on the logger.

            Parameters
                name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                Any

    class toil.realtimeLogger.RealtimeLogger(batchSystem, level=defaultLevel)
        Provide a logger that logs over UDP to the leader.

        To use in a Toil job, do:

            >>> from toil.realtimeLogger import RealtimeLogger
            >>> RealtimeLogger.info("This logging message goes straight to the leader")

        That's all a user of Toil would need to do. On the leader, Job.Runner.startToil() automatically starts the UDP server by using an instance of this class as a context manager.

        Parameters

            •   batchSystem (toil.batchSystems.abstractBatchSystem.AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>)

            •   level (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        envPrefix = 'TOIL_RT_LOGGING_'

        defaultLevel = 'INFO'

        lock

        loggingServer = None

        serverThread = None

        initialized = 0

        logger = None

        classmethod getLogger()
            Get the logger that logs real-time to the leader.

            Note that if the returned logger is used on the leader, you will see the message twice, since it still goes to the normal log handlers, too.

            Return type
                logging.Logger </usr/share/doc/python3/html/library/logging.html#logging.Logger>

        __enter__()

            Return type
                None

        __exit__(exc_type, exc_val, exc_tb)

            Parameters

                •   exc_type (Optional[type </usr/share/doc/python3/html/library/functions.html#type>[BaseException </usr/share/doc/python3/html/library/exceptions.html#BaseException>]])

                •   exc_val (Optional[BaseException </usr/share/doc/python3/html/library/exceptions.html#BaseException>])

                •   exc_tb (Optional[types.TracebackType </usr/share/doc/python3/html/library/types.html#types.TracebackType>])

            Return type
                None

  toil.resource

  Attributes
logger |

  Exceptions
ResourceException | Common base class for all non-exit exceptions.

  Classes
Resource | Represents a file or directory that will be deployed to each node before any jobs in the user script are invoked.
FileResource | A resource read from a file on the leader.
DirectoryResource | A resource read from a directory on the leader.
VirtualEnvResource | A resource read from a virtualenv on the leader.
ModuleDescriptor | A path to a Python module decomposed into a namedtuple of three elements

  Module Contents

    toil.resource.logger

    class toil.resource.Resource
        Bases: namedtuple('Resource', ('name', 'pathHash', 'url', 'contentHash'))

        Represents a file or directory that will be deployed to each node before any jobs in the user script are invoked.

        Each instance is a namedtuple with the following elements:

        The pathHash element contains the MD5 (in hexdigest form) of the path to the resource on the leader node. The path, and therefore its hash is unique within a job store.

        The url element is a "file:" or "http:" URL at which the resource can be obtained.

        The contentHash element is an MD5 checksum of the resource, allowing for validation and caching of resources.

        If the resource is a regular file, the type attribute will be 'file'.

        If the resource is a directory, the type attribute will be 'dir' and the URL will point at a ZIP archive of that directory.

        resourceEnvNamePrefix = 'JTRES_'

        rootDirPathEnvName = 'JTRES_ROOT'

        classmethod create(jobStore, leaderPath)
            Saves the content of the file or directory at the given path to the given job store and returns a resource object representing that content for the purpose of obtaining it again at a generic, public URL. This method should be invoked on the leader node.

            Parameters

                •   jobStore (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

                •   leaderPath (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                Resource <#toil.resource.Resource>

        refresh(jobStore)

            Parameters
                jobStore (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

            Return type
                Resource <#toil.resource.Resource>

        classmethod prepareSystem()
            Prepares this system for the downloading and lookup of resources. This method should only be invoked on a worker node. It is idempotent but not thread-safe.

            Return type
                None

        classmethod cleanSystem()
            Remove all downloaded, localized resources.

            Return type
                None

        register()
            Register this resource for later retrieval via lookup(), possibly in a child process.

            Return type
                None

        classmethod lookup(leaderPath)
            Return a resource object representing a resource created from a file or directory at the given path on the leader.

            This method should be invoked on the worker. The given path does not need to refer to an existing file or directory on the worker, it only identifies the resource within an instance of toil. This method returns None if no resource for the given path exists.

            Parameters
                leaderPath (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                Optional[Resource <#toil.resource.Resource>]

        download(callback=None)
            Download this resource from its URL to a file on the local system.

            This method should only be invoked on a worker node after the node was setup for accessing resources via prepareSystem().

            Parameters
                callback (Optional[Callable[[str </usr/share/doc/python3/html/library/stdtypes.html#str>], None]])

            Return type
                None

        property localPath: str </usr/share/doc/python3/html/library/stdtypes.html#str>

            Abstractmethod

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        Get the path to resource on the worker.

        The file or directory at the returned path may or may not yet exist. Invoking download() will ensure that it does.

        property localDirPath: str </usr/share/doc/python3/html/library/stdtypes.html#str>
            The path to the directory containing the resource on the worker.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        pickle()

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        classmethod unpickle(s)

            Parameters
                s (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                Resource <#toil.resource.Resource>

    class toil.resource.FileResource
        Bases: Resource

        A resource read from a file on the leader.

        property localPath: str </usr/share/doc/python3/html/library/stdtypes.html#str>
            Get the path to resource on the worker.

            The file or directory at the returned path may or may not yet exist. Invoking download() will ensure that it does.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

    class toil.resource.DirectoryResource
        Bases: Resource

        A resource read from a directory on the leader.

        The URL will point to a ZIP archive of the directory. All files in that directory (and any subdirectories) will be included. The directory may be a package but it does not need to be.

        property localPath: str </usr/share/doc/python3/html/library/stdtypes.html#str>
            Get the path to resource on the worker.

            The file or directory at the returned path may or may not yet exist. Invoking download() will ensure that it does.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

    class toil.resource.VirtualEnvResource
        Bases: DirectoryResource

        A resource read from a virtualenv on the leader.

        All modules and packages found in the virtualenv's site-packages directory will be included.

    class toil.resource.ModuleDescriptor
        Bases: namedtuple('ModuleDescriptor', ('dirPath', 'name', 'fromVirtualEnv'))

        A path to a Python module decomposed into a namedtuple of three elements

        •   dirPath, the path to the directory that should be added to sys.path before importing the module,

        •   moduleName, the fully qualified name of the module with leading package names separated by dot and

        >>> import toil.resource
        >>> ModuleDescriptor.forModule('toil.resource')
        ModuleDescriptor(dirPath='/.../src', name='toil.resource', fromVirtualEnv=False)

        >>> import subprocess, tempfile, os
        >>> dirPath = tempfile.mkdtemp()
        >>> path = os.path.join( dirPath, 'foo.py' )
        >>> with open(path,'w') as f:
        ...     _ = f.write('from toil.resource import ModuleDescriptor\n'
        ...                 'print(ModuleDescriptor.forModule(__name__))')
        >>> subprocess.check_output([ sys.executable, path ])
        b"ModuleDescriptor(dirPath='...', name='foo', fromVirtualEnv=False)\n"

        >>> from shutil import rmtree
        >>> rmtree( dirPath )

    Now test a collision. 'collections' is part of the standard library in Python 2 and 3. >>> dirPath = tempfile.mkdtemp() >>> path = os.path.join( dirPath, 'collections.py' ) >>> with open(path,'w') as f: ... _ = f.write('from toil.resource import ModuleDescriptorn' ... 'ModuleDescriptor.forModule(__name__)')

    This should fail and return exit status 1 due to the collision with the built-in module: >>> subprocess.call([ sys.executable, path ]) 1

    Clean up >>> rmtree( dirPath )

        dirPath: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        name: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        classmethod forModule(name)
            Return an instance of this class representing the module of the given name.

            If the given module name is "__main__", it will be translated to the actual file name of the top-level script without the .py or .pyc extension. This method assumes that the module with the specified name has already been loaded.

            Parameters
                name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                ModuleDescriptor <#toil.resource.ModuleDescriptor>

        property belongsToToil: bool </usr/share/doc/python3/html/library/functions.html#bool>
            True if this module is part of the Toil distribution

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        saveAsResourceTo(jobStore)
            Store the file containing this module--or even the Python package directory hierarchy containing that file--as a resource to the given job store and return the corresponding resource object. Should only be called on a leader node.

            Parameters
                jobStore (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

            Return type
                Resource <#toil.resource.Resource>

        localize()
            Check if this module was saved as a resource.

            If it was, return a new module descriptor that points to a local copy of that resource. Should only be called on a worker node. On the leader, this method returns this resource, i.e. self.

            Return type
                ModuleDescriptor <#toil.resource.ModuleDescriptor>

        globalize()
            Reverse the effect of localize().

            Return type
                ModuleDescriptor <#toil.resource.ModuleDescriptor>

        toCommand()

            Return type
                collections.abc.Sequence </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Sequence>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        classmethod fromCommand(command)

            Parameters
                command (collections.abc.Sequence </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Sequence>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                ModuleDescriptor <#toil.resource.ModuleDescriptor>

        makeLoadable()

            Return type
                ModuleDescriptor <#toil.resource.ModuleDescriptor>

        load()

            Return type
                Optional[types.ModuleType </usr/share/doc/python3/html/library/types.html#types.ModuleType>]

    exception toil.resource.ResourceException
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Common base class for all non-exit exceptions.

  toil.server

  Submodules

  toil.server.api_spec

  toil.server.app

  Attributes
logger |

  Functions
parser_with_server_options() |
create_app(args) | Create a "connexion.FlaskApp" instance with Toil server configurations.
start_server(args) | Start a Toil server.

  Module Contents

    toil.server.app.logger

    toil.server.app.parser_with_server_options()

        Return type
            argparse.ArgumentParser </usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser>

    toil.server.app.create_app(args)
        Create a "connexion.FlaskApp" instance with Toil server configurations.

        Parameters
            args (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)

        Return type
            connexion.FlaskApp

    toil.server.app.start_server(args)
        Start a Toil server.

        Parameters
            args (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)

        Return type
            None

  toil.server.celery_app

  Attributes
celery |

  Functions
create_celery_app() |

  Module Contents

    toil.server.celery_app.create_celery_app()

        Return type
            celery.Celery

    toil.server.celery_app.celery

  toil.server.cli

  Submodules

  toil.server.cli.wes_cwl_runner

  Attributes
logger |

  Classes
WESClientWithWorkflowEngineParameters | A modified version of the WESClient from the wes-service package that

  Functions
generate_attachment_path_names(paths) | Take in a list of path names and return a list of names with the common path
get_deps_from_cwltool(cwl_file[, input_file]) | Return a list of dependencies of the given workflow from cwltool.
submit_run(client, cwl_file[, input_file, engine_options]) | Given a CWL file, its input files, and an optional list of engine options,
poll_run(client, run_id) | Return True if the given workflow run is in a finished state.
print_logs_and_exit(client, run_id) | Fetch the workflow logs from the WES server, print the results, then exit
main() |

  Module Contents

    toil.server.cli.wes_cwl_runner.logger

    toil.server.cli.wes_cwl_runner.generate_attachment_path_names(paths)
        Take in a list of path names and return a list of names with the common path name stripped out, while preserving the input order. This guarantees that there are no relative paths that traverse up.

        For example, for the following CWL workflow where "hello.yaml" references a file "message.txt",
            ~/toil/workflows/hello.cwl ~/toil/input_files/hello.yaml ~/toil/input_files/message.txt

        This may be run with the command:
            toil-wes-cwl-runner hello.cwl ../input_files/hello.yaml

    Where "message.txt" is resolved to "../input_files/message.txt".

    We'd send the workflow file as "workflows/hello.cwl", and send the inputs as "input_files/hello.yaml" and "input_files/message.txt".

        Parameters
            paths (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- A list of absolute or relative path names. Relative paths are interpreted as relative to the current working directory.

        Returns
            The common path name and a list of minimal path names.

        Return type
            tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

    class toil.server.cli.wes_cwl_runner.WESClientWithWorkflowEngineParameters(endpoint, auth=None)
        Bases: wes_client.util.WESClient

        A modified version of the WESClient from the wes-service package that includes workflow_engine_parameters support.

        TODO: Propose a PR in wes-service to include workflow_engine_params.

        Parameters

            •   endpoint (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   auth (Optional[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

        get_version(extension, workflow_file)
            Determines the version of a .py, .wdl, or .cwl file.

            Parameters

                •   extension (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   workflow_file (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        parse_params(workflow_params_file)
            Parse the CWL input file into a dictionary to be attached to the body of the WES run request.

            Parameters
                workflow_params_file (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The URL or path to the CWL input file.

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]

        modify_param_paths(base_dir, workflow_params)
            Modify the file paths in the input workflow parameters to be relative to base_dir.

            Parameters

                •   base_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The base directory to make the file paths relative to. This should be the common ancestor of all attached files, which will become the root of the execution folder.

                •   workflow_params (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]) -- A dict containing the workflow parameters.

            Return type
                None

        build_wes_request(workflow_file, workflow_params_file, attachments, workflow_engine_parameters=None)
            Build the workflow run request to submit to WES.

            Parameters

                •   workflow_file (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The path or URL to the CWL workflow document. Only <file://> URL supported at the moment.

                •   workflow_params_file (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- The path or URL to the CWL input file.

                •   attachments (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) -- A list of local paths to files that will be uploaded to the server.

                •   workflow_engine_parameters (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) -- A list of engine parameters to set along with this workflow run.

            Returns
                A dictionary of parameters as the body of the request, and an iterable for the pairs of filename and file contents to upload to the server.

            Return type
                tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>], collections.abc.Iterable </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterable>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, io.BytesIO </usr/share/doc/python3/html/library/io.html#io.BytesIO>]]]]

        run_with_engine_options(workflow_file, workflow_params_file, attachments, workflow_engine_parameters)
            Composes and sends a post request that signals the WES server to run a workflow.

            Parameters

                •   workflow_file (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The path to the CWL workflow document.

                •   workflow_params_file (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- The path to the CWL input file.

                •   attachments (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) -- A list of local paths to files that will be uploaded to the server.

                •   workflow_engine_parameters (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) -- A list of engine parameters to set along with this workflow run.

            Returns
                The body of the post result as a dictionary.

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]

    toil.server.cli.wes_cwl_runner.get_deps_from_cwltool(cwl_file, input_file=None)
        Return a list of dependencies of the given workflow from cwltool.

        Parameters

            •   cwl_file (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The CWL file.

            •   input_file (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Omit to get the dependencies from the CWL file. If set, this returns the dependencies from the input file.

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.server.cli.wes_cwl_runner.submit_run(client, cwl_file, input_file=None, engine_options=None)
        Given a CWL file, its input files, and an optional list of engine options, submit the CWL workflow to the WES server via the WES client.

        This function also attempts to find the attachments from the CWL workflow and its input file, and attach them to the WES run request.

        Parameters

            •   client (WESClientWithWorkflowEngineParameters <#toil.server.cli.wes_cwl_runner.WESClientWithWorkflowEngineParameters>) -- The WES client.

            •   cwl_file (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The path to the CWL workflow document.

            •   input_file (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- The path to the CWL input file.

            •   engine_options (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) -- A list of engine parameters to set along with this workflow run.

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.server.cli.wes_cwl_runner.poll_run(client, run_id)
        Return True if the given workflow run is in a finished state.

        Parameters

            •   client (WESClientWithWorkflowEngineParameters <#toil.server.cli.wes_cwl_runner.WESClientWithWorkflowEngineParameters>)

            •   run_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.server.cli.wes_cwl_runner.print_logs_and_exit(client, run_id)
        Fetch the workflow logs from the WES server, print the results, then exit the program with the same exit code as the workflow run.

        Parameters

            •   client (WESClientWithWorkflowEngineParameters <#toil.server.cli.wes_cwl_runner.WESClientWithWorkflowEngineParameters>) -- The WES client.

            •   run_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The run_id of the target workflow.

        Return type
            None

    toil.server.cli.wes_cwl_runner.main()

        Return type
            None

  toil.server.utils

  Attributes
HAVE_S3 |
logger |
state_store_cache |
TERMINAL_STATES |
MAX_CANCELING_SECONDS |

  Classes
MemoryStateCache | An in-memory place to store workflow state.
AbstractStateStore | A place for the WES server to keep its state: the set of workflows that
MemoryStateStore | An in-memory place to store workflow state, for testing.
FileStateStore | A place to store workflow state that uses a POSIX-compatible file system.
S3StateStore | A place to store workflow state that uses an S3-compatible object store.
WorkflowStateStore | Slice of a state store for the state of a particular workflow.
WorkflowStateMachine | Class for managing the WES workflow state machine.

  Functions
get_iso_time() | Return the current time in ISO 8601 format.
link_file(src, dest) | Create a link to a file from src to dest.
download_file_from_internet(src, dest[, content_type]) | Download a file from the Internet and write it to dest.
download_file_from_s3(src, dest[, content_type]) | Download a file from Amazon S3 and write it to dest.
get_file_class(path) | Return the type of the file as a human readable string.
safe_read_file(file) | Safely read a file by acquiring a shared lock to prevent other processes
safe_write_file(file, s) | Safely write to a file by acquiring an exclusive lock to prevent other
connect_to_state_store(url) | Connect to a place to store state for workflows, defined by a URL.
connect_to_workflow_state_store(url, workflow_id) | Connect to a place to store state for the given workflow, in the state

  Module Contents

    toil.server.utils.HAVE_S3 = True

    toil.server.utils.logger

    toil.server.utils.get_iso_time()
        Return the current time in ISO 8601 format.

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.server.utils.link_file(src, dest)
        Create a link to a file from src to dest.

        Parameters

            •   src (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   dest (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            None

    toil.server.utils.download_file_from_internet(src, dest, content_type=None)
        Download a file from the Internet and write it to dest.

        Parameters

            •   src (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   dest (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   content_type (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            None

    toil.server.utils.download_file_from_s3(src, dest, content_type=None)
        Download a file from Amazon S3 and write it to dest.

        Parameters

            •   src (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   dest (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   content_type (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            None

    toil.server.utils.get_file_class(path)
        Return the type of the file as a human readable string.

        Parameters
            path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.server.utils.safe_read_file(file)
        Safely read a file by acquiring a shared lock to prevent other processes from writing to it while reading.

        Parameters
            file (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.server.utils.safe_write_file(file, s)
        Safely write to a file by acquiring an exclusive lock to prevent other processes from reading and writing to it while writing.

        Parameters

            •   file (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   s (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            None

    class toil.server.utils.MemoryStateCache
        An in-memory place to store workflow state.

        get(workflow_id, key)
            Get a key value from memory.

            Parameters

                •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        set(workflow_id, key, value)
            Set or clear a key value in memory.

            Parameters

                •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   value (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

    class toil.server.utils.AbstractStateStore
        A place for the WES server to keep its state: the set of workflows that exist and whether they are done or not.

        This is a key-value store, with keys namespaced by workflow ID. Concurrent access from multiple threads or processes is safe and globally consistent.

        Keys and workflow IDs are restricted to [-a-zA-Z0-9_], because backends may use them as path or URL components.

        Key values are either a string, or None if the key is not set.

        Workflow existence isn't a thing; nonexistent workflows just have None for all keys.

        Note that we don't yet have a cleanup operation: things are stored permanently. Even clearing all the keys may leave data behind.

        Also handles storage for a local cache, with a separate key namespace (not a read/write-through cache).

        TODO: Can we replace this with just using a JobStore eventually, when AWSJobStore no longer needs SimpleDB?

        abstractmethod get(workflow_id, key)
            Get the value of the given key for the given workflow, or None if the key is not set for the workflow.

            Parameters

                •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        abstractmethod set(workflow_id, key, value)
            Set the value of the given key for the given workflow. If the value is None, clear the key.

            Parameters

                •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   value (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

        read_cache(workflow_id, key)
            Read a value from a local cache, without checking the actual backend.

            Parameters

                •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        write_cache(workflow_id, key, value)
            Write a value to a local cache, without modifying the actual backend.

            Parameters

                •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   value (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

    class toil.server.utils.MemoryStateStore
        Bases: MemoryStateCache, AbstractStateStore

        An in-memory place to store workflow state, for testing.

        Inherits from MemoryStateCache first to provide implementations for AbstractStateStore.

    class toil.server.utils.FileStateStore(url)
        Bases: AbstractStateStore

        A place to store workflow state that uses a POSIX-compatible file system.

        Parameters
            url (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        get(workflow_id, key)
            Get a key value from the filesystem.

            Parameters

                •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        set(workflow_id, key, value)
            Set or clear a key value on the filesystem.

            Parameters

                •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   value (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

    class toil.server.utils.S3StateStore(url)
        Bases: AbstractStateStore

        A place to store workflow state that uses an S3-compatible object store.

        Parameters
            url (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        get(workflow_id, key)
            Get a key value from S3.

            Parameters

                •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        set(workflow_id, key, value)
            Set or clear a key value on S3.

            Parameters

                •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   value (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

    toil.server.utils.state_store_cache: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, AbstractStateStore <#toil.server.utils.AbstractStateStore>]

    toil.server.utils.connect_to_state_store(url)
        Connect to a place to store state for workflows, defined by a URL.

        URL may be a local file path or URL or an S3 URL.

        Parameters
            url (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            AbstractStateStore <#toil.server.utils.AbstractStateStore>

    class toil.server.utils.WorkflowStateStore(state_store, workflow_id)
        Slice of a state store for the state of a particular workflow.

        Parameters

            •   state_store (AbstractStateStore <#toil.server.utils.AbstractStateStore>)

            •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        get(key)
            Get the given item of workflow state.

            Parameters
                key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        set(key, value)
            Set the given item of workflow state.

            Parameters

                •   key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   value (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

        read_cache(key)
            Read a value from a local cache, without checking the actual backend.

            Parameters
                key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        write_cache(key, value)
            Write a value to a local cache, without modifying the actual backend.

            Parameters

                •   key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   value (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

    toil.server.utils.connect_to_workflow_state_store(url, workflow_id)
        Connect to a place to store state for the given workflow, in the state store defined by the given URL.

        Parameters

            •   url (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- A URL that can be used for connect_to_state_store()

            •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            WorkflowStateStore <#toil.server.utils.WorkflowStateStore>

    toil.server.utils.TERMINAL_STATES

    toil.server.utils.MAX_CANCELING_SECONDS = 30

    class toil.server.utils.WorkflowStateMachine(store)
        Class for managing the WES workflow state machine.

        This is the authority on the WES "state" of a workflow. You need one to read or change the state.

        Guaranteeing that only certain transitions can be observed is possible but not worth it. Instead, we just let updates clobber each other and grab and cache the first terminal state we see forever. If it becomes important that clients never see e.g. CANCELED -> COMPLETE or COMPLETE -> SYSTEM_ERROR, we can implement a real distributed state machine here.

        We do handle making sure that tasks don't get stuck in CANCELING.

        State can be:

        "UNKNOWN" "QUEUED" "INITIALIZING" "RUNNING" "PAUSED" "COMPLETE" "EXECUTOR_ERROR" "SYSTEM_ERROR" "CANCELED" "CANCELING"

        Uses the state store's local cache to prevent needing to read things we've seen already.

        Parameters
            store (WorkflowStateStore <#toil.server.utils.WorkflowStateStore>)

        send_enqueue()
            Send an enqueue message that would move from UNKNOWN to QUEUED.

            Return type
                None

        send_initialize()
            Send an initialize message that would move from QUEUED to INITIALIZING.

            Return type
                None

        send_run()
            Send a run message that would move from INITIALIZING to RUNNING.

            Return type
                None

        send_cancel()
            Send a cancel message that would move to CANCELING from any non-terminal state.

            Return type
                None

        send_canceled()
            Send a canceled message that would move to CANCELED from CANCELLING.

            Return type
                None

        send_complete()
            Send a complete message that would move from RUNNING to COMPLETE.

            Return type
                None

        send_executor_error()
            Send an executor_error message that would move from QUEUED, INITIALIZING, or RUNNING to EXECUTOR_ERROR.

            Return type
                None

        send_system_error()

            Send a system_error message that would move from QUEUED, INITIALIZING,
                or RUNNING to SYSTEM_ERROR.

            Return type
                None

        get_current_state()
            Get the current state of the workflow.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

  toil.server.wes

  Submodules

  toil.server.wes.abstract_backend

  Attributes
logger |
TaskLog |

  Exceptions
VersionNotImplementedException | Raised when the requested workflow version is not implemented.
MalformedRequestException | Raised when the request is malformed.
WorkflowNotFoundException | Raised when the requested run ID is not found.
WorkflowConflictException | Raised when the requested workflow is not in the expected state.
OperationForbidden | Raised when the request is forbidden.
WorkflowExecutionException | Raised when an internal error occurred during the execution of the workflow.

  Classes
WESBackend | A class to represent a GA4GH Workflow Execution Service (WES) API backend.

  Functions
handle_errors(func) | This decorator catches errors from the wrapped function and returns a JSON

  Module Contents

    toil.server.wes.abstract_backend.logger

    toil.server.wes.abstract_backend.TaskLog

    exception toil.server.wes.abstract_backend.VersionNotImplementedException(wf_type, version=None, supported_versions=None)
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Raised when the requested workflow version is not implemented.

        Parameters

            •   wf_type (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   version (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   supported_versions (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

    exception toil.server.wes.abstract_backend.MalformedRequestException(message)
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Raised when the request is malformed.

        Parameters
            message (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

    exception toil.server.wes.abstract_backend.WorkflowNotFoundException
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Raised when the requested run ID is not found.

    exception toil.server.wes.abstract_backend.WorkflowConflictException(run_id)
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Raised when the requested workflow is not in the expected state.

        Parameters
            run_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

    exception toil.server.wes.abstract_backend.OperationForbidden(message)
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Raised when the request is forbidden.

        Parameters
            message (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

    exception toil.server.wes.abstract_backend.WorkflowExecutionException(message)
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Raised when an internal error occurred during the execution of the workflow.

        Parameters
            message (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

    toil.server.wes.abstract_backend.handle_errors(func)
        This decorator catches errors from the wrapped function and returns a JSON formatted error message with the appropriate status code defined by the GA4GH WES spec.

        Parameters
            func (Callable[Ellipsis, Any])

        Return type
            Callable[Ellipsis, Any]

    class toil.server.wes.abstract_backend.WESBackend(options)
        A class to represent a GA4GH Workflow Execution Service (WES) API backend. Intended to be inherited. Subclasses should implement all abstract methods to handle user requests when they hit different endpoints.

        Parameters
            options (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        options

        resolve_operation_id(operation_id)
            Map an operationId defined in the OpenAPI or swagger yaml file to a function.

            Parameters
                operation_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The operation ID defined in the specification.

            Returns
                A function that should be called when the given endpoint is reached.

            Return type
                Any

        abstractmethod get_service_info()
            Get information about the Workflow Execution Service.

            GET /service-info

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]

        abstractmethod list_runs(page_size=None, page_token=None)
            List the workflow runs.

            GET /runs

            Parameters

                •   page_size (Optional[int </usr/share/doc/python3/html/library/functions.html#int>])

                •   page_token (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]

        abstractmethod run_workflow()
            Run a workflow. This endpoint creates a new workflow run and returns a RunId to monitor its progress.

            POST /runs

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        abstractmethod get_run_log(run_id)
            Get detailed info about a workflow run.

            GET /runs/{run_id}

            Parameters
                run_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]

        abstractmethod cancel_run(run_id)
            Cancel a running workflow.

            POST /runs/{run_id}/cancel

            Parameters
                run_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        abstractmethod get_run_status(run_id)
            Get quick status info about a workflow run, returning a simple result with the overall state of the workflow run.

            GET /runs/{run_id}/status

            Parameters
                run_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        static log_for_run(run_id, message)

            Parameters

                •   run_id (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   message (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        static secure_path(path)

            Parameters
                path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        collect_attachments(args, run_id, temp_dir)
            Collect attachments from the current request by staging uploaded files to temp_dir, and return the temp_dir and parsed body of the request.

            Parameters

                •   run_id (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- The run ID for logging.

                •   temp_dir (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- The directory where uploaded files should be staged. If None, a temporary directory is created.

                •   args (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any])

            Return type
                tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]]

  toil.server.wes.amazon_wes_utils

  Attributes
logger |
NOTICE |

  Classes
WorkflowPlan | These functions pass around dicts of a certain type, with data and files keys.
DataDict | Under data, there can be:
FilesDict | Under files, there can be:

  Functions
parse_workflow_zip_file(file, workflow_type) | Processes a workflow zip bundle
parse_workflow_manifest_file(manifest_file) | Reads a MANIFEST.json file for a workflow zip bundle
workflow_manifest_url_to_path(url[, parent_dir]) | Interpret a possibly-relative parsed URL, relative to the given parent directory.
task_filter(task, job_status) | AGC requires task names to be annotated with an AWS Batch job ID that they

  Module Contents

    toil.server.wes.amazon_wes_utils.logger

    toil.server.wes.amazon_wes_utils.NOTICE = Multiline-String

                """
                Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
                """

    class toil.server.wes.amazon_wes_utils.WorkflowPlan
        Bases: TypedDict

        These functions pass around dicts of a certain type, with data and files keys.

        data: DataDict <#toil.server.wes.amazon_wes_utils.DataDict>

        files: FilesDict <#toil.server.wes.amazon_wes_utils.FilesDict>

    class toil.server.wes.amazon_wes_utils.DataDict
        Bases: TypedDict

        Under data, there can be: * workflowUrl (required if no workflowSource): URL to main workflow code.

        workflowUrl: str </usr/share/doc/python3/html/library/stdtypes.html#str>

    class toil.server.wes.amazon_wes_utils.FilesDict
        Bases: TypedDict

        Under files, there can be: * workflowSource (required if no workflowUrl): Open binary-mode file for the main workflow code. * workflowInputFiles: List of open binary-mode file for input files. Expected to be JSONs. * workflowOptions: Open binary-mode file for a JSON of options sent along with the workflow. * workflowDependencies: Open binary-mode file for the zip the workflow came in, if any.

        workflowSource: IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]

        workflowInputFiles: list </usr/share/doc/python3/html/library/stdtypes.html#list>[IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]]

        workflowOptions: IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]

        workflowDependencies: IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]

    toil.server.wes.amazon_wes_utils.parse_workflow_zip_file(file, workflow_type)
        Processes a workflow zip bundle

        Parameters

            •   file (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- String or Path-like path to a workflow.zip file

            •   workflow_type (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- String, extension of workflow to expect (e.g. "wdl")

        Return type
            dict of data and files

    If the zip only contains a single file, that file is set as workflowSource

    If the zip contains multiple files with a MANIFEST.json file, the MANIFEST is used to determine appropriate data and file arguments. (See: parse_workflow_manifest_file())

        If the zip contains multiple files without a MANIFEST.json file:

            •   a main workflow file with an extension matching the workflow_type is expected and will be set as workflowSource

            •   optionally, if inputs*.json files are found in the root level of the zip, they will be set as workflowInputs(_d)* in the order they are found

            •   optionally, if an options.json file is found in the root level of the zip, it will be set as workflowOptions

    If the zip contains multiple files, the original zip is set as workflowDependencies

    toil.server.wes.amazon_wes_utils.parse_workflow_manifest_file(manifest_file)
        Reads a MANIFEST.json file for a workflow zip bundle

        Parameters
            manifest_file (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- String or Path-like path to a MANIFEST.json file

        Return type
            dict of data and files

    MANIFEST.json is expected to be formatted like:

                {
                    "mainWorkflowURL": "relpath/to/workflow",
                    "inputFileURLs": [
                        "relpath/to/input-file-1",
                        "relpath/to/input-file-2",
                        "relpath/to/input-file-3"
                    ],
                    "optionsFileURL": "relpath/to/option-file"
                }

    The mainWorkflowURL property that provides a relative file path in the zip to a workflow file, which will be set as workflowSource

    The inputFileURLs property is optional and provides a list of relative file paths in the zip to input.json files. The list is assumed to be in the order the inputs should be applied - e.g. higher list index is higher priority. If present, it will be used to set workflowInputs(_d) arguments.

    The optionsFileURL property is optional and provides a relative file path in the zip to an options.json file. If present, it will be used to set workflowOptions.

    toil.server.wes.amazon_wes_utils.workflow_manifest_url_to_path(url, parent_dir=None)
        Interpret a possibly-relative parsed URL, relative to the given parent directory.

        Parameters

            •   url (urllib.parse.ParseResult </usr/share/doc/python3/html/library/urllib.parse.html#urllib.parse.ParseResult>)

            •   parent_dir (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.server.wes.amazon_wes_utils.task_filter(task, job_status)
        AGC requires task names to be annotated with an AWS Batch job ID that they were run under. If it encounters an un-annotated task name, it will crash. See <<https://github.com/aws/amazon-genomics-cli/issues/494>>.

        This encodes the AWSBatchJobID annotation, from the AmazonBatchBatchSystem, into the task name of the given task, and returns the modified task. If no such annotation is available, the task is censored and None is returned.

        Parameters

            •   task (toil.server.wes.abstract_backend.TaskLog)

            •   job_status (toil.bus.JobStatus <#toil.bus.JobStatus>)

        Return type
            Optional[toil.server.wes.abstract_backend.TaskLog]

  toil.server.wes.tasks

  Attributes
logger |
WAIT_FOR_DEATH_TIMEOUT |
run_wes |

  Classes
ToilWorkflowRunner | A class to represent a workflow runner to run the requested workflow.
TaskRunner | Abstraction over the Celery API. Runs our run_wes task and allows canceling it.
MultiprocessingTaskRunner | Version of TaskRunner that just runs tasks with Multiprocessing.

  Functions
run_wes_task(base_scratch_dir, state_store_url, ...) | Run a requested workflow.
cancel_run(task_id) | Send a SIGTERM signal to the process that is running task_id.

  Module Contents

    toil.server.wes.tasks.logger

    toil.server.wes.tasks.WAIT_FOR_DEATH_TIMEOUT = 20

    class toil.server.wes.tasks.ToilWorkflowRunner(base_scratch_dir, state_store_url, workflow_id, request, engine_options)
        A class to represent a workflow runner to run the requested workflow.

        Responsible for parsing the user request into a shell command, executing that command, and collecting the outputs of the resulting workflow run.

        Parameters

            •   base_scratch_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   state_store_url (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   request (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any])

            •   engine_options (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        scratch_dir

        store

        state_machine

        request

        engine_options

        wf_type: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        version: str </usr/share/doc/python3/html/library/stdtypes.html#str>

        exec_dir

        out_dir

        default_job_store

        job_store

        write_scratch_file(filename, contents)
            Write a file to the scratch directory.

            Parameters

                •   filename (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   contents (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        get_state()

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        write_workflow(src_url)
            Fetch the workflow file from its source and write it to a destination file.

            Parameters
                src_url (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        sort_options(workflow_engine_parameters=None)
            Sort the command line arguments in the order that can be recognized by the workflow execution engine.

            Parameters
                workflow_engine_parameters (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]]) -- User-specified parameters for this particular workflow. Keys are command-line options, and values are option arguments, or None for options that are flags.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        initialize_run()
            Write workflow and input files and construct a list of shell commands to be executed. Return that list of shell commands that should be executed in order to complete this workflow run.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        call_cmd(cmd, cwd)
            Calls a command with Popen. Writes stdout, stderr, and the command to separate files.

            Parameters

                •   cmd (Union[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>], str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   cwd (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                subprocess.Popen </usr/share/doc/python3/html/library/subprocess.html#subprocess.Popen>[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]

        run()
            Construct a command to run a the requested workflow with the options, run it, and deposit the outputs in the output directory.

            Return type
                None

        write_output_files()
            Fetch all the files that this workflow generated and output information about them to outputs.json.

            Return type
                None

    toil.server.wes.tasks.run_wes_task(base_scratch_dir, state_store_url, workflow_id, request, engine_options)
        Run a requested workflow.

        Parameters

            •   base_scratch_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Directory where the workflow's scratch dir will live, under the workflow's ID.

            •   state_store_url (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- URL/path at which the server and Celery task communicate about workflow state.

            •   workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- ID of the workflow run.

            •   request (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any])

            •   engine_options (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Returns
            the state of the workflow run.

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.server.wes.tasks.run_wes

    toil.server.wes.tasks.cancel_run(task_id)
        Send a SIGTERM signal to the process that is running task_id.

        Parameters
            task_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            None

    class toil.server.wes.tasks.TaskRunner
        Abstraction over the Celery API. Runs our run_wes task and allows canceling it.

        We can swap this out in the server to allow testing without Celery.

        static run(args, task_id)
            Run the given task args with the given ID on Celery.

            Parameters

                •   args (tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>, dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any], list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

                •   task_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        static cancel(task_id)
            Cancel the task with the given ID on Celery.

            Parameters
                task_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        static is_ok(task_id)
            Make sure that the task running system is working for the given task. If the task system has detected an internal failure, return False.

            Parameters
                task_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

    class toil.server.wes.tasks.MultiprocessingTaskRunner
        Bases: TaskRunner

        Version of TaskRunner that just runs tasks with Multiprocessing.

        Can't use threading because there's no way to send a cancel signal or exception to a Python thread, if loops in the task (i.e. ToilWorkflowRunner) don't poll for it.

        static set_up_and_run_task(output_path, args)
            Set up logging for the process into the given file and then call run_wes_task with the given arguments.

            If the process finishes successfully, it will clean up the log, but if the process crashes, the caller must clean up the log.

            Parameters

                •   output_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   args (tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>, dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any], list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

            Return type
                None

        classmethod run(args, task_id)
            Run the given task args with the given ID.

            Parameters

                •   args (tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>, dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any], list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

                •   task_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        classmethod cancel(task_id)
            Cancel the task with the given ID.

            Parameters
                task_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        classmethod is_ok(task_id)
            Make sure that the task running system is working for the given task. If the task system has detected an internal failure, return False.

            Parameters
                task_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

  toil.server.wes.toil_backend

  Attributes
logger |

  Classes
ToilWorkflow |
ToilBackend | WES backend implemented for Toil to run CWL, WDL, or Toil workflows. This

  Module Contents

    toil.server.wes.toil_backend.logger

    class toil.server.wes.toil_backend.ToilWorkflow(base_work_dir, state_store_url, run_id)

        Parameters

            •   base_work_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   state_store_url (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   run_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        base_scratch_dir

        state_store_url

        run_id

        scratch_dir

        exec_dir

        store

        state_machine

        fetch_state(key: str </usr/share/doc/python3/html/library/stdtypes.html#str>, default: str </usr/share/doc/python3/html/library/stdtypes.html#str>) -> str </usr/share/doc/python3/html/library/stdtypes.html#str>

        fetch_state(key: str </usr/share/doc/python3/html/library/stdtypes.html#str>, default: None </usr/share/doc/python3/html/library/constants.html#None> = None) -> str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>
            Return the contents of the given key in the workflow's state store. If the key does not exist, the default value is returned.

        fetch_scratch(filename)
            Get a context manager for either a stream for the given file from the workflow's scratch directory, or None if it isn't there.

            Parameters
                filename (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                collections.abc.Generator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator>[Optional[TextIO], None, None]

        exists()
            Return True if the workflow run exists.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        get_state()
            Return the state of the current run.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        check_on_run(task_runner)
            Check to make sure nothing has gone wrong in the task runner for this workflow. If something has, log, and fail the workflow with an error.

            Parameters
                task_runner (type </usr/share/doc/python3/html/library/functions.html#type>[toil.server.wes.tasks.TaskRunner <#toil.server.wes.tasks.TaskRunner>])

            Return type
                None

        set_up_run()
            Set up necessary directories for the run.

            Return type
                None

        clean_up()
            Clean directory and files related to the run.

            Return type
                None

        queue_run(task_runner, request, options)
            This workflow should be ready to run. Hand this to the task system.

            Parameters

                •   task_runner (type </usr/share/doc/python3/html/library/functions.html#type>[toil.server.wes.tasks.TaskRunner <#toil.server.wes.tasks.TaskRunner>])

                •   request (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any])

                •   options (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

        get_output_files()
            Return a collection of output files that this workflow generated.

            Return type
                Any

        get_stdout_path()
            Return the path to the standard output log, relative to the run's scratch_dir, or None if it doesn't exist.

            Return type
                Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        get_stderr_path()
            Return the path to the standard output log, relative to the run's scratch_dir, or None if it doesn't exist.

            Return type
                Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        get_messages_path()
            Return the path to the bus message log, relative to the run's scratch_dir, or None if it doesn't exist.

            Return type
                Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        get_task_logs(filter_function=None)
            Return all the task log objects for the individual tasks in the workflow.

            Task names will be the job_type values from issued/completed/failed messages, with annotations from JobAnnotationMessage messages if available.

            Parameters
                filter_function (Optional[Callable[[toil.server.wes.abstract_backend.TaskLog, toil.bus.JobStatus <#toil.bus.JobStatus>], Optional[toil.server.wes.abstract_backend.TaskLog]]]) -- If set, will be called with each task log and its job annotations. Returns a modified copy of the task log to actually report, or None if the task log should be omitted.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Union[str </usr/share/doc/python3/html/library/stdtypes.html#str>, int </usr/share/doc/python3/html/library/functions.html#int>, None]]]

    class toil.server.wes.toil_backend.ToilBackend(work_dir, state_store, options, dest_bucket_base, bypass_celery=False, wes_dialect='standard')
        Bases: toil.server.wes.abstract_backend.WESBackend <#toil.server.wes.abstract_backend.WESBackend>

        WES backend implemented for Toil to run CWL, WDL, or Toil workflows. This class is responsible for validating and executing submitted workflows.

        Parameters

            •   work_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   state_store (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   options (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   dest_bucket_base (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   bypass_celery (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            •   wes_dialect (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        run_id_prefix = 'run-'

        task_runner

        wes_dialect = 'standard'

        dest_bucket_base

        work_dir = b'.'

        supported_versions

        get_runs()
            A generator of a list of run ids and their state.

            Return type
                collections.abc.Generator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>], None, None]

        get_state(run_id)
            Return the state of the workflow run with the given run ID. May raise an error if the workflow does not exist.

            Parameters
                run_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        get_service_info()
            Get information about the Workflow Execution Service.

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]

        list_runs(page_size=None, page_token=None)
            List the workflow runs.

            Parameters

                •   page_size (Optional[int </usr/share/doc/python3/html/library/functions.html#int>])

                •   page_token (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]

        run_workflow(**args)
            Run a workflow.

            Parameters
                args (Any)

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        get_run_log(run_id)
            Get detailed info about a workflow run.

            Parameters
                run_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]

        cancel_run(run_id)
            Cancel a running workflow.

            Parameters
                run_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        get_run_status(run_id)
            Get quick status info about a workflow run, returning a simple result with the overall state of the workflow run.

            Parameters
                run_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        get_stdout(run_id)
            Get the stdout of a workflow run as a static file.

            Parameters
                run_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                Any

        get_stderr(run_id)
            Get the stderr of a workflow run as a static file.

            Parameters
                run_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                Any

        get_health()
            Return successfully if the server is healthy.

            Return type
                werkzeug.wrappers.response.Response

        get_homepage()
            Provide a sensible result for / other than 404.

            Return type
                werkzeug.wrappers.response.Response

  toil.server.wsgi_app

  Classes
GunicornApplication | An entry point to integrate a Gunicorn WSGI server in Python. To start a

  Functions
run_app(app[, options]) | Run a Gunicorn WSGI server.

  Module Contents

    class toil.server.wsgi_app.GunicornApplication(app, options=None)
        Bases: gunicorn.app.base.BaseApplication

        An entry point to integrate a Gunicorn WSGI server in Python. To start a WSGI application with callable app, run the following code:

                WSGIApplication(app, options={
                    ...
                }).run()

    For more details, see: <https://docs.gunicorn.org/en/latest/custom.html>

        Parameters

            •   app (object </usr/share/doc/python3/html/library/functions.html#object>)

            •   options (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]])

        options

        application

        init(*args)

            Parameters
                args (Any)

            Return type
                None

        load_config()

            Return type
                None

        load()

            Return type
                object </usr/share/doc/python3/html/library/functions.html#object>

    toil.server.wsgi_app.run_app(app, options=None)
        Run a Gunicorn WSGI server.

        Parameters

            •   app (object </usr/share/doc/python3/html/library/functions.html#object>)

            •   options (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]])

        Return type
            None

  toil.serviceManager

  Attributes
logger |

  Classes
ServiceManager | Manages the scheduling of services.

  Module Contents

    toil.serviceManager.logger

    class toil.serviceManager.ServiceManager(job_store, toil_state)
        Manages the scheduling of services.

        Parameters

            •   job_store (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

            •   toil_state (toil.toilState.ToilState <#toil.toilState.ToilState>)

        services_are_starting(job_id)
            Check if services are being started.

            Returns
                True if the services for the given job are currently being started, and False otherwise.

            Parameters
                job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        get_job_count()
            Get the total number of jobs we are working on.

            (services and their parent non-service jobs)

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        start()
            Start the service scheduling thread.

            Return type
                None

        put_client(client_id)
            Schedule the services of a job asynchronously.

            When the job's services are running the ID for the job will be returned by toil.leader.ServiceManager.get_ready_client.

            Parameters
                client_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- ID of job with services to schedule.

            Return type
                None

        get_ready_client(maxWait)
            Fetch a ready client, waiting as needed.

            Parameters
                maxWait (float </usr/share/doc/python3/html/library/functions.html#float>) -- Time in seconds to wait to get a JobDescription before returning

            Returns
                the ID of a client whose services are running, or None if no such job is available.

            Return type
                Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        get_unservable_client(maxWait)
            Fetch a client whos services failed to start.

            Parameters
                maxWait (float </usr/share/doc/python3/html/library/functions.html#float>) -- Time in seconds to wait to get a JobDescription before returning

            Returns
                the ID of a client whose services failed to start, or None if no such job is available.

            Return type
                Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        get_startable_service(maxWait)
            Fetch a service job that is ready to start.

            Parameters
                maxWait (float </usr/share/doc/python3/html/library/functions.html#float>) -- Time in seconds to wait to get a job before returning.

            Returns
                the ID of a service job that the leader can start, or None if no such job exists.

            Return type
                Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        kill_services(service_ids, error=False)
            Stop all the given service jobs.

            Parameters

                •   services -- Service jobStoreIDs to kill

                •   error (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Whether to signal that the service failed with an error when stopping it.

                •   service_ids (collections.abc.Iterable </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterable>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

        is_active(service_id)
            Return true if the service job has not been told to terminate.

            Parameters
                service_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Service to check on

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        is_running(service_id)
            Return true if the service job has started and is active.

            Parameters

                •   service -- Service to check on

                •   service_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        check()
            Check on the service manager thread.

            Raises
                RuntimeError </usr/share/doc/python3/html/library/exceptions.html#RuntimeError> -- If the underlying thread has quit.

            Return type
                None

        shutdown()
            Terminate worker threads cleanly; starting and killing all service threads.

            Will block until all services are started and blocked.

            Return type
                None

  toil.statsAndLogging

  Attributes
logger |
root_logger |
toil_logger |
DEFAULT_LOGLEVEL |
TRACE |

  Classes
StatsAndLogging | A thread to aggregate statistics and logging.

  Functions
set_log_level(level[, set_logger]) | Sets the root logger level to a given string level (like "INFO").
install_log_color([set_logger]) | Make logs colored.
add_logging_options(parser[, default_level]) | Add logging options to set the global log level.
configure_root_logger() | Set up the root logger with handlers and formatting.
log_to_file(log_file, log_rotation) |
set_logging_from_options(options) |
suppress_exotic_logging(local_logger) | Attempts to suppress the loggers of all non-Toil packages by setting them to CRITICAL.

  Module Contents

    toil.statsAndLogging.logger

    toil.statsAndLogging.root_logger

    toil.statsAndLogging.toil_logger

    toil.statsAndLogging.DEFAULT_LOGLEVEL = 20

    toil.statsAndLogging.TRACE = 5

    class toil.statsAndLogging.StatsAndLogging(jobStore, config)
        A thread to aggregate statistics and logging.

        Parameters

            •   jobStore (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

            •   config (toil.common.Config <#toil.common.Config>)

        start()
            Start the stats and logging thread.

            Return type
                None

        classmethod formatLogStream(stream, stream_name)
            Given a stream of text or bytes, and the job name, job itself, or some other optional stringifyable identity info for the job, return a big text string with the formatted job log, suitable for printing for the user.

            We don't want to prefix every line of the job's log with our own logging info, or we get prefixes wider than any reasonable terminal and longer than the messages.

            Parameters

                •   stream (Union[IO[str </usr/share/doc/python3/html/library/stdtypes.html#str>], IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]]) -- The stream of text or bytes to print for the user.

                •   stream_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        classmethod logWithFormatting(stream_name, jobLogs, method=logger.debug, message=None)

            Parameters

                •   stream_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   jobLogs (Union[IO[str </usr/share/doc/python3/html/library/stdtypes.html#str>], IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]])

                •   method (Callable[[str </usr/share/doc/python3/html/library/stdtypes.html#str>], None])

                •   message (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

        classmethod writeLogFiles(jobNames, jobLogList, config, failed=False)

            Parameters

                •   jobNames (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   jobLogList (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   config (toil.common.Config <#toil.common.Config>)

                •   failed (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        classmethod statsAndLoggingAggregator(jobStore, stop, config)
            The following function is used for collating stats/reporting log messages from the workers. Works inside of a thread, collates as long as the stop flag is not True.

            Parameters

                •   jobStore (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

                •   stop (threading.Event </usr/share/doc/python3/html/library/threading.html#threading.Event>)

                •   config (toil.common.Config <#toil.common.Config>)

            Return type
                None

        check()
            Check on the stats and logging aggregator. :raise RuntimeError: If the underlying thread has quit.

            Return type
                None

        shutdown()
            Finish up the stats/logging aggregation thread.

            Return type
                None

    toil.statsAndLogging.set_log_level(level, set_logger=None)
        Sets the root logger level to a given string level (like "INFO").

        Parameters

            •   level (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   set_logger (Optional[logging.Logger </usr/share/doc/python3/html/library/logging.html#logging.Logger>])

        Return type
            None

    toil.statsAndLogging.install_log_color(set_logger=None)
        Make logs colored.

        Parameters
            set_logger (Optional[logging.Logger </usr/share/doc/python3/html/library/logging.html#logging.Logger>])

        Return type
            None

    toil.statsAndLogging.add_logging_options(parser, default_level=None)
        Add logging options to set the global log level.

        Parameters

            •   default_level (Optional[int </usr/share/doc/python3/html/library/functions.html#int>]) -- A logging level, like logging.INFO, to use as the default.

            •   parser (argparse.ArgumentParser </usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser>)

        Return type
            None

    toil.statsAndLogging.configure_root_logger()
        Set up the root logger with handlers and formatting.

        Should be called before any entry point tries to log anything, to ensure consistent formatting.

        Return type
            None

    toil.statsAndLogging.log_to_file(log_file, log_rotation)

        Parameters

            •   log_file (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   log_rotation (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            None

    toil.statsAndLogging.set_logging_from_options(options)

        Parameters
            options (Union[toil.common.Config <#toil.common.Config>, argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>])

        Return type
            None

    toil.statsAndLogging.suppress_exotic_logging(local_logger)
        Attempts to suppress the loggers of all non-Toil packages by setting them to CRITICAL.

        For example: 'requests_oauthlib', 'google', 'boto', 'websocket', 'oauthlib', etc.

        This will only suppress loggers that have already been instantiated and can be seen in the environment, except for the list declared in "always_suppress".

        This is important because some packages, particularly boto3, are not always instantiated yet in the environment when this is run, and so we create the logger and set the level preemptively.

        Parameters
            local_logger (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            None

  toil.test

Base testing class for Toil.

  Submodules

  toil.test.batchSystems

  Submodules

  toil.test.batchSystems.batchSystemTest

  Attributes
logger |
numCores |
preemptible |
defaultRequirements |

  Classes
BatchSystemPluginTest | Class for testing batch system plugin functionality.
hidden | Hide abstract base class from unittest's test case loader
AbstractBatchSystemJobTest | An abstract base class for batch system tests that use a full Toil workflow rather
KubernetesBatchSystemTest | Tests against the Kubernetes batch system
KubernetesBatchSystemBenchTest | Kubernetes batch system unit tests that don't need to actually talk to a cluster.
AWSBatchBatchSystemTest | Tests against the AWS Batch batch system
MesosBatchSystemTest | Tests against the Mesos batch system
SingleMachineBatchSystemTest | Tests against the single-machine batch system
MaxCoresSingleMachineBatchSystemTest | This test ensures that single machine batch system doesn't exceed the configured number
Service | Abstract class used to define the interface to a service.
GridEngineBatchSystemTest | Tests against the GridEngine batch system
SlurmBatchSystemTest | Tests against the Slurm batch system
LSFBatchSystemTest | Tests against the LSF batch system
TorqueBatchSystemTest | Tests against the Torque batch system
HTCondorBatchSystemTest | Tests against the HTCondor batch system
TestSingleMachineBatchSystemJob | Tests Toil workflow against the SingleMachine batch system
TestMesosBatchSystemJob | Tests Toil workflow against the Mesos batch system

  Functions
write_temp_file(s, temp_dir) | Dump a string into a temp file and return its path.
parentJob(job, cmd) |
childJob(job, cmd) |
grandChildJob(job, cmd) |
greatGrandChild(cmd) |
measureConcurrency(filepath[, sleep_time]) | Run in parallel to determine the number of concurrent tasks.
count(delta, file_path) | Increments counter file and returns the max number of times the file
getCounters(path) |
resetCounters(path) |
get_omp_threads() |

  Module Contents

    toil.test.batchSystems.batchSystemTest.logger

    toil.test.batchSystems.batchSystemTest.numCores = 2

    toil.test.batchSystems.batchSystemTest.preemptible = False

    toil.test.batchSystems.batchSystemTest.defaultRequirements

    class toil.test.batchSystems.batchSystemTest.BatchSystemPluginTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Class for testing batch system plugin functionality.

        tearDown()
            Hook method for deconstructing the test fixture after testing it.

            Return type
                None

        test_add_batch_system_factory()

            Return type
                None

    class toil.test.batchSystems.batchSystemTest.hidden
        Hide abstract base class from unittest's test case loader

        <http://stackoverflow.com/questions/1323455/python-unit-test-with-base-and-sub-class#answer-25695512>

        class AbstractBatchSystemTest(methodName='runTest')
            Bases: toil.test.ToilTest <#toil.test.ToilTest>

            A base test case with generic tests that every batch system should pass.

            Cannot assume that the batch system actually executes commands on the local machine/filesystem.

            abstractmethod createBatchSystem()

                Return type
                    toil.batchSystems.abstractBatchSystem.AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>

            supportsWallTime()

                Return type
                    bool </usr/share/doc/python3/html/library/functions.html#bool>

            classmethod createConfig()
                Returns a dummy config for the batch system tests. We need a workflowID to be set up since we are running tests without setting up a jobstore. This is the class version to be used when an instance is not available.

                Return type
                    toil.common.Config <#toil.common.Config>

            classmethod setUpClass()
                Hook method for setting up class fixture before running tests in the class.

                Return type
                    None

            setUp()
                Hook method for setting up the test fixture before exercising it.

                Return type
                    None

            tearDown()
                Hook method for deconstructing the test fixture after testing it.

                Return type
                    None

            get_max_startup_seconds()
                Get the number of seconds this test ought to wait for the first job to run. Some batch systems may need time to scale up.

                Return type
                    int </usr/share/doc/python3/html/library/functions.html#int>

            test_available_cores()

                Return type
                    None

            test_run_jobs()

                Return type
                    None

            test_set_env()

                Return type
                    None

            test_set_job_env()
                Test the mechanism for setting per-job environment variables to batch system jobs.

                Return type
                    None

            testCheckResourceRequest()

                Return type
                    None

            testScalableBatchSystem()

                Return type
                    None

        class AbstractGridEngineBatchSystemTest(methodName='runTest')
            Bases: AbstractBatchSystemTest

            An abstract class to reduce redundancy between Grid Engine, Slurm, and other similar batch systems

    class toil.test.batchSystems.batchSystemTest.AbstractBatchSystemJobTest
        An abstract base class for batch system tests that use a full Toil workflow rather than using the batch system directly.

        cpuCount

        allocatedCores

        sleepTime = 30

        abstractmethod getBatchSystemName()

            Return type
                (str </usr/share/doc/python3/html/library/stdtypes.html#str>, AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>)

        getOptions(tempDir)
            Configures options for Toil workflow and makes job store. :param str tempDir: path to test directory :return: Toil options object

            Parameters
                tempDir (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>

        testJobConcurrency(tmp_path)
            Tests that the batch system is allocating core resources properly for concurrent tasks.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_omp_threads(tmp_path)
            Test if the OMP_NUM_THREADS env var is set correctly based on jobs.cores.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

    class toil.test.batchSystems.batchSystemTest.KubernetesBatchSystemTest(methodName='runTest')
        Bases: hidden

        Tests against the Kubernetes batch system

        supportsWallTime()

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        createBatchSystem()

            Return type
                toil.batchSystems.abstractBatchSystem.AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>

    class toil.test.batchSystems.batchSystemTest.KubernetesBatchSystemBenchTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Kubernetes batch system unit tests that don't need to actually talk to a cluster.

        test_preemptability_constraints()
            Make sure we generate the right preemptability constraints.

            Return type
                None

        test_label_constraints()
            Make sure we generate the right preemptability constraints.

            Return type
                None

    class toil.test.batchSystems.batchSystemTest.AWSBatchBatchSystemTest(methodName='runTest')
        Bases: hidden

        Tests against the AWS Batch batch system

        supportsWallTime()

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        createBatchSystem()

            Return type
                toil.batchSystems.abstractBatchSystem.AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>

        get_max_startup_seconds()
            Get the number of seconds this test ought to wait for the first job to run. Some batch systems may need time to scale up.

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

    class toil.test.batchSystems.batchSystemTest.MesosBatchSystemTest(methodName='runTest')
        Bases: hidden, toil.batchSystems.mesos.test.MesosTestSupport <#toil.batchSystems.mesos.test.MesosTestSupport>

        Tests against the Mesos batch system

        batchSystem: toil.batchSystems.mesos.batchSystem.MesosBatchSystem <#toil.batchSystems.mesos.batchSystem.MesosBatchSystem>

        classmethod createConfig()
            needs to set mesos_endpoint to localhost for testing since the default is now the private IP address

            Return type
                toil.common.Config <#toil.common.Config>

        supportsWallTime()

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        createBatchSystem()

            Return type
                toil.batchSystems.mesos.batchSystem.MesosBatchSystem <#toil.batchSystems.mesos.batchSystem.MesosBatchSystem>

        tearDown()
            Hook method for deconstructing the test fixture after testing it.

            Return type
                None

        testIgnoreNode()

            Return type
                None

    toil.test.batchSystems.batchSystemTest.write_temp_file(s, temp_dir)
        Dump a string into a temp file and return its path.

        Parameters

            •   s (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   temp_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    class toil.test.batchSystems.batchSystemTest.SingleMachineBatchSystemTest(methodName='runTest')
        Bases: hidden

        Tests against the single-machine batch system

        supportsWallTime()

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        createBatchSystem()

            Return type
                toil.batchSystems.abstractBatchSystem.AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>

        testProcessEscape(hide=False)
            Test to make sure that child processes and their descendants go away when the Toil workflow stops.

            If hide is true, will try and hide the child processes to make them hard to stop.

            Parameters
                hide (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        testHidingProcessEscape()
            Test to make sure that child processes and their descendants go away when the Toil workflow stops, even if the job process stops and leaves children.

            Return type
                None

    class toil.test.batchSystems.batchSystemTest.MaxCoresSingleMachineBatchSystemTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        This test ensures that single machine batch system doesn't exceed the configured number cores

        classmethod setUpClass()
            Hook method for setting up class fixture before running tests in the class.

            Return type
                None

        setUp()
            Hook method for setting up the test fixture before exercising it.

            Return type
                None

        tearDown()
            Hook method for deconstructing the test fixture after testing it.

            Return type
                None

        scriptCommand()

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        test()

            Return type
                None

        testServices()

            Return type
                None

    toil.test.batchSystems.batchSystemTest.parentJob(job, cmd)

        Parameters

            •   job (toil.job.Job <#toil.job.Job>)

            •   cmd (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            None

    toil.test.batchSystems.batchSystemTest.childJob(job, cmd)

        Parameters

            •   job (toil.job.Job <#toil.job.Job>)

            •   cmd (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            None

    toil.test.batchSystems.batchSystemTest.grandChildJob(job, cmd)

        Parameters

            •   job (toil.job.Job <#toil.job.Job>)

            •   cmd (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            None

    toil.test.batchSystems.batchSystemTest.greatGrandChild(cmd)

        Parameters
            cmd (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            None

    class toil.test.batchSystems.batchSystemTest.Service(cmd)
        Bases: toil.job.Job.Service <#toil.job.Job.Service>

        Abstract class used to define the interface to a service.

        Should be subclassed by the user to define services.

        Is not executed as a job; runs within a ServiceHostJob.

        Parameters
            cmd (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        cmd

        start(job)
            Start the service.

            Parameters
                job (toil.job.ServiceHostJob <#toil.job.ServiceHostJob>) -- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files.

            Returns
                An object describing how to access the service. The object must be pickleable and will be used by jobs to access the service (see toil.job.Job.addService() <#toil.job.Job.addService>).

            Return type
                None

        check()
            Checks the service is still running.

            Raises
                exceptions.RuntimeError -- If the service failed, this will cause the service job to be labeled failed.

            Returns
                True if the service is still running, else False. If False then the service job will be terminated, and considered a success. Important point: if the service job exits due to a failure, it should raise a RuntimeError, not return False!

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        stop(job)
            Stops the service. Function can block until complete.

            Parameters
                job (toil.job.ServiceHostJob <#toil.job.ServiceHostJob>) -- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files.

            Return type
                None

    class toil.test.batchSystems.batchSystemTest.GridEngineBatchSystemTest(methodName='runTest')
        Bases: hidden

        Tests against the GridEngine batch system

        createBatchSystem()

            Return type
                toil.batchSystems.abstractBatchSystem.AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>

        tearDown()
            Hook method for deconstructing the test fixture after testing it.

            Return type
                None

    class toil.test.batchSystems.batchSystemTest.SlurmBatchSystemTest(methodName='runTest')
        Bases: hidden

        Tests against the Slurm batch system

        createBatchSystem()

            Return type
                toil.batchSystems.abstractBatchSystem.AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>

        tearDown()
            Hook method for deconstructing the test fixture after testing it.

            Return type
                None

    class toil.test.batchSystems.batchSystemTest.LSFBatchSystemTest(methodName='runTest')
        Bases: hidden

        Tests against the LSF batch system

        createBatchSystem()

            Return type
                toil.batchSystems.abstractBatchSystem.AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>

    class toil.test.batchSystems.batchSystemTest.TorqueBatchSystemTest(methodName='runTest')
        Bases: hidden

        Tests against the Torque batch system

        createBatchSystem()

            Return type
                toil.batchSystems.abstractBatchSystem.AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>

        tearDown()
            Hook method for deconstructing the test fixture after testing it.

            Return type
                None

    class toil.test.batchSystems.batchSystemTest.HTCondorBatchSystemTest(methodName='runTest')
        Bases: hidden

        Tests against the HTCondor batch system

        createBatchSystem()

            Return type
                toil.batchSystems.abstractBatchSystem.AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>

        tearDown()
            Hook method for deconstructing the test fixture after testing it.

            Return type
                None

    class toil.test.batchSystems.batchSystemTest.TestSingleMachineBatchSystemJob
        Bases: AbstractBatchSystemJobTest

        Tests Toil workflow against the SingleMachine batch system

        getBatchSystemName()

            Return type
                (str </usr/share/doc/python3/html/library/stdtypes.html#str>, AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>)

        testConcurrencyWithDisk(tmp_path)
            Tests that the batch system is allocating disk resources properly

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testNestedResourcesDoNotBlock(tmp_path)
            Resources are requested in the order Memory > Cpu > Disk. Test that unavailability of cpus for one job that is scheduled does not block another job that can run.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

    class toil.test.batchSystems.batchSystemTest.TestMesosBatchSystemJob
        Bases: AbstractBatchSystemJobTest, toil.batchSystems.mesos.test.MesosTestSupport <#toil.batchSystems.mesos.test.MesosTestSupport>

        Tests Toil workflow against the Mesos batch system

        mesos_support()

            Return type
                collections.abc.Generator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator>[None]

        getOptions(tempDir)
            Configures options for Toil workflow and makes job store. :param str tempDir: path to test directory :return: Toil options object

            Parameters
                tempDir (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>

        getBatchSystemName()

            Return type
                (str </usr/share/doc/python3/html/library/stdtypes.html#str>, AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>)

    toil.test.batchSystems.batchSystemTest.measureConcurrency(filepath, sleep_time=10)
        Run in parallel to determine the number of concurrent tasks. This code was copied from toil.batchSystemTestMaxCoresSingleMachineBatchSystemTest :param filepath: path to counter file :param sleep_time: number of seconds to sleep before counting down :return: max concurrency value

        Parameters

            •   filepath (toil.lib.misc.StrPath)

            •   sleep_time (int </usr/share/doc/python3/html/library/functions.html#int>)

        Return type
            int </usr/share/doc/python3/html/library/functions.html#int>

    toil.test.batchSystems.batchSystemTest.count(delta, file_path)
        Increments counter file and returns the max number of times the file has been modified. Counter data must be in the form: concurrent tasks, max concurrent tasks (counter should be initialized to 0,0)

        Parameters

            •   delta (int </usr/share/doc/python3/html/library/functions.html#int>) -- increment value

            •   file_path (toil.lib.misc.StrPath) -- path to shared counter file

        Returns
            max concurrent tasks

        Return type
            int </usr/share/doc/python3/html/library/functions.html#int>

    toil.test.batchSystems.batchSystemTest.getCounters(path)

        Parameters
            path (toil.lib.misc.StrPath)

        Return type
            tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[int </usr/share/doc/python3/html/library/functions.html#int>, int </usr/share/doc/python3/html/library/functions.html#int>]

    toil.test.batchSystems.batchSystemTest.resetCounters(path)

        Parameters
            path (toil.lib.misc.StrPath)

        Return type
            None

    toil.test.batchSystems.batchSystemTest.get_omp_threads()

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

  toil.test.batchSystems.batch_system_plugin_test

  Attributes
logger |

  Classes
FakeBatchSystem | Adds cleanup support when the last running job leaves a node, for batch
BatchSystemPluginTest | Legacy common base class for Toil tests.

  Module Contents

    toil.test.batchSystems.batch_system_plugin_test.logger

    class toil.test.batchSystems.batch_system_plugin_test.FakeBatchSystem(config, maxCores, maxMemory, maxDisk)
        Bases: toil.batchSystems.cleanup_support.BatchSystemCleanupSupport <#toil.batchSystems.cleanup_support.BatchSystemCleanupSupport>

        Adds cleanup support when the last running job leaves a node, for batch systems that can't provide it using the backing scheduler.

        Parameters

            •   config (toil.common.Config <#toil.common.Config>)

            •   maxCores (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxMemory (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   maxDisk (int </usr/share/doc/python3/html/library/functions.html#int>)

        classmethod supportsAutoDeployment()
            Whether this batch system supports auto-deployment of the user script itself.

            If it does, the setUserScript() can be invoked to set the resource object representing the user script.

            Note to implementors: If your implementation returns True here, it should also override

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        issueBatchJob(command, job_desc, job_environment=None)
            Issues a job with the specified command to the batch system and returns a unique job ID number.

            Parameters

                •   command (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the command to execute somewhere to run the Toil worker process

                •   job_desc (toil.job.JobDescription <#toil.job.JobDescription>) -- the JobDescription for the job being run

                •   job_environment (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) -- a collection of job-specific environment variables to be set on the worker.

            Returns
                a unique job ID number that can be used to reference the newly issued job

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        killBatchJobs(jobIDs)
            Kills the given job IDs. After returning, the killed jobs will not appear in the results of getRunningBatchJobIDs. The killed job will not be returned from getUpdatedBatchJob.

            Parameters
                jobIDs (list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>]) -- list of IDs of jobs to kill

            Return type
                None

        getIssuedBatchJobIDs()
            Gets all currently issued jobs

            Returns
                A list of jobs (as job ID numbers) currently issued (may be running, or may be waiting to be run). Despite the result being a list, the ordering should not be depended upon.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>]

        getRunningBatchJobIDs()
            Gets a map of jobs as job ID numbers that are currently running (not just waiting) and how long they have been running, in seconds.

            Returns
                dictionary with currently running job ID number keys and how many seconds they have been running as the value

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>]

        getUpdatedBatchJob(maxWait)
            Returns information about job that has updated its status (i.e. ceased running, either successfully or with an error). Each such job will be returned exactly once.

            Does not return info for jobs killed by killBatchJobs, although they may cause None to be returned earlier than maxWait.

            Parameters
                maxWait (int </usr/share/doc/python3/html/library/functions.html#int>) -- the number of seconds to block, waiting for a result

            Returns
                If a result is available, returns UpdatedBatchJobInfo. Otherwise it returns None. wallTime is the number of seconds (a strictly positive float) in wall-clock time the job ran for, or None if this batch system does not support tracking wall time.

            Return type
                Optional[toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo <#toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo>]

        shutdown()
            Called at the completion of a toil invocation. Should cleanly terminate all worker threads.

            Return type
                None

        classmethod add_options(parser)
            If this batch system provides any command line options, add them to the given parser.

            Parameters
                parser (configargparse.ArgumentParser)

            Return type
                None

        classmethod setOptions(setOption)
            Process command line or configuration options relevant to this batch system.

            Parameters
                setOption (toil.batchSystems.options.OptionSetter <#toil.batchSystems.options.OptionSetter>) -- A function with signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) returning nothing, used to update run configuration as a side effect.

            Return type
                None

    class toil.test.batchSystems.batch_system_plugin_test.BatchSystemPluginTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        tearDown()
            Hook method for deconstructing the test fixture after testing it.

            Return type
                None

        test_batchsystem_plugin_installable()
            Test that installing a batch system plugin works. :return:

  toil.test.batchSystems.test_gridengine

  Classes
FakeBatchSystem | Class that implements a minimal Batch System, needed to create a Worker (see below).
GridEngineTest | Class for unit-testing GridEngineBatchSystem

  Functions
call_qstat_or_qacct(args, **_) |

  Module Contents

    class toil.test.batchSystems.test_gridengine.FakeBatchSystem
        Class that implements a minimal Batch System, needed to create a Worker (see below).

        config

        getWaitDuration()

        with_retries(operation, *args, **kwargs)
            The grid engine batch system needs a with_retries function when running the GridEngineThread, so fake one

    toil.test.batchSystems.test_gridengine.call_qstat_or_qacct(args, **_)

    class toil.test.batchSystems.test_gridengine.GridEngineTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Class for unit-testing GridEngineBatchSystem

        setUp()
            Hook method for setting up the test fixture before exercising it.

        test_coalesce_job_exit_codes_one_exists()

        test_coalesce_job_exit_codes_one_still_running()

        test_coalesce_job_exit_codes_many_all_exist()

  toil.test.batchSystems.test_lsf_helper

lsfHelper.py shouldn't need a batch system and so the unit tests here should aim to run on any system.

  Classes
LSFHelperTest | Legacy common base class for Toil tests.

  Module Contents

    class toil.test.batchSystems.test_lsf_helper.LSFHelperTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        test_parse_mem_and_cmd_from_output()

  toil.test.batchSystems.test_slurm

  Attributes
logger |
JOB_BASE_TIME |

  Classes
FakeBatchSystem | Class that implements a minimal Batch System, needed to create a Worker (see below).
SlurmTest | Class for unit-testing SlurmBatchSystem

  Functions
call_either(args, **_) | Pretend to call either sacct or scontrol as appropriate.
call_sacct(args, **_) | The arguments passed to call_command when executing sacct are something like:
call_scontrol(args, **_) | The arguments passed to call_command when executing scontrol are:
call_sacct_raises(*_) | Fake that the sacct command fails by raising a CalledProcessErrorStderr
call_sinfo(*_) | Simulate asking for partition info from Slurm

  Module Contents

    toil.test.batchSystems.test_slurm.logger

    toil.test.batchSystems.test_slurm.JOB_BASE_TIME

    toil.test.batchSystems.test_slurm.call_either(args, **_)
        Pretend to call either sacct or scontrol as appropriate.

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.test.batchSystems.test_slurm.call_sacct(args, **_)
        The arguments passed to call_command when executing sacct are something like: ['sacct', '-n', '-j', '<comma-separated list of job-ids>', '--format', 'JobIDRaw,State,ExitCode', '-P', '-S', '1970-01-01'] The multi-line output is something like:

                1234|COMPLETED|0:0
                1234.batch|COMPLETED|0:0
                1235|PENDING|0:0
                1236|FAILED|0:2
                1236.extern|COMPLETED|0:0

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.test.batchSystems.test_slurm.call_scontrol(args, **_)
        The arguments passed to call_command when executing scontrol are: ['scontrol', 'show', 'job'] or ['scontrol', 'show', 'job', '<job-id>']

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.test.batchSystems.test_slurm.call_sacct_raises(*_)
        Fake that the sacct command fails by raising a CalledProcessErrorStderr

    toil.test.batchSystems.test_slurm.call_sinfo(*_)
        Simulate asking for partition info from Slurm

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    class toil.test.batchSystems.test_slurm.FakeBatchSystem
        Bases: toil.batchSystems.abstractBatchSystem.BatchSystemSupport <#toil.batchSystems.abstractBatchSystem.BatchSystemSupport>

        Class that implements a minimal Batch System, needed to create a Worker (see below).

        start_time

        getWaitDuration()

    class toil.test.batchSystems.test_slurm.SlurmTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Class for unit-testing SlurmBatchSystem

        setUp()
            Hook method for setting up the test fixture before exercising it.

        test_getJobDetailsFromSacct_one_exists()

        test_getJobDetailsFromSacct_one_not_exists()

        test_getJobDetailsFromSacct_many_all_exist()

        test_getJobDetailsFromSacct_many_some_exist()

        test_getJobDetailsFromSacct_many_none_exist()

        test_getJobDetailsFromSacct_argument_list_too_big()

        test_getJobDetailsFromScontrol_one_exists()

        test_getJobDetailsFromScontrol_one_not_exists()
            Asking for the job details of a single job that scontrol doesn't know about should raise an exception.

        test_getJobDetailsFromScontrol_many_all_exist()

        test_getJobDetailsFromScontrol_many_some_exist()

        test_getJobDetailsFromScontrol_many_none_exist()

        test_getJobExitCode_job_exists()

        test_getJobExitCode_job_not_exists()

        test_getJobExitCode_sacct_raises_job_exists()
            This test forces the use of scontrol to get job information, by letting sacct raise an exception.

        test_getJobExitCode_sacct_raises_job_not_exists()
            This test forces the use of scontrol to get job information, by letting sacct raise an exception. Next, scontrol should also raise because it doesn't know the job.

        test_coalesce_job_exit_codes_one_exists()

        test_coalesce_job_exit_codes_one_not_exists()

        test_coalesce_job_exit_codes_many_all_exist()

        test_coalesce_job_exit_codes_mix_sacct_scontrol()

        test_coalesce_job_exit_codes_some_exists()

        test_coalesce_job_exit_codes_sacct_raises_job_exists()
            This test forces the use of scontrol to get job information, by letting sacct raise an exception.

        test_coalesce_job_exit_codes_sacct_raises_job_not_exists()
            This test forces the use of scontrol to get job information, by letting sacct raise an exception. Next, scontrol should also raise because it doesn't know the job.

        test_PartitionSet_get_partition()

        test_PartitionSet_default_gpu_partition()

        test_prepareSbatch_partition()

        test_prepareSbatch_time()

        test_prepareSbatch_export()

        test_option_detector()

        test_any_option_detector()

  toil.test.cactus

  Submodules

  toil.test.cactus.test_cactus_integration

  Classes
CactusIntegrationTest | Run the Cactus Integration test on a Kubernetes AWS cluster

  Module Contents

    class toil.test.cactus.test_cactus_integration.CactusIntegrationTest(methodName)
        Bases: toil.test.provisioners.clusterTest.AbstractClusterTest <#toil.test.provisioners.clusterTest.AbstractClusterTest>

        Run the Cactus Integration test on a Kubernetes AWS cluster

        clusterName = 'cactus-test-'

        leaderNodeType = 't2.medium'

        clusterType = 'kubernetes'

        setUp()
            Set up for the test. Must be overridden to call this method and set self.jobStore.

        test_cactus_integration()

  toil.test.conftest

  Functions
rootpath(request) | Records the rootpath at the class level, for use on a unittest.TestCase.

  Module Contents

    toil.test.conftest.rootpath(request)
        Records the rootpath at the class level, for use on a unittest.TestCase.

        Parameters
            request (pytest.FixtureRequest)

        Return type
            None

  toil.test.cwl

  Submodules

  toil.test.cwl.conftest

  Attributes
logger |
collect_ignore |

  Functions
pytest_cwl_execute_test(config, processfile, jobfile) | Use Toil to execute CWL tests (equivalent to running toil-cwl-runner).

  Module Contents

    toil.test.cwl.conftest.logger

    toil.test.cwl.conftest.collect_ignore = ['spec']

    toil.test.cwl.conftest.pytest_cwl_execute_test(config, processfile, jobfile)
        Use Toil to execute CWL tests (equivalent to running toil-cwl-runner).

        Parameters

            •   config (cwltest.utils.CWLTestConfig)

            •   processfile (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   jobfile (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            Tuple[int </usr/share/doc/python3/html/library/functions.html#int>, Optional[Dict[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]]]

  toil.test.cwl.cwlTest

  Attributes
pkg_root |
log |
CONFORMANCE_TEST_TIMEOUT |
TesterFuncType |

  Classes
TestCWLWorkflow | CWL tests included in Toil that don't involve the whole CWL conformance
TestCWLv10Conformance | Run the CWL 1.0 conformance tests in various environments.
TestCWLv11Conformance | Run the CWL 1.1 conformance tests in various environments.
TestCWLv12Conformance | Run the CWL 1.2 conformance tests in various environments.
ImportWorkersMessageHandler | Detect whether any WorkerImportJob jobs ran during a workflow.

  Functions
run_conformance_tests(workDir, yml[, runner, caching, ...]) | Run the CWL conformance tests.
cwl_v1_0_spec(tmp_path) |
cwl_v1_1_spec(tmp_path) |
cwl_v1_2_spec(tmp_path) |
test_workflow_echo_string_scatter_stderr_log_dir(tmp_path) |
test_log_dir_echo_no_output(tmp_path) |
test_log_dir_echo_stderr(tmp_path) |
test_filename_conflict_resolution(tmp_path) |
test_filename_conflict_resolution_3_or_more(tmp_path) |
test_filename_conflict_detection(tmp_path) | Make sure we don't just stage files over each other when using a container.
test_filename_conflict_detection_at_root(tmp_path) | Make sure we don't just stage files over each other.
test_pick_value_with_one_null_value(caplog, tmp_path) | Make sure toil-cwl-runner does not false log a warning when pickValue is
test_workflow_echo_string(tmp_path) |
test_workflow_echo_string_scatter_capture_stdout(tmp_path) |
test_visit_top_cwl_class() |
test_visit_cwl_class_and_reduce() |
test_trim_mounts_op_nonredundant() | Make sure we don't remove all non-duplicate listings
test_trim_mounts_op_redundant() | Make sure we remove all duplicate listings
test_trim_mounts_op_partially_redundant() | Make sure we remove only the redundant listings in the CWL object and leave nonredundant listings intact
test_trim_mounts_op_mixed_urls_and_paths() | Ensure we remove redundant listings in certain edge cases
test_trim_mounts_op_decodable_paths() |
test_trim_mounts_op_multiple_encodings() |
test_download_structure(tmp_path) | Make sure that download_structure makes the right calls to what it thinks is the file store.
test_import_on_workers() |
test_missing_tmpdir_and_tmp_outdir(tmp_path) | tmpdir_prefix and tmp_outdir_prefix do not need to exist prior to running the workflow

  Module Contents

    toil.test.cwl.cwlTest.pkg_root = b'.'

    toil.test.cwl.cwlTest.log

    toil.test.cwl.cwlTest.CONFORMANCE_TEST_TIMEOUT = 10000

    toil.test.cwl.cwlTest.run_conformance_tests(workDir, yml, runner=None, caching=False, batchSystem=None, selected_tests=None, selected_tags=None, skipped_tests=None, extra_args=None, must_support_all_features=False, junit_file=None)
        Run the CWL conformance tests.

        Parameters

            •   workDir (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Directory to run tests in.

            •   yml (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- CWL test list YML to run tests from.

            •   runner (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- If set, use this cwl runner instead of the default toil-cwl-runner.

            •   caching (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If True, use Toil file store caching.

            •   batchSystem (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- If set, use this batch system instead of the default single_machine.

            •   selected_tests (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- If set, use this description of test numbers to run (comma-separated numbers or ranges)

            •   selected_tags (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- As an alternative to selected_tests, run tests with the given tags.

            •   skipped_tests (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Comma-separated string labels of tests to skip.

            •   extra_args (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) -- Provide these extra arguments to runner for each test.

            •   must_support_all_features (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If set, fail if some CWL optional features are unsupported.

            •   junit_file (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- JUnit XML file to write test info to.

        Return type
            None

    toil.test.cwl.cwlTest.TesterFuncType

    class toil.test.cwl.cwlTest.TestCWLWorkflow
        CWL tests included in Toil that don't involve the whole CWL conformance test suite. Tests Toil-specific functions like URL types supported for inputs.

        test_cwl_cmdline_input()
            Test that running a CWL workflow with inputs specified on the command line passes.

            Return type
                None

        revsort(cwl_filename, tester_fn, out_dir)

            Parameters

                •   cwl_filename (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   tester_fn (TesterFuncType)

                •   out_dir (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        revsort_no_checksum(cwl_filename, tester_fn, out_dir)

            Parameters

                •   cwl_filename (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   tester_fn (TesterFuncType)

                •   out_dir (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        download(inputs, tester_fn, out_dir)

            Parameters

                •   inputs (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   tester_fn (TesterFuncType)

                •   out_dir (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        load_contents(inputs, tester_fn, out_dir)

            Parameters

                •   inputs (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   tester_fn (TesterFuncType)

                •   out_dir (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        download_directory(inputs, tester_fn, out_dir)

            Parameters

                •   inputs (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   tester_fn (TesterFuncType)

                •   out_dir (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        download_subdirectory(inputs, tester_fn, out_dir)

            Parameters

                •   inputs (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   tester_fn (TesterFuncType)

                •   out_dir (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_mpi(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_s3_as_secondary_file(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_run_revsort(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_run_revsort_nochecksum(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_run_revsort_no_container(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_run_revsort2(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_run_revsort_debug_worker(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_run_colon_output(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_run_dockstore_trs(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_glob_dir_bypass_file_store(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_required_input_condition_protection(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_slurm_node_memory(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_download_s3(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_download_http(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_download_https(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_download_https_reference(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_download_file(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_download_directory_s3(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_download_directory_s3_reference(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_download_directory_file(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_download_subdirectory_s3(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_download_subdirectory_file(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_load_contents_s3(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_load_contents_http(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_load_contents_https(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_load_contents_file(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_bioconda(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_default_args(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_biocontainers(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_cuda(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_restart(tmp_path)
            Enable restarts with toil-cwl-runner -- run failing test, re-run correct test. Only implemented for single machine.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_caching(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_streamable(tmp_path, extra_args=None)
            Test that a file with 'streamable'=True is a named pipe. This is a CWL1.2 feature.

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   extra_args (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

            Return type
                None

        test_streamable_reference(tmp_path)
            Test that a streamable file is a stream even when passed around by URI.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_preemptible(tmp_path)
            Tests that the <http://arvados.org/cwl#UsePreemptible> extension is supported.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_preemptible_expression(tmp_path)
            Tests that the <http://arvados.org/cwl#UsePreemptible> extension is validated.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_missing_import(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_optional_secondary_files_exists(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_optional_secondary_files_missing(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

    toil.test.cwl.cwlTest.cwl_v1_0_spec(tmp_path)

        Parameters
            tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            collections.abc.Generator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator>[pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>]

    class toil.test.cwl.cwlTest.TestCWLv10Conformance
        Run the CWL 1.0 conformance tests in various environments.

        test_run_conformance_with_caching(cwl_v1_0_spec)

            Parameters
                cwl_v1_0_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_run_conformance(cwl_v1_0_spec, batchSystem=None, caching=False, selected_tests=None, skipped_tests=None, extra_args=None)

            Parameters

                •   cwl_v1_0_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   batchSystem (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   caching (bool </usr/share/doc/python3/html/library/functions.html#bool>)

                •   selected_tests (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   skipped_tests (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   extra_args (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

            Return type
                None

        test_lsf_cwl_conformance(cwl_v1_0_spec, caching=False)

            Parameters

                •   cwl_v1_0_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   caching (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        test_slurm_cwl_conformance(cwl_v1_0_spec, caching=False)

            Parameters

                •   cwl_v1_0_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   caching (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        test_torque_cwl_conformance(cwl_v1_0_spec, caching=False)

            Parameters

                •   cwl_v1_0_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   caching (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        test_gridengine_cwl_conformance(cwl_v1_0_spec, caching=False)

            Parameters

                •   cwl_v1_0_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   caching (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        test_mesos_cwl_conformance(cwl_v1_0_spec, caching=False)

            Parameters

                •   cwl_v1_0_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   caching (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        test_kubernetes_cwl_conformance(cwl_v1_0_spec, caching=False)

            Parameters

                •   cwl_v1_0_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   caching (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        test_lsf_cwl_conformance_with_caching(cwl_v1_0_spec)

            Parameters
                cwl_v1_0_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_slurm_cwl_conformance_with_caching(cwl_v1_0_spec)

            Parameters
                cwl_v1_0_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_torque_cwl_conformance_with_caching(cwl_v1_0_spec)

            Parameters
                cwl_v1_0_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_gridengine_cwl_conformance_with_caching(cwl_v1_0_spec)

            Parameters
                cwl_v1_0_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_mesos_cwl_conformance_with_caching(cwl_v1_0_spec)

            Parameters
                cwl_v1_0_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_kubernetes_cwl_conformance_with_caching(cwl_v1_0_spec)

            Parameters
                cwl_v1_0_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

    toil.test.cwl.cwlTest.cwl_v1_1_spec(tmp_path)

        Parameters
            tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            collections.abc.Generator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator>[pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>]

    class toil.test.cwl.cwlTest.TestCWLv11Conformance
        Run the CWL 1.1 conformance tests in various environments.

        test_run_conformance(cwl_v1_1_spec, caching=False, batchSystem=None, skipped_tests=None, extra_args=None)

            Parameters

                •   cwl_v1_1_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   caching (bool </usr/share/doc/python3/html/library/functions.html#bool>)

                •   batchSystem (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   skipped_tests (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   extra_args (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

            Return type
                None

        test_run_conformance_with_caching(cwl_v1_1_spec)

            Parameters
                cwl_v1_1_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_kubernetes_cwl_conformance(cwl_v1_1_spec, caching=False)

            Parameters

                •   cwl_v1_1_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   caching (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        test_kubernetes_cwl_conformance_with_caching(cwl_v1_1_spec)

            Parameters
                cwl_v1_1_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

    toil.test.cwl.cwlTest.cwl_v1_2_spec(tmp_path)

        Parameters
            tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            collections.abc.Generator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator>[pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>]

    class toil.test.cwl.cwlTest.TestCWLv12Conformance
        Run the CWL 1.2 conformance tests in various environments.

        test_run_conformance(cwl_v1_2_spec, runner=None, caching=False, batchSystem=None, selected_tests=None, skipped_tests=None, extra_args=None, must_support_all_features=False, junit_file=None)

            Parameters

                •   cwl_v1_2_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   runner (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   caching (bool </usr/share/doc/python3/html/library/functions.html#bool>)

                •   batchSystem (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   selected_tests (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   skipped_tests (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   extra_args (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

                •   must_support_all_features (bool </usr/share/doc/python3/html/library/functions.html#bool>)

                •   junit_file (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

        test_run_conformance_with_caching(cwl_v1_2_spec)

            Parameters
                cwl_v1_2_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_run_conformance_with_task_caching(cwl_v1_2_spec, tmp_path)

            Parameters

                •   cwl_v1_2_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_run_conformance_with_in_place_update(cwl_v1_2_spec)
            Make sure that with --bypass-file-store we properly support in place update on a single node, and that this doesn't break any other features.

            Parameters
                cwl_v1_2_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_kubernetes_cwl_conformance(cwl_v1_2_spec, caching=False, junit_file=None)

            Parameters

                •   cwl_v1_2_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   caching (bool </usr/share/doc/python3/html/library/functions.html#bool>)

                •   junit_file (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

        test_kubernetes_cwl_conformance_with_caching(cwl_v1_2_spec)

            Parameters
                cwl_v1_2_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_wes_server_cwl_conformance(cwl_v1_2_spec)
            Run the CWL conformance tests via WES. TOIL_WES_ENDPOINT must be specified. If the WES server requires authentication, set TOIL_WES_USER and TOIL_WES_PASSWORD.

            To run manually:

            TOIL_WES_ENDPOINT=http://localhost:8080 TOIL_WES_USER=test TOIL_WES_PASSWORD=password python -m pytest src/toil/test/cwl/cwlTest.py::TestCWLv12Conformance::test_wes_server_cwl_conformance -vv --log-level INFO --log-cli-level INFO

            Parameters
                cwl_v1_2_spec (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

    toil.test.cwl.cwlTest.test_workflow_echo_string_scatter_stderr_log_dir(tmp_path)

        Parameters
            tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            None

    toil.test.cwl.cwlTest.test_log_dir_echo_no_output(tmp_path)

        Parameters
            tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            None

    toil.test.cwl.cwlTest.test_log_dir_echo_stderr(tmp_path)

        Parameters
            tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            None

    toil.test.cwl.cwlTest.test_filename_conflict_resolution(tmp_path)

        Parameters
            tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            None

    toil.test.cwl.cwlTest.test_filename_conflict_resolution_3_or_more(tmp_path)

        Parameters
            tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            None

    toil.test.cwl.cwlTest.test_filename_conflict_detection(tmp_path)
        Make sure we don't just stage files over each other when using a container.

        Parameters
            tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            None

    toil.test.cwl.cwlTest.test_filename_conflict_detection_at_root(tmp_path)
        Make sure we don't just stage files over each other.

        Specifically, when using a container and the files are at the root of the work dir.

        Parameters
            tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            None

    toil.test.cwl.cwlTest.test_pick_value_with_one_null_value(caplog, tmp_path)
        Make sure toil-cwl-runner does not false log a warning when pickValue is used but outputSource only contains one null value. See: #3991.

        Parameters

            •   caplog (pytest.LogCaptureFixture)

            •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            None

    toil.test.cwl.cwlTest.test_workflow_echo_string(tmp_path)

        Parameters
            tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            None

    toil.test.cwl.cwlTest.test_workflow_echo_string_scatter_capture_stdout(tmp_path)

        Parameters
            tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            None

    toil.test.cwl.cwlTest.test_visit_top_cwl_class()

        Return type
            None

    toil.test.cwl.cwlTest.test_visit_cwl_class_and_reduce()

        Return type
            None

    toil.test.cwl.cwlTest.test_trim_mounts_op_nonredundant()
        Make sure we don't remove all non-duplicate listings

        Return type
            None

    toil.test.cwl.cwlTest.test_trim_mounts_op_redundant()
        Make sure we remove all duplicate listings

        Return type
            None

    toil.test.cwl.cwlTest.test_trim_mounts_op_partially_redundant()
        Make sure we remove only the redundant listings in the CWL object and leave nonredundant listings intact

        Return type
            None

    toil.test.cwl.cwlTest.test_trim_mounts_op_mixed_urls_and_paths()
        Ensure we remove redundant listings in certain edge cases

        Return type
            None

    toil.test.cwl.cwlTest.test_trim_mounts_op_decodable_paths()

        Return type
            None

    toil.test.cwl.cwlTest.test_trim_mounts_op_multiple_encodings()

        Return type
            None

    toil.test.cwl.cwlTest.test_download_structure(tmp_path)
        Make sure that download_structure makes the right calls to what it thinks is the file store.

        Parameters
            tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            None

    toil.test.cwl.cwlTest.test_import_on_workers()

        Return type
            None

    toil.test.cwl.cwlTest.test_missing_tmpdir_and_tmp_outdir(tmp_path)
        tmpdir_prefix and tmp_outdir_prefix do not need to exist prior to running the workflow

        Parameters
            tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            None

    class toil.test.cwl.cwlTest.ImportWorkersMessageHandler
        Bases: _stream_handler

        Detect whether any WorkerImportJob jobs ran during a workflow.

        detected = False

        emit(record)
            Emit a record.

            If a formatter is specified, it is used to format the record. The record is then written to the stream with a trailing newline. If exception information is present, it is formatted using traceback.print_exception and appended to the stream. If the stream has an 'encoding' attribute, it is used to determine how to do the output to the stream.

            Parameters
                record (logging.LogRecord </usr/share/doc/python3/html/library/logging.html#logging.LogRecord>)

            Return type
                None

  toil.test.docs

  Submodules

  toil.test.docs.scriptsTest

  Attributes
pkg_root |

  Classes
ToilDocumentationTest | Tests for scripts in the toil tutorials.

  Module Contents

    toil.test.docs.scriptsTest.pkg_root = b'.'

    class toil.test.docs.scriptsTest.ToilDocumentationTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Tests for scripts in the toil tutorials.

        classmethod setUpClass()
            Hook method for setting up class fixture before running tests in the class.

        tearDown()
            Hook method for deconstructing the test fixture after testing it.

            Return type
                None

        checkExitCode(script, extra_args=[])

            Parameters
                extra_args (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        checkExpectedOut(script, expectedOutput)

        checkExpectedPattern(script, expectedPattern)

        testStats()

        testDynamic()

        testEncapsulation()

        testEncapsulation2()

        testHelloworld()

        testInvokeworkflow()

        testInvokeworkflow2()

        testJobFunctions()

        testManaging()

        testManaging2()

        testMultiplejobs()

        testMultiplejobs2()

        testMultiplejobs3()

        testPromises2()

        testQuickstart()

        testRequirements()

        testArguments()

        testDocker()

        testPromises()

        testServices()

        testStaging()

  toil.test.jobStores

  Submodules

  toil.test.jobStores.jobStoreTest

  Attributes
logger |

  Classes
AbstractJobStoreTest | Hide abstract base class from unittest's test case loader
AbstractEncryptedJobStoreTest |
FileJobStoreTest | Legacy common base class for Toil tests.
GoogleJobStoreTest | Legacy common base class for Toil tests.
AWSJobStoreTest | Legacy common base class for Toil tests.
InvalidAWSJobStoreTest | Legacy common base class for Toil tests.
EncryptedAWSJobStoreTest | Legacy common base class for Toil tests.
StubHttpRequestHandler | Simple HTTP request handler with GET and HEAD commands.

  Functions
google_retry(x) |
tearDownModule() |

  Module Contents

    toil.test.jobStores.jobStoreTest.google_retry(x)

    toil.test.jobStores.jobStoreTest.logger

    toil.test.jobStores.jobStoreTest.tearDownModule()

    class toil.test.jobStores.jobStoreTest.AbstractJobStoreTest
        Hide abstract base class from unittest's test case loader

        <http://stackoverflow.com/questions/1323455/python-unit-test-with-base-and-sub-class#answer-25695512>

        class Test(methodName='runTest')
            Bases: toil.test.ToilTest <#toil.test.ToilTest>

            Legacy common base class for Toil tests.

            New tests should be made in the 'pytest' style and not use this class.

            When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

            classmethod setUpClass()
                Hook method for setting up class fixture before running tests in the class.

            setUp()
                Hook method for setting up the test fixture before exercising it.

            tearDown()
                Hook method for deconstructing the test fixture after testing it.

            testInitialState()
                Ensure proper handling of nonexistent files.

            testJobCreation()
                Test creation of a job.

                Does the job exist in the jobstore it is supposed to be in? Are its attributes what is expected?

            testConfigEquality()
                Ensure that the command line configurations are successfully loaded and stored.

                In setUp() self.jobstore1 is created and initialized. In this test, after creating newJobStore, .resume() will look for a previously instantiated job store and load its config options. This is expected to be equal but not the same object.

            testJobLoadEquality()
                Tests that a job created via one JobStore instance can be loaded from another.

            testChildLoadingEquality()
                Test that loading a child job operates as expected.

            testPersistantFilesToDelete()
                Make sure that updating a job persists filesToDelete.

                The following demonstrates the job update pattern, where files to be deleted atomically with a job update are referenced in "filesToDelete" array, which is persisted to disk first. If things go wrong during the update, this list of files to delete is used to ensure that the updated job and the files are never both visible at the same time.

            testUpdateBehavior()
                Tests the proper behavior during updating jobs.

            testJobDeletions()
                Tests the consequences of deleting jobs.

            testSharedFiles()
                Tests the sharing of files.

            testReadWriteSharedFilesTextMode()
                Checks if text mode is compatible for shared file streams.

            testReadWriteFileStreamTextMode()
                Checks if text mode is compatible for file streams.

            testPerJobFiles()
                Tests the behavior of files on jobs.

            testStatsAndLogging()
                Tests behavior of reading and writing stats and logging.

            testWriteLogFiles()
                Test writing log files.

            testBatchCreate()
                Test creation of many jobs.

            testGrowingAndShrinkingJob()
                Make sure jobs update correctly if they grow/shrink.

            externalStoreCache

            classmethod cleanUpExternalStores()

            mpTestPartSize = 5242880

            classmethod makeImportExportTests()

            testImportHttpFile()
                Test importing a file over HTTP.

            testImportFtpFile()
                Test importing a file over FTP

            testFileDeletion()
                Intended to cover the batch deletion of items in the AWSJobStore, but it doesn't hurt running it on the other job stores.

            testMultipartUploads()
                This test is meant to cover multi-part uploads in the AWSJobStore but it doesn't hurt running it against the other job stores as well.

            testZeroLengthFiles()
                Test reading and writing of empty files.

            testLargeFile()
                Test the reading and writing of large files.

            fetch_url(url)
                Fetch the given URL. Throw an error if it cannot be fetched in a reasonable number of attempts.

                Parameters
                    url (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                Return type
                    None

            assertUrl(url)

            testCleanCache()

            testPartialReadFromStream()
                Test whether readFileStream will deadlock on a partial read.

            testDestructionOfCorruptedJobStore()

            testDestructionIdempotence()

            testEmptyFileStoreIDIsReadable()
                Simply creates an empty fileStoreID and attempts to read from it.

    class toil.test.jobStores.jobStoreTest.AbstractEncryptedJobStoreTest

        class Test(methodName='runTest')
            Bases: AbstractJobStoreTest

            A test of job stores that use encryption

            setUp()
                Hook method for setting up the test fixture before exercising it.

            tearDown()
                Hook method for deconstructing the test fixture after testing it.

            testEncrypted()
                Create an encrypted file. Read it in encrypted mode then try with encryption off to ensure that it fails.

    class toil.test.jobStores.jobStoreTest.FileJobStoreTest(methodName='runTest')
        Bases: AbstractJobStoreTest

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        testPreserveFileName()
            Check that the fileID ends with the given file name.

        test_jobstore_init_preserves_symlink_path()
            Test that if we provide a fileJobStore with a symlink to a directory, it doesn't de-reference it.

        test_jobstore_does_not_leak_symlinks()
            Test that if we link imports into the FileJobStore, we can't get hardlinks to symlinks.

        test_file_link_imports()
            Test that imported files are symlinked when when expected

        test_symlink_read_control()
            Test that files are read by symlink when expected

    class toil.test.jobStores.jobStoreTest.GoogleJobStoreTest(methodName='runTest')
        Bases: AbstractJobStoreTest

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        projectID

        headers

    class toil.test.jobStores.jobStoreTest.AWSJobStoreTest(methodName='runTest')
        Bases: AbstractJobStoreTest

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        testOverlargeJob()

        testMultiThreadImportFile()
            Tests that importFile is thread-safe.

            Return type
                None

    class toil.test.jobStores.jobStoreTest.InvalidAWSJobStoreTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        testInvalidJobStoreName()

    class toil.test.jobStores.jobStoreTest.EncryptedAWSJobStoreTest(methodName='runTest')
        Bases: AWSJobStoreTest, AbstractEncryptedJobStoreTest

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

    class toil.test.jobStores.jobStoreTest.StubHttpRequestHandler(*args, directory=None, **kwargs)
        Bases: http.server.SimpleHTTPRequestHandler </usr/share/doc/python3/html/library/http.server.html#http.server.SimpleHTTPRequestHandler>

        Simple HTTP request handler with GET and HEAD commands.

        This serves files from the current directory and any of its subdirectories. The MIME type for files is determined by calling the .guess_type() method.

        The GET and HEAD requests are identical except that the HEAD request omits the actual contents of the file.

        fileContents = 'A good programmer looks both ways before crossing a one-way street'

        do_GET()
            Serve a GET request.

  toil.test.lib

  Submodules

  toil.test.lib.aws

  Submodules

  toil.test.lib.aws.test_iam

  Attributes
logger |

  Classes
IAMTest | Check that given permissions and associated functions perform correctly

  Module Contents

    toil.test.lib.aws.test_iam.logger

    class toil.test.lib.aws.test_iam.IAMTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Check that given permissions and associated functions perform correctly

        test_permissions_iam()

            Return type
                None

        test_negative_permissions_iam()

            Return type
                None

        test_wildcard_handling()

            Return type
                None

        test_get_policy_permissions()

            Return type
                None

        test_create_delete_iam_role()

            Return type
                None

  toil.test.lib.aws.test_s3

  Attributes
logger |

  Classes
S3Test | Confirm the workarounds for us-east-1.

  Module Contents

    toil.test.lib.aws.test_s3.logger

    class toil.test.lib.aws.test_s3.S3Test(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Confirm the workarounds for us-east-1.

        s3_resource: mypy_boto3_s3.S3ServiceResource | None </usr/share/doc/python3/html/library/constants.html#None>

        classmethod setUpClass()
            Hook method for setting up class fixture before running tests in the class.

            Return type
                None

        test_create_bucket()
            Test bucket creation for us-east-1.

            Return type
                None

        test_get_bucket_location_public_bucket()
            Test getting bucket location for a bucket we don't own.

            Return type
                None

  toil.test.lib.aws.test_utils

  Attributes
logger |

  Classes
TagGenerationTest | Test for tag generation from environment variables

  Module Contents

    toil.test.lib.aws.test_utils.logger

    class toil.test.lib.aws.test_utils.TagGenerationTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Test for tag generation from environment variables

        test_build_tag()

            Return type
                None

        test_empty_aws_tags()

            Return type
                None

        test_incorrect_json_object()

            Return type
                None

        test_incorrect_json_emoji()

            Return type
                None

        test_build_tag_with_tags()

            Return type
                None

  toil.test.lib.dockerTest

  Attributes
logger |

  Classes
TestDocker | Tests dockerCall and ensures no containers are left around.

  Module Contents

    toil.test.lib.dockerTest.logger

    class toil.test.lib.dockerTest.TestDocker
        Tests dockerCall and ensures no containers are left around.

        dockerTestLogLevel = 'INFO'

        testDockerClean(tmp_path, caching=False, detached=True, rm=True, deferParam=None)
            Run the test container that creates a file in the work dir, and sleeps for 5 minutes. Ensure that the calling job gets SIGKILLed after a minute, leaving behind the spooky/ghost/zombie container. Ensure that the container is killed on batch system shutdown (through the deferParam mechanism).

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   caching (bool </usr/share/doc/python3/html/library/functions.html#bool>)

                •   detached (bool </usr/share/doc/python3/html/library/functions.html#bool>)

                •   rm (bool </usr/share/doc/python3/html/library/functions.html#bool>)

                •   deferParam (Optional[int </usr/share/doc/python3/html/library/functions.html#int>])

            Return type
                None

        testDockerClean_CRx_FORGO(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerClean_CRx_STOP(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerClean_CRx_RM(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerClean_CRx_None(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerClean_CxD_FORGO(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerClean_CxD_STOP(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerClean_CxD_RM(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerClean_CxD_None(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerClean_Cxx_FORGO(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerClean_Cxx_STOP(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerClean_Cxx_RM(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerClean_Cxx_None(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerClean_xRx_FORGO(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerClean_xRx_STOP(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerClean_xRx_RM(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerClean_xRx_None(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerClean_xxD_FORGO(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerClean_xxD_STOP(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerClean_xxD_RM(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerClean_xxD_None(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerClean_xxx_FORGO(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerClean_xxx_STOP(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerClean_xxx_RM(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerClean_xxx_None(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerPipeChain(tmp_path, caching=False)
            Test for piping API for dockerCall(). Using this API (activated when list of argument lists is given as parameters), commands a piped together into a chain. ex: parameters=[ ['printf', 'x\n y\n'], ['wc', '-l'] ] should execute: printf 'x\n y\n' | wc -l

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   caching (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        testDockerPipeChainErrorDetection(tmp_path, caching=False)
            By default, executing cmd1 | cmd2 | ... | cmdN, will only return an error if cmdN fails. This can lead to all manor of errors being silently missed. This tests to make sure that the piping API for dockerCall() throws an exception if non-last commands in the chain fail.

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   caching (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        testNonCachingDockerChain(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testNonCachingDockerChainErrorDetection(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerLogs(tmp_path, stream=False, demux=False)
            Test for the different log outputs when deatch=False.

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   stream (bool </usr/share/doc/python3/html/library/functions.html#bool>)

                •   demux (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        testDockerLogs_Stream(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerLogs_Demux(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDockerLogs_Demux_Stream(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

  toil.test.lib.test_conversions

  Attributes
logger |

  Classes
ConversionTest | Legacy common base class for Toil tests.

  Module Contents

    toil.test.lib.test_conversions.logger

    class toil.test.lib.test_conversions.ConversionTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        test_convert()

        test_human2bytes()

        test_hms_duration_to_seconds()

  toil.test.lib.test_ec2

  Attributes
logger |

  Classes
FlatcarFeedTest | Test accessing the Flatcar AMI release feed, independent of the AWS API
AMITest | Legacy common base class for Toil tests.

  Module Contents

    toil.test.lib.test_ec2.logger

    class toil.test.lib.test_ec2.FlatcarFeedTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Test accessing the Flatcar AMI release feed, independent of the AWS API

        test_parse_archive_feed()
            Make sure we can get a Flatcar release from the Internet Archive.

        test_parse_beta_feed()
            Make sure we can get a Flatcar release from the beta channel.

        test_parse_stable_feed()
            Make sure we can get a Flatcar release from the stable channel.

    class toil.test.lib.test_ec2.AMITest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        classmethod setUpClass()
            Hook method for setting up class fixture before running tests in the class.

        test_fetch_flatcar()

        test_fetch_arm_flatcar()
            Test flatcar AMI finder architecture parameter.

  toil.test.lib.test_history

  Attributes
logger |

  Classes
TestHistory | Tests for Toil history tracking.

  Module Contents

    toil.test.lib.test_history.logger

    class toil.test.lib.test_history.TestHistory
        Tests for Toil history tracking.

        Each test gets its own history database.

        private_history_manager(tmp_path, monkeypatch)

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   monkeypatch (pytest.MonkeyPatch)

            Return type
                collections.abc.Generator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator>[None]

        make_fake_workflow(workflow_id)

            Parameters
                workflow_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        test_history_submittable_detection()
            Make sure that a submittable workflow shows up as such before submission and doesn't afterward.

            Return type
                None

        test_history_deletion()

            Return type
                None

        test_history_size_limit()
            Make sure the database size can be controlled.

            Return type
                None

  toil.test.lib.test_misc

  Attributes
logger |

  Classes
UserNameAvailableTest | Make sure we can get user names when they are available.
UserNameUnvailableTest | Make sure we can get something for a user name when user names are not
UserNameVeryBrokenTest | Make sure we can get something for a user name when user name fetching is

  Module Contents

    toil.test.lib.test_misc.logger

    class toil.test.lib.test_misc.UserNameAvailableTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Make sure we can get user names when they are available.

        test_get_user_name()

    class toil.test.lib.test_misc.UserNameUnvailableTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Make sure we can get something for a user name when user names are not available.

        setUp()
            Hook method for setting up the test fixture before exercising it.

        tearDown()
            Hook method for deconstructing the test fixture after testing it.

        test_get_user_name()

    class toil.test.lib.test_misc.UserNameVeryBrokenTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Make sure we can get something for a user name when user name fetching is broken in ways we did not expect.

        setUp()
            Hook method for setting up the test fixture before exercising it.

        tearDown()
            Hook method for deconstructing the test fixture after testing it.

        test_get_user_name()

  toil.test.lib.test_trs

  Attributes
logger |

  Classes
DockstoreLookupTest | Make sure we can look up workflows on Dockstore.

  Module Contents

    toil.test.lib.test_trs.logger

    class toil.test.lib.test_trs.DockstoreLookupTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Make sure we can look up workflows on Dockstore.

        read_result(url_or_path)
            Read a file or URL.

            Binary mode to allow testing for binary file support.

            This lets us test that we have the right workflow contents and not care how we are being shown them.

            Parameters
                url_or_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                IO[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>]

        test_lookup_from_page_url()

            Return type
                None

        test_lookup_from_trs_with_version()

            Return type
                None

        test_lookup_from_trs_no_version()

            Return type
                None

        test_get()

            Return type
                None

        test_get_from_trs_cached()

            Return type
                None

        test_lookup_from_trs_nonexistent_workflow()

            Return type
                None

        test_lookup_from_trs_nonexistent_workflow_bad_format()

            Return type
                None

        test_lookup_from_trs_nonexistent_version()

            Return type
                None

        test_get_nonexistent_workflow()

            Return type
                None

        test_get_nonexistent_version()

            Return type
                None

        test_get_nonexistent_workflow_bad_format()

            Return type
                None

  toil.test.lib.test_url

  Attributes
logger |

  Classes
TestURLAccess | Test URLAccess class handling read, list,

  Module Contents

    toil.test.lib.test_url.logger

    class toil.test.lib.test_url.TestURLAccess
        Test URLAccess class handling read, list, and checking the size/existence of resources at given URL

        test_get_url_access(httpserver)

            Parameters
                httpserver (pytest_httpserver.HTTPServer)

            Return type
                None

        test_get_size()

            Return type
                None

        test_get_is_directory()

            Return type
                None

        test_list_url()

            Return type
                None

        test_read_from_url()

            Return type
                None

        test_open_url()

            Return type
                None

  toil.test.lib.url_plugin_test

  Attributes
logger |

  Classes
FakeURLPlugin | Widget for accessing external storage (URLs).
TestURLAccess | Legacy common base class for Toil tests.

  Module Contents

    toil.test.lib.url_plugin_test.logger

    class toil.test.lib.url_plugin_test.FakeURLPlugin
        Bases: toil.lib.url.URLAccess <#toil.lib.url.URLAccess>

        Widget for accessing external storage (URLs).

    class toil.test.lib.url_plugin_test.TestURLAccess(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        setUp()
            Hook method for setting up the test fixture before exercising it.

            Return type
                None

        tearDown()
            Hook method for deconstructing the test fixture after testing it.

            Return type
                None

        test_url_exists()

            Return type
                None

        test_get_size()

            Return type
                None

        test_get_is_directory()

            Return type
                None

        test_list_url()

            Return type
                None

        test_read_from_url()

            Return type
                None

        test_open_url()

            Return type
                None

  toil.test.mesos

  Submodules

  toil.test.mesos.MesosDataStructuresTest

  Classes
DataStructuresTest | Legacy common base class for Toil tests.

  Module Contents

    class toil.test.mesos.MesosDataStructuresTest.DataStructuresTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        testJobQueue(testJobs=1000)
            The mesos JobQueue sorts MesosShape objects by requirement and this test ensures that that sorting is what is expected: non-preemptible jobs groups first, with priority given to large jobs.

  toil.test.mesos.helloWorld

A simple user script for Toil

  Attributes
childMessage |
parentMessage |

  Functions
hello_world(job) |
hello_world_child(job, hw) |
main() |

  Module Contents

    toil.test.mesos.helloWorld.childMessage = 'The child job is now running!'

    toil.test.mesos.helloWorld.parentMessage = 'The parent job is now running!'

    toil.test.mesos.helloWorld.hello_world(job)

    toil.test.mesos.helloWorld.hello_world_child(job, hw)

    toil.test.mesos.helloWorld.main()

  toil.test.mesos.stress

  Classes
LongTestJob | Class represents a unit of work in toil.
LongTestFollowOn | Class represents a unit of work in toil.
HelloWorldJob | Class represents a unit of work in toil.
HelloWorldFollowOn | Class represents a unit of work in toil.

  Functions
touchFile(fileStore) |
main(numJobs) |

  Module Contents

    toil.test.mesos.stress.touchFile(fileStore)

    class toil.test.mesos.stress.LongTestJob(numJobs)
        Bases: toil.job.Job <#toil.job.Job>

        Class represents a unit of work in toil.

        numJobs

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>.

    class toil.test.mesos.stress.LongTestFollowOn
        Bases: toil.job.Job <#toil.job.Job>

        Class represents a unit of work in toil.

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>.

    class toil.test.mesos.stress.HelloWorldJob(i)
        Bases: toil.job.Job <#toil.job.Job>

        Class represents a unit of work in toil.

        i

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>.

    class toil.test.mesos.stress.HelloWorldFollowOn(i)
        Bases: toil.job.Job <#toil.job.Job>

        Class represents a unit of work in toil.

        i

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>.

    toil.test.mesos.stress.main(numJobs)

  toil.test.options

  Submodules

  toil.test.options.options

  Classes
OptionsTest | Class to test functionality of all Toil options

  Module Contents

    class toil.test.options.options.OptionsTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Class to test functionality of all Toil options

        test_default_caching_slurm()
            Test to ensure that caching will be set to false when running on Slurm :return:

        test_caching_option_priority()
            Test to ensure that the --caching option takes priority over the default_caching() return value :return:

  toil.test.provisioners

  Submodules

  toil.test.provisioners.aws

  Submodules

  toil.test.provisioners.aws.awsProvisionerTest

  Attributes
log |

  Classes
TestAWSProvisionerBenchTest | Tests for the AWS provisioner that don't actually provision instances.
AbstractAWSAutoscaleTest | Legacy common base class for Toil tests.
AWSAutoscaleTest | Legacy common base class for Toil tests.
AWSStaticAutoscaleTest | Runs the tests on a statically provisioned cluster with autoscaling enabled.
AWSManagedAutoscaleTest | Runs the tests on a self-scaling Kubernetes cluster.
AWSAutoscaleTestMultipleNodeTypes | Legacy common base class for Toil tests.
AWSRestartTest | This test insures autoscaling works on a restarted Toil run.
PreemptibleDeficitCompensationTest | Legacy common base class for Toil tests.

  Functions
aws_zone() | Supply an appropriate AWS zone to work in to tests that need one.
aws_region(aws_zone) | Supply an appropriate AWS region to work in to tests that need one.
ec2_client(aws_region) | Supply an AWS EC2 client tests that need one.

  Module Contents

    toil.test.provisioners.aws.awsProvisionerTest.log

    toil.test.provisioners.aws.awsProvisionerTest.aws_zone()
        Supply an appropriate AWS zone to work in to tests that need one.

    toil.test.provisioners.aws.awsProvisionerTest.aws_region(aws_zone)
        Supply an appropriate AWS region to work in to tests that need one.

    toil.test.provisioners.aws.awsProvisionerTest.ec2_client(aws_region)
        Supply an AWS EC2 client tests that need one.

    class toil.test.provisioners.aws.awsProvisionerTest.TestAWSProvisionerBenchTest
        Tests for the AWS provisioner that don't actually provision instances.

        test_AMI_finding()

        test_read_write_global_files(aws_zone)
            Make sure the _write_file_to_cloud() and _read_file_from_cloud() functions of the AWS provisioner work as intended.

        test_get_spot_history(ec2_client)
            Make sure that we can download spot price history from AWS.

            Return type
                None

        test_get_aws_zone_from_spot_market(ec2_client)
            Make sure that we can process spot price history to pick a zone.

            Return type
                None

    class toil.test.provisioners.aws.awsProvisionerTest.AbstractAWSAutoscaleTest(methodName)
        Bases: toil.test.provisioners.clusterTest.AbstractClusterTest <#toil.test.provisioners.clusterTest.AbstractClusterTest>

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        instanceTypes = ['m5a.large']

        clusterName = 'aws-provisioner-test-'

        numWorkers = ['2']

        numSamples = 2

        spotBid = 0.15

        scriptDir = '/tmp/t'

        venvDir = '/tmp/venv'

        dataDir = '/tmp'

        scriptName = 'test_script.py'

        script()
            Return the full path to the user script on the leader.

        data(filename)
            Return the full path to the data file with the given name on the leader.

        rsyncUtil(src, dest)

            Parameters

                •   src (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   dest (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        getRootVolID()

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        putScript(content)
            Helper method for _getScript to inject a script file at the configured script path, from text.

            Parameters
                content (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

    class toil.test.provisioners.aws.awsProvisionerTest.AWSAutoscaleTest(name)
        Bases: AbstractAWSAutoscaleTest

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        clusterName = 'provisioner-test-'

        requestedLeaderStorage = 80

        scriptName = 'sort.py'

        setUp()
            Set up for the test. Must be overridden to call this method and set self.jobStore.

        launchCluster()

        getRootVolID()
            Adds in test to check that EBS volume is build with adequate size. Otherwise is functionally equivalent to parent. :return: volumeID

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        testAutoScale()

        testSpotAutoScale()

        testSpotAutoScaleBalancingTypes()

    class toil.test.provisioners.aws.awsProvisionerTest.AWSStaticAutoscaleTest(name)
        Bases: AWSAutoscaleTest

        Runs the tests on a statically provisioned cluster with autoscaling enabled.

        requestedNodeStorage = 20

        launchCluster()

    class toil.test.provisioners.aws.awsProvisionerTest.AWSManagedAutoscaleTest(name)
        Bases: AWSAutoscaleTest

        Runs the tests on a self-scaling Kubernetes cluster.

        requestedNodeStorage = 20

        launchCluster()

    class toil.test.provisioners.aws.awsProvisionerTest.AWSAutoscaleTestMultipleNodeTypes(name)
        Bases: AbstractAWSAutoscaleTest

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        clusterName = 'provisioner-test-'

        setUp()
            Set up for the test. Must be overridden to call this method and set self.jobStore.

        testAutoScale()

    class toil.test.provisioners.aws.awsProvisionerTest.AWSRestartTest(name)
        Bases: AbstractAWSAutoscaleTest

        This test insures autoscaling works on a restarted Toil run.

        clusterName = 'restart-test-'

        scriptName = 'restartScript.py'

        setUp()
            Set up for the test. Must be overridden to call this method and set self.jobStore.

        testAutoScaledCluster()

    class toil.test.provisioners.aws.awsProvisionerTest.PreemptibleDeficitCompensationTest(name)
        Bases: AbstractAWSAutoscaleTest

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        clusterName = 'deficit-test-'

        scriptName = 'userScript.py'

        setUp()
            Set up for the test. Must be overridden to call this method and set self.jobStore.

        test()

  toil.test.provisioners.clusterScalerTest

  Attributes
logger |
c4_8xlarge_preemptible |
c4_8xlarge |
r3_8xlarge |
r5_2xlarge |
r5_4xlarge |
t2_micro |

  Classes
BinPackingTest | Legacy common base class for Toil tests.
ClusterScalerTest | Legacy common base class for Toil tests.
ScalerThreadTest | Legacy common base class for Toil tests.
MockBatchSystemAndProvisioner | Mimics a leader, job batcher, provisioner and scalable batch system.

  Module Contents

    toil.test.provisioners.clusterScalerTest.logger

    toil.test.provisioners.clusterScalerTest.c4_8xlarge_preemptible

    toil.test.provisioners.clusterScalerTest.c4_8xlarge

    toil.test.provisioners.clusterScalerTest.r3_8xlarge

    toil.test.provisioners.clusterScalerTest.r5_2xlarge

    toil.test.provisioners.clusterScalerTest.r5_4xlarge

    toil.test.provisioners.clusterScalerTest.t2_micro

    class toil.test.provisioners.clusterScalerTest.BinPackingTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        setUp()
            Hook method for setting up the test fixture before exercising it.

        testPackingOneShape()
            Pack one shape and check that the resulting reservations look sane.

        testSorting()
            Test that sorting is correct: preemptible, then memory, then cores, then disk, then wallTime.

        testAddingInitialNode()
            Pack one shape when no nodes are available and confirm that we fit one node properly.

        testLowTargetTime()
            Test that a low targetTime (0) parallelizes jobs aggressively (1000 queued jobs require 1000 nodes).

            Ideally, low targetTime means: Start quickly and maximize parallelization after the cpu/disk/mem have been packed.

            Disk/cpu/mem packing is prioritized first, so we set job resource reqs so that each t2.micro (1 cpu/8G disk/1G RAM) can only run one job at a time with its resources.

            Each job is parametrized to take 300 seconds, so (the minimum of) 1 of them should fit into each node's 0 second window, so we expect 1000 nodes.

        testHighTargetTime()
            Test that a high targetTime (3600 seconds) maximizes packing within the targetTime.

            Ideally, high targetTime means: Maximize packing within the targetTime after the cpu/disk/mem have been packed.

            Disk/cpu/mem packing is prioritized first, so we set job resource reqs so that each t2.micro (1 cpu/8G disk/1G RAM) can only run one job at a time with its resources.

            Each job is parametrized to take 300 seconds, so 12 of them should fit into each node's 3600 second window. 1000/12 = 83.33, so we expect 84 nodes.

        testZeroResourceJobs()
            Test that jobs requiring zero cpu/disk/mem pack first, regardless of targetTime.

            Disk/cpu/mem packing is prioritized first, so we set job resource reqs so that each t2.micro (1 cpu/8G disk/1G RAM) can run a seemingly infinite number of jobs with its resources.

            Since all jobs should pack cpu/disk/mem-wise on a t2.micro, we expect only one t2.micro to be provisioned. If we raise this, as in testLowTargetTime, it will launch 1000 t2.micros.

        testLongRunningJobs()
            Test that jobs with long run times (especially service jobs) are aggressively parallelized.

            This is important, because services are one case where the degree of parallelization really, really matters. If you have multiple services, they may all need to be running simultaneously before any real work can be done.

            Despite setting globalTargetTime=3600, this should launch 1000 t2.micros because each job's estimated runtime (30000 seconds) extends well beyond 3600 seconds.

        run1000JobsOnMicros(jobCores, jobMem, jobDisk, jobTime, globalTargetTime)
            Test packing 1000 jobs on t2.micros. Depending on the targetTime and resources, these should pack differently.

        testPathologicalCase()
            Test a pathological case where only one node can be requested to fit months' worth of jobs.

            If the reservation is extended to fit a long job, and the bin-packer naively searches through all the reservation slices to find the first slice that fits, it will happily assign the first slot that fits the job, even if that slot occurs days in the future.

        testJobTooLargeForAllNodes()
            If a job is too large for all node types, the scaler should print a warning, but definitely not crash.

    class toil.test.provisioners.clusterScalerTest.ClusterScalerTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        setUp()
            Hook method for setting up the test fixture before exercising it.

        testRounding()
            Test to make sure the ClusterScaler's rounding rounds properly.

        testMaxNodes()
            Set the scaler to be very aggressive, give it a ton of jobs, and make sure it doesn't go over maxNodes.

        testMinNodes()
            Without any jobs queued, the scaler should still estimate "minNodes" nodes.

        testPreemptibleDeficitResponse()
            When a preemptible deficit was detected by a previous run of the loop, the scaler should add non-preemptible nodes to compensate in proportion to preemptibleCompensation.

        testPreemptibleDeficitIsSet()
            Make sure that updateClusterSize sets the preemptible deficit if it can't launch preemptible nodes properly. That way, the deficit can be communicated to the next run of estimateNodeCount.

        testNoLaunchingIfDeltaAlreadyMet()
            Check that the scaler doesn't try to launch "0" more instances if the delta was able to be met by unignoring nodes.

        testBetaInertia()

        test_overhead_accounting_large()
            If a node has a certain raw memory or disk capacity, that won't all be available when it actually comes up; some disk and memory will be used by the OS, and the backing scheduler (Mesos, Kubernetes, etc.).

            Make sure this overhead is accounted for for large nodes.

        test_overhead_accounting_small()
            If a node has a certain raw memory or disk capacity, that won't all be available when it actually comes up; some disk and memory will be used by the OS, and the backing scheduler (Mesos, Kubernetes, etc.).

            Make sure this overhead is accounted for for small nodes.

        test_overhead_accounting_observed()
            If a node has a certain raw memory or disk capacity, that won't all be available when it actually comes up; some disk and memory will be used by the OS, and the backing scheduler (Mesos, Kubernetes, etc.).

            Make sure this overhead is accounted for so that real-world observed failures cannot happen again.

    class toil.test.provisioners.clusterScalerTest.ScalerThreadTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        testClusterScaling()
            Test scaling for a batch of non-preemptible jobs and no preemptible jobs (makes debugging easier).

        testClusterScalingMultipleNodeTypes()

        testClusterScalingWithPreemptibleJobs()
            Test scaling simultaneously for a batch of preemptible and non-preemptible jobs.

    class toil.test.provisioners.clusterScalerTest.MockBatchSystemAndProvisioner(config, secondsPerJob)
        Bases: toil.batchSystems.abstractBatchSystem.AbstractScalableBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractScalableBatchSystem>, toil.provisioners.abstractProvisioner.AbstractProvisioner <#toil.provisioners.abstractProvisioner.AbstractProvisioner>

        Mimics a leader, job batcher, provisioner and scalable batch system.

        config

        secondsPerJob

        provisioner

        batchSystem

        jobQueue

        updatedJobsQueue

        jobBatchSystemIDToIssuedJob

        totalJobs = 0

        totalWorkerTime = 0.0

        toilMetrics = None

        nodesToWorker

        workers

        maxWorkers

        running = False

        leaderThread

        toilState

        start()

        shutDown()

        nodeInUse(nodeIP)
            Can be used to determine if a worker node is running any tasks. If the node is doesn't exist, this function should simply return False.

            Parameters
                nodeIP -- The worker nodes private IP address

            Returns
                True if the worker node has been issued any tasks, else False

        ignoreNode(nodeAddress)
            Stop sending jobs to this node. Used in autoscaling when the autoscaler is ready to terminate a node, but jobs are still running. This allows the node to be terminated after the current jobs have finished.

            Parameters
                nodeAddress -- IP address of node to ignore.

        unignoreNode(nodeAddress)
            Stop ignoring this address, presumably after a node with this address has been terminated. This allows for the possibility of a new node having the same address as a terminated one.

        supportedClusterTypes()
            Get all the cluster types that this provisioner implementation supports.

        createClusterSettings()
            Initialize class for a new cluster, to be deployed, when running outside the cloud.

        readClusterSettings()
            Initialize class from an existing cluster. This method assumes that the instance we are running on is the leader.

            Implementations must call _setLeaderWorkerAuthentication().

        setAutoscaledNodeTypes(node_types)
            Set node types, shapes and spot bids for Toil-managed autoscaling. :param nodeTypes: A list of node types, as parsed with parse_node_types.

            Parameters
                node_types (list </usr/share/doc/python3/html/library/stdtypes.html#list>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[set </usr/share/doc/python3/html/library/stdtypes.html#set>[toil.provisioners.abstractProvisioner.Shape <#toil.provisioners.abstractProvisioner.Shape>], Optional[float </usr/share/doc/python3/html/library/functions.html#float>]]])

        getProvisionedWorkers(instance_type=None, preemptible=None)
            Returns a list of Node objects, each representing a worker node in the cluster

            Parameters
                preemptible -- If True only return preemptible nodes else return non-preemptible nodes

            Returns
                list of Node

        terminateNodes(nodes)
            Terminate the nodes represented by given Node objects

            Parameters
                nodes -- list of Node objects

        remainingBillingInterval(node)

        addJob(jobShape, preemptible=False)
            Add a job to the job queue

        getNumberOfJobsIssued(preemptible=None)

        getJobs()

        getNodes(preemptible=False, timeout=600)
            Returns a dictionary mapping node identifiers of preemptible or non-preemptible nodes to NodeInfo objects, one for each node.

            Parameters

                •   preemptible (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>]) -- If True (False) only (non-)preemptible nodes will be returned. If None, all nodes will be returned.

                •   timeout (int </usr/share/doc/python3/html/library/functions.html#int>)

        addNodes(nodeTypes, numNodes, preemptible)
            Used to add worker nodes to the cluster

            Parameters

                •   numNodes -- The number of nodes to add

                •   preemptible -- whether or not the nodes will be preemptible

                •   spotBid -- The bid for preemptible nodes if applicable (this can be set in config, also).

                •   nodeTypes (set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Returns
                number of nodes successfully added

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        getNodeShape(nodeType, preemptible=False)
            The shape of a preemptible or non-preemptible node managed by this provisioner. The node shape defines key properties of a machine, such as its number of cores or the time between billing intervals.

            Parameters
                instance_type (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Instance type name to return the shape of.

        getWorkersInCluster(nodeShape)

        launchCluster(leaderNodeType, keyName, userTags=None, vpcSubnet=None, leaderStorage=50, nodeStorage=50, botoPath=None, **kwargs)
            Initialize a cluster and create a leader node.

            Implementations must call _setLeaderWorkerAuthentication() with the leader so that workers can be launched.

            Parameters

                •   leaderNodeType -- The leader instance.

                •   leaderStorage -- The amount of disk to allocate to the leader in gigabytes.

                •   owner -- Tag identifying the owner of the instances.

        destroyCluster()
            Terminates all nodes in the specified cluster and cleans up all resources associated with the cluster. :param clusterName: identifier of the cluster to terminate.

            Return type
                None

        getLeader()

            Returns
                The leader node.

        getNumberOfNodes(nodeType=None, preemptible=None)

  toil.test.provisioners.clusterTest

  Attributes
log |

  Classes
AbstractClusterTest | Legacy common base class for Toil tests.
CWLOnARMTest | Run the CWL 1.2 conformance tests on ARM specifically.

  Module Contents

    toil.test.provisioners.clusterTest.log

    class toil.test.provisioners.clusterTest.AbstractClusterTest(methodName)
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        Parameters
            methodName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        keyName

        clusterName = 'aws-provisioner-test-Instance of uuid.UUID'

        leaderNodeType = 't2.medium'

        clusterType = 'mesos'

        zone

        region

        aws

        venvDir = '/tmp/venv'

        python()
            Return the full path to the venv Python on the leader.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        pip()
            Return the full path to the venv pip on the leader.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        destroyCluster()
            Destroy the cluster we built, if it exists.

            Succeeds if the cluster does not currently exist.

            Return type
                None

        setUp()
            Set up for the test. Must be overridden to call this method and set self.jobStore.

            Return type
                None

        tearDown()
            Hook method for deconstructing the test fixture after testing it.

            Return type
                None

        sshUtil(command)
            Run the given command on the cluster. Raise subprocess.CalledProcessError if it fails.

            Parameters
                command (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

        rsync_util(from_file, to_file)
            Transfer a file to/from the cluster.

            The cluster-side path should have a ':' in front of it.

            Parameters

                •   from_file (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   to_file (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        createClusterUtil(args=None)

            Parameters
                args (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

            Return type
                None

        launchCluster()

            Return type
                None

    class toil.test.provisioners.clusterTest.CWLOnARMTest(methodName)
        Bases: AbstractClusterTest

        Run the CWL 1.2 conformance tests on ARM specifically.

        Parameters
            methodName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        clusterName = 'cwl-test-Instance of uuid.UUID'

        leaderNodeType = 't4g.2xlarge'

        clusterType = 'kubernetes'

        cwl_test_dir = '/tmp/toil/cwlTests'

        setUp()
            Set up for the test. Must be overridden to call this method and set self.jobStore.

            Return type
                None

        test_cwl_on_arm()

            Return type
                None

  toil.test.provisioners.gceProvisionerTest

  Attributes
log |

  Classes
AbstractGCEAutoscaleTest | Legacy common base class for Toil tests.
GCEAutoscaleTest | Legacy common base class for Toil tests.
GCEStaticAutoscaleTest | Runs the tests on a statically provisioned cluster with autoscaling enabled.
GCEAutoscaleTestMultipleNodeTypes | Legacy common base class for Toil tests.
GCERestartTest | This test insures autoscaling works on a restarted Toil run

  Module Contents

    toil.test.provisioners.gceProvisionerTest.log

    class toil.test.provisioners.gceProvisionerTest.AbstractGCEAutoscaleTest(methodName)
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        projectID

        sshUtil(command)

        rsyncUtil(src, dest)

        destroyClusterUtil()

        createClusterUtil(args=None)

        cleanJobStoreUtil()

        keyName

        botoDir

        googleZone = 'us-west1-a'

        leaderInstanceType = 'n1-standard-1'

        instanceTypes = ['n1-standard-2']

        numWorkers = ['2']

        numSamples = 2

        spotBid = 0.15

        setUp()
            Hook method for setting up the test fixture before exercising it.

        tearDown()
            Hook method for deconstructing the test fixture after testing it.

        launchCluster()

    class toil.test.provisioners.gceProvisionerTest.GCEAutoscaleTest(name)
        Bases: AbstractGCEAutoscaleTest

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        clusterName = 'provisioner-test-'

        requestedLeaderStorage = 80

        setUp()
            Hook method for setting up the test fixture before exercising it.

        launchCluster()

        testAutoScale()

        testSpotAutoScale()

    class toil.test.provisioners.gceProvisionerTest.GCEStaticAutoscaleTest(name)
        Bases: GCEAutoscaleTest

        Runs the tests on a statically provisioned cluster with autoscaling enabled.

        requestedNodeStorage = 20

        launchCluster()

    class toil.test.provisioners.gceProvisionerTest.GCEAutoscaleTestMultipleNodeTypes(name)
        Bases: AbstractGCEAutoscaleTest

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        clusterName = 'provisioner-test-'

        setUp()
            Hook method for setting up the test fixture before exercising it.

        testAutoScale()

    class toil.test.provisioners.gceProvisionerTest.GCERestartTest(name)
        Bases: AbstractGCEAutoscaleTest

        This test insures autoscaling works on a restarted Toil run

        clusterName = 'restart-test-'

        setUp()
            Hook method for setting up the test fixture before exercising it.

        testAutoScaledCluster()

  toil.test.provisioners.provisionerTest

  Attributes
log |

  Classes
ProvisionerTest | Legacy common base class for Toil tests.

  Module Contents

    toil.test.provisioners.provisionerTest.log

    class toil.test.provisioners.provisionerTest.ProvisionerTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        test_node_type_parsing()

            Return type
                None

  toil.test.provisioners.restartScript

  Attributes
parser |

  Functions
f0(job) |

  Module Contents

    toil.test.provisioners.restartScript.f0(job)

    toil.test.provisioners.restartScript.parser

  toil.test.server

  Submodules

  toil.test.server.serverTest

  Attributes
logger |

  Classes
ToilServerUtilsTest | Tests for the utility functions used by the Toil server.
hidden |
FileStateStoreTest | Test file-based state storage.
FileStateStoreURLTest | Test file-based state storage using URLs instead of local paths.
BucketUsingTest | Base class for tests that need a bucket.
AWSStateStoreTest | Test AWS-based state storage.
TrueDict | Dict that is truthy even when empty.
AbstractToilWESServerTest | Class for server tests that provides a self.app in testing mode.
ToilWESServerBenchTest | Tests for Toil's Workflow Execution Service API that don't run workflows.
ToilWESServerWorkflowTest | Tests of the WES server running workflows.
ToilWESServerCeleryWorkflowTest | End-to-end workflow-running tests against Celery.
ToilWESServerCeleryS3StateWorkflowTest | Test the server with Celery and state stored in S3.

  Module Contents

    toil.test.server.serverTest.logger

    class toil.test.server.serverTest.ToilServerUtilsTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Tests for the utility functions used by the Toil server.

        test_workflow_canceling_recovery()
            Make sure that a workflow in CANCELING state will be recovered to a terminal state eventually even if the workflow runner Celery task goes away without flipping the state.

    class toil.test.server.serverTest.hidden

        class AbstractStateStoreTest(methodName='runTest')
            Bases: toil.test.ToilTest <#toil.test.ToilTest>

            Basic tests for state stores.

            abstractmethod get_state_store()
                Make a state store to test, on a single fixed URL.

                Return type
                    AbstractStateStore <#toil.server.utils.AbstractStateStore>

            test_state_store()
                Make sure that the state store under test can store and load keys.

                Return type
                    None

    class toil.test.server.serverTest.FileStateStoreTest(methodName='runTest')
        Bases: hidden

        Test file-based state storage.

        setUp()
            Hook method for setting up the test fixture before exercising it.

            Return type
                None

        get_state_store()
            Make a state store to test, on a single fixed local path.

            Return type
                AbstractStateStore <#toil.server.utils.AbstractStateStore>

    class toil.test.server.serverTest.FileStateStoreURLTest(methodName='runTest')
        Bases: hidden

        Test file-based state storage using URLs instead of local paths.

        setUp()
            Hook method for setting up the test fixture before exercising it.

            Return type
                None

        get_state_store()
            Make a state store to test, on a single fixed URL.

            Return type
                AbstractStateStore <#toil.server.utils.AbstractStateStore>

    class toil.test.server.serverTest.BucketUsingTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Base class for tests that need a bucket.

        region: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        s3_resource: mypy_boto3_s3.S3ServiceResource | None </usr/share/doc/python3/html/library/constants.html#None>

        bucket: mypy_boto3_s3.service_resource.Bucket | None </usr/share/doc/python3/html/library/constants.html#None>

        bucket_name: str </usr/share/doc/python3/html/library/stdtypes.html#str> | None </usr/share/doc/python3/html/library/constants.html#None>

        classmethod setUpClass()
            Set up the class with a single pre-existing AWS bucket for all tests.

            Return type
                None

        classmethod tearDownClass()
            Hook method for deconstructing the class fixture after running all tests in the class.

            Return type
                None

    class toil.test.server.serverTest.AWSStateStoreTest(methodName='runTest')
        Bases: hidden, BucketUsingTest

        Test AWS-based state storage.

        bucket_path = 'prefix/of/keys'

        get_state_store()
            Make a state store to test, on a single fixed URL.

            Return type
                AbstractStateStore <#toil.server.utils.AbstractStateStore>

        test_state_store_paths()
            Make sure that the S3 state store puts things in the right places.

            We don't really care about the exact internal structure, but we do care about actually being under the path we are supposed to use.

            Return type
                None

    class toil.test.server.serverTest.TrueDict
        Bases: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>

        Dict that is truthy even when empty.

        Used as a workaround to set httpx post request encoding as recommended in <<https://github.com/encode/httpx/discussions/2399#discussioncomment-3814186>>.

        __bool__()
            Always say the object is truthy.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

    class toil.test.server.serverTest.AbstractToilWESServerTest(*args, **kwargs)
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Class for server tests that provides a self.app in testing mode.

        setUp()
            Hook method for setting up the test fixture before exercising it.

            Return type
                None

        tearDown()
            Hook method for deconstructing the test fixture after testing it.

            Return type
                None

    class toil.test.server.serverTest.ToilWESServerBenchTest(*args, **kwargs)
        Bases: AbstractToilWESServerTest

        Tests for Toil's Workflow Execution Service API that don't run workflows.

        test_home()
            Test the homepage endpoint.

            Return type
                None

        test_health()
            Test the health check endpoint.

            Return type
                None

        test_get_service_info()
            Test the GET /service-info endpoint.

            Return type
                None

    class toil.test.server.serverTest.ToilWESServerWorkflowTest(*args, **kwargs)
        Bases: AbstractToilWESServerTest

        Tests of the WES server running workflows.

        run_zip_workflow(zip_path, include_message=True, include_params=True)
            We have several zip file tests; this submits a zip file and makes sure it ran OK.

            If include_message is set to False, don't send a "message" argument in workflow_params. If include_params is also set to False, don't send workflow_params at all.

            Parameters

                •   zip_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   include_message (bool </usr/share/doc/python3/html/library/functions.html#bool>)

                •   include_params (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        test_run_workflow_relative_url_no_attachments_fails()
            Test run example CWL workflow from relative workflow URL but with no attachments.

            Return type
                None

        test_run_workflow_relative_url()
            Test run example CWL workflow from relative workflow URL.

            Return type
                None

        test_run_workflow_https_url()
            Test run example CWL workflow from the Internet.

            Return type
                None

        test_run_workflow_single_file_zip()
            Test run example CWL workflow from single-file ZIP.

            Return type
                None

        test_run_workflow_multi_file_zip()
            Test run example CWL workflow from multi-file ZIP.

            Return type
                None

        test_run_workflow_manifest_zip()
            Test run example CWL workflow from ZIP with manifest.

            Return type
                None

        test_run_workflow_inputs_zip()
            Test run example CWL workflow from ZIP without manifest but with inputs.

            Return type
                None

        test_run_workflow_manifest_and_inputs_zip()
            Test run example CWL workflow from ZIP with manifest and inputs.

            Return type
                None

        test_run_workflow_no_params_zip()
            Test run example CWL workflow from ZIP without workflow_params.

            Return type
                None

        test_run_and_cancel_workflows()
            Run two workflows, cancel one of them, and make sure they all exist.

            Return type
                None

    class toil.test.server.serverTest.ToilWESServerCeleryWorkflowTest(*args, **kwargs)
        Bases: ToilWESServerWorkflowTest

        End-to-end workflow-running tests against Celery.

    class toil.test.server.serverTest.ToilWESServerCeleryS3StateWorkflowTest(*args, **kwargs)
        Bases: ToilWESServerWorkflowTest, BucketUsingTest

        Test the server with Celery and state stored in S3.

        setUp()
            Hook method for setting up the test fixture before exercising it.

            Return type
                None

  toil.test.sort

  Submodules

  toil.test.sort.restart_sort

A demonstration of toil. Sorts the lines of a file into ascending order by doing a parallel merge sort. This is an intentionally buggy version that doesn't include restart() for testing purposes.

  Attributes
defaultLines |
defaultLineLen |
sortMemory |

  Functions
setup(job, inputFile, N, downCheckpoints, options) | Sets up the sort.
down(job, inputFileStoreID, N, path, downCheckpoints, ...) | Input is a file, a subdivision size N, and a path in the hierarchy of jobs.
up(job, inputFileID1, inputFileID2, path, options[, ...]) | Merges the two files and places them in the output.
sort(file) | Sorts the given file.
merge(fileHandle1, fileHandle2, outputFileHandle) | Merges together two files maintaining sorted order.
copySubRangeOfFile(inputFile, fileStart, fileEnd) | Copies the range (in bytes) between fileStart and fileEnd to the given
getMidPoint(file, fileStart, fileEnd) | Finds the point in the file to split.
makeFileToSort(fileName[, lines, lineLen]) |
main([options]) |

  Module Contents

    toil.test.sort.restart_sort.defaultLines = 1000

    toil.test.sort.restart_sort.defaultLineLen = 50

    toil.test.sort.restart_sort.sortMemory = '600M'

    toil.test.sort.restart_sort.setup(job, inputFile, N, downCheckpoints, options)
        Sets up the sort. Returns the FileID of the sorted file

    toil.test.sort.restart_sort.down(job, inputFileStoreID, N, path, downCheckpoints, options, memory=sortMemory)
        Input is a file, a subdivision size N, and a path in the hierarchy of jobs. If the range is larger than a threshold N the range is divided recursively and a follow on job is then created which merges back the results else the file is sorted and placed in the output.

    toil.test.sort.restart_sort.up(job, inputFileID1, inputFileID2, path, options, memory=sortMemory)
        Merges the two files and places them in the output.

    toil.test.sort.restart_sort.sort(file)
        Sorts the given file.

    toil.test.sort.restart_sort.merge(fileHandle1, fileHandle2, outputFileHandle)
        Merges together two files maintaining sorted order.

        All handles must be text-mode streams.

    toil.test.sort.restart_sort.copySubRangeOfFile(inputFile, fileStart, fileEnd)
        Copies the range (in bytes) between fileStart and fileEnd to the given output file handle.

    toil.test.sort.restart_sort.getMidPoint(file, fileStart, fileEnd)
        Finds the point in the file to split. Returns an int i such that fileStart <= i < fileEnd

    toil.test.sort.restart_sort.makeFileToSort(fileName, lines=defaultLines, lineLen=defaultLineLen)

    toil.test.sort.restart_sort.main(options=None)

  toil.test.sort.sort

A demonstration of toil. Sorts the lines of a file into ascending order by doing a parallel merge sort.

  Attributes
defaultLines |
defaultLineLen |
sortMemory |

  Functions
setup(job, inputFile, N, downCheckpoints, options) | Sets up the sort.
down(job, inputFileStoreID, N, path, downCheckpoints, ...) | Input is a file, a subdivision size N, and a path in the hierarchy of jobs.
up(job, inputFileID1, inputFileID2, path, options[, ...]) | Merges the two files and places them in the output.
sort(file) | Sorts the given file.
merge(fileHandle1, fileHandle2, outputFileHandle) | Merges together two files maintaining sorted order.
copySubRangeOfFile(inputFile, fileStart, fileEnd) | Copies the range (in bytes) between fileStart and fileEnd to the given
getMidPoint(file, fileStart, fileEnd) | Finds the point in the file to split.
makeFileToSort(fileName[, lines, lineLen]) |
main([options]) |

  Module Contents

    toil.test.sort.sort.defaultLines = 1000

    toil.test.sort.sort.defaultLineLen = 50

    toil.test.sort.sort.sortMemory = '600M'

    toil.test.sort.sort.setup(job, inputFile, N, downCheckpoints, options)
        Sets up the sort. Returns the FileID of the sorted file

    toil.test.sort.sort.down(job, inputFileStoreID, N, path, downCheckpoints, options, memory=sortMemory)
        Input is a file, a subdivision size N, and a path in the hierarchy of jobs. If the range is larger than a threshold N the range is divided recursively and a follow on job is then created which merges back the results else the file is sorted and placed in the output.

    toil.test.sort.sort.up(job, inputFileID1, inputFileID2, path, options, memory=sortMemory)
        Merges the two files and places them in the output.

    toil.test.sort.sort.sort(file)
        Sorts the given file.

    toil.test.sort.sort.merge(fileHandle1, fileHandle2, outputFileHandle)
        Merges together two files maintaining sorted order.

        All handles must be text-mode streams.

    toil.test.sort.sort.copySubRangeOfFile(inputFile, fileStart, fileEnd)
        Copies the range (in bytes) between fileStart and fileEnd to the given output file handle.

    toil.test.sort.sort.getMidPoint(file, fileStart, fileEnd)
        Finds the point in the file to split. Returns an int i such that fileStart <= i < fileEnd

    toil.test.sort.sort.makeFileToSort(fileName, lines=defaultLines, lineLen=defaultLineLen)

        Parameters

            •   fileName (toil.lib.misc.StrPath)

            •   lines (int </usr/share/doc/python3/html/library/functions.html#int>)

            •   lineLen (int </usr/share/doc/python3/html/library/functions.html#int>)

        Return type
            None

    toil.test.sort.sort.main(options=None)

  toil.test.sort.sortTest

  Attributes
logger |
defaultLineLen |
defaultLines |
defaultN |

  Classes
SortTest | Tests Toil by sorting a file in parallel on various combinations of job stores and batch

  Functions
runMain(options) | make sure the output file is deleted every time main is run

  Module Contents

    toil.test.sort.sortTest.logger

    toil.test.sort.sortTest.defaultLineLen

    toil.test.sort.sortTest.defaultLines

    toil.test.sort.sortTest.defaultN

    toil.test.sort.sortTest.runMain(options)
        make sure the output file is deleted every time main is run

    class toil.test.sort.sortTest.SortTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>, toil.batchSystems.mesos.test.MesosTestSupport <#toil.batchSystems.mesos.test.MesosTestSupport>

        Tests Toil by sorting a file in parallel on various combinations of job stores and batch systems.

        setUp()
            Hook method for setting up the test fixture before exercising it.

        tearDown()
            Hook method for deconstructing the test fixture after testing it.

        testAwsSingle()

        testAwsMesos()

        testFileMesos()

        testGoogleSingle()

        testGoogleMesos()

        testFileSingle()

        testFileSingleNonCaching()

        testFileSingleCheckpoints()

        testFileSingle10000()

        testFileGridEngine()

        testFileTorqueEngine()

        testNo = 5

        testSort()

        testMerge()

        testCopySubRangeOfFile()

        testGetMidPoint()

  toil.test.src

  Submodules

  toil.test.src.autoDeploymentTest

  Attributes
logger |

  Classes
AutoDeploymentTest | Tests various auto-deployment scenarios. Using the appliance, i.e. a docker container,

  Module Contents

    toil.test.src.autoDeploymentTest.logger

    class toil.test.src.autoDeploymentTest.AutoDeploymentTest(methodName='runTest')
        Bases: toil.test.ApplianceTestSupport <#toil.test.ApplianceTestSupport>

        Tests various auto-deployment scenarios. Using the appliance, i.e. a docker container, for these tests allows for running worker processes on the same node as the leader process while keeping their file systems separate from each other and the leader process. Separate file systems are crucial to prove that auto-deployment does its job.

        setUp()
            Hook method for setting up the test fixture before exercising it.

        sitePackages = 'venv/lib/python3.13/site-packages'

        testRestart()
            Test whether auto-deployment works on restart.

        testSplitRootPackages()
            Test whether auto-deployment works with a virtualenv in which jobs are defined in completely separate branches of the package hierarchy. Initially, auto-deployment did deploy the entire virtualenv but jobs could only be defined in one branch of the package hierarchy. We define a branch as the maximum set of fully qualified package paths that share the same first component. IOW, a.b and a.c are in the same branch, while a.b and d.c are not.

        testUserTypesInJobFunctionArgs()
            Test encapsulated, function-wrapping jobs where the function arguments reference user-defined types.

            Mainly written to cover <https://github.com/BD2KGenomics/toil/issues/1259> but then also revealed <https://github.com/BD2KGenomics/toil/issues/1278>.

        testDeferralWithConcurrentEncapsulation()
            Ensure that the following DAG succeeds:

                                  ┌───────────┐
                                  │ Root (W1) │
                                  └───────────┘
                                        │
                             ┌──────────┴─────────┐
                             ▼                    ▼
                    ┌────────────────┐ ┌────────────────────┐
                    │ Deferring (W2) │ │ Encapsulating (W3) │═══════════════╗
                    └────────────────┘ └────────────────────┘               ║
                                                  │                         ║
                                                  ▼                         ▼
                                        ┌───────────────────┐      ┌────────────────┐
                                        │ Encapsulated (W3) │      │ Follow-on (W6) │
                                        └───────────────────┘      └────────────────┘
                                                  │                         │
                                          ┌───────┴────────┐                │
                                          ▼                ▼                ▼
                                  ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
                                  │ Dummy 1 (W4) │ │ Dummy 2 (W5) │ │  Last (W6)   │
                                  └──────────────┘ └──────────────┘ └──────────────┘

        The Wn numbers denote the worker processes that a particular job is run in. Deferring adds a deferred function and then runs for a long time. The deferred function will be present in the cache state for the duration of Deferred. Follow-on is the generic Job instance that's added by encapsulating a job. It runs on the same worker node but in a separate worker process, as the first job in that worker. Because …

        1) it is the first job in its worker process (the user script has not been made available on the sys.path by a previous job in that worker) and

            2.  it shares the cache state with the Deferring job and

            3.  it is an instance of Job (and so does not introduce the user script to sys.path itself),

        … it might cause problems with deserializing a deferred function defined in the user script.

        Encapsulated has two children to ensure that Follow-on is run in a separate worker.

        testDeferralWithFailureAndEncapsulation()
            Ensure that the following DAG succeeds:

                                  ┌───────────┐
                                  │ Root (W1) │
                                  └───────────┘
                                        │
                             ┌──────────┴─────────┐
                             ▼                    ▼
                    ┌────────────────┐ ┌────────────────────┐
                    │ Deferring (W2) │ │ Encapsulating (W3) │═══════════════════════╗
                    └────────────────┘ └────────────────────┘                       ║
                                                  │                                 ║
                                                  ▼                                 ▼
                                        ┌───────────────────┐              ┌────────────────┐
                                        │ Encapsulated (W3) │════════════╗ │ Follow-on (W7) │
                                        └───────────────────┘            ║ └────────────────┘
                                                  │                      ║
                                           ┌──────┴──────┐               ║
                                           ▼             ▼               ▼
                                    ┌────────────┐┌────────────┐ ┌──────────────┐
                                    │ Dummy (W4) ││ Dummy (W5) │ │ Trigger (W6) │
                                    └────────────┘└────────────┘ └──────────────┘

        Trigger causes Deferring to crash. Follow-on runs next, detects Deferring's left-overs and runs the deferred function. Follow-on is an instance of Job and the first job in its worker process. This test ensures that despite these circumstances, the user script is loaded before the deferred functions defined in it are being run.

        Encapsulated has two children to ensure that Follow-on is run in a new worker. That's the only way to guarantee that the user script has not been loaded yet, which would cause the test to succeed coincidentally. We want to test that auto-deploying and loading of the user script are done properly before deferred functions are being run and before any jobs have been executed by that worker.

  toil.test.src.busTest

  Attributes
logger |

  Classes
TestMessageBus |

  Functions
failing_job_fn(job) | This function is guaranteed to fail.

  Module Contents

    toil.test.src.busTest.logger

    class toil.test.src.busTest.TestMessageBus

        test_enum_ints_in_file(tmp_path)
            Make sure writing bus messages to files works with enums.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_cross_thread_messaging()
            Make sure message bus works across threads.

            Return type
                None

        test_restart_without_bus_path(tmp_path)
            Test the ability to restart a workflow when the message bus path used by the previous attempt is gone.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

    toil.test.src.busTest.failing_job_fn(job)
        This function is guaranteed to fail.

        Parameters
            job (toil.job.Job <#toil.job.Job>)

        Return type
            NoReturn

  toil.test.src.checkpointTest

  Classes
CheckpointTest | Legacy common base class for Toil tests.
CheckRetryCount | Fail N times, succeed on the next try.
AlwaysFail | Class represents a unit of work in toil.
CheckpointFailsFirstTime | Class represents a unit of work in toil.
FailOnce | Fail the first time the workflow is run, but succeed thereafter.

  Module Contents

    class toil.test.src.checkpointTest.CheckpointTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        testCheckpointNotRetried()
            A checkpoint job should not be retried if the workflow has a retryCount of 0.

        testCheckpointRetriedOnce()
            A checkpoint job should be retried exactly once if the workflow has a retryCount of 1.

        testCheckpointedRestartSucceeds()
            A checkpointed job should succeed on restart of a failed run if its child job succeeds.

    class toil.test.src.checkpointTest.CheckRetryCount(numFailuresBeforeSuccess)
        Bases: toil.job.Job <#toil.job.Job>

        Fail N times, succeed on the next try.

        numFailuresBeforeSuccess

        getNumRetries(fileStore)
            Mark a retry in the fileStore, and return the number of retries so far.

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>.

    class toil.test.src.checkpointTest.AlwaysFail(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, preemptable=None, unitName='', checkpoint=False, displayName='', descriptionClass=None, local=None, files=None)
        Bases: toil.job.Job <#toil.job.Job>

        Class represents a unit of work in toil.

        Parameters

            •   memory (Optional[ParseableIndivisibleResource])

            •   cores (Optional[ParseableDivisibleResource])

            •   disk (Optional[ParseableIndivisibleResource])

            •   accelerators (Optional[ParseableAcceleratorRequirement])

            •   preemptible (Optional[ParseableFlag])

            •   preemptable (Optional[ParseableFlag])

            •   unitName (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   checkpoint (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>])

            •   displayName (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   descriptionClass (Optional[type </usr/share/doc/python3/html/library/functions.html#type>])

            •   local (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>])

            •   files (Optional[set </usr/share/doc/python3/html/library/stdtypes.html#set>[toil.fileStores.FileID <#toil.fileStores.FileID>]])

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>.

    class toil.test.src.checkpointTest.CheckpointFailsFirstTime
        Bases: toil.job.Job <#toil.job.Job>

        Class represents a unit of work in toil.

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>.

    class toil.test.src.checkpointTest.FailOnce(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, preemptable=None, unitName='', checkpoint=False, displayName='', descriptionClass=None, local=None, files=None)
        Bases: toil.job.Job <#toil.job.Job>

        Fail the first time the workflow is run, but succeed thereafter.

        Parameters

            •   memory (Optional[ParseableIndivisibleResource])

            •   cores (Optional[ParseableDivisibleResource])

            •   disk (Optional[ParseableIndivisibleResource])

            •   accelerators (Optional[ParseableAcceleratorRequirement])

            •   preemptible (Optional[ParseableFlag])

            •   preemptable (Optional[ParseableFlag])

            •   unitName (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   checkpoint (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>])

            •   displayName (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   descriptionClass (Optional[type </usr/share/doc/python3/html/library/functions.html#type>])

            •   local (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>])

            •   files (Optional[set </usr/share/doc/python3/html/library/stdtypes.html#set>[toil.fileStores.FileID <#toil.fileStores.FileID>]])

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>.

  toil.test.src.deferredFunctionTest

  Attributes
logger |

  Classes
TestDeferredFunction | Test the deferred function system.

  Functions
options(tmp_path) |

  Module Contents

    toil.test.src.deferredFunctionTest.logger

    toil.test.src.deferredFunctionTest.options(tmp_path)

        Parameters
            tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            collections.abc.Generator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator>[argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>]

    class toil.test.src.deferredFunctionTest.TestDeferredFunction
        Test the deferred function system.

        testDeferredFunctionRunsWithMethod(tmp_path, options)
            Refer docstring in _testDeferredFunctionRuns. Test with Method

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   options (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)

            Return type
                None

        testDeferredFunctionRunsWithClassMethod(tmp_path, options)
            Refer docstring in _testDeferredFunctionRuns. Test with Class Method

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   options (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)

            Return type
                None

        testDeferredFunctionRunsWithLambda(tmp_path, options)
            Refer docstring in _testDeferredFunctionRuns. Test with Lambda

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   options (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)

            Return type
                None

        testDeferredFunctionRunsWithFailures(options, tmp_path)
            Create 2 non local filesto use as flags. Create a job that registers a function that deletes one non local file. If that file exists, the job SIGKILLs itself. If it doesn't exist, the job registers a second deferred function to delete the second non local file and exits normally.

            Initially the first file exists, so the job should SIGKILL itself and neither deferred function will run (in fact, the second should not even be registered). On the restart, the first deferred function should run and the first file should not exist, but the second one should. We assert the presence of the second, then register the second deferred function and exit normally. At the end of the test, neither file should exist.

            Incidentally, this also tests for multiple registered deferred functions, and the case where a deferred function fails (since the first file doesn't exist on the retry).

            Parameters

                •   options (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testNewJobsCanHandleOtherJobDeaths(options, tmp_path)
            Create 2 non-local files and then create 2 jobs. The first job registers a deferred job to delete the second non-local file, deletes the first non-local file and then kills itself. The second job waits for the first file to be deleted, then sleeps for a few seconds and then spawns a child. the child of the second does nothing. However starting it should handle the untimely demise of the first job and run the registered deferred function that deletes the first file. We assert the absence of the two files at the end of the run.

            Parameters

                •   options (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testBatchSystemCleanupCanHandleWorkerDeaths(options, tmp_path)
            Create some non-local files. Create a job that registers a deferred function to delete the file and then kills its worker.

            Assert that the file is missing after the pipeline fails, because we're using a single-machine batch system and the leader's batch system cleanup will find and run the deferred function.

            Parameters

                •   options (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

  toil.test.src.dockerCheckTest

  Classes
DockerCheckTest | Tests checking whether a docker image exists or not.

  Module Contents

    class toil.test.src.dockerCheckTest.DockerCheckTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Tests checking whether a docker image exists or not.

        testOfficialUbuntuRepo()
            Image exists. This should pass.

        testBroadDockerRepo()
            Image exists. This should pass.

        testBroadDockerRepoBadTag()
            Bad tag. This should raise.

        testNonexistentRepo()
            Bad image. This should raise.

        testToilQuayRepo()
            Image exists. Should pass.

        testBadQuayRepoNTag()
            Bad repo and tag. This should raise.

        testBadQuayRepo()
            Bad repo. This should raise.

        testBadQuayTag()
            Bad tag. This should raise.

        testGoogleRepo()
            Image exists. Should pass.

        testBadGoogleRepo()
            Bad repo and tag. This should raise.

        testApplianceParser()
            Test that a specified appliance is parsed correctly.

  toil.test.src.environmentTest

  Attributes
logger |

  Classes
EnvironmentTest | Test to make sure that Toil's environment variable save and restore system

  Functions
signal_leader(job) | Make a file in the file store that the leader can see.
check_environment(job, try_name) | Fail if the test environment is wrong.
wait_a_bit(job) | Toil job that waits.
check_environment_repeatedly(job) | Toil job that checks the environment, waits, and checks it again, as
main([options]) | Run the actual workflow with the given options.

  Module Contents

    toil.test.src.environmentTest.logger

    class toil.test.src.environmentTest.EnvironmentTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Test to make sure that Toil's environment variable save and restore system (environment.pickle) works.

        The environment should be captured once at the start of the workflow and should be sent through based on that, not base don the leader's current environment when the job is launched.

        test_environment()

    toil.test.src.environmentTest.signal_leader(job)
        Make a file in the file store that the leader can see.

    toil.test.src.environmentTest.check_environment(job, try_name)
        Fail if the test environment is wrong.

        Parameters
            try_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

    toil.test.src.environmentTest.wait_a_bit(job)
        Toil job that waits.

    toil.test.src.environmentTest.check_environment_repeatedly(job)
        Toil job that checks the environment, waits, and checks it again, as separate invocations.

    toil.test.src.environmentTest.main(options=None)
        Run the actual workflow with the given options.

        Parameters
            options (Optional[argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>])

  toil.test.src.fileStoreTest

  Attributes
testingIsAutomatic |
logger |

  Classes
hidden | Hiding the abstract test classes from the Unittest loader so it can be inherited in different
NonCachingFileStoreTestWithFileJobStore | Abstract tests for the the various functions in
CachingFileStoreTestWithFileJobStore | Abstract tests for the the various cache-related functions in
NonCachingFileStoreTestWithAwsJobStore | Abstract tests for the the various functions in
CachingFileStoreTestWithAwsJobStore | Abstract tests for the the various cache-related functions in
NonCachingFileStoreTestWithGoogleJobStore | Abstract tests for the the various functions in
CachingFileStoreTestWithGoogleJobStore | Abstract tests for the the various cache-related functions in

  Module Contents

    toil.test.src.fileStoreTest.testingIsAutomatic = True

    toil.test.src.fileStoreTest.logger

    class toil.test.src.fileStoreTest.hidden
        Hiding the abstract test classes from the Unittest loader so it can be inherited in different test suites for the different job stores.

        class AbstractFileStoreTest(methodName='runTest')
            Bases: toil.test.ToilTest <#toil.test.ToilTest>

            An abstract base class for testing the various general functions described in :class:toil.fileStores.abstractFileStore.AbstractFileStore

            jobStoreType = None

            setUp()
                Hook method for setting up the test fixture before exercising it.

            options()
                Mint a fresh options object with default settings and a fresh jobstore.

            create_file(content, executable=False)

            testToilIsNotBroken()
                Runs a simple DAG to test if if any features other that caching were broken.

            testFileStoreLogging()
                Write a couple of files to the jobstore. Delete a couple of them. Read back written and locally deleted files.

            testFileStoreOperations()
                Write a couple of files to the jobstore. Delete a couple of them. Read back written and locally deleted files.

            testWriteReadGlobalFilePermissions()
                Ensures that uploaded files preserve their file permissions when they are downloaded again. This function checks that a written executable file maintains its executability after being read.

            testWriteExportFileCompatibility()
                Ensures that files created in a job preserve their executable permissions when they are exported from the leader.

            testImportReadFileCompatibility()
                Ensures that files imported to the leader preserve their executable permissions when they are read by the fileStore.

            testReadWriteFileStreamTextMode()
                Checks if text mode is compatible with file streams.

        class AbstractNonCachingFileStoreTest(methodName='runTest')
            Bases: AbstractFileStoreTest

            Abstract tests for the the various functions in :class:toil.fileStores.nonCachingFileStore.NonCachingFileStore. These tests are general enough that they can also be used for :class:toil.fileStores.CachingFileStore.

            options()
                Mint a fresh options object with default settings and a fresh jobstore.

        class AbstractCachingFileStoreTest(methodName='runTest')
            Bases: AbstractFileStoreTest

            Abstract tests for the the various cache-related functions in :class:toil.fileStores.cachingFileStore.CachingFileStore.

            options()
                Mint a fresh options object with default settings and a fresh jobstore.

            testExtremeCacheSetup()
                Try to create the cache with bad worker active and then have 10 child jobs try to run in the chain. This tests whether the cache is created properly even when the job crashes randomly.

            testCacheEvictionPartialEvict()
                Ensure the cache eviction happens as expected. Two files (20MB and 30MB) are written sequentially into the job store in separate jobs. The cache max is force set to 50MB. A Third Job requests 10MB of disk requiring eviction of the 1st file. Ensure that the behavior is as expected.

            testCacheEvictionTotalEvict()
                Ensure the cache eviction happens as expected. Two files (20MB and 30MB) are written sequentially into the job store in separate jobs. The cache max is force set to 50MB. A Third Job requests 10MB of disk requiring eviction of the 1st file. Ensure that the behavior is as expected.

            testCacheEvictionFailCase()
                Ensure the cache eviction happens as expected. Two files (20MB and 30MB) are written sequentially into the job store in separate jobs. The cache max is force set to 50MB. A Third Job requests 10MB of disk requiring eviction of the 1st file. Ensure that the behavior is as expected.

            testAsyncWriteWithCaching()
                Ensure the Async Writing of files happens as expected. The first Job forcefully modifies the cache size to 1GB. The second asks for 1GB of disk and writes a 900MB file into cache then rewrites it to the job store triggering an async write since the two unique jobstore IDs point to the same local file. Also, the second write is not cached since the first was written to cache, and there "isn't enough space" to cache the second. Immediately assert that the second write isn't cached, and is being asynchronously written to the job store.

                Attempting to get the file from the jobstore should not fail.

            testWriteNonLocalFileToJobStore()
                Write a file not in localTempDir to the job store. Such a file should not be cached. Ensure the file is not cached.

            testWriteLocalFileToJobStore()
                Write a file from the localTempDir to the job store. Such a file will be cached by default. Ensure the file is cached.

            testReadCacheMissFileFromJobStoreWithoutCachingReadFile()
                Read a file from the file store that does not have a corresponding cached copy. Do not cache the read file. Ensure the number of links on the file are appropriate.

            testReadCacheMissFileFromJobStoreWithCachingReadFile()
                Read a file from the file store that does not have a corresponding cached copy. Cache the read file. Ensure the number of links on the file are appropriate.

            testReadCachHitFileFromJobStore()
                Read a file from the file store that has a corresponding cached copy. Ensure the number of links on the file are appropriate.

            testMultipleJobsReadSameCacheHitGlobalFile()
                Write a local file to the job store (hence adding a copy to cache), then have 10 jobs read it. Assert cached file size never goes up, assert unused job required disk space is always (a multiple of job reqs) - (number of current file readers * filesize). At the end, assert the cache shows unused job-required space = 0.

            testMultipleJobsReadSameCacheMissGlobalFile()
                Write a non-local file to the job store(hence no cached copy), then have 10 jobs read it. Assert cached file size never goes up, assert unused job required disk space is always (a multiple of job reqs) - (number of current file readers * filesize). At the end, assert the cache shows unused job-required space = 0.

            testFileStoreExportFile()

            testReturnFileSizes()
                Write a couple of files to the jobstore. Delete a couple of them. Read back written and locally deleted files. Ensure that after every step that the cache is in a valid state.

            testReturnFileSizesWithBadWorker()
                Write a couple of files to the jobstore. Delete a couple of them. Read back written and locally deleted files. Ensure that after every step that the cache is in a valid state.

            testControlledFailedWorkerRetry()
                Conduct a couple of job store operations. Then die. Ensure that the restarted job is tracking values in the cache state file appropriately.

            testRemoveLocalMutablyReadFile()
                If a mutably read file is deleted by the user, it is ok.

            testRemoveLocalImmutablyReadFile()
                If an immutably read file is deleted by the user, it is not ok.

            testDeleteLocalFile()
                Test the deletion capabilities of deleteLocalFile

            testSimultaneousReadsUncachedStream()
                Test many simultaneous read attempts on a file created via a stream directly to the job store.

    class toil.test.src.fileStoreTest.NonCachingFileStoreTestWithFileJobStore(methodName='runTest')
        Bases: hidden

        Abstract tests for the the various functions in :class:toil.fileStores.nonCachingFileStore.NonCachingFileStore. These tests are general enough that they can also be used for :class:toil.fileStores.CachingFileStore.

        jobStoreType = 'file'

    class toil.test.src.fileStoreTest.CachingFileStoreTestWithFileJobStore(methodName='runTest')
        Bases: hidden

        Abstract tests for the the various cache-related functions in :class:toil.fileStores.cachingFileStore.CachingFileStore.

        jobStoreType = 'file'

    class toil.test.src.fileStoreTest.NonCachingFileStoreTestWithAwsJobStore(methodName='runTest')
        Bases: hidden

        Abstract tests for the the various functions in :class:toil.fileStores.nonCachingFileStore.NonCachingFileStore. These tests are general enough that they can also be used for :class:toil.fileStores.CachingFileStore.

        jobStoreType = 'aws'

    class toil.test.src.fileStoreTest.CachingFileStoreTestWithAwsJobStore(methodName='runTest')
        Bases: hidden

        Abstract tests for the the various cache-related functions in :class:toil.fileStores.cachingFileStore.CachingFileStore.

        jobStoreType = 'aws'

    class toil.test.src.fileStoreTest.NonCachingFileStoreTestWithGoogleJobStore(methodName='runTest')
        Bases: hidden

        Abstract tests for the the various functions in :class:toil.fileStores.nonCachingFileStore.NonCachingFileStore. These tests are general enough that they can also be used for :class:toil.fileStores.CachingFileStore.

        jobStoreType = 'google'

    class toil.test.src.fileStoreTest.CachingFileStoreTestWithGoogleJobStore(methodName='runTest')
        Bases: hidden

        Abstract tests for the the various cache-related functions in :class:toil.fileStores.cachingFileStore.CachingFileStore.

        jobStoreType = 'google'

  toil.test.src.helloWorldTest

  Classes
HelloWorldTest | Legacy common base class for Toil tests.
HelloWorld | Class represents a unit of work in toil.
FollowOn | Class represents a unit of work in toil.

  Functions
childFn(job) |

  Module Contents

    class toil.test.src.helloWorldTest.HelloWorldTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        testHelloWorld()

    class toil.test.src.helloWorldTest.HelloWorld
        Bases: toil.job.Job <#toil.job.Job>

        Class represents a unit of work in toil.

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>.

    toil.test.src.helloWorldTest.childFn(job)

    class toil.test.src.helloWorldTest.FollowOn(fileId)
        Bases: toil.job.Job <#toil.job.Job>

        Class represents a unit of work in toil.

        fileId

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>.

  toil.test.src.importExportFileTest

  Classes
TestImportExportFile |
RestartingJob | Class represents a unit of work in toil.

  Functions
create_file(tmp_path, content[, executable]) |

  Module Contents

    toil.test.src.importExportFileTest.create_file(tmp_path, content, executable=False)

        Parameters

            •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            •   content (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   executable (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>

    class toil.test.src.importExportFileTest.TestImportExportFile

        message_portion_1 = 'What do you get when you cross a seal and a polar bear?'

        message_portion_2 = ' A polar bear.'

        test_import_export_restart_true(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_import_export_restart_false(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_basic_import_export(tmp_path, subtests)
            Ensures that uploaded files preserve their file permissions when they are downloaded again. This function checks that an imported executable file maintains its executability after being exported.

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   subtests (pytest.Subtests)

            Return type
                None

    class toil.test.src.importExportFileTest.RestartingJob(msg_portion_file_id, trigger_file_id, message_portion_2)
        Bases: toil.job.Job <#toil.job.Job>

        Class represents a unit of work in toil.

        Parameters

            •   msg_portion_file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   trigger_file_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   message_portion_2 (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        msg_portion_file_id

        trigger_file_id

        message_portion_2

        run(file_store)
            Override this function to perform work and dynamically create successor jobs.

            Parameters

                •   fileStore -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

                •   file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

  toil.test.src.jobDescriptionTest

  Classes
JobDescriptionTest | Legacy common base class for Toil tests.

  Module Contents

    class toil.test.src.jobDescriptionTest.JobDescriptionTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        setUp()
            Hook method for setting up the test fixture before exercising it.

        tearDown()
            Hook method for deconstructing the test fixture after testing it.

        testJobDescription()
            Tests the public interface of a JobDescription.

        testJobDescriptionSequencing()

  toil.test.src.jobEncapsulationTest

  Classes
TestJobEncapsulation | Tests testing the EncapsulationJob class.

  Functions
noOp() |
encapsulatedJobFn(job, string, outFile) |

  Module Contents

    class toil.test.src.jobEncapsulationTest.TestJobEncapsulation
        Tests testing the EncapsulationJob class.

        testEncapsulation(tmp_path)
            Tests the Job.encapsulation method, which uses the EncapsulationJob class.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testAddChildEncapsulate()
            Make sure that the encapsulate child does not have two parents with unique roots.

            Return type
                None

    toil.test.src.jobEncapsulationTest.noOp()

        Return type
            None

    toil.test.src.jobEncapsulationTest.encapsulatedJobFn(job, string, outFile)

        Parameters

            •   job (toil.job.Job <#toil.job.Job>)

            •   string (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   outFile (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            toil.job.Promise <#toil.job.Promise>

  toil.test.src.jobFileStoreTest

  Attributes
logger |
PREFIX_LENGTH |
fileStoreString |
streamingFileStoreString |

  Classes
JobFileStoreTest | Tests testing the methods defined in :class:toil.fileStores.abstractFileStore.AbstractFileStore.

  Functions
fileTestJob(job, inputFileStoreIDs, testStrings, ...) | Test job exercises toil.fileStores.abstractFileStore.AbstractFileStore functions
simpleFileStoreJob(job) |
fileStoreChild(job, testID1, testID2) |

  Module Contents

    toil.test.src.jobFileStoreTest.logger

    toil.test.src.jobFileStoreTest.PREFIX_LENGTH = 200

    class toil.test.src.jobFileStoreTest.JobFileStoreTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Tests testing the methods defined in :class:toil.fileStores.abstractFileStore.AbstractFileStore.

        testCachingFileStore()

        testNonCachingFileStore()

        testJobFileStore()
            Tests case that about half the files are cached

        testJobFileStoreWithBadWorker()
            Tests case that about half the files are cached and the worker is randomly failing.

    toil.test.src.jobFileStoreTest.fileTestJob(job, inputFileStoreIDs, testStrings, chainLength)
        Test job exercises toil.fileStores.abstractFileStore.AbstractFileStore functions

    toil.test.src.jobFileStoreTest.fileStoreString = 'Testing writeGlobalFile'

    toil.test.src.jobFileStoreTest.streamingFileStoreString = 'Testing writeGlobalFileStream'

    toil.test.src.jobFileStoreTest.simpleFileStoreJob(job)

    toil.test.src.jobFileStoreTest.fileStoreChild(job, testID1, testID2)

  toil.test.src.jobServiceTest

  Attributes
logger |

  Classes
TestJobService | Tests testing the Job.Service class
TestPerfectServicet | Tests testing the Job.Service class
ToyService | Abstract class used to define the interface to a service.
ToySerializableService | Abstract class used to define the interface to a service.

  Functions
serviceTest(job, outFile, messageInt) | Creates one service and one accessing job, which communicate with two files to establish
serviceTestRecursive(job, outFile, messages) | Creates a chain of services and accessing jobs, each paired together.
serviceTestParallelRecursive(job, outFiles, messageBundles) | Creates multiple chains of services and accessing jobs.
serviceAccessor(job, communicationFiles, outFile, ...) | Writes a random integer iinto the inJobStoreFileID file, then tries 10 times reading
fnTest(strings, outputFile) | Function concatenates the strings together and writes them to the output file

  Module Contents

    toil.test.src.jobServiceTest.logger

    class toil.test.src.jobServiceTest.TestJobService
        Tests testing the Job.Service class

        testServiceSerialization(tmp_path)
            Tests that a service can receive a promise without producing a serialization error.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testService(tmp_path, checkpoint=False)
            Tests the creation of a Job.Service with random failures of the worker.

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   checkpoint (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        testServiceDeadlock(tmp_path)
            Creates a job with more services than maxServices, checks that deadlock is detected.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testServiceWithCheckpoints(tmp_path)
            Tests the creation of a Job.Service with random failures of the worker, making the root job use checkpointing to restart the subtree.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testServiceRecursive(tmp_path, checkpoint=True)
            Tests the creation of a Job.Service, creating a chain of services and accessing jobs. Randomly fails the worker.

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   checkpoint (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        testServiceParallelRecursive(tmp_path, checkpoint=True)
            Tests the creation of a Job.Service, creating parallel chains of services and accessing jobs. Randomly fails the worker.

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   checkpoint (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        runToil(tmp_path, rootJob, retryCount=1, badWorker=0.5, badWorkedFailInterval=0.1, maxServiceJobs=sys.maxsize, deadlockWait=60, max_attempts=50)

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   rootJob (toil.job.Job <#toil.job.Job>)

                •   retryCount (int </usr/share/doc/python3/html/library/functions.html#int>)

                •   badWorker (float </usr/share/doc/python3/html/library/functions.html#float>)

                •   badWorkedFailInterval (float </usr/share/doc/python3/html/library/functions.html#float>)

                •   maxServiceJobs (int </usr/share/doc/python3/html/library/functions.html#int>)

                •   deadlockWait (int </usr/share/doc/python3/html/library/functions.html#int>)

                •   max_attempts (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                None

    class toil.test.src.jobServiceTest.TestPerfectServicet
        Bases: TestJobService

        Tests testing the Job.Service class

        runToil(*args, **kwargs)
            Let us run all the tests in the other service test class, but without worker failures.

            Parameters

                •   args (Any)

                •   kwargs (Any)

            Return type
                None

    toil.test.src.jobServiceTest.serviceTest(job, outFile, messageInt)
        Creates one service and one accessing job, which communicate with two files to establish that both run concurrently.

        Parameters

            •   job (toil.job.Job <#toil.job.Job>)

            •   outFile (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            •   messageInt (int </usr/share/doc/python3/html/library/functions.html#int>)

        Return type
            None

    toil.test.src.jobServiceTest.serviceTestRecursive(job, outFile, messages)
        Creates a chain of services and accessing jobs, each paired together.

        Parameters

            •   job (toil.job.Job <#toil.job.Job>)

            •   outFile (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            •   messages (list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>])

        Return type
            None

    toil.test.src.jobServiceTest.serviceTestParallelRecursive(job, outFiles, messageBundles)
        Creates multiple chains of services and accessing jobs.

        Parameters

            •   job (toil.job.Job <#toil.job.Job>)

            •   outFiles (list </usr/share/doc/python3/html/library/stdtypes.html#list>[pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>])

            •   messageBundles (list </usr/share/doc/python3/html/library/stdtypes.html#list>[list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>]])

        Return type
            None

    class toil.test.src.jobServiceTest.ToyService(messageInt, *args, **kwargs)
        Bases: toil.job.Job.Service <#toil.job.Job.Service>

        Abstract class used to define the interface to a service.

        Should be subclassed by the user to define services.

        Is not executed as a job; runs within a ServiceHostJob.

        Parameters

            •   messageInt (int </usr/share/doc/python3/html/library/functions.html#int>)

            •   args (Any)

            •   kwargs (Any)

        messageInt

        start(job)
            Start the service.

            Parameters
                job (toil.job.ServiceHostJob <#toil.job.ServiceHostJob>) -- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files.

            Returns
                An object describing how to access the service. The object must be pickleable and will be used by jobs to access the service (see toil.job.Job.addService() <#toil.job.Job.addService>).

            Return type
                tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        stop(job)
            Stops the service. Function can block until complete.

            Parameters
                job (toil.job.Job <#toil.job.Job>) -- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files.

            Return type
                None

        check()
            Checks the service is still running.

            Raises
                exceptions.RuntimeError -- If the service failed, this will cause the service job to be labeled failed.

            Returns
                True if the service is still running, else False. If False then the service job will be terminated, and considered a success. Important point: if the service job exits due to a failure, it should raise a RuntimeError, not return False!

            Return type
                Literal[True]

        static serviceWorker(jobStore, terminate, error, inJobStoreID, outJobStoreID, messageInt)

            Parameters

                •   jobStore (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

                •   terminate (threading.Event </usr/share/doc/python3/html/library/threading.html#threading.Event>)

                •   error (threading.Event </usr/share/doc/python3/html/library/threading.html#threading.Event>)

                •   inJobStoreID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   outJobStoreID (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   messageInt (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                None

    toil.test.src.jobServiceTest.serviceAccessor(job, communicationFiles, outFile, to_subtract)
        Writes a random integer iinto the inJobStoreFileID file, then tries 10 times reading from outJobStoreFileID to get a pair of integers, the first equal to i the second written into the outputFile.

        Parameters

            •   job (toil.job.ServiceHostJob <#toil.job.ServiceHostJob>)

            •   communicationFiles (tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   outFile (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            •   to_subtract (int </usr/share/doc/python3/html/library/functions.html#int>)

        Return type
            None

    class toil.test.src.jobServiceTest.ToySerializableService(messageInt, *args, **kwargs)
        Bases: toil.job.Job.Service <#toil.job.Job.Service>

        Abstract class used to define the interface to a service.

        Should be subclassed by the user to define services.

        Is not executed as a job; runs within a ServiceHostJob.

        Parameters

            •   messageInt (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   args (Any)

            •   kwargs (Any)

        messageInt

        start(job)
            Start the service.

            Parameters
                job (toil.job.Job <#toil.job.Job>) -- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files.

            Returns
                An object describing how to access the service. The object must be pickleable and will be used by jobs to access the service (see toil.job.Job.addService() <#toil.job.Job.addService>).

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        stop(job)
            Stops the service. Function can block until complete.

            Parameters
                job (toil.job.Job <#toil.job.Job>) -- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files.

            Return type
                None

        check()
            Checks the service is still running.

            Raises
                exceptions.RuntimeError -- If the service failed, this will cause the service job to be labeled failed.

            Returns
                True if the service is still running, else False. If False then the service job will be terminated, and considered a success. Important point: if the service job exits due to a failure, it should raise a RuntimeError, not return False!

            Return type
                Literal[True]

    toil.test.src.jobServiceTest.fnTest(strings, outputFile)
        Function concatenates the strings together and writes them to the output file

        Parameters

            •   strings (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   outputFile (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            None

  toil.test.src.jobTest

  Classes
TestJob | Tests the job class.
TrivialService | Abstract class used to define the interface to a service.

  Functions
simpleJobFn(job, value) |
fn1Test(string, outputFile) | Function appends the next character after the last character in the given
fn2Test(pStrings, s, outputFile) | Function concatenates the strings in pStrings and s, in that order, and writes the result to
trivialParent(job) |
parent(job) |
diamond(job) |
child(job) |
errorChild(job) |

  Module Contents

    class toil.test.src.jobTest.TestJob
        Tests the job class.

        testStatic(tmp_path)
            Create a DAG of jobs non-dynamically and run it. DAG is:

                    A -> F
                    \-------
                    B -> D  \
                     \       \
                      ------- C -> E

        Follow on is marked by ->

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testStatic2(tmp_path)
            Create a DAG of jobs non-dynamically and run it. DAG is:

                    A -> F
                    \-------
                    B -> D  \
                     \       \
                      ------- C -> E

        Follow on is marked by ->

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testTrivialDAGConsistency(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDAGConsistency(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testSiblingDAGConsistency(tmp_path)
            Slightly more complex case. The stranded job's predecessors are siblings instead of parent/child.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testDeadlockDetection()
            Randomly generate job graphs with various types of cycle in them and check they cause an exception properly. Also check that multiple roots causes a deadlock exception.

            Return type
                None

        testNewCheckpointIsLeafVertexNonRootCase(tmp_path, subtests)
            Test for issue #1465: Detection of checkpoint jobs that are not leaf vertices identifies leaf vertices incorrectly

            Test verification of new checkpoint jobs being leaf vertices, starting with the following baseline workflow:

                    Parent
                      |
                    Child # Checkpoint=True

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   subtests (pytest.Subtests)

            Return type
                None

        testNewCheckpointIsLeafVertexRootCase(tmp_path, subtests)

            Test for issue #1466: Detection of checkpoint jobs that are not leaf vertices
                omits the workflow root job

        Test verification of a new checkpoint job being leaf vertex, starting with a baseline workflow of a single, root job:

                    Root # Checkpoint=True

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   subtests (pytest.Subtests)

            Return type
                None

        runNewCheckpointIsLeafVertexTest(tmp_path, subtests, createWorkflowFn)
            Test verification that a checkpoint job is a leaf vertex using both valid and invalid cases.

            Parameters

                •   createWorkflowFn (collections.abc.Callable </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Callable>[[], tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[toil.job.Job <#toil.job.Job>, toil.job.Job <#toil.job.Job>]]) --

                    function to create and new workflow and return a tuple of:

                    0.  the workflow root job

                    1.  a checkpoint job to test within the workflow

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   subtests (pytest.Subtests)

            Return type
                None

        runCheckpointVertexTest(workflowRootJob, checkpointJob, tmp_path, checkpointJobService=None, checkpointJobChild=None, checkpointJobFollowOn=None, expectedException=None)
            Modifies the checkpoint job according to the given parameters then runs the workflow, checking for the expected exception, if any.

            Parameters

                •   workflowRootJob (toil.job.Job <#toil.job.Job>)

                •   checkpointJob (toil.job.Job <#toil.job.Job>)

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   checkpointJobService (Optional[toil.job.Job.Service <#toil.job.Job.Service>])

                •   checkpointJobChild (Optional[toil.job.Job <#toil.job.Job>])

                •   checkpointJobFollowOn (Optional[toil.job.Job <#toil.job.Job>])

                •   expectedException (Optional[type </usr/share/doc/python3/html/library/functions.html#type>[Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>]])

            Return type
                None

        testEvaluatingRandomDAG(tmp_path)
            Randomly generate test input then check that the job graph can be run successfully, using the existence of promises to validate the run.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        static getRandomEdge(nodeNumber)

            Parameters
                nodeNumber (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[int </usr/share/doc/python3/html/library/functions.html#int>, int </usr/share/doc/python3/html/library/functions.html#int>]

        static makeRandomDAG(nodeNumber)
            Makes a random dag with "nodeNumber" nodes in which all nodes are connected. Return value is list of edges, each of form (a, b), where a and b are integers >= 0 < nodeNumber referring to nodes and the edge is from a to b.

            Parameters
                nodeNumber (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                set </usr/share/doc/python3/html/library/stdtypes.html#set>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[int </usr/share/doc/python3/html/library/functions.html#int>, int </usr/share/doc/python3/html/library/functions.html#int>]]

        static getAdjacencyList(nodeNumber, edges)
            Make adjacency list representation of edges

            Parameters

                •   nodeNumber (int </usr/share/doc/python3/html/library/functions.html#int>)

                •   edges (set </usr/share/doc/python3/html/library/stdtypes.html#set>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[int </usr/share/doc/python3/html/library/functions.html#int>, int </usr/share/doc/python3/html/library/functions.html#int>]])

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[set </usr/share/doc/python3/html/library/stdtypes.html#set>[int </usr/share/doc/python3/html/library/functions.html#int>]]

        reachable(node, adjacencyList)
            Find the set of nodes reachable from this node (including the node). Return is a set of integers.

            Parameters

                •   node (int </usr/share/doc/python3/html/library/functions.html#int>)

                •   adjacencyList (list </usr/share/doc/python3/html/library/stdtypes.html#list>[set </usr/share/doc/python3/html/library/stdtypes.html#set>[int </usr/share/doc/python3/html/library/functions.html#int>]])

            Return type
                set </usr/share/doc/python3/html/library/stdtypes.html#set>[int </usr/share/doc/python3/html/library/functions.html#int>]

        addRandomFollowOnEdges(childAdjacencyList)
            Adds random follow on edges to the graph, represented as an adjacency list. The follow on edges are returned as a set and their augmented edges are added to the adjacency list.

            Parameters
                childAdjacencyList (list </usr/share/doc/python3/html/library/stdtypes.html#list>[set </usr/share/doc/python3/html/library/stdtypes.html#set>[int </usr/share/doc/python3/html/library/functions.html#int>]])

            Return type
                set </usr/share/doc/python3/html/library/stdtypes.html#set>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[int </usr/share/doc/python3/html/library/functions.html#int>, int </usr/share/doc/python3/html/library/functions.html#int>]]

        makeJobGraph(nodeNumber, childEdges, followOnEdges, outPath, addServices=True)
            Converts a DAG into a job graph. childEdges and followOnEdges are the lists of child and followOn edges.

            Parameters

                •   nodeNumber (int </usr/share/doc/python3/html/library/functions.html#int>)

                •   childEdges (set </usr/share/doc/python3/html/library/stdtypes.html#set>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[int </usr/share/doc/python3/html/library/functions.html#int>, int </usr/share/doc/python3/html/library/functions.html#int>]])

                •   followOnEdges (set </usr/share/doc/python3/html/library/stdtypes.html#set>[tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[int </usr/share/doc/python3/html/library/functions.html#int>, int </usr/share/doc/python3/html/library/functions.html#int>]])

                •   outPath (Optional[pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>])

                •   addServices (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                toil.job.Job <#toil.job.Job>

        isAcyclic(adjacencyList)
            Returns true if there are any cycles in the graph, which is represented as an adjacency list.

            Parameters
                adjacencyList (list </usr/share/doc/python3/html/library/stdtypes.html#list>[set </usr/share/doc/python3/html/library/stdtypes.html#set>[int </usr/share/doc/python3/html/library/functions.html#int>]])

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.test.src.jobTest.simpleJobFn(job, value)

        Parameters

            •   job (toil.job.ServiceHostJob <#toil.job.ServiceHostJob>)

            •   value (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            None

    toil.test.src.jobTest.fn1Test(string, outputFile)
        Function appends the next character after the last character in the given string to the string, writes the string to a file, and returns it. For example, if string is "AB", we will write and return "ABC".

        Parameters

            •   string (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   outputFile (toil.lib.misc.FileDescriptorOrPath)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.test.src.jobTest.fn2Test(pStrings, s, outputFile)
        Function concatenates the strings in pStrings and s, in that order, and writes the result to the output file. Returns s.

        Parameters

            •   pStrings (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   s (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   outputFile (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.test.src.jobTest.trivialParent(job)

        Parameters
            job (toil.job.Job <#toil.job.Job>)

        Return type
            None

    toil.test.src.jobTest.parent(job)

        Parameters
            job (toil.job.Job <#toil.job.Job>)

        Return type
            None

    toil.test.src.jobTest.diamond(job)

        Parameters
            job (toil.job.Job <#toil.job.Job>)

        Return type
            None

    toil.test.src.jobTest.child(job)

        Parameters
            job (toil.job.Job <#toil.job.Job>)

        Return type
            None

    toil.test.src.jobTest.errorChild(job)

        Parameters
            job (toil.job.Job <#toil.job.Job>)

        Return type
            NoReturn

    class toil.test.src.jobTest.TrivialService(message, *args, **kwargs)
        Bases: toil.job.Job.Service <#toil.job.Job.Service>

        Abstract class used to define the interface to a service.

        Should be subclassed by the user to define services.

        Is not executed as a job; runs within a ServiceHostJob.

        Parameters

            •   message (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   args (Any)

            •   kwargs (Any)

        message

        start(job)
            Start the service.

            Parameters
                job (toil.job.ServiceHostJob <#toil.job.ServiceHostJob>) -- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files.

            Returns
                An object describing how to access the service. The object must be pickleable and will be used by jobs to access the service (see toil.job.Job.addService() <#toil.job.Job.addService>).

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        stop(job)
            Stops the service. Function can block until complete.

            Parameters
                job (toil.job.ServiceHostJob <#toil.job.ServiceHostJob>) -- The underlying host job that the service is being run in. Can be used to register deferred functions, or to access the fileStore for creating temporary files.

            Return type
                None

        check()
            Checks the service is still running.

            Raises
                exceptions.RuntimeError -- If the service failed, this will cause the service job to be labeled failed.

            Returns
                True if the service is still running, else False. If False then the service job will be terminated, and considered a success. Important point: if the service job exits due to a failure, it should raise a RuntimeError, not return False!

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

  toil.test.src.miscTests

  Attributes
log |

  Classes
TestMisc | This class contains miscellaneous tests that don't have enough content to be their own test
TestPanic |

  Module Contents

    toil.test.src.miscTests.log

    class toil.test.src.miscTests.TestMisc
        This class contains miscellaneous tests that don't have enough content to be their own test file, and that don't logically fit in with any of the other test suites.

        testIDStability()

            Return type
                None

        testGetSizeOfDirectoryWorks(tmp_path)
            A test to make sure toil.common.getDirSizeRecursively does not underestimate the amount of disk space needed.

            Disk space allocation varies from system to system. The computed value should always be equal to or slightly greater than the creation value. This test generates a number of random directories and randomly sized files to test this using getDirSizeRecursively.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_atomic_install(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_atomic_install_dev()

            Return type
                None

        test_atomic_context_ok(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_atomic_context_error(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_call_command_ok()

            Return type
                None

        test_call_command_err()

            Return type
                None

    class toil.test.src.miscTests.TestPanic

        test_panic_by_hand()

            Return type
                None

        test_panic()

            Return type
                None

        test_panic_with_secondary()

            Return type
                None

        test_nested_panic()

            Return type
                None

        try_and_panic_by_hand()

            Return type
                None

        try_and_panic()

            Return type
                None

        try_and_panic_with_secondary()

            Return type
                None

        try_and_nested_panic_with_secondary()

            Return type
                None

  toil.test.src.promisedRequirementTest

  Attributes
log |

  Classes
AbstractPromisedRequirementsTest | An abstract base class for testing Toil workflows with promised requirements.
TestSingleMachinePromisedRequirements | Tests against the SingleMachine batch system
TestMesosPromisedRequirements | Tests against the Mesos batch system

  Functions
maxConcurrency(job, cpuCount, filename, coresPerJob) | Returns the max number of concurrent tasks when using a PromisedRequirement instance
getOne() |
getThirtyTwoMb() |
logDiskUsage(job, funcName[, sleep]) | Logs the job's disk usage to master and sleeps for specified amount of time.

  Module Contents

    toil.test.src.promisedRequirementTest.log

    class toil.test.src.promisedRequirementTest.AbstractPromisedRequirementsTest
        Bases: toil.test.batchSystems.batchSystemTest.AbstractBatchSystemJobTest <#toil.test.batchSystems.batchSystemTest.AbstractBatchSystemJobTest>

        An abstract base class for testing Toil workflows with promised requirements.

        testConcurrencyDynamic(tmp_path)
            Asserts that promised core resources are allocated properly using a dynamic Toil workflow

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testConcurrencyStatic(tmp_path)
            Asserts that promised core resources are allocated properly using a static DAG

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        getOptions(tempDir, caching=True)
            Configures options for Toil workflow and makes job store. :param str tempDir: path to test directory :return: Toil options object

            Parameters

                •   tempDir (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   caching (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>

        getCounterPath(tempDir)
            Returns path to a counter file :param tempDir: path to test directory :return: path to counter file

            Parameters
                tempDir (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>

        testPromisesWithJobStoreFileObjects(tmp_path, caching=True)
            Check whether FileID objects are being pickled properly when used as return values of functions. Then ensure that lambdas of promised FileID objects can be used to describe the requirements of a subsequent job. This type of operation will be used commonly in Toil scripts. :return: None

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   caching (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                None

        testPromisesWithNonCachingFileStore(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testPromiseRequirementRaceStatic(tmp_path)
            Checks for a race condition when using promised requirements and child job functions.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

    toil.test.src.promisedRequirementTest.maxConcurrency(job, cpuCount, filename, coresPerJob)
        Returns the max number of concurrent tasks when using a PromisedRequirement instance to allocate the number of cores per job.

        Parameters

            •   cpuCount (int </usr/share/doc/python3/html/library/functions.html#int>) -- number of available cpus

            •   filename (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>) -- path to counter file

            •   coresPerJob (int </usr/share/doc/python3/html/library/functions.html#int>) -- number of cores assigned to each job

            •   job (toil.job.Job <#toil.job.Job>)

        Returns
            max concurrency value per CPU

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.job.Promise <#toil.job.Promise>]

    toil.test.src.promisedRequirementTest.getOne()

        Return type
            int </usr/share/doc/python3/html/library/functions.html#int>

    toil.test.src.promisedRequirementTest.getThirtyTwoMb()

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.test.src.promisedRequirementTest.logDiskUsage(job, funcName, sleep=0)
        Logs the job's disk usage to master and sleeps for specified amount of time.

        Returns
            job function's disk usage

        Parameters

            •   job (toil.job.JobFunctionWrappingJob <#toil.job.JobFunctionWrappingJob>)

            •   funcName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   sleep (int </usr/share/doc/python3/html/library/functions.html#int>)

        Return type
            int </usr/share/doc/python3/html/library/functions.html#int>

    class toil.test.src.promisedRequirementTest.TestSingleMachinePromisedRequirements
        Bases: AbstractPromisedRequirementsTest

        Tests against the SingleMachine batch system

        getBatchSystemName()

            Return type
                (str </usr/share/doc/python3/html/library/stdtypes.html#str>, AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>)

        tearDown()

            Return type
                None

    class toil.test.src.promisedRequirementTest.TestMesosPromisedRequirements
        Bases: AbstractPromisedRequirementsTest, toil.batchSystems.mesos.test.MesosTestSupport <#toil.batchSystems.mesos.test.MesosTestSupport>

        Tests against the Mesos batch system

        mesos_support()

            Return type
                collections.abc.Generator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator>[None]

        getOptions(tempDir, caching=True)
            Configures options for Toil workflow and makes job store. :param str tempDir: path to test directory :return: Toil options object

            Parameters

                •   tempDir (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   caching (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>

        getBatchSystemName()

            Return type
                (str </usr/share/doc/python3/html/library/stdtypes.html#str>, AbstractBatchSystem <#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem>)

  toil.test.src.promisesTest

  Classes
CachedUnpicklingJobStoreTest | Legacy common base class for Toil tests.
ChainedIndexedPromisesTest | Legacy common base class for Toil tests.
PathIndexingPromiseTest | Test support for indexing promises of arbitrarily nested data structures of lists, dicts and

  Functions
parent(job) |
child() |
a(job) |
b(job) |
c() |
d(job) |
e() |

  Module Contents

    class toil.test.src.promisesTest.CachedUnpicklingJobStoreTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        test()
            Runs two identical Toil workflows with different job store paths

    toil.test.src.promisesTest.parent(job)

    toil.test.src.promisesTest.child()

    class toil.test.src.promisesTest.ChainedIndexedPromisesTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        test()

    toil.test.src.promisesTest.a(job)

    toil.test.src.promisesTest.b(job)

    toil.test.src.promisesTest.c()

    class toil.test.src.promisesTest.PathIndexingPromiseTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Test support for indexing promises of arbitrarily nested data structures of lists, dicts and tuples, or any other object supporting the __getitem__() protocol.

        test()

    toil.test.src.promisesTest.d(job)

    toil.test.src.promisesTest.e()

  toil.test.src.realtimeLoggerTest

  Classes
RealtimeLoggerTest | Legacy common base class for Toil tests.
MessageDetector | Detect the secret message and set a flag.
LogTest | Class represents a unit of work in toil.

  Module Contents

    class toil.test.src.realtimeLoggerTest.RealtimeLoggerTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        testRealtimeLogger()

    class toil.test.src.realtimeLoggerTest.MessageDetector
        Bases: logging.StreamHandler </usr/share/doc/python3/html/library/logging.handlers.html#logging.StreamHandler>

        Detect the secret message and set a flag.

        detected = False

        overLogged = False

        emit(record)
            Emit a record.

            If a formatter is specified, it is used to format the record. The record is then written to the stream with a trailing newline. If exception information is present, it is formatted using traceback.print_exception and appended to the stream. If the stream has an 'encoding' attribute, it is used to determine how to do the output to the stream.

    class toil.test.src.realtimeLoggerTest.LogTest
        Bases: toil.job.Job <#toil.job.Job>

        Class represents a unit of work in toil.

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>.

  toil.test.src.regularLogTest

  Attributes
logger |

  Classes
RegularLogTest |

  Module Contents

    toil.test.src.regularLogTest.logger

    class toil.test.src.regularLogTest.RegularLogTest

        testLogToMaster()

            Return type
                None

        testWriteLogs(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testWriteGzipLogs(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testMultipleLogToMaster()

            Return type
                None

        testRegularLog()

            Return type
                None

  toil.test.src.resourceTest

  Classes
TestResource | Test module descriptors and resources derived from them.

  Functions
tempFileContaining(directory, content[, suffix]) | Write a file with the given contents, and keep it on disk as long as the context is active.

  Module Contents

    toil.test.src.resourceTest.tempFileContaining(directory, content, suffix='')
        Write a file with the given contents, and keep it on disk as long as the context is active. :param str content: The contents of the file. :param str suffix: The extension to use for the temporary file.

        Parameters

            •   directory (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            •   content (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   suffix (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    class toil.test.src.resourceTest.TestResource
        Test module descriptors and resources derived from them.

        testStandAlone(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testPackage(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testVirtualEnv(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testStandAloneInPackage(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testBuiltIn()

            Return type
                None

        testNonPyStandAlone(tmp_path)
            Asserts that Toil enforces the user script to have a .py or .pyc extension because that's the only way auto-deployment can re-import the module on a worker. See

            <https://github.com/BD2KGenomics/toil/issues/631> and <https://github.com/BD2KGenomics/toil/issues/858>

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

  toil.test.src.restartDAGTest

  Attributes
logger |

  Classes
TestRestartDAG | Tests that restarted job DAGs don't run children of jobs that failed in the first run till the

  Functions
passingFn(job[, file]) | This function is guaranteed to pass as it does nothing out of the ordinary.
failingFn(job, failType, file) | This function is guaranteed to fail via a raised assertion, or an os.kill

  Module Contents

    toil.test.src.restartDAGTest.logger

    class toil.test.src.restartDAGTest.TestRestartDAG
        Tests that restarted job DAGs don't run children of jobs that failed in the first run till the parent completes successfully in the restart.

        testRestartedWorkflowSchedulesCorrectJobsOnFailedParent(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testRestartedWorkflowSchedulesCorrectJobsOnKilledParent(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

    toil.test.src.restartDAGTest.passingFn(job, file=None)
        This function is guaranteed to pass as it does nothing out of the ordinary.

        If "file" is provided, it will be created.

        Parameters

            •   file (Optional[pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>]) -- The path of a file that must be created if provided.

            •   job (toil.job.Job <#toil.job.Job>)

        Return type
            None

    toil.test.src.restartDAGTest.failingFn(job, failType, file)
        This function is guaranteed to fail via a raised assertion, or an os.kill

        Parameters

            •   job (toil.job.Job <#toil.job.Job>) -- Job

            •   failType (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- 'raise' or 'kill

            •   file (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>) -- The path of a file that must be created.

        Return type
            None

  toil.test.src.resumabilityTest

  Classes
TestResumability | <https://github.com/BD2KGenomics/toil/issues/808>

  Functions
parent(job) | Set up a bunch of dummy child jobs, and a bad job that needs to be
chaining_parent(job) | Set up a failing job to chain to.
goodChild(job) | Does nothing.
badChild(job) | Fails the first time it's run, succeeds the second time.

  Module Contents

    class toil.test.src.resumabilityTest.TestResumability
        <https://github.com/BD2KGenomics/toil/issues/808>

        test(tmp_path)
            Tests that a toil workflow that fails once can be resumed without a NoSuchJobException.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_chaining(tmp_path)
            Tests that a job which is chained to and fails can resume and succeed.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

    toil.test.src.resumabilityTest.parent(job)
        Set up a bunch of dummy child jobs, and a bad job that needs to be restarted as the follow on.

        Parameters
            job (toil.job.Job <#toil.job.Job>)

        Return type
            None

    toil.test.src.resumabilityTest.chaining_parent(job)
        Set up a failing job to chain to.

        Parameters
            job (toil.job.Job <#toil.job.Job>)

        Return type
            None

    toil.test.src.resumabilityTest.goodChild(job)
        Does nothing.

        Parameters
            job (toil.job.Job <#toil.job.Job>)

        Return type
            None

    toil.test.src.resumabilityTest.badChild(job)
        Fails the first time it's run, succeeds the second time.

        Parameters
            job (toil.job.JobFunctionWrappingJob <#toil.job.JobFunctionWrappingJob>)

        Return type
            None

  toil.test.src.retainTempDirTest

  Classes
TestCleanWorkDir | Tests testing :class:toil.fileStores.abstractFileStore.AbstractFileStore

  Functions
tempFileTestJob(job) |
tempFileTestErrorJob(job) |

  Module Contents

    class toil.test.src.retainTempDirTest.TestCleanWorkDir
        Tests testing :class:toil.fileStores.abstractFileStore.AbstractFileStore

        testNever(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testAlways(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testOnErrorWithError(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testOnErrorWithNoError(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testOnSuccessWithError(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testOnSuccessWithSuccess(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

    toil.test.src.retainTempDirTest.tempFileTestJob(job)

        Parameters
            job (toil.job.JobFunctionWrappingJob <#toil.job.JobFunctionWrappingJob>)

        Return type
            None

    toil.test.src.retainTempDirTest.tempFileTestErrorJob(job)

        Parameters
            job (toil.job.JobFunctionWrappingJob <#toil.job.JobFunctionWrappingJob>)

        Return type
            None

  toil.test.src.systemTest

  Classes
TestSystem | Test various assumptions about the operating system's behavior.

  Module Contents

    class toil.test.src.systemTest.TestSystem
        Test various assumptions about the operating system's behavior.

        testAtomicityOfNonEmptyDirectoryRenames(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

  toil.test.src.threadingTest

  Attributes
log |

  Classes
TestThreading | Test Toil threading/synchronization tools.

  Module Contents

    toil.test.src.threadingTest.log

    class toil.test.src.threadingTest.TestThreading
        Test Toil threading/synchronization tools.

        testGlobalMutexOrdering(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testLastProcessStanding(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

  toil.test.src.toilContextManagerTest

  Classes
ToilContextManagerTest | Legacy common base class for Toil tests.
HelloWorld | Class represents a unit of work in toil.
FollowOn | Class represents a unit of work in toil.

  Functions
childFn(job) |

  Module Contents

    class toil.test.src.toilContextManagerTest.ToilContextManagerTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        setUp()
            Hook method for setting up the test fixture before exercising it.

        tearDown()
            Hook method for deconstructing the test fixture after testing it.

        testContextManger()

        testNoContextManger()

        testExportAfterFailedExport()

    class toil.test.src.toilContextManagerTest.HelloWorld
        Bases: toil.job.Job <#toil.job.Job>

        Class represents a unit of work in toil.

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>.

    toil.test.src.toilContextManagerTest.childFn(job)

    class toil.test.src.toilContextManagerTest.FollowOn(fileId)
        Bases: toil.job.Job <#toil.job.Job>

        Class represents a unit of work in toil.

        fileId

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>.

  toil.test.src.userDefinedJobArgTypeTest

  Classes
UserDefinedJobArgTypeTest | Test for issue #423 (Toil can't unpickle classes defined in user scripts) and variants
JobClass | Class represents a unit of work in toil.
Foo |

  Functions
jobFunction(job, level, foo) |
main() |

  Module Contents

    class toil.test.src.userDefinedJobArgTypeTest.UserDefinedJobArgTypeTest(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Test for issue #423 (Toil can't unpickle classes defined in user scripts) and variants thereof.

        <https://github.com/BD2KGenomics/toil/issues/423>

        setUp()
            Hook method for setting up the test fixture before exercising it.

        testJobFunction()
            Test with first job being a function

        testJobClass()
            Test with first job being an instance of a class

        testJobFunctionFromMain()
            Test with first job being a function defined in __main__

        testJobClassFromMain()
            Test with first job being an instance of a class defined in __main__

    class toil.test.src.userDefinedJobArgTypeTest.JobClass(level, foo)
        Bases: toil.job.Job <#toil.job.Job>

        Class represents a unit of work in toil.

        level

        foo

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>.

    toil.test.src.userDefinedJobArgTypeTest.jobFunction(job, level, foo)

    class toil.test.src.userDefinedJobArgTypeTest.Foo

        original_id

        assertIsCopy()

    toil.test.src.userDefinedJobArgTypeTest.main()

  toil.test.src.workerTest

  Classes
WorkerTests | Test miscellaneous units of the worker.

  Module Contents

    class toil.test.src.workerTest.WorkerTests(methodName='runTest')
        Bases: toil.test.ToilTest <#toil.test.ToilTest>

        Test miscellaneous units of the worker.

        setUp()
            Hook method for setting up the test fixture before exercising it.

        testNextChainable()
            Make sure chainable/non-chainable jobs are identified correctly.

  toil.test.utils

  Submodules

  toil.test.utils.toilDebugTest

  Attributes
logger |

  Classes
TestDebugJob | Test the toil debug-job command.

  Functions
workflow_debug_jobstore(tmp_path) |
testJobStoreContents(tmp_path) | Test toilDebugFile.printContentsOfJobStore().
fetchFiles(symLink, jobStoreDir, outputDir) | Fn for testFetchJobStoreFiles() and testFetchJobStoreFilesWSymlinks().
testFetchJobStoreFiles(tmp_path) | Test toilDebugFile.fetchJobStoreFiles() symlinks.

  Module Contents

    toil.test.utils.toilDebugTest.logger

    toil.test.utils.toilDebugTest.workflow_debug_jobstore(tmp_path)

        Parameters
            tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>

    toil.test.utils.toilDebugTest.testJobStoreContents(tmp_path)
        Test toilDebugFile.printContentsOfJobStore().

        Runs a workflow that imports 'B.txt' and 'mkFile.py' into the jobStore. 'A.txt', 'C.txt', 'ABC.txt' are then created. This checks to make sure these contents are found in the jobStore and printed.

        Parameters
            tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            None

    toil.test.utils.toilDebugTest.fetchFiles(symLink, jobStoreDir, outputDir)
        Fn for testFetchJobStoreFiles() and testFetchJobStoreFilesWSymlinks().

        Runs a workflow that imports 'B.txt' and 'mkFile.py' into the jobStore. 'A.txt', 'C.txt', 'ABC.txt' are then created. This test then attempts to get a list of these files and copy them over into our output diectory from the jobStore, confirm that they are present, and then delete them.

        Parameters

            •   symLink (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            •   jobStoreDir (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            •   outputDir (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            None

    toil.test.utils.toilDebugTest.testFetchJobStoreFiles(tmp_path)
        Test toilDebugFile.fetchJobStoreFiles() symlinks.

        Parameters
            tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            None

    class toil.test.utils.toilDebugTest.TestDebugJob
        Test the toil debug-job command.

        test_run_job(tmp_path)
            Make sure that we can use toil debug-job to try and run a job in-process.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_print_job_info(tmp_path)
            Make sure that we can use --printJobInfo to get information on a job from a job store.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_retrieve_task_directory(tmp_path)
            Make sure that we can use --retrieveTaskDirectory to get the input files for a job.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

  toil.test.utils.toilKillTest

  Attributes
logger |
pkg_root |

  Classes
TestToilKill | A set of test cases for "toil kill".
TestToilKillWithAWSJobStore | A set of test cases for "toil kill" using the AWS job store.

  Module Contents

    toil.test.utils.toilKillTest.logger

    toil.test.utils.toilKillTest.pkg_root = b'.'

    class toil.test.utils.toilKillTest.TestToilKill
        Bases: _ToilKillTest

        A set of test cases for "toil kill".

        test_cwl_toil_kill(tmp_path)
            Test "toil kill" on a CWL workflow with a 100 second sleep.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

    class toil.test.utils.toilKillTest.TestToilKillWithAWSJobStore
        Bases: _ToilKillTest

        A set of test cases for "toil kill" using the AWS job store.

        test_cwl_toil_kill()
            Test "toil kill" on a CWL workflow with a 100 second sleep.

            Return type
                None

  toil.test.utils.utilsTest

  Attributes
logger |

  Classes
TestUtils | Tests the utilities that toil ships with, e.g. stats and status, in conjunction with restart
RunTwoJobsPerWorker | Runs child job with same resources as self in an attempt to chain the jobs on the same worker

  Functions
unsortedFile(tmp_path) |
correctSort(unsortedFile) |
printUnicodeCharacter() |

  Module Contents

    toil.test.utils.utilsTest.logger

    toil.test.utils.utilsTest.unsortedFile(tmp_path)

        Parameters
            tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            collections.abc.Generator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator>[pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>]

    toil.test.utils.utilsTest.correctSort(unsortedFile)

        Parameters
            unsortedFile (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    class toil.test.utils.utilsTest.TestUtils
        Tests the utilities that toil ships with, e.g. stats and status, in conjunction with restart functionality.

        N: int </usr/share/doc/python3/html/library/functions.html#int> = 1000

        property toilMain: str </usr/share/doc/python3/html/library/stdtypes.html#str>

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        cleanCommand(jobstore)

            Parameters
                jobstore (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        statsCommand(jobstore)

            Parameters
                jobstore (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        statusCommand(jobstore, failIfNotComplete=False)

            Parameters

                •   jobstore (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   failIfNotComplete (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        test_config_functionality(tmp_path)
            Ensure that creating and reading back the config file works

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testAWSProvisionerUtils()
            Runs a number of the cluster utilities in sequence.

            Launches a cluster with custom tags. Verifies the tags exist. ssh's into the cluster. Does some weird string comparisons. Makes certain that TOIL_WORKDIR is set as expected in the ssh'ed cluster. Rsyncs a file and verifies it exists on the leader. Destroys the cluster.

            Returns

            Return type
                None

        testUtilsSort(tmp_path, unsortedFile, correctSort)
            Tests the status and stats commands of the toil command line utility using the sort example with the --restart flag.

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   unsortedFile (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   correctSort (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

        testUtilsStatsSort(tmp_path, unsortedFile, correctSort)
            Tests the stats commands on a complete run of the stats test.

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   unsortedFile (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   correctSort (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                None

        testUnicodeSupport(tmp_path)

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testMultipleJobsPerWorkerStats(tmp_path)
            Tests case where multiple jobs are run on 1 worker to ensure that all jobs report back their data

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        check_status(jobstore, status, status_fn, process=None, seconds=20)

            Parameters

                •   jobstore (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   status (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   status_fn (collections.abc.Callable </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Callable>[[str </usr/share/doc/python3/html/library/stdtypes.html#str>], str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   process (Optional[subprocess.Popen </usr/share/doc/python3/html/library/subprocess.html#subprocess.Popen>[Any]])

                •   seconds (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                None

        testGetPIDStatus(tmp_path, unsortedFile)
            Test that ToilStatus.getPIDStatus() behaves as expected.

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   unsortedFile (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testGetStatusFailedToilWF(tmp_path, unsortedFile)
            Test that ToilStatus.getStatus() behaves as expected with a failing Toil workflow. While this workflow could be called by importing and evoking its main function, doing so would remove the opportunity to test the 'RUNNING' functionality of getStatus().

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   unsortedFile (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testGetStatusFailedCWLWF(tmp_path)
            Test that ToilStatus.getStatus() behaves as expected with a failing CWL workflow.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testGetStatusSuccessfulCWLWF(tmp_path)
            Test that ToilStatus.getStatus() behaves as expected with a successful CWL workflow.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        testPrintJobLog(tmp_path, monkeypatch)
            Test that ToilStatus.printJobLog() reads the log from a failed command without error.

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   monkeypatch (pytest.MonkeyPatch)

            Return type
                None

        testRestartAttribute(tmp_path)
            Test that the job store is only destroyed when we observe a successful workflow run. The following simulates a failing workflow that attempts to resume without restart(). In this case, the job store should not be destroyed until restart() is called.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

    toil.test.utils.utilsTest.printUnicodeCharacter()

        Return type
            None

    class toil.test.utils.utilsTest.RunTwoJobsPerWorker(memory=None, cores=None, disk=None, accelerators=None, preemptible=None, preemptable=None, unitName='', checkpoint=False, displayName='', descriptionClass=None, local=None, files=None)
        Bases: toil.job.Job <#toil.job.Job>

        Runs child job with same resources as self in an attempt to chain the jobs on the same worker

        Parameters

            •   memory (Optional[ParseableIndivisibleResource])

            •   cores (Optional[ParseableDivisibleResource])

            •   disk (Optional[ParseableIndivisibleResource])

            •   accelerators (Optional[ParseableAcceleratorRequirement])

            •   preemptible (Optional[ParseableFlag])

            •   preemptable (Optional[ParseableFlag])

            •   unitName (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   checkpoint (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>])

            •   displayName (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   descriptionClass (Optional[type </usr/share/doc/python3/html/library/functions.html#type>])

            •   local (Optional[bool </usr/share/doc/python3/html/library/functions.html#bool>])

            •   files (Optional[set </usr/share/doc/python3/html/library/stdtypes.html#set>[toil.fileStores.FileID <#toil.fileStores.FileID>]])

        run(fileStore)
            Override this function to perform work and dynamically create successor jobs.

            Parameters
                fileStore (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>) -- Used to create local and globally sharable temporary files and to send log messages to the leader process.

            Returns
                The return value of the function can be passed to other jobs by means of toil.job.Job.rv() <#toil.job.Job.rv>.

            Return type
                None

  toil.test.wdl

  Submodules

  toil.test.wdl.wdltoil_test

  Attributes
logger |
WDL_CONFORMANCE_TEST_REPO |
WDL_CONFORMANCE_TEST_COMMIT |
WDL_CONFORMANCE_TESTS_UNSUPPORTED_BY_TOIL |
WDL_UNIT_TESTS_UNSUPPORTED_BY_TOIL |

  Classes
TestWDLConformance | WDL conformance tests for Toil.
TestWDL | Tests for Toil's MiniWDL-based implementation.
TestWDLToilBench | Tests for Toil's MiniWDL-based implementation that don't run workflows.

  Functions
wdl_conformance_test_repo(tmp_path) |

  Module Contents

    toil.test.wdl.wdltoil_test.logger

    toil.test.wdl.wdltoil_test.WDL_CONFORMANCE_TEST_REPO = 'https://github.com/DataBiosphere/wdl-conformance-tests.git'

    toil.test.wdl.wdltoil_test.WDL_CONFORMANCE_TEST_COMMIT = '46b5f85ee38ec60d0b8b9c35928b5104a2af83d5'

    toil.test.wdl.wdltoil_test.WDL_CONFORMANCE_TESTS_UNSUPPORTED_BY_TOIL = [16, 21, 64, 77]

    toil.test.wdl.wdltoil_test.WDL_UNIT_TESTS_UNSUPPORTED_BY_TOIL = [14, 19, 52, 58, 59, 66, 67, 68, 69, 87, 97, 105, 107, 108, 109, 110, 120, 131, 134, 144]

    toil.test.wdl.wdltoil_test.wdl_conformance_test_repo(tmp_path)

        Parameters
            tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

        Return type
            collections.abc.Generator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator>[pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>]

    class toil.test.wdl.wdltoil_test.TestWDLConformance
        WDL conformance tests for Toil.

        check(p)
            Make sure a call completed or explain why it failed.

            Parameters
                p (subprocess.CompletedProcess </usr/share/doc/python3/html/library/subprocess.html#subprocess.CompletedProcess>[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>])

            Return type
                None

        test_unit_tests_v11(wdl_conformance_test_repo)

            Parameters
                wdl_conformance_test_repo (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_conformance_tests_v10(wdl_conformance_test_repo)

            Parameters
                wdl_conformance_test_repo (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_conformance_tests_v11(wdl_conformance_test_repo)

            Parameters
                wdl_conformance_test_repo (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_conformance_tests_development(wdl_conformance_test_repo)

            Parameters
                wdl_conformance_test_repo (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_conformance_tests_integration(wdl_conformance_test_repo)

            Parameters
                wdl_conformance_test_repo (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

    class toil.test.wdl.wdltoil_test.TestWDL
        Tests for Toil's MiniWDL-based implementation.

        base_command

        test_MD5sum(tmp_path)
            Test if Toil produces the same outputs as known good outputs for WDL's GATK tutorial #1.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_url_to_file(tmp_path)
            Test if web URL strings can be coerced to usable Files.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_string_file_coercion(tmp_path)
            Test if input Files can be coerced to string and back.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_gather(tmp_path)
            Test files with the same name from different scatter tasks.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_wait(tmp_path)
            Test if Bash "wait" works in WDL scripts.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_restart(tmp_path)
            Test if a WDL workflow can be restarted and finish successfully.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_workflow_file_deletion(tmp_path)
            Test if Toil can delete non-output outputs at the end of a workflow.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_all_call_outputs(tmp_path)
            Test if Toil can collect all call outputs from a workflow that doesn't expose them.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_croo_detection(tmp_path)
            Test if Toil can detect and do something sensible with Cromwell Output Organizer workflows.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_caching(tmp_path)
            Test if Toil can cache task runs.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_url_to_optional_file(tmp_path, httpserver)
            Test if missing and error-producing URLs are handled correctly for optional File? values.

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   httpserver (pytest_httpserver.HTTPServer)

            Return type
                None

        test_missing_output_directory(tmp_path)
            Test if Toil can run a WDL workflow into a new directory.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_miniwdl_self_test(tmp_path, extra_args=None)
            Test if the MiniWDL self test runs and produces the expected output.

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   extra_args (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

            Return type
                None

        test_miniwdl_self_test_by_reference(tmp_path)
            Test if the MiniWDL self test works when passing input files by URL reference.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_dockstore_trs(tmp_path, extra_args=None)

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   extra_args (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

            Return type
                None

        test_dockstore_metrics_publication(tmp_path, extra_args=None)

            Parameters

                •   tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

                •   extra_args (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

            Return type
                None

        test_giraffe_deepvariant(tmp_path)
            Test if Giraffe and GPU DeepVariant run. This could take 25 minutes.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_giraffe(tmp_path)
            Test if Giraffe runs. This could take 12 minutes. Also we scale it down but it still demands lots of memory.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_gs_uri(tmp_path)
            Test if Toil can access Google Storage URIs.

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

        test_check(tmp_path)
            Test that Toil's lint check works

            Parameters
                tmp_path (pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>)

            Return type
                None

    class toil.test.wdl.wdltoil_test.TestWDLToilBench(methodName='runTest')
        Bases: unittest.TestCase </usr/share/doc/python3/html/library/unittest.html#unittest.TestCase>

        Tests for Toil's MiniWDL-based implementation that don't run workflows.

        test_coalesce()
            Test if WDLSectionJob can coalesce WDL decls.

            White box test; will need to be changed or removed if the WDL interpreter changes.

            Return type
                None

        make_string_expr(to_parse, expr_type=WDL.Expr.String)
            Parse pseudo-WDL for testing whitespace removal.

            Parameters

                •   to_parse (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   expr_type (type </usr/share/doc/python3/html/library/functions.html#type>[WDL.Expr.String])

            Return type
                WDL.Expr.String

        test_choose_human_readable_directory()
            Test to make sure that we pick sensible but non-colliding directories to put files in.

            Return type
                None

        test_uri_packing()
            Test to make sure Toil URI packing brings through the required information.

            Return type
                None

        test_disk_parse()
            Test to make sure the disk parsing is correct

            Return type
                None

  toil.test.wdl.wdltoil_test_kubernetes

  Attributes
logger |

  Classes
WDLKubernetesClusterTest | Ensure WDL works on the Kubernetes batchsystem.

  Module Contents

    toil.test.wdl.wdltoil_test_kubernetes.logger

    class toil.test.wdl.wdltoil_test_kubernetes.WDLKubernetesClusterTest(name)
        Bases: toil.test.provisioners.clusterTest.AbstractClusterTest <#toil.test.provisioners.clusterTest.AbstractClusterTest>

        Ensure WDL works on the Kubernetes batchsystem.

        Parameters
            name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        clusterName = 'wdl-integration-test-'

        leaderNodeType = 't2.medium'

        instanceTypes = ['t2.medium']

        clusterType = 'kubernetes'

        setUp()
            Set up for the test. Must be overridden to call this method and set self.jobStore.

            Return type
                None

        launchCluster()

            Return type
                None

        test_wdl_kubernetes_cluster()
            Test that a wdl workflow works on a kubernetes cluster. Launches a cluster with 1 worker. This runs a wdl workflow that performs an image pull on the worker. :return:

            Return type
                None

  Attributes
logger |
MT |
pneeds_rsync3 |
pneeds_online |
pneeds_aws_s3 |
pneeds_aws_ec2 |
pneeds_gridengine |
pneeds_torque |
pneeds_kubernetes |
pneeds_mesos |
pneeds_slurm |
pneeds_lsf |
pneeds_docker |
pneeds_local_cuda |
pneeds_docker_cuda |
pneeds_cwl |
pneeds_wdl |
pneeds_wes_server |
pintegrative |
pslow |
methodNamePartRegex |

  Exceptions
ProxyConnectionError | Common base class for all exceptions

  Classes
ToilTest | Legacy common base class for Toil tests.
ApplianceTestSupport | A Toil test that runs a user script on a minimal cluster of appliance containers.

  Functions
get_data(filename) | Returns an absolute path for a file from this package.
get_temp_file([suffix, rootDir]) | Return a string representing a temporary file, that must be manually deleted.
needs_env_var(var_name[, comment]) | Use as a decorator before test classes or methods to run only if the given
needs_rsync3(test_item) | Decorate classes or methods that depend on any features from rsync version 3.0.0+.
needs_online(test_item) | Use as a decorator before test classes or methods to run only if we are meant to talk to the Internet.
needs_aws_s3(test_item) | Use as a decorator before test classes or methods to run only if AWS S3 is usable.
needs_aws_ec2(test_item) | Use as a decorator before test classes or methods to run only if AWS EC2 is usable.
needs_aws_batch(test_item) | Use as a decorator before test classes or methods to run only if AWS Batch
needs_google_storage(test_item) | Use as a decorator before test classes or methods to run only if Google
needs_google_project(test_item) | Use as a decorator before test classes or methods to run only if we have a Google Cloud project set.
needs_gridengine(test_item) | Use as a decorator before test classes or methods to run only if GridEngine is installed.
needs_torque(test_item) | Use as a decorator before test classes or methods to run only if PBS/Torque is installed.
needs_kubernetes_installed(test_item) | Use as a decorator before test classes or methods to run only if Kubernetes is installed.
needs_kubernetes(test_item) | Use as a decorator before test classes or methods to run only if Kubernetes is installed and configured.
needs_mesos(test_item) | Use as a decorator before test classes or methods to run only if Mesos is installed.
needs_slurm(test_item) | Use as a decorator before test classes or methods to run only if Slurm is installed.
needs_htcondor(test_item) | Use a decorator before test classes or methods to run only if the HTCondor is installed.
needs_lsf(test_item) | Use as a decorator before test classes or methods to only run them if LSF is installed.
needs_java(test_item) | Use as a test decorator to run only if java is installed.
needs_docker(test_item) | Use as a decorator before test classes or methods to only run them if
needs_singularity(test_item) | Use as a decorator before test classes or methods to only run them if
needs_singularity_or_docker(test_item) | Use as a decorator before test classes or methods to only run them if
needs_local_cuda(test_item) | Use as a decorator before test classes or methods to only run them if
needs_docker_cuda(test_item) | Use as a decorator before test classes or methods to only run them if
needs_encryption(test_item) | Use as a decorator before test classes or methods to only run them if PyNaCl is installed
needs_cwl(test_item) | Use as a decorator before test classes or methods to only run them if CWLTool is installed
needs_wdl(test_item) | Use as a decorator before test classes or methods to only run them if miniwdl is installed
needs_server(test_item) | Use as a decorator before test classes or methods to only run them if Connexion is installed.
needs_celery_broker(test_item) | Use as a decorator before test classes or methods to run only if RabbitMQ is set up to take Celery jobs.
needs_wes_server(test_item) | Use as a decorator before test classes or methods to run only if a WES
needs_local_appliance(test_item) | Use as a decorator before test classes or methods to only run them if
needs_fetchable_appliance(test_item) | Use as a decorator before test classes or methods to only run them if
integrative(test_item) | Use this to decorate integration tests so as to skip them during regular builds.
slow(test_item) | Use this decorator to identify tests that are slow and not critical.
timeLimit(seconds) | Use to limit the execution time of a function.
make_tests(generalMethod, targetClass, **kwargs) | This method dynamically generates test methods using the generalMethod as a template. Each

  Package Contents

    exception toil.test.ProxyConnectionError
        Bases: BaseException </usr/share/doc/python3/html/library/exceptions.html#BaseException>

        Common base class for all exceptions

    toil.test.logger

    toil.test.get_data(filename)
        Returns an absolute path for a file from this package.

        Parameters
            filename (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            contextlib.AbstractContextManager </usr/share/doc/python3/html/library/contextlib.html#contextlib.AbstractContextManager>[pathlib.Path </usr/share/doc/python3/html/library/pathlib.html#pathlib.Path>]

    class toil.test.ToilTest(methodName='runTest')
        Bases: unittest.TestCase </usr/share/doc/python3/html/library/unittest.html#unittest.TestCase>

        Legacy common base class for Toil tests.

        New tests should be made in the 'pytest' style and not use this class.

        When running tests you may optionally set the TOIL_TEST_TEMP environment variable to the path of a directory where you want temporary test files be placed. The directory will be created if it doesn't exist. The path may be relative in which case it will be assumed to be relative to the project root. If TOIL_TEST_TEMP is not defined, temporary files and directories will be created in the system's default location for such files and any temporary files or directories left over from tests will be removed automatically removed during tear down. Otherwise, left-over files will not be removed.

        setup_method(method)

            Parameters
                method (Any)

            Return type
                None

        classmethod setUpClass()
            Hook method for setting up class fixture before running tests in the class.

            Return type
                None

        classmethod tearDownClass()
            Hook method for deconstructing the class fixture after running all tests in the class.

            Return type
                None

        setUp()
            Hook method for setting up the test fixture before exercising it.

            Return type
                None

        tearDown()
            Hook method for deconstructing the test fixture after testing it.

            Return type
                None

        classmethod awsRegion()
            Pick an appropriate AWS region.

            Use us-west-2 unless running on EC2, in which case use the region in which the instance is located

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.test.MT

    toil.test.get_temp_file(suffix='', rootDir=None)
        Return a string representing a temporary file, that must be manually deleted.

        Parameters

            •   suffix (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   rootDir (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.test.needs_env_var(var_name, comment=None)
        Use as a decorator before test classes or methods to run only if the given environment variable is set. Can include a comment saying what the variable should be set to.

        Parameters

            •   var_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   comment (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            Callable[[MT], MT]

    toil.test.needs_rsync3(test_item)
        Decorate classes or methods that depend on any features from rsync version 3.0.0+.

        Necessary because utilsTest.testAWSProvisionerUtils() uses option --protect-args which is only available in rsync 3

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.pneeds_rsync3

    toil.test.needs_online(test_item)
        Use as a decorator before test classes or methods to run only if we are meant to talk to the Internet.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.pneeds_online

    toil.test.needs_aws_s3(test_item)
        Use as a decorator before test classes or methods to run only if AWS S3 is usable.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.pneeds_aws_s3

    toil.test.needs_aws_ec2(test_item)
        Use as a decorator before test classes or methods to run only if AWS EC2 is usable.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.pneeds_aws_ec2

    toil.test.needs_aws_batch(test_item)
        Use as a decorator before test classes or methods to run only if AWS Batch is usable.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.needs_google_storage(test_item)
        Use as a decorator before test classes or methods to run only if Google Cloud is installed and we ought to be able to access public Google Storage URIs.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.needs_google_project(test_item)
        Use as a decorator before test classes or methods to run only if we have a Google Cloud project set.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.needs_gridengine(test_item)
        Use as a decorator before test classes or methods to run only if GridEngine is installed.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.pneeds_gridengine

    toil.test.needs_torque(test_item)
        Use as a decorator before test classes or methods to run only if PBS/Torque is installed.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.pneeds_torque

    toil.test.needs_kubernetes_installed(test_item)
        Use as a decorator before test classes or methods to run only if Kubernetes is installed.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.needs_kubernetes(test_item)
        Use as a decorator before test classes or methods to run only if Kubernetes is installed and configured.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.pneeds_kubernetes

    toil.test.needs_mesos(test_item)
        Use as a decorator before test classes or methods to run only if Mesos is installed.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.pneeds_mesos

    toil.test.needs_slurm(test_item)
        Use as a decorator before test classes or methods to run only if Slurm is installed.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.pneeds_slurm

    toil.test.needs_htcondor(test_item)
        Use a decorator before test classes or methods to run only if the HTCondor is installed.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.needs_lsf(test_item)
        Use as a decorator before test classes or methods to only run them if LSF is installed.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.pneeds_lsf

    toil.test.needs_java(test_item)
        Use as a test decorator to run only if java is installed.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.needs_docker(test_item)
        Use as a decorator before test classes or methods to only run them if docker is installed and docker-based tests are enabled.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.pneeds_docker

    toil.test.needs_singularity(test_item)
        Use as a decorator before test classes or methods to only run them if singularity is installed.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.needs_singularity_or_docker(test_item)
        Use as a decorator before test classes or methods to only run them if docker is installed and docker-based tests are enabled, or if Singularity is installed.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.needs_local_cuda(test_item)
        Use as a decorator before test classes or methods to only run them if a CUDA setup legible to cwltool (i.e. providing userspace nvidia-smi) is present.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.pneeds_local_cuda

    toil.test.needs_docker_cuda(test_item)
        Use as a decorator before test classes or methods to only run them if a CUDA setup is available through Docker.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.pneeds_docker_cuda

    toil.test.needs_encryption(test_item)
        Use as a decorator before test classes or methods to only run them if PyNaCl is installed and configured.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.needs_cwl(test_item)
        Use as a decorator before test classes or methods to only run them if CWLTool is installed and configured.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.pneeds_cwl

    toil.test.needs_wdl(test_item)
        Use as a decorator before test classes or methods to only run them if miniwdl is installed and configured.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.pneeds_wdl

    toil.test.needs_server(test_item)
        Use as a decorator before test classes or methods to only run them if Connexion is installed.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.needs_celery_broker(test_item)
        Use as a decorator before test classes or methods to run only if RabbitMQ is set up to take Celery jobs.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.needs_wes_server(test_item)
        Use as a decorator before test classes or methods to run only if a WES server is available to run against.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.pneeds_wes_server

    toil.test.needs_local_appliance(test_item)
        Use as a decorator before test classes or methods to only run them if the Toil appliance Docker image is downloaded.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.needs_fetchable_appliance(test_item)
        Use as a decorator before test classes or methods to only run them if the Toil appliance Docker image is able to be downloaded from the Internet.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.integrative(test_item)
        Use this to decorate integration tests so as to skip them during regular builds.

        We define integration tests as A) involving other, non-Toil software components that we develop and/or B) having a higher cost (time or money).

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.pintegrative

    toil.test.slow(test_item)
        Use this decorator to identify tests that are slow and not critical. Skip if TOIL_TEST_QUICK is true.

        Parameters
            test_item (MT)

        Return type
            MT

    toil.test.pslow

    toil.test.methodNamePartRegex

    toil.test.timeLimit(seconds)
        Use to limit the execution time of a function.

        Raises an exception if the execution of the function takes more than the specified amount of time. See <<http://stackoverflow.com/a/601168>>.

        Parameters
            seconds (int </usr/share/doc/python3/html/library/functions.html#int>) -- maximum allowable time, in seconds

        Return type
            collections.abc.Generator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Generator>[None, None, None]

        >>> import time
        >>> with timeLimit(2):
        ...    time.sleep(1)
        >>> import time
        >>> with timeLimit(1):
        ...    time.sleep(2)
        Traceback (most recent call last):
            ...
        RuntimeError: Timed out

    toil.test.make_tests(generalMethod, targetClass, **kwargs)
        This method dynamically generates test methods using the generalMethod as a template. Each generated function is the result of a unique combination of parameters applied to the generalMethod. Each of the parameters has a corresponding string that will be used to name the method. These generated functions are named in the scheme: test_[generalMethodName]___[ firstParamaterName]_[someValueName]__[secondParamaterName]_...

        The arguments following the generalMethodName should be a series of one or more dictionaries of the form {str : type, ...} where the key represents the name of the value. The names will be used to represent the permutation of values passed for each parameter in the generalMethod.

        The generated method names will list the parameters in lexicographic order by parameter name.

        Parameters

            •   generalMethod (Callable[[Any], Any]) -- A method that will be parameterized with values passed as kwargs. Note that the generalMethod must be a regular method.

            •   targetClass (Optional[Callable[[Any], Any]]) -- This represents the class to which the generated test methods will be bound. If no targetClass is specified the class of the generalMethod is assumed the target.

            •   kwargs (Any) -- a series of dictionaries defining values, and their respective names where each keyword is the name of a parameter in generalMethod.

        Return type
            None

        >>> class Foo:
        ...     def has(self, num, letter):
        ...         return num, letter
        ...
        ...     def hasOne(self, num):
        ...         return num

        >>> class Bar(Foo):
        ...     pass

        >>> make_tests(Foo.has, Bar, num={'one':1, 'two':2}, letter={'a':'a', 'b':'b'})

        >>> b = Bar()

    Note that num comes lexicographically before letter and so appears first in the generated method names.

        >>> assert b.test_has__letter_a__num_one() == b.has(1, 'a')

        >>> assert b.test_has__letter_b__num_one() == b.has(1, 'b')

        >>> assert b.test_has__letter_a__num_two() == b.has(2, 'a')

        >>> assert b.test_has__letter_b__num_two() == b.has(2, 'b')

        >>> f = Foo()

        >>> hasattr(f, 'test_has__num_one__letter_a')  # should be false because Foo has no test methods
        False

    class toil.test.ApplianceTestSupport(methodName='runTest')
        Bases: ToilTest

        A Toil test that runs a user script on a minimal cluster of appliance containers.

        i.e. one leader container and one worker container.

        class Appliance(outer, mounts, cleanMounts=False)
            Bases: toil.lib.threading.ExceptionalThread <#toil.lib.threading.ExceptionalThread>

            A thread whose join() method re-raises exceptions raised during run(). While join() is idempotent, the exception is only during the first invocation of join() that successfully joined the thread. If join() times out, no exception will be re reraised even though an exception might already have occurred in run().

            When subclassing this thread, override tryRun() instead of run().

                >>> def f():
                ...     assert 0
                >>> t = ExceptionalThread(target=f)
                >>> t.start()
                >>> t.join()
                Traceback (most recent call last):
                ...
                AssertionError

                >>> class MyThread(ExceptionalThread):
                ...     def tryRun( self ):
                ...         assert 0
                >>> t = MyThread()
                >>> t.start()
                >>> t.join()
                Traceback (most recent call last):
                ...
                AssertionError

            Parameters

                •   outer (ApplianceTestSupport <#toil.test.ApplianceTestSupport>)

                •   mounts (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   cleanMounts (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            lock

            outer

            mounts

            cleanMounts = False

            containerName = ''

            popen: subprocess.Popen </usr/share/doc/python3/html/library/subprocess.html#subprocess.Popen>[bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>] | None </usr/share/doc/python3/html/library/constants.html#None> = None

            __enter__()

                Return type
                    typing_extensions.Self

            __exit__(exc_type, exc_val, exc_tb)

                Parameters

                    •   exc_type (type </usr/share/doc/python3/html/library/functions.html#type>[BaseException </usr/share/doc/python3/html/library/exceptions.html#BaseException>])

                    •   exc_val (Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>)

                    •   exc_tb (Any)

                Return type
                    Literal[False]

            tryRun()

                Return type
                    None

            runOnAppliance(*args, **kwargs)

                Parameters

                    •   args (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                    •   kwargs (Any)

                Return type
                    None

            writeToAppliance(path, contents)

                Parameters

                    •   path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                    •   contents (Any)

                Return type
                    None

            deployScript(path, packagePath, script)
                Deploy a Python module on the appliance.

                Parameters

                    •   path (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the path (absolute or relative to the WORDIR of the appliance container) to the root of the package hierarchy where the given module should be placed. The given directory should be on the Python path.

                    •   packagePath (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the desired fully qualified module name (dotted form) of the module

                    •   script (str </usr/share/doc/python3/html/library/stdtypes.html#str>|callable) -- the contents of the Python module. If a callable is given, its source code will be extracted. This is a convenience that lets you embed user scripts into test code as nested function.

                Return type
                    None

        class LeaderThread(outer, mounts, cleanMounts=False)
            Bases: Appliance

            A thread whose join() method re-raises exceptions raised during run(). While join() is idempotent, the exception is only during the first invocation of join() that successfully joined the thread. If join() times out, no exception will be re reraised even though an exception might already have occurred in run().

            When subclassing this thread, override tryRun() instead of run().

                >>> def f():
                ...     assert 0
                >>> t = ExceptionalThread(target=f)
                >>> t.start()
                >>> t.join()
                Traceback (most recent call last):
                ...
                AssertionError

                >>> class MyThread(ExceptionalThread):
                ...     def tryRun( self ):
                ...         assert 0
                >>> t = MyThread()
                >>> t.start()
                >>> t.join()
                Traceback (most recent call last):
                ...
                AssertionError

            Parameters

                •   outer (ApplianceTestSupport <#toil.test.ApplianceTestSupport>)

                •   mounts (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   cleanMounts (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        class WorkerThread(outer, mounts, numCores)
            Bases: Appliance

            A thread whose join() method re-raises exceptions raised during run(). While join() is idempotent, the exception is only during the first invocation of join() that successfully joined the thread. If join() times out, no exception will be re reraised even though an exception might already have occurred in run().

            When subclassing this thread, override tryRun() instead of run().

                >>> def f():
                ...     assert 0
                >>> t = ExceptionalThread(target=f)
                >>> t.start()
                >>> t.join()
                Traceback (most recent call last):
                ...
                AssertionError

                >>> class MyThread(ExceptionalThread):
                ...     def tryRun( self ):
                ...         assert 0
                >>> t = MyThread()
                >>> t.start()
                >>> t.join()
                Traceback (most recent call last):
                ...
                AssertionError

            Parameters

                •   outer (ApplianceTestSupport <#toil.test.ApplianceTestSupport>)

                •   mounts (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   numCores (int </usr/share/doc/python3/html/library/functions.html#int>)

            numCores

  toil.toilState

  Attributes
logger |

  Classes
ToilState | Holds the leader's scheduling information.

  Module Contents

    toil.toilState.logger

    class toil.toilState.ToilState(jobStore)
        Holds the leader's scheduling information.

        But onlt that which does not need to be persisted back to the JobStore (such as information on completed and outstanding predecessors)

        Holds the true single copies of all JobDescription objects that the Leader and ServiceManager will use. The leader and service manager shouldn't do their own load() and update() calls on the JobStore; they should go through this class.

        Everything in the leader should reference JobDescriptions by ID.

        Only holds JobDescription objects, not Job objects, and those JobDescription objects only exist in single copies.

        Parameters
            jobStore (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

        bus

        successor_to_predecessors: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

        successorCounts: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, int </usr/share/doc/python3/html/library/functions.html#int>]

        service_to_client: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        servicesIssued: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

        jobs_issued: set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        totalFailedJobs: set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        hasFailedSuccessors: set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        failedSuccessors: set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        jobsToBeScheduledWithMultiplePredecessors: set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        load_workflow(rootJob, jobCache=None)
            Load the workflow rooted at the given job.

            If jobs are loaded that have updated and need to be dealt with by the leader, JobUpdatedMessage messages will be sent to the message bus.

            The jobCache is a map from jobStoreID to JobDescription or None. Is used to speed up the building of the state when loading initially from the JobStore, and is not preserved.

            Parameters

                •   rootJob (toil.job.JobDescription <#toil.job.JobDescription>) -- The description for the root job of the workflow being run.

                •   jobCache (Optional[dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.job.JobDescription <#toil.job.JobDescription>]]) -- A dict to cache downloaded job descriptions in, keyed by ID.

            Return type
                None

        job_exists(job_id)
            Test if the givin job exists now.

            Returns True if the given job exists right now, and false if it hasn't been created or it has been deleted elsewhere.

            Doesn't guarantee that the job will or will not be gettable, if racing another process, or if it is still cached.

            Parameters
                job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        get_job(job_id)
            Get the one true copy of the JobDescription with the given ID.

            Parameters
                job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                toil.job.JobDescription <#toil.job.JobDescription>

        commit_job(job_id)
            Save back any modifications made to a JobDescription.

            (one retrieved from get_job())

            Parameters
                job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        delete_job(job_id)
            Destroy a JobDescription.

            May raise an exception if the job could not be cleaned up (i.e. files belonging to it failed to delete).

            Parameters
                job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        reset_job(job_id)
            Discard any local modifications to a JobDescription.

            Will make modifications from other hosts visible.

            Parameters
                job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        reset_job_expecting_change(job_id, timeout)
            Discard any local modifications to a JobDescription.

            Will make modifications from other hosts visible.

            Will wait for up to timeout seconds for a modification (or deletion) from another host to actually be visible.

            Always replaces the JobDescription with what is stored in the job store, even if no modification ends up being visible.

            Returns True if an update was detected in time, and False otherwise.

            Parameters

                •   job_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   timeout (float </usr/share/doc/python3/html/library/functions.html#float>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        successors_pending(predecessor_id, count)
            Remember that the given job has the given number more pending successors.

            (that have not yet succeeded or failed.)

            Parameters

                •   predecessor_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   count (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                None

        successor_returned(predecessor_id)
            Remember that the given job has one fewer pending successors.

            (because one has succeeded or failed.)

            Parameters
                predecessor_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        count_pending_successors(predecessor_id)
            Count number of pending successors of the given job.

            Pending successors are those which have not yet succeeded or failed.

            Parameters
                predecessor_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

  toil.utils

  Submodules

  toil.utils.toilClean

Delete a job store used by a previous Toil workflow invocation.

  Attributes
logger |

  Functions
main() |

  Module Contents

    toil.utils.toilClean.logger

    toil.utils.toilClean.main()

        Return type
            None

  toil.utils.toilConfig

Create a config file with all default Toil options.

  Attributes
logger |

  Functions
main() |

  Module Contents

    toil.utils.toilConfig.logger

    toil.utils.toilConfig.main()

        Return type
            None

  toil.utils.toilDebugFile

Debug tool for copying files contained in a toil jobStore.

  Attributes
logger |

  Functions
fetchJobStoreFiles(jobStore, options) | Takes a list of file names as glob patterns, searches for these within a
printContentsOfJobStore(job_store[, job_id]) | Fetch a list of all files contained in the job store if nameOfJob is not
main() |

  Module Contents

    toil.utils.toilDebugFile.logger

    toil.utils.toilDebugFile.fetchJobStoreFiles(jobStore, options)
        Takes a list of file names as glob patterns, searches for these within a given directory, and attempts to take all of the files found and copy them into options.localFilePath.

        Parameters

            •   jobStore (toil.jobStores.fileJobStore.FileJobStore <#toil.jobStores.fileJobStore.FileJobStore>) -- A fileJobStore object.

            •   options.fetch -- List of file glob patterns to search for in the jobStore and copy into options.localFilePath.

            •   options.localFilePath -- Local directory to copy files into.

            •   options.jobStore -- The path to the jobStore directory.

            •   options (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)

        Return type
            None

    toil.utils.toilDebugFile.printContentsOfJobStore(job_store, job_id=None)
        Fetch a list of all files contained in the job store if nameOfJob is not declared, otherwise it only prints out the names of files for that specific job for which it can find a match. Also creates a log file of these file names in the current directory.

        Parameters

            •   job_store (toil.jobStores.fileJobStore.FileJobStore <#toil.jobStores.fileJobStore.FileJobStore>) -- Job store to ask for files from.

            •   job_id (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- Default is None, which prints out all files in the jobStore. If specified, it will print all jobStore files that have been written to the jobStore by that job.

        Return type
            None

    toil.utils.toilDebugFile.main()

        Return type
            None

  toil.utils.toilDebugJob

Debug tool for running a toil job locally.

  Attributes
logger |

  Functions
main() |

  Module Contents

    toil.utils.toilDebugJob.logger

    toil.utils.toilDebugJob.main()

        Return type
            None

  toil.utils.toilDestroyCluster

Terminates the specified cluster and associated resources.

  Attributes
logger |

  Functions
main() |

  Module Contents

    toil.utils.toilDestroyCluster.logger

    toil.utils.toilDestroyCluster.main()

        Return type
            None

  toil.utils.toilKill

Kills rogue toil processes.

  Attributes
logger |

  Functions
main() |

  Module Contents

    toil.utils.toilKill.logger

    toil.utils.toilKill.main()

        Return type
            None

  toil.utils.toilLaunchCluster

Launches a toil leader instance with the specified provisioner.

  Attributes
build_tag_dict_from_env |
logger |

  Functions
create_tags_dict(tags) |
main() |

  Module Contents

    toil.utils.toilLaunchCluster.build_tag_dict_from_env: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.utils.toilLaunchCluster.logger

    toil.utils.toilLaunchCluster.create_tags_dict(tags)

        Parameters
            tags (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.utils.toilLaunchCluster.main()

        Return type
            None

  toil.utils.toilMain

  Functions
main() |
get_or_die(module, name) | Get an object from a module or complain that it is missing.
loadModules() |
printHelp(modules) |
printVersion() |

  Module Contents

    toil.utils.toilMain.main()

        Return type
            None

    toil.utils.toilMain.get_or_die(module, name)
        Get an object from a module or complain that it is missing.

        Parameters

            •   module (types.ModuleType </usr/share/doc/python3/html/library/types.html#types.ModuleType>)

            •   name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            Any

    toil.utils.toilMain.loadModules()

        Return type
            dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, types.ModuleType </usr/share/doc/python3/html/library/types.html#types.ModuleType>]

    toil.utils.toilMain.printHelp(modules)

        Parameters
            modules (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, types.ModuleType </usr/share/doc/python3/html/library/types.html#types.ModuleType>])

        Return type
            None

    toil.utils.toilMain.printVersion()

        Return type
            None

  toil.utils.toilRsyncCluster

Rsyncs into the toil appliance container running on the leader of the cluster.

  Attributes
logger |

  Functions
main() |

  Module Contents

    toil.utils.toilRsyncCluster.logger

    toil.utils.toilRsyncCluster.main()

        Return type
            None

  toil.utils.toilServer

CLI entry for the Toil servers.

  Attributes
logger |

  Functions
main() |

  Module Contents

    toil.utils.toilServer.logger

    toil.utils.toilServer.main()

        Return type
            None

  toil.utils.toilSshCluster

SSH into the toil appliance container running on the leader of the cluster.

  Attributes
logger |

  Functions
have_ipv6() | Return True if the IPv6 loopback interface is useable.
main() |

  Module Contents

    toil.utils.toilSshCluster.logger

    toil.utils.toilSshCluster.have_ipv6()
        Return True if the IPv6 loopback interface is useable.

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.utils.toilSshCluster.main()

        Return type
            None

  toil.utils.toilStats

Reports statistical data about a given Toil workflow.

  Attributes
logger |
CATEGORIES |
CATEGORY_UNITS |
TITLES |
TIME_CATEGORIES |
SPACE_CATEGORIES |
COMPUTED_CATEGORIES |
LONG_FORMS |
sort_category_choices |
sort_field_choices |

  Classes
ColumnWidths | Convenience object that stores the width of columns for printing. Helps make things pretty.

  Functions
pad_str(s[, field]) | Pad the beginning of a string with spaces, if necessary.
pretty_space(k[, field, alone]) | Given input k as kibibytes, return a nicely formatted string.
pretty_time(t[, field, unit, alone]) | Given input t as seconds, return a nicely formatted string.
report_unit(unit) | Format a unit name for display.
report_time(t, options[, field, unit, alone]) | Given t seconds, report back the correct format as string.
report_space(k, options[, field, unit, alone]) | Given k kibibytes, report back the correct format as string.
report_number(n[, field, nan_value]) | Given a number, report back the correct format as string.
report(v, category, options[, field, alone]) | Report a value of the given category formatted as a string.
sprint_tag(key, tag, options[, columnWidths]) | Generate a pretty-print ready string from a JTTag().
decorate_title(category, title, options) | Add extra parts to the category titles.
decorate_subheader(category, columnWidths, options) | Add a marker to the correct field if the TITLE is sorted on.
get(tree, name) | Return a float value attribute NAME from TREE.
sort_jobs(jobTypes, options) | Return a jobTypes all sorted.
report_pretty_data(root, worker, job, job_types, options) | Print the important bits out.
compute_column_widths(job_types, worker, job, options) | Return a ColumnWidths() object with the correct max widths.
update_column_widths(tag, cw, options) | Update the column width attributes for this tag's fields.
build_element(element, items, item_name, defaults) | Create an element for output.
create_summary(element, containingItems, ...) | Figure out how many jobs (or contained items) ran on each worker (or containing item).
get_stats(jobStore) | Sum together all the stats information in the job store.
process_data(config, stats) | Collate the stats and report
report_data(tree, options) |
add_stats_options(parser) |
main() | Reports stats on the workflow, use with --stats option to toil.

  Module Contents

    toil.utils.toilStats.logger

    toil.utils.toilStats.CATEGORIES = ['time', 'clock', 'wait', 'memory', 'disk']

    toil.utils.toilStats.CATEGORY_UNITS

    toil.utils.toilStats.TITLES

    toil.utils.toilStats.TIME_CATEGORIES

    toil.utils.toilStats.SPACE_CATEGORIES

    toil.utils.toilStats.COMPUTED_CATEGORIES

    toil.utils.toilStats.LONG_FORMS

    class toil.utils.toilStats.ColumnWidths
        Convenience object that stores the width of columns for printing. Helps make things pretty.

        categories = ['time', 'clock', 'wait', 'memory', 'disk']

        fields_count = ['count', 'min', 'med', 'ave', 'max', 'total']

        fields = ['min', 'med', 'ave', 'max', 'total']

        data: dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, int </usr/share/doc/python3/html/library/functions.html#int>]

        title(category)
            Return the total printed length of this category item.

            Parameters
                category (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        get_width(category, field)

            Parameters

                •   category (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   field (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        set_width(category, field, width)

            Parameters

                •   category (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   field (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   width (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                None

        report()

            Return type
                None

    toil.utils.toilStats.pad_str(s, field=None)
        Pad the beginning of a string with spaces, if necessary.

        Parameters

            •   s (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   field (Optional[int </usr/share/doc/python3/html/library/functions.html#int>])

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.utils.toilStats.pretty_space(k, field=None, alone=False)
        Given input k as kibibytes, return a nicely formatted string.

        Parameters

            •   k (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   field (Optional[int </usr/share/doc/python3/html/library/functions.html#int>])

            •   alone (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.utils.toilStats.pretty_time(t, field=None, unit='s', alone=False)
        Given input t as seconds, return a nicely formatted string.

        Parameters

            •   t (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   field (Optional[int </usr/share/doc/python3/html/library/functions.html#int>])

            •   unit (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   alone (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.utils.toilStats.report_unit(unit)
        Format a unit name for display.

        Parameters
            unit (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.utils.toilStats.report_time(t, options, field=None, unit='s', alone=False)
        Given t seconds, report back the correct format as string.

        Parameters

            •   t (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   options (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)

            •   field (Optional[int </usr/share/doc/python3/html/library/functions.html#int>])

            •   unit (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   alone (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.utils.toilStats.report_space(k, options, field=None, unit='KiB', alone=False)
        Given k kibibytes, report back the correct format as string.

        If unit is set to B, convert to KiB first.

        Parameters

            •   k (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   options (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)

            •   field (Optional[int </usr/share/doc/python3/html/library/functions.html#int>])

            •   unit (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   alone (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.utils.toilStats.report_number(n, field=None, nan_value='NaN')
        Given a number, report back the correct format as string.

        If it is a NaN or None, use nan_value to represent it instead.

        Parameters

            •   n (Union[int </usr/share/doc/python3/html/library/functions.html#int>, float </usr/share/doc/python3/html/library/functions.html#float>, None])

            •   field (Optional[int </usr/share/doc/python3/html/library/functions.html#int>])

            •   nan_value (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.utils.toilStats.report(v, category, options, field=None, alone=False)
        Report a value of the given category formatted as a string.

        Uses the given field width if set.

        If alone is set, the field is being formatted outside a table and might need a unit.

        Parameters

            •   v (float </usr/share/doc/python3/html/library/functions.html#float>)

            •   category (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   options (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)

            •   field (Optional[int </usr/share/doc/python3/html/library/functions.html#int>])

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.utils.toilStats.sprint_tag(key, tag, options, columnWidths=None)
        Generate a pretty-print ready string from a JTTag().

        Parameters

            •   key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   tag (toil.lib.expando.Expando <#toil.lib.expando.Expando>)

            •   options (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)

            •   columnWidths (Optional[ColumnWidths <#toil.utils.toilStats.ColumnWidths>])

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.utils.toilStats.decorate_title(category, title, options)
        Add extra parts to the category titles.

        Add units to title if they won't appear in the formatted values. Add a marker to TITLE if the TITLE is sorted on.

        Parameters

            •   category (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   title (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   options (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.utils.toilStats.decorate_subheader(category, columnWidths, options)
        Add a marker to the correct field if the TITLE is sorted on.

        Parameters

            •   category (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   columnWidths (ColumnWidths <#toil.utils.toilStats.ColumnWidths>)

            •   options (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.utils.toilStats.get(tree, name)
        Return a float value attribute NAME from TREE.

        Parameters

            •   tree (toil.lib.expando.Expando <#toil.lib.expando.Expando>)

            •   name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            float </usr/share/doc/python3/html/library/functions.html#float>

    toil.utils.toilStats.sort_jobs(jobTypes, options)
        Return a jobTypes all sorted.

        Parameters

            •   jobTypes (list </usr/share/doc/python3/html/library/stdtypes.html#list>[Any])

            •   options (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[Any]

    toil.utils.toilStats.report_pretty_data(root, worker, job, job_types, options)
        Print the important bits out.

        Parameters

            •   root (toil.lib.expando.Expando <#toil.lib.expando.Expando>)

            •   worker (toil.lib.expando.Expando <#toil.lib.expando.Expando>)

            •   job (toil.lib.expando.Expando <#toil.lib.expando.Expando>)

            •   job_types (list </usr/share/doc/python3/html/library/stdtypes.html#list>[Any])

            •   options (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.utils.toilStats.compute_column_widths(job_types, worker, job, options)
        Return a ColumnWidths() object with the correct max widths.

        Parameters

            •   job_types (list </usr/share/doc/python3/html/library/stdtypes.html#list>[Any])

            •   worker (toil.lib.expando.Expando <#toil.lib.expando.Expando>)

            •   job (toil.lib.expando.Expando <#toil.lib.expando.Expando>)

            •   options (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)

        Return type
            ColumnWidths <#toil.utils.toilStats.ColumnWidths>

    toil.utils.toilStats.update_column_widths(tag, cw, options)
        Update the column width attributes for this tag's fields.

        Parameters

            •   tag (toil.lib.expando.Expando <#toil.lib.expando.Expando>)

            •   cw (ColumnWidths <#toil.utils.toilStats.ColumnWidths>)

            •   options (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)

        Return type
            None

    toil.utils.toilStats.build_element(element, items, item_name, defaults)
        Create an element for output.

        Parameters

            •   element (toil.lib.expando.Expando <#toil.lib.expando.Expando>)

            •   items (list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.job.Job <#toil.job.Job>])

            •   item_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   defaults (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, float </usr/share/doc/python3/html/library/functions.html#float>])

        Return type
            toil.lib.expando.Expando <#toil.lib.expando.Expando>

    toil.utils.toilStats.create_summary(element, containingItems, containingItemName, count_contained)
        Figure out how many jobs (or contained items) ran on each worker (or containing item).

        Stick a bunch of xxx_number_per_xxx stats into element to describe this.

        Parameters

            •   count_contained (Callable[[toil.lib.expando.Expando <#toil.lib.expando.Expando>], int </usr/share/doc/python3/html/library/functions.html#int>]) -- function that maps from containing item to number of contained items.

            •   element (toil.lib.expando.Expando <#toil.lib.expando.Expando>)

            •   containingItems (list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.lib.expando.Expando <#toil.lib.expando.Expando>])

            •   containingItemName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            None

    toil.utils.toilStats.get_stats(jobStore)
        Sum together all the stats information in the job store.

        Produces one object containing lists of the values from all the summed objects.

        Parameters
            jobStore (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>)

        Return type
            toil.lib.expando.Expando <#toil.lib.expando.Expando>

    toil.utils.toilStats.process_data(config, stats)
        Collate the stats and report

        Parameters

            •   config (toil.common.Config <#toil.common.Config>)

            •   stats (toil.lib.expando.Expando <#toil.lib.expando.Expando>)

        Return type
            toil.lib.expando.Expando <#toil.lib.expando.Expando>

    toil.utils.toilStats.report_data(tree, options)

        Parameters

            •   tree (toil.lib.expando.Expando <#toil.lib.expando.Expando>)

            •   options (argparse.Namespace </usr/share/doc/python3/html/library/argparse.html#argparse.Namespace>)

        Return type
            None

    toil.utils.toilStats.sort_category_choices = ['time', 'clock', 'wait', 'memory', 'disk', 'alpha', 'count']

    toil.utils.toilStats.sort_field_choices = ['min', 'med', 'ave', 'max', 'total']

    toil.utils.toilStats.add_stats_options(parser)

        Parameters
            parser (argparse.ArgumentParser </usr/share/doc/python3/html/library/argparse.html#argparse.ArgumentParser>)

        Return type
            None

    toil.utils.toilStats.main()
        Reports stats on the workflow, use with --stats option to toil.

        Return type
            None

  toil.utils.toilStatus

Tool for reporting on job status.

  Attributes
logger |

  Classes
ToilStatus | Tool for reporting on job status.

  Functions
main() | Reports the state of a Toil workflow.

  Module Contents

    toil.utils.toilStatus.logger

    class toil.utils.toilStatus.ToilStatus(jobStoreName, specifiedJobs=None)
        Tool for reporting on job status.

        Parameters

            •   jobStoreName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   specifiedJobs (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

        jobStoreName

        jobStore

        message_bus_path

        print_dot_chart()
            Print a dot output graph representing the workflow.

            Return type
                None

        printJobLog()
            Takes a list of jobs, finds their log files, and prints them to the terminal.

            Return type
                None

        printJobChildren()
            Takes a list of jobs, and prints their successors.

            Return type
                None

        printAggregateJobStats(properties, childNumber)
            Prints each job's ID, log file, remaining tries, and other properties.

            Parameters

                •   properties (list </usr/share/doc/python3/html/library/stdtypes.html#list>[set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) -- A set of string flag names for each job in self.jobsToReport.

                •   childNumber (list </usr/share/doc/python3/html/library/stdtypes.html#list>[int </usr/share/doc/python3/html/library/functions.html#int>]) -- A list of child counts for each job in self.jobsToReport.

            Return type
                None

        report_on_jobs()
            Gathers information about jobs such as its child jobs and status.

            Returns jobStats
                Dict containing some lists of jobs by category, and some lists of job properties for each job in self.jobsToReport.

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, Any]

        static getPIDStatus(jobStoreName)
            Determine the status of a process with a particular local pid.

            Checks to see if a process exists or not.

            Returns
                A string indicating the status of the PID of the workflow as stored in the jobstore.

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

            Parameters
                jobStoreName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        static getStatus(jobStoreName)
            Determine the status of a workflow.

            If the jobstore does not exist, this returns 'QUEUED', assuming it has not been created yet.

            Checks for the existence of files created in the toil.Leader.run(). In toil.Leader.run(), if a workflow completes with failed jobs, 'failed.log' is created, otherwise 'succeeded.log' is written. If neither of these exist, the leader is still running jobs.

            Returns
                A string indicating the status of the workflow. ['COMPLETED', 'RUNNING', 'ERROR', 'QUEUED']

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

            Parameters
                jobStoreName (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        print_running_jobs()
            Prints a list of the currently running jobs

            Return type
                None

        fetchRootJob()
            Fetches the root job from the jobStore that provides context for all other jobs.

            Exactly the same as the jobStore.loadRootJob() function, but with a different exit message if the root job is not found (indicating the workflow ran successfully to completion and certain stats cannot be gathered from it meaningfully such as which jobs are left to run).

            Raises
                JobException <#toil.job.JobException> -- if the root job does not exist.

            Return type
                toil.job.JobDescription <#toil.job.JobDescription>

        fetchUserJobs(jobs)
            Takes a user input array of jobs, verifies that they are in the jobStore and returns the array of jobsToReport.

            Parameters
                jobs (list </usr/share/doc/python3/html/library/stdtypes.html#list>) -- A list of jobs to be verified.

            Returns jobsToReport
                A list of jobs which are verified to be in the jobStore.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.job.JobDescription <#toil.job.JobDescription>]

        traverseJobGraph(rootJob, jobsToReport=None, foundJobStoreIDs=None)
            Find all current jobs in the jobStore and return them as an Array.

            Parameters

                •   rootJob (toil.job.JobDescription <#toil.job.JobDescription>) -- The root job of the workflow.

                •   jobsToReport (list </usr/share/doc/python3/html/library/stdtypes.html#list>) -- A list of jobNodes to be added to and returned.

                •   foundJobStoreIDs (set </usr/share/doc/python3/html/library/stdtypes.html#set>) -- A set of jobStoreIDs used to keep track of jobStoreIDs encountered in traversal.

            Returns jobsToReport
                The list of jobs currently in the job graph.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.job.JobDescription <#toil.job.JobDescription>]

    toil.utils.toilStatus.main()
        Reports the state of a Toil workflow.

        Return type
            None

  toil.utils.toilUpdateEC2Instances

Updates Toil's internal list of EC2 instance types.

  Attributes
logger |

  Functions
internet_connection() | Returns True if there is an internet connection present, and False otherwise.
main() |

  Module Contents

    toil.utils.toilUpdateEC2Instances.logger

    toil.utils.toilUpdateEC2Instances.internet_connection()
        Returns True if there is an internet connection present, and False otherwise.

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.utils.toilUpdateEC2Instances.main()

        Return type
            None

  toil.version

  Attributes
baseVersion |
cgcloudVersion |
version |
cacheTag |
mainCacheTag |
distVersion |
exactPython |
python |
dockerTag |
currentCommit |
dockerRegistry |
dockerName |
dirty |
cwltool_version |

  Module Contents

    toil.version.baseVersion = '9.1.2'

    toil.version.cgcloudVersion = '1.6.0a1.dev393'

    toil.version.version = '9.1.2-355b72ba0e425619c0367562caf2a337078dba65'

    toil.version.cacheTag = 'cache-local-py3.13'

    toil.version.mainCacheTag = 'cache-master-py3.13'

    toil.version.distVersion = '9.1.2'

    toil.version.exactPython = 'python3.13'

    toil.version.python = 'python3.13'

    toil.version.dockerTag = '9.1.2-355b72ba0e425619c0367562caf2a337078dba65-py3.13'

    toil.version.currentCommit = '355b72ba0e425619c0367562caf2a337078dba65'

    toil.version.dockerRegistry = 'quay.io/ucsc_cgl'

    toil.version.dockerName = 'toil'

    toil.version.dirty = False

    toil.version.cwltool_version = '3.1.20250715140722'

  toil.wdl

  Submodules

  toil.wdl.utils

  Functions
get_version(iterable) | Get the version of the WDL document.

  Module Contents

    toil.wdl.utils.get_version(iterable)
        Get the version of the WDL document.

        Parameters
            iterable (collections.abc.Iterable </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterable>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- An iterable that contains the lines of a WDL document.

        Returns
            The WDL version used in the workflow.

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

  toil.wdl.wdltoil

  Attributes
logger |
WDLINode |
AnyINode |
file_digest |
WDLContext |
F |
WDLBindings |
SHARED_PATH_ATTR |

  Exceptions
InsufficientMountDiskSpace | Common base class for all non-exit exceptions.

  Classes
INodeTransform | A type for a function that transforms a File or Directory to a modified copy or None.
ReadableFileObj | Protocol that is more specific than what file_digest takes as an argument.
FileDigester | Protocol for the features we need from hashlib.file_digest.
NonDownloadingSize | WDL size() implementation that avoids downloading files.
ToilWDLStdLibBase | Standard library implementation for WDL as run on Toil.
ToilWDLStdLibWorkflow | Standard library implementation for workflow scope.
ToilWDLStdLibTaskCommand | Standard library implementation to use inside a WDL task command evaluation.
ToilWDLStdLibTaskOutputs | Standard library implementation for WDL as run on Toil, with additional
WDLBaseJob | Base job class for all WDL-related jobs.
WDLTaskWrapperJob | Job that determines the resources needed to run a WDL job.
WDLTaskJob | Job that runs a WDL task.
WDLWorkflowNodeJob | Job that evaluates a WDL workflow node.
WDLWorkflowNodeListJob | Job that evaluates a list of WDL workflow nodes, which are in the same
WDLCombineBindingsJob | Job that collects the results from WDL workflow nodes and combines their
WDLWorkflowGraph | Represents a graph of WDL WorkflowNodes.
WDLSectionJob | Job that can create more graph for a section of the workflow.
WDLScatterJob | Job that evaluates a scatter in a WDL workflow. Runs the body for each
WDLArrayBindingsJob | Job that takes all new bindings created in an array of input environments,
WDLConditionalJob | Job that evaluates a conditional in a WDL workflow.
WDLWorkflowJob | Job that evaluates an entire WDL workflow.
WDLOutputsJob | Job which evaluates an outputs section for a workflow.
WDLStartJob | Job that evaluates an entire WDL workflow, and returns the workflow outputs
WDLInstallImportsJob | Class represents a unit of work in toil.
WDLImportWrapper | Job to organize importing files on workers instead of the leader. Responsible for extracting filenames and metadata,

  Functions
is_inode(value) | Determine if a WDL value is either a File or Directory.
wdl_error_reporter(task[, exit, log]) | Run code in a context where WDL errors will be reported with pretty formatting.
report_wdl_errors(task[, exit, log]) | Create a decorator to report WDL errors with the given task message.
toil_read_source(uri, path, importer) | Implementation of a MiniWDL read_source function that can use any
virtualized_equal(value1, value2) | Check if two WDL values are equal when taking into account file virtualization.
combine_bindings(all_bindings) | Combine variable bindings from multiple predecessor tasks into one set for
log_bindings(log_function, message, all_bindings) | Log bindings to the console, even if some are still promises.
get_supertype(types) | Get the supertype that can hold values of all the given types.
for_each_node(root) | Iterate over all WDL workflow nodes in the given node, including inputs,
recursive_dependencies(root) | Get the combined workflow_node_dependencies of root and everything under
parse_disks(spec, disks_spec) | Parse a WDL disk spec into a disk mount specification.
pack_toil_uri(file_id, task_path, parent, file_basename) | Encode a Toil file ID and metadata about who wrote it as a URI.
unpack_toil_uri(toil_uri) | Unpack a URI made by make_toil_uri.
clone_metadata(old_inode, new_inode) | Copy all Toil metadata from one WDL File/Directory to another.
make_inode(example_inode, value, expr) | Make a new File or Directory of the same type as the example with the given arguments.
set_inode_value(inode, new_value) | Return a copy of a WDL File/Directory with the value changed.
set_inode_nonexistent(inode, nonexistent) | Return a copy of a WDL File/Directory with the nonexistent flag changed.
get_inode_nonexistent(inode) | Return the nonexistent flag for a File/Direcotry.
set_inode_virtualized_value(inode, virtualized_value) | Return a copy of a WDL File/Directory with the virtualized_value attribute set.
get_inode_virtualized_value(inode) | Get the virtualized storage location for a File/Directory.
get_shared_fs_path(inode) | If a File/Directory has a shared filesystem path, get that path.
set_shared_fs_path(inode, path) | Return a copy of the given File/Directory with a shared filesystem path.
view_shared_fs_paths(bindings) | Given WDL bindings, return a copy where all files have their shared filesystem paths as their values.
poll_execution_cache(node, bindings) | Return the cached result of calling this workflow or task, and its key.
fill_execution_cache(cache_key, output_bindings, ...) | Cache the result of calling a workflow or task.
choose_human_readable_directory(root_dir, ...) | Select a good directory to save files from a task and source directory in.
evaluate_decls_to_bindings(decls, all_bindings, ...[, ...]) | Evaluate decls with a given bindings environment and standard library.
extract_inode_values(environment) | Get a list of all File or Directory object values in the given bindings.
extract_inode_virtualized_values(environment) | Get a list of all File/Directory object virtualized values in the bindings.
extract_toil_file_uris(environment) | Get the toilfile: URIs in the given bindings.
virtualize_inodes_in_bindings(environment, file_to_id, ...) | Fill in the virtualized_value fields for File/Directory objects.
convert_remote_files(environment, file_source, task_path) | Resolve relative-URI files in the given environment and import all files.
evaluate_named_expression(context, name, ...) | Evaluate an expression when we know the name of it.
evaluate_decl(node, environment, stdlib) | Evaluate the expression of a declaration node, or raise an error.
evaluate_call_inputs(context, expressions, ...[, ...]) | Evaluate a bunch of expressions with names, and make them into a fresh set of bindings. inputs_dict is a mapping of
evaluate_defaultable_decl(node, environment, stdlib) | If the name of the declaration is already defined in the environment, return its value. Otherwise, return the evaluated expression.
devirtualize_inodes(environment, stdlib) | Make sure all the File values embedded in the given bindings point to files
virtualize_inodes(environment, stdlib[, enforce_existence]) | Make sure all the File/Directory values embedded in the given bindings point to files
delete_dead_files(internal_bindings, ...) | Delete any files that are in the given bindings but not in the live list.
all_parents(path) | Yield all parents of the given path, up to the filesystem root.
add_paths(task_container, host_paths) | Based off of WDL.runtime.task_container.add_paths from miniwdl
drop_if_missing(inode, standard_library) | Return None if a File/Directory doesn't exist, or its path if it does.
missing_inode_dropper(standard_library) | Get a function to null out missing File/Directory values.
drop_missing_files(environment, standard_library) | Make sure all the File values embedded in the given bindings point to files
get_paths_in_bindings(environment) | Get the paths of all Files and Directories in the bindings.
map_over_inodes_in_bindings(environment, transform) | Run all File values embedded in the given bindings through the given
map_over_inodes_in_binding(binding, transform) | Run all File values' types and values embedded in the given binding's value through the given
remove_expr_from_value(value) | Remove the expression from a WDL value
map_over_typed_value(value, transform) | Apply a transform to a WDL value and all contained WDL values.
map_over_typed_inodes_in_value(value, transform) | Run all File values embedded in the given value through the given
ensure_null_inodes_are_nullable(value, original_value, ...) | Run through all nested values embedded in the given value and check that the null values are valid.
make_root_job(target, inputs, inputs_search_path, ...) |
main() | A Toil workflow to interpret WDL input files.

  Module Contents

    toil.wdl.wdltoil.logger

    toil.wdl.wdltoil.WDLINode

    toil.wdl.wdltoil.AnyINode

    class toil.wdl.wdltoil.INodeTransform
        Bases: Protocol

        A type for a function that transforms a File or Directory to a modified copy or None.

        If you use Callable[[AnyINode], AnyINode] as an argument type, it makes your function generic on the type variable; it doesn't mean that you take a function that is itself generic on the type variable. So we define a complicated type for functions that transform inodes to the same type of inodes.

        __call__(__file: WDL.Value.File) -> WDL.Value.File | None </usr/share/doc/python3/html/library/constants.html#None>

        __call__(__directory: WDL.Value.Directory) -> WDL.Value.Directory | None </usr/share/doc/python3/html/library/constants.html#None>

    toil.wdl.wdltoil.is_inode(value)
        Determine if a WDL value is either a File or Directory.

        Is a MyPy type guard, so code protected by this function in an if statement will convince MyPy that it can safely use what it passed to this function as a File-or-Directory.

        Parameters
            value (WDL.Value.Base)

        Return type
            typing_extensions.TypeGuard[WDLINode]

    class toil.wdl.wdltoil.ReadableFileObj
        Bases: Protocol

        Protocol that is more specific than what file_digest takes as an argument. Also guarantees a read() method.

        Would extend the protocol from Typeshed for hashlib but those are only declared for 3.11+.

        readinto(buf, /)

            Parameters
                buf (bytearray </usr/share/doc/python3/html/library/stdtypes.html#bytearray>)

            Return type
                int </usr/share/doc/python3/html/library/functions.html#int>

        readable()

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        read(number)

            Parameters
                number (int </usr/share/doc/python3/html/library/functions.html#int>)

            Return type
                bytes </usr/share/doc/python3/html/library/stdtypes.html#bytes>

    class toil.wdl.wdltoil.FileDigester
        Bases: Protocol

        Protocol for the features we need from hashlib.file_digest.

        __call__(__f, __alg_name)

            Parameters

                •   __f (ReadableFileObj <#toil.wdl.wdltoil.ReadableFileObj>)

                •   __alg_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                hashlib._Hash

    toil.wdl.wdltoil.file_digest: FileDigester <#toil.wdl.wdltoil.FileDigester>

    toil.wdl.wdltoil.WDLContext

    exception toil.wdl.wdltoil.InsufficientMountDiskSpace(mount_targets, desired_bytes, available_bytes)
        Bases: Exception </usr/share/doc/python3/html/library/exceptions.html#Exception>

        Common base class for all non-exit exceptions.

        Parameters

            •   mount_targets (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   desired_bytes (int </usr/share/doc/python3/html/library/functions.html#int>)

            •   available_bytes (int </usr/share/doc/python3/html/library/functions.html#int>)

    toil.wdl.wdltoil.wdl_error_reporter(task, exit=False, log=logger.critical)
        Run code in a context where WDL errors will be reported with pretty formatting.

        Parameters

            •   task (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   exit (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            •   log (Callable[[str </usr/share/doc/python3/html/library/stdtypes.html#str>], None])

        Return type
            Generator[None]

    toil.wdl.wdltoil.F

    toil.wdl.wdltoil.report_wdl_errors(task, exit=False, log=logger.critical)
        Create a decorator to report WDL errors with the given task message.

        Decorator can then be applied to a function, and if a WDL error happens it will say that it could not {task}.

        Parameters

            •   task (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   exit (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            •   log (Callable[[str </usr/share/doc/python3/html/library/stdtypes.html#str>], None])

        Return type
            Callable[[F], F]

    async toil.wdl.wdltoil.toil_read_source(uri, path, importer)
        Implementation of a MiniWDL read_source function that can use any filename or URL supported by Toil.

        Needs to be async because MiniWDL will await its result.

        Parameters

            •   uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   path (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   importer (WDL.Tree.Document | None)

        Return type
            WDL.Tree.ReadSourceResult

    toil.wdl.wdltoil.virtualized_equal(value1, value2)
        Check if two WDL values are equal when taking into account file virtualization.

        Treats virtualized and non-virtualized Files and Directories referring to the same underlying thing as equal.

        Parameters

            •   value1 (WDL.Value.Base) -- WDL value

            •   value2 (WDL.Value.Base) -- WDL value

        Returns
            Whether the two values are equal with file and directory virtualization accounted for

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.wdl.wdltoil.WDLBindings

    toil.wdl.wdltoil.combine_bindings(all_bindings)
        Combine variable bindings from multiple predecessor tasks into one set for the current task.

        Parameters
            all_bindings (Sequence[WDLBindings])

        Return type
            WDLBindings

    toil.wdl.wdltoil.log_bindings(log_function, message, all_bindings)
        Log bindings to the console, even if some are still promises.

        Parameters

            •   log_function (Callable[Ellipsis, None]) -- Function (like logger.info) to call to log data

            •   message (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Message to log before the bindings

            •   all_bindings (Sequence[toil.job.Promised[WDLBindings]]) -- A list of bindings or promises for bindings, to log

        Return type
            None

    toil.wdl.wdltoil.get_supertype(types)
        Get the supertype that can hold values of all the given types.

        Parameters
            types (Sequence[WDL.Type.Base])

        Return type
            WDL.Type.Base

    toil.wdl.wdltoil.for_each_node(root)
        Iterate over all WDL workflow nodes in the given node, including inputs, internal nodes of conditionals and scatters, and gather nodes.

        Parameters
            root (WDL.Tree.WorkflowNode)

        Return type
            Iterator[WDL.Tree.WorkflowNode]

    toil.wdl.wdltoil.recursive_dependencies(root)
        Get the combined workflow_node_dependencies of root and everything under it, which are not on anything in that subtree.

        Useful because section nodes can have internal nodes with dependencies not reflected in those of the section node itself.

        Parameters
            root (WDL.Tree.WorkflowNode)

        Return type
            set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.wdl.wdltoil.parse_disks(spec, disks_spec)
        Parse a WDL disk spec into a disk mount specification. :param spec: Disks spec to parse :param disks_spec: All disks spec as specified in the WDL file. Only used for better error messages. :return: Specified mount point (None if omitted or local-disk), number of units, size of unit (ex GB)

        Parameters

            •   spec (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   disks_spec (list </usr/share/doc/python3/html/library/stdtypes.html#list>[WDL.Value.String] | str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str> | None, float </usr/share/doc/python3/html/library/functions.html#float>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.wdl.wdltoil.pack_toil_uri(file_id, task_path, parent, file_basename)
        Encode a Toil file ID and metadata about who wrote it as a URI.

        The URI will start with the scheme in TOIL_URI_SCHEME.

        Parameters

            •   parent (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- bare path or URI to the parent of the file. Only one unique value may be used for a given parent location. Must be the same as the name parameter of toil.lib.directory.encode_directory() <#toil.lib.directory.encode_directory>. May be absolute or relative, but to avoid collisions should only be relative for worker temp storage.

            •   file_id (toil.fileStores.FileID <#toil.fileStores.FileID>)

            •   task_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   file_basename (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.wdl.wdltoil.unpack_toil_uri(toil_uri)
        Unpack a URI made by make_toil_uri.

        Returns
            the FileID, source task, source parent path or URI, and basename.

        Parameters
            toil_uri (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[toil.fileStores.FileID <#toil.fileStores.FileID>, str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.wdl.wdltoil.SHARED_PATH_ATTR = '_shared_fs_path'

    toil.wdl.wdltoil.clone_metadata(old_inode, new_inode)
        Copy all Toil metadata from one WDL File/Directory to another.

        Parameters

            •   old_inode (AnyINode)

            •   new_inode (AnyINode)

        Return type
            None

    toil.wdl.wdltoil.make_inode(example_inode, value, expr)
        Make a new File or Directory of the same type as the example with the given arguments.

        We use this because MyPy can't tell that type(a)(args) has the same type as a when a is typed with a TypeVar.

        Parameters

            •   example_inode (AnyINode)

            •   value (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   expr (Optional[WDL.Expr.Base])

        Return type
            AnyINode

    toil.wdl.wdltoil.set_inode_value(inode, new_value)
        Return a copy of a WDL File/Directory with the value changed.

        Preserves all Toil metadata.

        Parameters

            •   inode (AnyINode)

            •   new_value (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            AnyINode

    toil.wdl.wdltoil.set_inode_nonexistent(inode, nonexistent)
        Return a copy of a WDL File/Directory with the nonexistent flag changed.

        Preserves all Toil metadata.

        Parameters

            •   inode (AnyINode)

            •   nonexistent (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            AnyINode

    toil.wdl.wdltoil.get_inode_nonexistent(inode)
        Return the nonexistent flag for a File/Direcotry.

        Parameters
            inode (WDLINode)

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.wdl.wdltoil.set_inode_virtualized_value(inode, virtualized_value)
        Return a copy of a WDL File/Directory with the virtualized_value attribute set.

        Preserves all Toil metadata.

        Parameters

            •   inode (AnyINode)

            •   virtualized_value (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            AnyINode

    toil.wdl.wdltoil.get_inode_virtualized_value(inode)
        Get the virtualized storage location for a File/Directory.

        Parameters
            inode (WDLINode)

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.wdl.wdltoil.get_shared_fs_path(inode)
        If a File/Directory has a shared filesystem path, get that path.

        This will be the path the File was initially imported from, or the path that it has in the call cache.

        Parameters
            inode (WDLINode)

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.wdl.wdltoil.set_shared_fs_path(inode, path)
        Return a copy of the given File/Directory with a shared filesystem path.

        This should be the path it was initially imported from, or the path that it has in the call cache.

        Parameters

            •   inode (AnyINode)

            •   path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            AnyINode

    toil.wdl.wdltoil.view_shared_fs_paths(bindings)
        Given WDL bindings, return a copy where all files have their shared filesystem paths as their values.

        Parameters
            bindings (WDLBindings)

        Return type
            WDLBindings

    toil.wdl.wdltoil.poll_execution_cache(node, bindings)
        Return the cached result of calling this workflow or task, and its key.

        Returns None and the key if the cache has no result for us.

        Deals in un-namespaced bindings.

        Parameters

            •   node (Union[WDL.Tree.Workflow, WDL.Tree.Task])

            •   bindings (WDLBindings)

        Return type
            tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[WDLBindings | None, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.wdl.wdltoil.fill_execution_cache(cache_key, output_bindings, file_store, wdl_options, miniwdl_logger=None, miniwdl_config=None)
        Cache the result of calling a workflow or task.

        Deals in un-namespaced bindings.

        Returns
            possibly modified bindings to continue on with, that may reference the cache.

        Parameters

            •   cache_key (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   output_bindings (WDLBindings)

            •   file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            •   wdl_options (WDLContext)

            •   miniwdl_logger (Optional[logging.Logger </usr/share/doc/python3/html/library/logging.html#logging.Logger>])

            •   miniwdl_config (Optional[WDL.runtime.config.Loader])

        Return type
            WDLBindings

    toil.wdl.wdltoil.choose_human_readable_directory(root_dir, source_task_path, parent)
        Select a good directory to save files from a task and source directory in.

        The directories involved may not exist.

        Parameters

            •   root_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Directory that the path will be under

            •   source_task_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The dotted WDL name of whatever generated the file. We assume this is an acceptable filename component.

            •   parent (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Directory path or parent URI that the file came from. If a path, may be either absolute (on the worker or leader filesystem) or relative.

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.wdl.wdltoil.evaluate_decls_to_bindings(decls, all_bindings, standard_library, include_previous=False, drop_missing_files=False, expressions_are_defaults=False)
        Evaluate decls with a given bindings environment and standard library.

        Creates a new bindings object that only contains the bindings from the given decls. Guarantees that each decl in decls can access the variables defined by the previous ones.

        Parameters

            •   all_bindings (WDLBindings) -- Environment to use when evaluating decls

            •   decls (list </usr/share/doc/python3/html/library/stdtypes.html#list>[WDL.Tree.Decl]) -- Decls to evaluate

            •   standard_library (ToilWDLStdLibBase <#toil.wdl.wdltoil.ToilWDLStdLibBase>) -- Standard library

            •   include_previous (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Whether to include the existing environment in the new returned environment. This will be false for outputs where only defined decls should be included

            •   drop_missing_files (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- Whether to coerce nonexistent files to null. The coerced elements will be checked that the transformation is valid. Currently should only be enabled in output sections, see <https://github.com/openwdl/wdl/issues/673#issuecomment-2248828116>.

            •   expressions_are_defaults (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If True, value expressions in decls are treated as default values, and there may be existing values in the incoming environment that take precedence. If False, each decl is taken to be a fresh definition, and expressions are always evaluated and used.

        Returns
            New bindings object

        Return type
            WDLBindings

    class toil.wdl.wdltoil.NonDownloadingSize
        Bases: WDL.StdLib._Size

        WDL size() implementation that avoids downloading files.

        MiniWDL's default size() implementation downloads the whole file to get its size. We want to be able to get file sizes from code running on the leader, where there may not be space to download the whole file. So we override the fancy class that implements it so that we can handle sizes for FileIDs using the FileID's stored size info.

    toil.wdl.wdltoil.extract_inode_values(environment)
        Get a list of all File or Directory object values in the given bindings.

        Parameters
            environment (WDLBindings)

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.wdl.wdltoil.extract_inode_virtualized_values(environment)
        Get a list of all File/Directory object virtualized values in the bindings.

        If a value hasn't been virtualized, it won't contribute to the list.

        Parameters
            environment (WDLBindings)

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.wdl.wdltoil.extract_toil_file_uris(environment)
        Get the toilfile: URIs in the given bindings.

        Looks at for all Files in the given bindings, and all files inside Directories in the given bindings.

        Parameters
            environment (WDLBindings)

        Return type
            Iterable[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.wdl.wdltoil.virtualize_inodes_in_bindings(environment, file_to_id, file_to_metadata, task_path)
        Fill in the virtualized_value fields for File/Directory objects.

        Parameters

            •   environment (WDLBindings) -- Bindings to evaluate on. Will not be modified.

            •   file_to_id (Dict[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.fileStores.FileID <#toil.fileStores.FileID>]) -- Maps from imported URI to Toil FileID with the data.

            •   file_to_metadata (Dict[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.job.FileMetadata <#toil.job.FileMetadata>]) -- Maps from WDL-level file value to metadata about the file, including URI that would have been imported.

            •   task_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Returns
            new bindings object with the annotated File objects in it.

        Return type
            WDLBindings

    toil.wdl.wdltoil.convert_remote_files(environment, file_source, task_path, search_paths=None, import_remote_files=True, execution_dir=None)
        Resolve relative-URI files in the given environment and import all files.

        Returns an environment where each File's value is set to the URI it was found at, its virtualized value is set to what it was loaded into the filestore as (if applicable), and its shared filesystem path is set if it came from the local filesystem.

        Parameters

            •   environment (WDLBindings) -- Bindings to evaluate on

            •   file_source (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>) -- Context to search for files with

            •   task_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- Dotted WDL name of the user-level code doing the importing (probably the workflow name).

            •   search_paths (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) -- If set, try resolving input location relative to the URLs or directories in this list.

            •   import_remote_files (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If set, import files from remote locations. Else leave them as URI references.

            •   execution_dir (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            WDLBindings

    class toil.wdl.wdltoil.ToilWDLStdLibBase(file_store, wdl_options, share_files_with=None)
        Bases: WDL.StdLib.Base

        Standard library implementation for WDL as run on Toil.

        Parameters

            •   file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            •   wdl_options (WDLContext)

            •   share_files_with (ToilWDLStdLibBase <#toil.wdl.wdltoil.ToilWDLStdLibBase> | None)

        size

        property execution_dir: str </usr/share/doc/python3/html/library/stdtypes.html#str>

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        property task_path: str </usr/share/doc/python3/html/library/stdtypes.html#str>

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        get_local_paths()
            Get all the local paths of files devirtualized (or virtualized) through the stdlib.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        classmethod devirtualize_to(filename, dest_dir, file_source, wdl_options, devirtualized_to_virtualized, virtualized_to_devirtualized, export=None)
            Download or export a WDL virtualized filename/URL to the given directory.

            The destination directory must already exist. No other devirtualize_to call may be writing to it, including the case of another workflow writing the same task to the same place in the call cache at the same time.

            Makes sure sibling files stay siblings and files with the same name don't clobber each other. Makes sure Files or Directories within Directories stay at their proper place in the hierarchy. Called from within this class for tasks, and statically at the end of the workflow for outputs.

            Returns the local path to the file. If the file is already a local path, or if it already has an entry in virtualized_to_devirtualized, that path will be re-used instead of creating a new copy in dest_dir.

            The input filename could already be devirtualized. In this case, the filename should not be added to the cache.

            Parameters

                •   wdl_options (WDLContext) -- WDL options to carry through.

                •   export (bool </usr/share/doc/python3/html/library/functions.html#bool> | None) -- Always create exported copies of files rather than views that a FileStore might clean up.

                •   filename (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   dest_dir (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   file_source (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore> | toil.common.Toil <#toil.common.Toil>)

                •   devirtualized_to_virtualized (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   virtualized_to_devirtualized (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

    class toil.wdl.wdltoil.ToilWDLStdLibWorkflow(*args, **kwargs)
        Bases: ToilWDLStdLibBase

        Standard library implementation for workflow scope.

        Handles deduplicating files generated by write_* calls at workflow scope with copies already in the call cache, so that tasks that depend on them can also be fulfilled from the cache.

        Parameters

            •   args (Any)

            •   kwargs (Any)

    class toil.wdl.wdltoil.ToilWDLStdLibTaskCommand(file_store, container, wdl_options)
        Bases: ToilWDLStdLibBase

        Standard library implementation to use inside a WDL task command evaluation.

        Expects all the filenames in variable bindings to be container-side paths; these are the "virtualized" filenames, while the "devirtualized" filenames are host-side paths.

        Parameters

            •   file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            •   container (WDL.runtime.task_container.TaskContainer)

            •   wdl_options (WDLContext)

        container

    class toil.wdl.wdltoil.ToilWDLStdLibTaskOutputs(file_store, stdout_path, stderr_path, file_to_mountpoint, wdl_options, share_files_with=None)
        Bases: ToilWDLStdLibBase, WDL.StdLib.TaskOutputs

        Standard library implementation for WDL as run on Toil, with additional functions only allowed in task output sections.

        Parameters

            •   file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            •   stdout_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   stderr_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   file_to_mountpoint (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   wdl_options (WDLContext)

            •   share_files_with (ToilWDLStdLibBase <#toil.wdl.wdltoil.ToilWDLStdLibBase> | None)

        stdout_used()
            Return True if the standard output was read by the WDL.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        stderr_used()
            Return True if the standard error was read by the WDL.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.wdl.wdltoil.evaluate_named_expression(context, name, expected_type, expression, environment, stdlib)
        Evaluate an expression when we know the name of it.

        Parameters

            •   context (WDL.Error.SourceNode | WDL.Error.SourcePosition)

            •   name (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   expected_type (WDL.Type.Base | None)

            •   expression (WDL.Expr.Base | None)

            •   environment (WDLBindings)

            •   stdlib (WDL.StdLib.Base)

        Return type
            WDL.Value.Base

    toil.wdl.wdltoil.evaluate_decl(node, environment, stdlib)
        Evaluate the expression of a declaration node, or raise an error.

        Parameters

            •   node (WDL.Tree.Decl)

            •   environment (WDLBindings)

            •   stdlib (WDL.StdLib.Base)

        Return type
            WDL.Value.Base

    toil.wdl.wdltoil.evaluate_call_inputs(context, expressions, environment, stdlib, inputs_dict=None)
        Evaluate a bunch of expressions with names, and make them into a fresh set of bindings. inputs_dict is a mapping of variable names to their expected type for the input decls in a task.

        Parameters

            •   context (WDL.Error.SourceNode | WDL.Error.SourcePosition)

            •   expressions (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, WDL.Expr.Base])

            •   environment (WDLBindings)

            •   stdlib (WDL.StdLib.Base)

            •   inputs_dict (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, WDL.Type.Base] | None)

        Return type
            WDLBindings

    toil.wdl.wdltoil.evaluate_defaultable_decl(node, environment, stdlib)
        If the name of the declaration is already defined in the environment, return its value. Otherwise, return the evaluated expression.

        Parameters

            •   node (WDL.Tree.Decl)

            •   environment (WDLBindings)

            •   stdlib (WDL.StdLib.Base)

        Return type
            WDL.Value.Base

    toil.wdl.wdltoil.devirtualize_inodes(environment, stdlib)
        Make sure all the File values embedded in the given bindings point to files that are actually available to command line commands. The same virtual file always maps to the same devirtualized filename even with duplicates

        Parameters

            •   environment (WDLBindings)

            •   stdlib (ToilWDLStdLibBase <#toil.wdl.wdltoil.ToilWDLStdLibBase>)

        Return type
            WDLBindings

    toil.wdl.wdltoil.virtualize_inodes(environment, stdlib, enforce_existence=True)
        Make sure all the File/Directory values embedded in the given bindings point to files that are usable from other machines.

        Parameters

            •   environment (WDLBindings)

            •   stdlib (ToilWDLStdLibBase <#toil.wdl.wdltoil.ToilWDLStdLibBase>)

            •   enforce_existence (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            WDLBindings

    toil.wdl.wdltoil.delete_dead_files(internal_bindings, live_bindings_list, file_store)
        Delete any files that are in the given bindings but not in the live list.

        Scans the virtualized values of File and Directory objects anywhere in the bindings. Only tries to delete leaf files, not whole directories.

        Parameters

            •   internal_bindings (WDLBindings)

            •   live_bindings_list (list </usr/share/doc/python3/html/library/stdtypes.html#list>[WDLBindings])

            •   file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

        Return type
            None

    toil.wdl.wdltoil.all_parents(path)
        Yield all parents of the given path, up to the filesystem root.

        All yielded parents will end in "/".

        If the path is "/", yields the path itself.

        Otherwise, if the path ends in "/", does not yield the path itself.

        Parameters
            path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            Iterable[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.wdl.wdltoil.add_paths(task_container, host_paths)
        Based off of WDL.runtime.task_container.add_paths from miniwdl

        Comes up with a container path for each host path and fils in input_path_map and input_path_map_rev on the TaskContainer to map from host path to container path and visa versa.

        Makes sure directories have trailing slashes.

        Because of File and Directory sibling constraints, anything that's a child of something on the host needs to remain a child of the same thing in the container. MiniWDL's add_paths didn't do this.

        We also need to enforce that Directories that are at the top of the hierarchy of what's included are themselves siblings, if they were originally siblings.

        TODO: Deduplicate with the similar CWL mount deduplication code that's based on a notion of nonredundant mounts? But unlike that code, we want to list every File or Directory mentioned in the input, even if a mount is redundant. Probably. Because I'm not sure when/if the mappings we fill in are used for reverse lookups.

        Parameters

            •   task_container (WDL.runtime.task_container.TaskContainer)

            •   host_paths (Iterable[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            None

    toil.wdl.wdltoil.drop_if_missing(inode, standard_library)
        Return None if a File/Directory doesn't exist, or its path if it does.

        Parameters

            •   inode (WDLINode)

            •   standard_library (ToilWDLStdLibBase <#toil.wdl.wdltoil.ToilWDLStdLibBase>)

        Return type
            WDLINode | None

    toil.wdl.wdltoil.missing_inode_dropper(standard_library)
        Get a function to null out missing File/Directory values.

        A function to do this needs a standard library to get ahold of a current directory to use when resolving strings to paths.

        Parameters
            standard_library (ToilWDLStdLibBase <#toil.wdl.wdltoil.ToilWDLStdLibBase>)

        Return type
            INodeTransform <#toil.wdl.wdltoil.INodeTransform>

    toil.wdl.wdltoil.drop_missing_files(environment, standard_library)
        Make sure all the File values embedded in the given bindings point to files that exist, or are null.

        Files must not be virtualized.

        Parameters

            •   environment (WDLBindings)

            •   standard_library (ToilWDLStdLibBase <#toil.wdl.wdltoil.ToilWDLStdLibBase>)

        Return type
            WDLBindings

    toil.wdl.wdltoil.get_paths_in_bindings(environment)
        Get the paths of all Files and Directories in the bindings.

        Removes duplicates.

        TODO: Duplicative with WDL.runtime.task._fspaths.

        Parameters
            environment (WDLBindings)

        Return type
            list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.wdl.wdltoil.map_over_inodes_in_bindings(environment, transform)
        Run all File values embedded in the given bindings through the given transformation function.

        The transformation function must not mutate the original File.

        TODO: Replace with WDL.Value.rewrite_env_paths or WDL.Value.rewrite_files

        Parameters

            •   environment (WDLBindings)

            •   transform (INodeTransform <#toil.wdl.wdltoil.INodeTransform>)

        Return type
            WDLBindings

    toil.wdl.wdltoil.map_over_inodes_in_binding(binding, transform)
        Run all File values' types and values embedded in the given binding's value through the given transformation function.

        The transformation function must not mutate the original File.

        Parameters

            •   binding (WDL.Env.Binding[WDL.Value.Base])

            •   transform (INodeTransform <#toil.wdl.wdltoil.INodeTransform>)

        Return type
            WDL.Env.Binding[WDL.Value.Base]

    toil.wdl.wdltoil.remove_expr_from_value(value)
        Remove the expression from a WDL value :param value: Original WDL value :return: New WDL value without the expr field

        Parameters
            value (WDL.Value.Base)

        Return type
            WDL.Value.Base

    toil.wdl.wdltoil.map_over_typed_value(value, transform)
        Apply a transform to a WDL value and all contained WDL values. :param value: WDL value to transform :param transform: Function that takes a WDL value and returns a new WDL value :return: New transformed WDL value

        Parameters

            •   value (WDL.Value.Base)

            •   transform (Callable[[WDL.Value.Base], WDL.Value.Base])

        Return type
            WDL.Value.Base

    toil.wdl.wdltoil.map_over_typed_inodes_in_value(value, transform)
        Run all File values embedded in the given value through the given transformation function.

        The transformation function must not mutate the original File.

        If the transform returns None, the file value is changed to Null.

        The transform has access to the type information for the value, so it knows if it may return None, depending on if the value is optional or not.

        The transform is allowed to return None only if the mapping result won't actually be used, to allow for scans. So error checking needs to be part of the transform itself.

        Parameters

            •   value (WDL.Value.Base)

            •   transform (INodeTransform <#toil.wdl.wdltoil.INodeTransform>)

        Return type
            WDL.Value.Base

    toil.wdl.wdltoil.ensure_null_inodes_are_nullable(value, original_value, expected_type)
        Run through all nested values embedded in the given value and check that the null values are valid.

        If a null value is found that does not have a valid corresponding expected_type, raise an error

        (This is currently only used to check that null values arising from File/Directory coercion are in locations with a nullable type. If this is to be used elsewhere, the error message should be changed to describe the appropriate types and not just talk about files.)

        For example: If one of the nested values is null but the equivalent nested expected_type is not optional, a FileNotFoundError will be raised :param value: WDL base value to check. This is the WDL value that has been transformed and has the null elements :param original_value: The original WDL base value prior to the transformation. Only used for error messages :param expected_type: The WDL type of the value

        Parameters

            •   value (WDL.Value.Base)

            •   original_value (WDL.Value.Base)

            •   expected_type (WDL.Type.Base)

        Return type
            None

    class toil.wdl.wdltoil.WDLBaseJob(wdl_options, **kwargs)
        Bases: toil.job.Job <#toil.job.Job>

        Base job class for all WDL-related jobs.

        Responsible for post-processing returned bindings, to do things like add in null values for things not defined in a section. Post-processing operations can be added onto any job before it is saved, and will be applied as long as the job's run method calls postprocess().

        Also responsible for remembering the Toil WDL configuration keys and values.

        Parameters

            •   wdl_options (WDLContext)

            •   kwargs (Any)

        run(file_store)
            Run a WDL-related job.

            Remember to decorate non-trivial overrides with report_wdl_errors().

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                Any

        then_underlay(underlay)
            Apply an underlay of backup bindings to the result.

            Parameters
                underlay (toil.job.Promised[WDLBindings])

            Return type
                None

        then_remove(remove)
            Remove the given bindings from the result.

            Parameters
                remove (toil.job.Promised[WDLBindings])

            Return type
                None

        then_namespace(namespace)
            Put the result bindings into a namespace.

            Parameters
                namespace (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        then_overlay(overlay)
            Overlay the given bindings on top of the (possibly namespaced) result.

            Parameters
                overlay (toil.job.Promised[WDLBindings])

            Return type
                None

        remove_expr_from_bindings(bindings)

            Parameters
                bindings (WDLBindings)

            Return type
                WDLBindings

        postprocess(bindings)
            Apply queued changes to bindings.

            Should be applied by subclasses' run() implementations to their return values.

            Parameters
                bindings (WDLBindings)

            Return type
                WDLBindings

        defer_postprocessing(other)
            Give our postprocessing steps to a different job.

            Use this when you are returning a promise for bindings, on the job that issues the promise.

            Parameters
                other (WDLBaseJob <#toil.wdl.wdltoil.WDLBaseJob>)

            Return type
                None

    class toil.wdl.wdltoil.WDLTaskWrapperJob(task, prev_node_results, enclosing_bindings, task_id, wdl_options, **kwargs)
        Bases: WDLBaseJob

        Job that determines the resources needed to run a WDL job.

        Responsible for evaluating the input declarations for unspecified inputs, evaluating the runtime section, and scheduling or chaining to the real WDL job.

        All bindings are in terms of task-internal names.

        Parameters

            •   task (WDL.Tree.Task)

            •   prev_node_results (Sequence[toil.job.Promised[WDLBindings]])

            •   enclosing_bindings (WDLBindings)

            •   task_id (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   wdl_options (WDLContext)

            •   kwargs (Any)

        run(file_store)
            Evaluate inputs and runtime and schedule the task.

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                toil.job.Promised[WDLBindings]

    class toil.wdl.wdltoil.WDLTaskJob(task, task_internal_bindings, runtime_bindings, enclosing_bindings, task_id, mount_spec, wdl_options, cache_key=None, **kwargs)
        Bases: WDLBaseJob

        Job that runs a WDL task.

        Responsible for re-evaluating input declarations for unspecified inputs, evaluating the runtime section, re-scheduling if resources are not available, running any command, and evaluating the outputs.

        All bindings are in terms of task-internal names.

        Parameters

            •   task (WDL.Tree.Task)

            •   task_internal_bindings (toil.job.Promised[WDLBindings])

            •   runtime_bindings (toil.job.Promised[WDLBindings])

            •   enclosing_bindings (WDLBindings)

            •   task_id (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   mount_spec (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str> | None, int </usr/share/doc/python3/html/library/functions.html#int>])

            •   wdl_options (WDLContext)

            •   cache_key (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None)

            •   kwargs (Any)

        INJECTED_MESSAGE_DIR = '.toil_wdl_runtime'

        add_injections(command_string, task_container)
            Inject extra Bash code from the Toil WDL runtime into the command for the container.

            Currently doesn't implement the MiniWDL plugin system, but does add resource usage monitoring to Docker containers.

            Parameters

                •   command_string (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

                •   task_container (WDL.runtime.task_container.TaskContainer)

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        handle_injection_messages(outputs_library)
            Handle any data received from injected runtime code in the container.

            Parameters
                outputs_library (ToilWDLStdLibTaskOutputs <#toil.wdl.wdltoil.ToilWDLStdLibTaskOutputs>)

            Return type
                None

        handle_message_file(file_path)
            Handle a message file received from in-container injected code.

            Takes the host-side path of the file.

            Parameters
                file_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                None

        can_fake_root()
            Determine if --fakeroot is likely to work for Singularity.

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        can_mount_proc()
            Determine if --containall will work for Singularity. On Kubernetes, this will result in operation not permitted See: <https://github.com/apptainer/singularity/issues/5857>

            So if Kubernetes is detected, return False :return: bool

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        ensure_mount_point(file_store, mount_spec)
            Ensure the mount point sources are available.

            Will check if the mount point source has the requested amount of space available.

            Note: We are depending on Toil's job scheduling backend to error when the sum of multiple mount points disk requests is greater than the total available For example, if a task has two mount points request 100 GB each but there is only 100 GB available, the df check may pass but Toil should fail to schedule the jobs internally

            Parameters

                •   mount_spec (dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str> | None, int </usr/share/doc/python3/html/library/functions.html#int>]) -- Mount specification from the disks attribute in the WDL task. Is a dict where key is the mount point target and value is the size

                •   file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>) -- File store to create a tmp directory for the mount point source

            Returns
                Dict mapping mount point target to mount point source

            Return type
                dict </usr/share/doc/python3/html/library/stdtypes.html#dict>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        run(file_store)
            Actually run the task.

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                toil.job.Promised[WDLBindings]

    class toil.wdl.wdltoil.WDLWorkflowNodeJob(node, prev_node_results, wdl_options, **kwargs)
        Bases: WDLBaseJob

        Job that evaluates a WDL workflow node.

        Parameters

            •   node (WDL.Tree.WorkflowNode)

            •   prev_node_results (Sequence[toil.job.Promised[WDLBindings]])

            •   wdl_options (WDLContext)

            •   kwargs (Any)

        run(file_store)
            Actually execute the workflow node.

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                toil.job.Promised[WDLBindings]

    class toil.wdl.wdltoil.WDLWorkflowNodeListJob(nodes, prev_node_results, wdl_options, **kwargs)
        Bases: WDLBaseJob

        Job that evaluates a list of WDL workflow nodes, which are in the same scope and in a topological dependency order, and which do not call out to any other workflows or tasks or sections.

        Parameters

            •   nodes (list </usr/share/doc/python3/html/library/stdtypes.html#list>[WDL.Tree.WorkflowNode])

            •   prev_node_results (Sequence[toil.job.Promised[WDLBindings]])

            •   wdl_options (WDLContext)

            •   kwargs (Any)

        run(file_store)
            Actually execute the workflow nodes.

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                toil.job.Promised[WDLBindings]

    class toil.wdl.wdltoil.WDLCombineBindingsJob(prev_node_results, **kwargs)
        Bases: WDLBaseJob

        Job that collects the results from WDL workflow nodes and combines their environment changes.

        Parameters

            •   prev_node_results (Sequence[toil.job.Promised[WDLBindings]])

            •   kwargs (Any)

        run(file_store)
            Aggregate incoming results.

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                WDLBindings

    class toil.wdl.wdltoil.WDLWorkflowGraph(nodes)
        Represents a graph of WDL WorkflowNodes.

        Operates at a certain level of instantiation (i.e. sub-sections are represented by single nodes).

        Assumes all relevant nodes are provided; dependencies outside the provided nodes are assumed to be satisfied already.

        Parameters
            nodes (Sequence[WDL.Tree.WorkflowNode])

        real_id(node_id)
            Map multiple IDs for what we consider the same node to one ID.

            This elides/resolves gathers.

            Parameters
                node_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                str </usr/share/doc/python3/html/library/stdtypes.html#str>

        is_decl(node_id)
            Return True if a node represents a WDL declaration, and false otherwise.

            Parameters
                node_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                bool </usr/share/doc/python3/html/library/functions.html#bool>

        get(node_id)
            Get a node by ID.

            Parameters
                node_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                WDL.Tree.WorkflowNode

        get_dependencies(node_id)
            Get all the nodes that a node depends on, recursively (into the node if it has a body) but not transitively.

            Produces dependencies after resolving gathers and internal-to-section dependencies, on nodes that are also in this graph.

            Parameters
                node_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        get_transitive_dependencies(node_id)
            Get all the nodes that a node depends on, transitively.

            Parameters
                node_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            Return type
                set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        topological_order()
            Get a topological order of the nodes, based on their dependencies.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

        leaves()
            Get all the workflow node IDs that have no dependents in the graph.

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    class toil.wdl.wdltoil.WDLSectionJob(wdl_options, **kwargs)
        Bases: WDLBaseJob

        Job that can create more graph for a section of the workflow.

        Parameters

            •   wdl_options (WDLContext)

            •   kwargs (Any)

        static coalesce_nodes(order, section_graph)
            Given a topological order of WDL workflow node IDs, produce a list of lists of IDs, still in topological order, where each list of IDs can be run under a single Toil job.

            Parameters

                •   order (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

                •   section_graph (WDLWorkflowGraph <#toil.wdl.wdltoil.WDLWorkflowGraph>)

            Return type
                list </usr/share/doc/python3/html/library/stdtypes.html#list>[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]

        create_subgraph(nodes, gather_nodes, environment, local_environment=None, subscript=None)
            Make a Toil job to evaluate a subgraph inside a workflow or workflow section.

            Returns
                a child Job that will return the aggregated environment after running all the things in the section.

            Parameters

                •   gather_nodes (Sequence[WDL.Tree.Gather]) -- Names exposed by these will always be defined with something, even if the code that defines them does not actually run.

                •   environment (WDLBindings) -- Bindings in this environment will be used to evaluate the subgraph and will be passed through.

                •   local_environment (WDLBindings | None) -- Bindings in this environment will be used to evaluate the subgraph but will go out of scope at the end of the section.

                •   subscript (int </usr/share/doc/python3/html/library/functions.html#int> | None) -- If the subgraph is being evaluated multiple times, this should be a disambiguating integer for logging.

                •   nodes (Sequence[WDL.Tree.WorkflowNode])

            Return type
                WDLBaseJob <#toil.wdl.wdltoil.WDLBaseJob>

        make_gather_bindings(gathers, undefined)
            Given a collection of Gathers, create bindings from every identifier gathered, to the given "undefined" placeholder (which would be Null for a single execution of the body, or an empty array for a completely unexecuted scatter).

            These bindings can be overlaid with bindings from the actual execution, so that references to names defined in unexecuted code get a proper default undefined value, and not a KeyError at runtime.

            The information to do this comes from MiniWDL's "gathers" system: <<https://miniwdl.readthedocs.io/en/latest/WDL.html#WDL.Tree.WorkflowSection.gathers>>

            TODO: This approach will scale O(n^2) when run on n nested conditionals, because generating these bindings for the outer conditional will visit all the bindings from the inner ones.

            Parameters

                •   gathers (Sequence[WDL.Tree.Gather])

                •   undefined (WDL.Value.Base)

            Return type
                WDLBindings

    class toil.wdl.wdltoil.WDLScatterJob(scatter, prev_node_results, wdl_options, **kwargs)
        Bases: WDLSectionJob

        Job that evaluates a scatter in a WDL workflow. Runs the body for each value in an array, and makes arrays of the new bindings created in each instance of the body. If an instance of the body doesn't create a binding, it gets a null value in the corresponding array.

        Parameters

            •   scatter (WDL.Tree.Scatter)

            •   prev_node_results (Sequence[toil.job.Promised[WDLBindings]])

            •   wdl_options (WDLContext)

            •   kwargs (Any)

        run(file_store)
            Run the scatter.

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                toil.job.Promised[WDLBindings]

    class toil.wdl.wdltoil.WDLArrayBindingsJob(input_bindings, base_bindings, **kwargs)
        Bases: WDLBaseJob

        Job that takes all new bindings created in an array of input environments, relative to a base environment, and produces bindings where each new binding name is bound to an array of the values in all the input environments.

        Useful for producing the results of a scatter.

        Parameters

            •   input_bindings (Sequence[toil.job.Promised[WDLBindings]])

            •   base_bindings (WDLBindings)

            •   kwargs (Any)

        run(file_store)
            Actually produce the array-ified bindings now that promised values are available.

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                WDLBindings

    class toil.wdl.wdltoil.WDLConditionalJob(conditional, prev_node_results, wdl_options, **kwargs)
        Bases: WDLSectionJob

        Job that evaluates a conditional in a WDL workflow.

        Parameters

            •   conditional (WDL.Tree.Conditional)

            •   prev_node_results (Sequence[toil.job.Promised[WDLBindings]])

            •   wdl_options (WDLContext)

            •   kwargs (Any)

        run(file_store)
            Run the conditional.

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                toil.job.Promised[WDLBindings]

    class toil.wdl.wdltoil.WDLWorkflowJob(workflow, prev_node_results, enclosing_bindings, workflow_id, wdl_options, **kwargs)
        Bases: WDLSectionJob

        Job that evaluates an entire WDL workflow.

        Parameters

            •   workflow (WDL.Tree.Workflow)

            •   prev_node_results (Sequence[toil.job.Promised[WDLBindings]])

            •   enclosing_bindings (WDLBindings)

            •   workflow_id (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   wdl_options (WDLContext)

            •   kwargs (Any)

        run(file_store)
            Run the workflow. Return the result of the workflow.

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                toil.job.Promised[WDLBindings]

    class toil.wdl.wdltoil.WDLOutputsJob(workflow, bindings, enclosing_bindings, wdl_options, cache_key=None, **kwargs)
        Bases: WDLBaseJob

        Job which evaluates an outputs section for a workflow.

        Returns an environment with just the outputs bound, in no namespace.

        Parameters

            •   workflow (WDL.Tree.Workflow)

            •   bindings (toil.job.Promised[WDLBindings])

            •   enclosing_bindings (WDLBindings)

            •   wdl_options (WDLContext)

            •   cache_key (str </usr/share/doc/python3/html/library/stdtypes.html#str> | None)

            •   kwargs (Any)

        run(file_store)
            Make bindings for the outputs.

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                WDLBindings

    class toil.wdl.wdltoil.WDLStartJob(target, inputs, wdl_options, **kwargs)
        Bases: WDLSectionJob

        Job that evaluates an entire WDL workflow, and returns the workflow outputs namespaced with the workflow name. Inputs may or may not be namespaced with the workflow name; both forms are accepted.

        Parameters

            •   target (WDL.Tree.Workflow | WDL.Tree.Task)

            •   inputs (toil.job.Promised[WDLBindings])

            •   wdl_options (WDLContext)

            •   kwargs (Any)

        run(file_store)
            Actually build the subgraph.

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                toil.job.Promised[WDLBindings]

    class toil.wdl.wdltoil.WDLInstallImportsJob(task_path, inputs, import_data, **kwargs)
        Bases: toil.job.Job <#toil.job.Job>

        Class represents a unit of work in toil.

        Parameters

            •   task_path (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

            •   inputs (WDLBindings)

            •   import_data (toil.job.Promised[Tuple[Dict[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.fileStores.FileID <#toil.fileStores.FileID>], Dict[str </usr/share/doc/python3/html/library/stdtypes.html#str>, toil.job.FileMetadata <#toil.job.FileMetadata>]]])

            •   kwargs (Any)

        run(file_store)
            Convert the filenames in the workflow inputs ito the URIs :return: Promise of transformed workflow inputs

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                toil.job.Promised[WDLBindings]

    class toil.wdl.wdltoil.WDLImportWrapper(target, inputs, wdl_options, inputs_search_path, import_remote_files, import_workers_batchsize, import_workers_disk, **kwargs)
        Bases: WDLSectionJob

        Job to organize importing files on workers instead of the leader. Responsible for extracting filenames and metadata, calling ImportsJob, applying imports to input bindings, and scheduling the start workflow job

        This class is only used when runImportsOnWorkers is enabled.

        Parameters

            •   target (Union[WDL.Tree.Workflow, WDL.Tree.Task])

            •   inputs (WDLBindings)

            •   wdl_options (WDLContext)

            •   inputs_search_path (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   import_remote_files (bool </usr/share/doc/python3/html/library/functions.html#bool>)

            •   import_workers_batchsize (toil.job.ParseableIndivisibleResource)

            •   import_workers_disk (toil.job.ParseableIndivisibleResource)

            •   kwargs (Any)

        run(file_store)
            Run a WDL-related job.

            Remember to decorate non-trivial overrides with report_wdl_errors().

            Parameters
                file_store (toil.fileStores.abstractFileStore.AbstractFileStore <#toil.fileStores.abstractFileStore.AbstractFileStore>)

            Return type
                toil.job.Promised[WDLBindings]

    toil.wdl.wdltoil.make_root_job(target, inputs, inputs_search_path, toil, wdl_options, options)

        Parameters

            •   target (WDL.Tree.Workflow | WDL.Tree.Task)

            •   inputs (WDLBindings)

            •   inputs_search_path (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

            •   toil (toil.common.Toil <#toil.common.Toil>)

            •   wdl_options (WDLContext)

            •   options (configargparse.Namespace)

        Return type
            WDLSectionJob <#toil.wdl.wdltoil.WDLSectionJob>

    toil.wdl.wdltoil.main()
        A Toil workflow to interpret WDL input files.

        Return type
            None

  toil.worker

  Attributes
logger |

  Classes
StatsDict | Subclass of MagicExpando for type-checking purposes.

  Functions
nextChainable(predecessor, job_store, config) | Returns the next chainable job's JobDescription after the given predecessor
unstick_worker([interval, timeout]) | Thread function that tries to prevent the process from getting stuck.
workerScript(job_store, config, job_name, job_store_id) | Worker process script, runs a job.
parse_args(args) | Parse command-line arguments to the worker.
in_contexts(contexts) | Unpickle and enter all the pickled, base64-encoded context managers in the
main([argv]) |

  Module Contents

    toil.worker.logger

    class toil.worker.StatsDict(*args, **kwargs)
        Bases: toil.lib.expando.MagicExpando <#toil.lib.expando.MagicExpando>

        Subclass of MagicExpando for type-checking purposes.

        jobs: list </usr/share/doc/python3/html/library/stdtypes.html#list>[toil.lib.expando.MagicExpando <#toil.lib.expando.MagicExpando>]

    toil.worker.nextChainable(predecessor, job_store, config)
        Returns the next chainable job's JobDescription after the given predecessor JobDescription, if one exists, or None if the chain must terminate.

        Parameters

            •   predecessor (toil.job.JobDescription <#toil.job.JobDescription>) -- The job to chain from

            •   job_store (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>) -- The JobStore to fetch JobDescriptions from.

            •   config (toil.common.Config <#toil.common.Config>) -- The configuration for the current run.

        Return type
            Optional[toil.job.JobDescription <#toil.job.JobDescription>]

    toil.worker.unstick_worker(interval=120, timeout=120)
        Thread function that tries to prevent the process from getting stuck.

        Meant to be used as a daemon thread: does not have a shutdown signal but cleans up on exit.

        Parameters

            •   interval (float </usr/share/doc/python3/html/library/functions.html#float>) -- Try to unstick the process at intervals of this many seconds.

            •   timeout (float </usr/share/doc/python3/html/library/functions.html#float>) -- Stop child processes that take longer than this many seconds to finish.

        Return type
            None

    toil.worker.workerScript(job_store, config, job_name, job_store_id, redirect_output_to_log_file=True, local_worker_temp_dir=None, debug_flags=None)
        Worker process script, runs a job.

        Parameters

            •   job_store (toil.jobStores.abstractJobStore.AbstractJobStore <#toil.jobStores.abstractJobStore.AbstractJobStore>) -- The JobStore to fetch JobDescriptions from.

            •   config (toil.common.Config <#toil.common.Config>) -- The configuration for the current run.

            •   job_name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The "job name" (a user friendly name) of the job to be run

            •   job_store_id (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The job store ID of the job to be run

            •   redirect_output_to_log_file (bool </usr/share/doc/python3/html/library/functions.html#bool>) -- If False, log directly to the console instead of capturing job output.

            •   local_worker_temp_dir (Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]) -- The directory for the worker to work in. May be recursively removed after the job runs.

            •   debug_flags (Optional[set </usr/share/doc/python3/html/library/stdtypes.html#set>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]]) -- Flags to set on each job before running it.

        Return int
            1 if a job failed, or 0 if all jobs succeeded

        Return type
            int </usr/share/doc/python3/html/library/functions.html#int>

    toil.worker.parse_args(args)
        Parse command-line arguments to the worker.

        Parameters
            args (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            Any

    toil.worker.in_contexts(contexts)
        Unpickle and enter all the pickled, base64-encoded context managers in the given list. Then do the body, then leave them all.

        Parameters
            contexts (list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>])

        Return type
            collections.abc.Iterator </usr/share/doc/python3/html/library/collections.abc.html#collections.abc.Iterator>[None]

    toil.worker.main(argv=None)

        Parameters
            argv (Optional[list </usr/share/doc/python3/html/library/stdtypes.html#list>[str </usr/share/doc/python3/html/library/stdtypes.html#str>]])

        Return type
            None

  Attributes
log |
KNOWN_EXTANT_IMAGES |

  Exceptions
ApplianceImageNotFound | Error raised when using TOIL_APPLIANCE_SELF results in an HTTP error.

  Functions
which(cmd[, mode, path]) | Return the path with conforms to the given mode on the Path.
toilPackageDirPath() | Return the absolute path of the directory that corresponds to the top-level toil package.
inVirtualEnv() | Test if we are inside a virtualenv or Conda virtual environment.
resolveEntryPoint(entryPoint) | Find the path to the given entry point that should work on a worker.
physicalMemory() | Calculate the total amount of physical memory, in bytes.
physicalDisk(directory) |
applianceSelf([forceDockerAppliance]) | Return the fully qualified name of the Docker image to start Toil appliance containers from.
customDockerInitCmd() | Return the custom command set by the TOIL_CUSTOM_DOCKER_INIT_COMMAND environment variable.
customInitCmd() | Return the custom command set by the TOIL_CUSTOM_INIT_COMMAND environment variable.
lookupEnvVar(name, envName, defaultValue) | Look up environment variables that control Toil and log the result.
checkDockerImageExists(appliance) | Attempt to check a url registryName for the existence of a docker image with a given tag.
parseDockerAppliance(appliance) | Derive parsed registry, image reference, and tag from a docker image string.
checkDockerSchema(appliance) |
requestCheckRegularDocker(origAppliance, registryName, ...) | Check if an image exists using the requests library.
requestCheckDockerIo(origAppliance, imageName, tag) | Check docker.io to see if an image exists using the requests library.
logProcessContext(config) |

  Package Contents

    toil.log

    toil.which(cmd, mode=os.F_OK | os.X_OK, path=None)
        Return the path with conforms to the given mode on the Path.

        [Copy-pasted in from python3.6's shutil.which().]

        mode defaults to os.F_OK | os.X_OK. path defaults to the result of os.environ.get("PATH"), or can be overridden with a custom search path.

        Returns
            The path found, or None.

        Return type
            Optional[str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.toilPackageDirPath()
        Return the absolute path of the directory that corresponds to the top-level toil package.

        The return value is guaranteed to end in '/toil'.

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.inVirtualEnv()
        Test if we are inside a virtualenv or Conda virtual environment.

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.resolveEntryPoint(entryPoint)
        Find the path to the given entry point that should work on a worker.

        Returns
            The path found, which may be an absolute or a relative path.

        Parameters
            entryPoint (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.physicalMemory()
        Calculate the total amount of physical memory, in bytes.

            >>> n = physicalMemory()
            >>> n > 0
            True
            >>> n == physicalMemory()
            True

        Return type
            int </usr/share/doc/python3/html/library/functions.html#int>

    toil.physicalDisk(directory)

        Parameters
            directory (str </usr/share/doc/python3/html/library/stdtypes.html#str>)

        Return type
            int </usr/share/doc/python3/html/library/functions.html#int>

    toil.applianceSelf(forceDockerAppliance=False)
        Return the fully qualified name of the Docker image to start Toil appliance containers from.

        The result is determined by the current version of Toil and three environment variables: TOIL_DOCKER_REGISTRY, TOIL_DOCKER_NAME and TOIL_APPLIANCE_SELF.

        TOIL_DOCKER_REGISTRY specifies an account on a publicly hosted docker registry like Quay or Docker Hub. The default is UCSC's CGL account on Quay.io where the Toil team publishes the official appliance images. TOIL_DOCKER_NAME specifies the base name of the image. The default of toil will be adequate in most cases. TOIL_APPLIANCE_SELF fully qualifies the appliance image, complete with registry, image name and version tag, overriding both TOIL_DOCKER_NAME and TOIL_DOCKER_REGISTRY` as well as the version tag of the image. Setting TOIL_APPLIANCE_SELF will not be necessary in most cases.

        Parameters
            forceDockerAppliance (bool </usr/share/doc/python3/html/library/functions.html#bool>)

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.customDockerInitCmd()
        Return the custom command set by the TOIL_CUSTOM_DOCKER_INIT_COMMAND environment variable.

        The custom docker command is run prior to running the workers and/or the primary node's services.

        This can be useful for doing any custom initialization on instances (e.g. authenticating to private docker registries). Any single quotes are escaped and the command cannot contain a set of blacklisted chars (newline or tab).

        Returns
            The custom command, or an empty string is returned if the environment variable is not set.

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.customInitCmd()
        Return the custom command set by the TOIL_CUSTOM_INIT_COMMAND environment variable.

        The custom init command is run prior to running Toil appliance itself in workers and/or the primary node (i.e. this is run one stage before TOIL_CUSTOM_DOCKER_INIT_COMMAND).

        This can be useful for doing any custom initialization on instances (e.g. authenticating to private docker registries). Any single quotes are escaped and the command cannot contain a set of blacklisted chars (newline or tab).

        returns: the custom command or n empty string is returned if the environment variable is not set.

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.lookupEnvVar(name, envName, defaultValue)
        Look up environment variables that control Toil and log the result.

        Parameters

            •   name (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the human readable name of the variable

            •   envName (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the name of the environment variable to lookup

            •   defaultValue (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- the fall-back value

        Returns
            the value of the environment variable or the default value the variable is not set

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.checkDockerImageExists(appliance)
        Attempt to check a url registryName for the existence of a docker image with a given tag.

        Parameters
            appliance (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The url of a docker image's registry (with a tag) of the form: 'quay.io/<repo_path>:<tag>' or '<repo_path>:<tag>'. Examples: 'quay.io/ucsc_cgl/toil:latest', 'ubuntu:latest', or 'broadinstitute/genomes-in-the-cloud:2.0.0'.

        Returns
            Raises an exception if the docker image cannot be found or is invalid. Otherwise, it will return the appliance string.

        Return type
            str </usr/share/doc/python3/html/library/stdtypes.html#str>

    toil.parseDockerAppliance(appliance)
        Derive parsed registry, image reference, and tag from a docker image string.

        Example: "quay.io/ucsc_cgl/toil:latest" Should return: "quay.io", "ucsc_cgl/toil", "latest"

        If a registry is not defined, the default is: "docker.io" If a tag is not defined, the default is: "latest"

        Parameters
            appliance (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The full url of the docker image originally specified by the user (or the default). e.g. "quay.io/ucsc_cgl/toil:latest"

        Returns
            registryName, imageName, tag

        Return type
            tuple </usr/share/doc/python3/html/library/stdtypes.html#tuple>[str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>, str </usr/share/doc/python3/html/library/stdtypes.html#str>]

    toil.checkDockerSchema(appliance)

    exception toil.ApplianceImageNotFound(origAppliance, url, statusCode)
        Bases: docker.errors.ImageNotFound

        Error raised when using TOIL_APPLIANCE_SELF results in an HTTP error.

        Parameters

            •   origAppliance (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The full url of the docker image originally specified by the user (or the default). e.g. "quay.io/ucsc_cgl/toil:latest"

            •   url (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The URL at which the image's manifest is supposed to appear

            •   statusCode (int </usr/share/doc/python3/html/library/functions.html#int>) -- the failing HTTP status code returned by the URL

    toil.KNOWN_EXTANT_IMAGES

    toil.requestCheckRegularDocker(origAppliance, registryName, imageName, tag)
        Check if an image exists using the requests library.

        URL is based on the docker v2 schema <https://docs.docker.com/registry/spec/manifest-v2-2/>.

        This has the following format: https://{websitehostname}.io/v2/{repo}/manifests/{tag}

        Does not work with the official (docker.io) site, because they require an OAuth token, so a separate check is done for docker.io images.

        Parameters

            •   origAppliance (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The full url of the docker image originally specified by the user (or the default). For example, quay.io/ucsc_cgl/toil:latest.

            •   registryName (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The url of a docker image's registry. For example, quay.io.

            •   imageName (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The image, including path and excluding the tag. For example, ucsc_cgl/toil.

            •   tag (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The tag used at that docker image's registry. For example, latest.

        Raises
            ApplianceImageNotFound if no match is found.

        Returns
            Return True if match found.

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.requestCheckDockerIo(origAppliance, imageName, tag)
        Check docker.io to see if an image exists using the requests library.

        URL is based on the docker v2 schema. Requires that an access token be fetched first.

        Parameters

            •   origAppliance (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The full url of the docker image originally specified by the user (or the default). For example, ubuntu:latest.

            •   imageName (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The image, including path and excluding the tag. For example, ubuntu.

            •   tag (str </usr/share/doc/python3/html/library/stdtypes.html#str>) -- The tag used at that docker image's registry. For example, latest.

        Raises
            ApplianceImageNotFound if no match is found.

        Returns
            Return True if match found.

        Return type
            bool </usr/share/doc/python3/html/library/functions.html#bool>

    toil.logProcessContext(config)

        Parameters
            config (common.Config <#toil.common.Config>)

        Return type
            None

[1] Created with sphinx-autoapi <https://github.com/readthedocs/sphinx-autoapi>

    •   Index <>

    •   Search Page <>

Author
UCSC Computational Genomics Lab

Copyright
2015 – 2026 UCSC Computational Genomics Lab