# OAR 2.6 documentation

## OAR Use Cases

## Interactive jobs

### Job submission

    jdoe@idpot:~$ oarsub -I -l /nodes=3/core=1
    [ADMISSION RULE] Set default walltime to 7200.
    [ADMISSION RULE] Modify resource description with type constraints
    OAR_JOB_ID=4924
    Interactive mode : waiting...
    [2007-03-07 08:51:04] Starting...

    Connect to OAR job 4924 via the node idpot5.grenoble.grid5000.fr
    jdoe@idpot5:~$

### Connecting to the other cores

    jdoe@idpot5:~$ cat $OAR_NODEFILE
    idpot5.grenoble.grid5000.fr
    idpot8.grenoble.grid5000.fr
    idpot9.grenoble.grid5000.fr
    jdoe@idpot5:~$ oarsh idpot8
    Last login: Tue Mar  6 18:00:37 2007 from idpot.imag.fr
    jdoe@idpot8:~$ oarsh idpot9
    Last login: Wed Mar  7 08:48:30 2007 from idpot.imag.fr
    jdoe@idpot9:~$ oarsh idpot5
    Last login: Wed Mar  7 08:51:45 2007 from idpot5.imag.fr
    jdoe@idpot5:~$

### Copying a file from one node to another

    jdoe@idpot5:~$ hostname > /tmp/my_hostname
    jdoe@idpot5:~$ oarcp /tmp/my_hostname idpot8:/tmp/my_hostname
    jdoe@idpot5:~$ oarsh idpot8 cat /tmp/my_hostname
    idpot5
    jdoe@idpot5:~$

### Connecting to our job

    jdoe@idpot:~$ OAR_JOB_ID=4924 oarsh idpot9
    Last login: Wed Mar  7 08:52:09 2007 from idpot8.imag.fr
    jdoe@idpot9:~$ oarsh idpot5
    Last login: Wed Mar  7 08:52:18 2007 from idpot9.imag.fr
    jdoe@idpot5:~$

## Batch mode job

### Submission using a script

    jdoe@paramount:~$ oarsub -l core=10 ./runhpl
    Generate a job key...
    [ADMISSION RULE] Set default walltime to 3600.
    [ADMISSION RULE] Modify resource description with type constraints
    OAR_JOB_ID=199522

### Watching results

    jdoe@paramount:~$ cat OAR.199522.stdout
    ...

### Submission using an inline command

Sometimes it is very useful to run a little command in oarsub:

    jdoe@paramount:~$ oarsub -l core=1 'echo $PATH;which ssh'
    Generate a job key...
    [ADMISSION RULE] Set default walltime to 3600.
    [ADMISSION RULE] Modify resource description with type constraints
    OAR_JOB_ID=199523

### Watching results

    jdoe@paramount:~$ cat OAR.199523.stdout
    ...

## Reservations

The date format to pass to the -r option is YYYY-MM-DD HH:MM:SS:

    jdoe@paramount:~$ oarsub -l core=10 ./runhpl -r "2007-10-10 18:00:00"
    Generate a job key...
    [ADMISSION RULE] Set default walltime to 3600.
    [ADMISSION RULE] Modify resource description with type constraints
    OAR_JOB_ID=199524
    Reservation mode : waiting validation...
    Reservation valid --> OK
    jdoe@paramount:~$

## Examples of resource requests

### Using the resource hierarchy

- ask for 1 core on 15 nodes on a same cluster (total = 15 cores)

    oarsub -I -l /cluster=1/nodes=15/core=1

- ask for 1 core on 15 nodes on 2 clusters (total = 30 cores)

    oarsub -I -l /cluster=2/nodes=15/core=1

- ask for 1 core on 2 cpus on 15 nodes on a same cluster (total = 30 cores)

    oarsub -I -l /cluster=1/nodes=15/cpu=2/core=1

- ask for 10 cpus on 2 clusters (total = 20 cpus, information regarding the node ou core count depend on the topology of the machines)

    oarsub -I -l /cluster=2/cpu=10

- ask for 1 core on 3 different network switches (total = 3 cores)

    oarsub -I -l /switch=3/core=1

### Using properties

See OAR properties for a description of all available properties, and watch Monika.

- ask for 10 cores of the cluster azur

    oarsub -I -l core=10 -p "cluster='azur'"

- ask for 2 nodes with 4096 GB of memory and Infiniband 10G

    oarsub -I -p "memnode=4096 and ib10g='YES'" -l nodes=2

- ask for any 4 nodes except gdx-45

    oarsub -I -p "not host like 'gdx-45.%'" -l nodes=4

### Mixing every together

- ask for 1 core on 2 nodes on the same cluster with 4096 GB of memory and Infiniband 10G + 1 cpu on 2 nodes on the same switch with bicore processors for a walltime of 4 hours

    oarsub -I -l "{memnode=4096 and ib10g='YES'}/cluster=1/nodes=2/core=1+{nbcore=2}/switch=1/nodes=2/cpu=1,walltime=4:0:0"

#### Warning

1. walltime must always be the last argument of -l <…>

2. if no resource matches your request, oarsub will exit with the message

    Generate a job key...
    [ADMISSION RULE] Set default walltime to 3600.
    [ADMISSION RULE] Modify resource description with type constraints
    There are not enough resources for your request
    OAR_JOB_ID=-5
    Oarsub failed: please verify your request syntax or ask for support to your admin.

### Moldable jobs

- ask for 4 nodes and a walltime of 2 hours or 2 nodes and a walltime of 4 hours

    oarsub -I -l nodes=4,walltime=2 -l nodes=2,walltime=4

### Types of job

OAR features the concept of job “type”. For example:

- submit besteffort jobs

    for param in $(< ./paramlist); do
        oarsub -t besteffort -l core=1 "./my_script.sh $param"
    done

- ask for 4 nodes on the same cluster in order to deploy a customized environment:

    oarsub -I -l cluster=1/nodes=4,walltime=6 -t deploy

Check the man of oarsub to get the other job types.

## X11 forwarding

If you have a DISPLAY configured in your shell then oarsub will automatically forward the X11 to it.

For example:

    jdoe@idpot:~$ oarsub -I -l /nodes=2/core=1
    OAR_JOB_ID=4926
    Interactive mode : waiting...
    [2007-03-07 09:01:16] Starting...

    Initialize X11 forwarding...
    Connect to OAR job 4926 via the node idpot8.grenoble.grid5000.fr
    jdoe@idpot8:~$ xeyes &
    [1] 14656
    jdoe@idpot8:~$ cat $OAR_NODEFILE
    idpot8.grenoble.grid5000.fr
    idpot9.grenoble.grid5000.fr
    [1]+  Done                    xeyes
    jdoe@idpot8:~$ oarsh idpot9 xeyes
    Error: Can't open display:
    jdoe@idpot8:~$ oarsh -X idpot9 xeyes

## Using a parallel launcher: taktuk

Warning: Taktuk MUST BE installed on all nodes to test this point

### Shell 1

#### Unset DISPLAY so that X does not bother…

    jdoe@idpot:~$ unset DISPLAY

#### Job submission

    jdoe@idpot:~$ oarsub -I -l /nodes=20/core=1
    [ADMISSION RULE] Set default walltime to 7200.
    [ADMISSION RULE] Modify resource description with type constraints
    OAR_JOB_ID=4930
    Interactive mode : waiting...
    [2007-03-07 09:15:13] Starting...

    Connect to OAR job 4930 via the node idpot1.grenoble.grid5000.fr

#### Running the taktuk command

    jdoe@idpot1:~$ taktuk -c "oarsh" -f $OAR_FILE_NODES broadcast exec [ date ]
    idcalc12.grenoble.grid5000.fr-1: date (11567): output > Thu May  3 18:56:58 CEST 2007
    idcalc12.grenoble.grid5000.fr-1: date (11567): status > Exited with status 0
    idcalc4.grenoble.grid5000.fr-8: date (31172): output > Thu May  3 19:00:09 CEST 2007
    idcalc2.grenoble.grid5000.fr-2: date (32368): output > Thu May  3 19:01:56 CEST 2007
    idcalc3.grenoble.grid5000.fr-5: date (31607): output > Thu May  3 18:56:44 CEST 2007
    idcalc3.grenoble.grid5000.fr-5: date (31607): status > Exited with status 0
    idcalc7.grenoble.grid5000.fr-13: date (31188): output > Thu May  3 18:59:54 CEST 2007
    idcalc9.grenoble.grid5000.fr-15: date (32426): output > Thu May  3 18:56:45 CEST 2007
    idpot6.grenoble.grid5000.fr-20: date (16769): output > Thu May  3 18:59:54 CEST 2007
    idcalc4.grenoble.grid5000.fr-8: date (31172): status > Exited with status 0
    idcalc5.grenoble.grid5000.fr-9: date (10288): output > Thu May  3 18:56:39 CEST 2007
    idcalc5.grenoble.grid5000.fr-9: date (10288): status > Exited with status 0
    idcalc6.grenoble.grid5000.fr-11: date (11290): output > Thu May  3 18:57:52 CEST 2007
    idcalc6.grenoble.grid5000.fr-11: date (11290): status > Exited with status 0
    idcalc7.grenoble.grid5000.fr-13: date (31188): status > Exited with status 0
    idcalc8.grenoble.grid5000.fr-14: date (10450): output > Thu May  3 18:57:34 CEST 2007
    idcalc8.grenoble.grid5000.fr-14: date (10450): status > Exited with status 0
    idcalc9.grenoble.grid5000.fr-15: date (32426): status > Exited with status 0
    idpot1.grenoble.grid5000.fr-16: date (18316): output > Thu May  3 18:57:19 CEST 2007
    idpot1.grenoble.grid5000.fr-16: date (18316): status > Exited with status 0
    idpot10.grenoble.grid5000.fr-17: date (31547): output > Thu May  3 18:56:27 CEST 2007
    idpot10.grenoble.grid5000.fr-17: date (31547): status > Exited with status 0
    idpot2.grenoble.grid5000.fr-18: date (407): output > Thu May  3 18:56:21 CEST 2007
    idpot2.grenoble.grid5000.fr-18: date (407): status > Exited with status 0
    idpot4.grenoble.grid5000.fr-19: date (2229): output > Thu May  3 18:55:37 CEST 2007
    idpot4.grenoble.grid5000.fr-19: date (2229): status > Exited with status 0
    idpot6.grenoble.grid5000.fr-20: date (16769): status > Exited with status 0
    idcalc2.grenoble.grid5000.fr-2: date (32368): status > Exited with status 0
    idpot11.grenoble.grid5000.fr-6: date (12319): output > Thu May  3 18:59:54 CEST 2007
    idpot7.grenoble.grid5000.fr-10: date (7355): output > Thu May  3 18:57:39 CEST 2007
    idpot5.grenoble.grid5000.fr-12: date (13093): output > Thu May  3 18:57:23 CEST 2007
    idpot3.grenoble.grid5000.fr-3: date (509): output > Thu May  3 18:59:55 CEST 2007
    idpot3.grenoble.grid5000.fr-3: date (509): status > Exited with status 0
    idpot8.grenoble.grid5000.fr-4: date (13252): output > Thu May  3 18:56:32 CEST 2007
    idpot8.grenoble.grid5000.fr-4: date (13252): status > Exited with status 0
    idpot11.grenoble.grid5000.fr-6: date (12319): status > Exited with status 0
    idpot9.grenoble.grid5000.fr-7: date (17810): output > Thu May  3 18:57:42 CEST 2007
    idpot9.grenoble.grid5000.fr-7: date (17810): status > Exited with status 0
    idpot7.grenoble.grid5000.fr-10: date (7355): status > Exited with status 0
    idpot5.grenoble.grid5000.fr-12: date (13093): status > Exited with status 0

#### Setting the connector definitively and running taktuk again

    jdoe@idpot1:~$ export TAKTUK_CONNECTOR=oarsh
    jdoe@idpot1:~$ taktuk -m idpot3 -m idpot4 broadcast exec [ date ]
    idpot3-1: date (12293): output > Wed Mar  7 09:20:25 CET 2007
    idpot4-2: date (7508): output > Wed Mar  7 09:20:19 CET 2007
    idpot3-1: date (12293): status > Exited with status 0
    idpot4-2: date (7508): status > Exited with status 0

## Using MPI with OARSH

To use MPI, you must setup your MPI stack so that it uses OARSH instead of the default RSH or SSH connector. All required steps for the main different flavors of MPI are presented below.

### MPICH1

Mpich1 connector can be changed using the P4_RSHCOMMAND environment variable. This variable must be set in the shell configuration files. For instance for bash, within ~/.bashrc

    export P4_RSHCOMMAND=oarsh

Please consider setting the P4_GLOBMEMSIZE as well.

You can then run your mpich1 application:

    jdoe@idpot4:~/mpi/mpich$ mpirun.mpich -machinefile $OAR_FILE_NODES -np 6 ./hello
    Hello world from process 0 of 6 running on idpot4.grenoble.grid5000.fr
    Hello world from process 4 of 6 running on idpot6.grenoble.grid5000.fr
    Hello world from process 1 of 6 running on idpot4.grenoble.grid5000.fr
    Hello world from process 3 of 6 running on idpot5.grenoble.grid5000.fr
    Hello world from process 2 of 6 running on idpot5.grenoble.grid5000.fr
    Hello world from process 5 of 6 running on idpot6.grenoble.grid5000.fr

### MPICH2

Tested version: 1.0.5p2

MPICH2 uses daemons on nodes that may be started with the “mpdboot” command. This command takes oarsh has an argument (–rsh=oarsh) and all goes well:

    jdoe@idpot2:~/mpi/mpich/mpich2-1.0.5p2/bin$ ./mpicc -o hello ../../../hello.c
    jdoe@idpot2:~/mpi/mpich/mpich2-1.0.5p2/bin$ ./mpdboot --file=$OAR_NODEFILE --rsh=oarsh -n 2
    jdoe@idpot2:~/mpi/mpich/mpich2-1.0.5p2/bin$ ./mpdtrace -l
    idpot2_39441 (129.88.70.2)
    idpot4_36313 (129.88.70.4)
    jdoe@idpot2:~/mpi/mpich/mpich2-1.0.5p2/bin$ ./mpiexec -np 8 ./hello
    Hello world from process 0 of 8 running on idpot2
    Hello world from process 1 of 8 running on idpot4
    Hello world from process 3 of 8 running on idpot4
    Hello world from process 2 of 8 running on idpot2
    Hello world from process 5 of 8 running on idpot4
    Hello world from process 4 of 8 running on idpot2
    Hello world from process 6 of 8 running on idpot2
    Hello world from process 7 of 8 running on idpot4

### MVAPICH2

You can use the hydra launcher with the options “-launcher” and “-launcher-exec”, for example:

    mpiexec -launcher ssh -launcher-exec /usr/bin/oarsh -f $OAR_NODEFILE -n 4 ./app

### LAM/MPI

Tested version: 7.1.3

You can use export LAMRSH=oarsh before starting lamboot; otherwise the “lamboot” command takes -ssi boot_rsh_agent “oarsh” option has an argument (this is not in the manual!). Also note that OARSH doesn’t automatically sends the environnement of the user, so, you may need to specify the path to LAM distribution on the nodes with this option: -prefix

    jdoe@idpot2:~/mpi/lam$ ./bin/lamboot -prefix ~/mpi/lam \
                                     -ssi boot_rsh_agent "oarsh" \
                                     -d $OAR_FILE_NODES
    jdoe@idpot2:~/mpi/lam$ ./bin/mpirun -np 8 hello
    Hello world from process 2 of 8 running on idpot2
    Hello world from process 3 of 8 running on idpot2
    Hello world from process 0 of 8 running on idpot2
    Hello world from process 1 of 8 running on idpot2
    Hello world from process 4 of 8 running on idpot4
    Hello world from process 6 of 8 running on idpot4
    Hello world from process 5 of 8 running on idpot4
    Hello world from process 7 of 8 running on idpot4

### OpenMPI

Tested version: 1.1.4

The magic option to use with OpenMPI and OARSH is “-mca pls_rsh_agent “oarsh””. Also note that OpenMPI works with daemons that are started on the nodes (orted), but “mpirun” starts them on-demand. The “-prefix” option can help if OpenMPI is not installed in a standard path on the cluster nodes (you can replace the “-prefix” option by using the absolute path when invoking the “mpirun” command).

    jdoe@idpot2:~/mpi/openmpi$ ./bin/mpirun -prefix ~/mpi/openmpi \
                            -machinefile $OAR_FILE_NODES \
                            -mca pls_rsh_agent "oarsh" \
                            -np 8 hello
    Hello world from process 0 of 8 running on idpot2
    Hello world from process 4 of 8 running on idpot4
    Hello world from process 1 of 8 running on idpot2
    Hello world from process 5 of 8 running on idpot4
    Hello world from process 2 of 8 running on idpot2
    Hello world from process 6 of 8 running on idpot4
    Hello world from process 7 of 8 running on idpot4
    Hello world from process 3 of 8 running on idpot2

You can make the option “oarsh” automatically by adding it in a configuration file in the OpenMPI installation directory named “$OPENMPI_INSTALL_DIR/etc/openmpi-mca-params.conf”

    plm_rsh_agent=/usr/bin/oarsh

So, with this configuration, this is transparent for the users.

Note: In OpenMPI 1.6, “pls_rsh_agent” was replaced by “orte_rsh_agent”. Note: In OpenMPI 1.8, “orte_rsh_agent” was replaced by “plm_rsh_agent”.

### Intel MPI

Example using the hydra launcher:

    mpiexec.hydra -genvall -f $OAR_NODE_FILE -bootstrap-exec oarsh -env I_MPI_DEBUG 5 -n 8 ./ring

## Tests of the CPUSET mechanism

### Processus isolation

In this test, we run 4 “yes” commands in a job whose resources is only one core. (syntax tested with bash as the user’s shell)

    jdoe@idpot:~$ oarsub -l core=1 "yes > /dev/null & yes > /dev/null & yes > /dev/null & yes > /dev/null"
    [ADMISSION RULE] Set default walltime to 7200.
    [ADMISSION RULE] Modify resource description with type constraints
    OAR_JOB_ID=8683

Then we connect to the node and run ps or top for monitoring purposes:

    jdoe@idpot:~$ oarsub -C 8683
    Initialize X11 forwarding...
    Connect to OAR job 8683 via the node idpot9.grenoble.grid5000.fr
    jdoe@idpot9:~$ ps -eo fname,pcpu,psr | grep yes
    yes      23.2   1
    yes      23.1   1
    yes      24.0   1
    yes      23.0   1

This shows that the 4 processus are indeed restricted to the core the job was assigned to, as expected.

Don’t forget to delete your job:

    jdoe@idpot:~$ oardel 8683

## Using best effort mode jobs

### Best effort job campaign

OAR provides a way to specify that jobs are best effort, which means that the server can delete them if room is needed to fit other jobs. One can submit such jobs using the besteffort type of job.

For instance you can run a job campaign as follows:

    for param in $(< ./paramlist); do
        oarsub -t besteffort -l core=1 "./my_script.sh $param"
    done

In this example, the file ./paramlist contains a list of parameters for a parametric application.

The following demonstrates the mechanism.

### Best effort job mechanism

#### Running a besteffort job in a first shell

    jdoe@idpot:~$ oarsub -I -l nodes=23 -t besteffort
    [ADMISSION RULE] Added automatically besteffort resource constraint
    [ADMISSION RULE] Redirect automatically in the besteffort queue
    [ADMISSION RULE] Set default walltime to 7200.
    [ADMISSION RULE] Modify resource description with type constraints
    OAR_JOB_ID=9630
    Interactive mode : waiting...
    [2007-05-10 11:06:25] Starting...

    Initialize X11 forwarding...
    Connect to OAR job 9630 via the node idcalc1.grenoble.grid5000.fr

#### Running a non best effort job on the same set of resources in a second shell

    jdoe@idpot:~$ oarsub -I
    [ADMISSION RULE] Set default walltime to 7200.
    [ADMISSION RULE] Modify resource description with type constraints
    OAR_JOB_ID=9631
    Interactive mode : waiting...
    [2007-05-10 11:06:50] Start prediction: 2007-05-10 11:06:50 (Karma = 0.000)
    [2007-05-10 11:06:53] Starting...

    Initialize X11 forwarding...
    Connect to OAR job 9631 via the node idpot9.grenoble.grid5000.fr

As expected, meanwhile the best effort job was stopped (watch the first shell):

    jdoe@idcalc1:~$ bash: line 1: 23946 Killed                  /bin/bash -l
    Connection to idcalc1.grenoble.grid5000.fr closed.
    Disconnected from OAR job 9630
    jdoe@idpot:~$

## Testing the checkpointing trigger mechanism

### Writing the test script

Here is a script feature an infinite loop and a signal handler trigged by SIGUSR2 (default signal for OAR’s checkpointing mechanism).

    #!/bin/bash

    handler() { echo "Caught checkpoint signal at: `date`"; echo "Terminating."; exit 0; }
    trap handler SIGUSR2

    cat <<EOF
    Hostname: `hostname`
    Pid: $$
    Starting job at: `date`
    EOF
    while : ; do sleep 1; done

### Running the job

We run the job on 1 core, and a walltime of 1 hour, and ask the job to be checkpointed if it lasts (and it will indeed) more that walltime - 900 sec = 45 min.

    jdoe@idpot:~/oar-2.0/tests/checkpoint$ oarsub -l "core=1,walltime=1:0:0" --checkpoint 900 ./checkpoint.sh
    [ADMISSION RULE] Modify resource description with type constraints
    OAR_JOB_ID=9464
    jdoe@idpot:~/oar-2.0/tests/checkpoint$

### Result

Taking a look at the job output:

    jdoe@idpot:~/oar-2.0/tests/checkpoint$ cat OAR.9464.stdout
    Hostname: idpot9
    Pid: 26577
    Starting job at: Fri May  4 19:41:11 CEST 2007
    Caught checkpoint signal at: Fri May  4 20:26:12 CEST 2007
    Terminating.

The checkpointing signal was sent to the job 15 minutes before the walltime as expected so that the job can finish nicely.

### Interactive checkpointing

The oardel command provides the capability to raise a checkpoint event interactively to a job.

We submit the job again

    jdoe@idpot:~/oar-2.0/tests/checkpoint$ oarsub -l "core=1,walltime=1:0:0" --checkpoint 900 ./checkpoint.sh
    [ADMISSION RULE] Modify resource description with type constraints
    OAR_JOB_ID=9521

Then run the oardel -c #jobid command…

    jdoe@idpot:~/oar-2.0/tests/checkpoint$ oardel -c 9521
    Checkpointing the job 9521 ...DONE.
    The job 9521 was notified to checkpoint itself (send SIGUSR2).

And then watch the job’s output:

    jdoe@idpot:~/oar-2.0/tests/checkpoint$ cat OAR.9521.stdout
    Hostname: idpot9
    Pid: 1242
    Starting job at: Mon May  7 16:39:04 CEST 2007
    Caught checkpoint signal at: Mon May  7 16:39:24 CEST 2007
    Terminating.

The job terminated as expected.

## Testing the mechanism of dependency on an anterior job termination

### First Job

We run a first interactive job in a first Shell

    jdoe@idpot:~$ oarsub -I
    [ADMISSION RULE] Set default walltime to 7200.
    [ADMISSION RULE] Modify resource description with type constraints
    OAR_JOB_ID=9458
    Interactive mode : waiting...
    [2007-05-04 17:59:38] Starting...

    Initialize X11 forwarding...
    Connect to OAR job 9458 via the node idpot9.grenoble.grid5000.fr
    jdoe@idpot9:~$

And leave that job pending.

### Second Job

Then we run a second job in another Shell, with a dependence on the first one

    jdoe@idpot:~$ oarsub -I -a 9458
    [ADMISSION RULE] Set default walltime to 7200.
    [ADMISSION RULE] Modify resource description with type constraints
    OAR_JOB_ID=9459
    Interactive mode : waiting...
    [2007-05-04 17:59:55] Start prediction: 2007-05-04 19:59:39 (Karma = 4.469)

So this second job is waiting for the first job walltime (or sooner termination) to be reached to start.

### Job dependency in action

We do a logout on the first interactive job…

    jdoe@idpot9:~$ logout
    Connection to idpot9.grenoble.grid5000.fr closed.
    Disconnected from OAR job 9458
    jdoe@idpot:~$

… then watch the second Shell and see the second job starting

    [2007-05-04 18:05:05] Starting...

    Initialize X11 forwarding...
    Connect to OAR job 9459 via the node idpot7.grenoble.grid5000.fr

… as expected.