# Using Parallel Magics

IPython has a few magics to make working with your engines a bit nicer in IPython, e.g. via a Jupyter notebook.

As always, first we start a cluster (or connect to an existing one):

    import ipyparallel as ipp

    rc = ipp.Cluster(n=4).start_and_connect_sync()
    dv = rc[:]
    rc.ids

    Using existing profile dir: '/Users/benjaminrk/.ipython/profile_default'
    Starting 4 engines with <class 'ipyparallel.cluster.launcher.LocalEngineSetLauncher'>

    [0, 1, 2, 3]

Creating a Client registers the parallel magics `%px`, `%%px`, `%pxresult`, `%pxconfig`, and `%autopx`.
These magics are initially associated with a DirectView, always associated with all currently registered engines.

First, we can execute single lines remotely with `%px`:

    %px a=5

    %px print(a)

    [stdout:0] 5

    [stdout:1] 5

    [stdout:2] 5

    [stdout:3] 5

    %px a

    Out[2:3]: 5

    Out[0:3]: 5

    Out[3:3]: 5

    Out[1:3]: 5

    with dv.sync_imports():
        import sys

    importing sys on engine(s)

    %px from __future__ import print_function
    %px print("ERROR", file=sys.stderr)

    [stderr:0] ERROR

    [stderr:1] ERROR

    [stderr:3] ERROR

    [stderr:2] ERROR

You don’t have to wait for results. The `%pxconfig` magic lets you change the default blocking/targets for the `%px` magics:

    %pxconfig --noblock

    %px import time
    %px time.sleep(1)
    %px time.time()

    <AsyncResult(%px): pending>

But you will notice that this didn’t output the result of the last command. For this, we have `%pxresult`, which displays the output of the latest request:

    %pxresult

    Out[0:8]: 1635248507.7749069

    Out[1:8]: 1635248507.7802722

    Out[2:8]: 1635248507.780295

    Out[3:8]: 1635248507.780204

Remember, an IPython engine is IPython, so you can do magics remotely as well!

    %pxconfig --block
    %px %matplotlib inline

`%%px` can be used to lower the priority of the engines to improve system performance under heavy CPU load.

    %%px
    import psutil
    psutil.Process().nice(20 if psutil.POSIX else psutil.IDLE_PRIORITY_CLASS)

    %%px
    import numpy as np
    import matplotlib.pyplot as plt

`%%px` can also be used as a cell magic, for submitting whole blocks. This one acceps `--block` and `--noblock` flags to specify the blocking behavior, though the default is unchanged.

    dv.scatter('id', dv.targets, flatten=True)
    dv['stride'] = len(dv)

    %%px --noblock
    x = np.linspace(0,np.pi,1000)
    for n in range(id,12, stride):
        print(n)
        plt.plot(x,np.sin(n*x))
    plt.title("Plot %i" % id)

    <AsyncResult(%px): pending>

    %pxresult

    [stdout:0]
    0
    4
    8
    [stdout:1]
    1
    5
    9
    [stdout:2]
    2
    6
    10
    [stdout:3]
    3
    7
    11

    [output:0]

    [output:1]

    [output:2]

    [output:3]

    Out[0:13]: Text(0.5, 1.0, 'Plot 0')

    Out[1:13]: Text(0.5, 1.0, 'Plot 1')

    Out[2:13]: Text(0.5, 1.0, 'Plot 2')

    Out[3:13]: Text(0.5, 1.0, 'Plot 3')

It also lets you choose some amount of the grouping of the outputs with `--group-outputs`:

The choices are:

- `engine` - all of an engine’s output is collected together

- `type` - where stdout of each engine is grouped, etc. (the default)

- `order` - same as `type`, but individual displaypub outputs are interleaved. That is, it will output the first plot from each engine, then the second from each, etc.

    %%px --group-outputs=engine
    x = np.linspace(0,np.pi,1000)
    for n in range(id+1,12, stride):
        print(n)
        plt.figure()
        plt.plot(x,np.sin(n*x))
        plt.title("Plot %i" % n)

    [stdout:0] 1
    5
    9

    [stdout:1] 2
    6
    10

    [stdout:2] 3
    7
    11

    [stdout:3] 4
    8

    [output:3]

    [output:0]

    [output:1]

    [output:2]

    [output:3]

    [output:0]

    [output:1]

    [output:2]

    [output:0]

    [output:1]

    [output:2]

When you specify ‘order’, then individual display outputs (e.g. plots) will be interleaved.

`%pxresult` takes the same output-ordering arguments as `%%px`, so you can view the previous result in a variety of different ways with a few sequential calls to `%pxresult`:

    %pxresult --group-outputs=order

    [stdout:0]
    1
    5
    9
    [stdout:1]
    2
    6
    10
    [stdout:2]
    3
    7
    11
    [stdout:3]
    4
    8

    [output:0]

    [output:1]

    [output:2]

    [output:0]

    [output:1]

    [output:2]

    [output:0]

    [output:1]

    [output:2]

## Single-engine views

When a DirectView has a single target, the output is a bit simpler (no prefixes on stdout/err, etc.):

    from __future__ import print_function

    def generate_output():
        """function for testing output

        publishes two outputs of each type, and returns something
        """

        import sys,os
        from IPython.display import display, HTML, Math

        print("stdout")
        print("stderr", file=sys.stderr)

        display(HTML("<b>HTML</b>"))

        print("stdout2")
        print("stderr2", file=sys.stderr)

        display(Math(r"\alpha=\beta"))

        return os.getpid()

    dv['generate_output'] = generate_output

You can also have more than one set of parallel magics registered at a time.

The `View.activate()` method takes a suffix argument, which is added to `'px'`.

    e0 = rc[-1]
    e0.block = True
    e0.activate('0')

    %px0 generate_output()

    [stdout:3] stdout
    stdout2

    [stderr:3] stderr
    stderr2

    [output:3]

HTML

    [output:3]

\[\displaystyle \alpha=\beta\]

    Out[3:15]: 83115

    %px generate_output()

    [stdout:0] stdout
    stdout2

    [stdout:1] stdout
    stdout2

    [stdout:2] stdout
    stdout2

    [stderr:2] stderr
    stderr2

    [stdout:3] stdout
    stdout2

    [stderr:0] stderr
    stderr2

    [output:0]

HTML

    [stderr:1] stderr
    stderr2

    [output:1]

HTML

    [stderr:3] stderr
    stderr2

    [output:2]

HTML

    [output:3]

HTML

    [output:0]

\[\displaystyle \alpha=\beta\]

    [output:1]

\[\displaystyle \alpha=\beta\]

    Out[0:15]: 83112

    [output:2]

\[\displaystyle \alpha=\beta\]

    [output:3]

\[\displaystyle \alpha=\beta\]

    Out[1:15]: 83113

    Out[2:15]: 83114

    Out[3:16]: 83115

As mentioned above, we can redisplay those same results with various grouping:

    %pxresult --group-outputs order

    [stdout:0]
    stdout
    stdout2
    [stdout:1]
    stdout
    stdout2
    [stdout:2]
    stdout
    stdout2
    [stdout:3]
    stdout
    stdout2

    [stderr:0]
    stderr
    stderr2
    [stderr:1]
    stderr
    stderr2
    [stderr:2]
    stderr
    stderr2
    [stderr:3]
    stderr
    stderr2

    [output:0]

HTML

    [output:1]

HTML

    [output:2]

HTML

    [output:3]

HTML

    [output:0]

\[\displaystyle \alpha=\beta\]

    [output:1]

\[\displaystyle \alpha=\beta\]

    [output:2]

\[\displaystyle \alpha=\beta\]

    [output:3]

\[\displaystyle \alpha=\beta\]

    Out[0:15]: 83112

    Out[1:15]: 83113

    Out[2:15]: 83114

    Out[3:16]: 83115

    %pxresult --group-outputs engine

    [stdout:0]
    stdout
    stdout2

    [stderr:0]
    stderr
    stderr2

    [output:0]

HTML
\[\displaystyle \alpha=\beta\]

    Out[0:15]: 83112

    [stdout:1]
    stdout
    stdout2

    [stderr:1]
    stderr
    stderr2

    [output:1]

HTML
\[\displaystyle \alpha=\beta\]

    Out[1:15]: 83113

    [stdout:2]
    stdout
    stdout2

    [stderr:2]
    stderr
    stderr2

    [output:2]

HTML
\[\displaystyle \alpha=\beta\]

    Out[2:15]: 83114

    [stdout:3]
    stdout
    stdout2

    [stderr:3]
    stderr
    stderr2

    [output:3]

HTML
\[\displaystyle \alpha=\beta\]

    Out[3:16]: 83115

## Parallel Exceptions

When you raise exceptions with the parallel exception, the CompositeError raised locally will display your remote traceback.

    %%px
    from numpy.random import random
    A = random((100, 100, 'invalid shape'))

    [0:execute]:
    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    /var/folders/9p/clj0fc754y35m01btd46043c0000gn/T/ipykernel_83112/1064401740.py in <module>
          1 from numpy.random import random
    ----> 2 A = random((100, 100, 'invalid shape'))

    mtrand.pyx in numpy.random.mtrand.RandomState.random()

    mtrand.pyx in numpy.random.mtrand.RandomState.random_sample()

    _common.pyx in numpy.random._common.double_fill()

    TypeError: 'str' object cannot be interpreted as an integer
    [2:execute]:
    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    /var/folders/9p/clj0fc754y35m01btd46043c0000gn/T/ipykernel_83114/1064401740.py in <module>
          1 from numpy.random import random
    ----> 2 A = random((100, 100, 'invalid shape'))

    mtrand.pyx in numpy.random.mtrand.RandomState.random()

    mtrand.pyx in numpy.random.mtrand.RandomState.random_sample()

    _common.pyx in numpy.random._common.double_fill()

    TypeError: 'str' object cannot be interpreted as an integer
    [1:execute]:
    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    /var/folders/9p/clj0fc754y35m01btd46043c0000gn/T/ipykernel_83113/1064401740.py in <module>
          1 from numpy.random import random
    ----> 2 A = random((100, 100, 'invalid shape'))

    mtrand.pyx in numpy.random.mtrand.RandomState.random()

    mtrand.pyx in numpy.random.mtrand.RandomState.random_sample()

    _common.pyx in numpy.random._common.double_fill()

    TypeError: 'str' object cannot be interpreted as an integer
    [3:execute]:
    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    /var/folders/9p/clj0fc754y35m01btd46043c0000gn/T/ipykernel_83115/1064401740.py in <module>
          1 from numpy.random import random
    ----> 2 A = random((100, 100, 'invalid shape'))

    mtrand.pyx in numpy.random.mtrand.RandomState.random()

    mtrand.pyx in numpy.random.mtrand.RandomState.random_sample()

    _common.pyx in numpy.random._common.double_fill()

    TypeError: 'str' object cannot be interpreted as an integer

    4 errors

Sometimes, an error will occur on just one engine, while the rest are still working.

When this happens, you will see errors immediately, and can interrupt the execution:

    dv.scatter("rank", rc.ids, flatten=True)

    <AsyncResult(scatter): pending>

    %%px
    import time
    if rank == 0:
        raise RuntimeError("rank 0 failed!")
    time.sleep(10)

    [0:execute]:
    ---------------------------------------------------------------------------
    RuntimeError                              Traceback (most recent call last)
    /var/folders/9p/clj0fc754y35m01btd46043c0000gn/T/ipykernel_83112/2811384868.py in <module>
          1 import time
          2 if rank == 0:
    ----> 3     raise RuntimeError("rank 0 failed!")
          4 time.sleep(10)
          5

    RuntimeError: rank 0 failed!

    1 errors

## Remote Cell Magics

Remember, Engines are IPython too, so the cell that is run remotely by %%px can in turn use a cell magic.

    %%px
    %%timeit
    from numpy.random import random
    from numpy.linalg import norm
    A = random((100, 100))
    norm(A, 2)

    [stdout:1] 6.01 ms ± 596 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

    [stdout:0] 6.02 ms ± 549 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

    [stdout:3] 6.06 ms ± 580 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

    [stdout:2] 6.08 ms ± 430 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

## Local Execution

You can instruct `%%px` to also execute the cell locally. This is useful for interactive definitions, or if you want to load a data source everywhere, not just on the engines.

    %%px --local
    import os
    thispid = os.getpid()
    print(thispid)

    83088
    [stdout:0]
    83112
    [stdout:1]
    83113
    [stdout:2]
    83114
    [stdout:3]
    83115