# Using Parallel Magics

IPython has a few magics for working with your engines.

This assumes you have started an IPython cluster, either with the notebook interface, or the `ipcluster/controller/engine` commands.

    [1]:

    import ipyparallel as ipp

    rc = ipp.Cluster(n=4).start_and_connect_sync()
    dv = rc[:]
    rc.ids

    Starting 4 engines with <class 'ipyparallel.cluster.launcher.LocalEngineSetLauncher'>
    100%|██████████| 4/4 [00:05<00:00,  1.35s/engine]

    [1]:

    [0, 1, 2, 3]

Creating a Client registers the parallel magics `%px`, `%%px`, `%pxresult`, `pxconfig`, and `%autopx`.
These magics are initially associated with a DirectView always associated with all currently registered engines.

Now we can execute code remotely with `%px`:

    [2]:

    %px a=5

    [3]:

    %px print(a)

    [stdout:0] 5

    [stdout:2] 5

    [stdout:1] 5

    [stdout:3] 5

    [4]:

    %px a

    Out[0:3]: 5

    Out[1:3]: 5

    Out[2:3]: 5

    Out[3:3]: 5

    [5]:

    with dv.sync_imports():
        import sys

    importing sys on engine(s)

    [6]:

    %px from __future__ import print_function
    %px print("ERROR", file=sys.stderr)

    [stderr:1] ERROR

    [stderr:2] ERROR

    [stderr:0] ERROR

    [stderr:3] ERROR

You don’t have to wait for results. The `%pxconfig` magic lets you change the default blocking/targets for the `%px` magics:

    [7]:

    %pxconfig --noblock

    [8]:

    %px import time
    %px time.sleep(1)
    %px time.time()

    [8]:

    <AsyncResult: %px>

But you will notice that this didn’t output the result of the last command. For this, we have `%pxresult`, which displays the output of the latest request:

    [9]:

    %pxresult

    Out[0:8]: 1707467505.0044286

    Out[1:8]: 1707467504.9966855

    Out[2:8]: 1707467505.0078459

    Out[3:8]: 1707467505.0087497

Remember, an IPython engine is IPython, so you can do magics remotely as well!

    [10]:

    %pxconfig --block
    %px %matplotlib inline

    [stderr:3] Matplotlib created a temporary config/cache directory at /tmp/matplotlib-jvmargls because the default path (/sbuild-nonexistent/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.

    [stderr:2] Matplotlib created a temporary config/cache directory at /tmp/matplotlib-bqbhe237 because the default path (/sbuild-nonexistent/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.

    [stderr:0] Matplotlib created a temporary config/cache directory at /tmp/matplotlib-cffcmii_ because the default path (/sbuild-nonexistent/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.

    [stderr:1] Matplotlib created a temporary config/cache directory at /tmp/matplotlib-z__q1fb3 because the default path (/sbuild-nonexistent/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.

`%%px` can be used to lower the priority of the engines to improve system performance under heavy CPU load.

    [11]:

    %%px
    import psutil
    psutil.Process().nice(20 if psutil.POSIX else psutil.IDLE_PRIORITY_CLASS)

    [12]:

    %%px
    import numpy as np
    import matplotlib.pyplot as plt

`%%px` can also be used as a cell magic, for submitting whole blocks. This one acceps `--block` and `--noblock` flags to specify the blocking behavior, though the default is unchanged.

    [13]:

    dv.scatter('id', dv.targets, flatten=True)
    dv['stride'] = len(dv)

    [14]:

    %%px --noblock
    x = np.linspace(0,np.pi,1000)
    for n in range(id,12, stride):
        print(n)
        plt.plot(x,np.sin(n*x))
    plt.title("Plot %i" % id)

    [14]:

    <AsyncResult: %px>

    [15]:

    %pxresult

    [stdout:0]
    0
    4
    8
    [stdout:1]
    1
    5
    9
    [stdout:2]
    2
    6
    10
    [stdout:3]
    3
    7
    11

    [output:0]

    [output:1]

    [output:2]

    [output:3]

    Out[0:12]: Text(0.5, 1.0, 'Plot 0')

    Out[1:12]: Text(0.5, 1.0, 'Plot 1')

    Out[2:12]: Text(0.5, 1.0, 'Plot 2')

    Out[3:12]: Text(0.5, 1.0, 'Plot 3')

It also lets you choose some amount of the grouping of the outputs with `--group-outputs`:

The choices are:

- `engine` - all of an engine’s output is collected together

- `type` - where stdout of each engine is grouped, etc. (the default)

- `order` - same as `type`, but individual displaypub outputs are interleaved. That is, it will output the first plot from each engine, then the second from each, etc.

    [16]:

    %%px --group-outputs=engine
    x = np.linspace(0,np.pi,1000)
    for n in range(id+1,12, stride):
        print(n)
        plt.figure()
        plt.plot(x,np.sin(n*x))
        plt.title("Plot %i" % n)

    [stdout:0] 1
    5
    9

    [stdout:2] 3
    7
    11

    [stdout:3] 4
    8

    [stdout:1] 2
    6
    10

    [output:3]

    [output:0]

    [output:2]

    [output:1]

    [output:3]

    [output:0]

    [output:2]

    [output:1]

    [output:0]

    [output:2]

    [output:1]

When you specify ‘order’, then individual display outputs (e.g. plots) will be interleaved.

`%pxresult` takes the same output-ordering arguments as `%%px`, so you can view the previous result in a variety of different ways with a few sequential calls to `%pxresult`:

    [17]:

    %pxresult --group-outputs=order

    [stdout:0]
    1
    5
    9
    [stdout:1]
    2
    6
    10
    [stdout:2]
    3
    7
    11
    [stdout:3]
    4
    8

    [output:0]

    [output:1]

    [output:2]

    [output:0]

    [output:1]

    [output:2]

    [output:0]

    [output:1]

    [output:2]

## Single-engine views

When a DirectView has a single target, the output is a bit simpler (no prefixes on stdout/err, etc.):

    [18]:

    from __future__ import print_function

    def generate_output():
        """function for testing output

        publishes two outputs of each type, and returns something
        """

        import sys,os
        from IPython.display import display, HTML, Math

        print("stdout")
        print("stderr", file=sys.stderr)

        display(HTML("<b>HTML</b>"))

        print("stdout2")
        print("stderr2", file=sys.stderr)

        display(Math(r"\alpha=\beta"))

        return os.getpid()

    dv['generate_output'] = generate_output

You can also have more than one set of parallel magics registered at a time.

The `View.activate()` method takes a suffix argument, which is added to `'px'`.

    [19]:

    e0 = rc[-1]
    e0.block = True
    e0.activate('0')

    [20]:

    %px0 generate_output()

    [stdout:3] stdout
    stdout2

    [stderr:3] stderr
    stderr2

    [output:3]

HTML

    [output:3]

$\displaystyle \alpha=\beta$

    Out[3:14]: 15128

    [21]:

    %px generate_output()

    [stdout:0] stdout
    stdout2

    [stdout:1] stdout
    stdout2

    [stderr:0] stderr
    stderr2

    [stderr:1] stderr
    stderr2

    [stdout:2] stdout
    stdout2

    [output:1]

HTML

    [output:0]

HTML

    [stderr:2] stderr
    stderr2

    [stdout:3] stdout
    stdout2

    [output:1]

$\displaystyle \alpha=\beta$

    [output:2]

HTML

    [stderr:3] stderr
    stderr2

    [output:0]

$\displaystyle \alpha=\beta$

    [output:3]

HTML

    [output:2]

$\displaystyle \alpha=\beta$

    [output:3]

$\displaystyle \alpha=\beta$

    Out[0:14]: 15122

    Out[1:14]: 15124

    Out[2:14]: 15126

    Out[3:15]: 15128

As mentioned above, we can redisplay those same results with various grouping:

    [22]:

    %pxresult --group-outputs order

    [stdout:0]
    stdout
    stdout2
    [stdout:1]
    stdout
    stdout2
    [stdout:2]
    stdout
    stdout2
    [stdout:3]
    stdout
    stdout2

    [stderr:0]
    stderr
    stderr2
    [stderr:1]
    stderr
    stderr2
    [stderr:2]
    stderr
    stderr2
    [stderr:3]
    stderr
    stderr2

    [output:0]

HTML

    [output:1]

HTML

    [output:2]

HTML

    [output:3]

HTML

    [output:0]

$\displaystyle \alpha=\beta$

    [output:1]

$\displaystyle \alpha=\beta$

    [output:2]

$\displaystyle \alpha=\beta$

    [output:3]

$\displaystyle \alpha=\beta$

    Out[0:14]: 15122

    Out[1:14]: 15124

    Out[2:14]: 15126

    Out[3:15]: 15128

    [23]:

    %pxresult --group-outputs engine

    [stdout:0]
    stdout
    stdout2

    [stderr:0]
    stderr
    stderr2

    [output:0]

HTML
$\displaystyle \alpha=\beta$

    Out[0:14]: 15122

    [stdout:1]
    stdout
    stdout2

    [stderr:1]
    stderr
    stderr2

    [output:1]

HTML
$\displaystyle \alpha=\beta$

    Out[1:14]: 15124

    [stdout:2]
    stdout
    stdout2

    [stderr:2]
    stderr
    stderr2

    [output:2]

HTML
$\displaystyle \alpha=\beta$

    Out[2:14]: 15126

    [stdout:3]
    stdout
    stdout2

    [stderr:3]
    stderr
    stderr2

    [output:3]

HTML
$\displaystyle \alpha=\beta$

    Out[3:15]: 15128

## Parallel Exceptions

When you raise exceptions with the parallel exception, the CompositeError raised locally will display your remote traceback.

    [24]:

    %%px
    from numpy.random import random
    A = random((100, 100, 'invalid shape'))

    [0:execute]:
    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    Cell In[15], line 2
          1 from numpy.random import random
    ----> 2 A = random((100, 100, 'invalid shape'))

    File mtrand.pyx:445, in numpy.random.mtrand.RandomState.random()

    File mtrand.pyx:436, in numpy.random.mtrand.RandomState.random_sample()

    File _common.pyx:307, in numpy.random._common.double_fill()

    TypeError: 'str' object cannot be interpreted as an integer

    [1:execute]:
    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    Cell In[15], line 2
          1 from numpy.random import random
    ----> 2 A = random((100, 100, 'invalid shape'))

    File mtrand.pyx:445, in numpy.random.mtrand.RandomState.random()

    File mtrand.pyx:436, in numpy.random.mtrand.RandomState.random_sample()

    File _common.pyx:307, in numpy.random._common.double_fill()

    TypeError: 'str' object cannot be interpreted as an integer

    [2:execute]:
    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    Cell In[15], line 2
          1 from numpy.random import random
    ----> 2 A = random((100, 100, 'invalid shape'))

    File mtrand.pyx:445, in numpy.random.mtrand.RandomState.random()

    File mtrand.pyx:436, in numpy.random.mtrand.RandomState.random_sample()

    File _common.pyx:307, in numpy.random._common.double_fill()

    TypeError: 'str' object cannot be interpreted as an integer

    [3:execute]:
    ---------------------------------------------------------------------------
    TypeError                                 Traceback (most recent call last)
    Cell In[16], line 2
          1 from numpy.random import random
    ----> 2 A = random((100, 100, 'invalid shape'))

    File mtrand.pyx:445, in numpy.random.mtrand.RandomState.random()

    File mtrand.pyx:436, in numpy.random.mtrand.RandomState.random_sample()

    File _common.pyx:307, in numpy.random._common.double_fill()

    TypeError: 'str' object cannot be interpreted as an integer

## Remote Cell Magics

Remember, Engines are IPython too, so the cell that is run remotely by %%px can in turn use a cell magic.

    [25]:

    %%px
    %%timeit
    from numpy.random import random
    from numpy.linalg import norm
    A = random((100, 100))
    norm(A, 2)

    %px:   0%|          | 0/4 [00:09<?, ?tasks/s]

    [stdout:2] 1.42 ms ± 97.4 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

    %px:  25%|██▌       | 1/4 [00:09<00:00,  9.93tasks/s]

    [stdout:0] 1.42 ms ± 75.4 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

    %px:  50%|█████     | 2/4 [00:09<00:00,  9.94tasks/s]

    [stdout:1] 1.47 ms ± 144 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

    [stdout:3] 1.48 ms ± 174 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

    %px: 100%|██████████| 4/4 [00:09<00:00,  2.46s/tasks]

## Local Execution

You can instruct `%%px` to also execute the cell locally. This is useful for interactive definitions, or if you want to load a data source everywhere, not just on the engines.

    [26]:

    %%px --local
    import os
    thispid = os.getpid()
    print(thispid)

    15070
    [stdout:0]
    15122
    [stdout:1]
    15124
    [stdout:2]
    15126
    [stdout:3]
    15128