# Using widgets in IPython Parallel

IPython Parallel 7.1 introduces basic support for using Jupyter widgets from a notebook to engines.

This allows things like progress bars for incremental stages, when IPP’s own task-level progress doesn’t show you enough information.

As always, we start by creating and connecting to a cluster:

    [1]:

    import ipyparallel as ipp
    rc = ipp.Cluster(n=4).start_and_connect_sync()
    rc.activate()

    Using existing profile dir: '/Users/minrk/.ipython/profile_default'
    Starting 4 engines with <class 'ipyparallel.cluster.launcher.LocalEngineSetLauncher'>

    [1]:

    <DirectView all>

IPython widgets support updateable readouts of things like progress.

There are lots of widgets to choose from, but for our purposes, we are going to use `IntProgress`

    [2]:

    import time

    import ipywidgets as W
    from IPython.display import display

    progress = W.IntProgress(min=0, max=10, description="Step 0", readout=True)
    display(progress)

    for i in range(10):
        progress.value += 1
        progress.description = f"Step {progress.value}/{progress.max}"
        time.sleep(0.1)

    progress.bar_style = "success" # change color when it's done

IPython Parallel supports progress and interactive waits on AsyncResult objects, which is great when your tasks are small and you have a lot of them:

    [3]:

    view = rc.load_balanced_view()
    view.map_async(lambda x: x * 2, range(1000)).wait_interactive()

but the unit of progress is each function call.

If you have only a few large IPython tasks, you only get feedback when the whole task is done on a given engine:

    [4]:

    %%px --block
    import time

    def do_some_work(t):
        time.sleep(t)

    do_some_work(5)

Combining widgets with tasks allows us to show our progress in stages, including showing progress for every engine.

This way we get to see progress within the task, even though IPython Parallel can’t tell what’s going on within your black-box task

    [5]:

    rc[:].scatter("rank", rc.ids, flatten=True)

    [5]:

    <AsyncResult: scatter>

    [6]:

    %%px --block
    import random

    import ipywidgets as W
    from IPython.display import display

    n_steps = 10

    progress = W.IntProgress(max=n_steps, description=f"rank {rank}: 0")
    def _set_color(change):
        if change.new == progress.max:
            progress.bar_style = "success"
        else:
            progress.bar_style = ""
    progress.observe(_set_color, "value")
    display(progress)

    for i in range(n_steps):
        time.sleep(random.random())
        progress.value += 1
        progress.description = f"rank {rank}: {progress.value}/{progress.max}"

    [output:3]

    [output:2]

    [output:0]

    [output:1]

Producing progress bars for every engine can get very unwieldy if you have more than a few engines.

Here’s an example where only rank 0 reports its progress, giving a more succinct presentation, assuming rank 0 knows sufficient information about what’s going on (e.g. an MPI-based simulation).

    [7]:

    %%px --block
    import random
    import time

    import ipywidgets as W
    from IPython.display import display

    n_steps = 10

    if rank == 0:
        progress = W.IntProgress(max=n_steps, description="Step 0")
        def _set_color(change):
            if change.new == progress.max:
                progress.bar_style = "success"
            else:
                progress.bar_style = ""
        progress.observe(_set_color, "value")
        display(progress)

    for i in range(n_steps):
        time.sleep(random.random())
        if rank == 0:
            progress.value += 1
            progress.description = f"Step {progress.value}/{progress.max}"

    print(f"rank {rank}: done!")

    [output:0]

    [stdout:2] rank 2: done!
    [stdout:1] rank 1: done!
    [stdout:0] rank 0: done!
    [stdout:3] rank 3: done!

Widgets also support two-way communication, using buttons and text areas for input. These also work in IPython Parallel, allowing feedback between engines and a notebook.

    [8]:

    %%px
    button = W.Button(description=f"step")
    progress = W.IntProgress(max=5, description="0 / 5")
    count = 0

    def click_increment(_):
        progress.value += 1
        progress.description = f"{progress.value} / {progress.max}"
        if progress.value == progress.max:
            button.disabled = True
            progress.bar_style = "success"

    button.on_click(click_increment)

    display(W.HBox([W.Label(f"Rank {rank}"), button, progress]))

    [output:0]

    [output:1]

    [output:2]

    [output:3]

This lets you build interactive displays with parallel execution on remote engines. Useful? Maybe!