# Using IPython Parallel as a joblib backend

joblib is a tool for running tasks, which includes support for implementing custom parallel backends. IPython defines one such backend, so you can use IPython parallel with joblib.

The simplest way to set this up is a single call:

    import ipyparallel as ipp

    ipp.register_joblib_backend()

This registers the ‘ipyparallel’ backend with all the defaults.

    from joblib import Parallel, delayed

    def neg(x):
        return -x

    Parallel(backend='ipyparallel')(delayed(neg)(i) for i in range(32))

    Using existing profile dir: '/Users/minrk/.ipython/profile_default'
    Using existing profile dir: '/Users/minrk/.ipython/profile_default'
    Starting 4 engines with <class 'ipyparallel.cluster.launcher.LocalEngineSetLauncher'>

    Stopping engine(s): 1630490654
    engine set stopped 1630490654: {'0': {'exit_code': 0, 'pid': 14275, 'identifier': '0'}, '1': {'exit_code': 0, 'pid': 14276, 'identifier': '1'}, '2': {'exit_code': 0, 'pid': 14277, 'identifier': '2'}, '3': {'exit_code': 0, 'pid': 14278, 'identifier': '3'}}
    Stopping controller
    Controller stopped: {'exit_code': 0, 'pid': 14261, 'identifier': 'ipcontroller-1630490653-glt9-14256'}

    [0,
     -1,
     -2,
     -3,
     -4,
     -5,
     -6,
     -7,
     -8,
     -9,
     -10,
     -11,
     -12,
     -13,
     -14,
     -15,
     -16,
     -17,
     -18,
     -19,
     -20,
     -21,
     -22,
     -23,
     -24,
     -25,
     -26,
     -27,
     -28,
     -29,
     -30,
     -31]

You can also configure your own View, and register it explicitly, or even as the default:

    rc = ipp.Cluster(n=4).start_and_connect_sync()

    rc[:].use_cloudpickle()
    view = rc.load_balanced_view()
    view.register_joblib_backend(make_default=True)

    Using existing profile dir: '/Users/minrk/.ipython/profile_default'
    Starting 4 engines with <class 'ipyparallel.cluster.launcher.LocalEngineSetLauncher'>

    import os

    Parallel()(delayed(os.getpid)() for i in range(10))

    [14317, 14316, 14315, 14306, 14306, 14306, 14306, 14315, 14306, 14306]

Joblib also provides a context manager for selecting a particular backend:

    even = rc.load_balanced_view(targets=rc.ids[::2])
    even.register_joblib_backend('even')

    from joblib import parallel_backend

    with parallel_backend('even'):
        result = Parallel()(delayed(os.getpid)() for i in range(5))
    result

    [14316, 14306, 14316, 14306, 14316]