# execnet: Distributed Python deployment and communication

`home | install | examples | basic API | support`

# Basic local and remote communication

## Execute source code in subprocess, communicate through a channel

You can instantiate a subprocess gateway, execute code in it and bidirectionally send messages:

    >>> import execnet
    >>> gw = execnet.makegateway()
    >>> channel = gw.remote_exec("channel.send(channel.receive()+1)")
    >>> channel.send(1)
    >>> channel.receive()
    2

The initiating and the remote execution happen concurrently. `channel.receive()` operations return when input is available. `channel.send(data)` operations return when the message could be delivered to the IO system.

The initiating and the “other” process work use a share-nothing model and `channel.send|receive` are means to pass basic data messages between two processes.

## Remote-exec a function (avoiding inlined source part I)

You can send and remote execute parametrized pure functions like this:

    import execnet

    def multiplier(channel, factor):
        while not channel.isclosed():
            param = channel.receive()
            channel.send(param * factor)

    gw = execnet.makegateway()
    channel = gw.remote_exec(multiplier, factor=10)

    for i in range(5):
        channel.send(i)
        result = channel.receive()
        assert result == i * 10

    gw.exit()

The `multiplier` function executes remotely and establishes a loop multipliying incoming data with a constant factor passed in via keyword arguments to `remote_exec`.

Notes:

- unfortunately, you can not type this example interactively because `inspect.getsource(func)` fails for interactively defined functions.

- You will get an explicit error if you try to execute non-pure functions, i.e. functions that access any global state (which will not be available remotely as we have a share-nothing model between the nodes).

## Remote-exec a module (avoiding inlined source part II)

You can pass a module object to `remote_exec` in which case its source code will be sent. No dependencies will be transferred so the module must be self-contained or only use modules that are installed on the “other” side. Module code can detect if it is running in a remote_exec situation by checking for the special `__name__` attribute.

    # content of a module remote1.py

    if __name__ == "__channelexec__":
        channel.send("initialization complete")  # type: ignore[name-defined]

You can now remote-execute the module like this:

    >>> import execnet, remote1
    >>> gw = execnet.makegateway()
    >>> ch = gw.remote_exec(remote1)
    >>> print (ch.receive())
    initialization complete

which will print the ‘initialization complete’ string.

## Compare current working directories

A local subprocess gateway has the same working directory as the instantiatior:

    >>> import execnet, os
    >>> gw = execnet.makegateway()
    >>> ch = gw.remote_exec("import os; channel.send(os.getcwd())")
    >>> res = ch.receive()
    >>> assert res == os.getcwd()

“ssh” gateways default to the login home directory.

## Get information from remote SSH account

Use simple execution to obtain information from remote environments:

    >>> import execnet, os
    >>> gw = execnet.makegateway("ssh=codespeak.net")
    >>> channel = gw.remote_exec("""
    ...     import sys, os
    ...     channel.send((sys.platform, tuple(sys.version_info), os.getpid()))
    ... """)
    >>> platform, version_info, remote_pid = channel.receive()
    >>> platform
    'linux2'
    >>> version_info
    (2, 6, 6, 'final', 0)

## Use a callback instead of receive() and wait for completion

Set a channel callback to immediately react on incoming data:

    >>> import execnet
    >>> gw = execnet.makegateway()
    >>> channel = gw.remote_exec("for i in range(10): channel.send(i)")
    >>> l = []
    >>> channel.setcallback(l.append, endmarker=None)
    >>> channel.waitclose() # waits for closing, i.e. remote exec finish
    >>> l
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, None]

Note that the callback function will execute in the receiver thread so it should not block on IO or long to execute.

## Sending channels over channels

You can create and transfer a channel over an existing channel and use it to transfer information:

    >>> import execnet
    >>> gw = execnet.makegateway()
    >>> channel = gw.remote_exec("""
    ...        ch1, ch2 = channel.receive()
    ...        ch2.send("world")
    ...        ch1.send("hello")
    ... """)
    >>> c1 = gw.newchannel()    # create new channel
    >>> c2 = gw.newchannel()    # create another channel
    >>> channel.send((c1, c2))  # send them over
    >>> c1.receive()
    'hello'
    >>> c2.receive()
    'world'

## A simple command loop pattern

If you want the remote side to serve a number of synchronous function calls into your module you can setup a serving loop and write a local protocol.

    import os

    # contents of: remotecmd.py
    def simple(arg):
        return arg + 1

    def listdir(path):
        return os.listdir(path)

    if __name__ == "__channelexec__":
        for item in channel:  # type: ignore[name-defined]
            channel.send(eval(item))  # type: ignore[name-defined]

Then on the local side you can do:

    >>> import execnet, remotecmd
    >>> gw = execnet.makegateway()
    >>> ch = gw.remote_exec(remotecmd)
    >>> ch.send('simple(10)') # execute func-call remotely
    >>> ch.receive()
    11

Our remotecmd module starts up remote serving through the `for item in channel` loop which will terminate when the channel closes. It evaluates all incoming requests in the global name space and sends back the results.

## Instantiate gateways through sockets

In cases where you do not have SSH-access to a machine you need to download a small version-independent standalone socketserver.py script to provide a remote bootstrapping-point. You do not need to install the execnet package remotely. Simply run the script like this:

    python socketserver.py :8888   # bind to all IPs, port 8888

You can then instruct execnet on your local machine to bootstrap itself into the remote socket endpoint:

    import execnet
    gw = execnet.makegateway("socket=TARGET-IP:8888")

That’s it, you can now use the gateway object just like a popen- or SSH-based one.

## Receive file contents from remote SSH account

Here is some small server code that you can use to retrieve contents of remote files:

    # content of servefiles.py

    def servefiles(channel):
        for fn in channel:
            f = open(fn, "rb")
            channel.send(f.read())
            f.close()

    if __name__ == "__channelexec__":
        servefiles(channel)  # type: ignore[name-defined]

And here is some code to use it to retrieve remote contents:

    import execnet
    import servefiles
    gw = execnet.makegateway("ssh=codespeak.net")
    channel = gw.remote_exec(servefiles)

    for fn in ('/etc/passwd', '/etc/group'):
        channel.send(fn)
        content = channel.receive()
        print(fn)
        print(content)