# Complex Applications

Click is designed to assist with the creation of complex and simple CLI tools alike. However, the power of its design is the ability to arbitrarily nest systems together. For instance, if you have ever used Django, you will have realized that it provides a command line utility, but so does Celery. When using Celery with Django, there are two tools that need to interact with each other and be cross-configured.

In a theoretical world of two separate Click command line utilities, they could solve this problem by nesting one inside the other. For instance, the web framework could also load the commands for the message queue framework.

## Basic Concepts

To understand how this works, you need to understand two concepts: contexts and the calling convention.

### Contexts

Whenever a Click command is executed, a `Context` object is created which holds state for this particular invocation. It remembers parsed parameters, what command created it, which resources need to be cleaned up at the end of the function, and so forth. It can also optionally hold an application-defined object.

Context objects build a linked list until they hit the top one. Each context is linked to a parent context. This allows a command to work below another command and store its own information there without having to be afraid of altering up the state of the parent command.

Because the parent data is available, however, it is possible to navigate to it if needed.

Most of the time, you do not see the context object, but when writing more complex applications it comes in handy. This brings us to the next point.

### Calling Convention

When a Click command callback is executed, it’s passed all the non-hidden parameters as keyword arguments. Notably absent is the context. However, a callback can opt into being passed to the context object by marking itself with `pass_context()`.

So how do you invoke a command callback if you don’t know if it should receive the context or not? The answer is that the context itself provides a helper function (`Context.invoke()`) which can do this for you. It accepts the callback as first argument and then invokes the function correctly.

## Building a Git Clone

In this example, we want to build a command line tool that resembles a version control system. Systems like Git usually provide one over-arching command that already accepts some parameters and configuration, and then have extra subcommands that do other things.

### The Root Command

At the top level, we need a group that can hold all our commands. In this case, we use the basic `click.group()` which allows us to register other Click commands below it.

For this command, we also want to accept some parameters that configure the state of our tool:

    import os
    import click

    class Repo(object):
        def __init__(self, home=None, debug=False):
            self.home = os.path.abspath(home or '.')
            self.debug = debug

    @click.group()
    @click.option('--repo-home', envvar='REPO_HOME', default='.repo')
    @click.option('--debug/--no-debug', default=False,
                  envvar='REPO_DEBUG')
    @click.pass_context
    def cli(ctx, repo_home, debug):
        ctx.obj = Repo(repo_home, debug)

Let’s understand what this does. We create a group command which can have subcommands. When it is invoked, it will create an instance of a `Repo` class. This holds the state for our command line tool. In this case, it just remembers some parameters, but at this point it could also start loading configuration files and so on.

This state object is then remembered by the context as `obj`. This is a special attribute where commands are supposed to remember what they need to pass on to their children.

In order for this to work, we need to mark our function with `pass_context()`, because otherwise, the context object would be entirely hidden from us.

### The First Child Command

Let’s add our first child command to it, the clone command:

    @cli.command()
    @click.argument('src')
    @click.argument('dest', required=False)
    def clone(src, dest):
        pass

So now we have a clone command, but how do we get access to the repo? As you can imagine, one way is to use the `pass_context()` function which again will make our callback also get the context passed on which we memorized the repo. However, there is a second version of this decorator called `pass_obj()` which will just pass the stored object, (in our case the repo):

    @cli.command()
    @click.argument('src')
    @click.argument('dest', required=False)
    @click.pass_obj
    def clone(repo, src, dest):
        pass

### Interleaved Commands

While not relevant for the particular program we want to build, there is also quite good support for interleaving systems. Imagine for instance that there was a super cool plugin for our version control system that needed a lot of configuration and wanted to store its own configuration as `obj`. If we would then attach another command below that, we would all of a sudden get the plugin configuration instead of our repo object.

One obvious way to remedy this is to store a reference to the repo in the plugin, but then a command needs to be aware that it’s attached below such a plugin.

There is a much better system that can be built by taking advantage of the linked nature of contexts. We know that the plugin context is linked to the context that created our repo. Because of that, we can start a search for the last level where the object stored by the context was a repo.

Built-in support for this is provided by the `make_pass_decorator()` factory, which will create decorators for us that find objects (it internally calls into `Context.find_object()`). In our case, we know that we want to find the closest `Repo` object, so let’s make a decorator for this:

    pass_repo = click.make_pass_decorator(Repo)

If we now use `pass_repo` instead of `pass_obj`, we will always get a repo instead of something else:

    @cli.command()
    @click.argument('src')
    @click.argument('dest', required=False)
    @pass_repo
    def clone(repo, src, dest):
        pass

### Ensuring Object Creation

The above example only works if there was an outer command that created a `Repo` object and stored it in the context. For some more advanced use cases, this might become a problem. The default behavior of `make_pass_decorator()` is to call `Context.find_object()` which will find the object. If it can’t find the object, `make_pass_decorator()` will raise an error. The alternative behavior is to use `Context.ensure_object()` which will find the object, and if it cannot find it, will create one and store it in the innermost context. This behavior can also be enabled for `make_pass_decorator()` by passing `ensure=True`:

    pass_repo = click.make_pass_decorator(Repo, ensure=True)

In this case, the innermost context gets an object created if it is missing. This might replace objects being placed there earlier. In this case, the command stays executable, even if the outer command does not run. For this to work, the object type needs to have a constructor that accepts no arguments.

As such it runs standalone:

    @click.command()
    @pass_repo
    def cp(repo):
        click.echo(isinstance(repo, Repo))

As you can see:

    $ cp
    True

## Lazily Loading Subcommands

Large CLIs and CLIs with slow imports may benefit from deferring the loading of subcommands. The interfaces which support this mode of use are `MultiCommand.list_commands()` and `MultiCommand.get_command()`. A custom `MultiCommand` subclass can implement a lazy loader by storing extra data such that `MultiCommand.get_command()` is responsible for running imports.

Since the primary case for this is a `Group` which loads its subcommands lazily, the following example shows a lazy-group implementation.

Warning

Lazy loading of python code can result in hard to track down bugs, circular imports in order-dependent codebases, and other surprising behaviors. It is recommended that this technique only be used in concert with testing which will at least run the `--help` on each subcommand. That will guarantee that each subcommand can be loaded successfully.

### Defining the Lazy Group

The following `Group` subclass adds an attribute, `lazy_subcommands`, which stores a mapping from subcommand names to the information for importing them.

    # in lazy_group.py
    import importlib
    import click

    class LazyGroup(click.Group):
        def __init__(self, *args, lazy_subcommands=None, **kwargs):
            super().__init__(*args, **kwargs)
            # lazy_subcommands is a map of the form:
            #
            #   {command-name} -> {module-name}.{command-object-name}
            #
            self.lazy_subcommands = lazy_subcommands or {}

        def list_commands(self, ctx):
            base = super().list_commands(ctx)
            lazy = sorted(self.lazy_subcommands.keys())
            return base + lazy

        def get_command(self, ctx, cmd_name):
            if cmd_name in self.lazy_subcommands:
                return self._lazy_load(cmd_name)
            return super().get_command(ctx, cmd_name)

        def _lazy_load(self, cmd_name):
            # lazily loading a command, first get the module name and attribute name
            import_path = self.lazy_subcommands[cmd_name]
            modname, cmd_object_name = import_path.rsplit(".", 1)
            # do the import
            mod = importlib.import_module(modname)
            # get the Command object from that module
            cmd_object = getattr(mod, cmd_object_name)
            # check the result to make debugging easier
            if not isinstance(cmd_object, click.BaseCommand):
                raise ValueError(
                    f"Lazy loading of {import_path} failed by returning "
                    "a non-command object"
                )
            return cmd_object

### Using LazyGroup To Define a CLI

With `LazyGroup` defined, it’s now possible to write a group which lazily loads its subcommands like so:

    # in main.py
    import click
    from lazy_group import LazyGroup

    @click.group(
        cls=LazyGroup,
        lazy_subcommands={"foo": "foo.cli", "bar": "bar.cli"},
        help="main CLI command for lazy example",
    )
    def cli():
        pass

    # in foo.py
    import click

    @click.group(help="foo command for lazy example")
    def cli():
        pass

    # in bar.py
    import click
    from lazy_group import LazyGroup

    @click.group(
        cls=LazyGroup,
        lazy_subcommands={"baz": "baz.cli"},
        help="bar command for lazy example",
    )
    def cli():
        pass

    # in baz.py
    import click

    @click.group(help="baz command for lazy example")
    def cli():
        pass

### What triggers Lazy Loading?

There are several events which may trigger lazy loading by running the `MultiCommand.get_command()` function. Some are intuititve, and some are less so.

All cases are described with respect to the above example, assuming the main program name is `cli`.

1. Command resolution. If a user runs `cli bar baz`, this must first resolve `bar`, and then resolve `baz`. Each subcommand resolution step does a lazy load.

2. Helptext rendering. In order to get the short help description of subcommands, `cli --help` will load `foo` and `bar`. Note that it will still not load `baz`.

3. Shell completion. In order to get the subcommands of a lazy command, `cli <TAB>` will need to resolve the subcommands of `cli`. This process will trigger the lazy loads.

### Further Deferring Imports

It is possible to make the process even lazier, but it is generally more difficult the more you want to defer work.

For example, subcommands could be represented as a custom `BaseCommand` subclass which defers importing the command until it is invoked, but which provides `BaseCommand.get_short_help_str()` in order to support completions and helptext. More simply, commands can be constructed whose callback functions defer any actual work until after an import.

This command definition provides `foo`, but any of the work associated with importing the “real” callback function is deferred until invocation time:

    @click.command()
    @click.option("-n", type=int)
    @click.option("-w", type=str)
    def foo(n, w):
        from mylibrary import foo_concrete

        foo_concrete(n, w)

Because `click` builds helptext and usage info from options, arguments, and command attributes, it has no awareness that the underlying function is in any way handling a deferred import. Therefore, all `click`-provided utilities and functionality will work as normal on such a command.