Repeatable buildouts: controlling eggs used
===========================================

One of the goals of zc.buildout is to provide enough control to make
buildouts repeatable.  It should be possible to check the buildout
configuration files for a project into a version control system and
later use the checked in files to get the same buildout, subject to
changes in the environment outside the buildout.

An advantage of using Python eggs is that dependencies of eggs used are
automatically determined and used.  The automatic inclusion of
dependent distributions is at odds with the goal of repeatable
buildouts.

To support repeatable buildouts, a versions section can be created
with options for each distribution name who's version is to be fixed.
The section can then be specified via the buildout versions option.

To see how this works, we'll create two versions of a recipe egg:

    >>> mkdir('recipe')
    >>> write('recipe', 'recipe.py',
    ... '''
    ... import sys
    ... print_ = lambda *a: sys.stdout.write(' '.join(map(str, a))+'\\n')
    ... class Recipe:
    ...     def __init__(*a): pass
    ...     def install(self):
    ...         print_('recipe v1')
    ...         return ()
    ...     update = install
    ... ''')

    >>> write('recipe', 'setup.py',
    ... '''
    ... from setuptools import setup
    ... setup(name='spam', version='1', py_modules=['recipe'],
    ...       entry_points={'zc.buildout': ['default = recipe:Recipe']},
    ...       )
    ... ''')

    >>> write('recipe', 'README', '')

    >>> print_(system(buildout+' setup recipe bdist_egg')) # doctest: +ELLIPSIS
    Running setup script 'recipe/setup.py'.
    ...

    >>> rmdir('recipe', 'build')

    >>> write('recipe', 'recipe.py',
    ... '''
    ... import sys
    ... print_ = lambda *a: sys.stdout.write(' '.join(map(str, a))+'\\n')
    ... class Recipe:
    ...     def __init__(*a): pass
    ...     def install(self):
    ...         print_('recipe v2')
    ...         return ()
    ...     update = install
    ... ''')

    >>> write('recipe', 'setup.py',
    ... '''
    ... from setuptools import setup
    ... setup(name='spam', version='2', py_modules=['recipe'],
    ...       entry_points={'zc.buildout': ['default = recipe:Recipe']},
    ...       )
    ... ''')


    >>> print_(system(buildout+' setup recipe bdist_egg')) # doctest: +ELLIPSIS
    Running setup script 'recipe/setup.py'.
    ...

and we'll configure a buildout to use it:

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = foo
    ... find-links = %s
    ...
    ... [foo]
    ... recipe = spam
    ... ''' % join('recipe', 'dist'))

If we run the buildout, it will use version 2:

    >>> print_(system(buildout), end='')
    Getting distribution for 'spam'.
    Got spam 2.
    Installing foo.
    recipe v2

We can specify a versions section that lists our recipe and name it in
the buildout section:

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = foo
    ... find-links = %s
    ...
    ... [versions]
    ... spam = 1
    ... eggs = 2.2
    ...
    ... [foo]
    ... recipe = spam
    ... ''' % join('recipe', 'dist'))

Here we created a versions section listing the version 1 for the spam
distribution.  We told the buildout to use it by specifying release-1
as in the versions option.

Now, if we run the buildout, we'll use version 1 of the spam recipe:

    >>> print_(system(buildout), end='')
    Getting distribution for 'spam==1'.
    Got spam 1.
    Uninstalling foo.
    Installing foo.
    recipe v1

Running the buildout in verbose mode will help us get information
about versions used. If we run the buildout in verbose mode without
specifying a versions section:

    >>> print_(system(buildout+' buildout:versions= -v'), end='')
    Installing 'zc.buildout', 'wheel', 'pip', 'setuptools'.
    ...
    Installing 'spam'.
    We have the best distribution that satisfies 'spam'.
    Picked: spam = 2.
    Uninstalling foo.
    Installing foo.
    recipe v2

We'll get output that includes lines that tell us what versions
buildout chose a for us, like::

    zc.buildout.easy_install.picked: spam = 2

This allows us to discover versions that are picked dynamically, so
that we can fix them in a versions section.

If we run the buildout with the versions section:

    >>> print_(system(buildout+' -v'), end='')
    Installing 'zc.buildout', 'wheel', 'pip', 'setuptools'.
    ...
    Installing 'spam'.
    We have the distribution that satisfies 'spam==1'.
    Uninstalling foo.
    Installing foo.
    recipe v1

We won't get output for the spam distribution, which we didn't pick,
but we will get output for setuptools, which we didn't specify
versions for.

.. Edge case: version applied to range requirement:

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = foo
    ... find-links = %s
    ...
    ... [versions]
    ... spam = 1
    ... eggs = 2.2
    ...
    ... [foo]
    ... recipe = spam >0
    ... ''' % join('recipe', 'dist'))

    >>> print_(system(buildout+' -v'), end='')
    Installing 'zc.buildout', 'wheel', 'pip', 'setuptools'.
    ...
    Installing 'spam >0'.
    We have the distribution that satisfies 'spam==1'.
    Uninstalling foo.
    Installing foo.
    recipe v1

Edge case (issue #577) where a substitution inside the buildout section
which comes from a section with a recipe, then the versions versions
specifications were ignored, and the latest version installed, even if
allow-picked-versions is false.

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... allow-picked-versions = false
    ... #show-picked-versions = true
    ... parts = foo
    ... find-links = %s
    ... test = ${foo:option}
    ...
    ... [versions]
    ... spam = 1
    ...
    ... [foo]
    ... recipe = spam
    ... option = TEST
    ... ''' % join('recipe', 'dist'))

    >>> print_(system(buildout), end='') # doctest: +ELLIPSIS
    Uninstalling foo.
    Section `buildout` contains unused option(s): 'test'.
    ...
    Installing foo.
    recipe v1

You can request buildout to generate an error if it picks any
versions:

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = foo
    ... find-links = %s
    ... allow-picked-versions = false
    ...
    ... [versions]
    ... eggs = 2.2
    ...
    ... [foo]
    ... recipe = spam
    ... ''' % join('recipe', 'dist'))
    >>> print_(system(buildout), end='') # doctest: +ELLIPSIS
    While:
      Installing.
      Getting section foo.
      Initializing section foo.
      Installing recipe spam.
      Getting distribution for 'spam'.
    Error: Picked: spam = 2
    ...

We can name a version something else, if we wish, using the versions option:

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = foo
    ... find-links = %s
    ... versions = release1
    ...
    ... [release1]
    ... spam = 1
    ... eggs = 2.2
    ...
    ... [foo]
    ... recipe = spam
    ... ''' % join('recipe', 'dist'))
    >>> print_(system(buildout), end='') # doctest: +ELLIPSIS
    Uninstalling foo.
    Installing foo.
    recipe v1

We can also disable checking versions:

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = foo
    ... find-links = %s
    ... versions =
    ...
    ... [versions]
    ... spam = 1
    ... eggs = 2.2
    ...
    ... [foo]
    ... recipe = spam
    ... ''' % join('recipe', 'dist'))
    >>> print_(system(buildout), end='') # doctest: +ELLIPSIS
    Uninstalling foo.
    Installing foo.
    recipe v2


Easier reporting and managing of versions (new in buildout 2.0)
---------------------------------------------------------------

Since buildout 2.0, the functionality of the `buildout-versions
<http://packages.python.org/buildout-versions/>`_ extension is part of
buildout itself. This makes reporting and managing versions easier.

Buildout picks versions for pip and setuptools and for the tests, we need to grab the
version number:

    >>> import pkg_resources
    >>> req = pkg_resources.Requirement.parse('setuptools')
    >>> setuptools_version = pkg_resources.working_set.find(req).version
    >>> req = pkg_resources.Requirement.parse('pip')
    >>> pip_version = pkg_resources.working_set.find(req).version

If you set the ``show-picked-versions`` option, buildout will print
versions it picked at the end of its run:

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = foo
    ... find-links = %s
    ... show-picked-versions = true
    ...
    ... [versions]
    ...
    ... [foo]
    ... recipe = spam
    ... ''' % join('recipe', 'dist'))
    >>> print_(system(buildout), end='') # doctest: +ELLIPSIS
    Updating foo.
    recipe v2
    Versions had to be automatically picked.
    The following part definition lists the versions picked:
    [versions]
    spam = 2

When everything is pinned, no output is generated:

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = foo
    ... find-links = %s
    ... show-picked-versions = true
    ...
    ... [versions]
    ... pip = %s
    ... setuptools = %s
    ... spam = 2
    ...
    ... [foo]
    ... recipe = spam
    ... ''' % (join('recipe', 'dist'), pip_version, setuptools_version))
    >>> print_(system(buildout), end='') # doctest: +ELLIPSIS
    Updating foo.
    recipe v2

The Python package index is case-insensitive. Both
https://pypi.org/simple/Django/ and
https://pypi.org/simple/dJaNgO/ work. And distributions aren't always
naming themselves consistently case-wise. So all version names are normalized
and case differences won't impact the pinning:

    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = foo
    ... find-links = %s
    ... show-picked-versions = true
    ...
    ... [versions]
    ... pip = %s
    ... setuptools = %s
    ... Spam = 2
    ...
    ... [foo]
    ... recipe = spam
    ... ''' % (join('recipe', 'dist'), pip_version, setuptools_version))
    >>> print_(system(buildout), end='') # doctest: +ELLIPSIS
    Updating foo.
    recipe v2

Sometimes it is handy to have a separate file with versions. This is a regular
buildout file with a single ``[versions]`` section. You include it by
extending from that versions file:

    >>> write('my_versions.cfg',
    ... '''
    ... [versions]
    ... pip = %s
    ... setuptools = %s
    ... spam = 2
    ... ''' % (pip_version, setuptools_version))
    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = foo
    ... extends = my_versions.cfg
    ... find-links = %s
    ... show-picked-versions = true
    ...
    ... [foo]
    ... recipe = spam
    ... ''' % join('recipe', 'dist'))
    >>> print_(system(buildout), end='') # doctest: +ELLIPSIS
    Updating foo.
    recipe v2

If not everything is pinned and buildout has to pick versions, you can tell
buildout to append the versions to your versions file. It simply appends them
at the end.

    >>> write('my_versions.cfg',
    ... '''
    ... [versions]
    ... pip = %s
    ... setuptools = %s
    ... ''' % (pip_version, setuptools_version))
    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = foo
    ... extends = my_versions.cfg
    ... update-versions-file = my_versions.cfg
    ... find-links = %s
    ... show-picked-versions = true
    ...
    ... [foo]
    ... recipe = spam
    ... ''' % join('recipe', 'dist'))
    >>> print_(system(buildout), end='') # doctest: +ELLIPSIS
    Updating foo.
    recipe v2
    Versions had to be automatically picked.
    The following part definition lists the versions picked:
    [versions]
    spam = 2
    Picked versions have been written to my_versions.cfg

The versions file now contains the extra pin:

    >>> with open('my_versions.cfg') as f: print_(f.read()) # doctest: +ELLIPSIS
    <BLANKLINE>
    ...
    # Added by buildout at YYYY-MM-DD hh:mm:ss.dddddd
    spam = 2
    <BLANKLINE>

And re-running buildout doesn't report any picked versions anymore:

    >>> 'picked' in system(buildout)
    False

If you've enabled ``update-versions-file`` but not ``show-picked-versions``,
buildout will append the versions to your versions file anyway (without
printing them to the console):

    >>> write('my_versions.cfg',
    ... '''
    ... [versions]
    ... pip = %s
    ... setuptools = %s
    ... ''' % (pip_version, setuptools_version))
    >>> write('buildout.cfg',
    ... '''
    ... [buildout]
    ... parts = foo
    ... extends = my_versions.cfg
    ... update-versions-file = my_versions.cfg
    ... find-links = %s
    ... show-picked-versions = false
    ...
    ... [foo]
    ... recipe = spam
    ... ''' % join('recipe', 'dist'))
    >>> print_(system(buildout), end='') # doctest: +ELLIPSIS
    Updating foo.
    recipe v2
    Picked versions have been written to my_versions.cfg

The versions file contains the extra pin:

    >>> with open('my_versions.cfg') as f: print_(f.read()) # doctest: +ELLIPSIS
    <BLANKLINE>
    [versions]
    ...
    <BLANKLINE>
    # Added by buildout at YYYY-MM-DD hh:mm:ss.dddddd
    spam = 2
    <BLANKLINE>

Because buildout now includes buildout-versions' (and part of the older
buildout.dumppickedversions') functionality, it warns if these extensions are
configured.

    >>> write(sample_buildout, 'buildout.cfg',
    ... """
    ... [buildout]
    ... parts = foo
    ... extensions = buildout-versions
    ...
    ... [foo]
    ... recipe = spam
    ... """)
    >>> print_(system(buildout), end='') # doctest: +NORMALIZE_WHITESPACE
    While:
      Installing.
      Loading extensions.
      Error: Buildout now includes 'buildout-versions'
      (and part of the older 'buildout.dumppickedversions').
      Remove the extension from your configuration and look at the
      'show-picked-versions' option in buildout's documentation.
