Caching extended configuration
==============================

As mentioned in the general buildout documentation, configuration files can
extend each other, including the ability to download configuration being
extended from a URL. If desired, zc.buildout caches downloaded configuration
in order to be able to use it when run offline.

As we're going to talk about downloading things, let's start an HTTP server.
Also, all of the following will take place inside the sample buildout.

>>> server_data = tmpdir('server_data')
>>> server_url = start_server(server_data)
>>> cd(sample_buildout)

We also use a fresh directory for temporary files in order to make sure that
all temporary files have been cleaned up in the end:

>>> import tempfile
>>> old_tempdir = tempfile.tempdir
>>> tempfile.tempdir = tmpdir('tmp')


Basic use of the extends cache
------------------------------

We put some base configuration on a server and reference it from a sample
buildout:

>>> write(server_data, 'base.cfg', """\
... [buildout]
... parts =
... foo = bar
... """)

>>> write('buildout.cfg', """\
... [buildout]
... extends = %sbase.cfg
... """ % server_url)

When trying to run this buildout offline, we'll find that we cannot read all
of the required configuration:

>>> print_(system(buildout + ' -o'))
While:
  Initializing.
Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

Trying the same online, we can:

>>> print_(system(buildout))
Section `buildout` contains unused option(s): 'foo'.
This may be an indication for either a typo in the option's name or a bug in the used recipe.

As long as we haven't said anything about caching downloaded configuration,
nothing gets cached. Offline mode will still cause the buildout to fail:

>>> print_(system(buildout + ' -o'))
While:
  Initializing.
Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

Let's now specify a cache for base configuration files. This cache is
different from the download cache used by recipes for caching distributions
and other files; one might, however, use a namespace subdirectory of the
download cache for it. The configuration cache we specify will be created when
running buildout and the base.cfg file will be put in it (with the file name
being a hash of the complete URL):

>>> mkdir('cache')
>>> write('buildout.cfg', """\
... [buildout]
... extends = %sbase.cfg
... extends-cache = cache
... """ % server_url)

>>> print_(system(buildout))
Section `buildout` contains unused option(s): 'foo'.
This may be an indication for either a typo in the option's name or a bug in the used recipe.

>>> cache = join(sample_buildout, 'cache')
>>> ls(cache)
-  5aedc98d7e769290a29d654a591a3a45

>>> import os
>>> cat(cache, os.listdir(cache)[0])
[buildout]
parts =
foo = bar

We can now run buildout offline as it will read base.cfg from the cache:

>>> print_(system(buildout + ' -o'))
Section `buildout` contains unused option(s): 'foo'.
This may be an indication for either a typo in the option's name or a bug in the used recipe.

The cache is being used purely as a fall-back in case we are offline or don't
have access to a configuration file to be downloaded. As long as we are
online, buildout attempts to download a fresh copy of each file even if a
cached copy of the file exists. To see this, we put different configuration in
the same place on the server and run buildout in offline mode so it takes
base.cfg from the cache:

>>> write(server_data, 'base.cfg', """\
... [buildout]
... parts =
... bar = baz
... """)

>>> print_(system(buildout + ' -o'))
Section `buildout` contains unused option(s): 'foo'.
This may be an indication for either a typo in the option's name or a bug in the used recipe.

.. verify same behavior for install-from-cache:

   >>> print_(system(buildout + ' install-from-cache=true download-cache=.'))
   Section `buildout` contains unused option(s): 'foo'.
   This may be an indication for either a typo in the option's name or a bug in the used recipe.

In online mode, buildout will download and use the modified version:

>>> print_(system(buildout))
Section `buildout` contains unused option(s): 'bar'.
This may be an indication for either a typo in the option's name or a bug in the used recipe.

Trying offline mode again, the new version will be used as it has been put in
the cache now:

>>> print_(system(buildout + ' -o'))
Section `buildout` contains unused option(s): 'bar'.
This may be an indication for either a typo in the option's name or a bug in the used recipe.

Clean up:

>>> rmdir(cache)


Specifying extends cache and offline mode
-----------------------------------------

Normally, the values of buildout options such as the location of a download
cache or whether to use offline mode are determined by first reading the
user's default configuration, updating it with the project's configuration and
finally applying command-line options. User and project configuration are
assembled by reading a file such as ``~/.buildout/default.cfg``,
``buildout.cfg`` or a URL given on the command line, recursively (depth-first)
downloading any base configuration specified by the ``buildout:extends``
option read from each of those config files, and finally evaluating each
config file to provide default values for options not yet read.

This works fine for all options that do not influence how configuration is
downloaded in the first place. The ``extends-cache`` and ``offline`` options,
however, are treated differently from the procedure described in order to make
it simple and obvious to see where a particular configuration file came from
under any particular circumstances.

- Offline and extends-cache settings are read from the two root config files
  exclusively. Otherwise one could construct configuration files that, when
  read, imply that they should have been read from a different source than
  they have. Also, specifying the extends cache within a file that might have
  to be taken from the cache before being read wouldn't make a lot of sense.

- Offline and extends-cache settings given by the user's defaults apply to the
  process of assembling the project's configuration. If no extends cache has
  been specified by the user's default configuration, the project's root
  config file must be available, be it from disk or from the net.

- Offline mode turned on by the ``-o`` command line option is honored from
  the beginning even though command line options are applied to the
  configuration last. If offline mode is not requested by the command line, it
  may be switched on by either the user's or the project's config root.

Extends cache
~~~~~~~~~~~~~

Let's see the above rules in action. We create a new home directory for our
user and write user and project configuration that recursively extends online
bases, using different caches:

>>> mkdir('home')
>>> mkdir('home', '.buildout')
>>> mkdir('cache')
>>> mkdir('user-cache')
>>> home=join(sample_buildout, 'home')
>>> env=dict(HOME=home, USERPROFILE=home)
>>> import functools
>>> system = functools.partial(system, env=env)
>>> write('home', '.buildout', 'default.cfg', """\
... [buildout]
... extends = fancy_default.cfg
... extends-cache = user-cache
... """)
>>> write('home', '.buildout', 'fancy_default.cfg', """\
... [buildout]
... extends = %sbase_default.cfg
... """ % server_url)
>>> write(server_data, 'base_default.cfg', """\
... [buildout]
... foo = bar
... offline = false
... """)

>>> write('buildout.cfg', """\
... [buildout]
... extends = fancy.cfg
... extends-cache = cache
... """)
>>> write('fancy.cfg', """\
... [buildout]
... extends = %sbase.cfg
... """ % server_url)
>>> write(server_data, 'base.cfg', """\
... [buildout]
... parts =
... offline = false
... """)

Buildout will now assemble its configuration from all of these 6 files,
defaults first. The online resources end up in the respective extends caches:

>>> print_(system(buildout))
Section `buildout` contains unused option(s): 'foo'.
This may be an indication for either a typo in the option's name or a bug in the used recipe.

>>> ls('user-cache')
-  10e772cf422123ef6c64ae770f555740
>>> cat('user-cache', os.listdir('user-cache')[0])
[buildout]
foo = bar
offline = false

>>> ls('cache')
-  c72213127e6eb2208a3e1fc1dba771a7
>>> cat('cache', os.listdir('cache')[0])
[buildout]
parts =
offline = false

If, on the other hand, the extends caches are specified in files that get
extended themselves, they won't be used for assembling the configuration they
belong to (user's or project's, resp.). The extends cache specified by the
user's defaults does, however, apply to downloading project configuration.
Let's rewrite the config files, clean out the caches and re-run buildout:

>>> write('home', '.buildout', 'default.cfg', """\
... [buildout]
... extends = fancy_default.cfg
... """)
>>> write('home', '.buildout', 'fancy_default.cfg', """\
... [buildout]
... extends = %sbase_default.cfg
... extends-cache = user-cache
... """ % server_url)

>>> write('buildout.cfg', """\
... [buildout]
... extends = fancy.cfg
... """)
>>> write('fancy.cfg', """\
... [buildout]
... extends = %sbase.cfg
... extends-cache = cache
... """ % server_url)

>>> remove('user-cache', os.listdir('user-cache')[0])
>>> remove('cache', os.listdir('cache')[0])

>>> print_(system(buildout))
Section `buildout` contains unused option(s): 'foo'.
This may be an indication for either a typo in the option's name or a bug in the used recipe.

>>> ls('user-cache')
-  0548bad6002359532de37385bb532e26
>>> cat('user-cache', os.listdir('user-cache')[0])
[buildout]
parts =
offline = false

>>> ls('cache')

Clean up:

>>> rmdir('user-cache')
>>> rmdir('cache')

Offline mode and installation from cache
----------------------------------------

If we run buildout in offline mode now, it will fail because it cannot get at
the remote configuration file needed by the user's defaults:

>>> print_(system(buildout + ' -o'))
While:
  Initializing.
Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode.

Let's now successively turn on offline mode by different parts of the
configuration and see when buildout applies this setting in each case:

>>> write('home', '.buildout', 'default.cfg', """\
... [buildout]
... extends = fancy_default.cfg
... offline = true
... """)
>>> print_(system(buildout))
While:
  Initializing.
Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode.

>>> write('home', '.buildout', 'default.cfg', """\
... [buildout]
... extends = fancy_default.cfg
... """)
>>> write('home', '.buildout', 'fancy_default.cfg', """\
... [buildout]
... extends = %sbase_default.cfg
... offline = true
... """ % server_url)
>>> print_(system(buildout))
While:
  Initializing.
Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

>>> write('home', '.buildout', 'fancy_default.cfg', """\
... [buildout]
... extends = %sbase_default.cfg
... """ % server_url)
>>> write('buildout.cfg', """\
... [buildout]
... extends = fancy.cfg
... offline = true
... """)
>>> print_(system(buildout))
While:
  Initializing.
Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

>>> write('buildout.cfg', """\
... [buildout]
... extends = fancy.cfg
... """)
>>> write('fancy.cfg', """\
... [buildout]
... extends = %sbase.cfg
... offline = true
... """ % server_url)
>>> print_(system(buildout))
Section `buildout` contains unused option(s): 'foo'.
This may be an indication for either a typo in the option's name or a bug in the used recipe.

The ``install-from-cache`` option is treated accordingly:

>>> write('home', '.buildout', 'default.cfg', """\
... [buildout]
... extends = fancy_default.cfg
... install-from-cache = true
... """)
>>> print_(system(buildout))
While:
  Initializing.
Error: Couldn't download 'http://localhost/base_default.cfg' in offline mode.

>>> write('home', '.buildout', 'default.cfg', """\
... [buildout]
... extends = fancy_default.cfg
... """)
>>> write('home', '.buildout', 'fancy_default.cfg', """\
... [buildout]
... extends = %sbase_default.cfg
... install-from-cache = true
... """ % server_url)
>>> print_(system(buildout))
While:
  Initializing.
Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

>>> write('home', '.buildout', 'fancy_default.cfg', """\
... [buildout]
... extends = %sbase_default.cfg
... """ % server_url)
>>> write('buildout.cfg', """\
... [buildout]
... extends = fancy.cfg
... install-from-cache = true
... """)
>>> print_(system(buildout))
While:
  Initializing.
Error: Couldn't download 'http://localhost/base.cfg' in offline mode.

>>> write('buildout.cfg', """\
... [buildout]
... extends = fancy.cfg
... """)
>>> write('fancy.cfg', """\
... [buildout]
... extends = %sbase.cfg
... install-from-cache = true
... """ % server_url)
>>> print_(system(buildout))
While:
  Installing.
  Checking for upgrades.
An internal error occurred ...
ValueError: install_from_cache set to true with no download cache

>>> rmdir('home', '.buildout')

Newest and non-newest behavior for extends cache
-------------------------------------------------

While offline mode forbids network access completely, 'newest' mode determines
whether to look for updated versions of a resource even if some version of it
is already present locally. If we run buildout in newest mode
(``newest = true``), the configuration files are updated with each run:

>>> mkdir("cache")
>>> write(server_data, 'base.cfg', """\
... [buildout]
... parts =
... """)
>>> write('buildout.cfg', """\
... [buildout]
... extends-cache = cache
... extends = %sbase.cfg
... """ % server_url)
>>> print_(system(buildout))
>>> ls('cache')
-  5aedc98d7e769290a29d654a591a3a45
>>> cat('cache', os.listdir(cache)[0])
[buildout]
parts =

A change to ``base.cfg`` is picked up on the next buildout run:

>>> write(server_data, 'base.cfg', """\
... [buildout]
... parts =
... foo = bar
... """)
>>> print_(system(buildout + " -n"))
Section `buildout` contains unused option(s): 'foo'.
This may be an indication for either a typo in the option's name or a bug in the used recipe.
>>> cat('cache', os.listdir(cache)[0])
[buildout]
parts =
foo = bar

In contrast, when not using ``newest`` mode (``newest = false``), the files
already present in the extends cache will not be updated:

>>> write(server_data, 'base.cfg', """\
... [buildout]
... parts =
... """)
>>> print_(system(buildout + " -N"))
Section `buildout` contains unused option(s): 'foo'.
This may be an indication for either a typo in the option's name or a bug in the used recipe.
>>> cat('cache', os.listdir(cache)[0])
[buildout]
parts =
foo = bar

Even when updating base configuration files with a buildout run, any given
configuration file will be downloaded only once during that particular run. If
some base configuration file is extended more than once, its cached copy is
used:

>>> write(server_data, 'baseA.cfg', """\
... [buildout]
... extends = %sbase.cfg
... foo = bar
... """ % server_url)
>>> write(server_data, 'baseB.cfg', """\
... [buildout]
... extends-cache = cache
... extends = %sbase.cfg
... bar = foo
... """ % server_url)
>>> write('buildout.cfg', """\
... [buildout]
... extends-cache = cache
... newest = true
... extends = %sbaseA.cfg %sbaseB.cfg
... """ % (server_url, server_url))
>>> print_(system(buildout + " -n"))
Section `buildout` contains unused option(s): 'bar' 'foo'.
This may be an indication for either a typo in the option's name or a bug in the used recipe.

(XXX We patch download utility's API to produce readable output for the test;
a better solution would re-use the logging already done by the utility.)

>>> import zc.buildout
>>> old_download = zc.buildout.download.Download.download
>>> def wrapper_download(self, url, md5sum=None, path=None):
...   print_("The URL %s was downloaded." % url)
...   return old_download(url, md5sum, path)
>>> zc.buildout.download.Download.download = wrapper_download

>>> zc.buildout.buildout.main([])
The URL http://localhost/baseA.cfg was downloaded.
The URL http://localhost/base.cfg was downloaded.
The URL http://localhost/baseB.cfg was downloaded.
Not upgrading because not running a local buildout command.
Section `buildout` contains unused option(s): 'bar' 'foo'.
This may be an indication for either a typo in the option's name or a bug in the used recipe.

>>> zc.buildout.download.Download.download = old_download


The deprecated ``extended-by`` option
-------------------------------------

The ``buildout`` section used to recognize an option named ``extended-by``
that was deprecated at some point and removed in the 1.5 line. Since ignoring
this option silently was considered harmful as a matter of principle, a
UserError is raised if that option is encountered now:

>>> write(server_data, 'base.cfg', """\
... [buildout]
... parts =
... extended-by = foo.cfg
... """)
>>> print_(system(buildout))
While:
  Initializing.
Error: No-longer supported "extended-by" option found in http://localhost/base.cfg.


Reporting cached locations for downloads of faulty config files
---------------------------------------------------------------

A downloaded config file might be invalid. A cancelled buildout run, an
accidentally gzip-encoded download and so on.

>>> write(server_data, 'faulty.cfg', """\
... This is definitively not
... a proper() config file.
... """)

>>> write('buildout.cfg', """\
... [buildout]
... extends = %sfaulty.cfg
... """ % server_url)
>>> print_(system(buildout))
While:
  Initializing.
... File contains no section headers.
file: http://localhost/faulty.cfg (downloaded as ...), line: 1
'This is definitively not\n'


Failing if extends-cache contains ${section:variable}
-----------------------------------------------------

Because extends-cache is used to download extends, it may not contain ${section:variable}
as the state of all variables cannot be computed before all extends have been loaded.

>>> write(server_data, 'proper.cfg', """\
... [buildout]
... dummy = fjhfj
... """)

>>> write('buildout.cfg', """\
... [buildout]
... extends = %sproper.cfg
... extends-cache = ${buildout:dummy}
... """ % server_url)
>>> print_(system(buildout))
While:
  Initializing.
... ValueError: extends-cache '${buildout:dummy}' may not contain ${section:variable} to expand.


Clean up
--------

We should have cleaned up all temporary files created by downloading things:

>>> ls(tempfile.tempdir)

Reset the global temporary directory:

>>> tempfile.tempdir = old_tempdir
