# Storage finders

Graphite-API searches and fetches metrics from time series databases using an interface called finders. The default finder provided with Graphite-API is the one that integrates with Whisper databases.

Customizing finders can be done in the `finders` section of the Graphite-API configuration file:

    finders:
      - graphite_api.finders.whisper.WhisperFinder

Several values are allowed, to let you store different kinds of metrics at different places or smoothly handle transitions from one time series database to another.

The default finder reads data from a Whisper database.

## Custom finders

`finders` being a list of arbitrary python paths, it is relatively easy to write a custom finder if you want to read data from other places than Whisper. A finder is a python class with a `find_nodes()` method:

    class CustomFinder(object):
        def find_nodes(self, query):
            # ...

`query` is a `FindQuery` object. `find_nodes()` is the entry point when browsing the metrics tree. It must yield leaf or branch nodes matching the query:

    from graphite_api.node import LeafNode, BranchNode

    class CustomFinder(object):
        def find_nodes(self, query):
            # find some paths matching the query, then yield them
            # is_branch or is_leaf are predicates you need to implement
            for path in matches:
                if is_branch(path):
                    yield BranchNode(path)
                if is_leaf(path):
                    yield LeafNode(path, CustomReader(path))

`LeafNode` is created with a reader, which is the class responsible for fetching the datapoints for the given path. It is a simple class with 2 methods: `fetch()` and `get_intervals()`:

    from graphite_api.intervals import IntervalSet, Interval

    class CustomReader(object):
        __slots__ = ('path',)  # __slots__ is recommended to save memory on readers

        def __init__(self, path):
            self.path = path

        def fetch(self, start_time, end_time):
            # fetch data
            time_info = _from_, _to_, _step_
            return time_info, series

        def get_intervals(self):
            return IntervalSet([Interval(start, end)])

`fetch()` must return a list of 2 elements: the time info for the data and the datapoints themselves. The time info is a list of 3 items: the start time of the datapoints (in unix time), the end time and the time step (in seconds) between the datapoints.

The datapoints is a list of points found in the database for the required interval. There must be `(end - start) / step` points in the dataset even if the database has gaps: gaps can be filled with `None` values.

`get_intervals()` is a method that hints graphite-web about the time range available for this given metric in the database. It must return an `IntervalSet` of one or more `Interval` objects.

## Fetching multiple paths at once

If your storage backend allows it, fetching multiple paths at once is useful to avoid sequential fetches and save time and resources. This can be achieved in three steps:

- Subclass `LeafNode` and add a `__fetch_multi__` class attribute to your subclass:

      class CustomLeafNode(LeafNode):
          __fetch_multi__ = 'custom'

  The string `'custom'` is used to identify backends and needs to be unique per-backend.

- Add the `__fetch_multi__` attribute to your finder class:

      class CustomFinder(objects):
          __fetch_multi__ = 'custom'

- Implement a `fetch_multi()` method on your finder:

      class CustomFinder(objects):
          def fetch_multi(self, nodes, start_time, end_time):
              paths = [node.path for node in nodes]
              # fetch paths
              return time_info, series

  `time_info` is the same structure as the one returned by `fetch()`. `series` is a dictionnary with paths as keys and datapoints as values.

## Installing custom finders

In order for your custom finder to be importable, you need to package it under a namespace of your choice. Python packaging won’t be covered here but you can look at third-party finders to get some inspiration:

- Cyanite finder

## Configuration

Graphite-API instantiates finders and passes it its whole parsed configuration file, as a Python data structure. External finders can require extra sections in the configuration file to setup access to the time series database they communicate with. For instance, let’s say your `CustomFinder` needs two configuration parameters, a host and a user:

    class CustomFinder(object):
        def __init__(self, config):
            config.setdefault('custom', {})
            self.user = config['custom'].get('user', 'default')
            self.host = config['custom'].get('host', 'localhost')

The configuration file would look like:

    finders:
      - custom.CustomFinder
    custom:
      user: myuser
      host: example.com

When possible, try to use sane defaults that would “just work” for most common setups. Here if the `custom` section isn’t provided, the finder uses `default` as user and `localhost` as host.