# Helpers

## Date Ops

class curator.helpers.date_ops.TimestringSearch(timestring)

  Bases: `object`

  An object to allow repetitive search against a string, `searchme`, without having to repeatedly recreate the regex.

  Parameters:

    timestring (`strftime()`) – An `strftime` pattern

  get_epoch(searchme)
    Parameters:

      searchme – A string to be matched against `pattern` that matches `timestring`

    Returns:

      The epoch timestamp extracted from `searchme` by regex matching against `pattern`

    Return type:

      int or None

  pattern

    Object attribute. `re.compile(regex)` where `regex = r'(?P<date>{0})'.format(get_date_regex(timestring))`. Uses `get_date_regex()`

  timestring

    Object attribute preserving param `timestring`

curator.helpers.date_ops.absolute_date_range(unit, date_from, date_to, date_from_format=None, date_to_format=None)

  This function calculates a date range with an absolute time stamp for both the start time and the end time. These dates are converted to epoch time. The parameter `unit` is used when the same simplified date is used for both `date_from` and `date_to` to calculate the duration. For example, if `unit` is `months`, and `date_from` and `date_to` are both `2017.01`, then the entire month of January 2017 will be the absolute date range.

  Parameters:

    - unit (str) – One of `hours`, `days`, `weeks`, `months`, or `years`.

    - date_from (str) – The simplified date for the start of the range

    - date_to (str) – The simplified date for the end of the range.

    - date_from_format (str) – The `strftime()` string used to parse `date_from`

    - date_to_format (str) – The `strftime()` string used to parse `date_to`

  Returns:

    The epoch start time and end time of a date range

  Return type:

    tuple

curator.helpers.date_ops.date_range(unit, range_from, range_to, epoch=None, week_starts_on='sunday')

  This function calculates a date range with a distinct epoch time stamp of both the start time and the end time in counts of `unit` relative to the time at execution, if `epoch` is not set.

  If `unit` is `weeks`, you can also determine when a week begins using `week_starts_on`, which can be either `sunday` or `monday`.

  Parameters:

    - unit (str) – One of `hours`, `days`, `weeks`, `months`, or `years`.

    - range_from (int) – Count of `unit` in the past/future is the origin?

    - range_to (int) – Count of `unit` in the past/future is the end point?

    - epoch (int) – An epoch timestamp used to establish a point of reference for calculations.

    - week_starts_on (str) – Either `sunday` or `monday`. Default is `sunday`

  Returns:

    The epoch start time and end time of a date range

  Return type:

    tuple

curator.helpers.date_ops.datetime_to_epoch(mydate)

  Converts datetime into epoch seconds

  Parameters:

    mydate (`datetime`) – A Python datetime

  Returns:

    An epoch timestamp based on `mydate`

  Return type:

    int

curator.helpers.date_ops.fix_epoch(epoch)

  Fix value of `epoch` to be the count since the epoch in seconds only, which should be 10 or fewer digits long.

  Parameters:

    epoch (int) – An epoch timestamp, in epoch + milliseconds, or microsecond, or even nanoseconds.

  Returns:

    An epoch timestamp in seconds only, based on `epoch`

  Return type:

    int

curator.helpers.date_ops.get_date_regex(timestring)
  Parameters:

    timestring (`strftime()`) – An `strftime` pattern

  Returns:

    A regex string based on a provided `strftime()` `timestring`.

  Return type:

    str

curator.helpers.date_ops.get_datemath(client, datemath, random_element=None)
  Parameters:

    - client (`Elasticsearch`) – A client connection object

    - datemath (`datemath`) – An elasticsearch datemath string

    - random_element (str) – This allows external randomization of the name and is only useful for tests so that you can guarantee the output because you provided the random portion.

  Returns:

    the parsed index name from `datemath`

  Return type:

    str

curator.helpers.date_ops.get_datetime(index_timestamp, timestring)
  Parameters:

    - index_timestamp (str) – The index timestamp

    - timestring (`strftime()`) – An `strftime` pattern

  Returns:

    The datetime extracted from the index name, which is the index creation time.

  Return type:

    `datetime`

curator.helpers.date_ops.get_point_of_reference(unit, count, epoch=None)
  Parameters:

    - unit (str) – One of `seconds`, `minutes`, `hours`, `days`, `weeks`, `months`, or `years`.

    - unit_count (int) – The number of `units`. `unit_count` * `unit` will be calculated out to the relative number of seconds.

    - epoch (int) – An epoch timestamp used in conjunction with `unit` and `unit_count` to establish a point of reference for calculations.

  Returns:

    A point-of-reference timestamp in epoch + milliseconds by deriving from a `unit` and a `count`, and an optional reference timestamp, `epoch`

  Return type:

    int

curator.helpers.date_ops.get_unit_count_from_name(index_name, pattern)
  Parameters:

    - index_name (str) – An index name

    - pattern (str) – A regular expression pattern

  Returns:

    The unit count, if a match is able to be found in the name

  Return type:

    int

curator.helpers.date_ops.handle_iso_week_number(mydate, timestring, index_timestamp)
  Parameters:

    - mydate (`datetime`) – A Python datetime

    - timestring (`strftime()`) – An `strftime` pattern

    - index_timestamp (str) – The index timestamp

  Returns:

    The date of the previous week based on ISO week number

  Return type:

    `datetime`

curator.helpers.date_ops.isdatemath(data)
  Parameters:

    data (str) – An expression to validate as being datemath or not

  Returns:

    `True` if `data` is a valid datemath expression, else `False`

  Return type:

    bool

curator.helpers.date_ops.parse_date_pattern(name)

  Scan and parse `name` for `strftime()` strings, replacing them with the associated value when found, but otherwise returning lowercase values, as uppercase snapshot names are not allowed. It will detect if the first character is a `<`, which would indicate `name` is going to be using Elasticsearch date math syntax, and skip accordingly.

  The `strftime()` identifiers that Curator currently recognizes as acceptable include:

  - `Y`: A 4 digit year

  - `y`: A 2 digit year

  - `m`: The 2 digit month

  - `W`: The 2 digit week of the year

  - `d`: The 2 digit day of the month

  - `H`: The 2 digit hour of the day, in 24 hour notation

  - `M`: The 2 digit minute of the hour

  - `S`: The 2 digit number of second of the minute

  - `j`: The 3 digit day of the year

  Parameters:

    name (str) – A name, which can contain `strftime()` strings

  Returns:

    The parsed date pattern

  Return type:

    str

curator.helpers.date_ops.parse_datemath(client, value)

  Validate that `value` looks like proper datemath. If it passes this test, then try to ship it to Elasticsearch for real. It may yet fail this test, and if it does, it will raise a `ConfigurationError` exception. If it passes, return the fully parsed string.

  Parameters:

    - client (`Elasticsearch`) – A client connection object

    - value (str) – A string to check for datemath

  Returns:

    A datemath indexname, fully rendered by Elasticsearch

  Return type:

    str

## Getters

curator.helpers.getters.byte_size(num, suffix='B')
  Parameters:

    - num (int) – The number of byte

    - suffix (str) – An arbitrary suffix, like `Bytes`

  Returns:

    A formatted string indicating the size in bytes, with the proper unit, e.g. KB, MB, GB, TB, etc.

  Return type:

    float

curator.helpers.getters.get_alias_actions(oldidx, newidx, aliases)
  Parameters:

    - oldidx (str) – The old index name

    - newidx (str) – The new index name

    - aliases (dict) – The aliases

  Returns:

    A list of actions suitable for `update_aliases()` `actions` kwarg.

  Return type:

    list

curator.helpers.getters.get_data_tiers(client)

  Get all valid data tiers from the node roles of each node in the cluster by polling each node

  Parameters:

    client (`Elasticsearch`) – A client connection object

  Returns:

    The available data tiers in `tier: bool` form.

  Return type:

    dict

curator.helpers.getters.get_indices(client, search_pattern='*', include_hidden=False)

  Calls `indices()`

  Will use the provided `search_pattern` to get a list of indices from the cluster. If `include_hidden` is `True`, it will include hidden indices in ‘expand_wildcards’.

  :param

    client: A client connection object search_pattern: The index search pattern to use include_hidden: Include hidden indices in the list

  :type

    client: `Elasticsearch` search_pattern: str include_hidden: bool

  Returns:

    The matching list of indices from the cluster

  Return type:

    list

curator.helpers.getters.get_repository(client, repository='')

  Calls `get_repository()`

  Parameters:

    - client (`Elasticsearch`) – A client connection object

    - repository (str) – The Elasticsearch snapshot repository to use

  Returns:

    Configuration information for `repository`.

  Return type:

    dict

curator.helpers.getters.get_snapshot(client, repository=None, snapshot='')

  Calls `get()`

  Parameters:

    - client (`Elasticsearch`) – A client connection object

    - repository (str) – The Elasticsearch snapshot repository to use

    - snapshot (str) – The snapshot name, or a comma-separated list of snapshots

  Returns:

    Information about the provided `snapshot`, a snapshot (or a comma-separated list of snapshots). If no snapshot specified, it will collect info for all snapshots. If none exist, an empty `dict` will be returned.

  Return type:

    dict

curator.helpers.getters.get_snapshot_data(client, repository=None)

  Get all snapshots from repository and return a list. Calls `get()`

  Parameters:

    - client (`Elasticsearch`) – A client connection object

    - repository (str) – The Elasticsearch snapshot repository to use

  Returns:

    The list of all snapshots from `repository`

  Return type:

    list

curator.helpers.getters.get_tier_preference(client, target_tier='data_frozen')

  Do the tier preference thing in reverse order from coldest to hottest Based on the value of `target_tier`, build out the list to use.

  Parameters:

    - client (`Elasticsearch`) – A client connection object

    - target_tier (str) – The target data tier, e.g. data_warm.

  Returns:

    A suitable tier preference string in csv format

  Return type:

    str

curator.helpers.getters.get_write_index(client, alias)

  Calls `get_alias()`

  Parameters:

    - client (`Elasticsearch`) – A client connection object

    - alias (str) – An alias name

  Returns:

    The the index name associated with the alias that is designated `is_write_index`

  Return type:

    str

curator.helpers.getters.index_size(client, idx, value='total')

  Calls `stats()`

  Parameters:

    - client (`Elasticsearch`) – A client connection object

    - idx (str) – An index name

    - value (str) – One of either `primaries` or `total`

  Returns:

    The sum of either `primaries` or `total` shards for index `idx`

  Return type:

    integer

curator.helpers.getters.name_to_node_id(client, name)

  Calls `info()`

  Parameters:

    - client (`Elasticsearch`) – A client connection object

    - name (str) – The node `name`

  Returns:

    The node_id of the node identified by `name`

  Return type:

    str

curator.helpers.getters.node_id_to_name(client, node_id)

  Calls `info()`

  Parameters:

    - client (`Elasticsearch`) – A client connection object

    - node_id (str) – The node `node_id`

  Returns:

    The name of the node identified by `node_id`

  Return type:

    str

curator.helpers.getters.node_roles(client, node_id)

  Calls `info()`

  Parameters:

    - client (`Elasticsearch`) – A client connection object

    - node_id (str) – The node `node_id`

  Returns:

    The list of roles assigned to the node identified by `node_id`

  Return type:

    list

curator.helpers.getters.single_data_path(client, node_id)

  In order for a shrink to work, it should be on a single filesystem, as shards cannot span filesystems. Calls `stats()`

  Parameters:

    - client (`Elasticsearch`) – A client connection object

    - node_id (str) – The node `node_id`

  Returns:

    `True` if the node has a single filesystem, else `False`

  Return type:

    bool

## Testers

curator.helpers.testers.ilm_policy_check(client, alias)

  Test if alias is associated with an ILM policy

  Calls `get_settings()`

  Parameters:

    - client (`Elasticsearch`) – A client connection object

    - alias (str) – The alias name

  Return type:

    bool

curator.helpers.testers.repository_exists(client, repository=None)

  Calls `get_repository()`

  Parameters:

    - client (`Elasticsearch`) – A client connection object

    - repository (str) – The Elasticsearch snapshot repository to use

  Returns:

    `True` if `repository` exists, else `False`

  Return type:

    bool

curator.helpers.testers.rollable_alias(client, alias)

  Calls `get_alias()`

  Parameters:

    - client (`Elasticsearch`) – A client connection object

    - alias (str) – An Elasticsearch alias

  Returns:

    `True` or `False` depending on whether `alias` is an alias that points to an index that can be used by the `_rollover` API.

  Return type:

    bool

curator.helpers.testers.snapshot_running(client)

  Calls `get_repository()`

  Return `True` if a snapshot is in progress, and `False` if not

  Parameters:

    client (`Elasticsearch`) – A client connection object

  Return type:

    bool

curator.helpers.testers.validate_actions(data)

  Validate the `actions` configuration dictionary, as imported from actions.yml, for example.

  Parameters:

    data (dict) – The configuration dictionary

  Returns:

    The validated and sanitized configuration dictionary.

  Return type:

    dict

curator.helpers.testers.validate_filters(action, myfilters)

  Validate that myfilters are appropriate for the action type, e.g. no index filters applied to a snapshot list.

  Parameters:

    - action (str) – An action name

    - myfilters (list) – A list of filters to test.

  Returns:

    Validated list of filters

  Return type:

    list

curator.helpers.testers.verify_client_object(test)
  Parameters:

    test (`Elasticsearch`) – The variable or object to test

  Returns:

    `True` if `test` is a proper `Elasticsearch` client object and raise a `TypeError` exception if it is not.

  Return type:

    bool

curator.helpers.testers.verify_repository(client, repository=None)

  Do `verify_repository()` call. If it fails, raise a `RepositoryException`.

  Parameters:

    - client (`Elasticsearch`) – A client connection object

    - repository (str) – A repository name

  Return type:

    None

curator.helpers.testers.verify_snapshot_list(test)
  Parameters:

    test (`SnapshotList`) – The variable or object to test

  Returns:

    `None` if `test` is a proper `SnapshotList` object, else raise a `TypeError` exception.

  Return type:

    None

## Utils

curator.helpers.utils.chunk_index_list(indices)

  This utility chunks very large index lists into 3KB chunks. It measures the size as a csv string, then converts back into a list for the return value.

  Parameters:

    indices (list) – The list of indices

  Returns:

    A list of lists (each a piece of the original `indices`)

  Return type:

    list

curator.helpers.utils.report_failure(exception)

  Raise a `FailedExecution` exception and include the original error message.

  Parameters:

    exception (:py:exc:Exception) – The upstream exception.

  Return type:

    None

curator.helpers.utils.show_dry_run(ilo, action, **kwargs)

  Log dry run output with the action which would have been executed.

  Parameters:

    - ilo (`IndexList`) – An IndexList Object

    - action (str) – The `action` to be performed.

    - kwargs (dict) – Any other args to show in the log output

  Return type:

    None

curator.helpers.utils.to_csv(indices)
  Parameters:

    indices (list) – A list of indices to act on, or a single value, which could be in the format of a csv string already.

  Returns:

    A csv string from a list of indices, or a single value if only one value is present

  Return type:

    str

curator.helpers.utils.multitarget_fix(pattern: str) → str

  If pattern only has ‘-’ prefixed entries (excludes) prepend a wildcard to pattern

  Parameters:

    pattern (str) – The Elasticsearch multi-target syntax pattern

  Returns:

    The pattern, possibly with a wildcard prepended

  Return type:

    str

curator.helpers.utils.regex_loop(matchstr: str, indices: list) → list

  Loop through indices, Match against matchstr, return matches

  Parameters:

    - matchstr (str) – The Python regex pattern to match against

    - indices (list) – The list of indices to match against

  Returns:

    The list of matching indices

  Return type:

    list

curator.helpers.utils.multitarget_match(pattern: str, index_list: list) → list

  Convert Elasticsearch multi-target syntax `pattern` into Python regex patterns. Match against `index_list` and return the list of matches while excluding any negative matches.

  Parameters:

    - pattern (str) – The Elasticsearch multi-target syntax pattern

    - index_list (list) – The list of indices to match against

  Returns:

    The final resulting list of indices

  Return type:

    list

## Waiters

curator.helpers.waiters.health_check(client, **kwargs)

  This function calls client.cluster. `health()` and, based on the params provided, will return `True` or `False` depending on whether that particular keyword appears in the output, and has the expected value.

  If multiple keys are provided, all must match for a `True` response.

  Parameters:

    client (`Elasticsearch`) – A client connection object

  Return type:

    bool

curator.helpers.waiters.relocate_check(client, index)

  This function calls client.cluster. `state()` with a given index to check if all of the shards for that index are in the `STARTED` state. It will return `True` if all primary and replica shards are in the `STARTED` state, and it will return `False` if any shard is in a different state.

  Parameters:

    - client (`Elasticsearch`) – A client connection object

    - index (str) – The index name

  Return type:

    bool

curator.helpers.waiters.restore_check(client, index_list)

  This function calls client.indices. `recovery()` with the list of indices to check for complete recovery. It will return `True` if recovery of those indices is complete, and `False` otherwise. It is designed to fail fast: if a single shard is encountered that is still recovering (not in `DONE` stage), it will immediately return `False`, rather than complete iterating over the rest of the response.

  Parameters:

    - client (`Elasticsearch`) – A client connection object

    - index_list (list) – The list of indices to verify having been restored.

    - kwargs – Any additional keyword arguments to pass to the function

  Return type:

    bool

curator.helpers.waiters.snapshot_check(client, snapshot=None, repository=None)

  This function calls client.snapshot. `get()` and tests to see whether the snapshot is complete, and if so, with what status. It will log errors according to the result. If the snapshot is still `IN_PROGRESS`, it will return `False`. `SUCCESS` will be an `INFO` level message, `PARTIAL` nets a `WARNING` message, `FAILED` is an `ERROR`, message, and all others will be a `WARNING` level message.

  Parameters:

    - client (`Elasticsearch`) – A client connection object

    - snapshot (str) – The snapshot name

    - repository (str) – The repository name

  Return type:

    bool

curator.helpers.waiters.task_check(client, task_id=None)

  This function calls client.tasks. `get()` with the provided `task_id`. If the task data contains `'completed': True`, then it will return `True`. If the task is not completed, it will log some information about the task and return `False`

  Parameters:

    - client (`Elasticsearch`) – A client connection object

    - task_id (str) – The task id

  Return type:

    bool

curator.helpers.waiters.wait_for_it(client, action, task_id=None, snapshot=None, repository=None, index=None, index_list=None, wait_interval=9, max_wait=-1, **kwargs)

  This function becomes one place to do all `wait_for_completion` type behaviors

  Parameters:

    - client (`Elasticsearch`) – A client connection object

    - action (str) – The action name that will identify how to wait

    - task_id (str) – If the action provided a task_id, this is where it must be declared.

    - snapshot (str) – The name of the snapshot.

    - repository (str) – The Elasticsearch snapshot repository to use

    - wait_interval (int) – Seconds to wait between completion checks.

    - max_wait (int) – Maximum number of seconds to `wait_for_completion`

    - kwargs (dict) – Any additional keyword arguments to pass to the function

  Return type:

    None