
---------------------------------------------------------------------

Curator Reference

---------------------------------------------------------------------
---------------------------------------------------------------------

Table of Contents

I. Curator and Index Lifecycle Management
    1. ILM Actions
    2. ILM or Curator?
        2.1. Beats
        2.2. Logstash
    3. ILM and Curator!
II. About
    4. Origin
    5. Features
    6. Command-Line Interface (CLI)
    7. Application Program Interface (API)
    8. License
    9. Site Corrections
    10. Contributing
III. Versions
    11. Current Release
IV. Installation
    12. pip
        12.1. Upgrading with pip
        12.2. Installing a specific version with pip
        12.3. System-wide vs. User-only installation
    13. Installation from source
    14. Docker
V. Running Curator
    15. Command Line Interface
        15.1. Running Curator from Docker
    16. Singleton Command Line Interface
        16.1. Running Curator from Docker
        16.2. Command-line filtering
        16.3. Show Indices/Snapshots
    17. Exit Codes
VI. Configuration
    18. Environment Variables
        18.1. Examples
    19. Action File
        19.1. description
    20. Configuration File
        20.1. hosts
        20.2. cloud_id
        20.3. ca_certs
        20.4. client_cert
        20.5. client_key
        20.6. verify_certs
        20.7. request_timeout
        20.8. master_only
        20.9. username
        20.10. password
        20.11. id
        20.12. api_key
        20.13. token
        20.14. loglevel
        20.15. logfile
        20.16. logformat
        20.17. blacklist
VII. Actions
    21. Alias
        21.1. Required settings
        21.2. Optional settings
    22. Allocation
        22.1. Required settings
        22.2. Optional settings
    23. Close
        23.1. Optional settings
    24. Cluster Routing
        24.1. Required settings
        24.2. Optional settings
    25. Cold2Frozen
        25.1. index_settings
        25.2. ignore_index_settings
        25.3. Optional settings
    26. Create Index
        26.1. Manual naming
        26.2. Python strftime
        26.3. Date Math
        26.4. Extra Settings
        26.5. Required settings
        26.6. Optional settings
    27. Delete Indices
        27.1. Optional settings
    28. Delete Snapshots
        28.1. Required settings
        28.2. Optional settings
    29. Forcemerge
        29.1. Required settings
        29.2. Optional settings
    30. Index Settings
        30.1. Optional settings
    31. Open
        31.1. Optional settings
    32. Reindex
        32.1. Required settings
        32.2. Optional settings
        32.3. Compatibility
    33. Replicas
        33.1. Required settings
        33.2. Optional settings
    34. Restore
        34.1. Renaming indices on restore
        34.2. Extra settings
        34.3. Required settings
        34.4. Optional settings
    35. Rollover
        35.1. Extra settings
        35.2. Required settings
        35.3. Optional settings
    36. Shrink
        36.1. Required settings
        36.2. Optional settings
    37. Snapshot
        37.1. Required settings
        37.2. Optional settings
VIII. Options
    38. allocation_type
    39. allow_ilm_indices
    40. include_hidden
    41. continue_if_exception
    42. copy_aliases
    43. count
    44. delay
    45. delete_after
    46. delete_aliases
    47. skip_flush
    48. disable_action
    49. extra_settings
        49.1. alias
        49.2. create_index
        49.3. restore
        49.4. rollover
        49.5. shrink
    50. ignore_empty_list
    51. ignore_unavailable
        51.1. restore
        51.2. snapshot
        51.3. index_settings
    52. include_aliases
    53. include_global_state
        53.1. restore
        53.2. snapshot
    54. indices
        54.1. restore
    55. key
    56. max_age
    57. max_docs
    58. max_size
    59. max_num_segments
    60. max_wait
        60.1. allocation
        60.2. cluster_routing
        60.3. reindex
        60.4. replicas
        60.5. restore
        60.6. snapshot
    61. migration_prefix
    62. migration_suffix
    63. name
        63.1. date math
        63.2. strftime
        63.3. alias
        63.4. create_index
        63.5. snapshot
    64. new_index
        64.1. date_math
    65. node_filters
        65.1. permit_masters
        65.2. exclude_nodes
    66. number_of_replicas
    67. number_of_shards
    68. partial
    69. post_allocation
    70. preserve_existing
    71. refresh
    72. remote_certificate
    73. remote_client_cert
    74. remote_client_key
    75. remote_filters
    76. remote_url_prefix
    77. rename_pattern
    78. rename_replacement
    79. repository
        79.1. restore
        79.2. snapshot
    80. requests_per_second
    81. request_body
        81.1. Manual index selection
        81.2. Filter-Selected Indices
        81.3. Reindex From Remote
        81.4. Reindex From Remote With Filter-Selected Indices
        81.5. Reindex - Migration
        81.6. Other scenarios and options
    82. retry_count
    83. retry_interval
    84. routing_type
    85. search_pattern
    86. setting
    87. shrink_node
    88. shrink_prefix
    89. shrink_suffix
    90. slices
        90.1. Picking the number of slices
    91. skip_repo_fs_check
        91.1. restore
        91.2. snapshot
    92. timeout
    93. timeout_override
    94. value
        94.1. allocation
        94.2. cluster_routing
    95. wait_for_active_shards
        95.1. Reindex
        95.2. Rollover
        95.3. Shrink
    96. wait_for_completion
        96.1. allocation
        96.2. cluster_routing
        96.3. reindex
        96.4. replicas
        96.5. restore
        96.6. snapshot
    97. wait_for_rebalance
    98. wait_interval
        98.1. allocation
        98.2. cluster_routing
        98.3. reindex
        98.4. replicas
        98.5. restore
        98.6. snapshot
    99. warn_if_no_indices
IX. Filters
    100. filtertype
    101. age
        101.1. Age calculation
        101.2. name-based ages
        101.3. creation_date-based ages
        101.4. field_stats-based ages
        101.5. Required settings
        101.6. Dependent settings
        101.7. Optional settings
    102. alias
        102.1. Required settings
        102.2. Optional settings
    103. allocated
        103.1. Required settings
        103.2. Optional settings
    104. closed
        104.1. Optional settings
    105. count
        105.1. Age-based sorting
        105.2. Pattern-based sorting
        105.3. Reversing sorting
        105.4. Required settings
        105.5. Optional settings
        105.6. Index-only settings
    106. empty
        106.1. Optional settings
    107. forcemerged
        107.1. Required settings
        107.2. Optional settings
    108. kibana
        108.1. Optional settings
    109. none
    110. opened
        110.1. Optional settings
    111. pattern
        111.1. prefix
        111.2. suffix
        111.3. timestring
        111.4. regex
        111.5. Required settings
        111.6. Optional settings
    112. period
        112.1. Relative Periods
        112.2. Absolute Periods
        112.3. Required settings
        112.4. Dependent settings
        112.5. Optional settings
    113. space
        113.1. Deleting Indices By Space
        113.2. Age-based sorting
        113.3. Reversing sorting
        113.4. Required settings
        113.5. Optional settings
    114. state
        114.1. Required settings
        114.2. Optional settings
X. Filter Elements
    115. aliases
    116. allocation_type
    117.
    118. date_from
    119. date_from_format
    120. date_to
    121. date_to_format
    122. direction
    123. disk_space
    124. epoch
        124.1. Example
    125. exclude
        125.1. Examples
    126. field
    127. intersect
    128. key
    129. kind
        129.1. prefix
        129.2. suffix
        129.3. timestring
        129.4. regex
    130. max_num_segments
    131. pattern
    132. period_type
    133. range_from
    134. range_to
    135. reverse
    136. source
        136.1. name-based ages
        136.2. creation_date-based ages
        136.3. field_stats-based ages
    137. state
    138. stats_result
    139. timestring
        139.1. strftime
    140. threshold_behavior
    141. unit
    142. unit_count
    143. unit_count_pattern
    144. use_age
    145. value
    146. week_starts_on
XI. Examples
    147. alias
    148. allocation
    149. close
    150. cluster_routing
    151. create_index
    152. delete_indices
    153. delete_snapshots
    154. forcemerge
    155. index_settings
    156. open
    157. reindex
        157.1. Manually selected reindex of a single index
        157.2. Manually selected reindex of a multiple indices
        157.3. Filter-Selected Indices
        157.4. Reindex From Remote
        157.5. Reindex From Remote With Filter-Selected Indices
        157.6. Manually selected reindex of a single index with query
    158. replicas
    159. restore
    160. rollover
    161. shrink
    162. snapshot
XII. Frequently Asked Questions
    163. Q: How can I report an error in the documentation?
        163.1. A: Use the "Edit" link on any page
    164. Q: Can I delete only certain data from within indices?
        164.1. A: It’s complicated
    165. Q: Can Curator handle index names with strange characters?
        165.1. A: Yes!

Part I. Curator and Index Lifecycle Management

Beginning with Elasticsearch version 6.6, Elasticsearch has provided
Index Lifecycle Management (or, ILM) to users with at least a Basic
license. ILM provides users with many of the most common index
management features as a matter of policy, rather than execution time
analysis (which is how Curator works).

Table of Contents

1. ILM Actions
2. ILM or Curator?
    2.1. Beats
    2.2. Logstash
3. ILM and Curator!

Chapter 1. ILM Actions

ILM applies policy actions as indices enter time-oriented phases:

  * Hot
  * Warm
  * Cold
  * Delete

The policy actions include:

  * Set Priority
  * Rollover
  * Unfollow
  * Allocate
  * Read-Only
  * Force Merge
  * Shrink
  * Delete

Chapter 2. ILM or Curator?

If ILM provides the functionality to manage your index lifecycle, and
you have at least a Basic license, consider using ILM in place of
Curator. Many of the Stack components make use of ILM by default.

2.1. Beats

Note

All Beats share a similar ILM configuration. Filebeats is used as a
reference here.

Starting with version 7.0, Filebeat uses index lifecycle management
by default when it connects to a cluster that supports lifecycle
management. Filebeat loads the default policy automatically and
applies it to any indices created by Filebeat.

You can view and edit the policy in the Index lifecycle policies UI
in Kibana. For more information about working with the UI, see Index
lifecyle policies.

Read more about Filebeat and ILM here.

2.2. Logstash

Note

The Index Lifecycle Management feature requires version 9.3.1 or
higher of the logstash-output-elasticsearch plugin.

Logstash can use [Index Lifecycle Management](docs-content://
manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md)
to automate the management of indices over time.

The use of Index Lifecycle Management is controlled by the
ilm_enabled setting. By default, this will automatically detect
whether the Elasticsearch instance supports ILM, and will use it if
it is available. ilm_enabled can also be set to true or false to
override the automatic detection, or disable ILM.

Read more about [Logstash and ILM](logstash-docs-md://lsr/
plugins-outputs-elasticsearch.html#
plugins-outputs-elasticsearch-ilm).

Chapter 3. ILM and Curator!

Warning

Curator will not act on any index associated with an ILM policy
without setting allow_ilm_indices to true.

Curator and ILM can coexist. However, to prevent Curator from
accidentally interfering, or colliding with ILM policies, any index
associated with an ILM policy name is excluded by default. This is
true whether you have a Basic license or not, or whether the ILM
policy is enabled or not.

Curator can be configured to work with ILM-enabled indices by setting
the allow_ilm_indices option to true for any action.

Learn more about Index Lifecycle Management here.

Part II. About

Elasticsearch Curator helps you curate, or manage, your Elasticsearch
indices and snapshots by:

 1. Obtaining the full list of indices (or snapshots) from the
    cluster, as the actionable list
 2. Iterate through a list of user-defined filters to progressively
    remove indices (or snapshots) from this actionable list as
    needed.
 3. Perform various actions on the items which remain in the 
    actionable list.

Learn More:

  * Origin
  * Features
  * Command-Line Interface (CLI)
  * Application Program Interface (API)
  * License
  * Site Corrections
  * Contributing

Table of Contents

4. Origin
5. Features
6. Command-Line Interface (CLI)
7. Application Program Interface (API)
8. License
9. Site Corrections
10. Contributing

Chapter 4. Origin

Curator was first called clearESindices.py. Its sole function was to
delete indices. It was almost immediately renamed to
logstash_index_cleaner.py. After a time it was briefly relocated
under the logstash repository as expire_logs, at which point it began
to gain new functionality. Soon thereafter, Jordan Sissel was hired
by Elastic (then still Elasticsearch), as was the original author of
Curator. Not long after that it became Elasticsearch Curator and is
now hosted at https://github.com/elastic/curator

Curator now performs many operations on your Elasticsearch indices,
from delete to snapshot to shard allocation routing.

Chapter 5. Features

Curator allows for many different operations to be performed to both
indices and snapshots, including:

  * Add or remove indices (or both!) from an alias
  * Change shard routing allocation
  * Close indices
  * Create index
  * Delete indices
  * Delete snapshots
  * Open closed indices
  * forceMerge indices
  * reindex indices, including from remote clusters
  * Change the number of replicas per shard for indices
  * rollover indices
  * Take a snapshot (backup) of indices
  * Restore snapshots
  * Shrink indices

Chapter 6. Command-Line Interface (CLI)

Curator has always been a command-line tool. This site provides the
documentation for how to use Curator on the command-line.

Tip

Learn more about the command-line interface.

Chapter 7. Application Program Interface (API)

Curator ships with both an API and CLI tool. The API, or Application
Program Interface, allows you to write your own scripts to accomplish
similar goals—or even new and different things—with the same code
that Curator uses.

The API documentation is not on this site, but is available at http:/
/curator.readthedocs.io/. The Curator API is built using the
Elasticsearch Python API.

Chapter 8. License

Copyright (c) 2011-2025 Elasticsearch http://elastic.co

Licensed under the Apache License, Version 2.0 (the "License"); you
may not use this file except in compliance with the License. You may
obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied. See the License for the specific language governing
permissions and limitations under the License.

Chapter 9. Site Corrections

All documentation on this site allows for quick correction
submission.

To do this, use the "Edit" link to the right on any page.

  * You will need to log in with your GitHub account.
  * Make your corrections or additions to the documentation.
  * Please use the option to "Create a new branch for this commit and
    start a pull request."
  * Please make sure you have signed our Contributor License
    Agreement. We are not asking you to assign copyright to us, but
    to give us the right to distribute your code (even documentation
    corrections) without restriction. We ask this of all contributors
    in order to assure our users of the origin and continuing
    existence of the code. You only need to sign the CLA once. If you
    are uncomfortable with this, feel free to submit a GitHub Issue
    with your suggested correction instead.
  * Changes will be reviewed and merged, if acceptable.

Chapter 10. Contributing

We welcome contributions and bug fixes to Curator’s API and CLI.

We are grateful for the many contributors who have helped Curator
become what it is today.

Please read through our contribution guide, and the Curator readme
document.

A brief overview of the steps

  * fork the repo
  * make changes in your fork
  * add tests to cover your changes (if necessary)
  * run tests
  * sign the CLA
  * send a pull request!

Tip

To submit documentation fixes for this site, see Site Corrections

Part III. Versions

Elasticsearch Curator has been around for many different versions of
Elasticsearch. Earlier releases of Curator supported multiple
versions of Elasticsearch, but this is no longer the case.

Curator is now major version locked with Elasticsearch, which means
that if Curator’s major version is 9, it should support any
Elasticsearch 9.x release.

However, in many cases, the API calls that Curator makes do not
change between the current and previous major releases, or at least
the last few minor releases of the previous major release. This has
proven to be the case, and starting with Curator 8.0.18, Curator v8
has been tested to work with Elasticsearch 7.14+.

  * Current Release

Table of Contents

11. Current Release

Chapter 11. Current Release

The current version of Curator v9 is 9.0.0.

Part IV. Installation

Curator can be installed in a variety of ways, depending on what
meets your needs. It is important to note that Curator only requires
access to a client node in the Elasticsearch cluster to work. It does
not need to be installed on one of the nodes in the cluster.

  * pip, the easiest way to use and upgrade.
  * Source Code
  * Docker

Table of Contents

12. pip
    12.1. Upgrading with pip
    12.2. Installing a specific version with pip
    12.3. System-wide vs. User-only installation
13. Installation from source
14. Docker

Chapter 12. pip

This installation procedure requires a functional Python pip
executable and requires that the target machine has internet
connectivity for downloading Curator and the dependencies from The
Python Package Index.

pip install elasticsearch-curator

12.1. Upgrading with pip

If you already have Elasticsearch Curator installed, and want to
upgrade to the latest version, use the -U flag:

pip install -U elasticsearch-curator

12.2. Installing a specific version with pip

The -U flag uninstalls the current version (if any), then installs
the latest version, or a specified one. Specify a specific version by
adding == followed by the version you’d like to install, like this:

pip install -U elasticsearch-curator==X.Y.Z

For this release, you would type: pip install -U
elasticsearch-curator==9.0.0

12.3. System-wide vs. User-only installation

The above commands each imply a system-wide installation. This
usually requires super-user access, or the sudo command. There is a
way to install Curator into a path for just the current user, using
the --user flag.

pip install --user elasticsearch-curator

This will result in the curator end-point being installed in the
current user’s home directory, in the .local directory, in the bin
subdirectory. The full path might look something like this:

/home/user/.local/bin/curator

You can make an alias or a symlink to this so you can call it more
easily.

The --user flag can also be used in conjunction with the -U flag:

pip install -U --user elasticsearch-curator==X.Y.Z

 

Chapter 13. Installation from source

Installing or Curator from source tarball (rather than doing a git
clone) is also possible.

Download and install Curator from tarball:

 1. wget https://github.com/elastic/curator/archive/v9.0.0.tar.gz -O
    elasticsearch-curator.tar.gz
 2. pip install elasticsearch-curator.tar.gz

 

Chapter 14. Docker

Curator is periodically published to Docker Hub at untergeek/curator.

Download Curator Docker image:

 1. docker pull untergeek/curator:9.0.0

Part V. Running Curator

  * Command Line Interface
  * Singleton Command Line Interface
  * Exit Codes

Table of Contents

15. Command Line Interface
    15.1. Running Curator from Docker
16. Singleton Command Line Interface
    16.1. Running Curator from Docker
    16.2. Command-line filtering
    16.3. Show Indices/Snapshots
17. Exit Codes

Chapter 15. Command Line Interface

Most common client configuration settings are now available at the
command-line.

Important

While both the configuration file and the command-line arguments can
be used together, it is important to note that command-line options
will override file-based configuration of the same setting.

The most basic command-line arguments are as follows:

curator [--config CONFIG.YML] [--dry-run] ACTION_FILE.YML

The square braces indicate optional elements.

If --config and CONFIG.YML are not provided, Curator will look in ~
/.curator/curator.yml for the configuration file. ~ is the home
directory of the user executing Curator. In a Unix system, this might
be /home/username/.curator/curator.yml, while on a Windows system, it
might be C:\Users\username\.curator\curator.yml

If --dry-run is included, Curator will simulate the action(s) in
ACTION_FILE.YML as closely as possible without actually making any
changes. The results will be in the logfile, or STDOUT/command-line
if no logfile is specified.

ACTION_FILE.YML is a YAML actionfile.

For other client configuration options, command-line help is never
far away:

curator --help

The help output looks like this:

$ curator --help
Usage: curator [OPTIONS] ACTION_FILE

  Curator for Elasticsearch indices

  The default $HOME/.curator/curator.yml configuration file (--config) can be used but is not needed.

  Command-line settings will always override YAML configuration settings.

  Some less-frequently used client configuration options are now hidden. To see the full list, run:

      curator_cli -h

Options:
  --config PATH                   Path to configuration file.
  --hosts TEXT                    Elasticsearch URL to connect to.
  --cloud_id TEXT                 Elastic Cloud instance id
  --api_token TEXT                The base64 encoded API Key token
  --id TEXT                       API Key "id" value
  --api_key TEXT                  API Key "api_key" value
  --username TEXT                 Elasticsearch username
  --password TEXT                 Elasticsearch password
  --request_timeout FLOAT         Request timeout in seconds
  --verify_certs / --no-verify_certs
                                  Verify SSL/TLS certificate(s)  [default: verify_certs]
  --ca_certs TEXT                 Path to CA certificate file or directory
  --client_cert TEXT              Path to client certificate file
  --client_key TEXT               Path to client key file
  --dry-run                       Do not perform any changes.
  --loglevel [DEBUG|INFO|WARNING|ERROR|CRITICAL]
                                  Log level
  --logfile TEXT                  Log file
  --logformat [default|ecs]       Log output format
  -v, --version                   Show the version and exit.
  -h, --help                      Show this message and exit.

  Learn more at https://www.elastic.co/guide/en/elasticsearch/client/curator/8.0/command-line.html

You can use environment variables in your configuration files.

15.1. Running Curator from Docker

Running Curator from the command-line using Docker requires only a
few additional steps.

Should you desire to use them, Docker-based Curator requires you to
map a volume for your configuration and/or log files. Attempting to
read a YAML configuration file if you have neglected to volume map
your configuration directory to /.curator will not work.

It looks like this:

docker run [-t] --rm --name myimagename \
  -v /PATH/TO/MY/CONFIGS:/.curator      \
  untergeek/curator:mytag               \
--config /.curator/config.yml /.curator/actionfile.yml

Note

While testing, adding the -t flag will allocate a pseudo-tty,
allowing you to see terminal output that would otherwise be hidden.

Both of the files config.yml and actionfile.yml should already exist
in the path /PATH/TO/MY/CONFIGS before run time.

The --rm in the command means that the container (not the image) will
be deleted after completing execution. You definitely want this as
there is no reason to keep creating containers for each run. The
eventual cleanup from this would be unpleasant.

 

Chapter 16. Singleton Command Line Interface

The curator_cli command allows users to run a single, supported
action from the command-line, without needing either the client or
action YAML configuration file, though it does support using the
client configuration file if you want. As an important bonus, the
command-line options allow you to override the settings in the
curator.yml file!

Important

While both the configuration file and the command-line arguments can
be used together, it is important to note that command-line options
will override file-based configuration of the same setting.

$ curator_cli --help
Usage: curator_cli [OPTIONS] COMMAND [ARGS]...

  Curator CLI (Singleton Tool)

  Run a single action from the command-line.

  The default $HOME/.curator/curator.yml configuration file (--config) can be used but is not needed.

  Command-line settings will always override YAML configuration settings.

Options:
  --config PATH                   Path to configuration file.
  --hosts TEXT                    Elasticsearch URL to connect to.
  --cloud_id TEXT                 Elastic Cloud instance id
  --api_token TEXT                The base64 encoded API Key token
  --id TEXT                       API Key "id" value
  --api_key TEXT                  API Key "api_key" value
  --username TEXT                 Elasticsearch username
  --password TEXT                 Elasticsearch password
  --bearer_auth TEXT              Bearer authentication token
  --opaque_id TEXT                X-Opaque-Id HTTP header value
  --request_timeout FLOAT         Request timeout in seconds
  --http_compress / --no-http_compress
                                  Enable HTTP compression  [default: no-http_compress]
  --verify_certs / --no-verify_certs
                                  Verify SSL/TLS certificate(s)  [default: verify_certs]
  --ca_certs TEXT                 Path to CA certificate file or directory
  --client_cert TEXT              Path to client certificate file
  --client_key TEXT               Path to client key file
  --ssl_assert_hostname TEXT      Hostname or IP address to verify on the node's certificate.
  --ssl_assert_fingerprint TEXT   SHA-256 fingerprint of the node's certificate. If this value is given then root-of-trust
                                  verification isn't done and only the node's certificate fingerprint is verified.
  --ssl_version TEXT              Minimum acceptable TLS/SSL version
  --master-only / --no-master-only
                                  Only run if the single host provided is the elected master  [default: no-master-only]
  --skip_version_test / --no-skip_version_test
                                  Elasticsearch version compatibility check  [default: no-skip_version_test]
  --dry-run                       Do not perform any changes.
  --loglevel [DEBUG|INFO|WARNING|ERROR|CRITICAL]
                                  Log level
  --logfile TEXT                  Log file
  --logformat [default|ecs]       Log output format
  -v, --version                   Show the version and exit.
  -h, --help                      Show this message and exit.

Commands:
  alias             Add/Remove Indices to/from Alias
  allocation        Shard Routing Allocation
  close             Close Indices
  delete-indices    Delete Indices
  delete-snapshots  Delete Snapshots
  forcemerge        forceMerge Indices (reduce segment count)
  open              Open Indices
  replicas          Change Replica Count
  restore           Restore Indices
  rollover          Rollover Index associated with Alias
  show-indices      Show Indices
  show-snapshots    Show Snapshots
  shrink            Shrink Indices to --number_of_shards
  snapshot          Snapshot Indices

  Learn more at https://www.elastic.co/guide/en/elasticsearch/client/curator/8.0/singleton-cli.html

The option flags for the given commands match those used for the same
actions. The only difference is how filtering is handled.

16.1. Running Curator from Docker

Running curator_cli from the command-line using Docker requires only
a few additional steps.

Should you desire to use them, Docker-based curator_cli requires you
to map a volume for your configuration and/or log files. Attempting
to read a YAML configuration file if you have neglected to volume map
your configuration directory to /.curator will not work.

It looks like this:

docker run [-t] --rm --name myimagename  \
  --entrypoint /curator/curator_cli      \
  -v /PATH/TO/MY/CONFIGS:/.curator       \
  untergeek/curator:mytag                \
  --config /.curator/config.yml [OPTIONS] COMMAND [ARGS]...

Note

While testing, adding the -t flag will allocate a pseudo-tty,
allowing you to see terminal output that would otherwise be hidden.

The config.yml file should already exist in the path /PATH/TO/MY/
CONFIGS before run time.

The --rm in the command means that the container (not the image) will
be deleted after completing execution. You definitely want this as
there is no reason to keep creating containers for each run. The
eventual cleanup from this would be unpleasant.

16.2. Command-line filtering

Recent improvements in Curator include schema and setting validation.
With these improvements, it is possible to validate filters and their
many permutations if passed in a way that Curator can easily digest.

--filter_list TEXT  JSON string representing an array of filters.

This means that filters need to be passed as a single object, or an
array of objects in JSON format.

Single:

--filter_list '{"filtertype":"none"}'

Multiple:

--filter_list '[{"filtertype":"age","source":"creation_date","direction":"older","unit":"days","unit_count":13},{"filtertype":"pattern","kind":"prefix","value":"logstash"}]'

This preserves the power of chained filters, making them available on
the command line.

Note

You may need to escape all of the double quotes on some platforms, or
shells like PowerShell, for instance.

Caveats to this approach:

 1. Only one action can be taken at a time.
 2. Not all actions have singleton analogs. For example, Alias and
    Restore do not have singleton actions.

16.3. Show Indices/Snapshots

One feature that the singleton command offers that the other cannot
is to show which indices and snapshots are in the system. It’s a
great way to visually test your filters without causing any harm to
the system.

$ curator_cli show-indices --help
Usage: curator_cli show-indices [OPTIONS]

  Show indices

Options:
  --verbose           Show verbose output.
  --header            Print header if --verbose
  --epoch             Print time as epoch if --verbose
  --filter_list TEXT  JSON string representing an array of filters.
                      [required]
  --help              Show this message and exit.

  Learn more at https://www.elastic.co/guide/en/elasticsearch/client/curator/8.0/singleton-cli.html#_show_indicessnapshots

$ curator_cli show-snapshots --help
Usage: curator_cli show-snapshots [OPTIONS]

  Show snapshots

Options:
  --repository TEXT   Snapshot repository name  [required]
  --filter_list TEXT  JSON string representing an array of filters.
                      [required]
  --help              Show this message and exit.

  Learn more at https://www.elastic.co/guide/en/elasticsearch/client/curator/8.0/singleton-cli.html#_show_indicessnapshots

The show-snapshots command will only show snapshots matching the
provided filters. The show-indices command will also do this, but
also offers a few extra features.

  * --verbose adds state, total size of primary and all replicas, the
    document count, the number of primary and replica shards, and the
    creation date in ISO8601 format.
  * --header adds a header that shows the column names. This only
    occurs if --verbose is also selected.
  * --epoch changes the date format from ISO8601 to epoch time. If
    --header is also selected, the column header title will change to
    creation_date

There are no extra columns or --verbose output for the show-snapshots
command.

Without --epoch

Index               State     Size     Docs Pri Rep   Creation Timestamp
logstash-2016.10.20 close     0.0B        0   5   1 2016-10-20T00:00:03Z
logstash-2016.10.21  open  763.3MB  5860016   5   1 2016-10-21T00:00:03Z
logstash-2016.10.22  open  759.1MB  5858450   5   1 2016-10-22T00:00:04Z
logstash-2016.10.23  open  757.8MB  5857456   5   1 2016-10-23T00:00:04Z
logstash-2016.10.24  open  771.5MB  5859720   5   1 2016-10-24T00:00:00Z
logstash-2016.10.25  open  771.0MB  5860112   5   1 2016-10-25T00:00:01Z
logstash-2016.10.27  open  658.3MB  4872830   5   1 2016-10-27T00:00:03Z
logstash-2016.10.28  open  655.1MB  5237250   5   1 2016-10-28T00:00:00Z

With --epoch

Index               State     Size     Docs Pri Rep creation_date
logstash-2016.10.20 close     0.0B        0   5   1    1476921603
logstash-2016.10.21  open  763.3MB  5860016   5   1    1477008003
logstash-2016.10.22  open  759.1MB  5858450   5   1    1477094404
logstash-2016.10.23  open  757.8MB  5857456   5   1    1477180804
logstash-2016.10.24  open  771.5MB  5859720   5   1    1477267200
logstash-2016.10.25  open  771.0MB  5860112   5   1    1477353601
logstash-2016.10.27  open  658.3MB  4872830   5   1    1477526403
logstash-2016.10.28  open  655.1MB  5237250   5   1    1477612800

 

Chapter 17. Exit Codes

Exit codes will indicate success or failure.

  * 0 — Success
  * 1 — Failure
  * -1 - Exception raised that does not result in a 1 exit code.

 

Part VI. Configuration

These are the higher-level configuration settings used by the
configuration files. Actions and filters are documented separately.

  * Environment Variables
  * Action File
  * Configuration File

Table of Contents

18. Environment Variables
    18.1. Examples
19. Action File
    19.1. description
20. Configuration File
    20.1. hosts
    20.2. cloud_id
    20.3. ca_certs
    20.4. client_cert
    20.5. client_key
    20.6. verify_certs
    20.7. request_timeout
    20.8. master_only
    20.9. username
    20.10. password
    20.11. id
    20.12. api_key
    20.13. token
    20.14. loglevel
    20.15. logfile
    20.16. logformat
    20.17. blacklist

Chapter 18. Environment Variables

Warning

This functionality is experimental and may be changed or removed
completely in a future release.

You can use environment variable references in both the configuration
file and the action file to set values that need to be configurable
at runtime. To do this, use:

${VAR}

Where VAR is the name of the environment variable.

Each variable reference is replaced at startup by the value of the
environment variable. The replacement is case-sensitive and occurs
while the YAML file is parsed, but before configuration schema
validation. References to undefined variables are replaced by None
unless you specify a default value. To specify a default value, use:

${VAR:default_value}

Where default_value is the value to use if the environment variable
is undefined.

Unsupported use cases

When using environment variables, the value must only be the
environment variable.

Using extra text, such as:

logfile: ${LOGPATH}/extra/path/information/file.log

is not supported at this time.

18.1. Examples

Here are some examples of configurations that use environment
variables and what each configuration looks like after replacement:

+---------------------------------------------------------------+
|Config source     |Environment setting|Config after replacement|
|------------------+-------------------+------------------------|
|unit: ${UNIT}     |export UNIT=days   |unit: days              |
|------------------+-------------------+------------------------|
|unit: ${UNIT}     |no setting         |unit:                   |
|------------------+-------------------+------------------------|
|unit: ${UNIT:days}|no setting         |unit: days              |
|------------------+-------------------+------------------------|
|unit: ${UNIT:days}|export UNIT=hours  |unit: hours             |
+---------------------------------------------------------------+

Chapter 19. Action File

Note

You can use environment variables in your configuration files.

An action file has the following structure:

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: ACTION1
    description: OPTIONAL DESCRIPTION
    options:
      option1: value1
      ...
      optionN: valueN
      continue_if_exception: False
      disable_action: True
    filters:
    - filtertype: *first*
      filter_element1: value1
      ...
      filter_elementN: valueN
    - filtertype: *second*
      filter_element1: value1
      ...
      filter_elementN: valueN
  2:
    action: ACTION2
    description: OPTIONAL DESCRIPTION
    options:
      option1: value1
      ...
      optionN: valueN
      continue_if_exception: False
      disable_action: True
    filters:
    - filtertype: *first*
      filter_element1: value1
      ...
      filter_elementN: valueN
    - filtertype: *second*
      filter_element1: value1
      ...
      filter_elementN: valueN
  3:
    action: ACTION3
    ...
  4:
    action: ACTION4
    ...

It is a YAML configuration file. The root key must be actions, after
which there can be any number of actions, nested underneath numbers.
Actions will be taken in the order they are completed.

The high-level elements of each numbered action are:

  * action
  * description
  * options
  * filters

In the case of the alias action, there are two additional high-level
elements: add and remove, which are described in the alias action
documentation.

19.1. description

This is an optional description which can help describe what the
action and its filters are supposed to do.

description: >- I can make the description span multiple
    lines by putting ">-" at the beginning of the line,
    as seen above.  Subsequent lines must also be indented.
options:
  option1: ...

Chapter 20. Configuration File

Note

The default location of the configuration file is ~/.curator/
curator.yml, but another location can be specified using the --config
flag on the command-line.

Note

You can use environment variables in your configuration files.

The configuration file contains client connection and settings for
logging. It looks like this:

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
elasticsearch:
  client:
    hosts:
      - http://127.0.0.1:9200
    cloud_id:
    ca_certs:
    client_cert:
    client_key:
    verify_certs:
    request_timeout: 30
  other_settings:
    master_only: False
    username:
    password:
    api_key:
      id:
      api_key:
      token:

logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elastic_transport', 'urllib3']

It is a YAML configuration file. The two root keys must be
elasticsearch and logging. The subkeys of each of these will be
described here.

There are other keys available for the client subkey of the
elasticsearch root key, many of which are listed here. The most
commonly used ones (listed above) are described as follows:

20.1. hosts

Important

All hosts must be in HTTP[S]://FQDN.DOMAIN.TLD:PORT form or they will
be rejected and Curator will exit with an error. The only exception
to this is HTTPS://FQDN.DOMAIN.TLD (without port), in which case :443
is implicit, and is, in fact, appended automatically.

Warning

If both cloud_id and hosts keys are populated an exception will be
thrown and Curator will exit.

A hosts definition can be a single value:

hosts: http://127.0.0.1:9200

Or multiple values in the 3 acceptable YAML ways to render sequences,
or arrays:

Warning

Curator can only work with one cluster at a time. Including clients
from multiple clusters in the hosts setting will result in errors.

Flow:

hosts: [ "http://10.0.0.1:9200", "http://10.0.0.2:9200" ]

Spanning:

hosts: [ "http://10.0.0.1:9200",
    "http://10.0.0.2:9200" ]

Block:

hosts:
  - http://10.0.0.1:9200
  - http://10.0.0.2:9200

20.2. cloud_id

The value should encapsulated in quotes because of the included
colon:

cloud_id: 'deployment_name:BIG_HASH_VALUE'

Warning

If both cloud_id and hosts keys are populated an exception will be
thrown and Curator will exit.

20.3. ca_certs

This should be a file path to your CA certificate, or left empty.

ca_certs:

This setting allows the use of a specified CA certificate file to
validate the SSL certificate used by Elasticsearch.

There is no default.

File paths

File paths can be specified as follows:

For Windows:

'C:\path\to\file'

For Linux, BSD, Mac OS:

'/path/to/file'

Using single-quotes around your file path is encouraged, especially
with Windows file paths.

20.4. client_cert

This should be a file path to a client certificate (public key), or
left empty.

client_cert:

Allows the use of a specified SSL client cert file to authenticate to
Elasticsearch. The file may contain both an SSL client certificate
and an SSL key, in which case client_key is not used. If specifying
client_cert, and the file specified does not also contain the key,
use client_key to specify the file containing the SSL key. The file
must be in PEM format, and the key part, if used, must be an
unencrypted key in PEM format as well.

File paths

File paths can be specified as follows:

For Windows:

'C:\path\to\file'

For Linux, BSD, Mac OS:

'/path/to/file'

Using single-quotes around your file path is encouraged, especially
with Windows file paths.

20.5. client_key

This should be a file path to a client key (private key), or left
empty.

client_key:

Allows the use of a specified SSL client key file to authenticate to
Elasticsearch. If using client_cert and the file specified does not
also contain the key, use client_key to specify the file containing
the SSL key. The key file must be an unencrypted key in PEM format.

File paths

File paths can be specified as follows:

For Windows:

'C:\path\to\file'

For Linux, BSD, Mac OS:

'/path/to/file'

Using single-quotes around your file path is encouraged, especially
with Windows file paths.

20.6. verify_certs

This should be True, False or left empty.

verify_certs:

If access to your Elasticsearch instance is protected by SSL
encryption, you may set verify_certs to False to disable SSL
certificate verification.

Valid use cases for doing so include the use of self-signed
certificates that cannot be otherwise verified and would generate
error messages.

Warning

Setting verify_certs to False will likely result in a warning message
that your SSL certificates are not trusted. This is expected
behavior.

The default value is True.

20.7. request_timeout

This should be an integer number of seconds, or left empty.

request_timeout:

You can change the default client connection timeout value with this
setting.

The default value is 30 (seconds) should typically not be changed to
be very large. If a longer timeout is necessary for a given action,
such as snapshot, restore, or forcemerge, the client timeout can be
overridden on per action basis by setting timeout_override in the
action options. There are default override values for some of those
longer running actions.

20.8. master_only

This should be True, False or left empty.

master_only:

In some situations, primarily with automated deployments, it makes
sense to install Curator on every node. But you wouldn’t want it to
run on each node. By setting master_only to True, this is possible.
It tests for, and will only continue running on the node that is the
elected master.

Warning

If master_only is True, and hosts has more than one value, Curator
will raise an Exception. This setting should only be used with a
single host in hosts, as its utility centers around deploying to all
nodes in the cluster.

The default value is False.

20.9. username

The HTTP Basic Authentication username

20.10. password

The HTTP Basic Authentication password

20.11. id

This should be the id portion of an API Key pair.

api_key:
  id:

This setting combined with the other subkey api_key allows API Key
authentication to an Elasticsearch instance.

The default is empty.

20.12. api_key

This should be the api_key portion of an API Key pair.

api_key:
  api_key:

This setting combined with the other subkey id allows API Key
authentication to an Elasticsearch instance.

The default is empty.

20.13. token

This should be a base64 encoded representation of an API Key pair.

api_key:
  token:

This setting will override any values provided for the id or api_key
subkeys of api_key.

The default is empty.

20.14. loglevel

This should be CRITICAL, ERROR, WARNING, INFO, DEBUG, or left empty.

loglevel:

Set the minimum acceptable log severity to display.

  * CRITICAL will only display critical messages.
  * ERROR will only display error and critical messages.
  * WARNING will display error, warning, and critical messages.
  * INFO will display informational, error, warning, and critical
    messages.
  * DEBUG will display debug messages, in addition to all of the
    above.

The default value is INFO.

20.15. logfile

This should be a path to a log file, or left empty.

logfile:

File paths

File paths can be specified as follows:

For Windows:

'C:\path\to\file'

For Linux, BSD, Mac OS:

'/path/to/file'

Using single-quotes around your file path is encouraged, especially
with Windows file paths.

The default value is empty, which will result in logging to STDOUT,
or the console.

20.16. logformat

This should default, json, logstash, ecs or left empty.

logformat:

The default format looks like:

2016-04-22 11:53:09,972 INFO      Action #1: ACTIONNAME

The json or logstash formats look like:

{"@timestamp": "2016-04-22T11:54:29.033Z", "function": "cli", "linenum": 178,
"loglevel": "INFO", "message": "Action #1: ACTIONNAME", "name": "curator.cli"}

The ecs format looks like:

{"@timestamp":"2020-02-22T11:55:00.022Z","log.level":"info","message":"Action #1:
ACTIONNAME","ecs":{"version":"1.6.0"},"log":{"logger":"curator.cli","origin":
{"file":{"line":178,"name":"cli.py"},"function":"run"},"original":"Action #1:
ACTIONNAME"},"process":{"name":"MainProcess","pid":12345,"thread":
{"id":123456789886543,"name":"MainThread"}}}

The default value is default.

20.17. blacklist

This should be an empty array [], an array of log handler strings, or
left empty.

blacklist: ['elastic_transport', 'urllib3']

The default value is ['elastic_transport', 'urllib3'], which will
result in logs for the elastic_transport and urllib3 Python modules 
not being output. These can be quite verbose, so unless you need them
to debug an issue, you should accept the default value.

Tip

If you do need to troubleshoot an issue, set blacklist to [], which
is an empty array. Leaving it unset will result in the default
behavior, which is to filter out elastic_transport and urllib3 log
traffic.

Part VII. Actions

Actions are the tasks which Curator can perform on your indices.
Snapshots, once created, can only be deleted.

  * Alias
  * Allocation
  * Close
  * Cluster Routing
  * Cold2Frozen
  * Create Index
  * Delete Indices
  * Delete Snapshots
  * forceMerge
  * Index Settings
  * Open
  * Reindex
  * Replicas
  * Restore
  * Rollover
  * Shrink
  * Snapshot

Table of Contents

21. Alias
    21.1. Required settings
    21.2. Optional settings
22. Allocation
    22.1. Required settings
    22.2. Optional settings
23. Close
    23.1. Optional settings
24. Cluster Routing
    24.1. Required settings
    24.2. Optional settings
25. Cold2Frozen
    25.1. index_settings
    25.2. ignore_index_settings
    25.3. Optional settings
26. Create Index
    26.1. Manual naming
    26.2. Python strftime
    26.3. Date Math
    26.4. Extra Settings
    26.5. Required settings
    26.6. Optional settings
27. Delete Indices
    27.1. Optional settings
28. Delete Snapshots
    28.1. Required settings
    28.2. Optional settings
29. Forcemerge
    29.1. Required settings
    29.2. Optional settings
30. Index Settings
    30.1. Optional settings
31. Open
    31.1. Optional settings
32. Reindex
    32.1. Required settings
    32.2. Optional settings
    32.3. Compatibility
33. Replicas
    33.1. Required settings
    33.2. Optional settings
34. Restore
    34.1. Renaming indices on restore
    34.2. Extra settings
    34.3. Required settings
    34.4. Optional settings
35. Rollover
    35.1. Extra settings
    35.2. Required settings
    35.3. Optional settings
36. Shrink
    36.1. Required settings
    36.2. Optional settings
37. Snapshot
    37.1. Required settings
    37.2. Optional settings

Chapter 21. Alias

action: alias
description: "Add/Remove selected indices to or from the specified alias"
options:
  name: alias_name
add:
  filters:
  - filtertype: ...
remove:
  filters:
  - filtertype: ...

Note

Empty values and commented lines will result in the default value, if
any, being selected. If a setting is set, but not used by a given
action, it will be ignored.

This action adds and/or removes indices from the alias identified by
name

The filters under the add and remove directives define which indices
will be added and/or removed. This is an atomic action, so adds and
removes happen instantaneously.

The extra_settings option allows the addition of extra settings with
the add directive. These settings are ignored for remove. An example
of how these settings can be used to create a filtered alias might
be:

action: alias
description: "Add/Remove selected indices to or from the specified alias"
options:
  name: alias_name
  extra_settings:
    filter:
      term:
        user: kimchy
add:
  filters:
  - filtertype: ...
remove:
  filters:
  - filtertype: ...

Warning

Before creating a filtered alias, first ensure that the fields
already exist in the mapping.

Learn more about adding filtering and routing to aliases in the
Elasticsearch Alias API documentation.

21.1. Required settings

  * name

21.2. Optional settings

  * warn_if_no_indices
  * extra_settings
  * ignore_empty_list
  * timeout_override
  * continue_if_exception
  * disable_action

Tip

See an example of this action in an actionfile here.

Chapter 22. Allocation

action: allocation
description: "Apply shard allocation filtering rules to the specified indices"
options:
  key: ...
  value: ...
  allocation_type: ...
filters:
- filtertype: ...

Note

Empty values and commented lines will result in the default value, if
any, being selected. If a setting is set, but not used by a given
action, it will be ignored.

This action changes the shard routing allocation for the selected
indices.

See http://www.elastic.co/guide/en/elasticsearch/reference/9.1/
shard-allocation-filtering.html for more information.

You can optionally set wait_for_completion to True to have Curator
wait for the shard routing to complete before continuing:

action: allocation
description: "Apply shard allocation filtering rules to the specified indices"
options:
  key: ...
  value: ...
  allocation_type: ...
  wait_for_completion: True
  max_wait: 300
  wait_interval: 10
filters:
- filtertype: ...

This configuration will wait for a maximum of 300 seconds for shard
routing and reallocation to complete before giving up. A max_wait
value of -1 will wait indefinitely. Curator will poll for completion
at 10 second intervals, as defined by wait_interval.

22.1. Required settings

  * key

22.2. Optional settings

  * search_pattern
  * allocation_type
  * value
  * wait_for_completion
  * max_wait
  * wait_interval
  * ignore_empty_list
  * timeout_override
  * continue_if_exception
  * disable_action

Tip

See an example of this action in an actionfile here.

Chapter 23. Close

action: close
description: "Close selected indices"
options:
  delete_aliases: false
  skip_flush: false
filters:
- filtertype: ...

Note

Empty values and commented lines will result in the default value, if
any, being selected. If a setting is set, but not used by a given
action, it will be ignored.

This action closes the selected indices, and optionally deletes
associated aliases beforehand.

23.1. Optional settings

  * search_pattern
  * delete_aliases
  * skip_flush
  * ignore_empty_list
  * timeout_override
  * continue_if_exception
  * disable_action

Tip

See an example of this action in an actionfile here.

Chapter 24. Cluster Routing

action: cluster_routing
description: "Apply routing rules to the entire cluster"
options:
  routing_type:
  value: ...
  setting: enable

Note

Empty values and commented lines will result in the default value, if
any, being selected. If a setting is set, but not used by a given
action, it will be ignored.

This action changes the shard routing allocation for the selected
indices.

See http://www.elastic.co/guide/en/elasticsearch/reference/9.1/
shards-allocation.html for more information.

You can optionally set wait_for_completion to True to have Curator
wait for the shard routing to complete before continuing:

action: cluster_routing
description: "Apply routing rules to the entire cluster"
options:
  routing_type:
  value: ...
  setting: enable
  wait_for_completion: True
  max_wait: 300
  wait_interval: 10

This configuration will wait for a maximum of 300 seconds for shard
routing and reallocation to complete before giving up. A max_wait
value of -1 will wait indefinitely. Curator will poll for completion
at 10 second intervals, as defined by wait_interval.

24.1. Required settings

  * routing_type
  * value
  * setting Currently must be set to enable. This setting is a
    placeholder for potential future expansion.

24.2. Optional settings

  * wait_for_completion
  * max_wait
  * wait_interval
  * timeout_override
  * continue_if_exception
  * disable_action

Tip

See an example of this action in an actionfile here.

Chapter 25. Cold2Frozen

Important

This action is for an unusual case where an index is a mounted,
searchable snapshot in the cold tier and is not associated with an
ILM policy. This action will not work with an index associated with
an ILM policy regardless of the value of allow_ilm_indices.

action: cold2frozen
description: "Migrate non-ILM indices from the cold tier to the frozen tier"
options:
  index_settings: {}
  ignore_index_settings: []
  wait_for_completion: True
filters:
- filtertype: ...

Note

Empty values and commented lines will result in the default value, if
any, being selected. If a setting is set, but not used by a given
action, it will be ignored.

This action migrates the selected non-ILM indices from the cold tier
to the frozen tier. You may well ask why this action is here and why
it is limited to non-ILM indices. The answer is "redacted data." If
an index must be restored from the cold tier to be live so that
sensitive data can be redacted, at present, it must be disassociated
from an ILM policy to accomplish this. If you forcemerge and
re-snapshot the redacted index, you can still put it in the cold or
frozen tier, but it will not be associated with an ILM policy any
more. This custom action is for moving that manually re-mounted cold
tier index to the frozen tier, preserving the aliases it currently
has.

25.1. index_settings

Settings that should be added to the index when it is mounted. This
should be a YAML dictionary containing anything under what would
normally appear in settings.

See http://www.elastic.co/guide/en/elasticsearch/reference/9.1/
searchable-snapshots-api-mount-snapshot.html

action: cold2frozen
description: "Migrate non-ILM indices from the cold tier to the frozen tier"
options:
  index_settings:
    routing:
      allocation:
        include:
          _tier_preference: data_frozen
  ignore_index_settings: []
  wait_for_completion: True
filters:
- filtertype: ...

Note

If unset, the default behavior is to ensure that the _tier_preference
is data_frozen, if available. If it is not, Curator will assess which
data tiers are available in your cluster and use those from coldest
to warmest, e.g. data_cold,data_warm,data_hot. If none of these are
available, it will default to data_content.

25.2. ignore_index_settings

This should be a YAML list of index settings the migrated index
should ignore after mount.

See http://www.elastic.co/guide/en/elasticsearch/reference/9.1/
searchable-snapshots-api-mount-snapshot.html

action: cold2frozen
description: "Migrate non-ILM indices from the cold tier to the frozen tier"
options:
  index_settings:
  ignore_index_settings:
    - 'index.refresh_interval'
  wait_for_completion: True
filters:
- filtertype: ...

Note

If unset, the default behavior is to ensure that the
index.refresh_interval is ignored.

25.3. Optional settings

  * search_pattern
  * wait_for_completion
  * ignore_empty_list
  * timeout_override
  * continue_if_exception
  * disable_action

Chapter 26. Create Index

action: create_index
description: "Create index as named"
options:
  name: ...

Note

Empty values and commented lines will result in the default value, if
any, being selected. If a setting is set, but not used by a given
action, it will be ignored.

This action creates the named index. There are multiple different
ways to configure how the name is represented.

26.1. Manual naming

action: create_index
description: "Create index as named"
options:
  name: myindex
  # ...

In this case, what you see is what you get. An index named myindex
will be created

26.2. Python strftime

action: create_index
description: "Create index as named"
options:
  name: 'myindex-%Y.%m'
  # ...

For the create_index action, the name option can contain Python
strftime strings. The method for doing so is described in detail,
including which strftime strings are acceptable, in the documentation
for the name option.

26.3. Date Math

action: create_index
description: "Create index as named"
options:
  name: '<logstash-{now/d+1d}>'
  # ...

For the create_index action, the name option can be in Elasticsearch
date math format. This allows index names containing dates to use
deterministic math to set a date name in the past or the future.

For example, if today’s date were 2017-03-27, the name <logstash-{now
/d}> will create an index named logstash-2017.03.27. If you wanted to
create tomorrow’s index, you would use the name <logstash-{now/
d+1d}>, which adds 1 day. This pattern creates an index named
logstash-2017.03.28. For many more configuration options, read the
Elasticsearch date math documentation.

26.4. Extra Settings

The extra_settings option allows the addition of extra settings, such
as index settings and mappings. An example of how these settings can
be used to create an index might be:

action: create_index
description: "Create index as named"
options:
  name: myindex
  # ...
  extra_settings:
    settings:
      number_of_shards: 1
      number_of_replicas: 0
    mappings:
      type1:
        properties:
          field1:
            type: string
            index: not_analyzed

26.5. Required settings

  * name

26.6. Optional settings

  * extra_settings No default value. You can add any acceptable index
    settings and mappings as nested YAML. See the Elasticsearch
    Create Index API documentation for more information.
  * timeout_override
  * continue_if_exception
  * disable_action

Tip

See an example of this action in an actionfile here.

Chapter 27. Delete Indices

action: delete_indices
description: "Delete selected indices"
options:
  continue_if_exception: False
filters:
- filtertype: ...

Note

Empty values and commented lines will result in the default value, if
any, being selected. If a setting is set, but not used by a given
action, it will be ignored.

This action deletes the selected indices.

In clusters which are overcrowded with indices, or a high number of
shards per node, deletes can take a longer time to process. In such
cases, it may be helpful to set a higher timeout than is set in the
configuration file. You can override that request_timeout as follows:

action: delete_indices
description: "Delete selected indices"
options:
  timeout_override: 300
  continue_if_exception: False
filters:
- filtertype: ...

27.1. Optional settings

  * search_pattern
  * ignore_empty_list
  * timeout_override
  * continue_if_exception
  * disable_action

Tip

See an example of this action in an actionfile here.

Chapter 28. Delete Snapshots

action: delete_snapshots
description: "Delete selected snapshots from 'repository'"
options:
  repository: ...
  retry_interval: 120
  retry_count: 3
filters:
- filtertype: ...

Note

Empty values and commented lines will result in the default value, if
any, being selected. If a setting is set, but not used by a given
action, it will be ignored.

This action deletes the selected snapshots from the selected
repository. If a snapshot is currently underway, Curator will retry
up to retry_count times, with a delay of retry_interval seconds
between retries.

28.1. Required settings

  * repository

28.2. Optional settings

  * retry_interval
  * retry_count
  * ignore_empty_list
  * timeout_override
  * continue_if_exception
  * disable_action

Tip

See an example of this action in an actionfile here.

Chapter 29. Forcemerge

action: forcemerge
description: >-
  Perform a forceMerge on selected indices to 'max_num_segments' per shard
options:
  max_num_segments: 2
  timeout_override: 21600
filters:
- filtertype: ...

Note

Empty values and commented lines will result in the default value, if
any, being selected. If a setting is set, but not used by a given
action, it will be ignored.

This action performs a forceMerge on the selected indices, merging
them to max_num_segments per shard.

Warning

A forcemerge should never be executed on an index that is actively
receiving data. It should only ever be performed on indices where no
more documents are ever anticipated to be added in the future.

You can optionally pause between each merge for delay seconds to
allow the cluster to quiesce:

action: forcemerge
description: >-
  Perform a forceMerge on selected indices to 'max_num_segments' per shard
options:
  max_num_segments: 2
  timeout_override: 21600
  delay: 120
filters:
- filtertype: ...

29.1. Required settings

  * max_num_segments

29.2. Optional settings

  * search_pattern
  * delay
  * ignore_empty_list
  * timeout_override
  * continue_if_exception
  * disable_action

Tip

See an example of this action in an actionfile here.

Chapter 30. Index Settings

action: index_settings
description: "Change settings for selected indices"
options:
  index_settings:
    index:
      refresh_interval: 5s
  ignore_unavailable: False
  preserve_existing: False
filters:
- filtertype: ...

Note

Empty values and commented lines will result in the default value, if
any, being selected. If a setting is set, but not used by a given
action, it will be ignored.

This action updates the specified index settings for the selected
indices.

Important

While Elasticsearch allows for either dotted notation of index
settings, such as

PUT /indexname/_settings
{
  "index.blocks.read_only": true
}

or in nested structure, like this:

PUT /indexname/_settings
{
  "index": {
    "blocks": {
      "read_only": true
    }
  }
}

In order to appropriately detect static vs. dynamic settings, and to
be able to verify configurational integrity in the YAML file, Curator
does not support using dotted notation.

30.1. Optional settings

  * search_pattern
  * ignore_empty_list
  * timeout_override
  * continue_if_exception
  * disable_action
  * ignore_unavailable
  * preserve_existing

Tip

See an example of this action in an actionfile here.

Chapter 31. Open

action: open
description: "open selected indices"
options:
  continue_if_exception: False
filters:
- filtertype: ...

Note

Empty values and commented lines will result in the default value, if
any, being selected. If a setting is set, but not used by a given
action, it will be ignored.

This action opens the selected indices.

31.1. Optional settings

  * search_pattern
  * ignore_empty_list
  * timeout_override
  * continue_if_exception
  * disable_action

Tip

See an example of this action in an actionfile here.

Chapter 32. Reindex

actions:
  1:
    description: "Reindex index1 into index2"
    action: reindex
    options:
      wait_interval: 9
      max_wait: -1
      request_body:
        source:
          index: index1
        dest:
          index: index2
    filters:
    - filtertype: none

There are many options for the reindex option. The best place to
start is in the request_body documentation to see how to configure
this action. All other options are as follows.

32.1. Required settings

  * request_body

32.2. Optional settings

  * refresh
  * remote_certificate
  * remote_client_cert
  * remote_client_key
  * remote_filters
  * remote_url_prefix
  * request_body
  * requests_per_second
  * slices
  * timeout
  * wait_for_active_shards
  * wait_for_completion
  * max_wait
  * wait_interval
  * ignore_empty_list
  * timeout_override
  * continue_if_exception
  * disable_action
  * migration_prefix
  * migration_suffix

Tip

See an example of this action in an actionfile here.

32.3. Compatibility

Generally speaking, the Curator should be able to perform a remote
reindex from any version of Elasticsearch, 1.4 and newer. Strictly
speaking, the Reindex API in Elasticsearch is able to reindex from
older clusters, but Curator cannot be used to facilitate this due to
Curator’s dependency on changes released in 1.4.

Chapter 33. Replicas

action: replicas
description: >- Set the number of replicas per shard for selected
    indices to 'count'
options:
  count: ...
filters:
- filtertype: ...

Note

Empty values and commented lines will result in the default value, if
any, being selected. If a setting is set, but not used by a given
action, it will be ignored.

This action will set the number of replicas per shard to the value of
count.

You can optionally set wait_for_completion to True to have Curator
wait for the replication operation to complete before continuing:

action: replicas
description: >- Set the number of replicas per shard for selected
    indices to 'count'
options:
  count: ...
  wait_for_completion: True
  max_wait: 600
  wait_interval: 10
filters:
- filtertype: ...

This configuration will wait for a maximum of 600 seconds for all
index replicas to be complete before giving up. A max_wait value of
-1 will wait indefinitely. Curator will poll for completion at 10
second intervals, as defined by wait_interval.

33.1. Required settings

  * count

33.2. Optional settings

  * search_pattern
  * wait_for_completion
  * max_wait
  * wait_interval
  * ignore_empty_list
  * timeout_override
  * continue_if_exception
  * disable_action

Tip

See an example of this action in an actionfile here.

Chapter 34. Restore

actions:
  1:
    action: restore
    description: >-
      Restore all indices in the most recent snapshot with state SUCCESS.  Wait
      for the restore to complete before continuing.  Do not skip the repository
      filesystem access check.  Use the other options to define the index/shard
      settings for the restore.
    options:
      repository:
      # If name is blank, the most recent snapshot by age will be selected
      name:
      # If indices is blank, all indices in the snapshot will be restored
      indices:
      wait_for_completion: True
      max_wait: 3600
      wait_interval: 10
    filters:
    - filtertype: state
      state: SUCCESS
      exclude:
    - filtertype: ...

Note

Empty values and commented lines will result in the default value, if
any, being selected. If a setting is set, but not used by a given
action, it will be ignored.

This action will restore indices from the indicated repository, from
the most recent snapshot identified by the applied filters, or the
snapshot identified by name.

34.1. Renaming indices on restore

You can cause indices to be renamed at restore with the
rename_pattern and rename_replacement options:

actions:
  1:
    action: restore
    description: >-
      Restore all indices in the most recent snapshot with state SUCCESS.  Wait
      for the restore to complete before continuing.  Do not skip the repository
      filesystem access check.  Use the other options to define the index/shard
      settings for the restore.
    options:
      repository:
      # If name is blank, the most recent snapshot by age will be selected
      name:
      # If indices is blank, all indices in the snapshot will be restored
      indices:
      rename_pattern: 'index(.+)'
      rename_replacement: 'restored_index$1'
      wait_for_completion: True
      max_wait: 3600
      wait_interval: 10
    filters:
    - filtertype: state
      state: SUCCESS
      exclude:
    - filtertype: ...

In this configuration, Elasticsearch will capture whatever appears
after index and put it after restored_index. For example, if I was
restoring index-2017.03.01, the resulting index would be renamed to
restored_index-2017.03.01.

34.2. Extra settings

The extra_settings option allows the addition of extra settings, such
as index settings. An example of how these settings can be used to
change settings for an index being restored might be:

actions:
  1:
    action: restore
    description: >-
      Restore all indices in the most recent snapshot with state SUCCESS.  Wait
      for the restore to complete before continuing.  Do not skip the repository
      filesystem access check.  Use the other options to define the index/shard
      settings for the restore.
    options:
      repository:
      # If name is blank, the most recent snapshot by age will be selected
      name:
      # If indices is blank, all indices in the snapshot will be restored
      indices:
      extra_settings:
        index_settings:
          number_of_replicas: 0
      wait_for_completion: True
      max_wait: 3600
      wait_interval: 10
    filters:
    - filtertype: state
      state: SUCCESS
      exclude:
    - filtertype: ...

In this case, the number of replicas will be applied to the restored
indices.

For more information see the official Elasticsearch Documentation.

34.3. Required settings

  * repository

34.4. Optional settings

  * name
  * include_aliases
  * indices
  * ignore_unavailable
  * include_global_state
  * partial
  * rename_pattern
  * rename_replacement
  * extra_settings
  * wait_for_completion
  * max_wait
  * wait_interval
  * skip_repo_fs_check
  * ignore_empty_list
  * timeout_override
  * continue_if_exception
  * disable_action

Tip

See an example of this action in an actionfile here.

Chapter 35. Rollover

action: rollover
description: >-
  Rollover the index associated with alias 'aliasname', which should be in the
  form of prefix-000001 (or similar), or prefix-YYYY.MM.DD-1.
options:
  name: aliasname
  conditions:
    max_age: 1d
    max_docs: 1000000
    max_size: 5gb

This action uses the Elasticsearch Rollover API to create a new
index, if any of the described conditions are met.

Important

When choosing conditions, any one of max_age, max_docs, max_size, or
any combination of the three may be used. If multiple are used, then
the specified condition for any one of them must be matched for the
rollover to occur.

Warning

If one or more of the max_age, max_docs, or max_size options are
present, they must each have a value. Because there are no default
values, none of these conditions can be left empty, or Curator will
generate an error.

35.1. Extra settings

The extra_settings option allows the addition of extra index settings
(but not mappings). An example of how these settings can be used
might be:

action: rollover
description: >-
  Rollover the index associated with alias 'aliasname', which should be in the
  form of prefix-000001 (or similar), or prefix-YYYY.MM.DD-1.
options:
  name: aliasname
  conditions:
    max_age: 1d
    max_docs: 1000000
  extra_settings:
    index.number_of_shards: 3
    index.number_of_replicas: 1
  timeout_override:
  continue_if_exception: False
  disable_action: False

35.2. Required settings

  * name The alias name
  * max_age The maximum age that is allowed before triggering a
    rollover. This must be nested under conditions:. There is no
    default value. If this condition is specified, it must have a
    value, or Curator will generate an error.
  * max_docs The maximum number of documents allowed in an index
    before triggering a rollover. This must be nested under
    conditions:. There is no default value. If this condition is
    specified, it must have a value, or Curator will generate an
    error.
  * max_size The maximum size the index can be before a rollover is
    triggered. This must be nested under conditions:. There is no
    default value. If this condition is specified, it must have a
    value, or Curator will generate an error.

35.3. Optional settings

  * extra_settings No default value. You can add any acceptable index
    settings (not mappings) as nested YAML. See the Elasticsearch
    Create Index API documentation for more information.
  * new_index Specify a new index name.
  * timeout_override
  * continue_if_exception
  * disable_action

Tip

See an example of this action in an actionfile here.

Chapter 36. Shrink

action: shrink
description: >-
  Shrink selected indices on the node with the most available space.
  Delete source index after successful shrink, then reroute the shrunk
  index with the provided parameters.
options:
  ignore_empty_list: True
  shrink_node: DETERMINISTIC
  node_filters:
    permit_masters: False
    exclude_nodes: ['not_this_node']
  number_of_shards: 1
  number_of_replicas: 1
  shrink_prefix:
  shrink_suffix: '-shrink'
  delete_after: True
  post_allocation:
    allocation_type: include
    key: node_tag
    value: cold
  wait_for_active_shards: 1
  extra_settings:
    settings:
      index.codec: best_compression
  wait_for_completion: True
  wait_for_rebalance: True
  wait_interval: 9
  max_wait: -1
filters:
  - filtertype: ...

Note

Empty values and commented lines will result in the default value, if
any, being selected. If a setting is set, but not used by a given
action, it will be ignored.

Shrinking an index is a good way to reduce the total shard count in
your cluster. Several conditions need to be met in order for index
shrinking to take place:

  * The index must be marked as read-only
  * A (primary or replica) copy of every shard in the index must be
    relocated to the same node
  * The cluster must have health green
  * The target index must not exist
  * The number of primary shards in the target index must be a factor
    of the number of primary shards in the source index.
  * The source index must have more primary shards than the target
    index.
  * The index must not contain more than 2,147,483,519 documents in
    total across all shards that will be shrunk into a single shard
    on the target index as this is the maximum number of docs that
    can fit into a single shard.
  * The node handling the shrink process must have sufficient free
    disk space to accommodate a second copy of the existing index.

Curator will try to meet these conditions. If it is unable to meet
them all, it will not perform a shrink operation.

This action will shrink indices to the target index, the name of
which is the value of shrink_prefix + the source index name +
shrink_suffix. The resulting index will have number_of_shards primary
shards, and number_of_replicas replica shards.

The shrinking will take place on the node identified by shrink_node,
unless DETERMINISTIC is specified, in which case Curator will
evaluate all of the nodes to determine which one has the most free
space. If multiple indices are identified for shrinking by the filter
block, and DETERMINISTIC is specified, the node selection process
will be repeated for each successive index, preventing all of the
space being consumed on a single node.

By default, Curator will delete the source index after a successful
shrink. This can be disabled by setting delete_after to False. If the
source index, is not deleted after a successful shrink, Curator will
remove the read-only setting and the shard allocation routing applied
to the source index to put it on the shrink node. Curator will wait
for the shards to stop rerouting before continuing.

The post_allocation option applies to the target index after the
shrink is complete. If set, this shard allocation routing will be
applied (after a successful shrink) and Curator will wait for all
shards to stop rerouting before continuing.

The only extra_settings which are acceptable are settings and
aliases. Please note that in the example above, while
best_compression is being applied to the new index, it will not take
effect until new writes are made to the index, such as when
force-merging the shard to a single segment.

The other options are usually okay to leave at the defaults, but feel
free to change them as needed.

36.1. Required settings

  * shrink_node

36.2. Optional settings

  * search_pattern
  * continue_if_exception
  * ignore_empty_list
  * copy_aliases
  * delete_after
  * disable_action
  * extra_settings
  * node_filters
  * number_of_shards
  * number_of_replicas
  * post_allocation
  * shrink_prefix
  * shrink_suffix
  * timeout_override
  * wait_for_active_shards
  * wait_for_completion
  * wait_for_rebalance
  * max_wait
  * wait_interval

Tip

See an example of this action in an actionfile here.

Chapter 37. Snapshot

action: snapshot
description: >-
  Snapshot selected indices to 'repository' with the snapshot name or name
  pattern in 'name'.  Use all other options as assigned
options:
  repository: ...
  # Leaving name blank will result in the default 'curator-%Y%m%d%H%M%S'
  name:
  wait_for_completion: True
  max_wait: 3600
  wait_interval: 10
filters:
- filtertype: ...

Note

Empty values and commented lines will result in the default value, if
any, being selected. If a setting is set, but not used by a given
action, it will be ignored.

This action will snapshot indices to the indicated repository, with a
name, or name pattern, as identified by name.

The other options are usually okay to leave at the defaults, but feel
free to read about them and change them accordingly.

37.1. Required settings

  * repository

37.2. Optional settings

  * search_pattern
  * name
  * ignore_unavailable
  * include_global_state
  * partial
  * wait_for_completion
  * max_wait
  * wait_interval
  * skip_repo_fs_check
  * ignore_empty_list
  * timeout_override
  * continue_if_exception
  * disable_action

Tip

See an example of this action in an actionfile here.

Part VIII. Options

Options are settings used by actions.

  * allocation_type
  * allow_ilm_indices
  * continue_if_exception
  * count
  * delay
  * delete_aliases
  * skip_flush
  * disable_action
  * extra_settings
  * ignore_empty_list
  * ignore_unavailable
  * include_aliases
  * include_global_state
  * include_hidden
  * indices
  * key
  * max_age
  * max_docs
  * max_size
  * max_num_segments
  * max_wait
  * migration_prefix
  * migration_suffix
  * name
  * new_index
  * node_filters
  * number_of_replicas
  * number_of_shards
  * partial
  * refresh
  * remote_certificate
  * remote_client_cert
  * remote_client_key
  * remote_filters
  * remote_url_prefix
  * rename_pattern
  * rename_replacement
  * repository
  * request_body
  * requests_per_second
  * retry_count
  * retry_interval
  * routing_type
  * search_pattern
  * setting
  * shrink_node
  * slices
  * skip_repo_fs_check
  * timeout
  * timeout_override
  * value
  * wait_for_active_shards
  * wait_for_completion
  * wait_for_rebalance
  * wait_interval
  * warn_if_no_indices

You can use environment variables in your configuration files.

Table of Contents

38. allocation_type
39. allow_ilm_indices
40. include_hidden
41. continue_if_exception
42. copy_aliases
43. count
44. delay
45. delete_after
46. delete_aliases
47. skip_flush
48. disable_action
49. extra_settings
    49.1. alias
    49.2. create_index
    49.3. restore
    49.4. rollover
    49.5. shrink
50. ignore_empty_list
51. ignore_unavailable
    51.1. restore
    51.2. snapshot
    51.3. index_settings
52. include_aliases
53. include_global_state
    53.1. restore
    53.2. snapshot
54. indices
    54.1. restore
55. key
56. max_age
57. max_docs
58. max_size
59. max_num_segments
60. max_wait
    60.1. allocation
    60.2. cluster_routing
    60.3. reindex
    60.4. replicas
    60.5. restore
    60.6. snapshot
61. migration_prefix
62. migration_suffix
63. name
    63.1. date math
    63.2. strftime
    63.3. alias
    63.4. create_index
    63.5. snapshot
64. new_index
    64.1. date_math
65. node_filters
    65.1. permit_masters
    65.2. exclude_nodes
66. number_of_replicas
67. number_of_shards
68. partial
69. post_allocation
70. preserve_existing
71. refresh
72. remote_certificate
73. remote_client_cert
74. remote_client_key
75. remote_filters
76. remote_url_prefix
77. rename_pattern
78. rename_replacement
79. repository
    79.1. restore
    79.2. snapshot
80. requests_per_second
81. request_body
    81.1. Manual index selection
    81.2. Filter-Selected Indices
    81.3. Reindex From Remote
    81.4. Reindex From Remote With Filter-Selected Indices
    81.5. Reindex - Migration
    81.6. Other scenarios and options
82. retry_count
83. retry_interval
84. routing_type
85. search_pattern
86. setting
87. shrink_node
88. shrink_prefix
89. shrink_suffix
90. slices
    90.1. Picking the number of slices
91. skip_repo_fs_check
    91.1. restore
    91.2. snapshot
92. timeout
93. timeout_override
94. value
    94.1. allocation
    94.2. cluster_routing
95. wait_for_active_shards
    95.1. Reindex
    95.2. Rollover
    95.3. Shrink
96. wait_for_completion
    96.1. allocation
    96.2. cluster_routing
    96.3. reindex
    96.4. replicas
    96.5. restore
    96.6. snapshot
97. wait_for_rebalance
98. wait_interval
    98.1. allocation
    98.2. cluster_routing
    98.3. reindex
    98.4. replicas
    98.5. restore
    98.6. snapshot
99. warn_if_no_indices

Chapter 38. allocation_type

Note

This setting is used only when using the allocation action

action: allocation
description: "Apply shard allocation filtering rules to the specified indices"
options:
  key: ...
  value: ...
  allocation_type: ...
filters:
- filtertype: ...

The value of this setting must be one of require, include, or
exclude.

Read more about these settings at http://www.elastic.co/guide/en/
elasticsearch/reference/9.1/shard-allocation-filtering.html

The default value for this setting is require.

Chapter 39. allow_ilm_indices

This option allows Curator to manage ILM-enabled indices. Exercise
caution that ILM policies and Curator actions will not interfere with
each other.

Important

Read more about Curator and Index Lifecycle Management here.

action: delete_indices
description: "Delete the specified indices"
options:
  allow_ilm_indices: true
filters:
- filtertype: ...

The value of this setting must be either true or false.

The default value for this setting is false.

Chapter 40. include_hidden

This option allows Curator to act on indices with the setting hidden:
true, which is common with data_streams.

Important

If data_stream backing indices are matched by the search_pattern and/
or after the filters, any attempt to delete the active backing index
will result in an error code. The only way to delete all of a
data_stream is via the data_stream API.

action: delete_indices
description: "Delete the specified indices"
options:
  include_hidden: true
filters:
- filtertype: ...

The value of this setting must be either true or false.

The default value for this setting is false.

Chapter 41. continue_if_exception

Using ignore_empty_list rather than continue_if_exception

Curator has two general classifications of exceptions: Empty list
exceptions, and everything else. The empty list conditions are
curator.exception.NoIndices and curator.exception.NoSnapshots. The
continue_if_exception option only catches conditions other than empty
list conditions. In most cases, you will want to use
ignore_empty_list instead of continue_if_exception.

So why are there two kinds of exceptions? When Curator 4 was
released, the ability to continue in the event of any exception was
covered by the continue_if_exception option. However, an empty list
is a benign condition. In fact, it’s expected with brand new
clusters, or when new index patterns are added. The decision was made
to split the exceptions, and have a new option catch the empty lists.

See ignore_empty_list for more information.

Note

This setting is available in all actions.

action: delete_indices
description: "Delete selected indices"
options:
  continue_if_exception: False
filters:
- filtertype: ...

If continue_if_exception is set to True, Curator will attempt to
continue on to the next action, if any, even if an exception is
encountered. Curator will log but ignore the exception that was
raised.

The default value for this setting is False

Chapter 42. copy_aliases

Note

This setting is only used by the shrink action.

action: shrink
description: >-
  Shrink selected indices on the node with the most available space.
  Reimplement source index aliases on target index after successful shrink.
options:
  shrink_node: DETERMINISTIC
  copy_aliases: True
filters:
  - filtertype: ...

The default value of this setting is False. If True, after an index
has been successfully shrunk, any aliases associated with the source
index will be copied to the target index.

Chapter 43. count

Note

This setting is required when using the replicas action.

action: replicas
description: >- Set the number of replicas per shard for selected
    indices to 'count'
options:
  count: ...
filters:
- filtertype: ...

The value for this setting is the number of replicas to assign to
matching indices.

There is no default value. This setting must be set by the user or an
exception will be raised, and execution will halt.

Chapter 44. delay

Note

This setting is only used by the forceMerge action, and is optional.

action: forcemerge
description: >-
  Perform a forceMerge on selected indices to 'max_num_segments' per shard
options:
  max_num_segments: 2
  timeout_override: 21600
  delay: 120
filters:
- filtertype: ...

The value for this setting is the number of seconds to delay between
forceMerging indices, to allow the cluster to quiesce.

There is no default value.

Chapter 45. delete_after

Note

This setting is only used by the shrink action.

action: shrink
description: >-
  Shrink selected indices on the node with the most available space.
  Delete source index after successful shrink.
options:
  shrink_node: DETERMINISTIC
  delete_after: True
filters:
  - filtertype: ...

The default value of this setting is True. After an index has been
successfully shrunk, the source index will be deleted or preserved
based on the value of this setting.

Chapter 46. delete_aliases

Note

This setting is only used by the close action, and is optional.

action: close
description: "Close selected indices"
options:
  delete_aliases: False
filters:
- filtertype: ...

The value for this setting determines whether aliases will be deleted
from indices before closing.

The default value is False.

Chapter 47. skip_flush

Note

This setting is only used by the close action, and is optional.

action: close
description: "Close selected indices"
options:
  skip_flush: False
filters:
- filtertype: ...

If skip_flush is set to True, Curator will not flush indices to disk
before closing. This may be useful for closing red indices before
restoring.

The default value is False.

Chapter 48. disable_action

Note

This setting is available in all actions.

action: delete_indices
description: "Delete selected indices"
options:
  disable_action: False
filters:
- filtertype: ...

If disable_action is set to True, Curator will ignore the current
action. This may be useful for temporarily disabling actions in a
large configuration file.

The default value for this setting is False

Chapter 49. extra_settings

This setting should be nested YAML. The values beneath extra_settings
will be used by whichever action uses the option.

49.1. alias

action: alias
description: "Add/Remove selected indices to or from the specified alias"
options:
  name: alias_name
  extra_settings:
    filter:
      term:
        user: kimchy
add:
  filters:
  - filtertype: ...
remove:
  filters:
  - filtertype: ...

49.2. create_index

action: create_index
description: "Create index as named"
options:
  name: myindex
  # ...
  extra_settings:
    settings:
      number_of_shards: 1
      number_of_replicas: 0
    mappings:
      type1:
        properties:
          field1:
            type: string
            index: not_analyzed

49.3. restore

See the official Elasticsearch Documentation.

actions:
  1:
    action: restore
    description: >-
      Restore all indices in the most recent snapshot with state SUCCESS.  Wait
      for the restore to complete before continuing.  Do not skip the repository
      filesystem access check.  Use the other options to define the index/shard
      settings for the restore.
    options:
      repository:
      # If name is blank, the most recent snapshot by age will be selected
      name:
      # If indices is blank, all indices in the snapshot will be restored
      indices:
      extra_settings:
        index_settings:
          number_of_replicas: 0
      wait_for_completion: True
      max_wait: 3600
      wait_interval: 10
    filters:
    - filtertype: state
      state: SUCCESS
      exclude:
    - filtertype: ...

49.4. rollover

action: rollover
description: >-
  Rollover the index associated with alias 'name', which should be in the
  form of prefix-000001 (or similar), or prefix-yyyy.MM.DD-1.
options:
  name: aliasname
  conditions:
    max_age: 1d
    max_docs: 1000000
  extra_settings:
    index.number_of_shards: 3
    index.number_of_replicas: 1
  timeout_override:
  continue_if_exception: False
  disable_action: False

49.5. shrink

Note

Only settings and aliases are acceptable when used in shrink.

action: shrink
description: >-
  Shrink selected indices on the node with the most available space.
  Delete source index after successful shrink, then reroute the shrunk
  index with the provided parameters.
options:
  shrink_node: DETERMINISTIC
  extra_settings:
    settings:
      index.codec: best_compression
    aliases:
      my_alias: {}
filters:
  - filtertype: ...

There is no default value.

Chapter 50. ignore_empty_list

This setting must be either True or False.

action: delete_indices
description: "Delete selected indices"
options:
  ignore_empty_list: True
filters:
- filtertype: ...

Depending on your indices, and how you’ve filtered them, an empty
list may be presented to the action. This results in an error
condition.

When the ignore_empty_list option is set to True, the action will
exit with an INFO level log message indicating such. If
ignore_empty_list is set to False, an ERROR level message will be
logged, and Curator will exit with code 1.

The default value of this setting is False

Chapter 51. ignore_unavailable

Note

This setting is used by the snapshot, restore, and index_settings
actions.

This setting must be either True or False.

The default value of this setting is False

51.1. restore

actions:
  1:
    action: restore
    description: Restore my_index from my_snapshot in my_repository
    options:
      repository: my_repository
      name: my_snapshot
      indices: my_index
      ignore_unavailable: True
      wait_for_completion: True
      max_wait: 3600
      wait_interval: 10
    filters:
    - filtertype: state
      state: SUCCESS
      exclude:
    - filtertype: ...

When the ignore_unavailable option is False and an index is missing
the restore request will fail.

51.2. snapshot

action: snapshot
description: >-
  Snapshot selected indices to 'repository' with the snapshot name or name
  pattern in 'name'.  Use all other options as assigned
options:
  repository: my_repository
  name: my_snapshot
  ignore_unavailable: False
  wait_for_completion: True
  max_wait: 3600
  wait_interval: 10
filters:
- filtertype: ...

When the ignore_unavailable option is False and an index is missing,
the snapshot request will fail. This is not frequently a concern in
Curator, as it should only ever find indices that exist.

51.3. index_settings

action: index_settings
description: "Change settings for selected indices"
options:
  index_settings:
    index:
      refresh_interval: 5s
  ignore_unavailable: False
  preserve_existing: False
filters:
- filtertype: ...

When the ignore_unavailable option is False and an index is missing,
or if the request is to apply a static setting and the index is
opened, the index setting request will fail. The ignore_unavailable
option allows these indices to be skipped, when set to True.

Note

Dynamic index settings can be applied to either open or closed
indices.

Chapter 52. include_aliases

Note

This setting is only used by the restore action.

actions:
  1:
    action: restore
    description: Restore my_index from my_snapshot in my_repository
    options:
      repository: my_repository
      name: my_snapshot
      indices: my_index
      include_aliases: True
      wait_for_completion: True
      max_wait: 3600
      wait_interval: 10
    filters:
    - filtertype: state
      state: SUCCESS
      exclude:
    - filtertype: ...

This setting must be either True or False.

The value of this setting determines whether Elasticsearch should
include index aliases when restoring the snapshot.

The default value of this setting is False

Chapter 53. include_global_state

Note

This setting is used by the snapshot and restore actions.

This setting must be either True or False.

The value of this setting determines whether Elasticsearch should
include the global cluster state with the snapshot or restore.

When performing a snapshot, the default value of this setting is
True.

When performing a restore, the default value of this setting is
False.

53.1. restore

actions:
  1:
    action: restore
    description: Restore my_index from my_snapshot in my_repository
    options:
      repository: my_repository
      name: my_snapshot
      indices: my_index
      include_global_state: False
      wait_for_completion: True
      max_wait: 3600
      wait_interval: 10
    filters:
    - filtertype: state
      state: SUCCESS
      exclude:
    - filtertype: ...

53.2. snapshot

action: snapshot
description: >-
  Snapshot selected indices to 'repository' with the snapshot name or name
  pattern in 'name'.  Use all other options as assigned
options:
  repository: my_repository
  name: my_snapshot
  include_global_state: True
  wait_for_completion: True
  max_wait: 3600
  wait_interval: 10
filters:
- filtertype: ...

Chapter 54. indices

Note

This setting is only used by the restore action.

54.1. restore

actions:
  1:
    action: restore
    description: Restore my_index from my_snapshot in my_repository
    options:
      repository: my_repository
      name: my_snapshot
      indices: my_index
      wait_for_completion: True
      max_wait: 3600
      wait_interval: 10
    filters:
    - filtertype: state
      state: SUCCESS
      exclude:
    - filtertype: ...

This setting must be a list of indices to restore. Any valid YAML
format for lists are acceptable here. If indices is left empty, or
unset, all indices in the snapshot will be restored.

The default value of this setting is an empty setting.

Chapter 55. key

Note

This setting is required when using the allocation action.

action: allocation
description: "Apply shard allocation filtering rules to the specified indices"
options:
  key: ...
  value: ...
  allocation_type: ...
filters:
- filtertype: ...

The value of this setting should correspond to a node setting on one
or more nodes in your cluster.

For example, you might have set

node.tag: myvalue

in your elasticsearch.yml file for one or more of your nodes. To
match allocation in this case, set key to tag.

These special attributes are also supported:

+-------------------------------------------------------------------+
|attribute  |description                                            |
|-----------+-------------------------------------------------------|
|_name      |Match nodes by node name                               |
|-----------+-------------------------------------------------------|
|_host_ip   |Match nodes by host IP address (IP associated with     |
|           |hostname)                                              |
|-----------+-------------------------------------------------------|
|_publish_ip|Match nodes by publish IP address                      |
|-----------+-------------------------------------------------------|
|_ip        |Match either _host_ip or _publish_ip                   |
|-----------+-------------------------------------------------------|
|_host      |Match nodes by hostname                                |
+-------------------------------------------------------------------+

There is no default value. This setting must be set by the user or an
exception will be raised, and execution will halt.

Chapter 56. max_age

action: rollover
description: >-
  Rollover the index associated with alias 'aliasname', which should be in the
  form of prefix-000001 (or similar), or prefix-yyyy.MM.DD-1.
options:
  name: aliasname
  conditions:
    max_age: 1d

Note

At least one of max_age, max_docs, max_size or any combinations of
the three are required as conditions: for the Rollover action.

The maximum age that is allowed before triggering a rollover. It must
be nested under conditions: There is no default value. If this
condition is specified, it must have a value, or Curator will
generate an error.

Ages such as 1d for one day, or 30s for 30 seconds can be used.

Chapter 57. max_docs

action: rollover
description: >-
  Rollover the index associated with alias 'aliasname', which should be in the
  form of prefix-000001 (or similar), or prefix-yyyy.MM.DD-1.
options:
  name: aliasname
  conditions:
    max_docs: 1000000

Note

At least one of max_age, max_docs, max_size or any combinations of
the three are required as conditions: for the Rollover action.

The maximum number of documents allowed in an index before triggering
a rollover. It must be nested under conditions: There is no default
value. If this condition is specified, it must have a value, or
Curator will generate an error.

Chapter 58. max_size

action: rollover
description: >-
  Rollover the index associated with alias 'aliasname', which should be in the
  form of prefix-000001 (or similar), or prefix-yyyy.MM.DD-1.
options:
  name: aliasname
  conditions:
    max_size: 5gb

Note

At least one of max_age, max_docs, max_size or any combinations of
the three are required as conditions: for the Rollover action.

The maximum approximate size an index is allowed to be before
triggering a rollover. Sizes must use Elasticsearch approved
human-readable byte units. It must be nested under conditions: There
is no default value. If this condition is specified, it must have a
value, or Curator will generate an error.

Chapter 59. max_num_segments

Note

This setting is required when using the forceMerge action.

action: forcemerge
description: >-
  Perform a forceMerge on selected indices to 'max_num_segments' per shard
options:
  max_num_segments: 2
  timeout_override: 21600
filters:
- filtertype: ...

The value for this setting is the cutoff number of segments per
shard. Indices which have more than this number of segments per shard
will remain in the index list.

There is no default value. This setting must be set by the user or an
exception will be raised, and execution will halt.

Chapter 60. max_wait

Note

This setting is used by the allocation, cluster_routing, reindex,
replicas, restore, and snapshot actions.

This setting must be a positive integer, or -1.

This setting specifies how long in seconds to wait to see if the
action has completed before giving up. This option is used in
conjunction with wait_interval, which is the number of seconds to
wait between checking to see if the given action is complete.

The default value for this setting is -1, meaning that Curator will
wait indefinitely for the action to complete.

60.1. allocation

action: allocation
description: "Apply shard allocation filtering rules to the specified indices"
options:
  key: ...
  value: ...
  allocation_type: ...
  wait_for_completion: True
  max_wait: 300
  wait_interval: 10
filters:
- filtertype: ...

60.2. cluster_routing

action: cluster_routing
description: "Apply routing rules to the entire cluster"
options:
  routing_type:
  value: ...
  setting: enable
  wait_for_completion: True
  max_wait: 300
  wait_interval: 10

60.3. reindex

actions:
  1:
    description: "Reindex index1 into index2"
    action: reindex
    options:
      wait_interval: 9
      max_wait: -1
      request_body:
        source:
          index: index1
        dest:
          index: index2
    filters:
    - filtertype: none

60.4. replicas

action: replicas
description: >- Set the number of replicas per shard for selected
    indices to 'count'
options:
  count: ...
  wait_for_completion: True
  max_wait: 600
  wait_interval: 10
filters:
- filtertype: ...

60.5. restore

actions:
  1:
    action: restore
    description: Restore my_index from my_snapshot in my_repository
    options:
      repository: my_repository
      name: my_snapshot
      indices: my_index
      include_global_state: False
      wait_for_completion: True
      max_wait: 3600
      wait_interval: 10
    filters:
    - filtertype: state
      state: SUCCESS
      exclude:
    - filtertype: ...

60.6. snapshot

action: snapshot
description: >-
  Snapshot selected indices to 'repository' with the snapshot name or name
  pattern in 'name'.  Use all other options as assigned
options:
  repository: my_repository
  name: my_snapshot
  include_global_state: True
  wait_for_completion: True
  max_wait: 3600
  wait_interval: 10
filters:
- filtertype: ...

Chapter 61. migration_prefix

Note

This setting is used by the reindex action.

If the destination index is set to MIGRATION, Curator will reindex
all selected indices one by one until they have all been reindexed.
By configuring migration_prefix, a value can prepend each of those
index names. For example, if I were reindexing index1, index2, and
index3, and migration_prefix were set to new-, then the reindexed
indices would be named new-index1, new-index2, and new-index3:

actions:
  1:
    description: >-
      Reindex index1, index2, and index3 with a prefix of new-, resulting in
      indices named new-index1, new-index2, and new-index3
    action: reindex
    options:
      wait_interval: 9
      max_wait: -1
      migration_prefix: new-
      request_body:
        source:
          index: ["index1", "index2", "index3"]
        dest:
          index: MIGRATION
    filters:
    - filtertype: none

migration_prefix can be used in conjunction with Chapter 62, 
migration_suffix

Chapter 62. migration_suffix

Note

This setting is used by the reindex action.

If the destination index is set to MIGRATION, Curator will reindex
all selected indices one by one until they have all been reindexed.
By configuring migration_suffix, a value can be appended to each of
those index names. For example, if I were reindexing index1, index2,
and index3, and migration_suffix were set to -new, then the reindexed
indices would be named index1-new, index2-new, and index3-new:

actions:
  1:
    description: >-
      Reindex index1, index2, and index3 with a suffix of -new, resulting in
      indices named index1-new, index2-new, and index3-new
    action: reindex
    options:
      wait_interval: 9
      max_wait: -1
      migration_suffix: -new
      request_body:
        source:
          index: ["index1", "index2", "index3"]
        dest:
          index: MIGRATION
    filters:
    - filtertype: none

migration_suffix can be used in conjunction with Chapter 61, 
migration_prefix

Chapter 63. name

Note

This setting is used by the alias, create_index and snapshot,
actions.

The value of this setting is the name of the alias, snapshot, or
index, depending on which action makes use of name.

63.1. date math

This setting may be a valid Elasticsearch date math string.

A date math name takes the following form:

<static_name{date_math_expr{date_format|time_zone}}>

+--------------------------------+
|static_name   |is the static    |
|              |text part of the |
|              |name             |
|--------------+-----------------|
|date_math_expr|is a dynamic date|
|              |math expression  |
|              |that computes the|
|              |date dynamically |
|--------------+-----------------|
|date_format   |is the optional  |
|              |format in which  |
|              |the computed date|
|              |should be        |
|              |rendered.        |
|              |Defaults to      |
|              |yyyy.MM.dd.      |
|--------------+-----------------|
|time_zone     |is the optional  |
|              |time zone .      |
|              |Defaults to utc. |
+--------------------------------+

The following example shows different forms of date math names and
the final form they resolve to given the current time is 22rd March
2024 noon utc.

+--------------------------------+
|Expression  |Resolves to        |
|------------+-------------------|
|<logstash-  |logstash-2024.03.22|
|{now/d}>    |                   |
|------------+-------------------|
|<logstash-  |logstash-2024.03.01|
|{now/M}>    |                   |
|------------+-------------------|
|<logstash-  |logstash-2024.03   |
|{now/M      |                   |
|{yyyy.MM}}> |                   |
|------------+-------------------|
|<logstash-  |logstash-2024.02   |
|{now/M-1M   |                   |
|{yyyy.MM}}> |                   |
|------------+-------------------|
|<logstash-  |logstash-2024.03.23|
|{now/d      |                   |
|{yyyy.MM.dd||                   |
|+12:00}}>   |                   |
+--------------------------------+

63.2. strftime

This setting may alternately contain a valid Python strftime string.
Curator will extract the strftime identifiers and replace them with
the corresponding values.

The identifiers that Curator currently recognizes include:

+--------------------------------+
|Unit|Value                      |
|----+---------------------------|
|%Y  |4 digit year               |
|----+---------------------------|
|%y  |2 digit year               |
|----+---------------------------|
|%m  |2 digit month              |
|----+---------------------------|
|%W  |2 digit week of the year   |
|----+---------------------------|
|%d  |2 digit day of the month   |
|----+---------------------------|
|%H  |2 digit hour of the day, in|
|    |24 hour notation           |
|----+---------------------------|
|%M  |2 digit minute of the hour |
|----+---------------------------|
|%S  |2 digit second of the      |
|    |minute                     |
|----+---------------------------|
|%j  |3 digit day of the year    |
+--------------------------------+

63.3. alias

action: alias
description: "Add/Remove selected indices to or from the specified alias"
options:
  name: alias_name
add:
  filters:
  - filtertype: ...
remove:
  filters:
  - filtertype: ...

This option is required by the alias action, and has no default
setting in that context.

63.4. create_index

For the create_index action, there is no default setting, but you can
use strftime:

action: create_index
description: "Create index as named"
options:
  name: 'myindex-%Y.%m'
  # ...

or use Elasticsearch date math

action: create_index
description: "Create index as named"
options:
  name: '<logstash-{now/d+1d}>'
  # ...

to name your indices. See more in the create_index documenation.

63.5. snapshot

action: snapshot
description: >-
  Snapshot selected indices to 'repository' with the snapshot name or name
  pattern in 'name'.  Use all other options as assigned
options:
  repository: my_repository
  name:
  include_global_state: True
  wait_for_completion: True
  max_wait: 3600
  wait_interval: 10
filters:
- filtertype: ...

For the snapshot action, the default value of this setting is
curator-%Y%m%d%H%M%S

Chapter 64. new_index

Note

This optional setting is only used by the rollover action.

description: >-
  Rollover the index associated with alias 'name'.  Specify new index name using
  date math.
options:
  name: aliasname
  new_index: "<prefix-{now/d}-1>"
  conditions:
    max_age: 1d
  wait_for_active_shards: 1

Important

A new index name for rollover should still end with a dash followed
by an incrementable number, e.g. my_new_index-1, or if using date
math, <prefix-{now/d}-1> or <prefix-{now/d}-000001>

64.1. date_math

This setting may be a valid Elasticsearch date math string.

A date math name takes the following form:

<static_name{date_math_expr{date_format|time_zone}}>

+--------------------------------+
|static_name   |is the static    |
|              |text part of the |
|              |name             |
|--------------+-----------------|
|date_math_expr|is a dynamic date|
|              |math expression  |
|              |that computes the|
|              |date dynamically |
|--------------+-----------------|
|date_format   |is the optional  |
|              |format in which  |
|              |the computed date|
|              |should be        |
|              |rendered.        |
|              |Defaults to      |
|              |yyyy.MM.dd.      |
|--------------+-----------------|
|time_zone     |is the optional  |
|              |time zone .      |
|              |Defaults to utc. |
+--------------------------------+

The following example shows different forms of date math names and
the final form they resolve to given the current time is 22rd March
2024 noon utc.

+--------------------------------+
|Expression  |Resolves to        |
|------------+-------------------|
|<logstash-  |logstash-2024.03.22|
|{now/d}>    |                   |
|------------+-------------------|
|<logstash-  |logstash-2024.03.01|
|{now/M}>    |                   |
|------------+-------------------|
|<logstash-  |logstash-2024.03   |
|{now/M      |                   |
|{yyyy.MM}}> |                   |
|------------+-------------------|
|<logstash-  |logstash-2024.02   |
|{now/M-1M   |                   |
|{yyyy.MM}}> |                   |
|------------+-------------------|
|<logstash-  |logstash-2024.03.23|
|{now/d      |                   |
|{yyyy.MM.dd||                   |
|+12:00}}>   |                   |
+--------------------------------+

There is no default value for new_index.

Chapter 65. node_filters

Note

This setting is only used by the shrink action.

action: shrink
description: >-
  Shrink selected indices on the node with the most available space.
  Allow master/data nodes to be potential shrink targets, but exclude
  'named_node' from potential selection.
options:
  shrink_node: DETERMINISTIC
  node_filters:
    permit_masters: True
    exclude_nodes: ['named_node']
filters:
  - filtertype: ...

There is no default value for node_filters.

The current sub-options are as follows:

65.1. permit_masters

This option indicates whether the shrink action can select master
eligible nodes when using DETERMINISTIC as the value for shrink_node.
The default value is False. Please note that this will exclude the
elected master, as well as other master-eligible nodes.

Important

If you have a small cluster with only master/data nodes, you must set
permit_masters to True in order to select one of those nodes as a
potential shrink_node.

65.2. exclude_nodes

This option provides means to exclude nodes from selection when using
DETERMINISTIC as the value for shrink_node. It should be noted that
you can use a named node for shrink_node and then exclude it here,
and it will prevent a shrink from occurring.

Chapter 66. number_of_replicas

Note

This setting is only used by the shrink action.

action: shrink
description: >-
  Shrink selected indices on the node with the most available space.
  Set the number of replicas to 0.
options:
  shrink_node: DETERMINISTIC
  number_of_replicas: 0
filters:
  - filtertype: ...

The value of this setting determines the number of replica shards per
primary shard in the target index. The default value is 1.

Chapter 67. number_of_shards

Note

This setting is only used by the shrink action.

action: shrink
description: >-
  Shrink selected indices on the node with the most available space.
  Set the number of shards to 2.
options:
  shrink_node: DETERMINISTIC
  number_of_shards: 2
filters:
  - filtertype: ...

The value of this setting determines the number of primary shards in
the target index. The default value is 1.

Important

The value for number_of_shards must meet the following criteria:

  * It must be lower than the number of primary shards in the source
    index.
  * It must be a factor of the number of primary shards in the source
    index.

Chapter 68. partial

Note

This setting is only used by the snapshot action.

action: snapshot
description: >-
  Snapshot selected indices to 'repository' with the snapshot name or name
  pattern in 'name'.  Use all other options as assigned
options:
  repository: my_repository
  name: ...
  partial: False
  wait_for_completion: True
  max_wait: 3600
  wait_interval: 10
filters:
- filtertype: ...

This setting must be either True or False.

The entire snapshot will fail if one or more indices being added to
the snapshot do not have all primary shards available. This behavior
can be changed by setting partial to True.

The default value of this setting is False

Chapter 69. post_allocation

Note

This setting is only used by the shrink action.

action: shrink
description: >-
  Shrink selected indices on the node with the most available space.
  Apply shard routing allocation to target indices.
options:
  shrink_node: DETERMINISTIC
  post_allocation:
    allocation_type: include
    key: node_tag
    value: cold
filters:
  - filtertype: ...

The only permitted subkeys for post_allocation are the same options
used by the allocation action: allocation_type, key, and value.

If present, these values will be use to apply shard routing
allocation to the target index after shrinking.

There is no default value for post_allocation.

Chapter 70. preserve_existing

action: index_settings
description: "Change settings for selected indices"
options:
  index_settings:
    index:
      refresh_interval: 5s
  ignore_unavailable: False
  preserve_existing: False
filters:
- filtertype: ...

This setting must be either True or False.

If preserve_existing is set to True, and the configuration attempts
to push a setting to an index that already has any value for that
setting, the existing setting will be preserved, and remain
unchanged.

The default value of this setting is False

Chapter 71. refresh

Note

This setting is only used by the reindex action.

actions:
  1:
    description: "Reindex index1 into index2"
    action: reindex
    options:
      wait_interval: 9
      max_wait: -1
      refresh: True
      request_body:
        source:
          index: index1
        dest:
          index: index2
    filters:
    - filtertype: none

Setting refresh to True will cause all re-indexed indexes to be
refreshed. This differs from the Index API’s refresh parameter which
causes just the shard that received the new data to be refreshed.

Read more about this setting at http://www.elastic.co/guide/en/
elasticsearch/reference/9.1/docs-reindex.html

The default value is True.

Chapter 72. remote_certificate

This should be a file path to a CA certificate, or left empty.

actions:
  1:
    description: "Reindex index1 into index2"
    action: reindex
    options:
      wait_interval: 9
      max_wait: -1
      remote_certificate: /path/to/my/ca.cert
      remote_client_cert: /path/to/my/client.cert
      remote_client_key: /path/to/my/client.key
      request_body:
        source:
          remote:
            host: https://otherhost:9200
          index: index1
        dest:
          index: index2
    filters:
    - filtertype: none

Note

This option is only used by the Reindex action when performing a
remote reindex operation.

This setting allows the use of a specified CA certificate file to
validate the SSL certificate used by Elasticsearch.

There is no default.

Chapter 73. remote_client_cert

Note

This option is only used by the Reindex action when performing a
remote reindex operation.

This should be a file path to a client certificate (public key), or
left empty.

actions:
  1:
    description: "Reindex index1 into index2"
    action: reindex
    options:
      wait_interval: 9
      max_wait: -1
      remote_certificate: /path/to/my/ca.cert
      remote_client_cert: /path/to/my/client.cert
      remote_client_key: /path/to/my/client.key
      request_body:
        source:
          remote:
            host: https://otherhost:9200
          index: index1
        dest:
          index: index2
    filters:
    - filtertype: none

Allows the use of a specified SSL client cert file to authenticate to
Elasticsearch. The file may contain both an SSL client certificate
and an SSL key, in which case client_key is not used. If specifying
client_cert, and the file specified does not also contain the key,
use client_key to specify the file containing the SSL key. The file
must be in PEM format, and the key part, if used, must be an
unencrypted key in PEM format as well.

Chapter 74. remote_client_key

Note

This option is only used by the Reindex action when performing a
remote reindex operation.

This should be a file path to a client key (private key), or left
empty.

actions:
  1:
    description: "Reindex index1 into index2"
    action: reindex
    options:
      wait_interval: 9
      max_wait: -1
      remote_certificate: /path/to/my/ca.cert
      remote_client_cert: /path/to/my/client.cert
      remote_client_key: /path/to/my/client.key
      request_body:
        source:
          remote:
            host: https://otherhost:9200
          index: index1
        dest:
          index: index2
    filters:
    - filtertype: none

Allows the use of a specified SSL client key file to authenticate to
Elasticsearch. If using client_cert and the file specified does not
also contain the key, use client_key to specify the file containing
the SSL key. The key file must be an unencrypted key in PEM format.

Chapter 75. remote_filters

Note

This option is only used by the Reindex action when performing a
remote reindex operation.

This is an array of filters, exactly as with regular index selection:

actions:
  1:
    description: "Reindex matching indices into index2"
    action: reindex
    options:
      wait_interval: 9
      max_wait: -1
      request_body:
        source:
          remote:
            host: https://otherhost:9200
          index: REINDEX_SELECTION
        dest:
          index: index2
      remote_filters:
      - filtertype: *first*
        setting1: ...
        ...
        settingN: ...
      - filtertype: *second*
        setting1: ...
        ...
        settingN: ...
      - filtertype: *third*
    filters:
    - filtertype: none

This feature will only work when the source index is set to
REINDEX_SELECTION. It will select remote indices matching the filters
provided, and reindex them to the local cluster as the name provided
in the dest index. In this example, that is index2.

Chapter 76. remote_url_prefix

Note

This option is only used by the Reindex action when performing a
remote reindex operation.

This should be a single value or left empty.

actions:
  1:
    description: "Reindex index1 into index2"
    action: reindex
    options:
      wait_interval: 9
      max_wait: -1
      remote_url_prefix: my_prefix
      request_body:
        source:
          remote:
            host: https://otherhost:9200
          index: index1
        dest:
          index: index2
    filters:
    - filtertype: none

In some cases you may be obliged to connect to a remote Elasticsearch
cluster through a proxy of some kind. There may be a URL prefix
before the API URI items, e.g. http://example.com/elasticsearch/ as
opposed to http://localhost:9200. In such a case, set the
remote_url_prefix to the appropriate value, elasticsearch in this
example.

The default is an empty string.

Chapter 77. rename_pattern

Note

This setting is only used by the restore action.

from the Elasticsearch documentation

The rename_pattern and rename_replacement options can be also used to
rename indices on restore using regular expression that supports
referencing the original text as explained here.

actions:
  1:
    action: restore
    description: >-
      Restore all indices in the most recent snapshot with state SUCCESS.  Wait
      for the restore to complete before continuing.  Do not skip the repository
      filesystem access check.  Use the other options to define the index/shard
      settings for the restore.
    options:
      repository:
      # If name is blank, the most recent snapshot by age will be selected
      name:
      # If indices is blank, all indices in the snapshot will be restored
      indices:
      rename_pattern: 'index(.+)'
      rename_replacement: 'restored_index$1'
      wait_for_completion: True
      max_wait: 3600
      wait_interval: 10
    filters:
    - filtertype: state
      state: SUCCESS
      exclude:
    - filtertype: ...

In this configuration, Elasticsearch will capture whatever appears
after index and put it after restored_index. For example, if I was
restoring index-2017.03.01, the resulting index would be renamed to
restored_index-2017.03.01.

Read more about this setting at http://www.elastic.co/guide/en/
elasticsearch/reference/9.1/snapshots-restore-snapshot.html

There is no default value.

Chapter 78. rename_replacement

Note

This setting is only used by the restore action.

From the Elasticsearch documentation

The rename_pattern and rename_replacement options can be also used to
rename indices on restore using regular expression that supports
referencing the original text as explained here.

actions:
  1:
    action: restore
    description: >-
      Restore all indices in the most recent snapshot with state SUCCESS.  Wait
      for the restore to complete before continuing.  Do not skip the repository
      filesystem access check.  Use the other options to define the index/shard
      settings for the restore.
    options:
      repository:
      # If name is blank, the most recent snapshot by age will be selected
      name:
      # If indices is blank, all indices in the snapshot will be restored
      indices:
      rename_pattern: 'index(.+)'
      rename_replacement: 'restored_index$1'
      wait_for_completion: True
      max_wait: 3600
      wait_interval: 10
    filters:
    - filtertype: state
      state: SUCCESS
      exclude:
    - filtertype: ...

In this configuration, Elasticsearch will capture whatever appears
after index and put it after restored_index. For example, if I was
restoring index-2017.03.01, the resulting index would be renamed to
restored_index-2017.03.01.

Read more about this setting at http://www.elastic.co/guide/en/
elasticsearch/reference/9.1/snapshots-restore-snapshot.html

There is no default value.

Chapter 79. repository

Note

This setting is only used by the snapshot, and delete snapshots
actions.

There is no default value. This setting must be set by the user or an
exception will be raised, and execution will halt.

79.1. restore

actions:
  1:
    action: restore
    description: Restore my_index from my_snapshot in my_repository
    options:
      repository: my_repository
      name: my_snapshot
      indices: my_index
      wait_for_completion: True
      max_wait: 3600
      wait_interval: 10
    filters:
    - filtertype: state
      state: SUCCESS
      exclude:
    - filtertype: ...

79.2. snapshot

action: snapshot
description: >-
  Snapshot selected indices to 'repository' with the snapshot name or name
  pattern in 'name'.  Use all other options as assigned
options:
  repository: my_repository
  name: my_snapshot
  wait_for_completion: True
  max_wait: 3600
  wait_interval: 10
filters:
- filtertype: ...

Chapter 80. requests_per_second

Note

This option is only used by the Reindex action

actions:
  1:
    description: "Reindex index1 into index2"
    action: reindex
    options:
      wait_interval: 9
      max_wait: -1
      requests_per_second: -1
      request_body:
        source:
          index: index1
        dest:
          index: index2
    filters:
    - filtertype: none

requests_per_second can be set to any positive decimal number (1.4,
6, 1000, etc) and throttles the number of requests per second that
the reindex issues or it can be set to -1 to disable throttling.

The default value for this is option is -1.

Chapter 81. request_body

Note

This setting is only used by the reindex action.

81.1. Manual index selection

The request_body option is the heart of the reindex action. In here,
using YAML syntax, you recreate the body sent to Elasticsearch as
described in the official documentation. You can manually select
indices as with this example:

actions:
  1:
    description: "Reindex index1 into index2"
    action: reindex
    options:
      wait_interval: 9
      max_wait: -1
      request_body:
        source:
          index: index1
        dest:
          index: index2
    filters:
    - filtertype: none

You can also select multiple indices to reindex by making a list in
acceptable YAML syntax:

actions:
  1:
    description: "Reindex index1,index2,index3 into new_index"
    action: reindex
    options:
      wait_interval: 9
      max_wait: -1
      request_body:
        source:
          index: ['index1', 'index2', 'index3']
        dest:
          index: new_index
    filters:
    - filtertype: none

Important

You must set at least a none filter, or the action will fail. Do not
worry. If you’ve manually specified your indices, those are the only
ones which will be reindexed.

81.2. Filter-Selected Indices

Curator allows you to use all indices found by the filters section by
setting the source index to REINDEX_SELECTION, like this:

actions:
  1:
    description: >-
      Reindex all daily logstash indices from March 2017 into logstash-2017.03
    action: reindex
    options:
      wait_interval: 9
      max_wait: -1
      request_body:
        source:
          index: REINDEX_SELECTION
        dest:
          index: logstash-2017.03
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-2017.03.

81.3. Reindex From Remote

You can also reindex from remote:

actions:
  1:
    description: "Reindex remote index1 to local index1"
    action: reindex
    options:
      wait_interval: 9
      max_wait: -1
      request_body:
        source:
          remote:
            host: http://otherhost:9200
            username: myuser
            password: mypass
          index: index1
        dest:
          index: index1
    filters:
    - filtertype: none

Important

You must set at least a none filter, or the action will fail. Do not
worry. Only the indices you specified in source will be reindexed.

Curator will create a connection to the host specified as the host
key in the above example. It will determine which port to connect to,
and whether to use SSL by parsing the URL entered there. Because this
host is specifically used by Elasticsearch, and Curator is making a
separate connection, it is important to ensure that both Curator and
your Elasticsearch cluster have access to the remote host.

If you do not whitelist the remote cluster, you will not be able to
reindex. This can be done by adding the following line to your
elasticsearch.yml file:

reindex.remote.whitelist: remote_host_or_IP1:9200, remote_host_or_IP2:9200

or by adding this flag to the command-line when starting
Elasticsearch:

bin/elasticsearch -Edefault.reindex.remote.whitelist="remote_host_or_IP:9200"

Of course, be sure to substitute the correct host, IP, or port.

Other client connection arguments can also be supplied in the form of
action options:

  * remote_url_prefix
  * remote_certificate
  * remote_client_cert
  * remote_client_key

81.4. Reindex From Remote With Filter-Selected Indices

You can even reindex from remote with filter-selected indices on the
remote side:

actions:
  1:
    description: >-
      Reindex all remote daily logstash indices from March 2017 into local index
      logstash-2017.03
    action: reindex
    options:
      wait_interval: 9
      max_wait: -1
      request_body:
        source:
          remote:
            host: http://otherhost:9200
            username: myuser
            password: mypass
          index: REINDEX_SELECTION
        dest:
          index: logstash-2017.03
      remote_filters:
      - filtertype: pattern
        kind: prefix
        value: logstash-2017.03.
    filters:
    - filtertype: none

Important

Even though you are reindexing from remote, you must set at least a
none filter, or the action will fail. Do not worry. Only the indices
specified in source will be reindexed.

Curator will create a connection to the host specified as the host
key in the above example. It will determine which port to connect to,
and whether to use SSL by parsing the URL entered there. Because this
host is specifically used by Elasticsearch, and Curator is making a
separate connection, it is important to ensure that both Curator and
your Elasticsearch cluster have access to the remote host.

If you do not whitelist the remote cluster, you will not be able to
reindex. This can be done by adding the following line to your
elasticsearch.yml file:

reindex.remote.whitelist: remote_host_or_IP1:9200, remote_host_or_IP2:9200

or by adding this flag to the command-line when starting
Elasticsearch:

bin/elasticsearch -Edefault.reindex.remote.whitelist="remote_host_or_IP:9200"

Of course, be sure to substitute the correct host, IP, or port.

Other client connection arguments can also be supplied in the form of
action options:

  * remote_url_prefix
  * remote_certificate
  * remote_client_cert
  * remote_client_key

81.5. Reindex - Migration

Curator allows reindexing, particularly from remote, as a migration
path. This can be a very useful feature for migrating an older
cluster (1.4+) to a new cluster, on different hardware. It can also
be a useful tool for serially reindexing indices into newer mappings
in an automatable way.

Ordinarily, a reindex operation is from either one, or many indices,
to a single, named index. Assigning the dest index to MIGRATION tells
Curator to treat this reindex differently.

Important

If it is a local reindex, you must set either migration_prefix, or
migration_suffix, or both. This prevents collisions and other bad
things from happening. By assigning a prefix or a suffix (or both),
you can reindex any local indices to new versions of themselves, but
named differently.

It is true the Reindex API already has this functionality. Curator
includes this same functionality for convenience.

This example will reindex all of the remote indices matching
logstash-2017.03. into the local cluster, but preserve the original
index names, rather than merge all of the contents into a single
index. Internal to Curator, this results in multiple reindex actions:
one per index. All other available options and settings are
available.

actions:
  1:
    description: >-
      Reindex all remote daily logstash indices from March 2017 into local
      versions with the same index names.
    action: reindex
    options:
      wait_interval: 9
      max_wait: -1
      request_body:
        source:
          remote:
            host: http://otherhost:9200
            username: myuser
            password: mypass
          index: REINDEX_SELECTION
        dest:
          index: MIGRATION
      remote_filters:
      - filtertype: pattern
        kind: prefix
        value: logstash-2017.03.
    filters:
    - filtertype: none

Important

Even though you are reindexing from remote, you must set at least a
none filter, or the action will fail. Do not worry. Only the indices
specified in source will be reindexed.

Curator will create a connection to the host specified as the host
key in the above example. It will determine which port to connect to,
and whether to use SSL by parsing the URL entered there. Because this
host is specifically used by Elasticsearch, and Curator is making a
separate connection, it is important to ensure that both Curator and
your Elasticsearch cluster have access to the remote host.

If you do not whitelist the remote cluster, you will not be able to
reindex. This can be done by adding the following line to your
elasticsearch.yml file:

reindex.remote.whitelist: remote_host_or_IP1:9200, remote_host_or_IP2:9200

or by adding this flag to the command-line when starting
Elasticsearch:

bin/elasticsearch -Edefault.reindex.remote.whitelist="remote_host_or_IP:9200"

Of course, be sure to substitute the correct host, IP, or port.

Other client connection arguments can also be supplied in the form of
action options:

  * remote_url_prefix
  * remote_certificate
  * remote_client_cert
  * remote_client_key
  * migration_prefix
  * migration_suffix

81.6. Other scenarios and options

Nearly all scenarios supported by the reindex API are supported in
the request_body, including (but not limited to):

  * Pipelines
  * Scripting
  * Queries
  * Conflict resolution
  * Limiting by count
  * Versioning
  * Reindexing operation type (for example, create-only)

Read more about these, and more, at http://www.elastic.co/guide/en/
elasticsearch/reference/9.1/docs-reindex.html

Notable exceptions include:

  * You cannot manually specify slices. Instead, use the slices
    option for automated sliced reindexing.

Chapter 82. retry_count

Note

This setting is only used by the delete snapshots action.

action: delete_snapshots
description: "Delete selected snapshots from 'repository'"
options:
  repository: ...
  retry_interval: 120
  retry_count: 3
filters:
- filtertype: ...

The value of this setting is the number of times to retry deleting a
snapshot.

The default for this setting is 3.

Chapter 83. retry_interval

Note

This setting is only used by the delete snapshots action.

action: delete_snapshots
description: "Delete selected snapshots from 'repository'"
options:
  repository: ...
  retry_interval: 120
  retry_count: 3
filters:
- filtertype: ...

The value of this setting is the number of seconds to delay between
retries.

The default for this setting is 120.

Chapter 84. routing_type

Note

This setting is only used by the cluster_routing action.

action: cluster_routing
description: "Apply routing rules to the entire cluster"
options:
  routing_type:
  value: ...
  setting: enable
  wait_for_completion: True
  max_wait: 300
  wait_interval: 10

The value of this setting must be either allocation or rebalance

There is no default value. This setting must be set by the user or an
exception will be raised, and execution will halt.

Chapter 85. search_pattern

Note

This setting is only used by the Chapter 22, Allocation, Chapter 23, 
Close, Chapter 25, Cold2Frozen, Chapter 27, Delete Indices,
Chapter 29, Forcemerge, Chapter 30, Index Settings, Chapter 31, Open,
Chapter 33, Replicas, Chapter 36, Shrink, and Chapter 37, Snapshot
actions.

action: delete_indices
description: "Delete selected indices"
options:
  search_pattern: 'index-*,-index-00002,named-1'
filters:
- filtertype: ...

Supports Elasticsearch multi-target syntax.

The value of this option can be a comma-separated list of data
streams, indices, and aliases used to limit the request.

To target all data streams and indices, omit this parameter or use *.
If using wildcards it is highly recommended to encapsulate the entire
search pattern in single quotes, e.g. search_pattern: 'a*,b*,c*'

The default value is *.

Even though * is the default, system indices are filtered by default.
This appends the pattern:

EXCLUDE_SYSTEM = (
    '-.kibana*,-.security*,-.watch*,-.triggered_watch*,'
    '-.ml*,-.geoip_databases*,-.logstash*,-.tasks*'
)

Chapter 86. setting

Note

This setting is only used by the cluster_routing action.

action: cluster_routing
description: "Apply routing rules to the entire cluster"
options:
  routing_type:
  value: ...
  setting: enable
  wait_for_completion: True
  max_wait: 300
  wait_interval: 10

The value of this must be enable at present. It is a placeholder for
future expansion.

There is no default value. This setting must be set by the user or an
exception will be raised, and execution will halt.

Chapter 87. shrink_node

Note

This setting is only used by the shrink action.

action: shrink
description: >-
  Shrink selected indices on the node with the most available space, excluding
  master nodes and the node named 'not_this_node'
options:
  shrink_node: DETERMINISTIC
  node_filters:
    permit_masters: False
    exclude_nodes: ['not_this_node']
  shrink_suffix: '-shrink'
filters:
  - filtertype: ...

This setting is required. There is no default value.

The value of this setting must be the valid name of a node in your
Elasticsearch cluster, or DETERMINISTIC. If the value is
DETERMINISTIC, Curator will automatically select the data node with
the most available free space and make that the target node. Curator
will repeat this process for each successive index when the value is
DETERMINISTIC.

If node_filters, such as exclude_nodes are defined, those nodes will
not be considered as potential target nodes.

Chapter 88. shrink_prefix

Note

This setting is only used by the shrink action.

action: shrink
description: >-
  Shrink selected indices on the node with the most available space.
  Prepend target index names with 'foo-' and append a suffix of '-shrink'
options:
  shrink_node: DETERMINISTIC
  shrink_prefix: 'foo-'
  shrink_suffix: '-shrink'
filters:
  - filtertype: ...

There is no default value for this setting.

The value of this setting will be prepended to target index names. If
the source index were index, and the shrink_prefix were foo-, and the
shrink_suffix were -shrink, the resulting target index name would be
foo-index-shrink.

Chapter 89. shrink_suffix

Note

This setting is only used by the shrink action.

action: shrink
description: >-
  Shrink selected indices on the node with the most available space.
  Prepend target index names with 'foo-' and append a suffix of '-shrink'
options:
  shrink_node: DETERMINISTIC
  shrink_prefix: 'foo-'
  shrink_suffix: '-shrink'
filters:
  - filtertype: ...

The default value for this setting is -shrink.

The value of this setting will be appended to target index names. If
the source index were index, and the shrink_prefix were foo-, and the
shrink_suffix were -shrink, the resulting target index name would be
foo-index-shrink.

Chapter 90. slices

Note

This setting is only used by the reindex action.

This setting can speed up reindexing operations by using Sliced
Scroll to slice on the \_uid.

actions:
  1:
    description: "Reindex index1,index2,index3 into new_index"
    action: reindex
    options:
      wait_interval: 9
      max_wait: -1
      slices: 3
      request_body:
        source:
          index: ['index1', 'index2', 'index3']
        dest:
          index: new_index
    filters:
    - filtertype: none

90.1. Picking the number of slices

Here are a few recommendations around the number of slices to use:

  * Don’t use large numbers. 500 creates fairly massive CPU thrash,
    so Curator will not allow a number larger than this.
  * It is more efficient from a query performance standpoint to use
    some multiple of the number of shards in the source index.
  * Using exactly as many shards as are in the source index is the
    most efficient from a query performance standpoint.
  * Indexing performance should scale linearly across available
    resources with the number of slices.
  * Whether indexing or query performance dominates that process
    depends on lots of factors like the documents being reindexed and
    the cluster doing the reindexing.

Chapter 91. skip_repo_fs_check

Note

This setting is used by the snapshot and restore actions.

This setting must be either True or False.

The default value of this setting is False

91.1. restore

Each master and data node in the cluster must have write access to
the shared filesystem used by the repository for a snapshot to be
100% valid. For the purposes of a restore, this is a lightweight
attempt to ensure that all nodes are still actively able to write to
the repository, in hopes that snapshots were from all nodes. It is
not otherwise necessary for the purposes of a restore.

Some filesystems may take longer to respond to a check, which results
in a false positive for the filesystem access check. For these cases,
it is desirable to bypass this verification step, by setting this to
True.

actions:
  1:
    action: restore
    description: Restore my_index from my_snapshot in my_repository
    options:
      repository: my_repository
      name: my_snapshot
      indices: my_index
      skip_repo_fs_check: False
      wait_for_completion: True
      max_wait: 3600
      wait_interval: 10
    filters:
    - filtertype: state
      state: SUCCESS
      exclude:
    - filtertype: ...

91.2. snapshot

Each master and data node in the cluster must have write access to
the shared filesystem used by the repository for a snapshot to be
100% valid.

Some filesystems may take longer to respond to a check, which results
in a false positive for the filesystem access check. For these cases,
it is desirable to bypass this verification step, by setting this to
True.

action: snapshot
description: >-
  Snapshot selected indices to 'repository' with the snapshot name or name
  pattern in 'name'.  Use all other options as assigned
options:
  repository: my_repository
  name: my_snapshot
  skip_repo_fs_check: False
  wait_for_completion: True
  max_wait: 3600
  wait_interval: 10
filters:
- filtertype: ...

Chapter 92. timeout

Note

This setting is only used by the reindex action.

The timeout is the length in seconds each individual bulk request
should wait for shards that are unavailable. The default value is 60,
meaning 60 seconds.

actions:
  1:
    description: "Reindex index1,index2,index3 into new_index"
    action: reindex
    options:
      wait_interval: 9
      max_wait: -1
      timeout: 90
      request_body:
        source:
          index: ['index1', 'index2', 'index3']
        dest:
          index: new_index
    filters:
    - filtertype: none

Chapter 93. timeout_override

Note

This setting is available in all actions.

action: forcemerge
description: >-
  Perform a forceMerge on selected indices to 'max_num_segments' per shard
options:
  max_num_segments: 2
  timeout_override: 21600
filters:
- filtertype: ...

If timeout_override is unset in your configuration, some actions will
try to set a sane default value.

The following table shows these default values:

+-----------------------------------------------+
|Action Name     |Default timeout_override Value|
|----------------+------------------------------|
|close           |180                           |
|----------------+------------------------------|
|delete_snapshots|300                           |
|----------------+------------------------------|
|forcemerge      |21600                         |
|----------------+------------------------------|
|restore         |21600                         |
|----------------+------------------------------|
|snapshot        |21600                         |
+-----------------------------------------------+

All other actions have no default value for timeout_override.

This setting must be an integer number of seconds, or an error will
result.

This setting is particularly useful for the forceMerge action, as all
other actions have a new polling behavior when using
wait_for_completion that should reduce or prevent client timeouts.

Chapter 94. value

Note

This setting is optional when using the allocation action and
required when using the cluster_routing action.

94.1. allocation

For the allocation action, the value of this setting should
correspond to a node setting on one or more nodes in your cluster

For example, you might have set

node.tag: myvalue

in your elasticsearch.yml file for one or more of your nodes. To
match allocation in this case, set value to myvalue. Additonally, if
you used one of the special attribute names _ip, _name, _id, or _host
for key, value can match the one of the node ip addresses, names,
ids, or host names, respectively.

Note

To remove a routing allocation, the value of this setting should be
left empty, or the value setting not even included as an option.

For example, you might have set

PUT test/_settings
{
  "index.routing.allocation.exclude.size": "small"
}

to keep index test from allocating shards on nodes that have
node.tag: small. To remove this shard routing allocation setting, you
might use an action file similar to this:

---
  actions:
    1:
      action: allocation
      description: ->
        Unset 'index.routing.allocation.exclude.size' for index 'test' by
        passing an empty value.
      options:
        key: size
        value: ...
        allocation_type: exclude
      filters:
      - filtertype: pattern
        kind: regex
        value: '^test$'

94.2. cluster_routing

For the cluster_routing action, the acceptable values for this
setting depend on the value of routing_type.

action: cluster_routing
description: "Apply routing rules to the entire cluster"
options:
  routing_type: ...
  value: ...
  setting: enable
  wait_for_completion: True
  max_wait: 300
  wait_interval: 10

Acceptable values when routing_type is either allocation or rebalance
are all, primaries, and none (string, not NoneType).

If routing_type is allocation, this can also be new_primaries. If
routing_type is rebalance, then the value can also be replicas.

There is no default value. This setting must be set by the user or an
exception will be raised, and execution will halt.

Chapter 95. wait_for_active_shards

Note

This setting is used by the Reindex, Rollover, and Shrink actions.
Each use it similarly.

This setting determines the number of shard copies that must be
active before the client returns. The default value is 1, which
implies only the primary shards.

Set to all for all shard copies, otherwise set to any non-negative
value less than or equal to the total number of copies for the shard
(number of replicas + 1)

Read the Elasticsearch documentation for more information.

95.1. Reindex

actions:
  1:
    description: "Reindex index1,index2,index3 into new_index"
    action: reindex
    options:
      wait_interval: 9
      max_wait: -1
      wait_for_active_shards: 2
      request_body:
        source:
          index: ['index1', 'index2', 'index3']
        dest:
          index: new_index
    filters:
    - filtertype: none

95.2. Rollover

action: rollover
description: >-
  Rollover the index associated with alias 'name', which should be in the
  form of prefix-000001 (or similar), or prefix-yyyy.MM.DD-1.
options:
  name: aliasname
  conditions:
    max_age: 1d
    max_docs: 1000000
  wait_for_active_shards: 1
  extra_settings:
    index.number_of_shards: 3
    index.number_of_replicas: 1
  timeout_override:
  continue_if_exception: False
  disable_action: False

95.3. Shrink

action: shrink
description: >-
  Shrink selected indices on the node with the most available space.
  Prepend target index names with 'foo-' and append a suffix of '-shrink'
options:
  shrink_node: DETERMINISTIC
  wait_for_active_shards: all
filters:
  - filtertype: ...

Chapter 96. wait_for_completion

Note

This setting is used by the allocation, cluster_routing, reindex,
replicas, restore, and snapshot actions.

This setting must be either True or False.

This setting specifies whether or not the request should return
immediately or wait for the operation to complete before returning.

96.1. allocation

action: allocation
description: "Apply shard allocation filtering rules to the specified indices"
options:
  key: ...
  value: ...
  allocation_type: ...
  wait_for_completion: False
  max_wait: 300
  wait_interval: 10
filters:
- filtertype: ...

The default value for the allocation action is False.

96.2. cluster_routing

action: cluster_routing
description: "Apply routing rules to the entire cluster"
options:
  routing_type:
  value: ...
  setting: enable
  wait_for_completion: True
  max_wait: 300
  wait_interval: 10

The default value for the cluster_routing action is False.

96.3. reindex

actions:
  1:
    description: "Reindex index1 into index2"
    action: reindex
    options:
      wait_interval: 9
      max_wait: -1
      request_body:
        source:
          index: index1
        dest:
          index: index2
    filters:
    - filtertype: none

The default value for the reindex action is False.

96.4. replicas

action: replicas
description: >- Set the number of replicas per shard for selected
    indices to 'count'
options:
  count: ...
  wait_for_completion: True
  max_wait: 600
  wait_interval: 10
filters:
- filtertype: ...

The default value for the replicas action is False.

96.5. restore

actions:
  1:
    action: restore
    description: Restore my_index from my_snapshot in my_repository
    options:
      repository: my_repository
      name: my_snapshot
      indices: my_index
      wait_for_completion: True
      max_wait: 3600
      wait_interval: 10
    filters:
    - filtertype: state
      state: SUCCESS
      exclude:
    - filtertype: ...

The default value for the restore action is True.

96.6. snapshot

action: snapshot
description: >-
  Snapshot selected indices to 'repository' with the snapshot name or name
  pattern in 'name'.  Use all other options as assigned
options:
  repository: my_repository
  name: my_snapshot
  wait_for_completion: True
  max_wait: 3600
  wait_interval: 10
filters:
- filtertype: ...

The default value for the snapshot action is True.

Tip

During snapshot initialization, information about all previous
snapshots is loaded into the memory, which means that in large
repositories it may take several seconds (or even minutes) for this
command to return even if the wait_for_completion setting is set to
False.

Chapter 97. wait_for_rebalance

action: shrink
description: >-
  Shrink selected indices on the node with the most available space.
  Delete source index after successful shrink, then reroute the shrunk
  index with the provided parameters.
options:
  ignore_empty_list: True
  shrink_node: DETERMINISTIC
  node_filters:
    permit_masters: False
    exclude_nodes: ['not_this_node']
  number_of_shards: 1
  number_of_replicas: 1
  shrink_prefix:
  shrink_suffix: '-shrink'
  delete_after: True
  post_allocation:
    allocation_type: include
    key: node_tag
    value: cold
  wait_for_active_shards: 1
  extra_settings:
    settings:
      index.codec: best_compression
  wait_for_completion: True
  wait_for_rebalance: True
  wait_interval: 9
  max_wait: -1
filters:
  - filtertype: ...

Note

This setting is used by the shrink action.

This setting must be true or false.

Setting this to false will result in the shrink action only checking
that the index being shrunk has finished being relocated, and not
continue to wait for the cluster to fully rebalance all shards.

The default value for this setting is false.

Chapter 98. wait_interval

Note

This setting is used by the allocation, cluster_routing, reindex,
replicas, restore, and snapshot actions.

This setting must be a positive integer between 1 and 30.

This setting specifies how long to wait between checks to see if the
action has completed or not. This number should not be larger than
the client request_timeout or the timeout_override. As the default
client request_timeout value for is 30, this should be uncommon.

The default value for this setting is 9, meaning 9 seconds between
checks.

This option is generally used in conjunction with max_wait, which is
the maximum amount of time in seconds to wait for the given action to
complete.

98.1. allocation

action: allocation
description: "Apply shard allocation filtering rules to the specified indices"
options:
  key: ...
  value: ...
  allocation_type: ...
  wait_for_completion: False
  max_wait: 300
  wait_interval: 10
filters:
- filtertype: ...

98.2. cluster_routing

action: cluster_routing
description: "Apply routing rules to the entire cluster"
options:
  routing_type:
  value: ...
  setting: enable
  wait_for_completion: True
  max_wait: 300
  wait_interval: 10

98.3. reindex

actions:
  1:
    description: "Reindex index1 into index2"
    action: reindex
    options:
      wait_interval: 9
      max_wait: -1
      request_body:
        source:
          index: index1
        dest:
          index: index2
    filters:
    - filtertype: none

98.4. replicas

action: replicas
description: >- Set the number of replicas per shard for selected
    indices to 'count'
options:
  count: ...
  wait_for_completion: True
  max_wait: 600
  wait_interval: 10
filters:
- filtertype: ...

98.5. restore

actions:
  1:
    action: restore
    description: Restore my_index from my_snapshot in my_repository
    options:
      repository: my_repository
      name: my_snapshot
      indices: my_index
      wait_for_completion: True
      max_wait: 3600
      wait_interval: 10
    filters:
    - filtertype: state
      state: SUCCESS
      exclude:
    - filtertype: ...

98.6. snapshot

action: snapshot
description: >-
  Snapshot selected indices to 'repository' with the snapshot name or name
  pattern in 'name'.  Use all other options as assigned
options:
  repository: my_repository
  name: my_snapshot
  wait_for_completion: True
  max_wait: 3600
  wait_interval: 10
filters:
- filtertype: ...

Chapter 99. warn_if_no_indices

Note

This setting is only used by the alias action.

This setting must be either True or False.

The default value for this setting is False.

action: alias
description: "Add/Remove selected indices to or from the specified alias"
options:
  name: alias_name
  warn_if_no_indices: False
add:
  filters:
  - filtertype: ...
remove:
  filters:
  - filtertype: ...

This setting specifies whether or not the alias action should
continue with a warning or return immediately in the event that the
filters in the add or remove section result in an empty index list.

Improper use of this setting can yield undesirable results

Ideal use case: For example, you want to add the most recent seven
days of time-series indices into a lastweek alias, and remove indices
older than seven days from this same alias. If you do not not yet
have any indices older than seven days, this will result in an empty
index list condition which will prevent the entire alias action from
completing successfully. If warn_if_no_indices were set to True,
however, it would avert that potential outcome.

Potentially undesirable outcome: A non-beneficial case would be where
if warn_if_no_indices is set to True, and a misconfiguration results
in indices not being found, and therefore not being disassociated
from the alias. As a result, an alias that should only query one week
now references multiple weeks of data. If warn_if_no_indices were set
to False, this scenario would have been averted because the empty
list condition would have resulted in an error.

Part IX. Filters

Filters are the way to select only the indices (or snapshots) you
want.

Filter chaining

It is important to note that while filters can be chained, each is
linked by an implied logical AND operation. If you want to match from
one of several different patterns, as with a logical OR operation,
you can do so with the pattern filtertype using regex as the kind.

This example shows how to select multiple indices based on them
beginning with either alpha-, bravo-, or charlie-:

  filters:
  - filtertype: pattern
    kind: regex
    value: '^(alpha-|bravo-|charlie-).*$'

Explaining all of the different ways in which regular expressions can
be used is outside the scope of this document, but hopefully this
gives you some idea of how a regular expression pattern can be used
when a logical OR is desired.

The index filtertypes are:

  * age
  * alias
  * allocated
  * closed
  * count
  * empty
  * forcemerged
  * kibana
  * none
  * opened
  * pattern
  * period
  * space

The snapshot filtertypes are:

  * age
  * count
  * none
  * pattern
  * period
  * state

You can use environment variables in your configuration files.

Table of Contents

100. filtertype
101. age
    101.1. Age calculation
    101.2. name-based ages
    101.3. creation_date-based ages
    101.4. field_stats-based ages
    101.5. Required settings
    101.6. Dependent settings
    101.7. Optional settings
102. alias
    102.1. Required settings
    102.2. Optional settings
103. allocated
    103.1. Required settings
    103.2. Optional settings
104. closed
    104.1. Optional settings
105. count
    105.1. Age-based sorting
    105.2. Pattern-based sorting
    105.3. Reversing sorting
    105.4. Required settings
    105.5. Optional settings
    105.6. Index-only settings
106. empty
    106.1. Optional settings
107. forcemerged
    107.1. Required settings
    107.2. Optional settings
108. kibana
    108.1. Optional settings
109. none
110. opened
    110.1. Optional settings
111. pattern
    111.1. prefix
    111.2. suffix
    111.3. timestring
    111.4. regex
    111.5. Required settings
    111.6. Optional settings
112. period
    112.1. Relative Periods
    112.2. Absolute Periods
    112.3. Required settings
    112.4. Dependent settings
    112.5. Optional settings
113. space
    113.1. Deleting Indices By Space
    113.2. Age-based sorting
    113.3. Reversing sorting
    113.4. Required settings
    113.5. Optional settings
114. state
    114.1. Required settings
    114.2. Optional settings

Chapter 100. filtertype

Each filter is defined first by a filtertype. Each filtertype has its
own settings, or no settings at all. In a configuration file, filters
are defined as follows:

- filtertype: *first*
  setting1: ...
  ...
  settingN: ...
- filtertype: *second*
  setting1: ...
  ...
  settingN: ...
- filtertype: *third*

The - indicates in the YAML that this is an array element. Each
filtertype declaration must be preceded by a - for the filters to be
read properly. This is how Curator can chain filters together.
Anywhere filters can be used, multiple can be chained together in
this manner.

The index filtertypes are:

  * age
  * alias
  * allocated
  * closed
  * count
  * empty
  * forcemerged
  * kibana
  * none
  * opened
  * pattern
  * period
  * space

The snapshot filtertypes are:

  * age
  * count
  * none
  * pattern
  * period
  * state

Chapter 101. age

Note

Empty values and commented lines will result in the default value, if
any, being selected. If a setting is set, but not used by a given
filtertype, it may generate an error.

This filtertype will iterate over the actionable list and match
indices based on their age. They will remain in, or be removed from
the actionable list based on the value of exclude.

101.1. Age calculation

units are calculated as follows:

+--------------------------------+
|Unit   |Seconds |Note           |
|-------+--------+---------------|
|seconds|1       |One second     |
|-------+--------+---------------|
|minutes|60      |Calculated as  |
|       |        |60 seconds     |
|-------+--------+---------------|
|hours  |3600    |Calculated as  |
|       |        |60 minutes     |
|       |        |(60*60)        |
|-------+--------+---------------|
|days   |86400   |Calculated as  |
|       |        |24 hours       |
|       |        |(24*60*60)     |
|-------+--------+---------------|
|weeks  |604800  |Calculated as 7|
|       |        |days           |
|       |        |(7*24*60*60)   |
|-------+--------+---------------|
|months |2592000 |Calculated as  |
|       |        |30 days        |
|       |        |(30*24*60*60)  |
|-------+--------+---------------|
|years  |31536000|Calculated as  |
|       |        |365 days       |
|       |        |(365*24*60*60) |
+--------------------------------+

All calculations are in epoch time, which is the number of seconds
elapsed since 1 Jan 1970. If no epoch is specified in the filter,
then the current epoch time-which is always UTC-is used as the basis
for comparison.

As epoch time is always increasing, lower numbers indicate dates and
times in the past.

When age is calculated, unit is multiplied by unit_count to obtain a
total number of seconds to use as a differential.

For example, if the time at execution were 2017-04-07T15:00:00Z
(UTC), then the epoch timestamp would be 1491577200. If I had an age
filter defined like this:

 - filtertype: age
   source: creation_date
   direction: older
   unit: days
   unit_count: 3

The time differential would be 3*24*60*60 seconds, which is 259200
seconds. Subtracting this value from 1491577200 gives us 1491318000,
which is 2017-04-04T15:00:00Z (UTC), exactly 3 days in the past. The
creation_date of indices or snapshots is compared to this timestamp.
If it is older, it stays in the actionable list, otherwise it is
removed from the actionable list.

age filter vs. period filter

The time differential means of calculation can lead to frustration.

Setting unit to months, and unit_count to 3 will actually calculate
the age as 3*30*24*60*60, which is 7776000 seconds. This may be a big
deal. If the date is 2017-01-01T02:30:00Z, or 1483237800 in epoch
time. Subtracting 7776000 seconds makes 1475461800, which is
2016-10-03T02:30:00Z. If you were to try to match monthly indices,
index-2016.12, index-2016.11, 2016.10, 2016.09, etc., then both
index-2016.09 and index-2016.10 will be older than the cutoff date.
This may result in unintended behavior.

Another way this can cause issues is with weeks. Weekly indices may
start on Sunday or Monday. The age filter’s calculation doesn’t take
this into consideration, and merely tests the difference between
execution time and the timestamp on the index (from any source).

Another means of selecting indices and snapshots is the period
filter, which is perhaps a better choice for selecting weeks and
months as it compensates for these differences.

101.2. name-based ages

Using name as the source tells Curator to look for a timestring
within the index or snapshot name, and convert that into an epoch
timestamp (epoch implies UTC).

 - filtertype: age
   source: name
   direction: older
   timestring: '%Y.%m.%d'
   unit: days
   unit_count: 3

A word about regular expression matching with timestrings

Timestrings are parsed from strftime patterns, like %Y.%m.%d, into
regular expressions. For example, %Y is 4 digits, so the regular
expression for that looks like \d{4}, and %m is 2 digits, so the
regular expression is \d{2}.

What this means is that a simple timestring to match year and month,
%Y.%m will result in a regular expression like this: ^.*\d{4}\.\d
{2}.*$. This pattern will match any 4 digits, followed by a period .,
followed by 2 digits, occurring anywhere in the index name. This
means it will match monthly indices, like index-2016.12, as well as
daily indices, like index-2017.04.01, which may not be the intended
behavior.

To compensate for this, when selecting indices matching a subset of
another pattern, use a second filter with exclude set to True

- filtertype: pattern
 kind: timestring
 value: '%Y.%m'
- filtertype: pattern
 kind: timestring
 value: '%Y.%m.%d'
 exclude: True

This will prevent the %Y.%m pattern from matching the %Y.%m part of
the daily indices.

This applies whether using timestring as a mere pattern match, or as
part of date calculations.

101.3. creation_date-based ages

creation_date extracts the epoch time of index or snapshot creation.

 - filtertype: age
   source: creation_date
   direction: older
   unit: days
   unit_count: 3

101.4. field_stats-based ages

Note

source can only be field_stats when filtering indices.

In Curator 5.3 and older, source field_stats uses the Field Stats API
to calculate either the min_value or the max_value of the field as
the stats_result, and then use that value for age comparisons. In 5.4
and above, even though it is still called field_stats, it uses an
aggregation to calculate the same values, as the field_stats API is
no longer used in Elasticsearch 6.x and up.

field must be of type date in Elasticsearch.

 - filtertype: age
   source: field_stats
   direction: older
   unit: days
   unit_count: 3
   field: '@timestamp'
   stats_result: min_value

101.5. Required settings

  * source
  * direction
  * unit
  * unit_count

101.6. Dependent settings

  * timestring (required if source is name)
  * field (required if source is field_stats) [Indices only]
  * stats_result (only used if source is field_stats) [Indices only]

101.7. Optional settings

  * unit_count_pattern
  * epoch
  * exclude (default is False)

Chapter 102. alias

- filtertype: alias
  aliases: ...

Note

Empty values and commented lines will result in the default value, if
any, being selected. If a setting is set, but not used by a given
filtertype, it may generate an error.

This filtertype will iterate over the actionable list and match
indices based on whether they are associated with the given aliases,
which can be a single value, or an array. They will remain in, or be
removed from the actionable list based on the value of exclude.

Matching Indices and Aliases

Indices must be in all aliases to match.

If a list of aliases is provided (instead of only one), indices must
appear in all listed aliases or a 404 error will result, leading to
no indices being matched. In older versions, if the index was
associated with even one of the aliases in aliases, it would result
in a match.

102.1. Required settings

  * aliases

102.2. Optional settings

  * exclude

Chapter 103. allocated

- filtertype: allocated
  key: ...
  value: ...
  allocation_type:
  exclude: True

Note

Empty values and commented lines will result in the default value, if
any, being selected. If a setting is set, but not used by a given
filtertype, it may generate an error.

This filtertype will iterate over the actionable list and match
indices based on their shard routing allocation settings. They will
remain in, or be removed from the actionable list based on the value
of exclude.

By default the indices matched by the allocated filter will be
excluded since the exclude setting defaults to True.

To include matching indices rather than exclude, set the exclude
setting to False.

103.1. Required settings

  * key
  * value

103.2. Optional settings

  * allocation_type
  * exclude (default is True)

Chapter 104. closed

- filtertype: closed
  exclude: True

This filtertype will iterate over the actionable list and match
indices which are closed. They will remain in, or be removed from the
actionable list based on the value of exclude.

104.1. Optional settings

  * exclude (default is True)

Chapter 105. count

- filtertype: count
  count: 10

Note

Empty values and commented lines will result in the default value, if
any, being selected. If a setting is set, but not used by a given
filtertype, it may generate an error.

This filtertype will iterate over the actionable list of indices or
snapshots. They are ordered by age, or by alphabet, so as to
guarantee that the correct items will remain in, or be removed from
the actionable list based on the values of count, exclude, and
reverse.

105.1. Age-based sorting

For use cases where "like" items are being counted, and their name
pattern guarantees date sorting is equal to alphabetical sorting, it
is unnecessary to set use_age to True, as item names will be sorted
in reverse order by default. This means that the item count will
start beginning with the newest indices or snapshots, and proceed
through to the oldest.

Where this is not the case, the use_age setting can be used to ensure
that index or snapshot ages are properly considered for sorting:

- filtertype: count
  count: 10
  use_age: True
  source: creation_date

All of the age-related settings from the age filter are supported,
and the same restrictions apply with regard to filtering indices vs.
snapshots.

105.2. Pattern-based sorting

- filtertype: count
  count: 1
  pattern: '^(.*)-\d{6}$'
  reverse: true

This particular example will match indices following the basic
rollover pattern of indexname-######, and keep the highest numbered
index for each group.

For example, given indices a-000001, a-000002, a-000003 and b-000006,
and b-000007, the indices will would be matched are a-000003 and
b-000007. Indices that do not match the regular expression in pattern
will be automatically excluded.

This is particularly useful with indices created and managed using
the Rollover API, as you can select only the active indices with the
above example (exclude defaults to False). Setting exclude to True
with the above example will remove the active rollover indices,
leaving only those which have been rolled-over.

While this is perhaps most useful for the aforementioned scenario, it
can also be used with age-based indices as well.

105.3. Reversing sorting

Using the default configuration, reverse is True. Given These
indices:

index1
index2
index3
index4
index5

And this filter:

- filtertype: count
  count: 2

Indices index5 and index4 will be recognized as the 2 most recent,
and will be removed from the actionable list, leaving index1, index2,
and index3 to be acted on by the given action.

Similarly, given these indices:

index-2017.03.01
index-2017.03.02
index-2017.03.03
index-2017.03.04
index-2017.03.05

And this filter:

- filtertype: count
  count: 2
  use_age: True
  source: name
  timestring: '%Y.%m.%d'

The result will be similar. Indices index-2017.03.05 and
index-2017.03.04 will be recognized as the 2 most recent, and will be
removed from the actionable list, leaving index-2017.03.01,
index-2017.03.02, and index-2017.03.03 to be acted on by the given
action.

In some cases, you may wish to filter for the most recent indices. To
accomplish this you set reverse to False:

- filtertype: count
  count: 2
  reverse: False

This time indices index1 and index2 will be the 2 which will be
removed from the actionable list, leaving index3, index4, and index5
to be acted on by the given action.

Likewise with the age sorted indices:

- filtertype: count
  count: 2
  use_age: True
  source: name
  timestring: '%Y.%m.%d'
  reverse: True

Indices index-2017.03.01 and index-2017.03.02 will be the 2 which
will be removed from the actionable list, leaving index-2017.03.03,
index-2017.03.04, and index-2017.03.05 to be acted on by the given
action.

105.4. Required settings

  * count

105.5. Optional settings

  * reverse
  * use_age
  * pattern
  * source (required if use_age is True)
  * timestring (required if source is name)
  * exclude (default is True)

105.6. Index-only settings

  * field (required if source is field_stats)
  * stats_result (only used if source is field_stats)

Chapter 106. empty

- filtertype: empty
  exclude: False

This filtertype will iterate over the actionable list and match
indices which do not contain any documents. Indices that are closed
are automatically removed from consideration. They will remain in, or
be removed from the actionable list based on the value of exclude.

By default the indices matched by the empty filter will be excluded
since the exclude setting defaults to True. To include matching
indices rather than exclude, set the exclude setting to False.

106.1. Optional settings

  * exclude (default is True)

Chapter 107. forcemerged

- filtertype: forcemerged
  max_num_segments: 2
  exclude: True

Note

Empty values and commented lines will result in the default value, if
any, being selected. If a setting is set, but not used by a given
filtertype, it may generate an error.

This filtertype will iterate over the actionable list and match
indices which have max_num_segments segments per shard, or fewer.
They will remain in, or be removed from the actionable list based on
the value of exclude.

107.1. Required settings

  * max_num_segments

107.2. Optional settings

  * exclude (default is True)

Chapter 108. kibana

- filtertype: kibana
  exclude: True

This filtertype will remove any index matching the regular expression
^\.kibana.*$ from the list of indices, if present.

This filtertype will iterate over the actionable list and match
indices matching the regular expression ^\.kibana.*$. They will
remain in, or be removed from the actionable list based on the value
of exclude.

108.1. Optional settings

  * exclude (default is True)

Chapter 109. none

- filtertype: none

This filtertype will not filter anything, returning the full list of
indices or snapshots.

There are no settings for this filtertype.

Chapter 110. opened

- filtertype: opened
  exclude: True

This filtertype will iterate over the actionable list and match
indices which are opened. They will remain in, or be removed from the
actionable list based on the value of exclude.

110.1. Optional settings

  * exclude (default is True)

Chapter 111. pattern

- filtertype: pattern
 kind: ...
 value: ...

Note

Empty values and commented lines will result in the default value, if
any, being selected. If a setting is set, but not used by a given
filtertype, it may generate an error.

This filtertype will iterate over the actionable list and match
indices matching a given pattern. They will remain in, or be removed
from the actionable list based on the value of exclude.

Filter chaining

It is important to note that while filters can be chained, each is
linked by an implied logical AND operation. If you want to match from
one of several different patterns, as with a logical OR operation,
you can do so with the pattern filtertype using regex as the kind.

This example shows how to select multiple indices based on them
beginning with either alpha-, bravo-, or charlie-:

  filters:
  - filtertype: pattern
    kind: regex
    value: '^(alpha-|bravo-|charlie-).*$'

Explaining all of the different ways in which regular expressions can
be used is outside the scope of this document, but hopefully this
gives you some idea of how a regular expression pattern can be used
when a logical OR is desired.

The different kinds are described as follows:

111.1. prefix

To match all indices starting with logstash-:

- filtertype: pattern
 kind: prefix
 value: logstash-

To match all indices except those starting with logstash-:

- filtertype: pattern
 kind: prefix
 value: logstash-
 exclude: True

Note

Internally, the prefix value is used to create a regex pattern: ^
{0}.*$. Any special characters should be escaped with a backslash to
match literally.

111.2. suffix

To match all indices ending with -prod:

- filtertype: pattern
 kind: suffix
 value: -prod

To match all indices except those ending with -prod:

- filtertype: pattern
 kind: suffix
 value: -prod
 exclude: True

Note

Internally, the suffix value is used to create a regex pattern: ^.*
{0}$. Any special characters should be escaped with a backslash to
match literally.

111.3. timestring

Important

No age calculation takes place here. It is strictly a pattern match.

To match all indices with a Year.month.day pattern, like
index-2017.04.01:

- filtertype: pattern
 kind: timestring
 value: '%Y.%m.%d'

To match all indices except those with a Year.month.day pattern, like
index-2017.04.01:

- filtertype: pattern
 kind: timestring
 value: '%Y.%m.%d'
 exclude: True

A word about regular expression matching with timestrings

Timestrings are parsed from strftime patterns, like %Y.%m.%d, into
regular expressions. For example, %Y is 4 digits, so the regular
expression for that looks like \d{4}, and %m is 2 digits, so the
regular expression is \d{2}.

What this means is that a simple timestring to match year and month,
%Y.%m will result in a regular expression like this: ^.*\d{4}\.\d
{2}.*$. This pattern will match any 4 digits, followed by a period .,
followed by 2 digits, occurring anywhere in the index name. This
means it will match monthly indices, like index-2016.12, as well as
daily indices, like index-2017.04.01, which may not be the intended
behavior.

To compensate for this, when selecting indices matching a subset of
another pattern, use a second filter with exclude set to True

- filtertype: pattern
 kind: timestring
 value: '%Y.%m'
- filtertype: pattern
 kind: timestring
 value: '%Y.%m.%d'
 exclude: True

This will prevent the %Y.%m pattern from matching the %Y.%m part of
the daily indices.

This applies whether using timestring as a mere pattern match, or as
part of date calculations.

111.4. regex

This kind allows you to design a regular-expression to match indices
or snapshots:

To match all indices starting with a-, b-, or c-:

- filtertype: pattern
 kind: regex
 value: '^a-|^b-|^c-'

To match all indices except those starting with a-, b-, or c-:

- filtertype: pattern
 kind: regex
 value: '^a-|^b-|^c-'
 exclude: True

111.5. Required settings

  * kind
  * value

111.6. Optional settings

  * exclude (default is False)

Chapter 112. period

This filtertype will iterate over the actionable list and match
indices or snapshots based on whether they fit within the given time
range. They will remain in, or be removed from the actionable list
based on the value of exclude.

 - filtertype: period
   period_type: relative
   source: name
   range_from: -1
   range_to: -1
   timestring: '%Y.%m.%d'
   unit: weeks
   week_starts_on: sunday

Note

Empty values and commented lines will result in the default value, if
any, being selected. If a setting is set, but not used by a given
filtertype, it may generate an error.

112.1. Relative Periods

A relative period will be reckoned relative to execution time, unless
an epoch timestamp is provided. Reckoning is truncated to the most
recent whole unit, where a unit can be one of hours, days, weeks,
months, or years. For example, if I selected hours as my unit, and I
began execution at 02:35, then the point of reckoning would be 02:00.
This is relatively easy with days, months, and years, but slightly
more complicated with weeks. Some users may wish to reckon weeks by
the ISO standard, which starts weeks on Monday. Others may wish to
use Sunday as the first day of the week. Both are acceptable options
with the period filter. The default behavior for weeks is to have
Sunday be the start of the week. This can be overridden with
week_starts_on as follows:

 - filtertype: period
   period_type: relative
   source: name
   range_from: -1
   range_to: -1
   timestring: '%Y.%m.%d'
   unit: weeks
   week_starts_on: monday

range_from and range_to are counters of whole units. A negative
number indicates a whole unit in the past, while a positive number
indicates a whole unit in the future. A 0 indicates the present unit.
With such a timeline mentality, it is relatively easy to create a
date range that will meet your needs.

If the time of execution time is 2017-04-03T13:45:23.831, this table
will help you figure out what the previous whole unit, current unit,
and next whole unit will be, in ISO8601 format.

+-------------------------------------------------------------------+
|unit   |-1                 |0                  |+1                 |
|-------+-------------------+-------------------+-------------------|
|hours  |2017-04-03T12:00:00|2017-04-03T13:00:00|2017-04-03T14:00:00|
|-------+-------------------+-------------------+-------------------|
|days   |2017-04-02T00:00:00|2017-04-03T00:00:00|2017-04-04T00:00:00|
|-------+-------------------+-------------------+-------------------|
|weeks  |2017-03-26T00:00:00|2017-04-02T00:00:00|2017-04-09T00:00:00|
|sun    |                   |                   |                   |
|-------+-------------------+-------------------+-------------------|
|weeks  |2017-03-27T00:00:00|2017-04-03T00:00:00|2017-04-10T00:00:00|
|mon    |                   |                   |                   |
|-------+-------------------+-------------------+-------------------|
|months |2017-03-01T00:00:00|2017-04-01T00:00:00|2017-05-01T00:00:00|
|-------+-------------------+-------------------+-------------------|
|years  |2016-01-01T00:00:00|2017-01-01T00:00:00|2018-01-01T00:00:00|
+-------------------------------------------------------------------+

Ranges must be from older dates to newer dates, or smaller numbers
(including negative numbers) to larger numbers or Curator will return
an exception.

An example period filter demonstrating how to select all daily
indices by timestring found in the index name from last month might
look like this:

 - filtertype: period
   period_type: relative
   source: name
   range_from: -1
   range_to: -1
   timestring: '%Y.%m.%d'
   unit: months

Having range_from and range_to both be the same value will mean that
only that whole unit will be selected, in this case, a month.

Important

range_from and range_to are required for the relative pattern type.

112.2. Absolute Periods

 - filtertype: period
   period_type: absolute
   source: name
   timestring: '%Y.%m.%d'
   unit: months
   date_from: 2017.01
   date_from_format: '%Y.%m'
   date_to: 2017.01
   date_to_format: '%Y.%m'

In addition to relative periods, you can define absolute periods.
These are defined as the period between the date_from and the date_to
. For example, if date_from and date_to are both 2017.01, and unit is
months, all indices with a name, creation_date, or stats_result
(depending on the value of source) within the month of January 2017
will match.

The date_from is used to establish the beginning of the time period,
regardless of whether date_from_format is in hours, and the indices
you are trying to filter are in weeks or months. The format and date
of date_from will simply set the beginning of the time period.

The date_to, date_to_format, and unit work in conjunction to select
the end date. For example, if my date_to were 2017.04, and
date_to_format is %Y.%m to properly parse that date, it would follow
that unit would be months.

Important

The period filter is smart enough to calculate months and years
properly. If unit is not months or years, then your date range will
be unit seconds more than the beginning of the date_from date, minus
1 second, according to this table:

units are calculated as follows:

+--------------------------------+
|Unit   |Seconds |Note           |
|-------+--------+---------------|
|seconds|1       |One second     |
|-------+--------+---------------|
|minutes|60      |Calculated as  |
|       |        |60 seconds     |
|-------+--------+---------------|
|hours  |3600    |Calculated as  |
|       |        |60 minutes     |
|       |        |(60*60)        |
|-------+--------+---------------|
|days   |86400   |Calculated as  |
|       |        |24 hours       |
|       |        |(24*60*60)     |
|-------+--------+---------------|
|weeks  |604800  |Calculated as 7|
|       |        |days           |
|       |        |(7*24*60*60)   |
|-------+--------+---------------|
|months |2592000 |Calculated as  |
|       |        |30 days        |
|       |        |(30*24*60*60)  |
|-------+--------+---------------|
|years  |31536000|Calculated as  |
|       |        |365 days       |
|       |        |(365*24*60*60) |
+--------------------------------+

Specific date ranges can be captured with up to whole second
resolution:

 - filtertype: period
   period_type: absolute
   source: name
   timestring: '%Y.%m.%d.%H'
   unit: seconds
   date_from: 2017-01-01T00:00:00
   date_from_format: '%Y-%m-%dT%H:%M:%S'
   date_to: 2017-01-01T12:34:56
   date_to_format: '%Y-%m-%dT%H:%M:%S'

This example will capture indices with an hourly timestamp in their
name that fit between the date_from and date_to timestamps.

The identifiers that Curator currently recognizes include:

+--------------------------------+
|Unit|Value   |Note              |
|----+--------+------------------|
|%Y  |4 digit |                  |
|    |year    |                  |
|----+--------+------------------|
|%G  |4 digit |use instead of %Y |
|    |year    |when doing ISO    |
|    |        |Week calculations |
|----+--------+------------------|
|%y  |2 digit |                  |
|    |year    |                  |
|----+--------+------------------|
|%m  |2 digit |                  |
|    |month   |                  |
|----+--------+------------------|
|%W  |2 digit |                  |
|    |week of |                  |
|    |the year|                  |
|----+--------+------------------|
|%V  |2 digit |use instead of %W |
|    |week of |when doing ISO    |
|    |the year|Week calculations |
|----+--------+------------------|
|%d  |2 digit |                  |
|    |day of  |                  |
|    |the     |                  |
|    |month   |                  |
|----+--------+------------------|
|%H  |2 digit |24 hour notation  |
|    |hour    |                  |
|----+--------+------------------|
|%M  |2 digit |                  |
|    |minute  |                  |
|----+--------+------------------|
|%S  |2 digit |                  |
|    |second  |                  |
|----+--------+------------------|
|%j  |3 digit |                  |
|    |day of  |                  |
|    |the year|                  |
+--------------------------------+

112.3. Required settings

  * source
  * unit

112.4. Dependent settings

  * range_from
  * range_to
  * date_from
  * date_to
  * date_from_format
  * date_to_format
  * timestring (required if source is name)
  * field (required if source is field_stats) [Indices only]
  * stats_result (only used if source is field_stats) [Indices only]
  * intersect (optional if source is field_stats) [Indices only]

112.5. Optional settings

  * epoch
  * exclude (default is False)
  * week_starts_on

Chapter 113. space

This filtertype will iterate over the actionable list and match
indices when their cumulative disk consumption is greater_than
(default) or less_than than disk_space gigabytes. They are first
ordered by age, or by alphabet, so as to guarantee the oldest indices
are deleted first. They will remain in, or be removed from the
actionable list based on the value of exclude.

113.1. Deleting Indices By Space

- filtertype: space
  disk_space: 100

This filtertype is for those who want to retain indices based on disk
consumption, rather than by a set number of days. There are some
important caveats regarding this choice:

  * Elasticsearch cannot calculate the size of closed indices.
    Elasticsearch does not keep tabs on how much disk-space closed
    indices consume. If you close indices, your space calculations
    will be inaccurate.
  * Indices consume resources just by existing. You could run into
    performance and/or operational snags in Elasticsearch as the
    count of indices climbs.
  * You need to manually calculate how much space across all nodes.
    The total you give will be the sum of all space consumed across
    all nodes in your cluster. If you use shard allocation to put
    more shards or indices on a single node, it will not affect the
    total space reported by the cluster, but you may still run out of
    space on that node.

These are only a few of the caveats. This is still a valid use-case,
especially for those running a single-node test box.

For use cases where "like" indices are being counted, and their name
pattern guarantees date sorting is equal to alphabetical sorting, it
is unnecessary to set use_age to True, as index names will be sorted
in reverse order by default. For this case, this means that disk
space calculations will start beginning with the newest indices, and
proceeding through to the oldest.

113.2. Age-based sorting

- filtertype: space
  disk_space: 100
  use_age: True
  source: creation_date

For use cases where "like" indices are being counted, and their name
pattern guarantees date sorting is equal to alphabetical sorting, it
is unnecessary to set use_age to True, as index names will be sorted
in reverse order by default. For this case, this means that disk
space calculations will start beginning with the newest indices, and
proceeding through to the oldest.

Where this is not the case, the use_age setting can be used to ensure
that index or snapshot ages are properly considered for sorting:

All of the age-related settings from the age filter are supported.

113.3. Reversing sorting

Important

The reverse setting is ignored when use_age is True. When use_age is
True, sorting is always from newest to oldest, ensuring that old
indices are always selected first.

Using the default configuration, reverse is True. Given These
indices:

index1 10g
index2 10g
index3 10g
index4 10g
index5 10g

And this filter:

- filtertype: space
  disk_space: 21

The indices will be sorted alphabetically and iterated over in the
indicated order (the value of reverse) and the total disk space
compared after adding the size of each successive index. In this
example, that means that index5 will be added first, and the running
total of consumed disk space will be 10g. Since 10g is less than the
indicated threshold of 21, index5 will be removed from the actionable
list.

On the next iteration, the amount of space consumed by index4 will be
added, which brings the running total to 20g, which is still less
than the 21 threshold, so index4 is also removed from the actionable
list.

This process changes when the iteration adds the disk space consumed
by index3. Now the running total crosses the 21 threshold established
by disk_space (the running total is now 30g). Even though it is only
1g in excess of the total, index3 will remain in the actionable list.
The threshold is absolute.

The remaining indices, index2 and index1 will also be in excess of
the threshold, so they will also remain in the actionable list.

So in this example index1, index2, and index3 will be acted on by the
action for this block.

If you were to run this with loglevel set to DEBUG, you might see
messages like these in the output:

...Removed from actionable list: index5, summed disk usage is 10GB and disk limit is 21.0GB.
...Removed from actionable list: index4, summed disk usage is 20GB and disk limit is 21.0GB.
...Remains in actionable list: index3, summed disk usage is 30GB and disk limit is 21.0GB.
...Remains in actionable list: index2, summed disk usage is 40GB and disk limit is 21.0GB.
...Remains in actionable list: index1, summed disk usage is 50GB and disk limit is 21.0GB.

In some cases, you may wish to filter in the reverse order. To
accomplish this, you set reverse to False:

- filtertype: space
  disk_space: 21
  reverse: False

This time indices index1 and index2 will be the ones removed from the
actionable list, leaving index3, index4, and index5 to be acted on by
the given action.

If you were to run this with loglevel set to DEBUG, you might see
messages like these in the output:

...Removed from actionable list: index1, summed disk usage is 10GB and disk limit is 21.0GB.
...Removed from actionable list: index2, summed disk usage is 20GB and disk limit is 21.0GB.
...Remains in actionable list: index3, summed disk usage is 30GB and disk limit is 21.0GB.
...Remains in actionable list: index4, summed disk usage is 40GB and disk limit is 21.0GB.
...Remains in actionable list: index5, summed disk usage is 50GB and disk limit is 21.0GB.

113.4. Required settings

  * disk_space

113.5. Optional settings

  * reverse
  * use_age
  * source (required if use_age is True)
  * timestring (required if source is name)
  * threshold_behavior (default is greater_than)
  * field (required if source is field_stats)
  * stats_result (only used if source is field_stats)
  * exclude (default is False)

Chapter 114. state

- filtertype: state
  state: SUCCESS

Note

Empty values and commented lines will result in the default value, if
any, being selected. If a setting is set, but not used by a given
filtertype, it may generate an error.

This filtertype will iterate over the actionable list and match
snapshots based on the value of state. They will remain in, or be
removed from the actionable list based on the value of exclude.

114.1. Required settings

  * state

114.2. Optional settings

  * exclude (default is False)

Part X. Filter Elements

  * aliases
  * allocation_type
  * count
  * date_from
  * date_from_format
  * date_to
  * date_to_format
  * direction
  * disk_space
  * epoch
  * exclude
  * field
  * intersect
  * key
  * kind
  * max_num_segments
  * pattern
  * period_type
  * range_from
  * range_to
  * reverse
  * source
  * state
  * stats_result
  * timestring
  * threshold_behavior
  * unit
  * unit_count
  * unit_count_pattern
  * use_age
  * value
  * week_starts_on

You can use environment variables in your configuration files.

Table of Contents

115. aliases
116. allocation_type
117.
118. date_from
119. date_from_format
120. date_to
121. date_to_format
122. direction
123. disk_space
124. epoch
    124.1. Example
125. exclude
    125.1. Examples
126. field
127. intersect
128. key
129. kind
    129.1. prefix
    129.2. suffix
    129.3. timestring
    129.4. regex
130. max_num_segments
131. pattern
132. period_type
133. range_from
134. range_to
135. reverse
136. source
    136.1. name-based ages
    136.2. creation_date-based ages
    136.3. field_stats-based ages
137. state
138. stats_result
139. timestring
    139.1. strftime
140. threshold_behavior
141. unit
142. unit_count
143. unit_count_pattern
144. use_age
145. value
146. week_starts_on

Chapter 115. aliases

Matching Indices and Aliases

Indices must be in all aliases to match.

If a list of aliases is provided (instead of only one), indices must
appear in all listed aliases or a 404 error will result, leading to
no indices being matched. In older versions, if the index was
associated with even one of the aliases in aliases, it would result
in a match.

Note

This setting is used only when using the alias filter.

The value of this setting must be a single alias name, or a list of
alias names. This can be done in any of the ways YAML allows for
lists or arrays. Here are a few examples.

Single

filters:
- filtertype: alias
  aliases: my_alias
  exclude: False

List

  * Flow style:

    filters:
    - filtertype: alias
      aliases: [ my_alias, another_alias ]
      exclude: False

  * Block style:

    filters:
    - filtertype: alias
      aliases:
        - my_alias
        - another_alias
      exclude: False

There is no default value. This setting must be set by the user or an
exception will be raised, and execution will halt.

Chapter 116. allocation_type

Note

This setting is used only when using the allocated filter.

- filtertype: allocated
  key: ...
  value: ...
  allocation_type: require
  exclude: True

The value of this setting must be one of require, include, or
exclude.

Read more about these settings in the Elasticsearch documentation.

The default value for this setting is require.

Chapter 117. 

Note

This setting is only used with the count filtertype and is a required
setting.

- filtertype: count
  count: 10

The value for this setting is a number of indices or snapshots to
match.

Items will remain in the actionable list depending on the value of
exclude, and reverse.

There is no default value. This setting must be set by the user or an
exception will be raised, and execution will halt.

Chapter 118. date_from

Note

This setting is only used with the period filtertype when period_type
is absolute.

 - filtertype: period
   period_type: absolute
   source: name
   timestring: '%Y.%m.%d'
   unit: months
   date_from: 2017.01
   date_from_format: '%Y.%m'
   date_to: 2017.01
   date_to_format: '%Y.%m'

The value for this setting should be a date which can be easily
parsed using the strftime string in date_from_format:

The identifiers that Curator currently recognizes include:

+--------------------------------+
|Unit|Value   |Note              |
|----+--------+------------------|
|%Y  |4 digit |                  |
|    |year    |                  |
|----+--------+------------------|
|%G  |4 digit |use instead of %Y |
|    |year    |when doing ISO    |
|    |        |Week calculations |
|----+--------+------------------|
|%y  |2 digit |                  |
|    |year    |                  |
|----+--------+------------------|
|%m  |2 digit |                  |
|    |month   |                  |
|----+--------+------------------|
|%W  |2 digit |                  |
|    |week of |                  |
|    |the year|                  |
|----+--------+------------------|
|%V  |2 digit |use instead of %W |
|    |week of |when doing ISO    |
|    |the year|Week calculations |
|----+--------+------------------|
|%d  |2 digit |                  |
|    |day of  |                  |
|    |the     |                  |
|    |month   |                  |
|----+--------+------------------|
|%H  |2 digit |24 hour notation  |
|    |hour    |                  |
|----+--------+------------------|
|%M  |2 digit |                  |
|    |minute  |                  |
|----+--------+------------------|
|%S  |2 digit |                  |
|    |second  |                  |
|----+--------+------------------|
|%j  |3 digit |                  |
|    |day of  |                  |
|    |the year|                  |
+--------------------------------+

Chapter 119. date_from_format

Note

This setting is only used with the period filtertype when period_type
is absolute.

 - filtertype: period
   period_type: absolute
   source: name
   timestring: '%Y.%m.%d'
   unit: months
   date_from: 2017.01
   date_from_format: '%Y.%m'
   date_to: 2017.01
   date_to_format: '%Y.%m'

The value for this setting should be an strftime string which
corresponds to the date in date_from:

The identifiers that Curator currently recognizes include:

+--------------------------------+
|Unit|Value   |Note              |
|----+--------+------------------|
|%Y  |4 digit |                  |
|    |year    |                  |
|----+--------+------------------|
|%G  |4 digit |use instead of %Y |
|    |year    |when doing ISO    |
|    |        |Week calculations |
|----+--------+------------------|
|%y  |2 digit |                  |
|    |year    |                  |
|----+--------+------------------|
|%m  |2 digit |                  |
|    |month   |                  |
|----+--------+------------------|
|%W  |2 digit |                  |
|    |week of |                  |
|    |the year|                  |
|----+--------+------------------|
|%V  |2 digit |use instead of %W |
|    |week of |when doing ISO    |
|    |the year|Week calculations |
|----+--------+------------------|
|%d  |2 digit |                  |
|    |day of  |                  |
|    |the     |                  |
|    |month   |                  |
|----+--------+------------------|
|%H  |2 digit |24 hour notation  |
|    |hour    |                  |
|----+--------+------------------|
|%M  |2 digit |                  |
|    |minute  |                  |
|----+--------+------------------|
|%S  |2 digit |                  |
|    |second  |                  |
|----+--------+------------------|
|%j  |3 digit |                  |
|    |day of  |                  |
|    |the year|                  |
+--------------------------------+

Chapter 120. date_to

Note

This setting is only used with the period filtertype when period_type
is absolute.

 - filtertype: period
   period_type: absolute
   source: name
   timestring: '%Y.%m.%d'
   unit: months
   date_from: 2017.01
   date_from_format: '%Y.%m'
   date_to: 2017.01
   date_to_format: '%Y.%m'

The value for this setting should be a date which can be easily
parsed using the strftime string in date_to_format:

The identifiers that Curator currently recognizes include:

+--------------------------------+
|Unit|Value   |Note              |
|----+--------+------------------|
|%Y  |4 digit |                  |
|    |year    |                  |
|----+--------+------------------|
|%G  |4 digit |use instead of %Y |
|    |year    |when doing ISO    |
|    |        |Week calculations |
|----+--------+------------------|
|%y  |2 digit |                  |
|    |year    |                  |
|----+--------+------------------|
|%m  |2 digit |                  |
|    |month   |                  |
|----+--------+------------------|
|%W  |2 digit |                  |
|    |week of |                  |
|    |the year|                  |
|----+--------+------------------|
|%V  |2 digit |use instead of %W |
|    |week of |when doing ISO    |
|    |the year|Week calculations |
|----+--------+------------------|
|%d  |2 digit |                  |
|    |day of  |                  |
|    |the     |                  |
|    |month   |                  |
|----+--------+------------------|
|%H  |2 digit |24 hour notation  |
|    |hour    |                  |
|----+--------+------------------|
|%M  |2 digit |                  |
|    |minute  |                  |
|----+--------+------------------|
|%S  |2 digit |                  |
|    |second  |                  |
|----+--------+------------------|
|%j  |3 digit |                  |
|    |day of  |                  |
|    |the year|                  |
+--------------------------------+

Chapter 121. date_to_format

Note

This setting is only used with the period filtertype when period_type
is absolute.

 - filtertype: period
   period_type: absolute
   source: name
   timestring: '%Y.%m.%d'
   unit: months
   date_from: 2017.01
   date_from_format: '%Y.%m'
   date_to: 2017.01
   date_to_format: '%Y.%m'

The value for this setting should be an strftime string which
corresponds to the date in date_to:

The identifiers that Curator currently recognizes include:

+--------------------------------+
|Unit|Value   |Note              |
|----+--------+------------------|
|%Y  |4 digit |                  |
|    |year    |                  |
|----+--------+------------------|
|%G  |4 digit |use instead of %Y |
|    |year    |when doing ISO    |
|    |        |Week calculations |
|----+--------+------------------|
|%y  |2 digit |                  |
|    |year    |                  |
|----+--------+------------------|
|%m  |2 digit |                  |
|    |month   |                  |
|----+--------+------------------|
|%W  |2 digit |                  |
|    |week of |                  |
|    |the year|                  |
|----+--------+------------------|
|%V  |2 digit |use instead of %W |
|    |week of |when doing ISO    |
|    |the year|Week calculations |
|----+--------+------------------|
|%d  |2 digit |                  |
|    |day of  |                  |
|    |the     |                  |
|    |month   |                  |
|----+--------+------------------|
|%H  |2 digit |24 hour notation  |
|    |hour    |                  |
|----+--------+------------------|
|%M  |2 digit |                  |
|    |minute  |                  |
|----+--------+------------------|
|%S  |2 digit |                  |
|    |second  |                  |
|----+--------+------------------|
|%j  |3 digit |                  |
|    |day of  |                  |
|    |the year|                  |
+--------------------------------+

Chapter 122. direction

Note

This setting is only used with the age filtertype.

 - filtertype: age
   source: creation_date
   direction: older
   unit: days
   unit_count: 3

This setting must be either older or younger. This setting is used to
determine whether indices or snapshots are older or younger than the
reference point in time determined by unit, unit_count, and
optionally, epoch. If direction is older, then indices (or snapshots)
which are older than the reference point in time will be matched.
Likewise, if direction is younger, then indices (or snapshots) which
are younger than the reference point in time will be matched.

There is no default value. This setting must be set by the user or an
exception will be raised, and execution will halt.

Chapter 123. disk_space

Note

This setting is only used with the space filtertype and is a required
setting.

- filtertype: space
  disk_space: 100

The value for this setting is a number of gigabytes.

Indices in excess of this number of gigabytes will be matched.

There is no default value. This setting must be set by the user or an
exception will be raised, and execution will halt.

Chapter 124. epoch

Note

This setting is available in the age filtertype, and any filter which
has the use_age setting. This setting is strictly optional.

Tip

This setting is not common. It is most frequently used for testing.

unit, unit_count, and optionally, epoch, are used by Curator to
establish the moment in time point of reference with this formula:

point_of_reference = epoch - ((number of seconds in unit) * unit_count)

If epoch is unset, the current time is used. It is possible to set a
point of reference in the future by using a negative value for
unit_count.

124.1. Example

 - filtertype: age
   source: creation_date
   direction: older
   unit: days
   unit_count: 3
   epoch: 1491577200

The value for this setting must be an epoch timestamp. In this
example, the given epoch time of 1491577200 is 2017-04-04T15:00:00Z
(UTC). This will use 3 days older than that timestamp as the point of
reference for age comparisons.

Chapter 125. exclude

Note

This setting is available in all filter types.

If exclude is True, the filter will remove matches from the
actionable list. If exclude is False, then only matches will be kept
in the actionable list.

The default value for this setting is different for each filter type.

125.1. Examples

- filtertype: opened
  exclude: True

This filter will result in only closed indices being in the
actionable list.

- filtertype: opened
  exclude: False

This filter will result in only open indices being in the actionable
list.

Chapter 126. field

Note

This setting is available in the age filtertype, and any filter which
has the use_age setting. This setting is strictly optional.

 - filtertype: age
   source: field_stats
   direction: older
   unit: days
   unit_count: 3
   field: '@timestamp'
   stats_result: min_value

The value of this setting must be a timestamp field name. This field
must be present in the indices being filtered or an exception will be
raised, and execution will halt.

In Curator 5.3 and older, source field_stats uses the Field Stats API
to calculate either the min_value or the max_value of the field as
the stats_result, and then use that value for age comparisons. In 5.4
and above, even though it is still called field_stats, it uses an
aggregation to calculate the same values, as the field_stats API is
no longer used in Elasticsearch 6.x and up.

This setting is only used when source is field_stats.

The default value for this setting is @timestamp.

Chapter 127. intersect

Note

This setting is only available in the period filtertype. This setting
is strictly optional.

 - filtertype: period
   source: field_stats
   direction: older
   intersect: true
   unit: weeks
   range_from: -1
   range_to: -1
   field: '@timestamp'
   stats_result: min_value

The value of this setting must be True or False.

field_stats uses an aggregation query to calculate either the
min_value and the max_value of the field as the stats_result. If
intersect is True, then only indices where the min_value and the
max_value are within the range_from and range_to (relative to unit)
will match. This means that either min_value or max_value can be used
for stats_result when intersect is True with identical results.

This setting is only used when source is field_stats.

The default value for this setting is False.

Chapter 128. key

Note

This setting is required when using the allocated filtertype.

- filtertype: allocated
  key: ...
  value: ...
  allocation_type:
  exclude: True

The value of this setting should correspond to a node setting on one
or more nodes in your cluster.

For example, you might have set

node.tag: myvalue

in your elasticsearch.yml file for one or more of your nodes. To
match allocation in this case, set key to tag.

These special attributes are also supported:

+-------------------------------------------------------------------+
|attribute  |description                                            |
|-----------+-------------------------------------------------------|
|_name      |Match nodes by node name                               |
|-----------+-------------------------------------------------------|
|_host_ip   |Match nodes by host IP address (IP associated with     |
|           |hostname)                                              |
|-----------+-------------------------------------------------------|
|_publish_ip|Match nodes by publish IP address                      |
|-----------+-------------------------------------------------------|
|_ip        |Match either _host_ip or _publish_ip                   |
|-----------+-------------------------------------------------------|
|_host      |Match nodes by hostname                                |
+-------------------------------------------------------------------+

There is no default value. This setting must be set by the user or an
exception will be raised, and execution will halt.

Chapter 129. kind

Note

This setting is only used with the pattern filtertype and is a
required setting.

This setting tells the pattern what pattern type to match. Acceptable
values for this setting are prefix, suffix, timestring, and regex.

Filter chaining

It is important to note that while filters can be chained, each is
linked by an implied logical AND operation. If you want to match from
one of several different patterns, as with a logical OR operation,
you can do so with the pattern filtertype using regex as the kind.

This example shows how to select multiple indices based on them
beginning with either alpha-, bravo-, or charlie-:

  filters:
  - filtertype: pattern
    kind: regex
    value: '^(alpha-|bravo-|charlie-).*$'

Explaining all of the different ways in which regular expressions can
be used is outside the scope of this document, but hopefully this
gives you some idea of how a regular expression pattern can be used
when a logical OR is desired.

There is no default value. This setting must be set by the user or an
exception will be raised, and execution will halt.

The different kinds are described as follows:

129.1. prefix

To match all indices starting with logstash-:

- filtertype: pattern
 kind: prefix
 value: logstash-

To match all indices except those starting with logstash-:

- filtertype: pattern
 kind: prefix
 value: logstash-
 exclude: True

Note

Internally, the prefix value is used to create a regex pattern: ^
{0}.*$. Any special characters should be escaped with a backslash to
match literally.

129.2. suffix

To match all indices ending with -prod:

- filtertype: pattern
 kind: suffix
 value: -prod

To match all indices except those ending with -prod:

- filtertype: pattern
 kind: suffix
 value: -prod
 exclude: True

Note

Internally, the suffix value is used to create a regex pattern: ^.*
{0}$. Any special characters should be escaped with a backslash to
match literally.

129.3. timestring

Important

No age calculation takes place here. It is strictly a pattern match.

To match all indices with a Year.month.day pattern, like
index-2017.04.01:

- filtertype: pattern
 kind: timestring
 value: '%Y.%m.%d'

To match all indices except those with a Year.month.day pattern, like
index-2017.04.01:

- filtertype: pattern
 kind: timestring
 value: '%Y.%m.%d'
 exclude: True

A word about regular expression matching with timestrings

Timestrings are parsed from strftime patterns, like %Y.%m.%d, into
regular expressions. For example, %Y is 4 digits, so the regular
expression for that looks like \d{4}, and %m is 2 digits, so the
regular expression is \d{2}.

What this means is that a simple timestring to match year and month,
%Y.%m will result in a regular expression like this: ^.*\d{4}\.\d
{2}.*$. This pattern will match any 4 digits, followed by a period .,
followed by 2 digits, occurring anywhere in the index name. This
means it will match monthly indices, like index-2016.12, as well as
daily indices, like index-2017.04.01, which may not be the intended
behavior.

To compensate for this, when selecting indices matching a subset of
another pattern, use a second filter with exclude set to True

- filtertype: pattern
 kind: timestring
 value: '%Y.%m'
- filtertype: pattern
 kind: timestring
 value: '%Y.%m.%d'
 exclude: True

This will prevent the %Y.%m pattern from matching the %Y.%m part of
the daily indices.

This applies whether using timestring as a mere pattern match, or as
part of date calculations.

129.4. regex

This kind allows you to design a regular-expression to match indices
or snapshots:

To match all indices starting with a-, b-, or c-:

- filtertype: pattern
 kind: regex
 value: '^a-|^b-|^c-'

To match all indices except those starting with a-, b-, or c-:

- filtertype: pattern
 kind: regex
 value: '^a-|^b-|^c-'
 exclude: True

Chapter 130. max_num_segments

Note

This setting is only used with the forcemerged filtertype.

- filtertype: forcemerged
  max_num_segments: 2
  exclude: True

The value for this setting is the cutoff number of segments per
shard. Indices which have this number of segments per shard, or
fewer, will be actionable depending on the value of exclude, which is
True by default for the forcemerged filter type.

There is no default value. This setting must be set by the user or an
exception will be raised, and execution will halt.

Chapter 131. pattern

Note

This setting is only used with the count filtertype

- filtertype: count
  count: 1
  pattern: '^(.*)-\d{6}$'
  reverse: true

This particular example will match indices following the basic
rollover pattern of indexname-######, and keep the highest numbered
index for each group.

For example, given indices a-000001, a-000002, a-000003 and b-000006,
and b-000007, the indices will would be matched are a-000003 and
b-000007. Indices that do not match the regular expression in pattern
will be automatically excluded.

This is particularly useful with indices created and managed using
the Rollover API, as you can select only the active indices with the
above example (exclude defaults to False). Setting exclude to True
with the above example will remove the active rollover indices,
leaving only those which have been rolled-over.

While this is perhaps most useful for the aforementioned scenario, it
can also be used with age-based indices as well.

Items will remain in the actionable list depending on the value of
exclude, and reverse.

There is no default value. The value must include a capture group,
defined by parenthesis, or left empty. If a value is provided, and
there is no capture group, and exception will be raised and execution
will halt.

Chapter 132. period_type

Note

This setting is only used with the period filtertype

 - filtertype: period
   period_type: absolute
   source: name
   timestring: '%Y.%m.%d'
   unit: months
   date_from: 2017.01
   date_from_format: '%Y.%m'
   date_to: 2017.01
   date_to_format: '%Y.%m'

The value for this setting must be either relative or absolute. The
default value is relative.

Chapter 133. range_from

Note

This setting is only used with the period filtertype

 - filtertype: period
   source: name
   range_from: -1
   range_to: -1
   timestring: '%Y.%m.%d'
   unit: days

range_from and range_to are counters of whole units. A negative
number indicates a whole unit in the past, while a positive number
indicates a whole unit in the future. A 0 indicates the present unit.

Read more about this setting in context in the period filtertype
documentation, including examples.

Chapter 134. range_to

Note

This setting is only used with the period filtertype

 - filtertype: period
   source: name
   range_from: -1
   range_to: -1
   timestring: '%Y.%m.%d'
   unit: days

range_from and range_to are counters of whole units. A negative
number indicates a whole unit in the past, while a positive number
indicates a whole unit in the future. A 0 indicates the present unit.

Read more about this setting in context in the period filtertype
documentation, including examples.

Chapter 135. reverse

Note

This setting is used in the count and space filtertypes

This setting affects the sort order of the indices. True means
reverse-alphabetical. This means that if all index names share the
same pattern with a date—e.g. index-2016.03.01—older indices will be
selected first.

The default value of this setting is True.

This setting is ignored if use_age is True.

Tip

There are context-specific examples of how reverse works in the count
and space documentation.

Chapter 136. source

The source from which to derive the index or snapshot age. Can be one
of name, creation_date, or field_stats.

Note

This setting is only used with the age filtertype, or with the space
filtertype when use_age is set to True.

Note

When using the age filtertype, source requires direction, unit,
unit_count, and additionally, the optional setting, epoch.

136.1. name-based ages

Using name as the source tells Curator to look for a timestring
within the index or snapshot name, and convert that into an epoch
timestamp (epoch implies UTC).

 - filtertype: age
   source: name
   direction: older
   timestring: '%Y.%m.%d'
   unit: days
   unit_count: 3

A word about regular expression matching with timestrings

Timestrings are parsed from strftime patterns, like %Y.%m.%d, into
regular expressions. For example, %Y is 4 digits, so the regular
expression for that looks like \d{4}, and %m is 2 digits, so the
regular expression is \d{2}.

What this means is that a simple timestring to match year and month,
%Y.%m will result in a regular expression like this: ^.*\d{4}\.\d
{2}.*$. This pattern will match any 4 digits, followed by a period .,
followed by 2 digits, occurring anywhere in the index name. This
means it will match monthly indices, like index-2016.12, as well as
daily indices, like index-2017.04.01, which may not be the intended
behavior.

To compensate for this, when selecting indices matching a subset of
another pattern, use a second filter with exclude set to True

- filtertype: pattern
 kind: timestring
 value: '%Y.%m'
- filtertype: pattern
 kind: timestring
 value: '%Y.%m.%d'
 exclude: True

This will prevent the %Y.%m pattern from matching the %Y.%m part of
the daily indices.

This applies whether using timestring as a mere pattern match, or as
part of date calculations.

136.2. creation_date-based ages

creation_date extracts the epoch time of index or snapshot creation.

 - filtertype: age
   source: creation_date
   direction: older
   unit: days
   unit_count: 3

136.3. field_stats-based ages

Note

source can only be field_stats when filtering indices.

In Curator 5.3 and older, source field_stats uses the Field Stats API
to calculate either the min_value or the max_value of the field as
the stats_result, and then use that value for age comparisons. In 5.4
and above, even though it is still called field_stats, it uses an
aggregation to calculate the same values, as the field_stats API is
no longer used in Elasticsearch 6.x and up.

field must be of type date in Elasticsearch.

 - filtertype: age
   source: field_stats
   direction: older
   unit: days
   unit_count: 3
   field: '@timestamp'
   stats_result: min_value

Chapter 137. state

Note

This setting is only used with the state filtertype.

- filtertype: state
  state: SUCCESS

The value for this setting must be one of SUCCESS, PARTIAL, FAILED,
or IN_PROGRESS. This setting determines what kind of snapshots will
be passed.

The default value for this setting is SUCCESS.

Chapter 138. stats_result

Note

This setting is only used with the age filtertype.

 - filtertype: age
   source: field_stats
   direction: older
   unit: days
   unit_count: 3
   field: '@timestamp'
   stats_result: min_value

The value for this setting can be either min_value or max_value. This
setting is only used when source is field_stats, and determines
whether Curator will use the minimum or maximum value of field for
time calculations.

The default value for this setting is min_value.

Chapter 139. timestring

Note

This setting is only used with the age filtertype, or with the space
filtertype if use_age is set to True.

139.1. strftime

This setting must be a valid Python strftime string. It is used to
match and extract the timestamp in an index or snapshot name.

The identifiers that Curator currently recognizes include:

+--------------------------------+
|Unit|Value   |Note              |
|----+--------+------------------|
|%Y  |4 digit |                  |
|    |year    |                  |
|----+--------+------------------|
|%G  |4 digit |use instead of %Y |
|    |year    |when doing ISO    |
|    |        |Week calculations |
|----+--------+------------------|
|%y  |2 digit |                  |
|    |year    |                  |
|----+--------+------------------|
|%m  |2 digit |                  |
|    |month   |                  |
|----+--------+------------------|
|%W  |2 digit |                  |
|    |week of |                  |
|    |the year|                  |
|----+--------+------------------|
|%V  |2 digit |use instead of %W |
|    |week of |when doing ISO    |
|    |the year|Week calculations |
|----+--------+------------------|
|%d  |2 digit |                  |
|    |day of  |                  |
|    |the     |                  |
|    |month   |                  |
|----+--------+------------------|
|%H  |2 digit |24 hour notation  |
|    |hour    |                  |
|----+--------+------------------|
|%M  |2 digit |                  |
|    |minute  |                  |
|----+--------+------------------|
|%S  |2 digit |                  |
|    |second  |                  |
|----+--------+------------------|
|%j  |3 digit |                  |
|    |day of  |                  |
|    |the year|                  |
+--------------------------------+

These identifiers may be combined with each other, and/or separated
from each other with hyphens -, periods ., underscores _, or other
characters valid in an index name.

Each identifier must be preceded by a % character in the timestring.
For example, an index like index-2016.04.01 would use a timestring of
'%Y.%m.%d'.

When source is name, this setting must be set by the user or an
exception will be raised, and execution will halt. There is no
default value.

A word about regular expression matching with timestrings

Timestrings are parsed from strftime patterns, like %Y.%m.%d, into
regular expressions. For example, %Y is 4 digits, so the regular
expression for that looks like \d{4}, and %m is 2 digits, so the
regular expression is \d{2}.

What this means is that a simple timestring to match year and month,
%Y.%m will result in a regular expression like this: ^.*\d{4}\.\d
{2}.*$. This pattern will match any 4 digits, followed by a period .,
followed by 2 digits, occurring anywhere in the index name. This
means it will match monthly indices, like index-2016.12, as well as
daily indices, like index-2017.04.01, which may not be the intended
behavior.

To compensate for this, when selecting indices matching a subset of
another pattern, use a second filter with exclude set to True

- filtertype: pattern
 kind: timestring
 value: '%Y.%m'
- filtertype: pattern
 kind: timestring
 value: '%Y.%m.%d'
 exclude: True

This will prevent the %Y.%m pattern from matching the %Y.%m part of
the daily indices.

This applies whether using timestring as a mere pattern match, or as
part of date calculations.

Chapter 140. threshold_behavior

Note

This setting is only available in the space filtertype. This setting
is optional, and defaults to greater_than to preserve backwards
compatability.

- filtertype: space
  disk_space: 5
  threshold_behavior: less_than

The value for this setting is greater_than (default) or less_than.

When set to less_than, indices in less than disk_space gigabytes will
be matched. When set to greater_than (default), indices larger than
disk_space will be matched.

Chapter 141. unit

Note

This setting is used with the age filtertype, with the period
filtertype, or with the space filtertype if use_age is set to True.

 - filtertype: age
   source: creation_date
   direction: older
   unit: days
   unit_count: 3

This setting must be one of seconds, minutes, hours, days, weeks,
months, or years. The values seconds and minutes are not allowed with
the period filtertype and will result in an error condition if used
there.

For the age filtertype, or when use_age is set to True, unit,
unit_count, and optionally, epoch, are used by Curator to establish
the moment in time point of reference with this formula:

point_of_reference = epoch - ((number of seconds in unit) * unit_count)

units are calculated as follows:

+--------------------------------+
|Unit   |Seconds |Note           |
|-------+--------+---------------|
|seconds|1       |One second     |
|-------+--------+---------------|
|minutes|60      |Calculated as  |
|       |        |60 seconds     |
|-------+--------+---------------|
|hours  |3600    |Calculated as  |
|       |        |60 minutes     |
|       |        |(60*60)        |
|-------+--------+---------------|
|days   |86400   |Calculated as  |
|       |        |24 hours       |
|       |        |(24*60*60)     |
|-------+--------+---------------|
|weeks  |604800  |Calculated as 7|
|       |        |days           |
|       |        |(7*24*60*60)   |
|-------+--------+---------------|
|months |2592000 |Calculated as  |
|       |        |30 days        |
|       |        |(30*24*60*60)  |
|-------+--------+---------------|
|years  |31536000|Calculated as  |
|       |        |365 days       |
|       |        |(365*24*60*60) |
+--------------------------------+

If epoch is unset, the current time is used. It is possible to set a
point of reference in the future by using a negative value for
unit_count.

This setting must be set by the user or an exception will be raised,
and execution will halt.

Tip

See the age filter documentation for more information about time
calculation.

Chapter 142. unit_count

Note

This setting is only used with the age filtertype, or with the space
filtertype if use_age is set to True.

 - filtertype: age
   source: creation_date
   direction: older
   unit: days
   unit_count: 3

The value of this setting will be used as a multiplier for unit.

unit, unit_count, and optionally, epoch, are used by Curator to
establish the moment in time point of reference with this formula:

point_of_reference = epoch - ((number of seconds in unit) * unit_count)

units are calculated as follows:

+--------------------------------+
|Unit   |Seconds |Note           |
|-------+--------+---------------|
|seconds|1       |One second     |
|-------+--------+---------------|
|minutes|60      |Calculated as  |
|       |        |60 seconds     |
|-------+--------+---------------|
|hours  |3600    |Calculated as  |
|       |        |60 minutes     |
|       |        |(60*60)        |
|-------+--------+---------------|
|days   |86400   |Calculated as  |
|       |        |24 hours       |
|       |        |(24*60*60)     |
|-------+--------+---------------|
|weeks  |604800  |Calculated as 7|
|       |        |days           |
|       |        |(7*24*60*60)   |
|-------+--------+---------------|
|months |2592000 |Calculated as  |
|       |        |30 days        |
|       |        |(30*24*60*60)  |
|-------+--------+---------------|
|years  |31536000|Calculated as  |
|       |        |365 days       |
|       |        |(365*24*60*60) |
+--------------------------------+

If epoch is unset, the current time is used. It is possible to set a
point of reference in the future by using a negative value for
unit_count.

This setting must be set by the user or an exception will be raised,
and execution will halt.

If this setting is used in conjunction with unit_count_pattern, the
configured value will only be used as a fallback value in case the
pattern could not be matched. The value -1 has a special meaning in
this case and causes the index to be ignored when pattern matching
fails.

Tip

See the age filter documentation for more information about time
calculation.

Chapter 143. unit_count_pattern

Note

This setting is only used with the age filtertype to define, whether
the unit_count value is taken from the configuration or read from the
index name via a regular expression.

 - filtertype: age
   source: creation_date
   direction: older
   unit: days
   unit_count: 3
   unit_count_pattern: -([0-9]+)-

This setting can be used in cases where the value against which index
age should be assessed is not a static value but can be different for
every index. For this case, there is the option of extracting the
index specific value from the index names via a regular expression
defined in this parameter.

Consider for example the following index name patterns that contain
the retention time in their name: logstash-30-yyyy.mm.dd, 
logstash-12-yyyy.mm, _3_logstash-yyyy.mm.dd.

To extract a value from the index names, this setting will be
compiled as a regular expression and matched against index names, for
a successful match, the value of the first capture group from the
regular expression is used as the value for unit_count.

If there is any error during compiling or matching the expression, or
the expression does not contain a capture group, the value configured
in unit_count is used as a fallback value, unless it is set to -1, in
which case the index will be skipped.

Tip

Regular expressions and match groups are not explained here as they
are a fairly large and complex topic, but there are numerous
resources online that will help. Using an online tool for testing
regular expressions like regex101.com will help a lot when developing
patterns.

Examples

  * logstash-30-yyyy.mm.dd: Daily index that should be deleted after
    30 days, indices that don’t match the pattern will be deleted
    after 365 days

 - filtertype: age
   source: creation_date
   direction: older
   unit: days
   unit_count: 365
   unit_count_pattern: -([0-9]+)-

  * logstash-12-yyyy.mm: Monthly index that should be deleted after
    12 months, indices that don’t match the pattern will be deleted
    after 3 months

 - filtertype: age
   source: creation_date
   direction: older
   unit: months
   unit_count: 3
   unit_count_pattern: -([0-9]+)-

  * _3_logstash-yyyy.mm.dd: Daily index that should be deleted after
    3 years, indices that don’t match the pattern will be ignored

 - filtertype: age
   source: creation_date
   direction: older
   unit: years
   unit_count: -1
   unit_count_pattern: ^_([0-9]+)_

Important

Be sure to pay attention to the interaction of this parameter and
unit_count!

Chapter 144. use_age

- filtertype: count
  count: 10
  use_age: True
  source: creation_date

This setting allows filtering of indices by their age after other
considerations.

The default value of this setting is False

Note

Use of this setting requires the additional setting, source.

Tip

There are context-specific examples using use_age in the count and
space documentation.

Chapter 145. value

Note

This setting is only used with the pattern filtertype and is a
required setting. There is a separate value option associated with
the allocation action, and the allocated filtertype.

The value of this setting is used by kind as follows:

  * prefix: Search the first part of an index name for the provided
    value
  * suffix: Search the last part of an index name for the provided
    value
  * regex: Provide your own regular expression, and Curator will find
    the matches.
  * timestring: An strftime string to extrapolate and find indices
    that match. For example, given a timestring of '%Y.%m.%d',
    matching indices would include logstash-2016.04.01 and
    .marvel-2016.04.01, but not myindex-2016-04-01, as the pattern is
    different.

Important

Whatever you provide for value is always going to be a part of a
regular expression. The safest practice is to always encapsulate
within single quotes. For example: value: '-suffix', or value:
'prefix-'

There is no default value. This setting must be set by the user or an
exception will be raised, and execution will halt.

Tip

There are context-specific examples using value in the kind
documentation.

Chapter 146. week_starts_on

Note

This setting is only used with the period filtertype when period_type
is relative.

 - filtertype: period
   source: name
   range_from: -1
   range_to: -1
   timestring: '%Y.%m.%d'
   unit: weeks
   week_starts_on: sunday

The value of this setting indicates whether weeks should be measured
starting on sunday or monday. Though Monday is the ISO standard,
Sunday is frequently preferred.

This setting is only used when unit is set to weeks.

The default value for this setting is sunday.

Part XI. Examples

These examples should help illustrate how to build your own actions.

You can use environment variables in your configuration files.

  * alias
  * allocation
  * close
  * cluster_routing
  * create_index
  * delete_indices
  * delete_snapshots
  * forcemerge
  * open
  * reindex
  * replicas
  * restore
  * rollover
  * shrink
  * snapshot

Table of Contents

147. alias
148. allocation
149. close
150. cluster_routing
151. create_index
152. delete_indices
153. delete_snapshots
154. forcemerge
155. index_settings
156. open
157. reindex
    157.1. Manually selected reindex of a single index
    157.2. Manually selected reindex of a multiple indices
    157.3. Filter-Selected Indices
    157.4. Reindex From Remote
    157.5. Reindex From Remote With Filter-Selected Indices
    157.6. Manually selected reindex of a single index with query
158. replicas
159. restore
160. rollover
161. shrink
162. snapshot

Chapter 147. alias

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: alias
    description: >-
      Alias indices from last week, with a prefix of logstash- to 'last_week',
      remove indices from the previous week.
    options:
      name: last_week
      warn_if_no_indices: False
      disable_action: True
    add:
      filters:
      - filtertype: pattern
        kind: prefix
        value: logstash-
        exclude:
      - filtertype: period
        period_type: relative
        source: name
        range_from: -1
        range_to: -1
        timestring: '%Y.%m.%d'
        unit: weeks
        week_starts_on: sunday
    remove:
      filters:
      - filtertype: pattern
        kind: prefix
        value: logstash-
      - filtertype: period
        period_type: relative
        source: name
        range_from: -2
        range_to: -2
        timestring: '%Y.%m.%d'
        unit: weeks
        week_starts_on: sunday

Chapter 148. allocation

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: allocation
    description: >-
      Apply shard allocation routing to 'require' 'tag=cold' for hot/cold node
      setup for logstash- indices older than 3 days, based on index_creation
      date
    options:
      key: tag
      value: cold
      allocation_type: require
      disable_action: True
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 3

Chapter 149. close

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: close
    description: >-
      Close indices older than 30 days (based on index name), for logstash-
      prefixed indices.
    options:
      skip_flush: False
      delete_aliases: False
      ignore_sync_failures: True
      disable_action: True
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 30

Chapter 150. cluster_routing

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
#
# This action example has a blank spot at action ID 2.  This is to show that
# Curator can disable allocation before one or more actions, and then re-enable
# it afterward.
actions:
  1:
    action: cluster_routing
    description: >-
      Disable shard routing for the entire cluster.
    options:
      routing_type: allocation
      value: none
      setting: enable
      wait_for_completion: True
      disable_action: True
  2:
    action: (any other action details go here)
    ...
  3:
    action: cluster_routing
    description: >-
      Re-enable shard routing for the entire cluster.
    options:
      routing_type: allocation
      value: all
      setting: enable
      wait_for_completion: True
      disable_action: True

Chapter 151. create_index

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: create_index
    description: Create the index as named, with the specified extra settings.
    options:
      name: myindex
      extra_settings:
        settings:
          number_of_shards: 2
          number_of_replicas: 1
      disable_action: True

Chapter 152. delete_indices

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 45 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      disable_action: True
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 45

Chapter 153. delete_snapshots

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: delete_snapshots
    description: >-
      Delete snapshots from the selected repository older than 45 days
      (based on creation_date), for 'curator-' prefixed snapshots.
    options:
      repository:
      disable_action: True
    filters:
    - filtertype: pattern
      kind: prefix
      value: curator-
      exclude:
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 45

Chapter 154. forcemerge

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: forcemerge
    description: >-
      forceMerge logstash- prefixed indices older than 2 days (based on index
      creation_date) to 2 segments per shard.  Delay 120 seconds between each
      forceMerge operation to allow the cluster to quiesce. Skip indices that
      have already been forcemerged to the minimum number of segments to avoid
      reprocessing.
    options:
      max_num_segments: 2
      delay: 120
      timeout_override:
      continue_if_exception: False
      disable_action: True
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
      exclude:
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 2
      exclude:
    - filtertype: forcemerged
      max_num_segments: 2
      exclude:

Chapter 155. index_settings

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: index_settings
    description: >-
      Set Logstash indices older than 10 days to be read only (block writes)
    options:
      disable_action: True
      index_settings:
        index:
          blocks:
            write: True
      ignore_unavailable: False
      preserve_existing: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 10

Chapter 156. open

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: open
    description: >-
      Open indices older than 30 days but younger than 60 days (based on index
      name), for logstash- prefixed indices.
    options:
      disable_action: True
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 30
    - filtertype: age
      source: name
      direction: younger
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 60

Chapter 157. reindex

157.1. Manually selected reindex of a single index

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    description: "Reindex index1 into index2"
    action: reindex
    options:
      disable_action: True
      wait_interval: 9
      max_wait: -1
      request_body:
        source:
          index: index1
        dest:
          index: index2
    filters:
    - filtertype: none

157.2. Manually selected reindex of a multiple indices

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    description: "Reindex index1,index2,index3 into new_index"
    action: reindex
    options:
      disable_action: True
      wait_interval: 9
      max_wait: -1
      request_body:
        source:
          index: ['index1', 'index2', 'index3']
        dest:
          index: new_index
    filters:
    - filtertype: none

157.3. Filter-Selected Indices

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    description: >-
      'Reindex all daily logstash indices from March 2017 into logstash-2017.03'
    action: reindex
    options:
      disable_action: True
      wait_interval: 9
      max_wait: -1
      request_body:
        source:
          index: REINDEX_SELECTION
        dest:
          index: logstash-2017.03
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-2017.03.

157.4. Reindex From Remote

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    description: >-
      'Reindex all daily logstash indices from March 2017 into logstash-2017.03'
    action: reindex
    options:
      disable_action: True
      wait_interval: 9
      max_wait: -1
      request_body:
        source:
          remote:
            host: http://otherhost:9200
            username: myuser
            password: mypass
          index: index1
        dest:
          index: index1
    filters:
    - filtertype: none

157.5. Reindex From Remote With Filter-Selected Indices

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    description: >-
      Reindex all remote daily logstash indices from March 2017 into local index
      logstash-2017.03
    action: reindex
    options:
      disable_action: True
      wait_interval: 9
      max_wait: -1
      request_body:
        source:
          remote:
            host: http://otherhost:9200
            username: myuser
            password: mypass
          index: REINDEX_SELECTION
        dest:
          index: logstash-2017.03
      remote_filters:
      - filtertype: pattern
      kind: prefix
      value: logstash-2017.03.
    filters:
    - filtertype: none

157.6. Manually selected reindex of a single index with query

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    description: "Reindex index1 into index2"
    action: reindex
    options:
      disable_action: True
      wait_interval: 9
      max_wait: -1
      request_body:
        source:
          query:
            range:
              timestamp:
                gte: "now-1h"
          index: index1
        dest:
          index: index2
    filters:
    - filtertype: none

Chapter 158. replicas

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: replicas
    description: >-
      Reduce the replica count to 0 for logstash- prefixed indices older than
      10 days (based on index creation_date)
    options:
      count: 0
      wait_for_completion: True
      disable_action: True
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 10

Chapter 159. restore

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: restore
    description: >-
      Restore all indices in the most recent curator-* snapshot with state
      SUCCESS.  Wait for the restore to complete before continuing.  Do not skip
      the repository filesystem access check.  Use the other options to define
      the index/shard settings for the restore.
    options:
      repository:
      # If name is blank, the most recent snapshot by age will be selected
      name:
      # If indices is blank, all indices in the snapshot will be restored
      indices:
      include_aliases: False
      ignore_unavailable: False
      include_global_state: False
      partial: False
      rename_pattern:
      rename_replacement:
      extra_settings:
      wait_for_completion: True
      skip_repo_fs_check: True
      disable_action: True
    filters:
    - filtertype: pattern
      kind: prefix
      value: curator-
    - filtertype: state
      state: SUCCESS

Chapter 160. rollover

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: rollover
    description: >-
      Rollover the index associated with alias 'aliasname', which should be in the
      format of prefix-000001 (or similar), or prefix-YYYY.MM.DD-1.
    options:
      disable_action: True
      name: aliasname
      conditions:
        max_age: 1d
        max_docs: 1000000
        max_size: 50g
      extra_settings:
        index.number_of_shards: 3
        index.number_of_replicas: 1

Chapter 161. shrink

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: shrink
    description: >-
      Shrink logstash indices older than 21 days on the node with the most
      available space, excluding the node named 'not_this_node'.
      Delete each source index after successful shrink, then reroute the shrunk
      index with the provided parameters.
    options:
      disable_action: True
      ignore_empty_list: True
      shrink_node: DETERMINISTIC
      node_filters:
        permit_masters: False
        exclude_nodes: ['not_this_node']
      number_of_shards: 1
      number_of_replicas: 1
      shrink_prefix:
      shrink_suffix: '-shrink'
      delete_after: True
      post_allocation:
        allocation_type: include
        key: node_tag
        value: cold
      wait_for_active_shards: 1
      extra_settings:
        settings:
          index.codec: best_compression
      wait_for_completion: True
      wait_for_rebalance: True
      wait_interval: 9
      max_wait: -1
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 21

Chapter 162. snapshot

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: snapshot
    description: >-
      Snapshot logstash- prefixed indices older than 1 day (based on index
      creation_date) with the default snapshot name pattern of
      'curator-%Y%m%d%H%M%S'.  Wait for the snapshot to complete.  Do not skip
      the repository filesystem access check.  Use the other options to create
      the snapshot.
    options:
      repository:
      # Leaving name blank will result in the default 'curator-%Y%m%d%H%M%S'
      name:
      ignore_unavailable: False
      include_global_state: True
      partial: False
      wait_for_completion: True
      skip_repo_fs_check: False
      disable_action: True
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 1

Part XII. Frequently Asked Questions

This section will be updated as more frequently asked questions arise

  * How can I report an error in the documentation?
  * Can I delete only certain data from within indices?
  * Can Curator handle index names with strange characters?

Table of Contents

163. Q: How can I report an error in the documentation?
    163.1. A: Use the "Edit" link on any page
164. Q: Can I delete only certain data from within indices?
    164.1. A: It’s complicated
165. Q: Can Curator handle index names with strange characters?
    165.1. A: Yes!

Chapter 163. Q: How can I report an error in the documentation?

163.1. A: Use the "Edit" link on any page

See Site Corrections.

Chapter 164. Q: Can I delete only certain data from within indices?

164.1. A: It’s complicated

TL;DR: No. Curator can only delete entire indices.

Full answer:

As a thought exercise, think of Elasticsearch indices as being like
databases, or tablespaces within a database. If you had hundreds of
millions of rows to delete from your database, would you run a
separate DELETE from TABLE where date<YYYY.MM.dd to assemble hundreds
of millions of individual delete operations every day, or would you
partition your tables in a way that you could simply run DROP table
TABLENAME.YYYY.MM.dd? The strain on your database would be
astronomical on the former and next to nothing on the latter.
Elasticsearch works much the same way. While Elasticsearch can
technically do both methods, for use-cases with time-series data
(like logging), we recommend dropping entire indices vs. the
extremely I/O expensive search and delete method. Curator was created
to help fill that need.

While you can store different types within different indices (e.g.
syslog-2014.05.05, apache-2015.05.06), this gets very expensive, very
quickly in a totally different way. Each shard in Elasticsearch is a
Lucene index. Each index requires a portion of the heap to exist and
be kept current. If you have 3 daily indices with 5 primary shards
each, you suddenly have reduced the available heap space for shard
management by a factor of 3, having gone from 5 shards to 15, per
index, not counting multiple indexes per day. The ways to mitigate
this (if you pursue this route) include massive daily indexing boxes
and using shard allocation/routing to move indices to specific
members of the cluster where they can have less effect; keeping fewer
days of information; having more nodes in your cluster, and so forth.

Conclusion:

While it may be desirable to have different life-cycles for your
data, sometimes it’s just easier and cheaper to store everything as
long as the longest life-cycle you wish to maintain.

Post-script:

Even though it is neither recommended ^[1], nor best practices, it is
still possible to perform these search & delete operations yourself,
using the Delete-by-Query API. Curator will not be modified to
perform operations such as these, however. Curator is meant to manage
at the index level, rather than the data level.


---------------------------------------------------------------------

^[1] There are reasons Elasticsearch does not recommend this,
particularly for time-series data. For more information read http://
blog.mikemccandless.com/2011/02/
visualizing-lucenes-segment-merges.html and watch what happens to
your segments when you delete data.

Chapter 165. Q: Can Curator handle index names with strange
characters?

165.1. A: Yes!

This problem can be resolved by using the pattern filtertype with
kind set to regex, and value set to the needed regular expression.

The Problem:

Illegal characters make it hard to delete indices.

% curl logs.example.com:9200/_cat/indices
red    }?ebc-2015.04.08.03
                          sip-request{ 5 1         0  0     632b     316b
red    }?ebc-2015.04.08.03
                          sip-response 5 1         0  0     474b     237b
red    ?ebc-2015.04.08.02
                         sip-request{ 5 1         0  0     474b     316b
red
eb                               5 1         0  0     632b     316b
red    ?e                                5 1         0  0     632b     316b

 

You can see it looks like there are some tab characters and maybe
newline characters. This makes it hard to use the HTTP API to delete
the indices.

Dumping all the index settings out:

curl -XGET localhost:9200/*/_settings?pretty

 

…reveals the index names as the first key in the resulting JSON. In
this case, the names were very atypical:

}\b?\u0011ebc-2015.04.08.02\u000Bsip-request{
}\u0006?\u0011ebc-2015.04.08.03\u000Bsip-request{
}\u0003?\u0011ebc-2015.04.08.03\fsip-response
...

 

Curator lets you use regular expressions to select indices to perform
actions on.

Warning

Before attempting an action, see what will be affected by using the
--dry-run flag first.

To delete the first three from the above example, use '.*sip.*' as
your regular expression.

Note

In an actionfile, regular expressions and strftime date strings must
be encapsulated in single-quotes.

The next one is trickier. The real name of the index was \n\u0011eb.
The regular expression .*b$ did not work, but '\n.*' did.

The last index can be deleted with a regular expression of '.*e$'.

The resulting actionfile might look like this:

actions:
  1:
    description: Delete indices with strange characters that match regex '.*sip.*'
    action: delete_indices
    options:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: regex
      value: '.*sip.*'
  2:
    description: Delete indices with strange characters that match regex '\n.*'
    action: delete_indices
    options:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: regex
      value: '\n.*'
  3:
    description: Delete indices with strange characters that match regex '.*e$'
    action: delete_indices
    options:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: regex
      value: '.*e$'

