# OS-Faults + Rally

Combination of OS-Faults and Rally gives a powerful tool to test OpenStack high availability and fail-over under the load.

Fault injection is implemented with help of Rally Fault Injection Hook. Following is an example of Rally scenario performing Keystone authentication with restart of one of Memcached services:

    ---
      Authenticate.keystone:
        -
          runner:
            type: "constant_for_duration"
            duration: 30
            concurrency: 5
          context:
            users:
              tenants: 1
              users_per_tenant: 1
          hooks:
            -
              name: fault_injection
              args:
                action: restart memcached service on one node
              trigger:
                name: event
                args:
                  unit: iteration
                  at: [100]

The moment of fault injection can be specified as iteration number or in time relative to the beginning of the test:

    trigger:
      name: event
      args:
        unit: time
        at: [10]

Parameter action contains fault specification in human-friendly format, see Human API for details.

More on reliability testing of OpenStack:

> - Reliability Test Plan in OpenStack performance documentation
> - Keystone authentication with restart memcached report collected in OpenStack deployed by Fuel
> - Introduction into reliability metrics video cast