# Graceful Shutdown

Nova services have experimental graceful shutdown support on `SIGTERM`. When a service worker implementing an RPC server receives `SIGTERM`, that worker stops accepting new RPC requests and waits for in-progress tasks to reach a safe termination point before exiting. This reduces the risk of leaving instances or migrations of instances in an unwanted or unrecoverable state. If deployment has the multiple worker for the `nova-conductor` and `nova-scheduler` service, then new requests are handled by the other workers.

Important

The current implementation waits for the `manager_shutdown_timeout` time for in-progress tasks to complete. A future release will improve this by a proper task tracking system. As a result operations can be interrupted ungracefully if they do not complete within this timeout and can leave instances in a unwanted state.

## How graceful shutdown works for nova-compute service

When `nova-compute` receives `SIGTERM`, the following sequence occurs:

1. The primary RPC server (`compute` topic) stops accepting new requests.

2. The secondary RPC server (`compute-alt` topic) still active and handles the RPC requests needed to finish in-progress tasks.

3. The service manager waits up to `manager_shutdown_timeout` seconds for in-progress tasks to complete.

4. The secondary RPC server (`compute-alt` topic) is stopped.

5. The service is stopped.

For `nova-conductor` and `nova-scheduler`, the sequence is the same except there is only one RPC server and the further requests are handled by their other workers.

## The additional RabbitMQ queue for compute service

`nova-compute` service maintains two RPC servers:

- Primary server (`compute` topic): Handles all new incoming requests during normal operation. This server is stopped first when a shutdown begins.

- Secondary server (`compute-alt` topic): Receives requests for long-running operations that to be continued and completed during shutdown

Because a second RPC server, each compute node will have an additional RabbitMQ queue named `compute-alt.<hostname>`.

## Operations handled during shutdown

The following operations use the secondary RPC server so that they will be allowed to complete during a graceful shutdown:

- Live migration

- Cold migration

- Revert resize

- Cross-cell resize

- External instance events

- Get console output

When the compute node’s RPC version is older than 6.5, Nova automatically falls back to sending all operations to the primary RPC server. The secondary RPC server is not used in this case.

## Configuration

Two configuration options control graceful shutdown behaviour. Both are in the `[DEFAULT]` section of `nova.conf` of respective service.

`graceful_shutdown_timeout`

The overall time the service waits before forcefully exit. This is defaults to 180 seconds for each Nova services.

If the service is not exited by this time, the service is stopped instantaneously. The operators using the external system (e.g. k8s, systemd) to manage the Nova serviecs should ensure that their service stop timeouts are set to at least `graceful_shutdown_timeout` to avoid forcefully killing service before Nova finish its graceful shutdown.

`manager_shutdown_timeout`

This controls how long the service waits for in-progress tasks to finish during graceful shutdown.

This is defaults to 160 seconds for each service. This must be less than `graceful_shutdown_timeout`

Setting this option to `0` disables the wait entirely: the manager does not wait for in-progress tasks before proceeding with shutdown.

The operators may want to set the above config options value based on how long their typical long-running operations (e.g. live migrations) take to complete.

## Upgrade considerations

- The default value of `graceful_shutdown_timeout` has been raised from 60 seconds (the `oslo.service` default) to 180 seconds for all Nova services. If your service manager previously relied on the 60-second default, update its stop timeout to at least 180 seconds before upgrading.

- A new option `manager_shutdown_timeout` has been added with a default of 160 seconds. No action is required unless you want to change the value.

- `nova-compute` service creates an additional RabbitMQ queue (`compute-alt.<hostname>`) on startup. Ensure your message broker has capacity for the additional queues.

- During a rolling upgrade where some compute nodes are still running a version older than 6.5, Nova will fall back to routing all operations through the primary `compute` queue. The graceful shutdown feature only works when all compute nodes have been upgraded.

this page last updated: 2026-08-20 01:33:50

Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.

found an error? report a bug