# Troubleshoot Compute

Common problems for Compute typically involve misconfigured networking or credentials that are not sourced properly in the environment. Also, most flat networking configurations do not enable ping or ssh from a compute node to the instances that run on that node. Another common problem is trying to run 32-bit images on a 64-bit compute node. This section shows you how to troubleshoot Compute.

Todo

Move the sections below into sub-pages for readability.

- Orphaned resource allocations
- Rebuild placement DB
- Affinity policy violated with parallel requests

## Compute service logging

Compute stores a log file for each service in `/var/log/nova`. For example, `nova-compute.log` is the log for the `nova-compute` service. You can set the following options to format log strings for the `nova.log` module in the `nova.conf` file:

- `logging_context_format_string`

- `logging_default_format_string`

If the log level is set to `debug`, you can also specify `logging_debug_format_suffix` to append extra formatting. For information about what variables are available for the formatter, see Formatter Objects.

You have two logging options for OpenStack Compute based on configuration settings. In `nova.conf`, include the `logfile` option to enable logging. Alternatively you can set `use_syslog = 1` so that the nova daemon logs to syslog.

## Guru Meditation reports

A Guru Meditation report is sent by the Compute service upon receipt of the `SIGUSR2` signal (`SIGUSR1` before Mitaka). This report is a general-purpose error report that includes details about the current state of the service. The error report is sent to `stderr`.

For example, if you redirect error output to `nova-compute-err.log` using nova-compute 2>/var/log/nova/nova-compute-err.log, resulting in the process ID 8675, you can then run:

    # kill -USR2 8675

This command triggers the Guru Meditation report to be printed to `/var/log/nova/nova-compute-err.log`.

For the WSGI-based compute API and metadata API services, using signals is not trivial due to the web server’s own signal handling. An alternative to the signal-based approach that works equally well for freestanding and hosted entry points is to use a file-based trigger.

Configure the service to trigger the GMR by the modification time changes of a file or directory.

    [oslo_reports]
    file_event_handler=/var/lib/nova

Then the report can be triggered by touching the file or directory. The GMRb will be emitted in the same place where the service logs normally.

    touch /var/lib/nova

Note that some web servers freeze the request handler processes when there is no HTTP request to be handled. This prevent the file system monitoring loop to detect the change. So after touching the file make a HTTP request to the given WSGI application.

    openstack compute service list

For more details and a complete working example you can check openstack-must-gather.

The report has the following sections:

- Package: Displays information about the package to which the process belongs, including version information.

- Threads: Displays stack traces and thread IDs for each of the threads within the process.

- Green Threads: Displays stack traces for each of the green threads within the process (green threads do not have thread IDs).

- Configuration: Lists all configuration options currently accessible through the CONF object for the current process.

For more information, see Guru Meditation Reports.

## Common errors and fixes for Compute

The ask.openstack.org site offers a place to ask and answer questions, and you can also mark questions as frequently asked questions. This section describes some errors people have posted previously. Bugs are constantly being fixed, so online resources are a great way to get the most up-to-date errors and fixes.

## Credential errors, 401, and 403 forbidden errors

### Problem

Missing credentials cause a `403 forbidden` error.

### Solution

To resolve this issue, use one of these methods:

1. Manual method

   Gets the `novarc` file from the project ZIP file, saves existing credentials in case of override, and manually sources the `novarc` file.

2. Script method

   Generates `novarc` from the project ZIP file and sources it for you.

When you run `nova-api` the first time, it generates the certificate authority information, including `openssl.cnf`. If you start the CA services before this, you might not be able to create your ZIP file. Restart the services. When your CA information is available, create your ZIP file.

Also, check your HTTP proxy settings to see whether they cause problems with `novarc` creation.

## Live migration permission issues

### Problem

When live migrating an instance, you may see errors like the below:

    libvirtError: operation failed: Failed to connect to remote libvirt URI
    qemu+ssh://stack@cld6b16/system: Cannot recv data: Host key verification
    failed.: Connection reset by peer

### Solution

Ensure you have completed all the steps outlined in Configure SSH between compute nodes. In particular, it’s important to note that the `libvirt` process runs as `root` even though it may be connecting to a different user (`stack` in the above example). You can ensure everything is correctly configured by attempting to connect to the remote host via the `root` user. Using the above example once again:

    $ su - -c 'ssh stack@cld6b16'

## Instance errors

### Problem

Sometimes a particular instance shows `pending` or you cannot SSH to it. Sometimes the image itself is the problem. For example, when you use flat manager networking, you do not have a DHCP server and certain images do not support interface injection; you cannot connect to them.

### Solution

To fix instance errors use an image that does support this method, such as Ubuntu, which obtains an IP address correctly with FlatManager network settings.

To troubleshoot other possible problems with an instance, such as an instance that stays in a spawning state, check the directory for the particular instance under `/var/lib/nova/instances` on the `nova-compute` host and make sure that these files are present:

- `libvirt.xml`

- `disk`

- `disk-raw`

- `kernel`

- `ramdisk`

- `console.log`, after the instance starts.

If any files are missing, empty, or very small, the `nova-compute` service did not successfully download the images from the Image service.

Also check `nova-compute.log` for exceptions. Sometimes they do not appear in the console output.

Next, check the log file for the instance in the `/var/log/libvirt/qemu` directory to see if it exists and has any useful error messages in it.

Finally, from the `/var/lib/nova/instances` directory for the instance, see if this command returns an error:

    # virsh create libvirt.xml

## Empty log output for Linux instances

### Problem

You can view the log output of running instances from either the Log tab of the dashboard or the output of nova console-log. In some cases, the log output of a running Linux instance will be empty or only display a single character (for example, the `?` character).

This occurs when the Compute service attempts to retrieve the log output of the instance via a serial console while the instance itself is not configured to send output to the console.

### Solution

To rectify this, append the following parameters to kernel arguments specified in the instance’s boot loader:

    console=tty0 console=ttyS0,115200n8

Upon rebooting, the instance will be configured to send output to the Compute service.

## Reset the state of an instance

### Problem

Instances can remain in an intermediate state, such as `deleting`.

### Solution

You can use the nova reset-state command to manually reset the state of an instance to an error state. You can then delete the instance. For example:

    $ nova reset-state c6bbbf26-b40a-47e7-8d5c-eb17bf65c485
    $ openstack server delete c6bbbf26-b40a-47e7-8d5c-eb17bf65c485

You can also use the `--active` parameter to force the instance back to an active state instead of an error state. For example:

    $ nova reset-state --active c6bbbf26-b40a-47e7-8d5c-eb17bf65c485

## Injection problems

### Problem

Instances may boot slowly, or do not boot. File injection can cause this problem.

### Solution

To disable injection in libvirt, set the following in `nova.conf`:

    [libvirt]
    inject_partition = -2

Note

If you have not enabled the config drive and you want to make user-specified files available from the metadata server for to improve performance and avoid boot failure if injection fails, you must disable injection.

## Cannot find suitable emulator for x86_64

### Problem

When you attempt to create a VM, the error shows the VM is in the `BUILD` then `ERROR` state.

### Solution

On the KVM host, run cat /proc/cpuinfo. Make sure the `vmx` or `svm` flags are set.

Follow the instructions in the Enable KVM section in the Nova Configuration Reference to enable hardware virtualization support in your BIOS.

## Failed to attach volume after detaching

### Problem

Failed to attach a volume after detaching the same volume.

### Solution

You must change the device name on the nova-attach command. The VM might not clean up after a nova-detach command runs. This example shows how the nova-attach command fails when you use the `vdb`, `vdc`, or `vdd` device names:

    # ls -al /dev/disk/by-path/
    total 0
    drwxr-xr-x 2 root root 200 2012-08-29 17:33 .
    drwxr-xr-x 5 root root 100 2012-08-29 17:33 ..
    lrwxrwxrwx 1 root root 9 2012-08-29 17:33 pci-0000:00:04.0-virtio-pci-virtio0 -> ../../vda
    lrwxrwxrwx 1 root root 10 2012-08-29 17:33 pci-0000:00:04.0-virtio-pci-virtio0-part1 -> ../../vda1
    lrwxrwxrwx 1 root root 10 2012-08-29 17:33 pci-0000:00:04.0-virtio-pci-virtio0-part2 -> ../../vda2
    lrwxrwxrwx 1 root root 10 2012-08-29 17:33 pci-0000:00:04.0-virtio-pci-virtio0-part5 -> ../../vda5
    lrwxrwxrwx 1 root root 9 2012-08-29 17:33 pci-0000:00:06.0-virtio-pci-virtio2 -> ../../vdb
    lrwxrwxrwx 1 root root 9 2012-08-29 17:33 pci-0000:00:08.0-virtio-pci-virtio3 -> ../../vdc
    lrwxrwxrwx 1 root root 9 2012-08-29 17:33 pci-0000:00:09.0-virtio-pci-virtio4 -> ../../vdd
    lrwxrwxrwx 1 root root 10 2012-08-29 17:33 pci-0000:00:09.0-virtio-pci-virtio4-part1 -> ../../vdd1

You might also have this problem after attaching and detaching the same volume from the same VM with the same mount point multiple times. In this case, restart the KVM host.

## Failed to attach volume, systool is not installed

### Problem

This warning and error occurs if you do not have the required `sysfsutils` package installed on the compute node:

    WARNING nova.virt.libvirt.utils [req-1200f887-c82b-4e7c-a891-fac2e3735dbb\
    admin admin|req-1200f887-c82b-4e7c-a891-fac2e3735dbb admin admin] systool\
    is not installed
    ERROR nova.compute.manager [req-1200f887-c82b-4e7c-a891-fac2e3735dbb admin\
    admin|req-1200f887-c82b-4e7c-a891-fac2e3735dbb admin admin]
    [instance: df834b5a-8c3f-477a-be9b-47c97626555c|instance: df834b5a-8c3f-47\
    7a-be9b-47c97626555c]
    Failed to attach volume 13d5c633-903a-4764-a5a0-3336945b1db1 at /dev/vdk.

### Solution

Install the `sysfsutils` package on the compute node. For example:

    # apt-get install sysfsutils

## Failed to connect volume in FC SAN

### Problem

The compute node failed to connect to a volume in a Fibre Channel (FC) SAN configuration. The WWN may not be zoned correctly in your FC SAN that links the compute host to the storage array:

    ERROR nova.compute.manager [req-2ddd5297-e405-44ab-aed3-152cd2cfb8c2 admin\
    demo|req-2ddd5297-e405-44ab-aed3-152cd2cfb8c2 admin demo] [instance: 60ebd\
    6c7-c1e3-4bf0-8ef0-f07aa4c3d5f3|instance: 60ebd6c7-c1e3-4bf0-8ef0-f07aa4c3\
    d5f3]
    Failed to connect to volume 6f6a6a9c-dfcf-4c8d-b1a8-4445ff883200 while\
    attaching at /dev/vdjTRACE nova.compute.manager [instance: 60ebd6c7-c1e3-4\
    bf0-8ef0-f07aa4c3d5f3|instance: 60ebd6c7-c1e3-4bf0-8ef0-f07aa4c3d5f3]
    Traceback (most recent call last):...f07aa4c3d5f3\] ClientException: The\
    server has either erred or is incapable of performing the requested\
    operation.(HTTP 500)(Request-ID: req-71e5132b-21aa-46ee-b3cc-19b5b4ab2f00)

### Solution

The network administrator must configure the FC SAN fabric by correctly zoning the WWN (port names) from your compute node HBAs.

## Multipath call failed exit

### Problem

Multipath call failed exit. This warning occurs in the Compute log if you do not have the optional `multipath-tools` package installed on the compute node. This is an optional package and the volume attachment does work without the multipath tools installed. If the `multipath-tools` package is installed on the compute node, it is used to perform the volume attachment. The IDs in your message are unique to your system.

    WARNING nova.storage.linuxscsi [req-cac861e3-8b29-4143-8f1b-705d0084e571 \
    admin admin|req-cac861e3-8b29-4143-8f1b-705d0084e571 admin admin] \
    Multipath call failed exit (96)

### Solution

Install the `multipath-tools` package on the compute node. For example:

    # apt-get install multipath-tools

## Failed to Attach Volume, Missing sg_scan

### Problem

Failed to attach volume to an instance, `sg_scan` file not found. This error occurs when the sg3-utils package is not installed on the compute node. The IDs in your message are unique to your system:

    ERROR nova.compute.manager [req-cf2679fd-dd9e-4909-807f-48fe9bda3642 admin admin|req-cf2679fd-dd9e-4909-807f-48fe9bda3642 admin admin]
    [instance: 7d7c92e0-49fa-4a8e-87c7-73f22a9585d5|instance:  7d7c92e0-49fa-4a8e-87c7-73f22a9585d5]
    Failed to attach volume  4cc104c4-ac92-4bd6-9b95-c6686746414a at /dev/vdcTRACE nova.compute.manager
    [instance:  7d7c92e0-49fa-4a8e-87c7-73f22a9585d5|instance: 7d7c92e0-49fa-4a8e-87c7-73f22a9585d5]
    Stdout: '/usr/local/bin/nova-rootwrap: Executable not found: /usr/bin/sg_scan'

### Solution

Install the `sg3-utils` package on the compute node. For example:

    # apt-get install sg3-utils

## Requested microversions are ignored

### Problem

When making a request with a microversion beyond 2.1, for example:

    $ openstack --os-compute-api-version 2.15 server group create \
      --policy soft-anti-affinity my-soft-anti-group

It fails saying that “soft-anti-affinity” is not a valid policy, even thought it is allowed with the 2.15 microversion.

### Solution

Ensure the `compute` endpoint in the identity service catalog is pointing at `/v2.1` instead of `/v2`. The former route supports microversions, while the latter route is considered the legacy v2.0 compatibility-mode route which renders all requests as if they were made on the legacy v2.0 API.

this page last updated: 2026-04-21 19:00:00

Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.

found an error? report a bug