#### 2.2.1 Output States ¶

`runtest` flags the outcome of each test as one of these cases. See A POSIX compliant test framework for a discussion of how POSIX specifies the meanings of these cases.

PASS

  The most desirable outcome: the test was expected to succeed and did succeed.

XPASS

  A pleasant kind of failure: a test was expected to fail, but succeeded. This may indicate progress; inspect the test case to determine whether you should amend it to stop expecting failure.

FAIL

  A test failed, although it was expected to succeed. This may indicate regress; inspect the test case and the failing software to locate the bug.

XFAIL

  A test failed, but it was expected to fail. This result indicates no change in a known bug. If a test fails because the operating system where the test runs lacks some facility required by the test, the outcome is UNSUPPORTED instead.

UNRESOLVED

  Output from a test requires manual inspection; the testsuite could not automatically determine the outcome. For example, your tests can report this outcome is when a test does not complete as expected.

UNTESTED

  A test case is not yet complete, and in particular cannot yet produce a PASS or FAIL. You can also use this outcome in dummy “tests” that note explicitly the absence of a real test case for a particular property.

UNSUPPORTED

  A test depends on a conditionally available feature that does not exist (in the configured testing environment). For example, you can use this outcome to report on a test case that does not work on a particular target because its operating system support does not include a required subroutine.

`runtest` may also display the following messages:

ERROR

  Indicates a major problem (detected by the test case itself) in running the test. This is usually an unrecoverable error, such as a missing file or loss of communication to the target. POSIX testsuites should not emit this message; use UNSUPPORTED, UNTESTED, or UNRESOLVED instead, as appropriate.

WARNING

  Indicates a possible problem in running the test. Usually warnings correspond to recoverable errors, or display an important message about the following tests.

NOTE

  An informational message about the test case.