tasty-checklist-1.0.6.0: Check multiple items during a tasty test

Safe Haskell | Safe-Inferred
Language | Haskell2010

Test.Tasty.Checklist

Contents

- Checklist testing context
- Performing or Disabling checks
- Type-safe multi-check specifications
- Error reporting
- Displaying tested values

Description

This package provides the ability to run a Checklist of several "checks" during a single test. A "bad" check does not immediately result in a test failure; at the end of the test (passed or failed due to primary testing), all failed checks are reported (and any failed checks will result in an overall test failure at the end.

This type of checking can be very useful when needing to test various aspects of an operation that is complex to setup, has multiple effects, or where the checks are related such that knowing about the multiple failures makes debugging easier.

An alternative approach is to have some sort of common preparation code and use a separate test for each item. This module simply provides a convenient method to collate related items under the aegis of a single test.

This package also provides the `checkValues` function which can be used to check a number of derived values from a single input value via a checklist. This can be used to independently verify a number of record fields of a data structure or to validate related operations performed from a single input.

See the documentation for `check` and `checkValues` for examples of using this library. The tests in the source package also provide additional examples of usage.

Synopsis

- withChecklist :: (MonadIO m, MonadMask m) => Text -> (CanCheck => m a) -> m a
- type CanCheck = ?checker :: IORef [CheckResult]
- check :: (CanCheck, TestShow a, MonadIO m) => Text -> (a -> Bool) -> a -> m ()
- discardCheck :: (CanCheck, MonadIO m) => Text -> m ()
- checkValues :: CanCheck => TestShow dType => dType -> Assignment (DerivedVal dType) idx -> IO ()
- data DerivedVal i d where
  - Val :: (TestShow d, Eq d) => Text -> (i -> d) -> d -> DerivedVal i d
  - Got :: Text -> (i -> Bool) -> DerivedVal i Bool
  - Observe :: Eq d => Text -> (i -> d) -> d -> (d -> d -> String) -> DerivedVal i d
- data CheckResult
- data ChecklistFailures
- class TestShow v where
  - testShow :: v -> String
- testShowList :: TestShow v => [v] -> String
- multiLineDiff :: Text -> Text -> String

# Checklist testing context

withChecklist :: (MonadIO m, MonadMask m) => Text -> (CanCheck => m a) -> m a Source #

This should be used to wrap the test that contains checks. This initializes the environment needed for the checks to run, and on exit from the test, reports any (and all) failed checks as a test failure.

type CanCheck = ?checker :: IORef [CheckResult] Source #

A convenient Constraint to apply to functions that will perform checks (i.e. call `check` one or more times)

# Performing or Disabling checks

check :: (CanCheck, TestShow a, MonadIO m) => Text -> (a -> Bool) -> a -> m () Source #

This is used to run a check within the code. The first argument is the "name" of this check, the second is a function that takes a value and returns `True` if the value is OK, or `False` if the value fails the check. The last argument is the value to check.

    >>> :set -XOverloadedStrings
    >>> import Test.Tasty
    >>> import Test.Tasty.HUnit
    >>> :{
    >>> defaultMain $ testCase "odd numbers" $ withChecklist "odds" $ do
    >>> let three = 3 :: Int
    >>> check "three is odd" odd three
    >>> check "two is odd" odd (2 :: Int)
    >>> check "7 + 3 is odd" odd $ 7 + three
    >>> check "7 is odd" odd (7 :: Int)
    >>> :}
    odd numbers: FAIL
      Exception: ERROR: odds
        2 checks failed in this checklist:
        -Failed check 'two is odd' with: 2
        -Failed check '7 + 3 is odd' with: 10

    1 out of 1 tests failed (...s)
    *** Exception: ExitFailure 1

Any check failures are also printed to stdout (and omitted from the above for clarity). This is so that those failures are reported even if a more standard test assertion is used that prevents completion of the checklist. Thus, if an `assertEqual "values" three 7` had been added to the above, that would have been the only actual (and immediate) fail for the test, but any failing `check`s appearing before that `assertEqual` would still have printed.

discardCheck :: (CanCheck, MonadIO m) => Text -> m () Source #

Sometimes checks are provided in common testing code, often in setup/preparation for the main tests. In some cases, the check is not applicable for that particular test. This function can be used to discard any pending failures for the associated named check.

This is especially useful when a common code block is used to perform a set of checks: if a few of the common checks are not appropriate for the current situation, `discardCheck` can be used to throw away the results of those checks by matching on the check name.

# Type-safe multi-check specifications

Implementing a number of discrete `check` calls can be tedious, especially when they are validating different aspects of the same result value. To facilitate this, the `checkValues` function can be used along with a type-safe list of checks to perform.

To demonstrate this, first consider the following sample program, which has code that generates a complex `Struct` value, along with tests for various fields in that `Struct`.

    >>> :set -XPatternSynonyms
    >>> :set -XOverloadedStrings
    >>>
    >>> import Data.Parameterized.Context ( pattern Empty, pattern (:>) )
    >>> import Test.Tasty.Checklist
    >>> import Test.Tasty
    >>> import Test.Tasty.HUnit
    >>>
    >>> :{
    >>> data Struct = MyStruct { foo :: Int, bar :: Char, baz :: String }
    >>>
    >>> instance Show Struct where
    >>> show s = baz s <> " is " <> show (foo s) <> [bar s]
    >>> instance TestShow Struct where testShow = show
    >>>
    >>> someFun :: Int -> Struct
    >>> someFun n = MyStruct (n * 6)
    >>> (if n * 6 == 42 then '!' else '?')
    >>> "The answer to the universe"
    >>>
    >>> oddAnswer :: Struct -> Bool
    >>> oddAnswer = odd . foo
    >>>
    >>> test = testCase "someFun result" $
    >>> withChecklist "results for someFun" $
    >>> someFun 3 `checkValues`
    >>> (Empty
    >>> :> Val "foo" foo 42
    >>> :> Val "baz field" baz "The answer to the universe"
    >>> :> Val "shown" show "The answer to the universe is 42!"
    >>> :> Val "odd answer" oddAnswer False
    >>> :> Got "even answer" (not . oddAnswer)
    >>> :> Val "double-checking foo" foo 42
    >>> )
    >>> :}

This code will be used below to demonstrate various advanced checklist capabilities.

checkValues :: CanCheck => TestShow dType => dType -> Assignment (DerivedVal dType) idx -> IO () Source #

The `checkValues` is a checklist that tests various values that can be derived from the input value. The input value is provided, along with an `Assignment` list of extraction functions and the expected result value (and name) of that extraction. Each extraction is performed as a check within the checklist.

This is convenient to gather together a number of validations on a single datatype and represent them economically.

One example is testing the fields of a record structure, given the above code:

    >>> defaultMain test
    someFun result: FAIL
      Exception: ERROR: results for someFun
        3 checks failed in this checklist:
        --- Input for below: The answer to the universe is 18?
        -Failed check: foo
              expected:    42
              failed with: 18
        -Failed check: shown
              expected:    "The answer to the universe is 42!"
              failed with: "The answer to the universe is 18?"
        -Failed check: double-checking foo
              expected:    42
              failed with: 18

    1 out of 1 tests failed (...s)
    *** Exception: ExitFailure 1

In this case, several of the values checked were correct, but more than one was wrong. Helpfully, this test output lists all the wrong answers for the single input provided.

data DerivedVal i d where Source #

Each entry in the `Assignment` list for `checkValues` should be one of these `DerivedVal` values.

The `i` type parameter is the input type, and the `d` is the value derived from that input type.

Constructors

Val :: (TestShow d, Eq d) => Text -> (i -> d) -> d -> DerivedVal i d | Val allows specification of a description string, an extraction function, and the expected value to be extracted. The `checkValues` function will add a Failure if the expected value is not obtained.

Got :: Text -> (i -> Bool) -> DerivedVal i Bool

Got allows specification of a description string and an extraction function. The `checkValues` function will add a Failure if the extraction result is False.

    Val "what" f True === Got "what" f

Observe :: Eq d => Text -> (i -> d) -> d -> (d -> d -> String) -> DerivedVal i d

Observe performs the same checking as Val except the TestShow information for the actual and expected values are not as useful (e.g. they are lengthy, multi-line, or gibberish) so instead this allows the specification of a function that will take the supplied expected value and the result of the extraction function (the actual), respectively, and generate its own description of the failure.

# Error reporting

data CheckResult Source #

The internal `CheckResult` captures the failure information for a check

#### Instances

Instances details

Show CheckResult Source # |
Instance details Defined in Test.Tasty.Checklist Methods showsPrec :: Int -> CheckResult -> ShowS Source # show :: CheckResult -> String Source # showList :: [CheckResult] -> ShowS Source #

data ChecklistFailures Source #

The ChecklistFailures exception is thrown if any checks have failed during testing.

#### Instances

Instances details

Exception ChecklistFailures Source # |
Instance details Defined in Test.Tasty.Checklist Methods toException :: ChecklistFailures -> SomeException Source # fromException :: SomeException -> Maybe ChecklistFailures Source # displayException :: ChecklistFailures -> String Source #
Show ChecklistFailures Source # |
Instance details Defined in Test.Tasty.Checklist Methods showsPrec :: Int -> ChecklistFailures -> ShowS Source # show :: ChecklistFailures -> String Source # showList :: [ChecklistFailures] -> ShowS Source #

# Displaying tested values

class TestShow v where Source #

The `TestShow` class is defined to provide a way for the various data objects tested by this module to be displayed when tests fail. The default `testShow` will use a `Show` instance, but this can be overridden if there are alternate ways to display a particular object (e.g. pretty-printing, etc.)

Minimal complete definition

Nothing

Methods

testShow :: v -> String Source #

default testShow :: Show v => v -> String Source #

#### Instances

Instances details

TestShow String Source # |
Instance details Defined in Test.Tasty.Checklist Methods testShow :: String -> String Source #
TestShow Integer Source # |
Instance details Defined in Test.Tasty.Checklist Methods testShow :: Integer -> String Source #
TestShow () Source # |
Instance details Defined in Test.Tasty.Checklist Methods testShow :: () -> String Source #
TestShow Bool Source # |
Instance details Defined in Test.Tasty.Checklist Methods testShow :: Bool -> String Source #
TestShow Char Source # |
Instance details Defined in Test.Tasty.Checklist Methods testShow :: Char -> String Source #
TestShow Float Source # |
Instance details Defined in Test.Tasty.Checklist Methods testShow :: Float -> String Source #
TestShow Int Source # |
Instance details Defined in Test.Tasty.Checklist Methods testShow :: Int -> String Source #
(TestShow a, TestShow b) => TestShow (a, b) Source # |
Instance details Defined in Test.Tasty.Checklist Methods testShow :: (a, b) -> String Source #
(TestShow a, TestShow b, TestShow c) => TestShow (a, b, c) Source # |
Instance details Defined in Test.Tasty.Checklist Methods testShow :: (a, b, c) -> String Source #

testShowList :: TestShow v => [v] -> String Source #

A helper function for defining a testShow for lists of items.

    instance TestShow [Int] where testShow = testShowList

multiLineDiff :: Text -> Text -> String Source #

The multiLineDiff is another helper function that can be used to format a line-by-line difference display of two Text representations. This is provided as a convenience function to help format large text regions for easier comparison.