criterion-1.1.0.0: Robust, reliable performance measurement and analysis

Copyright | (c) 2009-2014 Bryan O'Sullivan
License | BSD-style
Maintainer | bos@serpentine.com
Stability | experimental
Portability | GHC
Safe Haskell | None
Language | Haskell98

Criterion.Analysis

Description

Analysis code for benchmarks.

Synopsis

- data Outliers = Outliers {
  - samplesSeen :: !Int64
  - lowSevere :: !Int64
  - lowMild :: !Int64
  - highMild :: !Int64
  - highSevere :: !Int64
  }
- data OutlierEffect
  - = Unaffected
  - | Slight
  - | Moderate
  - | Severe
- data OutlierVariance = OutlierVariance {
  - ovEffect :: OutlierEffect
  - ovDesc :: String
  - ovFraction :: Double
  }
- data SampleAnalysis = SampleAnalysis {
  - anRegress :: [Regression]
  - anOverhead :: Double
  - anMean :: Estimate
  - anStdDev :: Estimate
  - anOutlierVar :: OutlierVariance
  }
- analyseSample :: Int -> String -> Vector Measured -> ExceptT String Criterion Report
- scale :: Double -> SampleAnalysis -> SampleAnalysis
- analyseMean :: Sample -> Int -> Criterion Double
- countOutliers :: Outliers -> Int64
- classifyOutliers :: Sample -> Outliers
- noteOutliers :: Outliers -> Criterion ()
- outlierVariance :: Estimate -> Estimate -> Double -> OutlierVariance
- resolveAccessors :: [String] -> Either String [(String, Measured -> Maybe Double)]
- validateAccessors :: [String] -> String -> Either String [(String, Measured -> Maybe Double)]
- regress :: GenIO -> [String] -> String -> Vector Measured -> ExceptT String Criterion Regression

# Documentation

data Outliers Source

Outliers from sample data, calculated using the boxplot technique.

Constructors

Outliers |

Fields

samplesSeen :: !Int64
lowSevere :: !Int64

  More than 3 times the interquartile range (IQR) below the first quartile.

lowMild :: !Int64

  Between 1.5 and 3 times the IQR below the first quartile.

highMild :: !Int64

  Between 1.5 and 3 times the IQR above the third quartile.

highSevere :: !Int64

  More than 3 times the IQR above the third quartile.

Instances

Eq Outliers Source |
Data Outliers Source |
Read Outliers Source |
Show Outliers Source |
Generic Outliers Source |
ToJSON Outliers Source |
FromJSON Outliers Source |
Monoid Outliers Source |
Binary Outliers Source |
NFData Outliers Source |
type Rep Outliers Source |

data OutlierEffect Source

A description of the extent to which outliers in the sample data affect the sample mean and standard deviation.

Constructors

Unaffected | Less than 1% effect.
Slight | Between 1% and 10%.
Moderate | Between 10% and 50%.
Severe | Above 50% (i.e. measurements are useless).

Instances

Eq OutlierEffect Source |
Data OutlierEffect Source |
Ord OutlierEffect Source |
Read OutlierEffect Source |
Show OutlierEffect Source |
Generic OutlierEffect Source |
ToJSON OutlierEffect Source |
FromJSON OutlierEffect Source |
Binary OutlierEffect Source |
NFData OutlierEffect Source |
type Rep OutlierEffect Source |

data OutlierVariance Source

Analysis of the extent to which outliers in a sample affect its standard deviation (and to some extent, its mean).

Constructors

OutlierVariance |

Fields

ovEffect :: OutlierEffect

  Qualitative description of effect.

ovDesc :: String

  Brief textual description of effect.

ovFraction :: Double

  Quantitative description of effect (a fraction between 0 and 1).

Instances

Eq OutlierVariance Source |
Data OutlierVariance Source |
Read OutlierVariance Source |
Show OutlierVariance Source |
Generic OutlierVariance Source |
ToJSON OutlierVariance Source |
FromJSON OutlierVariance Source |
Binary OutlierVariance Source |
NFData OutlierVariance Source |
type Rep OutlierVariance Source |

data SampleAnalysis Source

Result of a bootstrap analysis of a non-parametric sample.

Constructors

SampleAnalysis |

Fields

anRegress :: [Regression]

  Estimates calculated via linear regression.

anOverhead :: Double

  Estimated measurement overhead, in seconds. Estimation is performed via linear regression.

anMean :: Estimate

  Estimated mean.

anStdDev :: Estimate

  Estimated standard deviation.

anOutlierVar :: OutlierVariance

  Description of the effects of outliers on the estimated variance.

Instances

Eq SampleAnalysis Source |
Data SampleAnalysis Source |
Read SampleAnalysis Source |
Show SampleAnalysis Source |
Generic SampleAnalysis Source |
ToJSON SampleAnalysis Source |
FromJSON SampleAnalysis Source |
Binary SampleAnalysis Source |
NFData SampleAnalysis Source |
type Rep SampleAnalysis Source |

analyseSample Source

Arguments

:: Int | Experiment number.
-> String | Experiment name.
-> Vector Measured | Sample data.
-> ExceptT String Criterion Report |

Perform an analysis of a measurement.

scale Source

Arguments

:: Double | Value to multiply by.
-> SampleAnalysis |
-> SampleAnalysis |

Multiply the `Estimate`s in an analysis by the given value, using `scale`.

analyseMean Source

Arguments

:: Sample |
-> Int | Number of iterations used to compute the sample.
-> Criterion Double |

Display the mean of a `Sample`, and characterise the outliers present in the sample.

countOutliers :: Outliers -> Int64 Source

Count the total number of outliers in a sample.

classifyOutliers :: Sample -> Outliers Source

Classify outliers in a data set, using the boxplot technique.

noteOutliers :: Outliers -> Criterion () Source

Display a report of the `Outliers` present in a `Sample`.

outlierVariance Source

Arguments

:: Estimate | Bootstrap estimate of sample mean.
-> Estimate | Bootstrap estimate of sample standard deviation.
-> Double | Number of original iterations.
-> OutlierVariance |

Compute the extent to which outliers in the sample data affect the sample mean and standard deviation.

resolveAccessors :: [String] -> Either String [(String, Measured -> Maybe Double)] Source

Given a list of accessor names (see `measureKeys`), return either a mapping from accessor name to function or an error message if any names are wrong.

validateAccessors Source

Arguments

:: [String] | Predictor names.
-> String | Responder name.
-> Either String [(String, Measured -> Maybe Double)] |

Given predictor and responder names, do some basic validation, then hand back the relevant accessors.

regress Source

Arguments

:: GenIO |
-> [String] | Predictor names.
-> String | Responder name.
-> Vector Measured |
-> ExceptT String Criterion Regression |

Regress the given predictors against the responder.

Errors may be returned under various circumstances, such as invalid names or lack of needed data.

See `olsRegress` for details of the regression performed.