# mir_eval.pattern

Pattern discovery involves the identification of musical patterns (i.e. short fragments or melodic ideas that repeat at least twice) both from audio and symbolic representations. The metrics used to evaluate pattern discovery systems attempt to quantify the ability of the algorithm to not only determine the present patterns in a piece, but also to find all of their occurrences.

Based on the methods described here:

  T. Collins. MIREX task: Discovery of repeated themes & sections. http://www.music-ir.org/mirex/wiki/2013:Discovery_of_Repeated_Themes_&_Sections, 2013.

## Conventions

The input format can be automatically generated by calling `mir_eval.io.load_patterns()`. This format is a list of a list of tuples. The first list collections patterns, each of which is a list of occurrences, and each occurrence is a list of MIDI onset tuples of `(onset_time, mid_note)`

A pattern is a list of occurrences. The first occurrence must be the prototype of that pattern (i.e. the most representative of all the occurrences). An occurrence is a list of tuples containing the onset time and the midi note number.

## Metrics

- `mir_eval.pattern.standard_FPR()`: Strict metric in order to find the possibly transposed patterns of exact length. This is the only metric that considers transposed patterns.

- `mir_eval.pattern.establishment_FPR()`: Evaluates the amount of patterns that were successfully identified by the estimated results, no matter how many occurrences they found. In other words, this metric captures how the algorithm successfully established that a pattern repeated at least twice, and this pattern is also found in the reference annotation.

- `mir_eval.pattern.occurrence_FPR()`: Evaluation of how well an estimation can effectively identify all the occurrences of the found patterns, independently of how many patterns have been discovered. This metric has a threshold parameter that indicates how similar two occurrences must be in order to be considered equal. In MIREX, this evaluation is run twice, with thresholds .75 and .5.

- `mir_eval.pattern.three_layer_FPR()`: Aims to evaluate the general similarity between the reference and the estimations, combining both the establishment of patterns and the retrieval of its occurrences in a single F1 score.

- `mir_eval.pattern.first_n_three_layer_P()`: Computes the three-layer precision for the first N patterns only in order to measure the ability of the algorithm to sort the identified patterns based on their relevance.

- `mir_eval.pattern.first_n_target_proportion_R()`: Computes the target proportion recall for the first N patterns only in order to measure the ability of the algorithm to sort the identified patterns based on their relevance.

mir_eval.pattern.validate(reference_patterns, estimated_patterns)

  Check that the input annotations to a metric look like valid pattern lists, and throws helpful errors if not.

  Parameters:
    reference_patterns list

      The reference patterns using the format returned by `mir_eval.io.load_patterns()`

    estimated_patterns list

      The estimated patterns in the same format

mir_eval.pattern.standard_FPR(reference_patterns, estimated_patterns, tol=1e-05)

  Compute the standard F1 Score, Precision and Recall.

  This metric checks if the prototype patterns of the reference match possible translated patterns in the prototype patterns of the estimations. Since the sizes of these prototypes must be equal, this metric is quite restrictive and it tends to be 0 in most of 2013 MIREX results.

  Parameters:
    reference_patterns list

      The reference patterns using the format returned by `mir_eval.io.load_patterns()`

    estimated_patterns list

      The estimated patterns in the same format

    tol float

      Tolerance level when comparing reference against estimation. Default parameter is the one found in the original matlab code by Tom Collins used for MIREX 2013. (Default value = 1e-5)

  Returns:
    f_measure float

      The standard F1 Score

    precision float

      The standard Precision

    recall float

      The standard Recall

  Examples

      >>> ref_patterns = mir_eval.io.load_patterns("ref_pattern.txt")
      >>> est_patterns = mir_eval.io.load_patterns("est_pattern.txt")
      >>> F, P, R = mir_eval.pattern.standard_FPR(ref_patterns, est_patterns)

mir_eval.pattern.establishment_FPR(reference_patterns, estimated_patterns, similarity_metric='cardinality_score')

  Compute the establishment F1 Score, Precision and Recall.

  Parameters:
    reference_patterns list

      The reference patterns in the format returned by `mir_eval.io.load_patterns()`

    estimated_patterns list

      The estimated patterns in the same format

    similarity_metric str

      A string representing the metric to be used when computing the similarity matrix. Accepted values:

      > - “cardinality_score”: Count of the intersection between occurrences.

      (Default value = “cardinality_score”)

  Returns:
    f_measure float

      The establishment F1 Score

    precision float

      The establishment Precision

    recall float

      The establishment Recall

  Examples

      >>> ref_patterns = mir_eval.io.load_patterns("ref_pattern.txt")
      >>> est_patterns = mir_eval.io.load_patterns("est_pattern.txt")
      >>> F, P, R = mir_eval.pattern.establishment_FPR(ref_patterns,
      ...                                              est_patterns)

mir_eval.pattern.occurrence_FPR(reference_patterns, estimated_patterns, thres=0.75, similarity_metric='cardinality_score')

  Compute the occurrence F1 Score, Precision and Recall.

  Parameters:
    reference_patterns list

      The reference patterns in the format returned by `mir_eval.io.load_patterns()`

    estimated_patterns list

      The estimated patterns in the same format

    thres float

      How similar two occurrences must be in order to be considered equal (Default value = .75)

    similarity_metric str

      A string representing the metric to be used when computing the similarity matrix. Accepted values:

      > - “cardinality_score”: Count of the intersection between occurrences.

      (Default value = “cardinality_score”)

  Returns:
    f_measure float

      The occurrence F1 Score

    precision float

      The occurrence Precision

    recall float

      The occurrence Recall

  Examples

      >>> ref_patterns = mir_eval.io.load_patterns("ref_pattern.txt")
      >>> est_patterns = mir_eval.io.load_patterns("est_pattern.txt")
      >>> F, P, R = mir_eval.pattern.occurrence_FPR(ref_patterns,
      ...                                           est_patterns)

mir_eval.pattern.three_layer_FPR(reference_patterns, estimated_patterns)

  Three Layer F1 Score, Precision and Recall. As described by Meridith.

  Parameters:
    reference_patterns list

      The reference patterns in the format returned by `mir_eval.io.load_patterns()`

    estimated_patterns list

      The estimated patterns in the same format

  Returns:
    f_measure float

      The three-layer F1 Score

    precision float

      The three-layer Precision

    recall float

      The three-layer Recall

  Examples

      >>> ref_patterns = mir_eval.io.load_patterns("ref_pattern.txt")
      >>> est_patterns = mir_eval.io.load_patterns("est_pattern.txt")
      >>> F, P, R = mir_eval.pattern.three_layer_FPR(ref_patterns,
      ...                                            est_patterns)

mir_eval.pattern.first_n_three_layer_P(reference_patterns, estimated_patterns, n=5)

  First n three-layer precision.

  This metric is basically the same as the three-layer FPR but it is only applied to the first n estimated patterns, and it only returns the precision. In MIREX and typically, n = 5.

  Parameters:
    reference_patterns list

      The reference patterns in the format returned by `mir_eval.io.load_patterns()`

    estimated_patterns list

      The estimated patterns in the same format

    n int

      Number of patterns to consider from the estimated results, in the order they appear in the matrix (Default value = 5)

  Returns:
    precision float

      The first n three-layer Precision

  Examples

      >>> ref_patterns = mir_eval.io.load_patterns("ref_pattern.txt")
      >>> est_patterns = mir_eval.io.load_patterns("est_pattern.txt")
      >>> P = mir_eval.pattern.first_n_three_layer_P(ref_patterns,
      ...                                            est_patterns, n=5)

mir_eval.pattern.first_n_target_proportion_R(reference_patterns, estimated_patterns, n=5)

  First n target proportion establishment recall metric.

  This metric is similar is similar to the establishment FPR score, but it only takes into account the first n estimated patterns and it only outputs the Recall value of it.

  Parameters:
    reference_patterns list

      The reference patterns in the format returned by `mir_eval.io.load_patterns()`

    estimated_patterns list

      The estimated patterns in the same format

    n int

      Number of patterns to consider from the estimated results, in the order they appear in the matrix. (Default value = 5)

  Returns:
    recall float

      The first n target proportion Recall.

  Examples

      >>> ref_patterns = mir_eval.io.load_patterns("ref_pattern.txt")
      >>> est_patterns = mir_eval.io.load_patterns("est_pattern.txt")
      >>> R = mir_eval.pattern.first_n_target_proportion_R(
      ...                                 ref_patterns, est_patterns, n=5)

mir_eval.pattern.evaluate(ref_patterns, est_patterns, **kwargs)

  Load data and perform the evaluation.

  Parameters:
    ref_patterns list

      The reference patterns in the format returned by `mir_eval.io.load_patterns()`

    est_patterns list

      The estimated patterns in the same format

    **kwargs

      Additional keyword arguments which will be passed to the appropriate metric or preprocessing functions.

  Returns:
    scores dict

      Dictionary of scores, where the key is the metric name (str) and the value is the (float) score achieved.

  Examples

      >>> ref_patterns = mir_eval.io.load_patterns("ref_pattern.txt")
      >>> est_patterns = mir_eval.io.load_patterns("est_pattern.txt")
      >>> scores = mir_eval.pattern.evaluate(ref_patterns, est_patterns)