The ChrF metric.

ChrF is a metric for evaluating the similarity between a target and reference text based on similarity of character and word n-grams.

Configuration

Configuration for the parameters of the ChrF metric:

title: "ChrfConfig"
type: "object"
properties:
  char_order:
    type: "integer"
    description: >
      "The n-gram order of character n-grams to consider. "
      "(default: 6)"
  word_order:
    type: "integer"
    description: >
      "The n-gram order of words to consider. "
      "(default: 0)"
  whitespace:
    type: "boolean"
    description: "Whether to consider whitespace. (default: False)"
  beta:
    type: "number"
    description: >
      "The importance of recall with respect to precision. "
      "(default: 2)"

Data

Accepted data format of the ChrF metric:

title: "ChrfData"
type: "object"
properties:
  target:
    type: "string"
    description: "Target text to evaluate."
  references:
    type: "array"
    description: "The references to evaluate the target against."
    items:
      type: "string"
required:
  - "target"
  - "references"

Results

Format of the results of the ChrF metric:

title: "ChrfResult"
type: "object"
$defs:
  ChrfStats:
    type: "object"
    properties:
      value:
        type: "number"
        description: "ChrF score for the entire dataset."
      correct:
        type: "array"
        description: >
          "List of counts of correct ngrams, 1 <= n <= max_ngram_order. "
          "When smoothing is used, these will be the adjusted counts after smoothing."
        items:
          type: "number"
      total:
        type: "array"
        description: >
          "List of counts of total ngrams, 1 <= n <= max_ngram_order. "
          "When smoothing is used, these will be the adjusted counts after smoothing."
        items:
          type: "number"
      precisions:
        type: "array"
        description: "List of precisions, 1 <= n <= max_ngram_order"
        items:
          type: "number"
      brevity_penalty:
        type: "number"
        description: "The brevity penalty."
      target_length:
        type: "number"
        description: "The cumulative system length."
      reference_length:
        type: "number"
        description: "The cumulative reference length."
    required:
      - "value"
properties:
  overall:
    $ref: "#/$defs/ChrfStats"
  examples:
    type: "array"
    items:
      $ref: "#/$defs/ChrfStats"
required:
  - "overall"
  - "examples"