The BARTScore metric.

BARTScore is a metric for evaluating text via text generation based on the BART sequence-to-sequence pre-trained model.

Configuration

Configuration for the parameters of the BARTScore metric:

title: "BartScoreConfig"
type: "object"
properties:
  variety:
    type: "string"
    description: "Name of the metric variety to be used."
    enum:
      - "reference_to_target"
      - "target_to_reference"
      - "reference_target_bidirectional"
      - "source_to_target"
      - "target_to_source"
  model:
    type: "string"
    pattern: "^[a-z0-9-/]+$"
    description: "Model name"
  language:
    type: "string"
    pattern: "^[a-z]{3}$"
    description: >
      "Three-letter abbreviation of the language in ISO 639-3 language code)"
      "format: https://en.wikipedia.org/wiki/ISO_639-3\n"
      "For example, English is 'eng'."
  multi_reference_aggregation:
    type: "string"
    description: >
      "The variety of aggregation to use when multiple references are given. "
    enum:
      - "max"
      - "mean"
      - "min"

For the model parameter, you can use the following:

  • facebook/bart-large-cnn

More models will be coming soon! Please get in contact if you’re interested in using a different model.

Data

Accepted data format of the BARTScore metric. Note that there is a size limit of 250 examples per query. If you want to submit more examples, you can use multiple queries.

title: "BartScoreData"
type: "object"
properties:
  target:
    type: "string"
    description: "Input text to evaluate."
  source:
    type: "string"
    description: "Source text."
  references:
    type: "array"
    description: "Gold reference texts"
    items:
      type: "string"
required:
  - "target"

Results

Format of the results of the BARTScore metric:

title: "BartScoreResult"
type: "object"
$defs:
  BartScoreStats:
    type: "object"
    properties:
      value:
        type: "number"
        description: "The main BartScore value."
    required:
      - "value"
properties:
  overall:
    $ref: "#/$defs/BartScoreStats"
  examples:
    type: "array"
    items:
      $ref: "#/$defs/BartScoreStats"
required:
  - "overall"
  - "examples"