The BARTScore metric.
BARTScore is a metric for evaluating text via text generation based on the BART sequence-to-sequence pre-trained model.
Configuration
Configuration for the parameters of the BARTScore metric:
title: "BartScoreConfig"
type: "object"
properties:
variety:
type: "string"
description: "Name of the metric variety to be used."
enum:
- "reference_to_target"
- "target_to_reference"
- "reference_target_bidirectional"
- "source_to_target"
- "target_to_source"
model:
type: "string"
pattern: "^[a-z0-9-/]+$"
description: "Model name"
language:
type: "string"
pattern: "^[a-z]{3}$"
description: >
"Three-letter abbreviation of the language in ISO 639-3 language code)"
"format: https://en.wikipedia.org/wiki/ISO_639-3\n"
"For example, English is 'eng'."
multi_reference_aggregation:
type: "string"
description: >
"The variety of aggregation to use when multiple references are given. "
enum:
- "max"
- "mean"
- "min"
For the model parameter, you can use the following:
facebook/bart-large-cnn
More models will be coming soon! Please get in contact if you’re interested in using a different model.
Data
Accepted data format of the BARTScore metric. Note that there is a size limit of 250 examples per query. If you want to submit more examples, you can use multiple queries.
title: "BartScoreData"
type: "object"
properties:
target:
type: "string"
description: "Input text to evaluate."
source:
type: "string"
description: "Source text."
references:
type: "array"
description: "Gold reference texts"
items:
type: "string"
required:
- "target"
Results
Format of the results of the BARTScore metric:
title: "BartScoreResult"
type: "object"
$defs:
BartScoreStats:
type: "object"
properties:
value:
type: "number"
description: "The main BartScore value."
required:
- "value"
properties:
overall:
$ref: "#/$defs/BartScoreStats"
examples:
type: "array"
items:
$ref: "#/$defs/BartScoreStats"
required:
- "overall"
- "examples"