The COMET metric.

COMET is a metric mainly used for machine translation evaluation based on learned embeddings.

Configuration

Configuration for the parameters of the COMET metric:

title: "CometConfig"
type: "object"
properties:
  model:
    type: "string"
    pattern: "^[a-z0-9-_/]+$"
    description: "Model name"
  multi_reference_aggregation:
    type: "string"
    description: >
      "The variety of aggregation to use when multiple references are given. "
    enum:
      - "max"
      - "mean"
      - "min"

For the model parameter, you can use the following:

  • unbabel_comet/wmt21-comet-qe-da

More models are coming soon! Please get in contact with us if there are other models you’d like to use.

Data

Accepted data format of the COMET metric. Note that there is a size limit of 250 examples per query. If you want to submit more examples, you can use multiple queries.

title: "CometData"
type: "object"
properties:
  target:
    type: "string"
    description: "Input text to evaluate."
  source:
    type: "string"
    description: "Source text."
  references:
    type: "array"
    description: "Gold reference texts"
    items:
      type: "string"
required:
  - "target"

Results

Format of the results of the COMET metric:

title: "CometResult"
type: "object"
$defs:
  CometStats:
    type: "object"
    properties:
      value:
        type: "number"
        description: "The main Comet value."
    required:
      - "value"
properties:
  overall:
    $ref: "#/$defs/CometStats"
  examples:
    type: "array"
    items:
      $ref: "#/$defs/CometStats"
required:
  - "overall"
  - "examples"