Ranking Metrics Jobs

Configuration Properties

Calculate relevance metrics (nDCG and so on) by replaying ground truth queries against catalog data using variants from an experiment.

Configuration Properties

Property Description, Type

groundTruthConfig

Configure ground truth dataset

required

Configure properties for Ground truth dataset

type: object

object attributes: {
  docIdField : {
    display name: Doc ID field
    type: string
    default value: 'doc_id_s'
    description : Field containing ranked doc id's
    }
  filterQueries : {
    display name: Filter queries
    type: array of string
    default value: 'aggr_type_s:ground_truth'
    description : Solr filter queries to apply against Ground truth collection
    }
  inputCollection : {
    display name: Ground Truth Input Collection
    type: string
    description : Input collection representing ground truth dataset
    minLength : 1
    }
  queryField : {
    display name: Query field
    type: string
    default value: 'query_s'
    description : Query field in the collection
    }
  weightField : {
    display name: Weight Field
    type: string
    default value: 'weight_d'
    description : Field representing the weight of document to the query
    }
  }

Spark Job ID

required

The ID for this Spark job. Used in the API to reference this job. Allowed characters: a-z, A-Z, dash (-) and underscore (_)

type: string

maxLength: 128

pattern: ^[A-Za-z0-9_\-]+$

metricsPerQuery

Calculate metrics per query

Calculate ranking metrics per each query in ground truth set and save them to Solr collection

type: boolean

default value: 'true'

outputCollection

Output collection

required

Output collection to save the ranking metrics to

type: string

minLength: 1

rankingExperimentConfig

Configure experiment

required

Configure properties for the experiment

type: object

object attributes: {
  defaultProfile : {
    display name: Default Query Profile
    type: string
    description : Default query profile to use if not specified in experiment variants
    }
  docIdField : {
    display name: Doc Id Field
    type: string
    default value: 'id'
    description : Doc id field to retrieve values (Must return values that match the ground truth data)
    }
  experimentId : {
    display name: Experiment ID
    type: string
    description : Calculate ranking metrics using variants from experiment
    minLength : 1
    }
  experimentObjectiveName : {
    display name: Experiment metric name
    type: string
    description : Experiment objective name
    minLength : 1
    }
  inputCollection : {
    display name: Input Collection
    type: string
    description : Collection to run the experiment on
    minLength : 1
    }
  queryPipelines : {
    display name: Query pipelines
    type: array of string
    description : Pipeline variants for experiment
    }
  }

rankingPositionK

Ranking Position @K

Ranking position at K for metrics calculation

type: integer

default value: '10'

type

Spark Job Type

required

type: string

default value: 'ranking_metrics'

enum: { ranking_metrics }