Product Selector

Fusion 5.12
    Fusion 5.12

    Ranking Metrics Jobs

    Use this job to calculate relevance metrics by replaying ground truth queries against catalog data using variants from an experiment. Metrics include Normalized Discounted Cumulative Gain (nDCG) and others.

    To create a Ranking Metrics job, sign in to Fusion and click Collections > Jobs. Then click Add+ and in the Experiment Evaluation Jobs section, select Ranking Metrics. You can enter basic and advanced parameters to configure the job. If the field has a default value, it is populated when you click to add the job.

    Basic parameters

    To enter advanced parameters in the UI, click Advanced. Those parameters are described in the advanced parameters section.

    • Spark job ID. The unique ID for the Spark job that references this job in the API. This is the id field in the configuration file. Required field.

    • Output collection. The Solr collection where the job output is stored. The job will write the output to this collection. This is the outputCollection field in the configuration file. Required field.

    • Ground Truth Parameters. This section includes this parameter:

      • Ground truth input collection. The collection that stores the ground truth dataset this job accesses. This is the inputCollection field in the configuration file. Required field.

    • Ranking Experiment Parameters. This section includes the following parameters:

      • Ranking experiment input collection. The collection that stores the experiment data this job accesses. This is the rankingExperimentConfig inputCollection field in the configuration file. Optional field.

      • Experiment ID. The identifier for the experiment that stores the variants this job uses to calculate ranking metrics. This is the rankingExperimentConfig experimentId field in the configuration file. Optional field.

      • Experiment metric name. The name of the purpose (objective) of the experiment this job accesses to calculate ranking metrics. This is the rankingExperimentConfig experimentObjectiveName field in the configuration file. Optional field.

      • Default query profile. The name of the query profile this job defaults to if the value is not specified in the experiment variants. This is the rankingExperimentConfig defaultProfile field in the configuration file. Optional field.

    Advanced parameters

    If you click the Advanced toggle, the following optional fields are displayed in the UI.

    • Spark Settings. This section lets you enter parameter name:parameter value options to use in this job. This is the sparkConfig field in the configuration file.

    • Ranking position @K. The number of returned or recommended items that are ranked (based on the relevancy rating) that are used for metrics calculation. This is the rankingPositionK field in the configuration file.

    • Calculate metrics per query. If this checkbox is selected (set to true), the job calculates the ranking metrics per query in the ground truth dataset, and saves the metrics data to the Output collection designated for this job. This is the metricsPerQuery field in the configuration file.

    • Ground Truth Parameters. The advanced option adds these parameters:

      • Filter queries. The Solr filter queries this job applies against the ground truth collection to calculate ranking metrics. This is the groundTruthConfig filterQueries field in the configuration file.

      • Query field. The query field in the ground truth collection. This is the groundTruthConfig queryField field in the configuration file.

      • Doc ID field. This field contains the ranked document IDs in the collection. This is the groundTruthConfig docIdField field in the configuration file.

      • Weight field. This field contains the weight of the document as it relates to the query. This is the groundTruthConfig weightField field in the configuration file.

    • Ranking Experiment Parameters. The advanced option adds these parameters:

      • Query pipelines. These are the query pipelines for the experiment that stores the variants this job uses to calculate ranking metrics. This is the rankingExperimentConfig queryPipelines field in the configuration file.

      • Doc ID field. This field contains the values (that match the ground truth data) this job uses to calculate ranking metrics. This is the rankingExperimentConfig docIdField field in the configuration file.

    Configuration properties

    use this job to calculate relevance metrics (nDCG etc..) by replaying ground truth queries (see ground truth job) against catalog data using variants from an experiment.

    id - stringrequired

    The ID for this Spark job. Used in the API to reference this job. Allowed characters: a-z, A-Z, dash (-) and underscore (_). Maximum length: 63 characters.

    <= 63 characters

    Match pattern: [a-zA-Z][_\-a-zA-Z0-9]*[a-zA-Z0-9]?

    sparkConfig - array[object]

    Spark configuration settings.

    object attributes:{key required : {
     display name: Parameter Name
     type: string
    }    value : {
     display name: Parameter Value
     type: string
    }    }

    groundTruthConfig - Configure ground truth dataset

    Configure properties for Ground truth dataset

    inputCollection - string

    Input collection representing ground truth dataset

    >= 1 characters

    filterQueries - array[string]

    Solr filter queries to apply against Ground truth collection

    Default: "type:ground_truth"

    queryField - string

    Query field in the collection

    Default: query

    docIdField - string

    Field containing ranked doc id's

    Default: docId

    weightField - string

    Field representing the weight of document to the query

    Default: weight

    rankingExperimentConfig - Configure experiment

    Configure properties for the experiment

    inputCollection - string

    Collection to run the experiment on

    >= 1 characters

    queryPipelines - array[string]

    Pipeline variants for experiment

    docIdField - string

    Doc id field to retrieve values (Must return values that match the ground truth data)

    Default: id

    experimentId - string

    Calculate ranking metrics using variants from experiment

    >= 1 characters

    experimentObjectiveName - string

    Experiment objective name

    >= 1 characters

    defaultProfile - string

    Default query profile to use if not specified in experiment variants

    outputCollection - stringrequired

    Output collection to save the ranking metrics to

    >= 1 characters

    rankingPositionK - integer

    Ranking position at K for metrics calculation

    Default: 10

    metricsPerQuery - boolean

    Calculate ranking metrics per each query in ground truth set and save them to Solr collection

    Default: true

    type - stringrequired

    Default: ranking_metrics

    Allowed values: ranking_metrics