Use this job to calculate relevance metrics by replaying ground truth queries against catalog data using variants from an experiment. Metrics include Normalized Discounted Cumulative Gain (nDCG) and others. To create a Ranking Metrics job, sign in to Fusion and click Collections > Jobs. Then click Add+ and in the Experiment Evaluation Jobs section, select Ranking Metrics. You can enter basic and advanced parameters to configure the job. If the field has a default value, it is populated when you click to add the job.

​
Basic parameters

To enter advanced parameters in the UI, click Advanced. Those parameters are described in the advanced parameters section.
  • Spark job ID. The unique ID for the Spark job that references this job in the API. This is the id field in the configuration file. Required field.
  • Output collection. The Solr collection where the job output is stored. The job will write the output to this collection. This is the outputCollection field in the configuration file. Required field.
  • Ground Truth Parameters. This section includes this parameter:
    • Ground truth input collection. The collection that stores the ground truth dataset this job accesses. This is the inputCollection field in the configuration file. Required field.
  • Ranking Experiment Parameters. This section includes the following parameters:
    • Ranking experiment input collection. The collection that stores the experiment data this job accesses. This is the rankingExperimentConfig inputCollection field in the configuration file. Optional field.
    • Experiment ID. The identifier for the experiment that stores the variants this job uses to calculate ranking metrics. This is the rankingExperimentConfig experimentId field in the configuration file. Optional field.
    • Experiment metric name. The name of the purpose (objective) of the experiment this job accesses to calculate ranking metrics. This is the rankingExperimentConfig experimentObjectiveName field in the configuration file. Optional field.
    • Default query profile. The name of the query profile this job defaults to if the value is not specified in the experiment variants. This is the rankingExperimentConfig defaultProfile field in the configuration file. Optional field.

​
Advanced parameters

If you click the Advanced toggle, the following optional fields are displayed in the UI.
  • Spark Settings. This section lets you enter parameter name:parameter value options to use in this job. This is the sparkConfig field in the configuration file.
  • Ranking position @K. The number of returned or recommended items that are ranked (based on the relevancy rating) that are used for metrics calculation. This is the rankingPositionK field in the configuration file.
  • Calculate metrics per query. If this checkbox is selected (set to true), the job calculates the ranking metrics per query in the ground truth dataset, and saves the metrics data to the Output collection designated for this job. This is the metricsPerQuery field in the configuration file.
  • Ground Truth Parameters. The advanced option adds these parameters:
    • Filter queries. The Solr filter queries this job applies against the ground truth collection to calculate ranking metrics. This is the groundTruthConfig filterQueries field in the configuration file.
    • Query field. The query field in the ground truth collection. This is the groundTruthConfig queryField field in the configuration file.
    • Doc ID field. This field contains the ranked document IDs in the collection. This is the groundTruthConfig docIdField field in the configuration file.
    • Weight field. This field contains the weight of the document as it relates to the query. This is the groundTruthConfig weightField field in the configuration file.
  • Ranking Experiment Parameters. The advanced option adds these parameters:
    • Query pipelines. These are the query pipelines for the experiment that stores the variants this job uses to calculate ranking metrics. This is the rankingExperimentConfig queryPipelines field in the configuration file.
    • Doc ID field. This field contains the values (that match the ground truth data) this job uses to calculate ranking metrics. This is the rankingExperimentConfig docIdField field in the configuration file.

​
Configuration properties