Ranking MetricsJob configuration specifications
Use this job to calculate relevance metrics by replaying ground truth queries against catalog data using variants from an experiment. Metrics include Normalized Discounted Cumulative Gain (nDCG) and others.
To create a Ranking Metrics job, sign in to Managed Fusion and click Collections > Jobs. Then click Add+ and in the Experiment Evaluation Jobs section, select Ranking Metrics. You can enter basic and advanced parameters to configure the job. If the field has a default value, it is populated when you click to add the job.
Basic parameters
To enter advanced parameters in the UI, click Advanced. Those parameters are described in the advanced parameters section. |
-
Spark job ID. The unique ID for the Spark job that references this job in the API. This is the
id
field in the configuration file. Required field. -
Output collection. The Solr collection where the job output is stored. The job will write the output to this collection. This is the
outputCollection
field in the configuration file. Required field. -
Ground Truth Parameters. This section includes this parameter:
-
Ground truth input collection. The collection that stores the ground truth dataset this job accesses. This is the
inputCollection
field in the configuration file. Required field.
-
-
Ranking Experiment Parameters. This section includes the following parameters:
-
Ranking experiment input collection. The collection that stores the experiment data this job accesses. This is the
rankingExperimentConfig
inputCollection
field in the configuration file. Optional field. -
Experiment ID. The identifier for the experiment that stores the variants this job uses to calculate ranking metrics. This is the
rankingExperimentConfig
experimentId
field in the configuration file. Optional field. -
Experiment metric name. The name of the purpose (objective) of the experiment this job accesses to calculate ranking metrics. This is the
rankingExperimentConfig
experimentObjectiveName
field in the configuration file. Optional field. -
Default query profile. The name of the query profile this job defaults to if the value is not specified in the experiment variants. This is the
rankingExperimentConfig
defaultProfile
field in the configuration file. Optional field.
-
Advanced parameters
If you click the Advanced toggle, the following optional fields are displayed in the UI.
-
Spark Settings. This section lets you enter
parameter name:parameter value
options to use in this job. This is thesparkConfig
field in the configuration file. -
Ranking position @K. The number of returned or recommended items that are ranked (based on the relevancy rating) that are used for metrics calculation. This is the
rankingPositionK
field in the configuration file. -
Calculate metrics per query. If this checkbox is selected (set to
true
), the job calculates the ranking metrics per query in the ground truth dataset, and saves the metrics data to the Output collection designated for this job. This is themetricsPerQuery
field in the configuration file. -
Ground Truth Parameters. The advanced option adds these parameters:
-
Filter queries. The Solr filter queries this job applies against the ground truth collection to calculate ranking metrics. This is the
groundTruthConfig
filterQueries
field in the configuration file. -
Query field. The query field in the ground truth collection. This is the
groundTruthConfig
queryField
field in the configuration file. -
Doc ID field. This field contains the ranked document IDs in the collection. This is the
groundTruthConfig
docIdField
field in the configuration file. -
Weight field. This field contains the weight of the document as it relates to the query. This is the
groundTruthConfig
weightField
field in the configuration file.
-
-
Ranking Experiment Parameters. The advanced option adds these parameters:
-
Query pipelines. These are the query pipelines for the experiment that stores the variants this job uses to calculate ranking metrics. This is the
rankingExperimentConfig
queryPipelines
field in the configuration file. -
Doc ID field. This field contains the values (that match the ground truth data) this job uses to calculate ranking metrics. This is the
rankingExperimentConfig
docIdField
field in the configuration file.
-