Use an Experiment stage

Here we describe how to set up an experiment that uses an Experiment query pipeline stage. This approach is an alternative to setting up an experiment and a query profile that references it. Using an Experiment query pipeline stage is a bit more complicated.

How an Experiment stage works

An Experiment stage applies the experiment actions in-line in the query pipeline (wherever it is located in the pipeline), instead of performing the actions before passing queries to a query pipeline or pipelines.

An Experiment stage that apportions traffic among query pipelines is similar to a Run Query Pipeline stage, but the processing is conditional. Queries that lack a user ID parameter aren’t sent processed by the stage and aren’t sent to other pipelines (if that is what the stage does).

For the primary pipeline to process queries that don’t include a user identifier, it must contain a Solr Query stage as the last stage. If the Experiment query stage references other pipelines, then there are two options:

  • Solr Query stage as the last stage in the variant pipelines – The variant pipelines send queries to Solr. Control doesn’t return to the primary pipeline. In the primary pipeline, the Experiment stage must be the second-to-last stage. The last stage must be the Solr Query stage.

  • No Solr Query stage in the variant pipelines – The variant pipelines don’t send queries to Solr. Control returns from the variant pipelines to the primary pipeline. In the primary pipeline, the Experiment stage can be in any position except the last.

Before you begin

Before you set up an experiment, you must already have:

  • A search app – The key aspect of the search app for experiments is that the search app identifies users in some way. A user ID might be associated with users persistently (this is best) or only for the current session. Searches and subsequent actions by anonymous users aren’t sent through an experiment.

  • A Fusion app – The Fusion app provides the search functionality you want to provide. Below, you’ll modify the app to include an experiment.

  • Data for users to search – The app should have data that users can search and search results that users can interact with. Typically, users will search a single dataset in the sense that different users aren’t given search results from different datasets. But in an experiment, different experiment variants can use data in different collections.

  • Results for users to interact with – Experiment metrics depend on users interacting with search results, for example, clicking on them. A search app uses signals to report the interactions to Fusion.

  • A plan for the experiment – This plan includes which control and variants to compare, the projected traffic, sample sizes, experiment duration, metrics, and decision criteria.

Basically, you need a working system in some environment on which you want to perform experiments, and a plan for experiment variants and for deciding which results are best.

1. Create an experiment

Create an experiment. The experiment defines variants and metrics, as well as the user ID parameter and the base collection for signals:

  1. Navigate to Analytics Analytics > Experiments.

  2. Click New.

  3. Enter an arbitrary ID (name) for the experiment.

  4. Verify that the unique ID parameter is correct. It’s the parameter that uniquely identifies each user. The default is userId. Correct the parameter if necessary.

  5. Choose the base collection for signals. Signals resulting from requests that flow through the experiment are stored in the _signals collection associated with this collection.

  6. (Optional) Enter a description for the experiment.

    Configure experiment in Experiments

  7. Add variants. Click Add Variant to add each non-control variant in your experiment.

    Add variants in experiment

  8. For each variant:

    1. Enter an arbitrary name. For the first variant, which is the control, Fusion uses the name control. You can change that name if you wish.

    2. Click Specify what varies and specify what varies. Items you select are visible in the variant UI and have a green check mark in the dropdown menu. You can vary the query pipeline, query parameters (URL parameters to add to the query), and/or the collection.

      Specify what varies

    3. (For query parameters) Click New params. In the dialog box, specify the Parameter Name, Parameter Value, and Update Policy for each parameter (append, default, remove, or replace).

  1. Add metrics. For each metric:

    1. Click Add Metric and select the type of metric.

    2. Fill in information for the metric.

    Add metrics in experiment

  2. Click Save to save the experiment.

2. Set up an Experiment stage

If part or all of what you will vary in the experiment is encompassed by differences in query pipelines, create additional pipelines for experiment variants. You can’t use the default query pipeline (the pipeline to which you are adding the Experiment stage) as one of the variants. That pipeline will be a part of all variants. Fusion directs traffic that doesn’t identify users through the default pipeline but not through the experiment.

To set up an experiment stage
  1. Navigate to Querying Querying > Query Pipelines.

  2. Click the name of the pipeline to which you want to add the Experiment stage.

  3. Click Add a new pipeline stage and select Experiment stage.

    Add an experiment stage

  4. (Optional) Specify a label for the stage.

  5. (Optional) Specify a condition that must be satisfied for queries to pass through the experiment.

  6. Under Experiment ID, choose the experiment.

  7. (Optional) Specify the percent of traffic to include in the experiment.

  8. Click Save.

  9. Drag the stage to where you want it in the pipeline.

  10. Click Save.

You control the experiment in Analytics Analytics > Experiments.