Signals API

The Signals API accepts a set of signals, encoded as JSON objects, for indexing into a signals collection.

Normally, signals are indexed just like ordinary documents, through a configured datasource and index pipeline. This API is provided for cases where it is more convenient to index signals directly.

To aggregate signals, use a SQL aggregation job, which is a kind of Spark job.

Tip
You can manage aggregation jobs in the Jobs manager and Scheduler in the Fusion UI, or with the Spark jobs API. We recommend using the Fusion UI.

Signals types and structure

There are three main types of signals:

The signal type parameter can also take arbitrary values for custom signal types. For example, you can create special signals for purchase events, cart addition/subtraction events, "favorite" or "like" events, customer service events, and so on. Custom signals can be analyzed in App Insights just like pre-defined signal types.

Request signals

A request signal is generated by a front-end search app and captures the raw user query and other contextual information about a user and their journey through the search app. A request signal should have the following fields:

[
  {
    "id":"288fe4f7-6680-403e-8d18-27647cdd9989",
    "timestamp":1518717749409,
    "type":"request",
    "params":{
      "user_id":"admin",
      "session":"ef4e00cd-91bb-45b4-be80-e81f9f9c5b27",
      "query":"USER QUERY HERE",
      "app_id":"SEARCH APP ID",
      "ip_address":"0:0:0:0:0:0:0:1",
      "host":"Lucids-MacBook-Pro-5.local",
      "filter":[
        "field1/value",
        ...
      ],
      "filter_field":[
        "field1"
      ]
    }
  }
]

Additional optional fields are used by App Insights. In the raw signal, optional fields should be inside the params object. Optional fields are as follows:

"page_title":"Fusion Search",
"path":"/search",
"browser_type":"Browser",
"browser_version":"64.0.3282.140",
"browser_name":"Chrome",
"referrer":"http://localhost:8080/",
"ctx_prev_uri":"/",
"ctx_prev_query":"",
"ctx_prev_path":"/",
"os_manufacturer":"Apple Inc.",
"os_name":"Mac OS X",
"os_id":"778",
"os_device":"Computer",
"os_group":"Mac OS X"

Response signals

Response signals are automatically generated by a query pipeline when the signals feature is enabled for a collection.

Note
Front-end search applications should not send response signals to Fusion directly, as those would conflict with the auto-generated signals.

A response signal has the following explicit fields, plus any additional query parameters sent by the search application for a query:

Field Name Description Example

id

The x-fusion-query-id generated by the query-pipeline used for associating click signals with queries in experiments and aggregation jobs.

TwWCn3Dz

type

Signal type

response

response_type

Used by Insights to determine if this query had results or was empty

results | empty

session

User session ID; the search app should pass the session ID in the query params for a query

UUID

query

The actual query string sent to Solr from Fusion

ipad

query_orig_s

The incoming query from the search app before it is enriched by the query pipeline

ipad

query_id

A hash generated from the session, query, and filters fields; used as a rollup key in Insights to group activity by a specific

SHA1 hash

filters_s

Filter queries sent to Solr; the Fusion SearchLogger component combines multiple fq parameters into a single value delimited by " $ "

{!tag=format}format:(vhs) $ {!tag=type}type:(movie)

filter

Reformatted filter queries for use by App Insights

field1/value

user_id

User ID; the search app should pass the user_id in the query params

admin

doc_ids_s

A comma-delimited list of document IDs returned for the page of results; this field is used by Fusion Spark jobs, such as the ground truth job, to perform click/skip analysis

123,456,789

pipeline_id

Fusion query pipeline that processed this query

_system

collection

Fusion collection

my_collection

qtime

Query time from Solr, in milliseconds

10

rows

Number of rows requested for this query

10

hits

Total number of documents matching the query

10000

totaltime

Total processing time of this query in milliseconds, includes Solr qtime and Fusion query processing time

15

timestamp_tdt

Timestamp when the query request was received by Fusion

2018-02-15T18:17:42.560Z

res_offset

Offset of results; this field is used by experiment metrics to calculate MRR

0

params.*

Any other query param sent from the search app to Fusion that was not already mapped to a declared field

params.defType_s=edismax

Fusion’s experiment framework relies heavily on response signals and the linking between response and clicks signals using the fusion_query_id.

Click signals

Click signals are sent from the search app to Fusion. All click signals should include a fusion_query_id field pulled from the query response header x-fusion-query-id. In addition, click signals should include the following fields:

[
  {
    "id":"SOME UUID HERE",
    "timestamp":1518725351750,
    "type":"click",
    "params":{
      "fusion_query_id":"ABkaEA11",
      "user_id":"admin",
      "session":"b3a15101-9e30-4e28-8a23-d1f663c2ee06",
      "query":"tiger woods",
      "ctype":"result",
      "res_offset":0,
      "filter":[
        "type/Game"
      ],
      "ip_address":"0:0:0:0:0:0:0:1",
      "host":"Lucids-MacBook-Pro-5.local",
      "doc_id":"9502308",
      "app_id":"SEARCH APP ID",
      "res_pos":1,
      "filter_field":[
        "type"
      ]
    }
  }
]

Additional optional fields are used by App Insights. In the raw signal, optional fields should be inside the params object. Optional fields are as follows:

"browser_type":"Browser",
"browser_version":"64.0.3282.140",
"browser_name":"Chrome",
"referrer":"http://localhost:8080/",
"ctx_prev_uri":"/",
"ctx_prev_query":"",
"ctx_prev_path":"/",
"os_manufacturer":"Apple Inc.",
"os_name":"Mac OS X",
"os_id":"778",
"os_device":"Computer",
"os_group":"Mac OS X"
"url":"http://localhost:8080/#/product/9502308",
"label":"Tiger Woods PGA Tour 09 All-Play - Nintendo Wii",

Examples

Send two signal events to record user clicks:

REQUEST

curl \
 -u user:pass -X POST -H 'Content-type:application/json' -d @- \
http://localhost:8764/api/signals/docs?commit=true \
<<EOF
[
 {"params": {
      "query": "Televisiones Panasonic  50 pulgadas",
      "filterQueries": ["cat00000","abcat0100000", "abcat0101000", "abcat0101001"],
      "docId": "2125233" },
 "type":"click",
 "timestamp": "2011-09-01T23:44:52.533000Z"
 },
 {"params": {
       "query": "Sharp",
       "filterQueries": ["cat00000", "abcat0100000", "abcat0101000", "abcat0101001"],
       "docId": "2009324" },
  "type":"click",
  "timestamp": "2011-09-05T12:25:37.420000Z"
 }
]
EOF

A successful request results in events being added to the signals collection. For the above example, the events will be represented as follows:

 {
  "responseHeader":{
    "status":0,
    "QTime":1,
    "params":{
      "indent":"true",
      "q":"doc_id_s: 2125233",
      "wt":"json"}
 },
  "response":{"numFound":1198,"start":0,"docs":[
      {
        "id": "7aee7b1f-5cde-4957-b73c-c15881f559ec",
        "filters_s": "abcat0100000 $ abcat0101000 $ abcat0101001 $ cat00000",
        "query_orig_s": "Televisiones Panasonic  50 pulgadas",
        "params.user_s": "000000df17cd56a5df4a94074e133c9d4739fae3",
        "doc_id_s": "2125233",
        "params.query_time__s": "2011-09-01T23:43:59.752Z",
        "query_t": "televisiones panasonic 50 pulgadas",
        "query_s": "televisiones panasonic 50 pulgadas",
        "filters_orig_ss": [
          "abcat0100000",
          "abcat0101000",
          "abcat0101001",
          "cat00000"
        ],
        "flag_s": "EVENT",
        "type_s": "click",
        "attr_params.filterQueries_": [
          "cat00000",
          "abcat0100000",
          "abcat0101000",
          "abcat0101001"
        ],
        "timestamp_dt": "2011-09-01T23:44:52.533Z",
        "_version_": 1478892846857584600
      },
      {
        "id": "6789a209-f5b5-457e-9df6-8033b8f7f317",
        "filters_s": "abcat0100000 $ abcat0101000 $ abcat0101001 $ cat00000",
        "query_orig_s": "Sharp",
        "params.user_s": "000001928162247ffaf63185cd8b2a244c78e7c6",
        "doc_id_s": "2009324",
        "params.query_time__s": "2011-09-05T12:25:01.187Z",
        "query_t": "sharp",
        "query_s": "sharp",
        "filters_orig_ss": [
          "abcat0100000",
          "abcat0101000",
          "abcat0101001",
          "cat00000"
        ],
        "flag_s": "EVENT",
        "type_s": "click",
        "attr_params.filterQueries_": [
          "cat00000",
          "abcat0100000",
          "abcat0101000",
          "abcat0101001"
        ],
        "timestamp_dt": "2011-09-05T12:25:37.42Z",
        "_version_": 1478892846859681800
      }
    ]
  }
}