Product Selector

Fusion 5.12
    Fusion 5.12

    Format Signals Index Stage

    The Format Signals stage normalizes both the fields and field contents of a PipelineDocument to ensure that certain pre-defined fields for signals are populated.

    Date/time parsing and formatting

    Timestamp data can be obtained from the following fields, in this order of precedence:

    • timestamp

    • timestamp_tdt

    • timestamp_dt

    • epoch. value in this field is treated as a number of milliseconds since epoch, and UTC zone is assumed.

    It is now possible to specify the locale to be used for parsing timestamps by setting the "timestampLocale" property in the stage configuration. If this property is null then the default platform locale will be used.

    Output document will carry the following two fields:

    • "timestamp" - containing the ISO8601 timestamp

    • "tz_timestamp_txt" - containing the "zoned format" of the timestamp with normalized components.

    This stage does not define a list of allowed types.

    Example Stage Specification

    The Format Signals stage defined as part of the default 'signals_ingest' pipeline included with Fusion:

    {
        "type" : "format-signals",
        "id" : "ingest-signals",
        "flatten" : true,
        "undefinedType" : "general",
        "skip" : false,
        "label" : "format-signals",
        "type" : "format-signals"
      }

    Configuration

    When entering configuration values in the UI, use unescaped characters, such as \t for the tab character. When entering configuration values in the API, use escaped characters, such as \\t for the tab character.

    This stage makes sure that incoming documents are properly formatted signal events.

    skip - boolean

    Set to true to skip this stage.

    Default: false

    label - string

    A unique label for this stage.

    <= 255 characters

    condition - string

    Define a conditional script that must result in true or false. This can be used to determine if the stage should process or not.

    flatten - boolean

    Flatten nested values.

    Default: true

    allowedTypes - array[string]

    List of allowed signal types.

    undefinedType - string

    Signal type when undefined. Null discards events with undefined type.

    timestampLocale - string

    Use this locale when parsing timestamp information. Null uses platform default locale.