Aggregation Jobs

Configuration Properties

Define an aggregation job.

Configuration Properties

Property Description, Type

aggregationTime

Aggregation Time

Timestamp to use for the aggregation results. Defaults to NOW.

type: string

format: date-time

definition

Aggregation Settings

Defines the type of aggregation to perform, either SQL or legacy. SQL aggregations allow you to use ANSI SQL 2003, including numerous built-in functions to define your aggregation and rollup logic. The legacy aggregation option is based on pre-Fusion 4.0 features and will be removed in Fusion 4.1.

type: object

object attributes: {
  aggregates : {
    display name: Aggregates
    type: array of object
    }
  aggregator : {
    display name: Aggregator
    type: string
    description : Aggregator implementation to use. This is either one of the symbolic names (simple, click, em) or a fully-qualified class name of a class extending EventAggregator. If not set then 'simple' is used.
    minLength : 1
    }
  groupingFields : {
    display name: Grouping Fields
    type: array of string
    description : The fields to group on
    }
  id : {
    display name: Aggregation
    type: string
    description : The unique id of the Aggregation definition. If not set, then a UUID will be generated.
    minLength : 1
    }
  outputCollection : {
    display name: Output Collection
    type: string
    description : The collection to write the aggregates to on output. This property is required if the selected output / rollup pipeline requires it (the default pipeline does). A special value of '-' disables the output.
    minLength : 1
    }
  outputPipeline : {
    display name: Output Pipeline
    type: string
    default value: '_system'
    description : What pipeline to use to process the output. If not set then '_system' pipeline will be used.
    minLength : 1
    }
  outputRollup : {
    display name: Rollup Output
    type: boolean
    default value: 'true'
    description : Roll-up current results with all previous results for this aggregation id, which are available in "outputCollection".
    }
  parameters : {
    display name: Parameters
    type: array of object
    }
  rollupAggregator : {
    display name: Rollup Aggregator
    type: string
    description : The aggregator to use when rolling up. If not set then the same aggregator will be used for roll-up.
    minLength : 1
    }
  rollupPipeline : {
    display name: Rollup Pipeline
    type: string
    description : Pipeline to use for processing results of roll-up. This is by default the same indexing pipeline used for processing the aggregation results.
    minLength : 1
    }
  rollupSql : {
    display name: Rollup SQL
    type: string
    description : Use SQL to perform a rollup of previously aggregated docs. If left blank, the aggregation framework will supply a default SQL query to rollup aggregated metrics.
    minLength : 1
    }
  selectQuery : {
    display name: Query
    type: string
    default value: '*:*'
    description : The query to select the desired signals. If not set then '*:*' will be used, or equivalent.
    minLength : 1
    }
  signalTypes : {
    display name: Signal Types
    type: array of string
    description : The signal types. If not set then any signal type is selected
    }
  sort : {
    display name: Sort Criteria
    type: string
    description : The criteria to sort on within a group. If not set then sort order is by id, ascending.
    minLength : 1
    }
  sourceCatchup : {
    display name: Catchup
    type: boolean
    default value: 'true'
    description : If true, only aggregate the signals since the last time the job was successfully run. If there is a record of such previous run then this overrides the starting time of time range set in 'timeRange' property.
    }
  sourceRemove : {
    display name: Remove Source
    type: boolean
    default value: 'false'
    description : If true, the processed source signals will be removed after aggregation. Default is false.
    }
  sql : {
    display name: SQL
    type: string
    description : Use SQL to perform the aggregation. You do not need to include a time range filter in the WHERE clause as it gets applied automatically before executing the SQL statement.
    minLength : 1
    }
  statsFields : {
    display name: Stats fields
    type: array of string
    description : List of numeric fields in results for which to compute overall statistics. Not supported for SQL aggregations.
    }
  timeRange : {
    display name: Time Range
    type: string
    description : The time range to select signals on.
    minLength : 1
    }
  typeFieldName : {
    display name: Type Field
    type: string
    description : Name of the signal type field; defaults to 'type'
    }
  }

object attributes: {
   : {
    display name:
    type:
    }
   : {
    display name:
    type:
    }
  }

Spark Job ID

required

The ID for this Spark job. Used in the API to reference this job

type: string

maxLength: 128

pattern: ^[A-Za-z0-9_\-]+$

inputCollection

Source Collection

required

Collection containing signals to be aggregated.

type: string

readOptions

Read Options

Additional configuration settings to fine-tune how input records are read for this aggregation.

type: array of object

object attributes: {
  key (required) : {
    display name: Parameter Name
    type: string
    }
  value : {
    display name: Parameter Value
    type: string
    }
  }

referenceTime

Reference Time

Timestamp to use for computing decays and to determine the value of NOW.

type: string

format: date-time

rows

Batch Size

Number of rows to read from the source collection per request.

type: integer

default value: '10000'

skipCheckEnabled

Job Skip Check Enabled?

If the catch-up flag is enabled and this field is checked, the job framework will execute a fast Solr query to determine if this run can be skipped.

type: boolean

default value: 'true'

type

Spark Job Type

required

type: string

default value: 'aggregation'

enum: { aggregation }

Was this page helpful?