Cluster Labeling Jobs

Configuration Properties

Use this job when you already have clusters or well-defined document categories, and you want to discover and attach keywords to see representative words within those existing clusters. (If you want to create new clusters, use the Document Clustering job.)

Configuration Properties

Property Description, Type

analyzerConfig

Lucene Analyzer Schema

LuceneTextAnalyzer schema for tokenization (JSON-encoded)

type: string

default value: '{ "analyzers": [{ "name": "StdTokLowerStop","charFilters": [ { "type": "htmlstrip" } ],"tokenizer": { "type": "standard" },"filters": [{ "type": "lowercase" },{ "type": "KStem" },{ "type": "length", "min": "2", "max": "32767" },{ "type": "fusionstop", "ignoreCase": "true", "format": "snowball", "words": "org/apache/lucene/analysis/snowball/english_stop.txt" }] }],"fields": [{ "regex": ".+", "analyzer": "StdTokLowerStop" } ]}'

minLength: 1

clusterIdField

Existing Document Category Field

required

Field that contains your existing cluster IDs or document categories.

type: string

minLength: 1

clusterLabelField

Top Unique Terms Field Name

Output field name for top frequent terms that are (mostly) unique for each cluster.

type: string

default value: 'cluster_label'

dataFormat

Data format

required

Spark-compatible format which training data comes in (like 'solr', 'hdfs', 'file', 'parquet' etc)

type: string

default value: 'solr'

minLength: 1

fieldToVectorize

Field to detect keywords from

required

Field containing data from which to discover keywords for the cluster

type: string

minLength: 1

freqTermField

Top Frequent Terms Field Name

Output field name for top frequent terms in each cluster. These may overlap with other clusters.

type: string

default value: 'freq_terms'

Spark Job ID

required

The ID for this Spark job. Used in the API to reference this job. Allowed characters: a-z, A-Z, dash (-) and underscore (_). Maximum length: 63 characters.

type: string

maxLength: 63

pattern: [a-zA-Z][_\-a-zA-Z0-9]*[a-zA-Z0-9]?

maxDF

Max Doc Support

Max number of documents the term can show up. value<1.0 denotes a percentage, value=1.0 denotes 100%, value>1.0 denotes the exact number.

type: number

default value: '0.75'

minDF

Min Doc Support

Min number of documents the term has to show up. value<1.0 denotes a percentage, value=1.0 denotes 100%, value>1.0 denotes the exact number.

type: number

default value: '5.0'

modelId

Model ID

Identifier for the model to be trained; uses the supplied Spark Job ID if not provided.

type: string

minLength: 1

norm

Vector normalization

p-norm to normalize vectors with (choose -1 to turn normalization off)

type: integer

default value: '2'

enum: { -1 0 1 2 }

numKeywordsPerLabel

Number of Keywords for Each Cluster

Number of Keywords needed for labeling each cluster.

type: integer

default value: '5'

outputCollection

Output Collection

required

Solr Collection to store output data to

type: string

minLength: 1

overwriteOutput

Overwrite Output

Overwrite output collection

type: boolean

default value: 'true'

randomSeed

Random seed

For any deterministic pseudorandom number generation

type: integer

default value: '1234'

readOptions

Read Options

Options used when reading input from Solr or other sources.

type: array of object

object attributes: {
  key (required) : {
    display name: Parameter Name
    type: string
    }
  value : {
    display name: Parameter Value
    type: string
    }
  }

sourceFields

Fields to Load

Solr fields to load (comma-delimited). Leave empty to allow the job to select the required fields to load at runtime.

type: string

sparkConfig

Spark Settings

Spark configuration settings.

type: array of object

object attributes: {
  key (required) : {
    display name: Parameter Name
    type: string
    }
  value : {
    display name: Parameter Value
    type: string
    }
  }

stopwordsList

List of stopwords

Stopwords defined in Lucene analyzer config

type: array of string

trainingCollection

Training Collection

required

Solr Collection containing documents with defined categories or clusters

type: string

minLength: 1

trainingDataFilterQuery

Training data filter query

Solr query to use when loading training data if using Solr, Spark SQL expression for all other data sources

type: string

default value: '*:*'

trainingDataFrameConfigOptions

Dataframe Config Options

Additional spark dataframe loading configuration options

type: object

object attributes: {
}

trainingDataSamplingFraction

Training data sampling fraction

Fraction of the training data to use

type: number

default value: '1.0'

exclusiveMaximum: false

maximum: 1.0

type

Spark Job Type

required

type: string

default value: 'cluster_labeling'

enum: { cluster_labeling }

writeOptions

Write Options

Options used when writing output to Solr.

type: array of object

object attributes: {
  key (required) : {
    display name: Parameter Name
    type: string
    }
  value : {
    display name: Parameter Value
    type: string
    }
  }