Custom Spark Jobs

Configuration Properties

Run a custom Spark job.

Configuration Properties

Property Description, Type

deleteOnExit

Delete launcher dir

Remove all temp files on exit

type: boolean

default value: 'true'

files

Additional Files

Blob resource (files) to be placed in the working directory of each executor

type: array of string

Spark Job ID

required

The ID for this Spark job. Used in the API to reference this job

type: string

maxLength: 128

pattern: ^[A-Za-z0-9_\-]+$

javaOptions

Java options

Java options to pass to Spark driver/executor

type: array of object

object attributes: {
  key (required) : {
    display name: Parameter Name
    type: string
    }
  value : {
    display name: Parameter Value
    type: string
    }
  }

mainClass

Main class name

Application's main class (for Java/Scala)

type: string

pythonFiles

Python Files

Blob resource (.zip, .egg, .py files) to place on the PYTHONPATH for Python apps

type: array of string

resourceName

Blob Resource (JAR or python file)

required

Name of the resource uploaded to Blob store. This should match with the Blob name

type: string

sparkConfig

Spark Settings

Spark configuration settings.

type: array of object

default value: '{"key"=>"spark.executor.memory", "value"=>"2g"}{"key"=>"spark.driver.memory", "value"=>"2g"}{"key"=>"spark.logConf", "value"=>"true"}{"key"=>"spark.eventLog.enabled", "value"=>"true"}{"key"=>"spark.eventLog.compress", "value"=>"true"}{"key"=>"spark.scheduler.mode", "value"=>"FAIR"}'

object attributes: {
  key (required) : {
    display name: Parameter Name
    type: string
    }
  value : {
    display name: Parameter Value
    type: string
    }
  }

submitArgs

Spark args

Additional options to pass to the Spark Submit when running this job.

type: array of string

type

Spark Job Type

required

type: string

default value: 'custom_spark_job'

enum: { custom_spark_job }

verboseReporting

Verbose reporting

Enables verbose reporting for SparkSubmit

type: boolean

default value: 'true'