Product Selector

Fusion 5.11
    Fusion 5.11

    Custom Spark Jobs

    Run a custom Spark job.

    Use this job when you want to run a custom spark job.

    id - stringrequired

    The ID for this Spark job. Used in the API to reference this job. Allowed characters: a-z, A-Z, dash (-) and underscore (_)

    <= 128 characters

    Match pattern: ^[A-Za-z0-9_\-]+$

    resourceName - stringrequired

    Name of the resource uploaded to Blob store. This should match with the Blob name

    >= 1 characters

    mainClass - string

    Application's main class (for Java/Scala)

    files - array[string]

    Blob resource (files) to be placed in the working directory of each executor

    pythonFiles - array[string]

    Blob resource (.zip, .egg, .py files) to place on the PYTHONPATH for Python apps

    sparkConfig - array[object]

    Spark configuration settings.

    Default: {"key":"spark.executor.memory","value":"2g"}{"key":"spark.driver.memory","value":"2g"}{"key":"spark.logConf","value":"true"}{"key":"spark.eventLog.enabled","value":"true"}{"key":"spark.eventLog.compress","value":"true"}{"key":"spark.scheduler.mode","value":"FAIR"}

    object attributes:{key required : {
     display name: Parameter Name
     type: string
    }
    value : {
     display name: Parameter Value
     type: string
    }
    }

    submitArgs - array[string]

    Additional options to pass to the Spark Submit when running this job.

    javaOptions - array[object]

    Java options to pass to Spark driver/executor

    object attributes:{key required : {
     display name: Parameter Name
     type: string
    }
    value : {
     display name: Parameter Value
     type: string
    }
    }

    verboseReporting - boolean

    Enables verbose reporting for SparkSubmit

    Default: true

    deleteOnExit - boolean

    Remove all temp files on exit

    Default: true

    envOptions - array[object]

    Set environment variables for driver

    object attributes:{key required : {
     display name: Parameter Name
     type: string
    }
    value : {
     display name: Parameter Value
     type: string
    }
    }

    type - stringrequired

    Default: custom_spark_job

    Allowed values: custom_spark_job