Product Selector

Fusion 5.9
    Fusion 5.9

    Custom Spark Jobs

    Custom Spark jobs run Java ARchive (JAR) files uploaded to the blob store.

    To configure a custom Spark job, sign in to Fusion and click Collections > Jobs. Then click Add+ and in the Custom and Other Jobs section, select Custom Spark Job. You can enter basic and advanced parameters to configure the job. If the field has a default value, it is populated when you click to add the job.

    Basic parameters

    To enter advanced parameters in the UI, click Advanced. Those parameters are described in the advanced parameters section.

    • Spark job ID. The unique ID for the Spark job that references this job in the API. This is the id field in the configuration file. Required field.

    • Class name. The fully-qualified name of the Java/Scala class to invoke for this job. This is the klassName field in the configuration file. Required field.

    • Script ARGs. The arguments (ARGs), which are additional options to pass to the application when running this job. This is the submitArgs field in the configuration file. Optional field.

    Advanced parameters

    If you click the Advanced toggle, the following optional fields are displayed in the UI.

    • Spark Settings. This section lets you enter parameter name:parameter value options to configure Spark settings. This is the sparkConfig field in the configuration file.

    • Scala Script. The value in this text field overrides running className.main(args) default behavior. This is the script field in the configuration file.

    Alternatives to custom Spark jobs are Custom Python Jobs and Scala Script Jobs.

    Configuration properties

    Use this job when you want to run a custom JAR on Spark

    id - stringrequired

    The ID for this Spark job. Used in the API to reference this job. Allowed characters: a-z, A-Z, dash (-) and underscore (_). Maximum length: 63 characters.

    <= 63 characters

    Match pattern: [a-zA-Z][_\-a-zA-Z0-9]*[a-zA-Z0-9]?

    sparkConfig - array[object]

    Spark configuration settings.

    object attributes:{key required : {
     display name: Parameter Name
     type: string
    }    value : {
     display name: Parameter Value
     type: string
    }    }

    klassName - stringrequired

    Fully-qualified name of the Java/Scala class to invoke

    submitArgs - array[string]

    Additional options to pass to the application when running this job.

    script - string

    Use this text field if you want to override the default behaviour, which is to run className.main(args)

    type - stringrequired

    Default: custom_spark_scala_job

    Allowed values: custom_spark_scala_job