Skip to main content
A Spark “driver” is an application that creates a SparkContext for executing one or more Spark jobs. The following diagram shows the driver’s role in running a Spark job: Spark driver In Fusion 5, Kubernetes is the cluster manager. Fusion runs Spark in native Kubernetes mode, so there is no long-running Spark cluster. If your Spark job performs any collect operations, then the result of the collect (or collectAsMap) is sent back to the driver from all the executors. Consequently, if the result of the collect is too big too fit into memory, you will encounter OOM issues (or other memory related problems) when running your job.

How Fusion runs Spark drivers

When a Spark job runs, the job-launcher service starts a driver pod for that job run. The driver requests executor pods, and Kubernetes schedules them. If another job starts while the first one is running, Fusion starts a separate driver pod and executor pods for it. When the job finishes, the driver pod stops. A Kubernetes cron job removes successfully completed driver pods every 15 minutes. When no jobs are running, no Spark pods are running.

Configure drivers

You can configure drivers for all jobs or for a single job:
  • All jobs. Set the default driver and executor settings in the job-launcher Helm values:
    • executorInstances sets the number of executors. The default is 2.
    • sparkNamespace sets the namespace where driver and executor pods run. The default is the Fusion release namespace.
    • sparkNodeSelector sets the node labels that driver and executor pods must match. This value is separate from nodeSelector, which schedules the job-launcher pod itself.
    • spark.podTemplates.custom.driver and spark.podTemplates.custom.executor set custom pod templates for driver and executor pods. Pod templates take effect only when spark.podTemplates.enabled is true.
    For other default settings, such as driver and executor memory, see Spark config defaults.
  • A single job. Add Spark settings, such as spark.driver.memory or spark.driver.cores, to the job’s Spark Settings field or the sparkConfig list in the job configuration. For examples, see Spark operations how-tos. To control which nodes run a job’s driver and executor pods, see Node Selectors.

Custom jobs

Every custom Spark job runs in its own driver pod. Fusion supports these custom job types:

Logs and troubleshooting

To get driver logs, view the Spark UI for a running job, or clean up driver pods manually, see the “Get Logs for a Spark Job” and “Clean Up Spark Driver Pods” sections in Spark operations how-tos.