> ## Documentation Index
> Fetch the complete documentation index at: https://doc.lucidworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Spark Jobs

export const LwTemplate = ({title = "Key questions to get you started", icon = "sparkles", cta = "Powered by Agent Studio", linkHref = "https://lucidworks.com/demo/?utm_source=docs&utm_medium=referral&utm_campaign=docs_cta_ai"}) => {
  const [isLoaded, setIsLoaded] = useState(false);
  useEffect(() => {
    const timer = setTimeout(() => {
      setIsLoaded(true);
    }, 500);
    return () => clearTimeout(timer);
  }, []);
  return <div className="lw-template-container">
      <Card title={title} icon={icon}>
        {isLoaded && <span dangerouslySetInnerHTML={{
    __html: `<lw-template id="a029c1a9-28be-427e-b0e1-5d918920246a"></lw-template
            >`
  }} />}
        <Link href={linkHref} className="agent-studio-link text-left text-gray-600 gap-2 dark:text-gray-400 text-sm font-medium flex flex-row items-center hover:text-primary dark:hover:text-primary-light group-hover:text-primary group-hover:dark:text-primary-light">Powered by Lucidworks Agent Studio</Link>
      </Card>
    </div>;
};

[localhost link]: http://localhost:3000/docs/4/fusion-server/concepts/jobs/spark-jobs

[mintlify link]: https://doc.lucidworks.com/docs/4/fusion-server/concepts/jobs/spark-jobs

[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/187

Apache Spark can power a wide variety of data analysis jobs. In Fusion, Spark jobs are especially useful for generating [recommendations](/docs/4/fusion-ai/concepts/boosting/overview).

<LwTemplate />

## Spark job subtypes

For the Spark [job type](/docs/4/fusion-server/concepts/jobs/overview), the available subtypes are listed below.

* [Aggregation](/docs/4/fusion-server/reference/jobs/aggregation)\
  Define an aggregation job.
* [Custom Spark job](/docs/4/fusion-server/reference/jobs/custom-spark-job)\
  Run a custom Spark job.
* [Script](/docs/4/fusion-server/reference/jobs/script)\
  Run a custom Scala script as a Fusion job.

<Note>
  [Additional Spark jobs](/docs/4/fusion-ai/reference/jobs/overview) are available with a [Fusion AI](/docs/4/fusion-ai/overview) license.
</Note>

## Spark job configuration

Spark jobs can be created and modified using the [Fusion UI](/docs/4/fusion-server/concepts/jobs/overview) or the [Spark Jobs API](/docs/4/fusion-server/reference/api/spark-jobs-api). They can be scheduled using the Fusion UI or the [Jobs API](/docs/4/fusion-server/reference/api/spark-jobs-api).

For the complete list of configuration parameters for all Spark job subtypes, see the [Jobs Configuration Reference](/docs/4/fusion-server/concepts/jobs/overview).

## Learn more

<Accordion title="Troubleshoot Custom Spark Jobs">
  To troubleshoot problems with a custom job, start by looking for errors in the script-job driver log, `/opt/fusion/latest.x*/var/log/api/spark-driver-launcher.log` (on Unix) or `C:\lucidworks\var\fusion\latest.*x\var\log\api\spark-driver-launcher.log` (on Windows).

  For general Spark job troubleshooting steps, see Spark troubleshooting.
</Accordion>