Skip to main content
Apache Spark can power a wide variety of data analysis jobs. In Lucidworks Search, Spark jobs are especially useful for generating recommendations.

Spark job subtypes

For the Spark job type, the available subtypes are listed below.
  • SQL Aggregation A Spark SQL aggregation job where user-defined parameters are injected into a built-in SQL template at runtime.
  • Custom Python The Custom Python job provides user the ability to run Python code via Lucidworks Search. Python version requirements depend on your Spark version (Python 3.10 for Spark 3.4.1 and Python 3.7.3 for Spark 3.2.2).
  • Script This job lets you run a custom Scala script in Lucidworks Search.

Spark version configuration

Lucidworks Search releases 5.9.10 and later use Spark 3.4.1 by default. In Lucidworks Search releases 5.9.12 and later, you can configure Spark to use version 3.2.2 for compatibility with legacy Python and Scala environments. See the Lucidworks Search 5.9.12 release notes for details. Contact Lucidworks to request Spark 3.2.2 for your deployment.
Spark 3.4.1 is incompatible with Azure Blob Storage when accessed via the deprecated wasbs:// protocol due to a Jetty version conflict. Affected jobs fail with a java.lang.NoSuchMethodError that references org.eclipse.jetty.util.log.Log.getProperties(). If your jobs rely on wasbs://, request Spark 3.2.2 as a short-term fix. For the long term, migrate to the abfss:// protocol, which is fully supported in Spark 3.4.1.
See Additional Spark jobs for more information.

Spark job configuration

Spark jobs can be created and modified using the Lucidworks Search UI or the Spark Jobs API. They can be scheduled using the Lucidworks Search UI or the Spark Jobs API. For the complete list of configuration parameters for all Spark job subtypes, see the Jobs Configuration Reference.

Machine learning jobs

Lucidworks Search provides these job types to perform machine learning tasks.

Signals analysis

These jobs analyze a collection of signals in order to perform query rewriting, signals aggregation, or experiment analysis.
  • Ground Truth
    Ground truth or gold standard datasets are used in the ground truth jobs and query relevance metrics to define a specific set of documents.

Query rewriting

These jobs produce data that can be used for query rewriting or to inform updates to the synonyms.txt file.
  • Head/Tail Analysis
    Perform head/tail analysis of queries from collections of raw or aggregated signals, to identify underperforming queries and the reasons. This information is valuable for improving overall conversions, Solr configurations, auto-suggest, product catalogs, and SEO/SEM strategies, in order to improve conversion rates.
  • Phrase Extraction
    Identify multi-word phrases in signals.
  • Synonym Detection
    Use this job to generate pairs of synonyms and pairs of similar queries. Two words are considered potential synonyms when they are used in a similar context in similar queries.
  • Token and Phrase Spell Correction
    Detect misspellings in queries or documents using the numbers of occurrences of words and phrases.

Signals aggregation

  • SQL Aggregation
    A Spark SQL aggregation job where user-defined parameters are injected into a built-in SQL template at runtime.

Experiment analysis

Collaborative recommenders

These jobs analyze signals and generate matrices used to provide collaborative recommendations.
  • BPR Recommender
    Use this job when you want to compute user recommendations or item similarities using a Bayesian Personalized Ranking (BPR) recommender algorithm.
  • Query-to-Query Session-Based Similarity
    This recommender is based on co-occurrence of queries in the context of clicked documents and sessions. It is useful when your data shows that users tend to search for similar items in a single search session. This method of generating query-to-query recommendations is faster and more reliable than the Query-to-Query Similarity recommender job, and is session-based unlike the similar queries previously generated as part of the Synonym Detection job.

Content-based recommenders

Content-based recommenders create matrices of similar items based on their content.
  • Content-Based Recommender
    Use this job when you want to compute item similarities based on their content, such as product descriptions.

Content analysis

  • Cluster Labeling
    Cluster labeling jobs are run against your data collections, and are used:
    • When clusters or well-defined document categories already exist
    • When you want to discover and attach keywords to see representative words within existing clusters
  • Document Clustering
    The Document Clustering job uses an unsupervised machine learning algorithm to group documents into clusters based on similarities in their content. You can enable more efficient document exploration by using these clusters as facets, high-level summaries or themes, or to recommend other documents from the same cluster. The job can automatically group similar documents in all kinds of content, such as clinical trials, legal documents, book reviews, blogs, scientific papers, and products.
  • Classification
    This job analyzes how your existing documents are categorized and produces a classification model that can be used to predict the categories of new documents at index time.
  • Outlier Detection
    Outlier detection jobs are run against your data collections, and also perform the following actions:
    • Identify information that significantly differs from other data in the collection
    • Attach labels to designate each outlier group

Data ingest

The Parallel Bulk Loader (PBL) job enables bulk ingestion of structured and semi-structured data from big data systems, NoSQL databases, and common file formats like Parquet and Avro. Datasources the PBL uses include not only common file formats, but Solr databases, JDBC-compliant databases, MongoDB databases and more. In addition, the PBL distributes the load across the Lucidworks Search Spark cluster to optimize performance. And because no parsing is needed, indexing performance is also maximized by writing directly to Solr. For more information about usage and detailed configuration, see Parallel Bulk Loader configuration reference.

Learn more

Each Spark job run has its own driver pod. You can get the driver logs from the API or from Grafana.

Get driver logs from the API

Use the Spark pod logs endpoint of the Job Rest Server API to retrieve the initial logs about the driver pod starting up:

Get driver logs in Grafana

  1. Sign in to your Grafana environment and go to Explore.
  2. Select the Loki data source. The name of the Loki data source may be different in your deployment, such as Loki-stack.
  3. Add the pod label and select the driver pod for the job run. Driver pod names start with driver-.
  4. Set the time range to cover the job run.
Driver pods from earlier job runs stay in the pod list. If you don’t see any logs, check that you selected the driver pod for the job run in your time range.
You can also select driver pods in the Pods selector of the Fusion Log Viewer dashboard.For more information about building log queries, see Grafana Loki exploration.
Spark deletes failed and successful executor pods. Lucidworks Search removes successfully completed driver pods every 15 minutes.