Workload Isolation with Multiple Node Pools

You can run all Fusion services on a single Node Pool and Kubernetes will do its best to balance resource utilization across the nodes. However, Lucidworks recommends defining multiple Node Pools to separate services into "workload partitions" based on the type of traffic a service receives. Specifically, the Fusion Helm chart supports three optional partitions: search, analytics, and system. Workload isolation with Node Pools allows you to optimize resource utilization across the cluster to achieve better scalability, balance, and minimize infrastructure costs. It also helps with monitoring as you have better control over the traffic handled by each node in the cluster. To use this feature, you must define separate NodePools in your Kubernetes cluster ahead of time.


As depicted in Figure 1, the search partition hosts the API gateway (aka proxy), query pipelines, ML model service, and a Solr StatefulSet that hosts collections that support high volume, low-latency reads, such as your primary search collection and the signals_aggr collection which serves signal boosting lookups during query execution. The search partition is where you want to invest in better hardware, such as using nodes with SSDs for better query performance; typically, SSDs would not be needed for analytics nodes. The services deployed in the search partition often have Horizontal Pod Autoscalers (HPA) configured. We’ll cover how to configure the HPA for search-oriented services in Configure Replicas and Horizontal Pod Auto-Scaling.

When using multiple node pools to isolate / partition workloads, the Fusion Helm chart defines multiple StatefulSets for Solr. Each Solr StatefulSet uses the same Zookeeper connect string so are considered to be in the same Solr cluster; the partitioning of collections based on workload and zone is done with a Solr auto-scaling policy. The auto-scaling policy also ensures replicas get placed evenly between multiple availability zones (typically 3 for HA) so that your Fusion cluster can withstand the loss of one AZ and remain operational.


The analytics partition hosts the Spark driver & executor pods, Spark job management services (job-rest-service and job-launcher), index pipelines, and a Solr StatefulSet for hosting analytics-oriented collections, such as the signals collection. The signals collection typically experiences high write volume (to track user activity) and batch-oriented read requests from Spark jobs that do large table scans on the collection throughout the day. In addition, the analytics Solr pods may have different resource settings than the search Solr pods, i.e. you don’t need as much memory for these as they’re not serving facet queries and other memory intensive workloads in Solr.

When running in GKE, separating the Spark driver and executor pods into a dedicated Node Pool backed by preemptible nodes is a common pattern for reducing costs while increasing the compute capacity for running Spark jobs. You can also do this on EKS with spot instances. We’ll cover this approach in more detail in the Spark Ops section.

The system partition hosts all other Fusion services, such as the various stateless UI services (e.g. rules-ui), Prometheus/Grafana, as well as Solr pods hosting system collections like system_blobs. Lucidworks recommends running your Zookeeper ensemble in the system partition.

The analytics, search, and system partitions are simply a recommended starting point—​you can extend upon this model to refine your pod allocation by adding more Node Pools as needed. For instance, running Spark jobs on a dedicated pool of preemptible nodes is a pattern we’ve had great success with in our own K8s clusters at Lucidworks.