Multiple Zones for High Availability

Fusion relies heavily on Zookeeper to maintain quorum, which implies you need at least 3 ZK pods in an ensemble. In the Fusion Helm chart, Zookeeper is deployed as a StatefulSet. With GKE, you can launch a regional cluster that distributes nodes across 3 availability zones. With a multi-zone setup, your cluster can withstand the loss of one zone without experiencing downtime; you may experience degraded performance from losing ⅓ of your total compute capacity assuming your pods are distributed evenly across zones. The multi-zonal cluster also ensures higher availability for the Kubernetes control plane services.

Lucidworks provides affinity rules to ensure multiple pods per service get distributed across multiple zones, see Pod Affinity Rules

When running in a multi-zone cluster, each Solr node has a solr_zone system property set to the zone it is running in, such as -Dsolr_zone=us-west1-a. We’ll cover how to use the solr_zone property to distribute replicas across zones in the Solr Auto-scaling Policy section. Setting the solr_zone property for Solr pods requires the Solr service account to have a ClusterRoleBinding that allows it to get node metadata from the K8s API service.