Multiple Zones for High Availability

• When scheduling, prefer to put a pod on a node in an availability zone that doesn’t have a running instance of this component.
• Require that pods are deployed on a host that doesn’t have a running instance of the component that is being scheduled.

MY_VALUES="${MY_VALUES} --values gke_search_f5_fusion_affinity.yaml"

LucidAcademyLucidworks offers free training to help you get started.The Course for Preparing for Fusion Implementation focuses on the key elements for a successful implementation, defining your business requirements, preparing clean data, and involving the right personnel:

Visit the LucidAcademy to see the full training catalog.

​

Prerequisites

• A local copy of the fusion-cloud-native repository. This must be up-to-date with the latest master branch.
• Any cloud provider-specific command line tools, such as gcloud or aws, and kubectl.
  See the platform-specific instructions linked above, or check with your cloud provider.
• Helm v3
  • To install on a Mac:
  Copy

  brew upgrade kubernetes-helm
  

  • For other operating systems, download from Helm Releases.
  • Verify your installation:
  Copy

  helm version --short
  v3.0.0+ge29ce2a
  

• Kubernetes namespace
  • Collect the following information about your Kubernetes environment:
    • CLUSTER: Cluster name (passed to our setup scripts using the -c arg)
    • NAMESPACE: Kubernetes namespace where to install Fusion; a namespace should only contain lowercase letters (a-z), digits (0-9), or dash. No periods or underscores allowed.
• (optional) Clarify your organization’s DockerHub policy. The Fusion Helm chart points to public Docker images on DockerHub. Your organization may not allow Kubernetes to pull images directly from DockerHub or may require extra security scanning before loading images into production clusters.
  Consult your Kubernetes and Docker admin team to find how to get the Fusion images loaded into a registry that’s accessible to your cluster. You can update the image for each service using the custom values YAML file.

​

Custom values YAML file

1. Clone the fusion-cloud-native repository: git clone https://github.com/lucidworks/fusion-cloud-native
2. Run the customize_fusion_values.sh script.
   Copy

   ./customize_fusion_values.sh  --provider <provider> -c <cluster> -n <namespace> \
    --num-solr 3 \
    --solr-disk-gb 100 \
    --node-pool <node_selector> \
    --prometheus true \
    --with-resource-limits \
    --with-affinity-rules
   

   Pass the --help parameter to see script usage details.
   The script creates the following files:

   File	Description
   <provider>_<cluster>_<namespace>_fusion_values.yaml	Main custom values YAML used to override Helm chart defaults for Fusion microservices.
   <provider>_<cluster>_<namespace>_monitoring_values.yaml	Custom values yaml used to configure Prometheus and Grafana.
   <provider>_<cluster>_<namespace>_fusion_resources.yaml	Resource requests and limits for all Microservices.
   <provider>_<cluster>_<namespace>_fusion_affinity.yaml	Pod affinity rules to ensure multiple replicas for a single service are evenly distributed across zones and nodes.
   <provider>_<cluster>_<namespace>_upgrade_fusion.sh	Script used to install and/or upgrade Fusion using the aforementioned custom values YAML files.

   For an explanation of these placeholder values, see Configuration Values below.
3. Add the new files to version control. You will make changes to it over time as you fine-tune your Fusion installation. You will also need it to perform upgrades. If you try to upgrade your Fusion installation and don’t provide the custom values YAML, your deployment will revert to chart defaults.
   Review the <provider>_<cluster>_<release>_fusion_values.yaml file to familiarize yourself with its structure and contents. Notice it contains a separate section for each of the Fusion microservices. The example configuration of the query-pipeline service below illustrates some important concepts about the custom values YAML file.
   Copy

   query-pipeline: ①
   enabled: true ②
   nodeSelector: ③
       cloud.google.com/gke-nodepool: default-pool
   javaToolOptions: "..." ④
   pod: ⑤
       annotations:
       prometheus.io/port: "8787"
       prometheus.io/scrape: "true"
       prometheus.io/path: "/actuator/prometheus"
   

​

Deployment-specific values

​

Flags

kubectl label <NODE_ID> fusion_node_type=<NODE_LABEL>

​

Configure Solr sizing

solr:
  resources:                    # Set resource limits for Solr to help K8s pod scheduling;
    limits:                     # these limits are not just for the Solr process in the pod,
      cpu: "7700m"              # so allow ample memory for loading index files into the OS cache (mmap)
      memory: "26Gi"
    requests:
      cpu: "7000m"
      memory: "25Gi"
  logLevel: WARN
  nodeSelector:
    fusion_node_type: search    # Run this Solr StatefulSet in the "search" node pool
  exporter:
    enabled: true               # Enable the Solr metrics exporter (for Prometheus) and
                                # schedule on the default node pool (system partition)
    podAnnotations:
      prometheus.io/scrape: "true"
      prometheus.io/port: "9983"
      prometheus.io/path: "/metrics"
    nodeSelector:
      cloud.google.com/gke-nodepool: default-pool
  image:
    tag: 8.4.1
  updateStrategy:
    type: "RollingUpdate"
  javaMem: "-Xmx3g -Dfusion_node_type=system" # Configure memory settings for Solr
  solrGcTune: "-XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UseStringDeduplication -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=150 -XX:+UseLargePages -XX:+AlwaysPreTouch"
  volumeClaimTemplates:
    storageSize: "100Gi"        # Size of the Solr disk
  replicaCount: 6               # Number of Solr pods to run in this StatefulSet

zookeeper:
  nodeSelector:
    cloud.google.com/gke-nodepool: default-pool
  replicaCount: 3               # Number of Zookeepers
  persistence:
    size: 20Gi
  resources: {}
  env:
    ZK_HEAP_SIZE: 1G
    ZOO_AUTOPURGE_PURGEINTERVAL: 1

​

Configure storage class for Solr pods (optional)

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
 name: solr-gke-storage-regional
provisioner: kubernetes.io/gce-pd
parameters:
 type: pd-standard
 replication-type: regional-pd
 zones: us-west1-b, us-west1-c

kubectl apply -f storageClass.yaml

solr:
  volumeClaimTemplates:
    storageClassName: solr-gke-storage-regional
    storageSize: 250Gi

​

Configure multiple node pools

solr:
  nodePools:
    - name: "" ①
    - name: "analytics" ②
      javaMem: "-Xmx6g"
      replicaCount: 6
      storageSize: "100Gi"
      nodeSelector:
        fusion_node_type: analytics ③
      resources:
        requests:
          cpu: 2
          memory: 12Gi
        limits:
          cpu: 3
          memory: 12Gi
    - name: "search" ④
      javaMem: "-Xms11g -Xmx11g"
      replicaCount: 12
      storageSize: "50Gi"
      nodeSelector:
        fusion_node_type: search ⑤
      resources:
        limits:
          cpu: "7700m"
          memory: "26Gi"
        requests:
          cpu: "7000m"
          memory: "25Gi"
  nodeSelector:
    cloud.google.com/gke-nodepool: default-pool ⑥
...

​

Solr auto-scaling policy

​

Pod network policy

​

On-premises private Docker registries

1. Transfer the public images from DockerHub to your private Docker registry.
2. Establish a workstation that has access to DockerHub. This workstation must connect to your internal Docker registry, most likely via VPN connection. In this example, the workstation is referred to as envoy.
3. Install Docker on envoy. You need at least 100GB of free disk for Docker.
4. Pull all of the images from DockerHub to envoy’s local registry. For example, to pull the query pipeline image, run docker pull lucidworks/query-pipeline:5.9.0. See docker pull --help for more information about pulling Docker images.
5. Establish a connection from envoy to the private Docker registry, most likely via a VPN connection. In this example, the private Docker registry is referred to as <internal-private-registry>.
6. Push the images from envoy’s Docker registry to the private registry. This will take a long time.
   1. You’ll need to re-tag all images for the internal registry. For example, to tag the query-pipeline image, run:
   Copy

   docker tag lucidworks/query-pipeline:5.9.0 <internal-private-registry>/query-pipeline:5.9.0
   

   2. Push each image to the internal repo:
   Copy

   docker push <internal-private-registry>/query-pipeline:5.9.0
   

7. Install the Docker registry secret in Kubernetes. Create the Docker registry secret in the Kubernetes namespace where you want to install Fusion:
   Copy

   SECRET_NAME=<internal-private-secret>
   REPO=<internal-private-registry>
   
   kubectl create secret docker-registry "${SECRET_NAME}" \
    --namespace "${NAMESPACE}" \
    --docker-server="${REPO}" \
    --docker-username=${REPO_USER} \
    --docker-password=${REPO_PASS} \
    --docker-email=${REPO_USER}
   

   For details, see the Kubernetes article Pull an Image from a Private Registry.
8. Update the custom values YAML for your cluster to point to your private registry and secret to allow Kubernetes to pull images. For example:
   Copy

   query-pipeline:
    image:
      imagePullSecrets:
        - name: <internal-private-secret>
      repository: <internal-private-registry>
   

   Repeat the process for all Fusion services.

​

Customize Helm Chart

kubectl patch sa default -n $NAMESPACE \
  -p '"imagePullSecrets": [{"name": "<internal-private-secret>" }]'

kubectl patch sa default -n $NAMESPACE \
  -p "'imagePullSecrets': [{'name': '<internal-private-secret>'}]"

​

Add additional trusted certificate(s) to Fusion’s indexing and querying services (optional)

1. Create a new YAML file for your custom trusted certificates. The customcerts.yaml file is the example file in these instructions.
2. Add the custom certificate(s) in the YAML file created in the previous step. For example:
   Copy

   trustedCertificates:
    enabled: true
    files:
      some.cert: |-
        -----BEGIN CERTIFICATE-----
        MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
        (...)
        EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
        -----END CERTIFICATE-----
      other.cert: |-
        -----BEGIN CERTIFICATE-----
        MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
        (...)
        EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
        -----END CERTIFICATE---------
   

3. Update the indexing or querying service by running the following Helm command. Replace EXAMPLE-VALUES-FILE.yaml with your previous values file.
   Copy

   helm upgrade --install --namespace ${EXAMPLE-NAMESPACE} ${HELM-RELEASE} ${HELM-CHART-PATH} --values EXAMPLE-VALUES-FILE.yaml --values customcerts.yaml
   

4. Verify the indexing or querying pod has a new init-container with the name import-certs.

​

Add additional trusted certificate(s) for connectors to allow crawling of web resources with SSL/TLS enabled (optional)

classic-rest-service:
  trustedCertificates:
    enabled: true
    files:
      some.cert: |-
        -----BEGIN CERTIFICATE-----
        MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
        (...)
        EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
        -----END CERTIFICATE-----
      other.cert: |-
        -----BEGIN CERTIFICATE-----
        MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
        (...)
        EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
        -----END CERTIFICATE---------
connector-plugin:
  trustedCertificates:
    enabled: true
    files:
      some.cert: |-
        -----BEGIN CERTIFICATE-----
        MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
        (...)
        EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
        -----END CERTIFICATE-----
      other.cert: |-
        -----BEGIN CERTIFICATE-----
        MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
        (...)
        EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
        -----END CERTIFICATE---------

​

Generating the certificate on linux command line

openssl s_client -servername remote.server.net -connect remote.server.net:443 </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' >$fusion_home/apps/jetty/connectors/etc/yourcertname.crt

​

Generating the certificate using Firefox web browser

1. Navigate to the SharePoint host.
2. Click the in the address bar, then click the icon.
3. Next, navigate to More Information > View Certificate > Export.
   Save the file to the following folder: $fusion_home/apps/jetty/connectors/etc/yourcertname.crt

​

Generating the certificate using Chrome web browser

1. Navigate to Chrome menu > More Tools > Developer Tools > Security Tab. This will display the Security overview.
2. Click the View certificate button.
3. Save the file to the following folder:

​

Generating the certificate using powershell

$fusion_home = c:\your\fusion\install\directory
$webRequest = [Net.WebRequest]::Create("https://your-hostname")
try { $webRequest.GetResponse() } catch {}
$cert = $webRequest.ServicePoint.Certificate
$bytes = $cert.Export([Security.Cryptography.X509Certificates.X509ContentType]::Cert)
set-content -value $bytes -encoding byte -path "$fusion_home\apps\jetty\connectors\etc\yourcertname.binary.crt"
certutil -encode "$fusion_home\apps\jetty\connectors\etc\yourcertname.binary.crt" "$fusion_home\apps\jetty\connectors\etc\yourcertname.crt"
rm "$fusion_home\apps\jetty\connectors\etc\yourcertname.binary.crt" -f

​

Install Fusion 5 on Kubernetes

./gke_search_f5_upgrade_fusion.sh

​

Monitoring Fusion with Prometheus and Grafana

1. Run the customize_fusion_values.sh script with the --prometheus true option. This creates an extra custom values YAML file for installing Prometheus and Grafana, <provider>_<cluster>_<namespace>_monitoring_values.yaml. For example: gke_search_f5_monitoring_values.yaml.
2. Commit the YAML file to version control.
3. Review its contents to ensure that the settings suit your needs. For example, decide how long you want to keep metrics. The default is 36 hours.
   See the Prometheus documentation and Grafana documentation for details.
4. Run the install_prom.sh script to install Prometheus & Grafana in your namespace. Include the provider, cluster name, namespace, and helm release as in the example below:
   Copy

   ./install_prom.sh --provider gke -c search -n f5 -r 5-5-1
   

   Pass the --help parameter to see script usage details.
   The Grafana dashboards from monitoring/grafana are installed automatically by the install_prom.sh script.