> ## Documentation Index > Fetch the complete documentation index at: https://doc.lucidworks.com/llms.txt > Use this file to discover all available pages before exploring further. # Which Kubernetes? export const LwTemplate = ({title = "Key questions to get you started", icon = "sparkles", cta = "Powered by Agent Studio", linkHref = "https://lucidworks.com/demo/?utm_source=docs&utm_medium=referral&utm_campaign=docs_cta_ai"}) => { const [isLoaded, setIsLoaded] = useState(false); useEffect(() => { const timer = setTimeout(() => { setIsLoaded(true); }, 500); return () => clearTimeout(timer); }, []); return
{isLoaded && ` }} />} Powered by Lucidworks Agent Studio
; }; [localhost link]: http://localhost:3000/docs/5/fusion/operations/survival-guide/platforms [mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/survival-guide/platforms [old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/148 fusion services diagram The image above is for illustrative purposes only. It depicts services available in Fusion 5.9, but your actual implementation may differ. To help users get started with Fusion 5, Lucidworks provides setup scripts for Google Kubernetes Engine (GKE), Amazon Elastic Kubernetes Service (EKS), and Azure Kubernetes Service (AKS). It’s possible to use other Kubernetes providers, but the instructions and details provided in this article target the Kubernetes providers listed within release notes. To run Fusion 5, use your cloud provider’s stable release channel. Although GKE refers to the latest stable version as the "stable release channel," your cloud provider may use different terminology. For more information, see: * [GKE Release Channels](https://cloud.google.com/kubernetes-engine/docs/concepts/release-channels) * [EKS Kubernetes Release Calendar](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-release-calendar) * [AKS Kubernetes Release Calendar](https://docs.microsoft.com/en-us/azure/aks/supported-kubernetes-versions#aks-kubernetes-release-calendar) ## More information Here are instructions for deploying on some popular Kubernetes platforms: Fusion supports deployment on Google Kubernetes Engine (GKE). This topic explains how to deploy a Fusion cluster on GKE using the [`setup_f5_gke.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_gke.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository. ## Prerequisites This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes. ### Release Name and Namespace Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) Kubernetes namespace to install Fusion into. Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*. However, please *do not* install more than one Fusion release in the same namespace. All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_ Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default. ### Install Helm Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster. Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform. On MacOS, you can do: ```bash theme={"dark"} brew install kubernetes-helm ``` If you already have helm installed, make sure you're using the latest version: ```bash theme={"dark"} brew upgrade kubernetes-helm ``` For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/) The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`. #### Helm User Permissions If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory. When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.: ```bash theme={"dark"} alias k=kubectl ``` To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into: ```bash theme={"dark"} k create namespace fusion-namespace ``` Apply the `role.yaml` and `cluster-role.yaml` files to that namespace ```bash theme={"dark"} k apply -f cluster-role.yaml k config set-context --current --namespace=$NAMESPACE k apply -f role.yaml ``` Then bind the rolebinding and clusterolebinding to the install user: ```bash wrap theme={"dark"} k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user ``` You will then be able to run the `helm install` command as the `` ### Clone fusion-cloud-native from GitHub You should clone this repo from github as you'll need to run the scripts on your local workstation: ```bash theme={"dark"} git clone https://github.com/lucidworks/fusion-cloud-native.git ``` You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts. ```bash theme={"dark"} cd fusion-cloud-native git pull ``` Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip). Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory. ## Prerequisites This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes. ### Release Name and Namespace Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) Kubernetes namespace to install Fusion into. Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*. However, please *do not* install more than one Fusion release in the same namespace. All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_ Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default. ### Install Helm Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster. Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform. On MacOS, you can do: ```bash theme={"dark"} brew install kubernetes-helm ``` If you already have helm installed, make sure you're using the latest version: ```bash theme={"dark"} brew upgrade kubernetes-helm ``` For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/) The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`. #### Helm User Permissions If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory. When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.: ```bash theme={"dark"} alias k=kubectl ``` To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into: ```bash theme={"dark"} k create namespace fusion-namespace ``` Apply the `role.yaml` and `cluster-role.yaml` files to that namespace ```bash theme={"dark"} k apply -f cluster-role.yaml k config set-context --current --namespace=$NAMESPACE k apply -f role.yaml ``` Then bind the rolebinding and clusterolebinding to the install user: ```bash wrap theme={"dark"} k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user ``` You will then be able to run the `helm install` command as the `` ### Clone fusion-cloud-native from GitHub You should clone this repo from github as you'll need to run the scripts on your local workstation: ```bash theme={"dark"} git clone https://github.com/lucidworks/fusion-cloud-native.git ``` You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts. ```bash theme={"dark"} cd fusion-cloud-native git pull ``` Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip). Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory. The [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_gke.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_gke.sh) `setup_f5_gke.sh` script provided in this repo is strictly optional. The script is mainly to help those new to Kubernetes and/or Fusion get started quickly. If you're already familiar with K8s, Helm, and GKE, then you can skip the script and just use Helm directly to install Fusion into an existing cluster or one you create yourself using the process described `<>`. ### Set up the Google Cloud SDK (one time only) If you've already installed the `gcloud` command-line tools, you can skip to `<>`. These steps set up your local Google Cloud SDK environment so that you're ready to use the command-line tools to manage your Fusion deployment. Usually, you only need to perform these setup steps once. After that, you're ready to [create a cluster](#create-a-single-node-demo-cluster). For a nice getting started tutorial for GKE, see: [https://cloud.google.com/kubernetes-engine/docs/deploy-app-cluster](https://cloud.google.com/kubernetes-engine/docs/deploy-app-cluster) How to set up the Google Cloud SDK: 1. [https://console.cloud.google.com/apis/library/container.googleapis.com?q=kubernetes%20engine](https://console.cloud.google.com/apis/library/container.googleapis.com?q=kubernetes%20engine) 2. Log in to Google Cloud: `gcloud auth login` 3. Set up the Google Cloud SDK: 1. `gcloud config set compute/zone ` If you are working with regional clusters instead of zone clusters, use `gcloud config set compute/region ` instead. 2. `gcloud config set core/account ` 3. *New GKE projects only:* `gcloud projects create ` If you have already created a project, for example in [https://console.cloud.google.com/](https://console.cloud.google.com/), then skip to the next step. 4. `gcloud config set project ` Make sure you install the Kubernetes command-line tool `kubectl` using: ```bash theme={"dark"} gcloud components install kubectl gcloud components update ``` ### Create a single-node demo cluster Run the [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_gke.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_gke.sh) `setup_f5_gke.sh` script to install Fusion 5.x in a GKE cluster. To create a new, single-node *demo* cluster and install Fusion, simply do: ```bash theme={"dark"} ./setup_f5_gke.sh -c -p --create demo ``` Use the `--help` option to see script usage. If you want the script to create a cluster for you, then you need to pass the `--create` option with either `demo` or `multi_az`. If you don't want the script to create a cluster, then you need to create a cluster before running the script and simply pass the name of the existing cluster using the `-c` parameter. If you pass `--create demo` to the script, then we create a single node GKE cluster (defaults to using `n1-standard-8` node type). The minimum node type you'll need for a 1 node cluster is an `n1-standard-8` (on GKE) which has 8 CPU and 30 GB of memory. This is cutting it very close in terms of resources as you also need to host all of the Kubernetes system pods on this same node. Obviously, this works for kicking the tires on Fusion 5.1 but is not sufficient for production workloads. You can change the instance type using the `-i` parameter; see: [https://cloud.google.com/compute/docs/regions-zones/#available](https://cloud.google.com/compute/docs/regions-zones/#available) for a list of which machine types are available in your desired region. If not provided the script generates a custom values file named `gke___fusion_values.yaml` which you can use to customize the Fusion chart.\_\_ \#WARNING# If using Helm V2, the `setup_f5_gke.sh` script installs Helm's `tiller` component into your GKE cluster with the cluster admin role. If you don't want this, then please upgrade to Helm v3. If you see an error similar to the following, then wait a few seconds and try running the `setup_f5_gke.sh` script again with the same arguments as this is usually a transient issue: ```bash wrap theme={"dark"} Error: could not get apiVersions from Kubernetes: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request ``` After running the `setup_f5_gke.sh` script, proceed to the `<>` section below. When you're ready to deploy Fusion to a production-like environment, see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview). ### Create a three-node regional cluster to withstand a zone outage With a three-node regional cluster, nodes are deployed across three separate availability zones. ```bash wrap theme={"dark"} ./setup_f5_gke.sh -c -p -n --region --create multi_az ``` * `` value should be the name of a non-existent cluster; the script will create the new cluster. * `` must match the name of an existing project in GKE. Run `gcloud config get-value project` to get this value, or see the [GKE setup instructions](#set-up-the-google-cloud-sdk-one-time-only). * `` Kubernetes namespace to install Fusion into, defaults to `default` with release `f5` * `` value should be the name of a GKE region, defaults to `us-west1`. Run `gcloud config get-value compute/zone` to get this value, or see the [GKE setup instructions](#set-up-the-google-cloud-sdk-one-time-only) to set the value. In this configuration, Kubernetes deploys a ZooKeeper and Solr pod on each of the three nodes, which allows the cluster to retain ZK quorum and remain operational after losing one node, such as during an outage in one availability zone. When running in a multi-zone cluster, each Solr node has the `solr_zone` system property set to the zone it is running in, such as `-Dsolr_zone=us-west1-a`. After running the `setup_f5_gke.sh` script, proceed to the `<>` section below. When you're ready to deploy Fusion to a production-like environment, see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview). #### GKE Ingress and TLS The Fusion proxy service provides authentication and serves as an API gateway for accessing all other Fusion services. It's typical to use an Ingress for TLS termination in front of the proxy service. The `setup_f5_gke.sh` supports creating an Ingress with a TLS cert for a domain you own by passing: `-t -h ` After the script runs, you need to create an A record in GCP's DNS service to map your domain name to the Ingress IP. Once this occurs, our script setup uses [https://letsencrypt.org/](https://letsencrypt.org/) to issue a TLS cert for your Ingress. To see the status of the Let's Encrypt issued certificate, do: ```bash theme={"dark"} kubectl get managedcertificates -n -o yaml ``` Please refer to the Kubernetes documentation on configuring an Ingress for GKE: [https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer](https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer) The GCP Ingress defaults to a 30 second timeout, which can lead to false negatives for long running requests such as importing apps. To configure the timeout for the backend in kubernetes: Create a BackendConfig object in your namespace: ```yaml theme={"dark"} --- apiVersion: cloud.google.com/v1beta1 kind: BackendConfig metadata: name: backend_config_name spec: timeoutSec: 120 connectionDraining: drainingTimeoutSec: 60 ``` Then make sure that the following entries are in the right place in your values.yaml file: ```yaml theme={"dark"} api-gateway: service: annotations: beta.cloud.google.com/backend-config: '{"ports": {"6764":"backend_config_name"}}' ``` and upgrade your release to apply the configuration changes #### Ingresses and externalTrafficPolicy When running a fusion cluster behind an externally controlled LoadBalancer it can be advantageous to configure the `externalTrafficPolicy` of the `proxy` service to `Local`. This preserves the client source IP and avoids a second hop for LoadBalancer and NodePort type services, but risks potentially imbalanced traffic spreading. Although when running in a cluster with a dedicated pool for spark jobs that can scale up and down freely it can prevent unwanted request failures. This behaviour can be altered with the `api-gateway.service.externalTrafficPolicy` value, which is set to `Local` if the example values file is used. **You must use `externalTrafficPolicy`=`Local` for the Trusted HTTP Realm to work correctly.** If you are already using a custom 'values.yaml' file, create an entry for `externalTrafficPolicy` under `api-gateway` service. ```yaml theme={"dark"} api-gateway: service: externalTrafficPolicy: Local ``` #### Considerations when using the nginx ingress controller If you are using the `nginx` ingress controller to fulfil your ingress definitions there are a couple of options that are recommended to be set in the configmap: ```bash wrap theme={"dark"} enable-underscores-in-headers: "true" # Fusion can return some headers that have underscores, these have to be explicitly enabled in nginx proxy-body-size: "0" # By default nginx places a maximum size on request bodies, either increase as needed or disable by setting to 0 proxy-read-timeout: "300" # Increases the timeout for potential slow queries. ``` #### Custom values There are some example values files that can be used as a starting point for resources, affinity and replica count configuration in the `example-values` folder. These can be passed to the install script using the `--values` option, for example: ```bash wrap theme={"dark"} ./setup_f5_gke.sh -c -p -r -n \ --values example-values/affinity.yaml --values example-values/resources.yaml --values example-values/replicas.yaml ``` The `--values` option can be passed multiple times, if the same configuration property is contained within multiple `values` files then the values from the latest file passed as a `--values` option are used. ##### Connectors custom values If you are using Fusion 5.9 or later, you can specify resources and replica count per connector. This allows you to set different resource limits for each connector. If you do not set custom values for a connector, that connector uses the default values. Set each connector's resource values in the `connector-plugin` section under `pluginValues`. The `pluginValues` section is a list of plugins and its resources. The following sample shows an example. ```yaml theme={"dark"} pluginValues: - id: "plugin-id" <1> resources: <2> limits: cpu: "2" memory: "3Gi" requests: cpu: "250m" memory: "2Gi" replicaCount: 1 <3> ``` * `<1>` The plugin ID. The plugin ID must match the plugin ID on the plugin ZIP file. without the `lucidworks.` prefix. For example, if the plugin ID on the plugin ZIP file is `lucidworks.sharepoint-optimized`, the plugin ID is `sharepoint-optimized`. * `<2>` The resources settings. You may specify the limits, the requests, and the CPU and memory for each. * `<3>` The number of replicas per connector. This value is 1 by default. After editing the `connector-plugin` section, you must reinstall the affected connector. #### Upgrades and Ingress If you used the `-t -h ` options when installing your cluster, our script created an additional values yaml file named `tls-values.yaml`. To make things easier for you when upgrading, you should add the settings from this file into your main custom values yaml file, e.g.: ```yaml theme={"dark"} api-gateway: service: type: "NodePort" ingress: enabled: true host: "" tls: enabled: true annotations: "networking.gke.io/managed-certificates": "-managed-certificate" "kubernetes.io/ingress.class": "gce" ``` This way you don't have to remember to pass the additional `tls-values.yaml` file when upgrading. ## Verifying the Fusion Installation In this section, we provide some tips on how to verify the Fusion installation. Check if the Fusion Admin UI is available at `\https://:6764/admin/`. Let's review some useful kubectl commands. ### Enhance the K8s Command-line Experience Here is a list of tools we found useful for improving your command-line experience with Kubernetes: * krew (kubectl plugin mgr): [https://github.com/kubernetes-sigs/krew/](https://github.com/kubernetes-sigs/krew/) * kube-ps1 (show current context on command line prompt): [https://github.com/jonmosco/kube-ps1](https://github.com/jonmosco/kube-ps1) * kubectx / kubens (switch between clusters / namespaces): [https://github.com/ahmetb/kubectx](https://github.com/ahmetb/kubectx) ### Useful kubectl commands kubectl reference: [https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands) Set the namespace for `kubectl` if not using the default: ```bash theme={"dark"} kubectl config set-context --current --namespace= ``` **This saves you from having to pass `-n` with every command.** Get a list of running pods: `k get pods` Get logs for a pod using a label: `k logs –l app.kubernetes.io/component=query-pipeline` Get pod deployment spec and details: `k get pods -o yaml` Get details about a pod events: `k describe po ` Port forward to a specific pod: `k port-forward 8983:8983` SSH into a pod: `k exec -it -- /bin/bash` CPU/Memory usage report for pods: `k top pods` Forcefully kill a pod: `k delete po --force --grace-period 0` Scale up (or down) a deployment: `k scale deployment.v1.apps/ --replicas=N` Get a list of pod versions: `k get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq` ### Check Fusion Pods and Services Once the install script completes, you can check that all pods and services are available using: ```bash theme={"dark"} kubectl get pods ``` If all goes well, you should see a list of pods similar to: ```bash wrap theme={"dark"} NAME READY STATUS RESTARTS AGE seldon-controller-manager-6675874894-qxwrv 1/1 Running 0 8m45s f5-admin-ui-74d794f4f8-m5jms 1/1 Running 0 8m45s f5-ambassador-fd6b9b5dc-7ghf6 1/1 Running 0 8m43s f5-api-gateway-6b9998b9c-tmchk 1/1 Running 0 8m45s f5-auth-ui-7565564b4c-rdc74 1/1 Running 0 8m42s f5-classic-rest-service-0 1/1 Running 3 8m44s f5-devops-ui-77bb867ffb-fbzxd 1/1 Running 0 8m42s f5-fusion-admin-78b8f8fc7f-4d7l8 1/1 Running 0 8m42s f5-fusion-indexing-599c8d448-xzsvm 1/1 Running 0 8m44s f5-insights-665fd9f6fc-g5psw 1/1 Running 0 8m43s f5-job-launcher-84dd4c5c96-p8528 1/1 Running 0 8m44s f5-job-rest-server-6d44d964b8-xtnxw 1/1 Running 0 8m45s f5-logstash-0 1/1 Running 0 8m45s f5-ml-model-service-6987dc94c9-9ppp8 2/2 Running 1 8m45s f5-monitoring-grafana-5d499dbb58-pzw72 1/1 Running 0 10m f5-monitoring-prometheus-kube-state-metrics-54d6678dv9h7h 1/1 Running 0 10m f5-monitoring-prometheus-pushgateway-7d65c65b85-vwrwf 1/1 Running 0 10m f5-monitoring-prometheus-server-0 2/2 Running 0 10m f5-pm-ui-86cbc5bb65-nd2n8 1/1 Running 0 8m44s f5-pulsar-bookkeeper-0 1/1 Running 0 8m45s f5-pulsar-broker-b56cc776f-56msx 1/1 Running 0 8m45s f5-query-pipeline-5d75d7d5f4-l2mdf 1/1 Running 0 8m43s f5-connectors-7bb6cfc65f-7wfs2 1/1 Running 0 8m42s f5-connectors-backend-987fdc648-dldwv 1/1 Running 0 8m45s f5-rules-ui-6b9d55b78f-9hzzj 1/1 Running 0 8m43s f5-solr-0 1/1 Running 0 8m44s f5-solr-exporter-c4687c785-jsm7x 1/1 Running 0 8m45s f5-ui-6cdbcc68c6-rj9cq 1/1 Running 0 8m45s f5-webapps-6d6bb9bfd-hm4qx 1/1 Running 0 8m45s f5-workflow-controller-7b66679fb7-sjbvp 1/1 Running 0 8m44s f5-zookeeper-0 1/1 Running 0 8m45s ``` The number of pods per deployment / statefulset will vary based on your cluster size and replicaCount settings in your custom values YAML file. Also, don't worry if you see some pods having been restarted as that just means they were too slow to come up and Kubernetes killed and restarted them. You do want to see at least one pod running for every service. If a pod is not running after waiting a sufficient amount of time, use `kubectl logs ` to see the logs for that pod; to see the logs for previous versions of a pod, use: `kubectl logs -p`. You can also look at the actions Kubernetes performed on the pod using `kubectl describe po `. To see a list of Fusion services, do: ```bash theme={"dark"} kubectl get svc ``` For an overview of the various Fusion 5 microservices, see: [Fusion microservices](/docs/5/fusion/reference/microservices). Once you're ready to build a Fusion cluster for production, please see see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview). ### Upgrading with Zero Downtime One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. Fusion 5 allows customers to upgrade from Fusion 5.x.y to a later 5.x.z version on a live cluster with zero downtime or disruption of service. When Kubernetes performs a rolling update to an individual microservice, there will be a mix of old and new services in the cluster concurrently (only briefly in most cases) and requests from other services will be routed to both versions. Consequently, Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same 5.x line of releases. We also ensure stored configuration remains compatible in the same 5.x release line. Lucidworks releases minor updates to individual services frequently, so our customers can pull in those upgrades using Helm at their discretion. To upgrade your cluster at any time, use the `--upgrade` option with our setup scripts in this repo. The scripts in this repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks. To see what would be upgraded, you can pass the `--dry-run` option to the script. ### Grafana Dashboards Get the initial Grafana password from a K8s secret by doing: ```bash wrap theme={"dark"} kubectl get secret --namespace "${NAMESPACE}" ${RELEASE}-monitoring-grafana \ -o jsonpath="{.data.admin-password}" | base64 --decode ; echo ``` With Grafana, you can either setup a temporary port-forward to a Grafana pod or expose Grafana on an external IP using a K8s LoadBalancer. To define a LoadBalancer, do (replace \${RELEASE} with your Helm release label): ```bash wrap theme={"dark"} kubectl expose deployment ${RELEASE}-monitoring-grafana --type=LoadBalancer --name=grafana --port=3000 --target-port=3000 ``` You can use `kubectl get services --namespace ` to determine when the load balancer is setup and its IP address. Direct your browser to `http://:3000` and enter the username `admin@localhost` and the password that was returned in the previous step. This will log you into the application. It is recommended that you create another administrative user with a more desirable password. The dashboards and datasoure will be setup for you in grafana, simply navigate to `Dashboards` -> `Manage` to view the available dashboards ## Learn more The course for **Deploying Fusion 5** focuses on the prerequisite software needed to deploy Fusion, the necessary setup steps, and the physical act of deployment. Fusion supports deployment on Amazon Elastic Kubernetes Service (EKS). This topic explains how to deploy a Fusion cluster on EKS using the [`setup_f5_eks.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_eks.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository. In addition, this topic provides information about how to [configure IAM roles for the service account](#configure-iam-roles-for-the-service-account). ## Prerequisites This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes. ### Release Name and Namespace Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) Kubernetes namespace to install Fusion into. Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*. However, please *do not* install more than one Fusion release in the same namespace. All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_ Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default. ### Install Helm Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster. Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform. On MacOS, you can do: ```bash theme={"dark"} brew install kubernetes-helm ``` If you already have helm installed, make sure you're using the latest version: ```bash theme={"dark"} brew upgrade kubernetes-helm ``` For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/) The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`. #### Helm User Permissions If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory. When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.: ```bash theme={"dark"} alias k=kubectl ``` To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into: ```bash theme={"dark"} k create namespace fusion-namespace ``` Apply the `role.yaml` and `cluster-role.yaml` files to that namespace ```bash theme={"dark"} k apply -f cluster-role.yaml k config set-context --current --namespace=$NAMESPACE k apply -f role.yaml ``` Then bind the rolebinding and clusterolebinding to the install user: ```bash wrap theme={"dark"} k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user ``` You will then be able to run the `helm install` command as the `` ### Clone fusion-cloud-native from GitHub You should clone this repo from github as you'll need to run the scripts on your local workstation: ```bash theme={"dark"} git clone https://github.com/lucidworks/fusion-cloud-native.git ``` You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts. ```bash theme={"dark"} cd fusion-cloud-native git pull ``` Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip). Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory. The [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_eks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_eks.sh) `setup_f5_eks.sh` script provided in this repo is strictly optional. The script is mainly to help those new to Kubernetes and/or Fusion get started quickly. If you're already familiar with K8s, Helm, and EKS, then you use Helm directly to install Fusion into an existing cluster or one you create yourself using the process described `<>`. If you're new to Amazon Web Services (AWS), then please visit the Amazon Web Services [https://aws.amazon.com/getting-started/](https://aws.amazon.com/getting-started/) to set up an account. If you're new to Kubernetes and EKS, then we recommend going through Amazon's [https://eksworkshop.com/introduction/](https://eksworkshop.com/introduction/) before proceeding with Fusion. ### Set up the AWS CLI tools Before launching an EKS cluster, you need to install and configure `kubectl`, `aws`, `eksctl`, `aws-iam-authenticator` using the links provided below: Required AWS Command-line Tools: * kubectl: [https://kubernetes.io/docs/tasks/tools/install-kubectl/](https://kubernetes.io/docs/tasks/tools/install-kubectl/) * aws: [https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) * eksctl: [https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html) * aws-iam-authenticator: [https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html](https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html) Run `aws configure` to configure a profile for authenticating to AWS. You'll use the profile name you configure in this step, which defaults to `default`, as the `-p` argument to the `setup_f5_eks.sh` script in the next section. When working in Ubuntu, avoid using the eksctl snap version. Alternative sources can have different versions that could cause command failures. Also, always make sure you are using the latest version for each one of the required tools. ### Set up Fusion on EKS To create a cluster in EKS the following IAM policies are required: * AmazonEC2FullAccess * AWSCloudFormationFullAccess EKS Permissions: ```bash theme={"dark"} eks:CreateCluster eks:DeleteCluster eks:DescribeCluster eks:DescribeUpdate eks:ListClusters eks:ListUpdates eks:UpdateClusterVersion ``` VPC Permissions: ```bash theme={"dark"} ec2:AssociateSubnetCidrBlock ec2:AssociateVpcCidrBlock ec2:AttachInternetGateway ec2:CreateInternetGateway ec2:CreateSubnet ec2:CreateVpc ec2:CreateVpcEndpoint ec2:DeleteInternetGateway ec2:DeleteSubnet ec2:DeleteVpc ec2:DeleteVpcEndpoints ec2:DescribeSubnets ec2:DescribeVpcAttribute ec2:DescribeVpcs ec2:DetachInternetGateway ec2:DisassociateSubnetCidrBlock ec2:DisassociateVpcCidrBlock ec2:ModifySubnetAttribute ec2:ModifyVpcAttribute ec2:ModifyVpcEndpoint ``` .IAM Permissions ```bash theme={"dark"} iam:AddRoleToInstanceProfile iam:AttachRolePolicy iam:CreateInstanceProfile iam:CreatePolicy iam:CreatePolicyVersion iam:CreateRole iam:DeleteInstanceProfile iam:DeletePolicy iam:DeletePolicyVersion iam:DeleteRole iam:DeleteRolePolicy iam:DetachRolePolicy iam:GetInstanceProfile iam:GetPolicy iam:GetPolicyVersion iam:GetRole iam:GetRolePolicy iam:ListInstanceProfiles iam:ListInstanceProfilesForRole iam:PassRole iam:PutRolePolicy iam:RemoveRoleFromInstanceProfile iam:TagRole iam:UntagRole ``` Download and run the [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_eks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_eks.sh) `setup_f5_eks.sh` script to install Fusion 5.x in an EKS cluster. This script does not support multiple node pools and should not be used for production clusters. * To create a new cluster and install Fusion, run the following command: ``` ./setup_f5_eks.sh -c my-eks-cluster -p profile-name -n fusion-namespace --create demo ``` * Replace `my-eks-cluster`, `profile-name`, and `fusion-namespace` with your cluster, profile, and namespace values. * Pass the `--create` option with either `demo` or `multi_az`. * To use an existing cluster and install Fusion, run the following command: ``` ./setup_f5_eks.sh -c cluster-name -p profile-name ``` * Replace `cluster-name` with the name of the cluster you already created. * Replace `profile-name` with the name of your profile. The profile is automatically set to `default` if you ran the AWS configure command without giving the profile a name. Use the `--help` option to see full script usage. If using Helm V2, the `setup_f5_eks.sh` script installs Helm's `tiller` component into your EKS cluster with the cluster admin role. If you don't want this, then please upgrade to Helm v3. The `setup_f5_eks.sh` script creates a service account that provides S3 read-only permissions to the created pods. After running the `setup_f5_eks.sh` script, proceed to the `<>` section below. #### EKS cluster overview The EKS cluster is created using `eksctl` ([https://eksctl.io/](https://eksctl.io/)). By default it will setup the following resources in your AWS account: * A dedicated VPC for the EKS cluster in the specified region with CIDR: `192.168.0.0/16` * 3 Public and 3 Private subnets within the created VPC, each with a `/19` CIDR range, along with the corresponding route tables. * A NAT gateway in each Public subnet * An Auto Scaling Group of the instance type specified by the script, which defaults to `m5.2xlarge`, with 3 instances spanning the public subnets. See [https://eksctl.io/usage/vpc-networking/](https://eksctl.io/usage/vpc-networking/) for more information on the networking setup. #### EKS Ingress The `setup_f5_eks.sh` script exposes the Fusion proxy service on an external DNS name provided by an ELB over HTTP. This is done for demo or getting started purposes. However, you're strongly encouraged to configure a K8s Ingress with TLS termination in front of the proxy service. See: [https://aws.amazon.com/premiumsupport/knowledge-center/terminate-https-traffic-eks-acm/](https://aws.amazon.com/premiumsupport/knowledge-center/terminate-https-traffic-eks-acm/) Our EKS script creates a classic ELB for exposing fusion proxy service. In case you need to change this behavior and use [https://github.com/kubernetes-sigs/aws-load-balancer-controller](https://github.com/kubernetes-sigs/aws-load-balancer-controller) instead you can use the following parameters when running the `setup_f5_eks.sh` script: ```bash theme={"dark"} --deploy-alb # Tells the script to deploy an ALB ``` By default the `kube-system` namespace is being used for installing the `aws-load-balancer-controller` because pods `priorityClassName` is set to `system-cluster-critical`. In case you need to deploy an internal ALB you can use the `--internal-alb` option. This will create the nodes in the internal subnets. Fusion will be reachable from an AWS instance located in any of the external subnets on the same VPC. To use an ALB also an ingress with a DNS name is required, you can use the `-h` option to create an ingress with the required DNS name. Finally, use Route 53 or your DNS provider for creating an A ALIAS DNS record for your DNS name pointing to the ingress ADRESS. You can get the address listing the ingress using the command `kubectl get ing`. ## Verifying the Fusion Installation In this section, we provide some tips on how to verify the Fusion installation. Check if the Fusion Admin UI is available at `\https://:6764/admin/`. Let's review some useful kubectl commands. ### Enhance the K8s Command-line Experience Here is a list of tools we found useful for improving your command-line experience with Kubernetes: * krew (kubectl plugin mgr): [https://github.com/kubernetes-sigs/krew/](https://github.com/kubernetes-sigs/krew/) * kube-ps1 (show current context on command line prompt): [https://github.com/jonmosco/kube-ps1](https://github.com/jonmosco/kube-ps1) * kubectx / kubens (switch between clusters / namespaces): [https://github.com/ahmetb/kubectx](https://github.com/ahmetb/kubectx) ### Useful kubectl commands kubectl reference: [https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands) Set the namespace for `kubectl` if not using the default: ```bash theme={"dark"} kubectl config set-context --current --namespace= ``` **This saves you from having to pass `-n` with every command.** Get a list of running pods: `k get pods` Get logs for a pod using a label: `k logs –l app.kubernetes.io/component=query-pipeline` Get pod deployment spec and details: `k get pods -o yaml` Get details about a pod events: `k describe po ` Port forward to a specific pod: `k port-forward 8983:8983` SSH into a pod: `k exec -it -- /bin/bash` CPU/Memory usage report for pods: `k top pods` Forcefully kill a pod: `k delete po --force --grace-period 0` Scale up (or down) a deployment: `k scale deployment.v1.apps/ --replicas=N` Get a list of pod versions: `k get po -o jsonpath='{..image}' tr -s '[[:space:]]' '\n' sort uniq` ### Check Fusion Pods and Services Once the install script completes, you can check that all pods and services are available using: ```bash theme={"dark"} kubectl get pods ``` If all goes well, you should see a list of pods similar to: ```bash wrap theme={"dark"} NAME READY STATUS RESTARTS AGE seldon-controller-manager-6675874894-qxwrv 1/1 Running 0 8m45s f5-admin-ui-74d794f4f8-m5jms 1/1 Running 0 8m45s f5-ambassador-fd6b9b5dc-7ghf6 1/1 Running 0 8m43s f5-api-gateway-6b9998b9c-tmchk 1/1 Running 0 8m45s f5-auth-ui-7565564b4c-rdc74 1/1 Running 0 8m42s f5-classic-rest-service-0 1/1 Running 3 8m44s f5-devops-ui-77bb867ffb-fbzxd 1/1 Running 0 8m42s f5-fusion-admin-78b8f8fc7f-4d7l8 1/1 Running 0 8m42s f5-fusion-indexing-599c8d448-xzsvm 1/1 Running 0 8m44s f5-insights-665fd9f6fc-g5psw 1/1 Running 0 8m43s f5-job-launcher-84dd4c5c96-p8528 1/1 Running 0 8m44s f5-job-rest-server-6d44d964b8-xtnxw 1/1 Running 0 8m45s f5-logstash-0 1/1 Running 0 8m45s f5-ml-model-service-6987dc94c9-9ppp8 2/2 Running 1 8m45s f5-monitoring-grafana-5d499dbb58-pzw72 1/1 Running 0 10m f5-monitoring-prometheus-kube-state-metrics-54d6678dv9h7h 1/1 Running 0 10m f5-monitoring-prometheus-pushgateway-7d65c65b85-vwrwf 1/1 Running 0 10m f5-monitoring-prometheus-server-0 2/2 Running 0 10m f5-pm-ui-86cbc5bb65-nd2n8 1/1 Running 0 8m44s f5-pulsar-bookkeeper-0 1/1 Running 0 8m45s f5-pulsar-broker-b56cc776f-56msx 1/1 Running 0 8m45s f5-query-pipeline-5d75d7d5f4-l2mdf 1/1 Running 0 8m43s f5-connectors-7bb6cfc65f-7wfs2 1/1 Running 0 8m42s f5-connectors-backend-987fdc648-dldwv 1/1 Running 0 8m45s f5-rules-ui-6b9d55b78f-9hzzj 1/1 Running 0 8m43s f5-solr-0 1/1 Running 0 8m44s f5-solr-exporter-c4687c785-jsm7x 1/1 Running 0 8m45s f5-ui-6cdbcc68c6-rj9cq 1/1 Running 0 8m45s f5-webapps-6d6bb9bfd-hm4qx 1/1 Running 0 8m45s f5-workflow-controller-7b66679fb7-sjbvp 1/1 Running 0 8m44s f5-zookeeper-0 1/1 Running 0 8m45s ``` The number of pods per deployment / statefulset will vary based on your cluster size and replicaCount settings in your custom values YAML file. Also, don't worry if you see some pods having been restarted as that just means they were too slow to come up and Kubernetes killed and restarted them. You do want to see at least one pod running for every service. If a pod is not running after waiting a sufficient amount of time, use `kubectl logs ` to see the logs for that pod; to see the logs for previous versions of a pod, use: `kubectl logs -p`. You can also look at the actions Kubernetes performed on the pod using `kubectl describe po `. To see a list of Fusion services, do: ```bash theme={"dark"} kubectl get svc ``` For an overview of the various Fusion 5 microservices, see: [Fusion microservices](/docs/5/fusion/reference/microservices). Once you're ready to build a Fusion cluster for production, please see see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview). ### Upgrading with Zero Downtime One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. Fusion 5 allows customers to upgrade from Fusion 5.x.y to a later 5.x.z version on a live cluster with zero downtime or disruption of service. When Kubernetes performs a rolling update to an individual microservice, there will be a mix of old and new services in the cluster concurrently (only briefly in most cases) and requests from other services will be routed to both versions. Consequently, Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same 5.x line of releases. We also ensure stored configuration remains compatible in the same 5.x release line. Lucidworks releases minor updates to individual services frequently, so our customers can pull in those upgrades using Helm at their discretion. To upgrade your cluster at any time, use the `--upgrade` option with our setup scripts in this repo. The scripts in this repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks. To see what would be upgraded, you can pass the `--dry-run` option to the script. ### Grafana Dashboards Get the initial Grafana password from a K8s secret by doing: ```bash wrap theme={"dark"} kubectl get secret --namespace "${NAMESPACE}" ${RELEASE}-monitoring-grafana \ -o jsonpath="{.data.admin-password}" base64 --decode ; echo ``` With Grafana, you can either setup a temporary port-forward to a Grafana pod or expose Grafana on an external IP using a K8s LoadBalancer. To define a LoadBalancer, do (replace \${RELEASE} with your Helm release label): ```bash wrap theme={"dark"} kubectl expose deployment ${RELEASE}-monitoring-grafana --type=LoadBalancer --name=grafana --port=3000 --target-port=3000 ``` You can use `kubectl get services --namespace ` to determine when the load balancer is setup and its IP address. Direct your browser to `http://:3000` and enter the username `admin@localhost` and the password that was returned in the previous step. This will log you into the application. It is recommended that you create another administrative user with a more desirable password. The dashboards and datasoure will be setup for you in grafana, simply navigate to `Dashboards` -> `Manage` to view the available dashboards ## Configure IAM roles for the service account Configuring IAM roles lets you utilize the Amazon Web Services Security Token Service (AWS STS) for short-term authentication credentials to access services like the Amazon S3 simple storage service. To configure IAM roles, your user account must be granted `admin` permissions or `IAM:FullAccess`. Complete the following steps: 1. To create the OpenID Connect (OIDC) provider, run the following command: ```bash wrap theme={"dark"} eksctl utils associate-iam-oidc-provider --cluster cluster_name --approve ``` 2. To create an IAM role for the service account associated with the plugin-pod, run the following command: ```bash wrap theme={"dark"} eksctl create iamserviceaccount --name f5-connector-plugin --namespace default --cluster cluster_name --attach-policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess --approve --override-existing-serviceaccounts ``` This command: * Creates an IAM role and attaches the target policy. * Updates the existing Kubernetes `f5-connector-plugin` service account and annotates it with the IAM role. * Uses the existing `policy/AmazonS3ReadOnlyAccess` policy. If the IAM role was already created without the command, and you want to associate the service account, run the following command: ```bash wrap theme={"dark"} kubectl annotate serviceaccount -n default f5-connector-plugin eks.amazonaws.com/role-arn=arn:aws:iam::411271863668:role/FUS_ROLE --overwrite=true ``` To utilize this feature, create a data source with the settings in **S3 Authentication Settings > AWS Instance Credentials Authentication Settings**. For detailed installation information, [AWS S3 V2 connector](/docs/fusion-connectors/connectors/amazon-aws-s3-v2). For more information, see: * [Welcome to the AWS Security Token Service API Reference](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html) * [Temporary security credentials in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html) * [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html) * [What is Amazon EKS?](https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html) * [Fine-grained IAM roles for service accounts](https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts) ## Learn more The course for **Deploying Fusion 5** focuses on the prerequisite software needed to deploy Fusion, the necessary setup steps, and the physical act of deployment. Fusion supports deployment on Azure Kubernetes Service (AKS). This topic explains how to deploy a Fusion cluster on AKS using the [`setup_f5_aks.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_aks.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository. The `setup_f5_aks.sh` script is the basic foundation for getting started and proof-of-concept purposes. For information about custom values in a production-ready environment, see Custom values YAML file. ## Prerequisites This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes. ### Release Name and Namespace Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) namespace to install Fusion into. Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*. However, please *do not* install more than one Fusion release in the same namespace. All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_ Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default. ### Install Helm Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster. Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform. On MacOS, you can do: ```bash theme={"dark"} brew install kubernetes-helm ``` If you already have helm installed, make sure you're using the latest version: ```bash theme={"dark"} brew upgrade kubernetes-helm ``` For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/) The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`. #### Helm User Permissions If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory. When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.: ```bash theme={"dark"} alias k=kubectl ``` To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into: ```bash theme={"dark"} k create namespace fusion-namespace ``` Apply the `role.yaml` and `cluster-role.yaml` files to that namespace ```bash theme={"dark"} k apply -f cluster-role.yaml k config set-context --current --namespace=$NAMESPACE k apply -f role.yaml ``` Then bind the rolebinding and clusterolebinding to the install user: ```bash wrap theme={"dark"} k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user ``` You will then be able to run the `helm install` command as the `` ### Clone fusion-cloud-native from GitHub You should clone this repo from github as you'll need to run the scripts on your local workstation: ```bash theme={"dark"} git clone https://github.com/lucidworks/fusion-cloud-native.git ``` You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts. ```bash theme={"dark"} cd fusion-cloud-native git pull ``` Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip). Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory. The [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_aks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_aks.sh) script provided in this repo is strictly optional. The script is mainly to help those new to Kubernetes and/or Fusion get started quickly. If you're already familiar with K8s, Helm, and AKS, then you use Helm directly to install Fusion into an existing cluster or one you create yourself using the process described `<>`. If you're new to Azure, then please visit [https://azure.microsoft.com/en-us/free/search/](https://azure.microsoft.com/en-us/free/search/) to set up an account. ### Set up the AKS CLI tools Before launching an AKS cluster, you need to install and configure `kubectl` and `az` using the links provided below: Required AKS Command-line Tools: * `kubectl`: [https://kubernetes.io/docs/tasks/tools/install-kubectl/](https://kubernetes.io/docs/tasks/tools/install-kubectl/) * `az`: [https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest) To confirm your account access and command-line tools are set up correctly, run the `az login` command (`az login –help` to see available options). ### Azure Prerequisites To launch a cluster in AKS (or pretty much do anything with Azure) you need to setup a Resource Group. Resource Groups are a way of organizing and managing related resources in Azure. For more information about resource groups, see [https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview#resource-groups](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview#resource-groups). You also need to choose a location where you want to spin up your AKS cluster, such as `westus2`. For a list of locations you can choose, see [https://azure.microsoft.com/en-us/global-infrastructure/locations/](https://azure.microsoft.com/en-us/global-infrastructure/locations/). Use the Azure console in your browser to create a resource group, or simply do: ```bash theme={"dark"} az group create -g $AZURE_RESOURCE_GROUP -l $AZURE_LOCATION ``` To recap, you should have the following requirements in place: * Azure Account set up. * `azure-cli` (`az`) command-line tools installed. * `az` login working. * Created an Azure Resource Group and selected a location to launch the cluster. ### Set up Fusion on AKS Download and run the [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_aks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_aks.sh) to install Fusion 5.x in a AKS cluster. To create a new cluster and install Fusion, simply do: ```bash theme={"dark"} ./setup_f5_aks.sh -c -p ``` If you don't want the script to create a cluster, then you need to create a cluster before running the script and simply pass the name of the existing cluster using the `-c` parameter. Use the `--help` option to see full script usage. By default, our script installs Fusion into the default namespace; think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster in separate namespaces. However, please do not install more than one Fusion release in the same namespace. You can override the namespace using the `-n` option. In addition, our script uses f5 for the Helm release name; you can customize this using the `-r` option. Helm uses the release name you provide to track a specific instance of an installation, allowing you to perform updates and rollback changes for that specific release only. You can also pass the `--preview` option to the script, which enables soon-to-be-released features for AKS, such as deploying a multi-zone cluster across 3 availability zones for higher availability guarantees. For more information about the Availability Zone feature, see [https://docs.microsoft.com/en-us/azure/aks/availability-zones](https://docs.microsoft.com/en-us/azure/aks/availability-zones). It takes a while for AKS to spin up the new cluster. The cluster will have three Standard\_D4\_v3 nodes which have 4 CPU cores and 16 GB of memory. Behind the scenes, our script calls the `az aks create` command. If using Helm V2, the `setup_f5_aks.sh` script installs Helm's `tiller` component into your AKS cluster with the cluster admin role. If you don't want this, then please upgrade to Helm v3. After running the `setup_f5_aks.sh` script, proceed to `<>`. #### AKS Ingress The `setup_f5_aks.sh` script exposes the Fusion proxy service on an external IP over HTTP. This is done for demo or getting started purposes. However, you're strongly encouraged to configure a K8s Ingress with TLS termination in front of the proxy service. Use the `-t` and `-h ` options to have our script create an Ingress with a TLS certificate issued by Let's Encrypt. #### Upgrades and Ingress IMPORTANT: If you used the `-t -h ` options when installing your cluster, our script created an additional values yaml file named `tls-values.yaml`. To make things easier for you when upgrading, you should add the settings from this file into your main custom values yaml file. For example: ```yaml theme={"dark"} api-gateway: service: type: "NodePort" ingress: enabled: true host: "" tls: enabled: true annotations: "networking.gke.io/managed-certificates": "-managed-certificate" "kubernetes.io/ingress.class": "gce" ``` This way, you don't have to remember to pass the additional `tls-values.yaml` file when upgrading. ## Verifying the Fusion Installation In this section, we provide some tips on how to verify the Fusion installation. Check if the Fusion Admin UI is available at `\https://:6764/admin/`. Let's review some useful kubectl commands. ### Enhance the K8s Command-line Experience Here is a list of tools we found useful for improving your command-line experience with Kubernetes: * krew (kubectl plugin mgr): [https://github.com/kubernetes-sigs/krew/](https://github.com/kubernetes-sigs/krew/) * kube-ps1 (show current context on command line prompt): [https://github.com/jonmosco/kube-ps1](https://github.com/jonmosco/kube-ps1) * kubectx / kubens (switch between clusters / namespaces): [https://github.com/ahmetb/kubectx](https://github.com/ahmetb/kubectx) ### Useful kubectl commands kubectl reference: [https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands) Set the namespace for `kubectl` if not using the default: ```bash theme={"dark"} kubectl config set-context --current --namespace= ``` **This saves you from having to pass `-n` with every command.** Get a list of running pods: `k get pods` Get logs for a pod using a label: `k logs –l app.kubernetes.io/component=query-pipeline` Get pod deployment spec and details: `k get pods -o yaml` Get details about a pod events: `k describe po ` Port forward to a specific pod: `k port-forward 8983:8983` SSH into a pod: `k exec -it -- /bin/bash` CPU/Memory usage report for pods: `k top pods` Forcefully kill a pod: `k delete po --force --grace-period 0` Scale up (or down) a deployment: `k scale deployment.v1.apps/ --replicas=N` Get a list of pod versions: `k get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq` ### Check Fusion Pods and Services Once the install script completes, you can check that all pods and services are available using: ```bash theme={"dark"} kubectl get pods ``` If all goes well, you should see a list of pods similar to: ```bash wrap theme={"dark"} NAME READY STATUS RESTARTS AGE seldon-controller-manager-6675874894-qxwrv 1/1 Running 0 8m45s f5-admin-ui-74d794f4f8-m5jms 1/1 Running 0 8m45s f5-ambassador-fd6b9b5dc-7ghf6 1/1 Running 0 8m43s f5-api-gateway-6b9998b9c-tmchk 1/1 Running 0 8m45s f5-auth-ui-7565564b4c-rdc74 1/1 Running 0 8m42s f5-classic-rest-service-0 1/1 Running 3 8m44s f5-devops-ui-77bb867ffb-fbzxd 1/1 Running 0 8m42s f5-fusion-admin-78b8f8fc7f-4d7l8 1/1 Running 0 8m42s f5-fusion-indexing-599c8d448-xzsvm 1/1 Running 0 8m44s f5-insights-665fd9f6fc-g5psw 1/1 Running 0 8m43s f5-job-launcher-84dd4c5c96-p8528 1/1 Running 0 8m44s f5-job-rest-server-6d44d964b8-xtnxw 1/1 Running 0 8m45s f5-logstash-0 1/1 Running 0 8m45s f5-ml-model-service-6987dc94c9-9ppp8 2/2 Running 1 8m45s f5-monitoring-grafana-5d499dbb58-pzw72 1/1 Running 0 10m f5-monitoring-prometheus-kube-state-metrics-54d6678dv9h7h 1/1 Running 0 10m f5-monitoring-prometheus-pushgateway-7d65c65b85-vwrwf 1/1 Running 0 10m f5-monitoring-prometheus-server-0 2/2 Running 0 10m f5-pm-ui-86cbc5bb65-nd2n8 1/1 Running 0 8m44s f5-pulsar-bookkeeper-0 1/1 Running 0 8m45s f5-pulsar-broker-b56cc776f-56msx 1/1 Running 0 8m45s f5-query-pipeline-5d75d7d5f4-l2mdf 1/1 Running 0 8m43s f5-connectors-7bb6cfc65f-7wfs2 1/1 Running 0 8m42s f5-connectors-backend-987fdc648-dldwv 1/1 Running 0 8m45s f5-rules-ui-6b9d55b78f-9hzzj 1/1 Running 0 8m43s f5-solr-0 1/1 Running 0 8m44s f5-solr-exporter-c4687c785-jsm7x 1/1 Running 0 8m45s f5-ui-6cdbcc68c6-rj9cq 1/1 Running 0 8m45s f5-webapps-6d6bb9bfd-hm4qx 1/1 Running 0 8m45s f5-workflow-controller-7b66679fb7-sjbvp 1/1 Running 0 8m44s f5-zookeeper-0 1/1 Running 0 8m45s ``` The number of pods per deployment / statefulset will vary based on your cluster size and replicaCount settings in your custom values YAML file. Also, don't worry if you see some pods having been restarted as that just means they were too slow to come up and Kubernetes killed and restarted them. You do want to see at least one pod running for every service. If a pod is not running after waiting a sufficient amount of time, use `kubectl logs ` to see the logs for that pod; to see the logs for previous versions of a pod, use: `kubectl logs -p`. You can also look at the actions Kubernetes performed on the pod using `kubectl describe po `. To see a list of Fusion services, do: ```bash theme={"dark"} kubectl get svc ``` For an overview of the various Fusion 5 microservices, see: [Fusion microservices](/docs/5/fusion/reference/microservices). Once you're ready to build a Fusion cluster for production, please see see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview). ### Upgrading with Zero Downtime One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. Fusion 5 allows customers to upgrade from Fusion 5.x.y to a later 5.x.z version on a live cluster with zero downtime or disruption of service. When Kubernetes performs a rolling update to an individual microservice, there will be a mix of old and new services in the cluster concurrently (only briefly in most cases) and requests from other services will be routed to both versions. Consequently, Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same 5.x line of releases. We also ensure stored configuration remains compatible in the same 5.x release line. Lucidworks releases minor updates to individual services frequently, so our customers can pull in those upgrades using Helm at their discretion. To upgrade your cluster at any time, use the `--upgrade` option with our setup scripts in this repo. The scripts in this repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks. To see what would be upgraded, you can pass the `--dry-run` option to the script. ### Grafana Dashboards Get the initial Grafana password from a K8s secret by doing: ```bash wrap theme={"dark"} kubectl get secret --namespace "${NAMESPACE}" ${RELEASE}-monitoring-grafana \ -o jsonpath="{.data.admin-password}" | base64 --decode ; echo ``` With Grafana, you can either setup a temporary port-forward to a Grafana pod or expose Grafana on an external IP using a K8s LoadBalancer. To define a LoadBalancer, do (replace \${RELEASE} with your Helm release label): ```bash wrap theme={"dark"} kubectl expose deployment ${RELEASE}-monitoring-grafana --type=LoadBalancer --name=grafana --port=3000 --target-port=3000 ``` You can use `kubectl get services --namespace ` to determine when the load balancer is setup and its IP address. Direct your browser to `http://:3000` and enter the username `admin@localhost` and the password that was returned in the previous step. This will log you into the application. It is recommended that you create another administrative user with a more desirable password. The dashboards and datasoure will be setup for you in grafana, simply navigate to `Dashboards` -> `Manage` to view the available dashboards ## More deployment options * How to deploy Fusion 5 in Google Kubernetes Engine * How to deploy Fusion 5 in Amazon Elastic Kubernetes Service * How to deploy Fusion on 5 other Kubernetes platforms ## Frequently Asked Questions **Can the stateful database services, for example, MySQL, be supported by an Azure PaaS service?** This option is not supported. In theory, it may be possible to implement this function. **Is it possible to use cross-zone storage solutions rather than volumes, such as Microsoft Azure file storage, for stateful services?** While in theory it may be possible to implement this functionality, the configuration has not been tested by Lucidworks and is not supported. ## Learn more The course for **Deploying Fusion 5** focuses on the prerequisite software needed to deploy Fusion, the necessary setup steps, and the physical act of deployment. The [`setup_f5_k8s.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_k8s.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository provides deployment support for any Kubernetes platform, including on-premise, private cloud, public cloud, and hybrid platforms. This script is used by the [`setup_f5_gke.sh`](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_gke.sh), [`setup_f5_eks.sh`](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_eks.sh), and [`setup_f5_aks.sh`](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_aks.sh) scripts, which provide additional platform-specific support for Google Kubernetes Engine (GKE), Amazon Elastic Kubernetes Service (EKS), and Azure Kubernetes Service (AKS). This topic explains how to deploy a Fusion cluster in Kubernetes using the [`setup_f5_k8s.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_k8s.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository. If you’re deploying on-premises or using a localized repository, you’ll need to use a private repository for Docker images. ## Prerequisites This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes. ### Release Name and Namespace Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) Kubernetes namespace to install Fusion into. Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*. However, please *do not* install more than one Fusion release in the same namespace. All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_ Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default. ### Install Helm Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster. Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform. On MacOS, you can do: ```bash theme={"dark"} brew install kubernetes-helm ``` If you already have helm installed, make sure you're using the latest version: ```bash theme={"dark"} brew upgrade kubernetes-helm ``` For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/) The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`. #### Helm User Permissions If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory. When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.: ```bash theme={"dark"} alias k=kubectl ``` To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into: ```bash theme={"dark"} k create namespace fusion-namespace ``` Apply the `role.yaml` and `cluster-role.yaml` files to that namespace ```bash theme={"dark"} k apply -f cluster-role.yaml k config set-context --current --namespace=$NAMESPACE k apply -f role.yaml ``` Then bind the rolebinding and clusterolebinding to the install user: ```bash wrap theme={"dark"} k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user ``` You will then be able to run the `helm install` command as the `` ### Clone fusion-cloud-native from GitHub You should clone this repo from github as you'll need to run the scripts on your local workstation: ```bash theme={"dark"} git clone https://github.com/lucidworks/fusion-cloud-native.git ``` You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts. ```bash theme={"dark"} cd fusion-cloud-native git pull ``` Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip). Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory. ## Deployment If you're not running on a managed K8s platform like GKE, AKS, or EKS, you can use Helm to install the Fusion chart to an existing Kubernetes cluster. Fusion version 5.5 now includes support for the Rancher Kubernetes Engine (RKE) platform. Before deploying Fusion to RKE, you must download and install the [latest RKE software](https://rancher.com/docs/rke/latest/en/). After configuring your cluster, you can proceed with the Helm v3 installation. You must have a working cluster configured before performing the Helm v3 installation. ### Use Helm v3 to Install Fusion You should upgrade to the latest version of Helm v3 for working with Fusion. If you need to keep Helm V2 for other clusters, ensure Helm V3 is ahead of Helm V2 in your working shell's PATH before proceeding. #### Customize Fusion Chart Settings Fusion aims to be well-configured out-of-the-box, but you can customize any of the built-in settings using a custom values `YAML` file. If you use one of our setup scripts, such as `setup_f5_gke.sh`, then it will create a custom values YAML file for you the first time you run it using [https://github.com/lucidworks/fusion-cloud-native/blob/master/customize\_fusion\_values.yaml.example](https://github.com/lucidworks/fusion-cloud-native/blob/master/customize_fusion_values.yaml.example) as a template. If you're working with Helm directly and not using one of our setup scripts, then run the [https://github.com/lucidworks/fusion-cloud-native/blob/master/customize\_fusion\_values.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/customize_fusion_values.sh) script to create a custom values YAML file from our [https://github.com/lucidworks/fusion-cloud-native/blob/master/customize\_fusion\_values.yaml.example](https://github.com/lucidworks/fusion-cloud-native/blob/master/customize_fusion_values.yaml.example) template as a starting point: ```bash theme={"dark"} ./customize_fusion_values.sh -c -n \ --provider --num-solr 1 --node-pool "" ``` Pass `--help` for usage details. In this example: * `` is the K8s platform you're running on, such as `gke` * `` is the name of your cluster * `` is the K8s namespace where you plan to install Fusion The `--node-pool` option specifies the node selector label for determining which nodes to run Fusion pods. You can pass `"{}"` to let Kubernetes decide which nodes to schedule pods on. This file is referred to as `${MY_VALUES}` in the commands belo. Replace the filename with the correct filename for your environment. Keep this file handy, as you'll need it to customize Fusion settings and upgrade to a newer version. Review the settings in the custom values YAML file to ensure the defaults are appropriate for your environment, including the number of Solr and Zookeeper replicas. Add the Lucidworks Helm repo: ```bash theme={"dark"} helm repo add lucidworks https://charts.lucidworks.com ``` The `customize_fusion_values.sh` script creates an upgrade script to install/upgrade Fusion into Kubernetes using Helm. Look in the directory where you ran `customize_fusion_values.sh` for a script named like: `___upgrade_fusion.sh`. Run this script to install Fusion. ## Verifying the Fusion Installation In this section, we provide some tips on how to verify the Fusion installation. Check if the Fusion Admin UI is available at `\https://:6764/admin/`. Let's review some useful kubectl commands. ### Enhance the K8s Command-line Experience Here is a list of tools we found useful for improving your command-line experience with Kubernetes: * krew (kubectl plugin mgr): [https://github.com/kubernetes-sigs/krew/](https://github.com/kubernetes-sigs/krew/) * kube-ps1 (show current context on command line prompt): [https://github.com/jonmosco/kube-ps1](https://github.com/jonmosco/kube-ps1) * kubectx / kubens (switch between clusters / namespaces): [https://github.com/ahmetb/kubectx](https://github.com/ahmetb/kubectx) ### Useful kubectl commands kubectl reference: [https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands) Set the namespace for `kubectl` if not using the default: ```bash theme={"dark"} kubectl config set-context --current --namespace= ``` **This saves you from having to pass `-n` with every command.** Get a list of running pods: `k get pods` Get logs for a pod using a label: `k logs –l app.kubernetes.io/component=query-pipeline` Get pod deployment spec and details: `k get pods -o yaml` Get details about a pod events: `k describe po ` Port forward to a specific pod: `k port-forward 8983:8983` SSH into a pod: `k exec -it -- /bin/bash` CPU/Memory usage report for pods: `k top pods` Forcefully kill a pod: `k delete po --force --grace-period 0` Scale up (or down) a deployment: `k scale deployment.v1.apps/ --replicas=N` Get a list of pod versions: `k get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq` ### Check Fusion Pods and Services Once the install script completes, you can check that all pods and services are available using: ```bash theme={"dark"} kubectl get pods ``` If all goes well, you should see a list of pods similar to: ```bash wrap theme={"dark"} NAME READY STATUS RESTARTS AGE seldon-controller-manager-6675874894-qxwrv 1/1 Running 0 8m45s f5-admin-ui-74d794f4f8-m5jms 1/1 Running 0 8m45s f5-ambassador-fd6b9b5dc-7ghf6 1/1 Running 0 8m43s f5-api-gateway-6b9998b9c-tmchk 1/1 Running 0 8m45s f5-auth-ui-7565564b4c-rdc74 1/1 Running 0 8m42s f5-classic-rest-service-0 1/1 Running 3 8m44s f5-devops-ui-77bb867ffb-fbzxd 1/1 Running 0 8m42s f5-fusion-admin-78b8f8fc7f-4d7l8 1/1 Running 0 8m42s f5-fusion-indexing-599c8d448-xzsvm 1/1 Running 0 8m44s f5-insights-665fd9f6fc-g5psw 1/1 Running 0 8m43s f5-job-launcher-84dd4c5c96-p8528 1/1 Running 0 8m44s f5-job-rest-server-6d44d964b8-xtnxw 1/1 Running 0 8m45s f5-logstash-0 1/1 Running 0 8m45s f5-ml-model-service-6987dc94c9-9ppp8 2/2 Running 1 8m45s f5-monitoring-grafana-5d499dbb58-pzw72 1/1 Running 0 10m f5-monitoring-prometheus-kube-state-metrics-54d6678dv9h7h 1/1 Running 0 10m f5-monitoring-prometheus-pushgateway-7d65c65b85-vwrwf 1/1 Running 0 10m f5-monitoring-prometheus-server-0 2/2 Running 0 10m f5-pm-ui-86cbc5bb65-nd2n8 1/1 Running 0 8m44s f5-pulsar-bookkeeper-0 1/1 Running 0 8m45s f5-pulsar-broker-b56cc776f-56msx 1/1 Running 0 8m45s f5-query-pipeline-5d75d7d5f4-l2mdf 1/1 Running 0 8m43s f5-connectors-7bb6cfc65f-7wfs2 1/1 Running 0 8m42s f5-connectors-backend-987fdc648-dldwv 1/1 Running 0 8m45s f5-rules-ui-6b9d55b78f-9hzzj 1/1 Running 0 8m43s f5-solr-0 1/1 Running 0 8m44s f5-solr-exporter-c4687c785-jsm7x 1/1 Running 0 8m45s f5-ui-6cdbcc68c6-rj9cq 1/1 Running 0 8m45s f5-webapps-6d6bb9bfd-hm4qx 1/1 Running 0 8m45s f5-workflow-controller-7b66679fb7-sjbvp 1/1 Running 0 8m44s f5-zookeeper-0 1/1 Running 0 8m45s ``` The number of pods per deployment / statefulset will vary based on your cluster size and replicaCount settings in your custom values YAML file. Also, don't worry if you see some pods having been restarted as that just means they were too slow to come up and Kubernetes killed and restarted them. You do want to see at least one pod running for every service. If a pod is not running after waiting a sufficient amount of time, use `kubectl logs ` to see the logs for that pod; to see the logs for previous versions of a pod, use: `kubectl logs -p`. You can also look at the actions Kubernetes performed on the pod using `kubectl describe po `. To see a list of Fusion services, do: ```bash theme={"dark"} kubectl get svc ``` For an overview of the various Fusion 5 microservices, see: [Fusion microservices](/docs/5/fusion/reference/microservices). Once you're ready to build a Fusion cluster for production, please see see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview). ### Upgrading with Zero Downtime One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. Fusion 5 allows customers to upgrade from Fusion 5.x.y to a later 5.x.z version on a live cluster with zero downtime or disruption of service. When Kubernetes performs a rolling update to an individual microservice, there will be a mix of old and new services in the cluster concurrently (only briefly in most cases) and requests from other services will be routed to both versions. Consequently, Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same 5.x line of releases. We also ensure stored configuration remains compatible in the same 5.x release line. Lucidworks releases minor updates to individual services frequently, so our customers can pull in those upgrades using Helm at their discretion. To upgrade your cluster at any time, use the `--upgrade` option with our setup scripts in this repo. The scripts in this repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks. To see what would be upgraded, you can pass the `--dry-run` option to the script. ### Grafana Dashboards Get the initial Grafana password from a K8s secret by doing: ```bash wrap theme={"dark"} kubectl get secret --namespace "${NAMESPACE}" ${RELEASE}-monitoring-grafana \ -o jsonpath="{.data.admin-password}" | base64 --decode ; echo ``` With Grafana, you can either setup a temporary port-forward to a Grafana pod or expose Grafana on an external IP using a K8s LoadBalancer. To define a LoadBalancer, do (replace \${RELEASE} with your Helm release label): ```bash wrap theme={"dark"} kubectl expose deployment ${RELEASE}-monitoring-grafana --type=LoadBalancer --name=grafana --port=3000 --target-port=3000 ``` You can use `kubectl get services --namespace ` to determine when the load balancer is setup and its IP address. Direct your browser to `http://:3000` and enter the username `admin@localhost` and the password that was returned in the previous step. This will log you into the application. It is recommended that you create another administrative user with a more desirable password. The dashboards and datasoure will be setup for you in grafana, simply navigate to `Dashboards` -> `Manage` to view the available dashboards ## Learn more The course for **Deploying Fusion 5** focuses on the prerequisite software needed to deploy Fusion, the necessary setup steps, and the physical act of deployment.