Monitoring

Fusion has functions built in for observability and monitoring with Fusion’s tools or third-party monitoring tools. Fusion system metrics are logged and available in the UI, in log files, and through the REST API. You can use the following third-party tools to process and display metrics:

Prometheus is a solution that is installed during the Fusion installation. Prometheus records metrics, stores them in a time series datastore, and then uses queries to return data for visualizations and alerts. For more information, see the Prometheus documentation.
Grafana is a tool installed during the Fusion installation. Grafana provides visualizations of multiple datasources and connects to Prometheus, which lets you access your Fusion data when you create or edit the Grafana dashboards. Fusion provides pre-configured Grafana dashboards, but you can also create custom dashboards to analyze metrics further. For more information, see the Grafana documentation.
Loki is an optional, additional tool that stores, aggregates, and queries logs. You can filter logs based on keywords from specific Fusion namespaces, services, and other elements. For more information, see the Grafana’s Loki documentation.

For more information about configuring these tools, see Configure Grafana, Prometheus, Promtail, and Loki in Fusion.

Configure Grafana, Prometheus, Promtail, and Loki in Fusion

Before you perform these installation instructions, you must delete any existing persistent volume claims (PVCs) related to Prometheus, Grafana, Promtail, and Loki on your namespace.

Clone the `fusion-cloud-native` repository

Open a terminal window and run following command:

git clone https://github.com/lucidworks/fusion-cloud-native.git

Install Grafana

In your local fusion-cloud-native repository, run the following command for your <cluster> and <namespace>:

./install-prom.sh -c <cluster> -n <namespace>

The following is a sample output. The errors are related to resource limits on the sample cluster, and can be ignored. Similar errors may display for your cluster, and do not impact Grafana logging.

Adding the stable chart repo to helm repo list
"prometheus-community" already exists with the same configuration, skipping
"grafana" already exists with the same configuration, skipping

Installing Prometheus and Grafana for monitoring Fusion metrics ... this can take a few minutes.

Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "ckotzbauer" chart repository
...Successfully got an update from the "lucidworks" chart repository
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "prometheus-community" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 2 charts
Downloading prometheus from repo https://prometheus-community.github.io/helm-charts
Downloading grafana from repo https://grafana.github.io/helm-charts
Deleting outdated charts
Release "fe-foundry-monitoring" does not exist. Installing it now.
Error: context deadline exceeded


Successfully installed Prometheus and Grafana into the fe-foundry namespace.

NAME                 	NAMESPACE 	REVISION	UPDATED                             	STATUS  	CHART                  	APP VERSION
fe-foundry           	fe-foundry	11      	2023-08-07 15:10:55.373825 -0700 PDT	deployed	fusion-5.8.0           	5.8.0
fe-foundry-jupyter   	fe-foundry	2       	2023-07-20 11:29:38.481329 -0700 PDT	deployed	fusion-jupyter-0.2.5   	1.0
fe-foundry-monitoring	fe-foundry	1       	2023-08-10 11:41:06.113257 -0700 PDT	failed  	fusion-monitoring-1.0.1	1.0.1

Using the Grafana service endpoint in the newly-installed Grafana helm release, run the following command:

kubectl get services

The following is a sample output.

NAME          TYPE           CLUSTER-IP     EXTERNAL-IP    PORT(S)          AGE
grafana       LoadBalancer   <IP Address>   <IP Address>  3000:32589/TCP    87m

If the output does not display, run the following command to expose Grafana, including an EXTERNAL_IP for your Grafana LoadBalancer service:

kubectl expose deployment <grafana-deployment-name> --type=LoadBalancer --name=grafana --port=3000 --target-port=3000

Install Loki

To obtain Loki from the helm chart repository, run the following command for the unique <loki-release-name> for your cluster:

helm upgrade --install <loki-release-name> --namespace=<namespace> grafana/loki-stack

If you do no enter the <loki-release-name> correctly, an error similar to the following displays:

Error: rendered manifests contain a resource that already exists. Unable to continue with install: PodSecurityPolicy "loki" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-namespace" must equal "fe-foundry": current value is "ps-intl".

If the helm upgrade is successful, the following is a sample output.

Release "fe-foundry-loki" does not exist. Installing it now.
W0810 11:47:07.890370   39624 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0810 11:47:09.396246   39624 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME: fe-foundry-loki
LAST DEPLOYED: Thu Aug 10 11:47:07 2023
NAMESPACE: fe-foundry
STATUS: deployed
REVISION: 1
NOTES:
The Loki stack has been deployed to your cluster. Loki can now be added as a datasource in Grafana.

See http://docs.grafana.org/features/datasources/loki/ for more detail.

Obtain Admin credentials for Grafana

After you validate Grafana is running by accessing <EXTERNAL-IP>:3000, run the following command to obtain an <admin_password> for your Grafana instance:
```
kubectl get secret --namespace <namespace> <release_name>-monitoring-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```
Sign in to Grafana and change the password for security purposes.
Run the following command to display the promtail pods that are running:
```
kubectl get pods | grep -i promtail | nl
```
Promtail pods must match the number of Kubernetes nodes since an instance of Promtail runs on each node.

Add the Loki datsource

Sign in to Grafana and in the toolbar, click the arrow below Home to display all of the options.
In the Configuration section, click Data sources.
Click Add new data source.
In the search bar for the data source, enter Loki.
In the URL field on the Settings screen, enter your unique <loki-release-name:port>. The default port for Loki is 3100.
If you encounter issues with the <loki-release-name:port> information, open a terminal and run kubectl get services | grep loki to display a list of every service with a name that contains loki along with its associated IP address and port.
Complete the other fields and click Save & test.

Introduction to Fusion

Getting Data In

Getting Data Out

Operations

Reference

Developer Docs

Neural Hybrid Search

Release Notes

Clone the `fusion-cloud-native` repository

Install Grafana

Install Loki

Obtain Admin credentials for Grafana

Add the Loki datsource