> ## Documentation Index > Fetch the complete documentation index at: https://doc.lucidworks.com/llms.txt > Use this file to discover all available pages before exploring further. # Monitoring export const LwTemplate = ({title = "Key questions to get you started", icon = "sparkles", cta = "Powered by Agent Studio", linkHref = "https://lucidworks.com/demo/?utm_source=docs&utm_medium=referral&utm_campaign=docs_cta_ai"}) => { const [isLoaded, setIsLoaded] = useState(false); useEffect(() => { const timer = setTimeout(() => { setIsLoaded(true); }, 500); return () => clearTimeout(timer); }, []); return
{isLoaded && ` }} />} Powered by Lucidworks Agent Studio
; }; [localhost link]: http://localhost:3000/docs/5/fusion/operations/monitoring-and-reporting/overview [mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/monitoring-and-reporting/overview [old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/132 Fusion has functions built in for observability and monitoring with Fusion’s tools or third-party monitoring tools. Fusion system metrics are logged and available in the UI, in log files, and through the REST API. You can use the following third-party tools to process and display metrics: * Prometheus is a solution that is installed during the Fusion installation. Prometheus records metrics, stores them in a time series datastore, and then uses queries to return data for visualizations and alerts. For more information, see the [Prometheus documentation](https://prometheus.io/docs). * Grafana is a tool installed during the Fusion installation. Grafana provides visualizations of multiple datasources and connects to Prometheus, which lets you access your Fusion data when you create or edit the Grafana dashboards. Fusion provides pre-configured Grafana dashboards, but you can also create custom dashboards to analyze metrics further. For more information, see the [Grafana documentation](https://grafana.com/docs). * Loki is an optional, additional tool that stores, aggregates, and queries logs. You can filter logs based on keywords from specific Fusion namespaces, services, and other elements. For more information, see the [Grafana’s Loki documentation](https://grafana.com/docs/loki). For more information about configuring these tools, see **Configure Grafana, Prometheus, Promtail, and Loki in Fusion**. Before you perform these installation instructions, you must delete any existing persistent volume claims (PVCs) related to Prometheus, Grafana, Promtail, and Loki on your namespace. ## Clone the `fusion-cloud-native` repository Open a terminal window and run following command: ``` git clone https://github.com/lucidworks/fusion-cloud-native.git ``` ## Install Grafana 1. In your local `fusion-cloud-native` repository, run the following command for your `` and ``: ``` ./install-prom.sh -c -n ``` The following is a sample output. The errors are related to resource limits on the sample cluster, and can be ignored. Similar errors may display for your cluster, and do not impact Grafana logging. ``` Adding the stable chart repo to helm repo list "prometheus-community" already exists with the same configuration, skipping "grafana" already exists with the same configuration, skipping Installing Prometheus and Grafana for monitoring Fusion metrics ... this can take a few minutes. Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "ckotzbauer" chart repository ...Successfully got an update from the "lucidworks" chart repository ...Successfully got an update from the "grafana" chart repository ...Successfully got an update from the "prometheus-community" chart repository Update Complete. ⎈Happy Helming!⎈ Saving 2 charts Downloading prometheus from repo https://prometheus-community.github.io/helm-charts Downloading grafana from repo https://grafana.github.io/helm-charts Deleting outdated charts Release "fe-foundry-monitoring" does not exist. Installing it now. Error: context deadline exceeded Successfully installed Prometheus and Grafana into the fe-foundry namespace. NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION fe-foundry fe-foundry 11 2023-08-07 15:10:55.373825 -0700 PDT deployed fusion-5.8.0 5.8.0 fe-foundry-jupyter fe-foundry 2 2023-07-20 11:29:38.481329 -0700 PDT deployed fusion-jupyter-0.2.5 1.0 fe-foundry-monitoring fe-foundry 1 2023-08-10 11:41:06.113257 -0700 PDT failed fusion-monitoring-1.0.1 1.0.1 ``` 2. Using the Grafana service endpoint in the newly-installed Grafana helm release, run the following command: ``` kubectl get services ``` The following is a sample output. ``` NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE grafana LoadBalancer 3000:32589/TCP 87m ``` If the output does not display, run the following command to expose Grafana, including an `EXTERNAL_IP` for your Grafana LoadBalancer service: ``` kubectl expose deployment --type=LoadBalancer --name=grafana --port=3000 --target-port=3000 ``` ## Install Loki To obtain Loki from the helm chart repository, run the following command for the unique `` for your cluster: ``` helm upgrade --install --namespace= grafana/loki-stack ``` If you do no enter the `` correctly, an error similar to the following displays: ``` Error: rendered manifests contain a resource that already exists. Unable to continue with install: PodSecurityPolicy "loki" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-namespace" must equal "fe-foundry": current value is "ps-intl". ``` If the helm upgrade is successful, the following is a sample output. ``` Release "fe-foundry-loki" does not exist. Installing it now. W0810 11:47:07.890370 39624 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ W0810 11:47:09.396246 39624 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ NAME: fe-foundry-loki LAST DEPLOYED: Thu Aug 10 11:47:07 2023 NAMESPACE: fe-foundry STATUS: deployed REVISION: 1 NOTES: The Loki stack has been deployed to your cluster. Loki can now be added as a datasource in Grafana. See http://docs.grafana.org/features/datasources/loki/ for more detail. ``` ## Obtain Admin credentials for Grafana 1. After you validate Grafana is running by accessing `:3000`, run the following command to obtain an `` for your Grafana instance: ``` kubectl get secret --namespace -monitoring-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo ``` 2. Sign in to Grafana and change the password for security purposes. 3. Run the following command to display the promtail pods that are running: ``` kubectl get pods | grep -i promtail | nl ``` Promtail pods must match the number of Kubernetes nodes since an instance of Promtail runs on each node. ## Add the Loki datsource 1. Sign in to Grafana and in the toolbar, click the arrow below Home to display all of the options. 2. In the Configuration section, click **Data sources**. 3. Click **Add new data source**. 4. In the search bar for the data source, enter **Loki**. 5. In the URL field on the Settings screen, enter your unique ``. The default port for Loki is `3100`. If you encounter issues with the `` information, open a terminal and run `kubectl get services | grep loki` to display a list of every service with a name that contains `loki` along with its associated IP address and port. 6. Complete the other fields and click **Save & test**. ## Splunk logging If your policy requires Splunk instead of Grafana, use the instructions in **Logging to Splunk**. Fusion services write log messages to `stdout` by default. You can use OpenShift as your cluster’s logging pipeline to send container `stdout` and `stderr` to Splunk. ### Forward `stdout` to Splunk 1. Set Fusion logging values in your custom values YAML to enable JSON logs. This is optional, but recommended to make parsing easier. ```yaml theme={"dark"} # values.yaml global: logging: jsonOutput: true ``` 2. Configure your logging stack to ship pod `stdout` to Splunk. ### OpenShift If Fusion runs on OpenShift and already writes to `stdout`, use OpenShift’s log forwarding to Splunk HTTP Event Collector (HEC). 1. Create a secret with your Splunk HEC token: ```bash theme={"dark"} oc -n openshift-logging create secret generic splunk-hec-secret \ --from-literal=hecToken=YOUR_HEC_TOKEN ``` 2. Create or update a ClusterLogForwarder to send application logs to Splunk: ```yaml theme={"dark"} apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: name: instance namespace: openshift-logging spec: outputs: - name: splunk type: splunk url: https://splunk.example.com:8088 secret: name: splunk-hec-secret pipelines: - name: app-to-splunk inputRefs: [ application ] outputRefs: [ splunk ] ``` 3. Verify ingestion in Splunk. Search for Kubernetes labels such as `kubernetes.pod_name` or `app.kubernetes.io/name`. OpenShift provides its own collection and forwarding path so you do not need to install extra agents inside Fusion pods.