> ## Documentation Index
> Fetch the complete documentation index at: https://doc.lucidworks.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Prometheus, Grafana, and Loki in Fusion
export const LwTemplate = ({title = "Key questions to get you started", icon = "sparkles", cta = "Powered by Agent Studio", linkHref = "https://lucidworks.com/demo/?utm_source=docs&utm_medium=referral&utm_campaign=docs_cta_ai"}) => {
const [isLoaded, setIsLoaded] = useState(false);
useEffect(() => {
const timer = setTimeout(() => {
setIsLoaded(true);
}, 500);
return () => clearTimeout(timer);
}, []);
return
{isLoaded && `
}} />}
Powered by Lucidworks Agent Studio
;
};
[localhost link]: http://localhost:3000/docs/5/fusion/operations/monitoring-and-reporting/prometheus-grafana-loki
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/monitoring-and-reporting/prometheus-grafana-loki
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/131
Fusion 5.6 and later utilizes Prometheus, Grafana, and Loki for metrics and log analytics.
## Prometheus
Prometheus provides a powerful solution for recording metrics, persisting them to a pluggable time series datastore, and querying them for both visualizations and alerting purposes. Metrics data is pulled from Fusion and stored in a time series datastore.
For more information, see the [Prometheus documentation](https://prometheus.io/docs).
## Grafana
Grafana provides an industry-standard metric visualization tool that supports complex visualizations over any number of different pluggable data sources, organized into panels and managed dashboards. Grafana connects Prometheus, allowing you to access your Fusion data when creating or editing panels in your dashboards.
Pre-configured Grafana dashboards are provided with Fusion deployments. However, custom dashboards extend your metrics analysis capabilities.
Lucidworks Search offers read-only access for Grafana. If you are a Lucidworks Search customer, you cannot create custom dashboards.
For more information, see the [Grafana documentation](https://grafana.com/docs).
## Loki
Loki is a log aggregation system designed to store and query logs. With Loki, you can define a log stream from Fusion’s namespaces, services, pods, and more. Then, you can filter the log stream for key data, such as keyword matches for "error."
For more information, see the [Grafana’s Loki documentation](https://grafana.com/docs/loki).
## Setup and configuration
Prometheus and Grafana are enabled during Fusion installation, and Loki is an optional addition to the Prometheus and Grafana integration. For more information, see **Configure Grafana, Prometheus, Promtail, and Loki in Fusion**.
Before you perform these installation instructions, you must delete any existing persistent volume claims (PVCs) related to Prometheus, Grafana, Promtail, and Loki on your namespace.
## Clone the `fusion-cloud-native` repository
Open a terminal window and run following command:
```
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
## Install Grafana
1. In your local `fusion-cloud-native` repository, run the following command for your `` and ``:
```
./install-prom.sh -c -n
```
The following is a sample output. The errors are related to resource limits on the sample cluster, and can be ignored. Similar errors may display for your cluster, and do not impact Grafana logging.
```
Adding the stable chart repo to helm repo list
"prometheus-community" already exists with the same configuration, skipping
"grafana" already exists with the same configuration, skipping
Installing Prometheus and Grafana for monitoring Fusion metrics ... this can take a few minutes.
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "ckotzbauer" chart repository
...Successfully got an update from the "lucidworks" chart repository
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "prometheus-community" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 2 charts
Downloading prometheus from repo https://prometheus-community.github.io/helm-charts
Downloading grafana from repo https://grafana.github.io/helm-charts
Deleting outdated charts
Release "fe-foundry-monitoring" does not exist. Installing it now.
Error: context deadline exceeded
Successfully installed Prometheus and Grafana into the fe-foundry namespace.
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
fe-foundry fe-foundry 11 2023-08-07 15:10:55.373825 -0700 PDT deployed fusion-5.8.0 5.8.0
fe-foundry-jupyter fe-foundry 2 2023-07-20 11:29:38.481329 -0700 PDT deployed fusion-jupyter-0.2.5 1.0
fe-foundry-monitoring fe-foundry 1 2023-08-10 11:41:06.113257 -0700 PDT failed fusion-monitoring-1.0.1 1.0.1
```
2. Using the Grafana service endpoint in the newly-installed Grafana helm release, run the following command:
```
kubectl get services
```
The following is a sample output.
```
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
grafana LoadBalancer 3000:32589/TCP 87m
```
If the output does not display, run the following command to expose Grafana, including an `EXTERNAL_IP` for your Grafana LoadBalancer service:
```
kubectl expose deployment --type=LoadBalancer --name=grafana --port=3000 --target-port=3000
```
## Install Loki
To obtain Loki from the helm chart repository, run the following command for the unique `` for your cluster:
```
helm upgrade --install --namespace= grafana/loki-stack
```
If you do no enter the `` correctly, an error similar to the following displays:
```
Error: rendered manifests contain a resource that already exists. Unable to continue with install: PodSecurityPolicy "loki" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-namespace" must equal "fe-foundry": current value is "ps-intl".
```
If the helm upgrade is successful, the following is a sample output.
```
Release "fe-foundry-loki" does not exist. Installing it now.
W0810 11:47:07.890370 39624 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0810 11:47:09.396246 39624 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME: fe-foundry-loki
LAST DEPLOYED: Thu Aug 10 11:47:07 2023
NAMESPACE: fe-foundry
STATUS: deployed
REVISION: 1
NOTES:
The Loki stack has been deployed to your cluster. Loki can now be added as a datasource in Grafana.
See http://docs.grafana.org/features/datasources/loki/ for more detail.
```
## Obtain Admin credentials for Grafana
1. After you validate Grafana is running by accessing `:3000`, run the following command to obtain an `` for your Grafana instance:
```
kubectl get secret --namespace -monitoring-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```
2. Sign in to Grafana and change the password for security purposes.
3. Run the following command to display the promtail pods that are running:
```
kubectl get pods | grep -i promtail | nl
```
Promtail pods must match the number of Kubernetes nodes since an instance of Promtail runs on each node.
## Add the Loki datsource
1. Sign in to Grafana and in the toolbar, click the arrow below Home to display all of the options.
2. In the Configuration section, click **Data sources**.
3. Click **Add new data source**.
4. In the search bar for the data source, enter **Loki**.
5. In the URL field on the Settings screen, enter your unique ``. The default port for Loki is `3100`.
If you encounter issues with the `` information, open a terminal and run `kubectl get services | grep loki` to display a list of every service with a name that contains `loki` along with its associated IP address and port.
6. Complete the other fields and click **Save & test**.
* [Grafana Loki for log exploration](/docs/5/fusion/operations/monitoring-and-reporting/grafana-loki)