Skip to main content
The DevOps Center is a suite of tools for monitoring, troubleshooting, and incident investigation. It consists of a set of dashboards and an interactive log viewer, providing views into this Fusion cluster’s hosts and services using metrics and events.
  • Metrics come from the system_monitor collection.
    See System Metrics for details about this collection.
  • Events come from the system_logs collection.
You can Export Data from the DevOps Center all metrics and events for any selected time period.
The DevOps Center can export metrics and events in CSV format for the currently-selected time period. Exported metrics and events are useful for collaborating with other Fusion administrators, using third-party analysis tools, or sending troubleshooting data to the Lucidworks support team.
  1. How to export data from the DevOps Center
  2. In the Fusion UI, navigate to System > DevOps Center > Log Viewer.
  3. Select the time period that is relevant to the incident you are interested in. Log files will be exported according to the tab you are viewing.
Export CSV Files buttonClick Export service logs in the Service Logs tab or Export request logs in the Request Logs tab to download a .csv file containing the logs (shown here with example timestamps):
  • serviceLogs[2019-02-07T22:28:49.000Z-2019-02-07T22:43:49.695Z].csv
  • requestLogs[2019-02-07T22:28:49.000Z-2019-02-07T22:43:49.695Z].csv
In any Fusion app, you can open the DevOps Center by navigating to System > DevOps Center: Navigating to the DevOps Center
In the DevOps Center, dates and times are local, as determined by your browser. In the Log Viewer, you can toggle between local time and UTC.

​
System requirements

The DevOps Center is included as part of Fusion Server 4.2 and later. It requires the following:
  • Your Web browser must support HTML5.
  • Every node that you want to monitor must be running Fusion Server version 4.2.0 or later.
  • Every node that you want to monitor must be running the agent and log-shipper services.
The DevOps Center is enabled by Fusion’s default configuration. If you find that the DevOps Center is not correctly populated with data, see Troubleshooting.

​
Cluster dashboard

The first screen that displays is a high-level dashboard showing indicators of general health of your Fusion cluster. DevOps Center Cluster dashboard The Cluster dashboard includes this data:
  • Fusion license information
  • The number of hosts, collections, and Solr clusters in this cluster
  • The cluster’s service status
  • Average and max values during the selected time interval for the following metrics:
    • Query rate
    • Response time
    • Index size
    • Indexing rate
    • Active sessions

​
Hosts dashboard

Data about individual hosts is available in the Hosts dashboard. The initial view includes all hosts. DevOps Center Hosts dashboard From here you can filter the set of hosts displayed:
  • Select one or more status codes to display only the hosts with those statuses.
  • Select one or more services to display only the hosts on which they are running.
Click on any host’s IP address to drill down to additional visualizations about that host: Single-host view In this single-host view, you can see the following: Host details
  • Cores
  • IP address
  • Operating system
  • Uptime
  • CPU usage
  • CPU load
  • Memory total
  • Memory free
  • Disk total
  • Disk free
  • ulimit files
Host graphs and tables
  • CPU utilization
  • CPU load
  • Per-service CPU utilization
  • Time spent in garbage collection (GC)
  • Free memory
  • Free disk space
  • Overview of services running on the host

​
Services dashboard

The Services dashboard displays the status of each of the Fusion services: “OK” for services that are running and “Bad” for services that are unreachable. Metrics are also displayed for each service, and these vary depending on the service. A service is marked “Bad” if it does not return metrics after several expected reporting intervals, that is, after a few minutes. DevOps Center Services dashboard

​
Datasources dashboard

The Datasources dashboard displays the status, number of fetched documents, number of failed documents, and number of indexed documents for each Fusion datasource. By default, the dashboard displays only the datasources for the currently-selected app. Select one or more apps on the left to view all of the datasources in those apps. DevOps Datasources dashboard

​
The Log Viewer

The Log Viewer displays service logs and request logs from Fusion’s system_logs collection. DevOps Log Viewer dashboard By default, timestamps are displayed in local time, as determined by your browser. To view timestamps in UTC (unless default.timezone is set to another time zone in fusion.properties), set LOCAL TIMEZONE to “Off”. Auto-Refesh is off by default, to display static log data. To display real-time logs, set this to “On”. See Export Data from the DevOps Center to learn how to export log data.

​
Service logs

Service log files are written to the filesystem by each running service, such as var/log/api/api.log, var/log/proxy/proxy.log, and so on. Fusion indexes them in the system_logs collection with type=java. In the DevOps Center, you can filter service logs by:
  • Service
  • Log level
  • Host

​
Request logs

HTTP request log files are written to the filesystem by Jetty, at var/log/proxy/jetty_request_<date>.log. Fusion indexes them in the system_logs collection with type=http. In the DevOps Center, you can filter request logs by:
  • HTTP status code
  • HTTP method
  • Host

​
Learn more

The DevOps Center is enabled by Fusion’s default configuration. If you find that the DevOps Center is not correctly populated with data, check the following:
  • Check for browser extensions that block HTML5 autoplay.
  • Check that the Fusion log-shipper service is enabled, as it is by default: 
    fusion/latest.x/bin/log-shipper status
    If it is not enabled, make sure it is included in the list of default services in fusion.cors (fusion.properties in Fusion 4.x), as in this example: 
    group.default = zookeeper, solr, api, connectors-classic, connectors-rpc, proxy, webapps, admin-ui, sql, log-shipper
    If you are running some services on separate nodes, make sure those nodes are also running Fusion’s agent and log-shipper.
  • In fusion.cors (fusion.properties in Fusion 4.x), check that collectMetrics is “true” and collectMetricsIntervalSecs is set to a reasonable interval: 
    # capture host and services metrics
# and ship them to system_monitor Solr collection
default.collectMetrics = true
default.collectMetricsIntervalSecs = 30