> ## Documentation Index
> Fetch the complete documentation index at: https://doc.lucidworks.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Cluster Monitoring and Troubleshooting

export const LwTemplate = ({title = "Key questions to get you started", icon = "sparkles", cta = "Powered by Agent Studio", linkHref = "https://lucidworks.com/demo/?utm_source=docs&utm_medium=referral&utm_campaign=docs_cta_ai"}) => {
  const [isLoaded, setIsLoaded] = useState(false);
  useEffect(() => {
    const timer = setTimeout(() => {
      setIsLoaded(true);
    }, 500);
    return () => clearTimeout(timer);
  }, []);
  return <div className="lw-template-container">
      <Card title={title} icon={icon}>
        {isLoaded && <span dangerouslySetInnerHTML={{
    __html: `<lw-template id="a029c1a9-28be-427e-b0e1-5d918920246a"></lw-template
            >`
  }} />}
        <Link href={linkHref} className="agent-studio-link text-left text-gray-600 gap-2 dark:text-gray-400 text-sm font-medium flex flex-row items-center hover:text-primary dark:hover:text-primary-light group-hover:text-primary group-hover:dark:text-primary-light">Powered by Lucidworks Agent Studio</Link>
      </Card>
    </div>;
};

[localhost link]: http://localhost:3000/docs/4/fusion-server/concepts/system/monitoring/cluster-monitoring

[mintlify link]: https://doc.lucidworks.com/docs/4/fusion-server/concepts/system/monitoring/cluster-monitoring

[old doc.lw link]: https://doc.lucidworks.com/fusion-server/4.2/133

<LwTemplate />

## Fusion structure and components

Fusion consists of a collection of services running on one or more nodes in a network, as summarized below.

<Note>
  The default ports listed under **Runs on** below can be altered in `fusion.properties` file located in `/fusion/4.2.x/conf/`.
</Note>

### Fusion services

| Service          | Runs on                                 | Description                                                                                                                                                                                                                              |
| ---------------- | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Fusion UI        | All nodes in Fusion cluster. Port 8763. | Fusion Admin Web App.                                                                                                                                                                                                                    |
| API              | All nodes in Fusion cluster. Port 8765. | The API service fulfills REST API requests, processes pipeline stages, and starts, stop, and sometimes monitors scheduled jobs.                                                                                                          |
| Connectors       | Fusion nodes. Port 8984.                | The connectors service runs connector jobs and processes pipeline stages.                                                                                                                                                                |
| Proxy            | All nodes in Fusion cluster. Port 8764. | The proxy service authenticates API requests sent to port 8764 and dispatches to an API or connectors service over port 8765 or 8984.                                                                                                    |
| Apache Solr      | All Signals nodes. Port 8983.           | Solr for various Fusion collections.  `signals`, `_aggr`, `logs`, `various system* collections` such as models, blobs, history, messages, metrics, etc. On larger installations, Solr is often installed and run separately from Fusion. |
| Apache Spark     | Signals nodes. Various ports.           | The Apache Spark service provides large-scale data processing jobs for Spark SQL, aggregation, machine learning, etc.                                                                                                                    |
| Web Apps         | All nodes in Fusion cluster. Port 8780. | The web apps service runs installed App Studio and/or Rules Editor web applications (war).                                                                                                                                               |
| Apache Zookeeper | Port 9983.                              | Apache ZooKeeper configures and manages all Fusion components in a single Fusion deployment. ZooKeeper is often installed and run separately from Fusion. In this case, it typically runs on port 2181.                                  |

### Typical directory structure

```bash theme={"dark"}
├── bin   # scripts for start|stop|status of Fusion services
├── conf  # contains config for Fusion processes and logging
├── data  # data directories (often linked or mounted to larger disk)
│   ├── connectors
│   ├── nlp
│   ├── solr
│   └── zookeeper
└── var  # default location for log files.
    ├── log
       ├── admin-ui
       ├── agent  # agent used to start/stop services
       ├── api    # API service calls, jobs, and pipeline stage output
       ├── connectors
       │   │   ├── connectors-classic  # Connector and pipeline output
       │   │   └── connectors-rpc
       ├── log-shipper
       ├── proxy
       ├── solr
       ├── spark-master
       ├── spark-worker
       ├── sql
       ├── webapps
       └── zookeeper
```

In general, each log directory contains a log named after the directory followed by a `.log` suffix. garbage collection (GC) logs are also located in these directories. The GC logs show internal timings for Java garbage collection cycles, as well as logs showing Jetty requests and errors for the given service. Each Fusion service has a `log4j2.xml` settings file located in `/fusion/{fusion}.x/conf` which controls the particulars of log retention and contents.

By default, file rotation is enabled in the `log4j2.xml` settings file. File rotation ensures a new log file is created under the following conditions:

* A 24-hour period has elapsed
* A log file has exceeded 100MB