Directories and Logs

Directories

The directory where the Fusion files go for a specific version of Fusion is the Fusion home directory. The Fusion home directory is a version-numbered directory (for example, 3.1.0) below the directory fusion. This installation strategy lets you install multiple versions of Fusion and switch between them. The Fusion home directory is the directory fusion.

The directories found in the Fusion home directory fusion/3.0.x are:

Name Description

apps

Fusion components 3rd-party distributions used by Fusion, including jar files and plugins

bin

Master script to run Fusion, and per-component run scripts

conf

Configuration files for Fusion and ZooKeeper that contain parameters settings tuned for common use cases

data

Default location of data stores used by Fusion apps

docs

License information

examples

Fusion signals example

init

systemd and upstart scripts and configurations for Linux

scripts

Developer utilities, including diagnostic scripts, for Linux and Windows. See scripts/diag/linux/README and scripts/diag/win64/README.txt for details.

var

Logfiles and system files created by Fusion components, as well as .pid files for each running process

To simplify access to the latest version of Fusion and to files in the bin, conf, and var directories, Fusion creates a symbolic link latest to the latest version and symbolic links bin, conf, and var to latest/bin, latest/conf, and latest/var respectively.

For example, if latest is 3.1.0, then instead of entering this command to change to the bin directory:

$ cd /path/to/fusion/3.1.0/bin

You could just type:

$ cd /path/to/fusion/bin

To avoid possible confusion in the documentation, we spell out the path below the fusion directory.

From the fusion directory, you can view the symbolic links by typing:

$ ls -l . | grep "\->"

To change the version of Fusion to which the symbolic links refer, unlink and relink latest, for example:

$ cd /path/to/fusion
$ unlink latest
$ ln -s 3.1.2 latest

Logfiles

Logfiles are found in directories under fusion/3.0.x/var/log. Because the Fusion components run in separate JVMs, each component has its own set of logfiles and files that monitor all garbage-collection events for that process.

Name Description

api

Fusion REST API services logging and error messages. This log shows the result of service requests submitted to the REST API directly via HTTP and indirectly via the Fusion UI.

connectors

Fusion connector services logging and error messages. Fusion index pipeline logging stages write to this file.

solr

Messages from Solr

spark-master

Spark-master logs

spark-worker

Spark-worker logs

ui

Fusion UI messages

zookeeper

ZooKeeper messages

Every component logs all messages to a logfile named <component>.log. For example, the full path to the logfile for the connectors services is:

fusion/3.0.x/var/log/connectors/connectors.log

In addition to component logfiles, every component maintains a set of garbage-collection logfiles which are used for resource tuning. The garbage-collection logfiles are named gc_<YYYYMMDD>_<PID>.log.<CT>. In addition, the current garbage-collection logfile has suffix .current.

The Fusion REST API, UI, connectors services, and Solr all run inside a Jetty server. The Jetty server logs are also written to that component’s logfile directory. The Jetty server logs are named:

  • jetty-YYYY_MM_DD.request.log

  • jetty-YYYY_MM_DD.stderrout.log

Fusion uses the Apache Log4j 2 logging framework with Jetty to log each of the Fusion components. Logging is configured via an xml configuration file named log4j2.xml. Log levels, frequencies, and log rotation policy can be configured by changing these configuration files:

API service

fusion/3.0.x/conf/api-log4j2.xml

connectors

fusion/3.0.x/conf/connectors-log4j2.xml

Solr

fusion/3.0.x/conf/solr-log4j2.xml

Spark

fusion/3.0.x/conf/spark-driver-log4j2.xml

fusion/3.0.x/conf/spark-master-agent-log4j2.xml

fusion/3.0.x/conf/spark-master-log4j2.xml

fusion/3.0.x/conf/spark-worker-agent-log4j2.xml

fusion/3.0.x/conf/spark-worker-log4j2.xml

UI

fusion/3.0.x/conf/ui-log4j2.xml

ZooKeeper

fusion/3.0.x/conf/zk-log4j2.xml

The Log4j2 Configuration guide provides documentation and examples of all logging configuration options.