Directories and Logs

Directories

The directories found in the home directory of your Fusion installation ($FUSION) are:

Name Description

apps

Fusion components 3rd-party distributions used by Fusion, including jarfiles and plugins.

bin

master script to run Fusion, per-component run scripts.

conf

configuration files for Fusion and ZooKeeper which contain parameters settings tuned for common use cases.

data

default location of data stores used by Fusion apps.

docs

license information.

examples

Fusion signals example.

init

systemd and upstart scripts and configurations for Linux.

scripts

Developer utilities, including diagnostic scripts, for Linux and Windows. See scripts/diag/linux/README and scripts/diag/win64/README.txt for details.

var

logfiles and system files created by Fusion components as well as .pid files for each running process.

Logfiles

Logfiles are found in directories under $FUSION/var/log. Because the Fusion components run in separate JVMs, each component has its own set of logfiles and files which monitor all garbage collection events for that process.

Name Description

api

Fusion REST-API services logging and error messages. This log shows the result of service requests submitted to the REST-API directly via HTTP and indirectly via the Fusion UI.

connectors

Fusion connector services logging and error messages. Fusion index pipeline logging stages write to this file.

solr

messages from Solr

spark-master

Spark-master logs

spark-worker

Spark-worker logs

ui

Fusion UI messages.

zookeeper

ZooKeeper messages

Every component logs all messages to a logfile named "<component>.log", e.g., the full path to the logfile for the connectors services is:

$FUSION/var/log/connectors/connectors.log

In addition to component logfiles, every component maintains a set of gc logfiles which are used for resource tuning. The gc logfiles are named "gc_<YYYYMMDD>_<PID>.log.<CT>, in addition the current gc logfile has suffix ".current".

The Fusion REST-API, UI, connectors services and Solr are all running inside a jetty server, and the jetty server logs are also written to that component’s logfile directory. The Jetty server logs are named:

  • jetty-YYYY_MM_DD.request.log

  • jetty-YYYY_MM_DD.stderrout.log

Fusion uses the Apache Log4j 2 logging framework with Jetty to log each of the Fusion components. Logging is configured via an xml configuration file named log4j2.xml. Log levels, frequencies, and log rotation policy can be configured by changing these configuration files:

API service

$FUSION/conf/api-log4j2.xml

connectors

$FUSION/conf/connectors-log4j2.xml

Solr

$FUSION/conf/solr-log4j2.xml

Spark

$FUSION/conf/spark-driver-log4j2.xml

$FUSION/conf/spark-master-agent-log4j2.xml

$FUSION/conf/spark-master-log4j2.xml

$FUSION/conf/spark-worker-agent-log4j2.xml

$FUSION/conf/spark-worker-log4j2.xml

UI

$FUSION/conf/ui-log4j2.xml

ZooKeeper

$FUSION/conf/zk-log4j2.xml

The Log4j2 Configuration guide provides documentation and examples of all logging configuration options.