Skip to main content
Compatible with Fusion version: 4.0.0 through 4.2.6
Deprecation and removal noticeThis connector is deprecated as of Fusion 4.2 and is removed or expected to be removed as of Fusion 5.0.For more information about deprecations and removals, including possible alternatives, see Deprecations and Removals.
There is also a non-MapReduce enabled connector for HDFS filesystem; see the HDFS Connector Configuration Reference for details.

​
How Hadoop connectors work

The Hadoop crawlers take full advantage of the scaling abilities of the MapReduce architecture and will use all of the nodes available in the cluster just like any other MapReduce job. This has significant ramifications for performance since it is designed to move a lot of content, in parallel, as fast as possible (depending on the system’s capabilities), from its raw state to the Fusion index. The Hadoop crawlers work in stages:
  1. Create one or more SequenceFiles from the raw content. This can be done in one of two ways:
    • If the source files are available in a shared Hadoop filesystem, prepare a list of source files and their locations as a SequenceFile. The raw contents of each file are not processed until step 2.
    • If the source files are not available, prepare a list of source files and the raw content, stored as a Behemoth document. This process is currently done sequentially and can take a significant amount of time if there is a large number of documents and/or if they are very large.
  2. Run a MapReduce job to extract text and metadata from the raw content using Apache Tika. This is similar to the Fusion approach of extracting content from crawled documents, except it is done with MapReduce.
  3. Run a MapReduce job to send the extracted content from HDFS to the index pipeline for further processing.
The first step of the crawl process converts the input content into a SequenceFile. In order to do this, the entire contents of that file must be read into memory so that it can be written out in the SequenceFile. Thus, you should be careful to ensure that the system does not load into memory a file that is larger than the Java heap size of the process. In certain cases, Behemoth can work with existing files such as SequenceFiles to convert them to Behemoth SequenceFiles. Contact Lucidworks for possible alternative approaches. The processing approach is currently “all or nothing” when it comes to ingesting the raw content and all steps must be completed each time, regardless of whether the raw content has changed. Future versions may allow the crawler to restart from the SequenceFile conversion process. In the meantime, incremental crawling is not supported for this connector.

​
Fusion login configuration file

The Fusion login config file required by the datasource configuration parameter login_config is a Java Authentication and Authorization Service (JAAS) configuration file which needs to be present on every mapper/reducer node which will inject data to Fusion. Here is a sample file that describes the structure expected by Fusion: 
FusionClient {
 com.sun.security.auth.module.Krb5LoginModule required
 useKeyTab=true
 useTicketCache=false
 storeKey=false
 keyTab="/home/keytabs/hadoop.keytab";
};
FusionClient is the application name and can be set to anything. Be sure to set the login_app_name parameter to the same value if you change it. The other parameters can be configured as required and the keyTab value should point to the location on the node where the keytab file can be found.

​
Learn more

The Apache Hadoop 2 Connector is a MapReduce-enabled crawler that is compatible with Apache Hadoop v2.x.The connector services must be able to access the Hadoop client in file $HADOOP_HOME/bin/hadoop, so it must either be installed on one of the nodes of the Hadoop cluster (such as the nameNode), or a client supported by your specific distribution must be installed on the same server as the connectors. The Hadoop client must be configured to access the Hadoop cluster so the crawler is able to access the Hadoop cluster for content processing.
Instructions for setting up any of the supported Hadoop distributions is beyond the scope of this document. We recommend reading one of the many tutorials found online or one of the books on Hadoop.
This connector writes to the hadoop.tmp.dir and the /tmp directory in HDFS, so Fusion should be started by a user who has read/write permissions for both.

​
Permission issues

Using any flavor of Hadoop, you will need to be aware of the way Hadoop and systems based on Hadoop (such as CDH, MapR, etc.) handle permissions for services that communicate with other nodes.Hadoop services execute under specific user credentials: a quadruplet consisting of user name, group name, numeric user id, numeric group id. Installations that follow the manual usually use user ‘mapr’ and group ‘mapr’ (or similar), with numeric ids assigned by the operating system (e.g., uid=1000, gid=20). When the system is configured to enforce user permissions (which is the default in some systems), any client that connects to Hadoop services has to use a quadruplet that exists on the server. This means that ALL values in this quadruplet must be equal between the client and the server, i.e., an account with the same user, group, uid, and gid must exist on both client and server machines.When a client attempts to access a resource on Hadoop filesystems (or the JobTracker, which also uses this authentication method) it sends its credentials, which are looked up on the server, and if an exactly matching record is found then those local permissions will be used to determine read/write access. If no such account is found then the user is treated as “other” in the sense of the permission model.This means that the crawlers for the HDFS data source should be able to crawl Hadoop or MapR filesystems without any authentication, as long as there is a read (and execute for directories) access for “other” users granted on the target resources. Authenticated users will be able to access resources owned by their equivalent account.However, the Hadoop crawling described on this page require write access to a /tmp directory to use as a working directory. In many cases, this directory does not exist, or if it does, it does not have write access to “other” (not authenticated) users. Therefore users of these data sources should make sure that there is a /tmp directory on the target filesystem that is writable using their local user credentials, be it a recognized user, group, or “other”. If a local user is recognized by the server then it is enough to create a /tmp directory that is owned by that user. If there is no such user, then the /tmp directory must be modified to have write permissions for “other” users. The working directory can be modified to be another directory that can be used for temporary working storage that has the correct permissions.

​
Configuration for a Kerberos Hadoop cluster

Kerberos is a system that provides authenticated access for users and services on a network. Instead of sending passwords in plaintext over the network, encrypted passwords are used to generate time-sensitive tickets which are used for authentication. Kerberos uses symmetric-key cryptography and a trusted third party called a Key Distribution Center (KDC) to authenticate users to a suite of network services. When a user authenticates to the KDC, the KDC sends a set of credentials (a ticket) specific to that session back to the user’s machine.To work with a Kerberized Hadoop cluster you must have a set of credentials. These are generated by running the “kinit” program. The datasource can be configured to run this program, in which case, the following information must be specified: the full path to the program, the Kerberos principal name, the location of a keytab file and the name of the file in which to store the ticket.

​
Configuration

When entering configuration values in the UI, use unescaped characters, such as \t for the tab character. When entering configuration values in the API, use escaped characters, such as \\t for the tab character.