With Managed Fusion, your data is indexed in a set of collections in Managed Fusion’s Solr core. A primary collection holds your searchable content, such as your product catalog, knowledge base, blog articles, product reviews, and so on. A set of secondary collections are associated with your primary collection to hold related data that Managed Fusion can use to enhance the relevancy of your search results. This topic shows you how different types of data flow through Managed Fusion to be indexed in your primary and secondary collections.

1. Index your content

No matter where your content is located, Managed Fusion can index it. The Managed Fusion collection where your searchable content is indexed is called the primary collection. Secondary collections are automatically created to hold related data, such as signals, machine learning job output, and so on. There are a few ways to index your searchable content to the primary collection:
  • Lucidworks Connectors
    Lucidworks has a wide variety of connectors for many types of data sources.
    Once the connector fetches your data, parsers read it before passing it to the index pipeline.
    • The gRpc remote framework is configured on the client side and only works with V2 type connectors.
    • V1 classic connectors support ingesting data using IP white lists, VPN tunnels, and public channels.
  • The Parallel Bulk Loader (PBL)
    The PBL can send your data to an index pipeline or directly to the primary collection, depending on whether the data requires transformation before indexing. It does not support parsers and is not recommended for production environments.
  • Import Data with the REST API
    Send your content to an index profile using the Managed Fusion REST API. Index profiles are saved configurations of parsers and index pipelines.
It is often possible to get documents into Managed Fusion by configuring a datasource with the appropriate connector: Connectors Configuration Reference.But if there are obstacles to using connectors, it can be simpler to index documents with a REST API call to an index profile or pipeline.

Push documents to Managed Fusion using index profiles

Index profiles allow you to send documents to a consistent endpoint (the profile alias) and change the backend index pipeline as needed. The profile is also a simple way to use one pipeline for multiple collections without any one collection “owning” the pipeline. See Managed Fusion Index Profiles.You can send documents directly to an index using the Index REST API. The request path is:
/api/apps/APP_NAME/index/INDEX_PROFILE
These requests are sent as a POST request. The request header specifies the format of the contents of the request body. Create an index profile in the Managed Fusion UI.To send a streaming list of JSON documents, you can send the JSON file that holds these objects to the API listed above with application/json as the content type. If your JSON file is a list or array of many items, the endpoint operates in a streaming way and indexes the docs as necessary.

Send data to an index profile that is part of an app

Accessing an index profile through an app lets a Managed Fusion admin secure and manage all objects on a per-app basis. Security is then determined by whether a user can access an app. This is the recommended way to manage permissions in Managed Fusion.The syntax for sending documents to an index profile that is part of an app is as follows:
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/json' https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/apps/APP_NAME/index/INDEX_PROFILE --data-binary @my-json-data.json
Spaces in an app name become underscores. Spaces in an index profile name become hyphens.
To prevent the terminal from displaying all the data and metadata it indexes—useful if you are indexing a large file—you can optionally append ?echo=false to the URL.Be sure to set the content type header properly for the content being sent. Some frequently used content types are:
  • Text: application/json, application/xml
  • PDF documents: application/pdf
  • MS Office:
    • DOCX: application/vnd.openxmlformats-officedocument.wordprocessingml.document
    • XLSX: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
    • PPTX: application/vnd.vnd.openxmlformats-officedocument.presentationml.presentation
    • More types: http://filext.com/faq/office_mime_types.php

Example: Send JSON data to an index profile under an app

In $FUSION_HOME/apps/solr-dist/example/exampledocs you can find a few sample documents. This example uses one of these, books.json.To push JSON data to an index profile under an app:
  1. Create an index profile. In the Managed Fusion UI, click Indexing > Index Profiles and follow the prompts.
  2. From the directory containing books.json, enter the following, substituting your values for username, password, and index profile name:
    curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/json' https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/apps/APP_NAME/index/INDEX_PROFILE?echo=false --data-binary @books.json
    
  3. Test that your data has made it into Managed Fusion:
    1. Log into the Managed Fusion UI.
    2. Navigate to the app where you sent your data.
    3. Navigate to the Query Workbench.
    4. Search for \*:*.
    5. Select relevant Display Fields, for example author and name.

Example: Send JSON data without defining an app

In most cases it is best to delegate permissions on a per-app basis. But if your use case requires it, you can push data to Managed Fusion without defining an app.To send JSON data without app security, issue the following curl command:
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/json' https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/index/INDEX_PROFILE --data-binary @my-json-data.json

Example: Send XML data to an index profile with an app

To send XML data to an app, use the following:
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/xml' https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/apps/APP_NAME/index/INDEX_PROFILE --data-binary @my-xml-file.xml
You can also create documents using the PipelineDocument JSON notation.

Send documents to an index pipeline

Although sending documents to an index profile is recommended, if your use case requires it, you can send documents directly to an index pipeline.

Specify a parser

When you push data to a pipeline, you can specify the name of the parser by adding a parserId querystring parameter to the URL. For example: https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/index-pipelines/INDEX_PIPELINE/collections/COLLECTION_NAME/index?parserId=PARSER.If you do not specify a parser, and you are indexing outside of an app (https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/index-pipelines/...), then the _system parser is used.If you do not specify a parser, and you are indexing in an app context (https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/apps/APP_NAME/index-pipelines/...), then the parser with the same name as the app is used.

Indexing CSV Files

In the usual case, to index a CSV or TSV file, the file is split into records, one per row, and each row is indexed as a separate document.
Index your content

Indexing your content

The index pipeline consists of one or more configurable index pipeline stages, each performing a different type of transformation on the incoming data. Each connector has a default index pipeline, but you can modify these or create new ones. The last stage in any index pipeline should be the Solr Indexer stage, which submits the documents to Solr for indexing.

2. Index your signals

Signals are event records that provide historical data about user behavior, such as clicks, likes, purchases, and so on. You don’t need to index signals about query responses; Managed Fusion indexes response signals automatically by default. If you are using App Studio or App Insights, then you need to index request signals. Learn more about signal types and required fields. To index your signals, you send them to Managed Fusion using the Signals API, which points to the hidden index pipeline designed especially for signals.
Index your signals

Indexing your signals

Raw signals are indexed in a secondary collection called COLLECTION_NAME_signals. For example, if your primary collection is called Products, then the raw signals collection is Products_signals. When you Enable or disable signals, Managed Fusion creates jobs and secondary collections for analyzing and aggregating your raw signals. Some of this data enables query rewriting and automatic boosting, while other data becomes useful when you enable recommendations.
You can enable and disable signals using the Managed Fusion UI or the REST API.
When you disable signals, the aggregation jobs are deleted, but the _signals and _signals_aggr collections are not, your legacy signal data remains intact.

Using the UI

When you create a collection using the Fusion UI, signals are enabled and a signals collection created by default. You can also enable and disable signals for existing collections using the Collections Manager.Enable signals for a collection
  1. In the Fusion workspace, navigate to Collections > Collections Manager.
  2. Hover over the primary collection for which you want to enable signals.
  3. Click Configure to open the drop-down menu. Enable Signals
  4. Click Enable Signals.
    The Enable Signals window appears, with a list of collections and jobs that are created when you enable signals.
    Enable Signals
  5. Click Enable Signals.
Disable signals for a collection
  1. In the Fusion workspace, navigate to Collections > Collections Manager.
  2. Hover over the primary collection for which you want to disable signals.
  3. Click Configure to open the drop-down menu.
  4. Click Disable Signals.
    The Disable Signals window appears, with a list of jobs that are created when you enable signals.
  5. Click Disable Signals.
    Your _signals and _signals_aggr collections remain intact so that you can access your legacy signals data.

Using the Collection Features API

Using the API, the /collections/{collection}/features/{feature} endpoint enables or disables signals for any collection:Check whether signals are enabled for a collection
curl -u USERNAME:PASSWORD https://EXAMPLE_COMPANY.b.lucidworks.cloud:{api-port}/api/collections/COLLECTION_NAME/features/signals
Enable signals for a collection
curl -u USERNAME:PASSWORD -X PUT -H "Content-type: application/json" -d '{"enabled" : true}' https://https://EXAMPLE_COMPANY.b.lucidworks.cloud:{api-port}/api/collections/COLLECTION_NAME/features/signals
Disable signals for a collection
curl -u USERNAME:PASSWORD -X PUT -H "Content-type: application/json" -d '{"enabled" : false}' https://https://EXAMPLE_COMPANY.b.lucidworks.cloud:{api-port}/api/collections/COLLECTION_NAME/features/signals
Enable signals jobs

Managed Fusion jobs that index signals-related data

4. Managed Fusion jobs that index recommendations

When you enable recommendations, another set of jobs and secondary collections is created.
Enable recommendations jobs

Managed Fusion jobs that index recommendations

The default recommendation jobs read data from different collections depending on the type of recommendations being generated:
For more recommendations, try Identify trending items or queries.

What’s next?

See Query data to learn about querying your content and recommendations.
LucidAcademyLucidworks offers free training to help you get started.The Quick Learning for Indexing Menu Tour focuses on the Indexing Menu features and functionality along with a brief description of each screen available in the menu:
Indexing Menu TourPlay Button
Visit the LucidAcademy to see the full training catalog.
LucidAcademyLucidworks offers free training to help you get started.The Quick Learning for Anatomy of Indexing focuses on the Fusion objects that work together to index your data:
Anatomy of IndexingPlay Button
Visit the LucidAcademy to see the full training catalog.
LucidAcademyLucidworks offers free training to help you get started.The Course for Ingesting Data focuses on understanding connectors and how to configure datasources, parsers, and index pipelines:
Ingesting DataPlay Button
Visit the LucidAcademy to see the full training catalog.