Skip to main content
The Index Pipelines API provides methods for managing a set of named index pipelines. Every pipeline is made up of one or more stages. Stages can be defined during the creation of a pipeline, or stages can be defined separately and included into one or more pipelines. For details of the REST API for index stages, see Index Stages API. Document processing proceeds stage by stage in a linear fashion. The order of the stages in a pipeline is the order in which they were defined. At installation, Managed Fusion includes several pre-configured pipelines. See Index Pipelines for details on these default pipelines. For more information about structuring documents for indexing, see Import Data with the REST API.
It is often possible to get documents into Fusion Server by configuring a datasource with the appropriate connector.But if there are obstacles to using connectors, it can be simpler to index documents with a REST API call to an index profile or pipeline.

​
Push documents to Fusion using index profiles

Index profiles allow you to send documents to a consistent endpoint (the profile alias) and change the backend index pipeline as needed. The profile is also a simple way to use one pipeline for multiple collections without any one collection “owning” the pipeline.

​
Send data to an index profile that is part of an app

Accessing an index profile through an app lets a Fusion admin secure and manage all objects on a per-app basis. Security is then determined by whether a user can access an app. This is the recommended way to manage permissions in Fusion.The syntax for sending documents to an index profile that is part of an app is as follows:
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/json' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE --data-binary @my-json-data.json
Spaces in an app name become underscores. Spaces in an index profile name become hyphens.
To prevent the terminal from displaying all the data and metadata it indexes—useful if you are indexing a large file—you can optionally append ?echo=false to the URL.Be sure to set the content type header properly for the content being sent. Some frequently used content types are:
  • Text: application/json, application/xml
  • PDF documents: application/pdf
  • MS Office:
    • DOCX: application/vnd.openxmlformats-officedocument.wordprocessingml.document
    • XLSX: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
    • PPTX: application/vnd.vnd.openxmlformats-officedocument.presentationml.presentation
    • More types: http://filext.com/faq/office_mime_types.php

​
Example: Send JSON data to an index profile under an app

In $FUSION_HOME/apps/solr-dist/example/exampledocs you can find a few sample documents. This example uses one of these, books.json.To push JSON data to an index profile under an app:
  1. Create an index profile. In the Fusion UI, click Indexing > Index Profiles and follow the prompts.
  2. From the directory containing books.json, enter the following, substituting your values for username, password, and index profile name: 
    curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/json' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE?echo=false --data-binary @books.json
  3. Test that your data has made it into Fusion:
    1. Log into the Fusion UI.
    2. Navigate to the app where you sent your data.
    3. Navigate to the Query Workbench.
    4. Search for \*:*.
    5. Select relevant Display Fields, for example author and name.

​
Example: Send JSON data without defining an app

In most cases it is best to delegate permissions on a per-app basis. But if your use case requires it, you can push data to Fusion without defining an app.To send JSON data without app security, issue the following curl command:
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/json' https://FUSION_HOST:FUSION_PORT/api/index/INDEX_PROFILE --data-binary @my-json-data.json

​
Example: Send XML data to an index profile with an app

To send XML data to an app, use the following:
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/xml' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE --data-binary @my-xml-file.xml
In Fusion 5, documents can be created on the fly using the PipelineDocument JSON notation.

​
Remove documents

​
Example 1

The following example removes content:
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/vnd.lucidworks-document' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE --data-binary @del-json-data.json

​
Example 2

A more specific example removes data from books.json. To delete “The Lightning Thief” and “The Sea of Monsters” from the index, use their id values in the JSON file.The del-json-data.json file to delete the two books:
[{ "id": "978-0641723445","commands": [{"name": "delete","params": {}}]},{ "id": "978-1423103349","commands": [{"name": "delete","params": {}}, {"name": "commit","params": {}}]}]
The ?echo=false can be used to turn off the response to the terminal.

​
Example 3

Another example to delete items using the Push API is:
curl -u admin:XXX -X POST  'http://FUSION_HOST:FUSION_PORT/api/apps/APP/index/INDEX' -H 'Content-Type: application/vnd.lucidworks-document' -d '[
  {
    "id": "1663838589-44",
    "commands":
    [
      {
        "name": "delete",
        "params":
        {}
      },
      {
        "name": "commit",
        "params":
        {}
      }
    ]
  }, ...
]'

​
Send documents to an index pipeline

Although sending documents to an index profile is recommended, if your use case requires it, you can send documents directly to an index pipeline.For more information about index pipeline REST API reference documentation, select the link for your Fusion release:

​
Specify a parser

When you push data to a pipeline, you can specify the name of the parser by adding a parserId querystring parameter to the URL. For example: https://FUSION_HOST:FUSION_PORT/api/index-pipelines/INDEX_PIPELINE/collections/COLLECTION_NAME/index?parserId=PARSER.If you do not specify a parser, and you are indexing outside of an app (https://FUSION_HOST:FUSION_PORT/api/index-pipelines/...), then the _system parser is used.If you do not specify a parser, and you are indexing in an app context (https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index-pipelines/...), then the parser with the same name as the app is used.

​
Indexing CSV Files

In the usual case, to index a CSV or TSV file, the file is split into records, one per row, and each row is indexed as a separate document.
For more information, view the API specification.

​
Examples

The full set of fields for each stage type are not included in the following examples. See Index Pipeline Stages Reference for more information about the fields available for each stage.
List the ‘default’ pipeline: REQUEST 
curl -u USERNAME:PASSWORD https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/index-pipelines/default
RESPONSE 
{
  "id" : "default",
  "stages" : [ {
    "type" : "solr-index",
    "id" : "solr-default",
    "skip" : false
  } ]
}
Create the pipeline in the “MyApp” app REQUEST 
curl -u USERNAME:PASSWORD -X POST -H 'Content-type: application/json' -d '{
    "id": "my-index-pipeline",
    "stages": [
        {
            "id": "field_mapping",
            "type": "field-mapping",
            "reservedFieldsMappingAllowed": false
        },
        {
            "id": "solr_dynamic_field_mapping",
            "type": "solr-dynamic-field-name-mapping",
            "advancedTextFieldsIndexing": true
        },
        {
            "id": "solr_indexing",
            "type": "solr-index",
            "fieldsList": false
        }
    ]
}' https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/apps/MyApp/index-pipelines
RESPONSE 
{
    "id": "my-index-pipeline",
    "stages": [
        {
            "id": "field_mapping",
            "translationMappings": [],
            "retentionMappings": [],
            "updateMappings": [],
            "unmappedRule": {
                "keep": true,
                "delete": false,
                "fieldToMoveValuesTo": "",
                "fieldToCopyValuesTo": "",
                "valueToAddToUnmappedFields": "",
                "valueToSetOnUnmappedFields": ""
            },
            "reservedFieldsMappingAllowed": false,
            "type": "field-mapping",
            "skip": false,
            "secretSourceStageId": "field_mapping"
        },
        {
            "id": "solr_dynamic_field_mapping",
            "duplicateSingleValuedFields": false,
            "fieldsToExclude": [],
            "advancedTextFieldsIndexing": true,
            "maxTextLengthForAdvancedIndexing": 100,
            "type": "solr-dynamic-field-name-mapping",
            "skip": false,
            "secretSourceStageId": "solr_dynamic_field_mapping"
        },
        {
            "id": "solr_indexing",
            "enforceSchema": true,
            "fieldsList": false,
            "dateFormats": [],
            "params": [],
            "bufferDocsForSolr": true,
            "allowBlacklistedRequestParameters": false,
            "type": "solr-index",
            "skip": false,
            "secretSourceStageId": "solr_indexing"
        }
    ],
    "properties": {
        "secretSourcePipelineId": "my-index-pipeline"
    }
}
⌘I