# Introduction
Source: https://doc.lucidworks.com/docs/lw-platform/index
Discover how to deploy, customize, and scale Lucidworks products for AI-powered search and insights across B2B, B2C, knowledge management, customer support, and more.
Reach out with your current search challenges and what you hope to achieve with a search and discovery solution.
## Discover
Get started quickly with product-specific guides and tools for building intelligent search, analytics, and AI applications.
Advanced AI-powered data discovery and search
Combine lexical precision with semantic understanding for more relevant results
Build fast, intuitive, and configurable search experiences
Prototype and deploy enterprise-grade AI agents
Optimize search results and improve product discovery
Transform data into actionable intelligence
Low-code signal capture for advanced analytics
# Latest changes
Source: https://doc.lucidworks.com/docs/lw-platform/latest-changes
Lucidworks products are constantly evolving to deliver new features, improvements, and fixes. This page highlights the latest changes to help you stay current with product updates and enhancements.
[localhost link]: http://localhost:3000/docs/lw-platform/latest-changes
[mintlify link]: https://doc.lucidworks.com/docs/lw-platform/latest-changes
# Search Results Layout Settings
The Commerce Studio Settings feature now provides the option to display search results in a list, or in the current default grid pattern.
The **Layout** section lets you select to display search results in a vertical list or a grid pattern.
For example, if you select the **List** option, the search results display in a vertical list on the Editor screen.
For more information, see [Commerce Studio Settings Layout](/docs/lw-platform/lw-cs/cstudio-settings#layout).
# Configure the Vector Similarity Threshold and Vector Result Count of Neural Hybrid Search
The Lucidworks Platform Core Settings Relevance option has been enhanced with two new fields:
* **Vector Similarity Threshold** lets you set the minimum of how closely a document must match your query's meaning to be included in results. Enter a decimal value between `0` and `1`. A higher minimum value returns only results that strongly match the query's meaning. A lower minimum value returns more results because results do not have to match the query's meaning as closely.
* **Vector Result Count** specifies how many candidate results the vector (semantic) search retrieves before the results are merged and re-ranked with the lexical results. Enter an integer between `0` and `2000`. A higher value assesses more semantic candidate results, which improves relevant results. A lower value assesses fewer semantic candidate results, which may decrease relevant results.
For more information, see [Relevance](/docs/lw-platform/lw-core/sites-relevance) and [NHS Quickstart](/docs/lw-platform/lw-core/quickstart).
# Exclude rules from Commerce Studio Editor results preview
The Execute section of the Commerce Studio Editor screen now gives you the option to include or exclude a rule from the query being previewed. This lets you test to see the search results if the rule is removed from a query without actually deleting the rule.
Click the **Preview** toggle `on` to execute that rule in the query, or click **Preview** `off` to exclude it from the query. The query results display based on the rules included in the query.
Your live site is *never* affected by setting the **Preview** toggle to `off`.
# Customize the UI Studio composition canvas
The Customize Section option of UI Studio lets you set a background color and size for a composition that matches your organization's branding or for a specific search experience.
When you access UI Studio, click **Builder**, and select the composition to customize, the Composition Editor screen displays.
Click the main canvas and then click the gear icon to open the **Customize Section** panel. Set a background color for the composition. You can also or adjust the section dimensions. These settings can be refined at any time.
# Add a UI Studio composition
The Create New UI feature of UI Studio lets you [add a new composition](/docs/lw-platform/lw-ui-studio/compositions#add-a-composition) to build a new search experience page for your users. This UI option is a user-friendly method to add a search page customized to your organization and doesn't require engineering resources.
When you access UI Studio and click **Builder**, the **Create New UI** option lets you name the new composition.
Add the name and description and click **Create**.
The new composition is listed on the Compositions screen that lets you [add and customize components](/docs/lw-platform/lw-ui-studio/compositions#build-a-composition) such as a search bar, facets, and a results section.
# Unified Platform Login
If you're using Lucidworks Search, you can now access Fusion instances and Grafana dashboards directly from Lucidworks Platform using single sign-on (SSO) authentication.
Unified Platform Login eliminates the need to log in separately to each system and provides a seamless, secure authentication experience powered by Okta.
When you click the **Managed** button from Lucidworks Platform's Indexes tab, you're authenticated automatically and redirected to your Fusion instance without entering additional credentials.
The same SSO experience extends to Grafana dashboards accessed with the **View Metrics** option.
All authentication infrastructure is managed by Lucidworks—no manual security realm, OIDC, or group mapping configuration required.
Unified Platform Login works with Lucidworks Search.
It does not apply to self-hosted Fusion.
If you need elevated access to perform administrative tasks in an instance of Lucidworks Search, request role elevation from Lucidworks Support or your Customer Success Manager.
After the role is granted, log in again with Lucidworks Platform to activate your new permissions.
See [Fusion instances](/docs/lw-platform/lw-platform/indexes#fusion-instances) for complete details.
# Draft edits for all rule types in Commerce Studio
Commerce Studio now supports draft edits for every rule type, so merchandisers can update a published rule without taking it offline. When you open a published rule for editing, the published version stays active on your site and your changes are captured as a draft. You can then **Publish** the changes immediately, **Save as Draft** to preview the impact in the Editor, or **Cancel** to discard the edits.
The live rule keeps serving traffic while edits are staged, so you can refine merchandising logic before it reaches users.
On the Rules screen, a published rule with a draft appears as two rows so it is always clear which version is live and which has unpublished changes. Bulk actions, filters, and row-level actions all reflect draft state.
For more information, see [Manage rules](/docs/lw-platform/lw-cs/cstudio-rules#manage-rules).
# Part Number detection
The Part number detection feature is a specialized query processing feature that automatically detects part number patterns in search queries, and then optimizes search behavior to balance lexical precision with semantic understanding.
In part number searches, lexical matching is the more dominant function and highly accurate, lexically focused results are returned. Semantic matching is minimized for part number searches, but is not impacted for general queries.
Part number detection employs a Fusion LWAI Prediction stage you create. You must [configure the stage to use the Part Number Classification custom-trained model](/docs/lw-platform/lw-ai/lw-ai-custom-embedding-model-training/create-and-deploy-custom-models#create-a-new-model).
To further optimize part number searches, you can complete the following actions:
* Configure any other Fusion stage with conditional statements that return the boolean from part number detection (if applicable to your business).
* Tune the Neural Hybrid Query stage to give more weight to lexical search.
## Use cases
* Business-to-consumer organizations that offer products identified with model or part numbers can use part number detection to accurately return results to help users select the correct item, which increases customer satisfaction and reduces product returns.
* Business-to-business organizations that rely heavily on part numbers, SKUs, or product codes, use part number detection to identify the products they distribute. The correct selection of products is essential to their customers, and enable targeted purchases, encourage repeat orders, and build the long-term customer relationships that help B2B organizations thrive.
* Knowledge management searches use part number detection to return results from your company's knowledge bases, technical documents, inventory catalogs, and other documents. Easy and fast access to this information increases productivity and access to accurate, up-to-date information.
## Examples
When the stages are configured, the part number detection feature can be used in Fusion or in Lucidworks AI.
The following example reflects when a part number is detected (`true`):
The following example reflects when a part number is not detected (`false`):
For more information, see the following topics:
* [Fusion Part Number Detection in Neural Hybrid Search](/docs/5/fusion/hybrid-search/part-number-detection-in-nhs)
* [Lucidworks Search Part Number Detection in Neural Hybrid Search](/docs/lucidworks-search/11-vector-search/part-number-detection-in-nhs)
* [Lucidworks AI Prediction API Part Number Detection use case](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/part-number-detection-use-case)
* [Lucidworks AI Prediction API Part Number Detection API specification](/api-reference/get-predictions/part-number-detection-use-case)
# Introducing UI Studio
Lucidworks UI Studio is a no-code visual experience builder for creating and launching enterprise-grade, AI-powered search applications on the Lucidworks Platform. Teams can design branded, multi-page search experiences using drag-and-drop components, live previews, and configurable search integrations without relying on engineering resources.
Unlike traditional search UI frameworks and developer SDKs that require frontend development expertise, UI Studio empowers search administrators, digital teams, and business users to visually create and manage search experiences independently. Organizations can rapidly launch search interfaces that are accurate, scalable, branded, and production-ready.
Accept user queries and trigger search across your datastore.
Display search results in List, Grid, or Table views with sorting and pagination.
Let users narrow results by filtering on field values.
Add navigation, export, or action buttons to your pages.
Display images with optional links, borders, and fallback support.
Add static or dynamic text content with rich styling options.
UI Studio supports both SaaS and self-hosted Fusion deployments, and integrates natively with Lucidworks Search, AI-powered relevance, typeahead, version control, and secure publishing.
## Key capabilities
* Drag-and-drop creation of branded, multi-page search applications
* Built-in integration with Lucidworks Search, AI-powered relevance, and typeahead
* Live preview with real production data during configuration
* Visual theming and branding controls for consistent enterprise design
* Secure publishing to Lucidworks-managed hosting in minutes
* Version control and rollback for safe iteration
* Enterprise authentication support including SSO-secured experiences
## Use cases
B2B buyers work with large, complex product catalogs and require precise access to technical specifications, parts, and contract terms. UI Studio enables B2B organizations to rapidly deploy branded search experiences without dedicating frontend engineering resources. Search administrators can configure facets, sorting, and relevance tuning to match complex procurement workflows and publish updates independently. This reduces implementation friction, shortens time-to-value, and allows engineering teams to focus on higher-priority initiatives.
B2C customers expect fast, intuitive, and visually consistent search experiences. UI Studio allows ecommerce and digital teams to create branded product discovery experiences with configurable search bars, facets, typeahead, and results pages. Teams can preview changes against live data, iterate quickly without code, and publish confidently with built-in version control. This improves product findability, reduces cart abandonment, and accelerates time-to-launch for new storefronts and campaigns.
Enterprise knowledge management programs often span multiple portals: employee search, technical documentation, supplier resources, and customer support. UI Studio enables organizations to launch and maintain each of these experiences independently, without parallel frontend development projects. Search administrators can manage layouts, components, and pages through a visual interface while automatically capturing behavioral signals to improve relevance over time.
## Dive in
Explore the UI Studio documentation to learn how to build, customize, and publish enterprise-grade search experiences without writing code.
See UI Studio in action and learn how your team can launch enterprise search experiences faster without frontend development.
# Bulk operations API for Commerce Studio
The Rules and Rewrites API for Commerce Studio now supports bulk operations, so you can publish, unpublish, edit, delete, or migrate multiple rules in a single request instead of calling the API once per rule.
Bulk operations run asynchronously.
You submit a request that targets rules by ID, external ID, or a filter, then poll the operation status until it completes.
These capabilities support enterprise use cases where per-rule API calls do not scale, such as:
* Publishing or unpublishing large sets of rules for a seasonal campaign
* Applying the same edit across every rule that shares a tag
* Deleting obsolete rules in a single batch
* Migrating a rule set to another Commerce Studio instance
Bulk operations require an authentication token with the `commercestudio.bulkoperations` scope.
See [Bulk operations](/docs/lw-platform/lw-cs/cs-rules-rewrites-api#bulk-operations) for usage instructions and example requests.
For complete endpoint specifications, see the [Bulk Operations API reference](/api-reference/bulk-operations/list-bulk-operations).
# Product Impressions Metrics
Product impressions measure the number of times a product displays in search results in response to user queries. The product is counted each time it displays, even if users do not interact further with the product.
You can use impressions metrics to help your organization perform these important functions:
* Track impression changes after merchandising updates or catalog modifications to monitor trends over time
* Analyze impressions for different search categories or product types and segment by query type
* Identify high-impression products that may need pricing or promotion adjustments
* Include impression analysis in regular merchandising reviews to maintain optimal product exposure
You can configure impressions metrics in [Analytics Studio metrics](/docs/lw-platform/lw-analytics/analytics-screen#param-top-performing-products) for top performing products.
You can also configure [Commerce Studio inline analytics](/docs/lw-platform/lw-cs/cstudio-settings#inline-analytics) to display impressions on product cards.
For more information about how to use impressions metrics to optimize search experiences and product visibility, see [Impressions metrics](/docs/lw-platform/lw-cs/cstudio-editor#impressions-metrics).
# AND logic for attribute rules in Commerce Studio
Commerce Studio now supports **AND logic** across attribute field value conditions in [Boost by Attribute](/docs/lw-platform/lw-cs/cstudio-rules#boost), [Bury by Attribute](/docs/lw-platform/lw-cs/cstudio-rules#bury), and [Block by Attribute](/docs/lw-platform/lw-cs/cstudio-rules#block) rules, enabling precise targeting of specific product combinations.
Previously, attribute rules could only use OR logic between different attributes. For example, `Color = Black OR Type = Pants` would affect all black items and all pants, even if the items didn't meet both conditions.
Now you can create rules that require multiple attributes to match simultaneously in one rule:
* `(Color: Black, Red) AND (Type: Shirt)` — Boost only shirts that are black or red
* `(Brand: Nike) AND (Gender: Women) AND (Category: Running)` — Boost women's Nike running shoes
* `(Clearance: True) AND (Available: 0)` — Bury out-of-stock clearance items
This enhancement enables merchandising scenarios such as:
* Boosting seasonal product combinations like winter coats or summer dresses across multiple brands or colors
* Burying clearance items that are out of stock to reduce customer frustration
* Promoting new arrivals within specific high-margin categories
* Burying products that do not meet a product speficiation in the query
Existing attribute rules continue to use OR logic between attributes, ensuring backward compatibility for older rules.
For more information, see [Commerce Studio Rules](/docs/lw-platform/lw-cs/cstudio-rules).
This feature requires Fusion or Lucidworks Search 5.17.1 or later.
# Numeric field operators for Commerce Studio rules
Commerce Studio now supports numeric comparison operators for field value conditions in [Boost by Attribute](/docs/lw-platform/lw-cs/cstudio-rules#boost), [Bury by Attribute](/docs/lw-platform/lw-cs/cstudio-rules#bury), and [Block by Attribute](/docs/lw-platform/lw-cs/cstudio-rules#block) rules.
The Commerce Studio UI now provides operators for creating attribute-based rules with numerical field comparisons.
Previously, configuring numeric conditions required writing Solr queries directly.
This enhancement enables merchandising scenarios such as:
* Boosting products with inventory levels that exceed a threshold
* Burying items with low customer ratings
* Blocking products priced outside acceptable ranges
* Promoting items within specific price brackets during sales
See [Field value operators](/docs/lw-platform/lw-cs/cstudio-rules#field-value-operators) for complete details about configuring numeric comparison conditions.
This feature requires Fusion or Lucidworks Search 5.17.1 or later.
# Platform Signals integration
The Boost with Signals query pipeline stage in Fusion and Lucidworks Search now supports Platform Signals as a data source, enabling you to use the Signals Beacon or Platform Signals API to feed behavioral data directly into search relevance boosting.
Previously, only native signals were supported, requiring more cumbersome parallel signal infrastructure to influence search ranking.
This integration eliminates that duplication.
Customers already capturing user behavior through Platform Signals can now use that same data to drive query boosting, turning behavioral analytics into active search optimization without additional infrastructure.
**Fusion**: See [Platform Signals](/docs/5/fusion/getting-data-in/platform-signals/overview) for configuration details.
**Lucidworks Search**: See [Platform Signals](/docs/lucidworks-search/04-move-data-in/platform-signals/overview).
# Form Builder for rules
Commerce Studio now includes a Form Builder for creating rules directly on the Rules screen, without opening the Editor. When you know exactly what rule you want to create and do not need to see search results in context, the Form Builder is faster than the Visual Builder workflow.
Use the Form Builder when you want to define conditions and actions without a live preview.
See [Form Builder](/docs/lw-platform/lw-cs/cstudio-rules#manual-builder) in the Rules documentation for configuration details.
# Conversational Q\&A and FAQ agents
Two new Agent Studio agents are now available. Both the Conversational Q\&A agent and the Conversational Q\&A FAQ agent are AI-powered and live on your organization's product detail pages. Customers can ask natural-language questions about a product, and the agent responds with accurate, real-time answers grounded in that product's own documentation, such as datasheets, manuals, specifications, and other supporting PDF technical content.
These agents save your customers time-consuming and difficult research to obtain the information they need to help make a purchasing decision. They also enhance customer satisfaction, increase the potential for conversions, and improve the relationship between your customers and your organization.
In addition, the Conversational Q\&A FAQ agent provides frequently asked questions based on questions users may ask about the product or information. This gives the user more information about their query, which helps minimize customer support calls and product returns.
You can quickly and easily create multiple agents that enhance your ability to provide information 24/7 to customers and internal purposes. For example, you may have one agent for customers about product information, and another for employees to get information about organization policies.
# Core Settings for multiple query profiles
[Core Settings](/docs/lw-platform/lw-core/overview) now supports managing multiple query profiles within a single site, enabling you to configure and maintain distinct search experiences for different use cases.
When your site has multiple query profiles, a query profile selector appears at the top of the Core Settings interface.
You can switch between profiles to configure each search experience independently without affecting other profiles.
This capability supports enterprise deployments that require any of the following:
* Separate search configurations for different websites or applications
* Different ranking strategies for various content types
* Dedicated search behaviors for internal versus public users
* Distinct search experiences for multiple audiences or regions
The query profile selector only appears when your site has multiple query profiles configured.
Sites with a single query profile continue to work exactly as before with no interface changes.
See [Query profiles](/docs/lw-platform/lw-core/overview#query-profiles) for complete details about managing multiple search configurations.
# Rules and Rewrites API
Commerce Studio now provides a public-facing Rules and Rewrites API that enables programmatic management of merchandising rules and query rewrites.
This API supports enterprise automation workflows, enabling you to create, update, and synchronize rules across multiple environments through CI/CD pipelines, internal tools, and custom integrations.
The API introduces external ID support, allowing you to assign your own identifiers to rules and rewrites for consistent cross-environment synchronization.
For example, you can create a rule with an external ID like `summer-promo-2026` in development, then deploy the same rule to QA and production using that identifier to ensure changes propagate correctly across all environments.
These capabilities address enterprise use cases where manual UI-based rule management does not scale:
* **Automated deployments**: Integrate rule management into CI/CD workflows for consistent, repeatable deployments.
* **Cross-environment synchronization**: Maintain identical rule sets across development, QA, and production.
* **Internal system integration**: Connect Commerce Studio to merchandising platforms, promotion systems, or custom tooling.
* **Large-scale rule management**: Efficiently manage rule sets that are impractical to maintain manually.
Rate limiting protects platform stability, and the API follows RESTful conventions with support for create, read, update, and delete operations.
See [Rules and Rewrites API](/docs/lw-platform/lw-cs/cs-rules-rewrites-api) for authentication details, key concepts, and usage instructions.
For complete endpoint specifications, see the [Rules and Rewrites API reference](/api-reference/rule-management/list-rules).
# Average Click Position
Commerce Studio and Analytics Studio now display the average ranking position of products in the search results when the customer clicks those products.
A lower average click position is better, because that means customers are finding the product near the top of the results, which provides the following benefits:
* Increased customer satisfaction with the search experience
* Enhanced conversion rates
* Minimized cart abandonment and loss of sales to competitors
If the product position needs to display higher, your ecommerce team can modify or create rules to increase that product's visibility.
This metric helps ecommerce teams validate merchandising rules, optimize product visibility, and measure the impact of search configuration changes on customer engagement and conversion rates.
## Average click position graph in Commerce Studio
The Ranking Insights panel in Commerce Studio displays average click position of the query specified in the Conditions Manager section of the Editor screen.
The metric is calculated by dividing the sum of all clicked positions by the total number of clicks for the current query and time period selected in the Editor screen.
## Average click position graph in Analytics Studio
The Analytics Studio Signals Analytics screen displays an average click position graph that measures the average ranking position for all products for the designated time period.
The metric is calculated by dividing the sum of all clicked positions by the total number of clicks for the time period displayed.
See [Types of metrics](/docs/lw-platform/lw-analytics/analytics-screen#types-of-metrics) for more information.
# Starts With and Ends With query matching
Commerce Studio now supports **Starts With** and **Ends With** query matching methods, giving you more precise control over when rules trigger.
These new matching options complement the existing **Is**, **Contains**, and **Contains Phrase** methods by enabling prefix- and suffix-based targeting strategies.
These enhancements enable the following capabilities:
* Target branded searches when queries begin with specific brand names.
* Apply location-specific logic when queries end with "near me".
* Trigger help content when queries end with "manual," "pdf," or "replacement".
* Reduce rule sprawl by creating precise, structured query patterns instead of multiple broader rules.
Both matching methods support multi-word phrases with word order respect, ensuring predictable behavior.
For example, a rule set to "\[starts with] winter jackets" triggers for "winter jackets sale" but not "jackets winter sale."
This update improves merchandising precision without increasing technical complexity, maintaining the same familiar workflow you use today.
See [Rules](/docs/lw-platform/lw-cs/cstudio-rules#conditions) for complete details about configuring query matching methods, including examples for e-commerce and knowledge management use cases.
This feature requires Fusion or Lucidworks Search 5.17.0 or later.
# Generative AI using Lucidworks-managed keys
Lucidworks now offers the option to execute generative AI models using Lucidworks-managed API keys, which eliminates the need for your organization to manage third-party credentials across multiple large language model (LLM) providers such as OpenAI, Anthropic, Google, and Azure.
This business option delivers significant operational and strategic advantages in a simplified deployment that eliminates complex third-party integrations and allows your teams to focus on business goals rather than credential management.
Your organization benefits from reduced costs with predictable, fixed pricing instead of unpredictable per-token charges across multiple vendors. This lowers administrative overhead for managing vendor relationships and billing accounts.
This Lucidworks-managed service handles the complete lifecycle of Gen-AI model execution through the Lucidworks AI Gateway for Gen-AI use cases in the Prediction API and Async
Prediction API.
Your organization will configure dedicated Fusion pipelines using LWAI Prediction index and query stages with `Use Case` and `Model` values provided by Lucidworks, while Lucidworks manages the underlying infrastructure, API key lifecycle, and model routing.
Contact Lucidworks to find out about feature availability per your Lucidworks Platform contract.
# Advanced grouping
Commerce Studio now supports advanced grouping, a conditional, rule-based approach to product variant grouping that gives you dynamic control over how products are grouped in search results.
Unlike standard variants that apply uniformly across all products and queries, advanced grouping uses rules to trigger different grouping parameters based on conditions such as queries, categories, or other criteria.
For example, you can group products by color for fashion queries while grouping by product ID for electronics queries.
This dynamic capability enables you to create context-aware grouping strategies that adapt to different search scenarios, improving product discovery and the customer experience.
See [Advanced grouping](/docs/lw-platform/lw-cs/cstudio-rules#advanced-grouping) in the Rules documentation for complete configuration details.
# Category management for Commerce Studio
Because category-level optimization plays a critical role in guiding product discovery and driving conversion, Commerce Studio now includes a dedicated Category Management workspace that lets merchandisers easily curate and optimize category pages without complex Fusion configuration.
You can select categories using both readable names and IDs, apply category-level rules and conditions, preview changes before publishing, and measure performance with integrated analytics.
This streamlined experience empowers your merchandising team to manage browse experiences with the same speed, control, and insight available for search—reducing reliance on engineering and accelerating time to impact.
You can configure the category field on the [Settings](/docs/lw-platform/lw-cs/cstudio-settings) screen, then [use category conditions](/docs/lw-platform/lw-cs/cstudio-editor#use-category-conditions) in the Editor to access the available categories and optimize the results for each one:
Contact Lucidworks to find out about feature availability per your Lucidworks Platform contract.
# Typeahead
The new Typeahead feature is now available in Lucidworks Platform Core Settings to help customers find results faster with real-time query suggestions as they type.
The feature supports Included and Excluded suggestion lists to control which typeahead suggestions appear as customers type. Users can manually add terms to both lists while the Platform automatically populates the table with the most popular terms from a linked [Signals Store](/docs/lw-platform/lw-analytics/signals/signals-store).
As your customer enters text into the search box on your site, the Typeahead feature uses the Included suggestions list to predict possible customer queries and displays suggestions in real time. When the customer clicks on the suggestion, results for the selected term display.
This function helps customers find relevant results faster by predicting what they are trying to search.
See [Typeahead](/docs/lw-platform/lw-core/sites-typeahead) for information to enter and manage suggestions. To retrieve a full or partial list of suggestions for your organization, see [Typeahead Service API](/docs/lw-platform/lw-core/typeahead-api).
# AI Ranking Insights
AI Ranking Insights is a new Commerce Studio feature that displays a clear, business-friendly explanation about why a product or content is ranked in a given position.
When you click a search result, and then click **AI Ranking Insights**, this feature uses a large language model (LLM) to interpret primary ranking drivers, such as text relevance, rules, boosts, rewrites, and AI-driven signals like Neural Hybrid Search (NHS), in Fusion's debug response. Specifically, AI Ranking Insights identifies key relevancy influencers including merchandising rules, matching fields, and matching logic, and then returns the reasoning in a readable format so merchandisers and search managers can quickly identify the key factors influencing a search result's relevancy.
For example, if you click on a bathroom faucet handle pinned as first in a search for "bathroom faucet", the response is:
"Bathroom Faucet Handle ranks #1 out of 12550 total results because of a pin rule that explicitly places this specific product at the top of the results for the query bathroom faucets. This merchandising rule overrides the organic search ranking to ensure the product appears in the first position."
Using this information, the merchandiser or search manager can modify the factors of this and other search results, if needed, to rank other results higher than the bathroom faucet handle.
See [AI Ranking Insights](/docs/lw-platform/lw-cs/cs-ai-ranking-insights) for complete details and instructions to use this feature.
# Image Data Enrichment
The new Image Data Enrichment feature in Lucidworks AI automatically enhances product catalogs with richer attributes by analyzing each item's images and text fields to infer relevant details.
By enriching your catalog's categories, keywords, and synonyms, you make it easier for customers to find what they're looking for—driving higher conversions and reducing search abandonment.
This feature is especially powerful when your product catalog is too large for manually adding details to every item.
See [Image Data Enrichment](/docs/lw-platform/lw-ai/data-enrichment) for complete details and instructions for enabling the feature.
You can use this feature with or without Fusion.
# Streaming responses for the LWAI Prediction API
The LWAI Prediction API now supports streaming responses, enabling applications to receive model output as it is generated. This improves responsiveness for interactive workloads such as chat experiences and real-time content generation. Clients can enable streaming by adding the `Accept: text/event-stream` header to synchronous prediction requests, which returns model output as a sequence of Server-Sent Events (SSE).
Streaming is available only when calling the LWAI Prediction API directly as Fusion does not yet support streaming responses.
See the [Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/overview#enable-streaming) for examples of curl calls and streaming output.
# Accelerated indexing with Lucidworks AI batch vectorization
Lucidworks AI continues to advance the performance of vector-based indexing. The latest release introduces new batch vectorization capability that significantly improves embedding throughput. Fusion 5.9.15 now supports this capability through the [LWAI Batch Vectorize Index Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/lwai-batch-vectorize-index-stage), enabling faster and more scalable document processing. This integration shows how Fusion and Lucidworks AI evolve together to deliver higher performance for AI-driven indexing.
Upgrade to Fusion 5.9.15 to take advantage of the new **LWAI Batch Vectorize Index Stage** and achieve faster, more scalable indexing performance. Follow the Fusion upgrade guide to complete your upgrade.
Lucidworks Search customers can request an upgrade to Fusion 5.9.15 to enable the new **LWAI Batch Vectorize Index Stage** and benefit from significantly improved indexing performance. Contact your Lucidworks representative to initiate the upgrade process.
The **LWAI Batch Vectorize Index Stage** groups documents and sends them together to the Lucidworks AI embedding endpoint. By batching requests, it minimizes API overhead, increases throughput, and improves overall pipeline efficiency.
## Performance improvements
Benchmark testing shows that using the**LWAI Batch Vectorize Index Stage** achieved indexing speeds up to 10 times faster than the single-document approach. Indexing rates reached as high as 95 documents per second, compared to fewer than 8 documents per second previously. Even with larger embeddings, throughput consistently exceeded 80 documents per second.
## How it works
The **LWAI Batch Vectorize Index Stage** buffers documents until either the maximum batch size or the maximum wait time is reached. The batch is then sent to the Lucidworks AI embedding service in a single API call, reducing overhead and enabling more consistent scaling.
The new stage allows multiple documents per API call, reducing latency and improving performance through Lucidworks AI’s optimized embedding infrastructure.
## Integrated product development
This release demonstrates how Lucidworks AI innovations directly influence Fusion’s evolution. The **LWAI Batch Vectorize Index Stage** enables Fusion users to take advantage of Lucidworks AI’s optimized batching and embedding capabilities without changing existing configurations.
This release highlights how Fusion and Lucidworks AI are being developed in parallel to deliver practical improvements. By adopting the new Batch Vectorize stage, Lucidworks AI users benefit from faster indexing and greater scalability without needing to change their embedding configuration.
For details on setup and configuration, see the [LWAI Batch Vectorize Index Stage documentation](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/lwai-batch-vectorize-index-stage).
**Introducing Agent Studio**
Lucidworks is pleased to announce the launch of Agent Studio, a new capability on Lucidworks Platform that enables organizations to create and deploy AI-powered virtual agents for their websites. These prebuilt, customizable agents address a wide range of use cases, including product question-and-answer experiences and generative AI summarization.
### Designed for B2B and B2C
Agent Studio supports both B2B and B2C commerce strategies. For B2B organizations, virtual agents can guide customers through complex product specifications, compatibility requirements, and contract terms, helping shorten sales cycles. For B2C retailers, agents can answer product-specific questions about features, ingredients, or usage guidelines, giving customers the confidence to complete purchases.
### Business impact
Virtual agents are available 24/7 to handle high inquiry volumes, reducing wait times and live interaction demands. This enables staff to focus on resolving complex cases, accelerating transaction completion, and improving customer satisfaction. By integrating directly into product pages, agents deliver accurate, contextual information without requiring customers to search multiple documents.
### Key features
* Library of prebuilt agent types, including Product Q\&A agents optimized for industry and use case.
* Integration with Google Cloud Platform (GCP) datasources for secure, cloud-based data ingestion.
* Simple deployment with a script and HTML snippet.
* Multiple options for product ID extraction to ensure accurate, product-specific answers.
* Customizable layout through CSS to match website branding.
### Get started
Agent Studio is available now in Lucidworks Platform. Select an agent type from the library, connect your datasource, choose an AI model for your industry, and implement the agent with a few lines of code. You can preview and refine the agent before it goes live, ensuring accurate, optimized responses.
Explore the Agent Studio documentation to discover how to create, customize, and deploy AI-powered virtual agents for your business.
Core Settings is a new capability that simplifies how organizations design and manage search experiences. It provides a single place to configure critical elements of search, including relevance, boost, and typeahead, without requiring code or infrastructure changes.
This release introduces an intuitive workflow that makes advanced features accessible in minutes. For example, teams can adjust Neural Hybrid Search by setting the balance between lexical and semantic approaches through a simple slider. What previously required technical expertise can now be managed through a straightforward interface, giving more teams direct control over search quality.
The benefits of Core Settings extend beyond configuration. The preferences defined here establish a consistent foundation that carries through to Commerce Studio and other applications. This creates a shared understanding of the optimal search setup across environments, ensuring users and customers encounter relevant, high quality results.
B2B buyers often work with large, complex catalogs and need precise access to products or parts. Core Settings enables fine control over relevance, making it easier to surface the right items quickly. Boosting rules and typeahead help reduce friction in product discovery and speed up workflows, which supports efficiency for procurement teams and sales operations.
B2C customers expect fast, personalized experiences that feel intuitive. Core Settings allows merchandising and digital teams to tune search in ways that highlight trending products, balance semantic understanding with exact keyword matches, and provide responsive typeahead suggestions. These adjustments improve product findability, reduce abandonment, and support higher conversion rates across retail and consumer channels.
Core Settings enables organizations to move faster, align teams around a unified search strategy, and deliver improved experiences to both business buyers and consumers.
Use Core Settings to quickly define how search works across your applications. Adjust relevance, apply boosting rules, enable typeahead, and fine tune Neural Hybrid Search with no coding necessary.
**Lucidworks AI now supports chunking**
Lucidworks AI now uses document chunking to deliver more precise search results and AI-generated responses from large files. Chunking breaks documents into smaller, meaningful sections so the system can identify and return the most relevant part, rather than the entire document. This capability is available in Fusion 5.9.12 and later.
This delivers:
* **More accurate answers**: Generative AI uses the most relevant sections, improving accuracy and reducing irrelevant output.
* **Better context**: Responses are grounded in the specific parts of a document that match the prompt.
* **Fresher content**: Updates and removals apply at the section level, ensuring AI uses current information.
* **Improved ranking**: Search and AI results can prioritize the best-matching chunks instead of entire documents.
Five chunking strategies are available:
* **dynamic-newline**: Breaks text at newline characters, adjusting chunk size automatically.
* **dynamic-sentence**: Groups sentences together with flexible chunk length based on context.
* **regex-splitter**: Uses a custom regular expression to define chunk boundaries.
* **semantic**: Organizes content into sections based on meaning and related ideas.
* **sentence**: Creates chunks from single sentences or a set number of sentences, based on configuration.
In B2B and B2C commerce, these strategies help AI find exact product details, specifications, or terms hidden deep in large catalogs, manuals, and contracts, improving customer experiences and sales efficiency.
Discover how to use Lucidworks AI to chunk documents for improved retrieval, relevance, and ranking in Fusion 5.9.
# Agent Studio
Source: https://doc.lucidworks.com/docs/lw-platform/lw-agent-studio/overview
Agent Studio lets you create and implement AI-powered virtual agents for your website. The agents are pre-built, highly customizable AI solutions that cover a wide array of use cases such as a product question and answer chatbot or conversational agents that respond to multiple questions from a user in a single session.
Agent Studio agents can be implemented to improve customer experiences with your website and enhance organizational goals for B2B, B2C, and knowledge management industries.
Virtual agents provide the following benefits:
* Available 24/7 for inquiries from customers all over the world and respond with accurate, standardized information tailored to those customer queries.
* Handle high volumes of customer inquiries, which lessens wait times for customers who choose to call and reduces the number of live customer interactions. These factors allow your personnel to resolve more complex customer issues, which in turn, helps shorten the timeframe to complete purchase transactions or obtain the information your organization provides.
For examples about how the virtual agents can be used in your organization, see the section about the [Library of available agents](#library-of-available-agents).
## Agent configuration overview
Agent Studio is hosted on Lucidworks Platform and is easily configured and implemented. You create an agent by choosing from a [library of prebuilt agents](#library-of-available-agents), and then select from a list of generative AI models, industry options, and datasources, all hosted in the cloud. Then, you complete the implementation by placing a small script and HTML markup on your site, based on when you want the virtual agent to load.
Agent Studio does not support custom large language model (LLM) training.
## Library of available agents
The following is a list of available agent types you can select to create custom virtual agents.
* **[Product Q\&A](#product-q\&a-agent)**: Generates frequently asked questions to simplify and enhance user understanding of documents or products.
* **[Conversational Q\&A](#conversational-q\&a-agent)**: Assesses natural language user queries for a product and generates accurate, real-time answers grounded in that product's own documentation, such as datasheets, manuals, specifications, and other supporting PDF technical content.
* **[Conversational Q\&A FAQ](#conversational-q\&a-faq-agent)**: Provides the same functions as the conversational Q\&A agent, but also generates a series of frequently asked questions.
## Agent use cases and configuration
This section provides detailed information about each agent type.
For a video walkthrough of the setup steps for a new agent, click **Get Started** below.
### Product Q\&A agent
Based on your industry, the Agent Studio **Product Q\&A** agent generates an interactive question and answer solution that provides information from your website about products you offer or information your organization provides.
The following examples describe how you can use an Agent Studio Product Q\&A agent.
If you are a B2B organization that provides industrial technology components and a computer server manufacturer queries about fiber optic connectors, the agent displays questions that a customer may have about a specific fiber optic connector on its product detail page, based on the information ingested about that product in the datasource from documents such as instruction manuals and specification files. These questions could relate to bandwidth capabilities, equipment compatibility, and warranty terms. The customer can then select the question to view the answer without reading multiple documents to find the answer.
If you are a B2C organization that provides skin care products, the agent displays questions that a customer may have about a specific skin care product on its product detail page, based on the information ingested about that product in the datasource from documents such as ingredient lists and benefits of using that product. These questions could relate to whether that product is vegan or suitable for daily use. The customer can then select the question and view the answer, increasing customer confidence and the likelihood of a purchase.
To implement an agent, you must first set up a datasource for the agent.
Agent Studio ingests client datasource information stored in a datasource outside of Lucidworks Platform and uses that data in a specified model. You must enter the datasource details where your data is stored when you create an agent.
Agent Studio requires Google Cloud Storage (GCS) buckets as a datasource.
If your organization has not created a GCS bucket to use as a datasource, you need to create one.
For detailed information, see [Create a GCS bucket](https://cloud.google.com/storage/docs/creating-buckets).
Your data must be in JSONL *and* PDF format.
In a JSONL file, each line is a complete JSON expression representing a single document, as in the example below:
```json wrap theme={"dark"}
{"name": "Puffy Winter Jacket", "product_id": "7a45q835", "description": "This classic down-filled jacket provides maximum insulation combined with lightweight comfort.", "documents": ["https://storage.googleapis.com/my-storage/puffy-winter-jacket.pdf"]}
{"name": "Extra Puffy Winter Jacket", "product_id": "9b82c222", "description": "This classic down-filled jacket provides even more insulation combined with lightweight comfort.", "documents": ["https://storage.googleapis.com/my-storage/puffy-winter-jacket.pdf"]}
```
All fields are required. Each `product_id` must be unique. JSONL lines are not separated by commas. Enter multiple PDF files as an array of strings. If a document does not have an associated PDF file, a blank PDF file is accepted.
To create an Agent Studio agent:
1. From the **Agent Studio** screen, select **Agents** and click **+ Create New**.
2. Enter a descriptive **Name** for the agent, select an [agent type](#library-of-available-agents), and then click **Next**.
3. From the **Industry Domain** list, select the industry that best matches your business.
4. In the **Tasks** field, select the AI model to execute for that agent. The recommended model is selected by default because it is deemed to be the most relevant based on the industry selection, but you can select a different model. Click **Next**.
5. On the **Select a data connector** screen, click the connector and then click **Next**. For example, select **GCS Bucket**.
6. On the **Configure GCS bucket to import data** screen, fill out all required fields.
* In the **Name** field, enter the name of your datasource as you want it to display in Lucidworks Platform.
* In the **Processing Region** field, select the region for Lucidworks to process your data. This is not necessarily the region you created your GCS bucket in. The default value is `us-southcarolina`.
* In the **Bucket Name** field, enter the name of your GCS bucket as it displays in GCS.
* In the **Service Account JSON Key** field, enter your Google Cloud Platform (GCP) key in JSON format to allow access to your GCS bucket.
7. Click **Verify Connection** to verify the connection to the GCS bucket.
You can add multiple GCS buckets to the agent to expand the information the model uses for responses. To add another GCS bucket, click **Add New** on the Configure GCS bucket screen. If multiple GCS buckets exist and you want to remove a specific GCS bucket from your agent, select that bucket and click **Remove**.
8. When you’ve successfully created the agent, select **Get Agent Code** to [implement the agent](#agent-studio-agent-implementation). Click **Exit** and verify the agent displays on the Agents screen of Agent Studio.
You can also click [**Preview Agent** to test the agent](#preview-and-test-an-agent) after it is configured.
Major changes to an agent, such as updating the model or changing data sources, will generate a new `template-id`. While the old `template-id` remains functional, it may not reflect the results of your changes. Update your frontend code with the new `template-id` from the agent's Details tab to ensure you're using the latest configuration.
To implement and manage the agent, follow the instructions in the following sections:
* [Agent Studio agent implementation](#agent-studio-agent-implementation)
* [Additional agent management tasks](#additional-agent-management-tasks)
### Conversational Q\&A agent
The Agent Studio **Conversational Q\&A** agent is an AI-powered chat agent that lives on your organization's product detail pages. Customers can ask natural-language questions about a product, and the agent responds with accurate, real-time answers grounded in that product's own documentation, such as datasheets, manuals, specifications, and other supporting PDF technical content.
The following examples describe how you can use an Agent Studio Conversational Q\&A agent.
If you are a B2B organization that provides computer components for enterprise organizations, your customers may ask questions about the specifications and compatibility documents for your products prior to purchase. The Conversational Q\&A agent can interpret user questions, analyze the documentation, and quickly provide accurate, factually-based answers. Any follow-up questions are answered based on the information on your site.
This saves your customers time-consuming and difficult research to obtain the information they need to help make a purchasing decision. This agent enhances customer satisfaction, increases the potential for conversions, and improves the long-term relationship between your customers and your organization.
The Conversational Q\&A agent is most effective for B2C organizations that offer products with complex documentation such as technical specifications and requirements. For example, a customer asks about the installation and warranty for an uninterruptible power supply for their office computer. The agent interprets the user's questions and returns relevant, concise information derived from the documents on your site. As the customer enters additional questions, the agent responds with answers to those inquiries. This enhances the potential for sales, and minimizes customer support calls or product returns.
The Conversational Q\&A agent excels at knowledge management use cases, helping employees and stakeholders quickly surface institutional knowledge from internal documentation, policies, and procedures. For example, an HR team member asks about the company's parental leave policy and accrual rules. Instead of searching through multiple documents, the agent interprets the question and returns a clear, accurate summary drawn from official HR documentation. As the employee asks follow-up questions such as eligibility criteria or how to submit a request, the agent continues the conversation with contextually relevant answers. This reduces time spent searching for information and keeps teams aligned on internal processes.
To implement an agent, you must first set up a datasource for the agent.
Agent Studio ingests client datasource information stored in a datasource outside of Lucidworks Platform and uses that data in a specified model. You must enter the datasource details where your data is stored when you create an agent.
Agent Studio requires Google Cloud Storage (GCS) buckets as a datasource.
If your organization has not created a GCS bucket to use as a datasource, you need to create one.
For detailed information, see [Create a GCS bucket](https://cloud.google.com/storage/docs/creating-buckets).
Your data must be in JSONL *and* PDF format.
In a JSONL file, each line is a complete JSON expression representing a single document, as in the example below:
```json wrap theme={"dark"}
{"name": "Puffy Winter Jacket", "product_id": "7a45q835", "description": "This classic down-filled jacket provides maximum insulation combined with lightweight comfort.", "documents": ["https://storage.googleapis.com/my-storage/puffy-winter-jacket.pdf"]}
{"name": "Extra Puffy Winter Jacket", "product_id": "9b82c222", "description": "This classic down-filled jacket provides even more insulation combined with lightweight comfort.", "documents": ["https://storage.googleapis.com/my-storage/puffy-winter-jacket.pdf"]}
```
All fields are required. Each `product_id` must be unique. JSONL lines are not separated by commas. Enter multiple PDF files as an array of strings. If a document does not have an associated PDF file, a blank PDF file is accepted.
To create an Agent Studio agent:
1. From the **Agent Studio** screen, select **Agents** and click **+ Create New**.
2. Enter a descriptive **Name** for the agent, select an [agent type](#library-of-available-agents), and then click **Next**.
3. From the **Industry Domain** list, select the industry that best matches your business.
4. In the **Tasks** field, select the AI model to execute for that agent. The recommended model is selected by default because it is deemed to be the most relevant based on the industry selection, but you can select a different model. Click **Next**.
5. On the **Select a data connector** screen, click the connector and then click **Next**. For example, select **GCS Bucket**.
6. On the **Configure GCS bucket to import data** screen, fill out all required fields.
* In the **Name** field, enter the name of your datasource as you want it to display in Lucidworks Platform.
* In the **Processing Region** field, select the region for Lucidworks to process your data. This is not necessarily the region you created your GCS bucket in. The default value is `us-southcarolina`.
* In the **Bucket Name** field, enter the name of your GCS bucket as it displays in GCS.
* In the **Service Account JSON Key** field, enter your Google Cloud Platform (GCP) key in JSON format to allow access to your GCS bucket.
7. Click **Verify Connection** to verify the connection to the GCS bucket.
You can add multiple GCS buckets to the agent to expand the information the model uses for responses. To add another GCS bucket, click **Add New** on the Configure GCS bucket screen. If multiple GCS buckets exist and you want to remove a specific GCS bucket from your agent, select that bucket and click **Remove**.
8. When you’ve successfully created the agent, select **Get Agent Code** to [implement the agent](#agent-studio-agent-implementation). Click **Exit** and verify the agent displays on the Agents screen of Agent Studio.
You can also click [**Preview Agent** to test the agent](#preview-and-test-an-agent) after it is configured.
Major changes to an agent, such as updating the model or changing data sources, will generate a new `template-id`. While the old `template-id` remains functional, it may not reflect the results of your changes. Update your frontend code with the new `template-id` from the agent's Details tab to ensure you're using the latest configuration.
To implement and manage the agent, follow the instructions in the following sections:
* [Agent Studio agent implementation](#agent-studio-agent-implementation)
* [Additional agent management tasks](#additional-agent-management-tasks)
### Conversational Q\&A FAQ agent
The Agent Studio **Conversational Q\&A FAQ** agent provides the following information:
* **Answers to initial and follow-up user queries.** The agent assesses user queries and then analyzes product information, specifications, and other documents on your site to generate standardized, accurate, content-based answers. If the user enters additional questions, the agent continues the conversation and responds based on your site's information.
* **Frequently asked questions.** The agent generates a series of frequently asked questions that the current user is most likely to ask. This gives the user more information about their query, which can help minimize customer support calls and product returns.
The following examples describe how you can use an Agent Studio Conversational Q\&A FAQ agent.
If you are a B2B organization that provides computer components for enterprise organizations, your customers may ask questions about the specifications and compatibility documents for your products prior to purchase. The Conversational Q\&A FAQ agent can interpret user questions, analyze the documentation, and quickly provide accurate, factually-based answers. The agent responds to any follow-up queries to continue the conversation, providing standardized answers based on your site's documentation.
In addition, the agent predicts and surfaces several relevant questions based on the product content itself, which may provide information the current user might not have thought to ask. This gives the user more information to understand the product, and helps minimize customer support calls and product returns.
This agent saves your customers time-consuming and difficult research to obtain the information they need to help make a purchasing decision. This agent enhances customer satisfaction, increases the potential for conversions, and improves the long-term relationship between your customers and your organization.
If you are a B2C organization that sells exercise equipment, a customer could ask about a specific treadmill. The agent reviews product information and provides answers to the user query. If the customer enters additional questions about price, payment options, and delivery options, the agent responds with fact-based answers from the information on your site.
The agent may also provide a list of frequently asked questions about financing options, assembly instructions, and recommended assemblers.
This conversation and section of frequently asked questions can help the customer make the choice that is best for them, which increases customer satisfaction and minimizes support interactions and product returns.
If your organization wants to provide human resource documents such as sick leave policies or code of conduct requirements, the agent can answer employee questions in natural language and surface the most relevant content from those documents. It also generates frequently asked questions based on the available documentation, helping employees self-serve before reaching out to HR. This reduces the burden on HR teams and ensures employees consistently receive accurate, policy-aligned answers.
To implement an agent, you must first set up a datasource for the agent.
Agent Studio ingests client datasource information stored in a datasource outside of Lucidworks Platform and uses that data in a specified model. You must enter the datasource details where your data is stored when you create an agent.
Agent Studio requires Google Cloud Storage (GCS) buckets as a datasource.
If your organization has not created a GCS bucket to use as a datasource, you need to create one.
For detailed information, see [Create a GCS bucket](https://cloud.google.com/storage/docs/creating-buckets).
Your data must be in JSONL *and* PDF format.
In a JSONL file, each line is a complete JSON expression representing a single document, as in the example below:
```json wrap theme={"dark"}
{"name": "Puffy Winter Jacket", "product_id": "7a45q835", "description": "This classic down-filled jacket provides maximum insulation combined with lightweight comfort.", "documents": ["https://storage.googleapis.com/my-storage/puffy-winter-jacket.pdf"]}
{"name": "Extra Puffy Winter Jacket", "product_id": "9b82c222", "description": "This classic down-filled jacket provides even more insulation combined with lightweight comfort.", "documents": ["https://storage.googleapis.com/my-storage/puffy-winter-jacket.pdf"]}
```
All fields are required. Each `product_id` must be unique. JSONL lines are not separated by commas. Enter multiple PDF files as an array of strings. If a document does not have an associated PDF file, a blank PDF file is accepted.
To create an Agent Studio agent:
1. From the **Agent Studio** screen, select **Agents** and click **+ Create New**.
2. Enter a descriptive **Name** for the agent, select an [agent type](#library-of-available-agents), and then click **Next**.
3. From the **Industry Domain** list, select the industry that best matches your business.
4. In the **Tasks** field, select the AI model to execute for the **Q\&A Generation** (FAQ) feature of the agent. The recommended model is selected by default because it is deemed to be the most relevant based on the industry selection, but you can select a different model.
5. In the **Tasks** field, select the AI model to execute for the **Conversation Generation** (conversational Q\&A) feature of the agent. The recommended model is selected by default because it is deemed to be the most relevant based on the industry selection, but you can select a different model. Click **Next**.
6. On the **Select a data connector** screen, click the connector and then click **Next**. For example, select **GCS Bucket**.
7. On the **Configure GCS bucket to import data** screen, fill out all required fields.
* In the **Name** field, enter the name of your datasource as you want it to display in Lucidworks Platform.
* In the **Processing Region** field, select the region for Lucidworks to process your data. This is not necessarily the region you created your GCS bucket in. The default value is `us-southcarolina`.
* In the **Bucket Name** field, enter the name of your GCS bucket as it displays in GCS.
* In the **Service Account JSON Key** field, enter your Google Cloud Platform (GCP) key in JSON format to allow access to your GCS bucket.
8. Click **Verify Connection** to verify the connection to the GCS bucket.
You can add multiple GCS buckets to the agent to expand the information the model uses for responses. To add another GCS bucket, click **Add New** on the Configure GCS bucket screen. If multiple GCS buckets exist and you want to remove a specific GCS bucket from your agent, select that bucket and click **Remove**.
9. When you’ve successfully created the agent, select **Get Agent Code** to [implement the agent](#agent-studio-agent-implementation). Click **Exit** and verify the agent displays on the Agents screen of Agent Studio.
You can also click [**Preview Agent** to test the agent](#preview-and-test-an-agent) after it is configured.
Major changes to an agent, such as updating the model or changing data sources, will generate a new `template-id`. While the old `template-id` remains functional, it may not reflect the results of your changes. Update your frontend code with the new `template-id` from the agent's Details tab to ensure you're using the latest configuration.
To implement and manage the agent, follow the instructions in the following sections:
* [Agent Studio agent implementation](#agent-studio-agent-implementation)
* [Additional agent management tasks](#additional-agent-management-tasks)
## Agent Studio agent implementation
The script, component markup, and CSS code for customization may be different based on the agent type you create. Please copy the information in the Agent Details section for each specific agent you create.
After creating your agent, you can access the code you need to implement the virtual agent on your website.
There are two considerations when implementing the virtual agent:
* Where you [place the script and HTML snippet](#options-to-obtain-the-script) that enables the option to display the virtual agent
* How the virtual agent [obtains the product ID](#options-to-obtain-the-product-id) so it provides accurate information
The following sections describe the options to place the script, obtain the product ID, and format the agent.
To implement the agent on your site, obtain the script and HTML snippet from Lucidworks Platform.
To add the script to your website, complete the following steps:
1. Navigate to Agent Studio, click **Agents**, and then click the agent you want to implement.
2. In the **Agent Details** section, scroll to the **Agent Code** section, and in the **Add the Script Tag** area, click **Copy**.
3. Paste the script into the `` or `` section of your website’s HTML code.
You have two options for placing the script: in the `` or `` of your HTML. The location of the script determines when it will load. For best performance, place the script in the `` of your website’s HTML. If you place the script in the `` of your website’s HTML, the script loads before your entire HTML page body loads, along with any other scripts your `` may contain. The agent still only displays on the individual pages where you have also placed the HTML markup snippet in the ``.
The following example shows how to include the script in the ``. It includes one option for the HTML snippet to obtain the product ID, which is described in [options to obtain the product ID](#options-to-obtain-the-product-id):
```html theme={"dark"}
```
You may also place the script in the `` of your website’s HTML. If you place the script in the `` of specific pages, the script loads only on the pages that contain the script, but the script loads after all the other scripts in your ``. Even in this case, the agent only displays if you have also placed the HTML markup snippet on that page’s ``. However, the order that you place the script and HTML markup snippet relative to other items on your page can affect the order in which other page elements load. For this reason, this placement is not the recommended one.
The following example shows how to include the script in the ``. It includes one option for the HTML snippet to obtain the product ID, which is described in [options to obtain the product ID](#options-to-obtain-the-product-id):
```html theme={"dark"}
```
No matter where you place the script, the `` HTML snippet must be present in the `` of a page in order for the agent to display.
To use the Signals beacon on the same page as your agent, you need additional script configuration.
See [Configure the beacon with Agent Studio or UI Studio](/docs/lw-platform/lw-analytics/signals/signals-beacon#configure-the-beacon-with-agent-studio-or-ui-studio).
The Product Q\&A, Conversational Q\&A, and Conversational Q\&A FAQ agents let your customers ask questions about a specific product. The agents know the exact product by obtaining the product’s unique ID. The product ID is included in the HTML snippet that you add to the `` of your webpage. There are three ways the agents obtain the product ID.
This method is suitable when the product ID is consistently found in the page URL as a query parameter, or when you want to avoid manually injecting it into an HTML snippet.
Use this method in combination with the [option you chose to obtain the script](#options-to-obtain-the-script).
After you place the script in either the `` or `` section:
1. Scroll to the **Add Component Markup** section and the in **Automatic Extraction** area, click **Copy** to copy the HTML snippet.
2. Paste the `` HTML snippet in the `` of every page where you want the virtual agent to display.
3. In the **Specify the Product ID** section, enter the **URL Query Parameters** value for the `product-id`. For example, if your URL is `https://www.example.com/sale?q=solvents&product=1234` and your product ID is 1234, enter `product`.
4. Verify the agent displays on the appropriate pages of your website.
5. Test the agent again by entering various queries to determine if the agent returns appropriate responses.
You would use this method if, for example, the product ID is not in the URL, or at least not in the query parameter. Or if you do not want to inject the product ID into an HTML snippet. You might also use this method if the product ID is already built into the page through CSS selectors and you know how to isolate it.
Use this method in combination with the [option you chose to obtain the script](#options-to-obtain-the-script).
You must be able to extract the exact product ID through CSS selectors. For example, if the product ID is 1234, then you must be able to extract `1234` from your CSS selectors without any additional text, such as `product-id: 1234`.
After you place the script in either the `` or `` section:
1. Scroll to the **Add Component Markup** section and the in **Automatic Extraction** area, click **Copy** to copy the HTML snippet.
2. Paste the `` HTML snippet in the `` of every page where you want the virtual agent to display.
3. In the **Specify the Product ID** section, enter the **CSS Selector** elements of the CSS rule that contains the `product_id`. The following example uses `.product .product-name @data-value` that extracts a product ID of 1234.
```html theme={"dark"}
Product Name
```
4. Verify the agent displays on the appropriate pages of your website.
5. Test the agent again by entering various queries to determine if the agent returns appropriate responses.
This option is primarily used if the product ID exists in the product detail page but is not made available in either the HTML or as a query parameter. Even when the agent cannot easily retrieve the product ID, it can be injected dynamically via markup.
Use this method in combination with the [option you chose to obtain the script](#options-to-obtain-the-script).
After you place the script in either the `` or `` section:
1. Scroll to the **Add Component Markup** section and the **Markup Attribute** area, click **Copy** to copy the HTML snippet.
2. Paste the `` HTML snippet in the `` of every page where you want the virtual agent to display. The example code sample uses `https://www.example.com/sale/1234` and `product-id` in the snippet. If your `product-id` attribute name is different than what is copied in the snippet, manually change the name of your attribute.
```html theme={"dark"}
```
3. Verify the agent displays on the appropriate pages of your website.
4. Test the agent again by entering various queries to determine if the agent returns appropriate responses.
The agent details screen also contains a code block for CSS customization. You can use this code block to customize the appearance of the agent so it matches your website’s existing layout.
The following code sample displays some custom styling options for the agent. Based on the agent type, the styling options may vary.
All agent customizations must be contained within the `.lw-template` properties.
```css theme={"dark"}
/* Custom styling for the agent template */
.lw-template {
/* Container dimensions */
min-height: 300px;
max-height: 800px;
min-width: 300px;
max-width: 600px;
/* Optional styling examples: */
--qna-font-family: Arial, sans-serif;
--qna-container-border-radius: 10px;
--qna-header-font-size: 32px;
--qna-header-icon-color: #a19eff;
--qna-answer-bubble-bg-color: #add8e6;
}
```
## Additional agent management tasks
This option lets you preview and test an agent to determine if its responses accurately reflect the information from your website and datasource before you implement it.
Navigate to Agent Studio, click the **Agents** tab, and then select the agent you want to preview or test.
When you select an existing agent, the Agent Details display the following sections:
* **Agent Information** such as name, type, and model for the agent.
* **Agent Preview** to test the agent to determine if the responses accurately reflect the information on your website.
* **Agent Code** to embed the agent into your web page.
Enter information in the **Agent Preview** section to test the response for the agent.
To make changes to the agent:
1. Navigate to Agent Studio, click **Agents**, hold the pointer over the agent you want to change, and click the pencil icon.
2. Enter the changes to each of the screens, clicking **Next** to advance to the next screen, and then click **Exit** at the final screen to save the changes.
3. Verify the changes by testing the agent using the steps in the [Agent Preview](#preview-and-test-an-agent) section. If necessary, you can refine the agent until you determine that the returned responses are accurate and optimized.
If you no longer want to use an agent, you can delete it. To delete an agent:
1. Navigate to Agent Studio, click **Agents**, hold the pointer over the agent you want to change, and then click the trash can icon.
2. Confirm the deletion and then review the Agents screen and verify that the agent no longer displays.
Deleting an agent is permanent. If you want to use a deleted agent, you must add it again as a new agent.
# Quickstart
Source: https://doc.lucidworks.com/docs/lw-platform/lw-agent-studio/quickstart
This quickstart explains how to create an embeddable AI agent.
Agent Studio provides a simple interface for quickly creating an intelligent agent that can answer customer questions right inside your website.
For a video walkthrough of the setup steps for a new agent, click **Get Started** below:
Here's how it works:
What business goals will this agent help you meet?
In what context will customers encounter this agent?
What types of questions do your customers ask?
Knowing the answers to these questions helps you choose the right type of agent and the right content to train it for maximum effectiveness.
Agent Studio reads data directly from a GCS bucket, not from Fusion.
Your data must be in JSONL *and* PDF format.
In a JSONL file, each line is a complete JSON expression representing a single document, as in the example below:
```json wrap theme={"dark"}
{"name": "Puffy Winter Jacket", "product_id": "7a45q835", "description": "This classic down-filled jacket provides maximum insulation combined with lightweight comfort.", "documents": ["https://storage.googleapis.com/my-storage/puffy-winter-jacket.pdf"]}
```
Your library includes one or more pre-built agent templates, depending on your Lucidworks subscription.
Navigate to the Agent Studio Library screen and click the agent that's best suited for your use case.
You can [contact Lucidworks](mailto:support@lucidworks.com) if you need additional agent templates for the evolving needs of your business.
Agent Studio guides you through some simple setup steps to connect your agent to your data.
You can also configure options that help train your agent to provide the most helpful answers for your customers, such as the industry domain and the AI model to use.
When you finish configuring your agent, Lucidworks AI needs some time to perform the initial training that fully enables it.
While training is in progress, you can update your website code as explained in the final step below.
You can also preview and test your agent right inside Agent Studio.
Click the **Agents** tab, and then select the agent you want to preview or test. The Agent Details display. The **Agent Preview** section lets you test the agent to determine if the responses accurately reflect the information on your website.
Now you're ready to embed the agent where your customers will see it.
Click **Get Agent Code** to get ready-to-use code snippets that you can copy and paste into your site code:
* Copy the `
```
Every site works a little differently. The Signals Beacon is flexible, so you can set it up in the way that fits your site best.
To get started, configure the Signals Beacon to collect these user insights:
When users search for products, they land on a search results page. They might use facets to narrow down what they see.
Capture the user's search query and selected facets. Map the query and facet parameters from the URL to the `query` field in the Signals Beacon configuration.
```html URL theme={"dark"}
https://example.com/search?q=USER_QUERY&fq=color:("red","pink")
```
```json Query Config highlight={22-30} theme={"dark"}
{
"attributes": {
"idQueryParam": "pid",
"numberTypeFields": [
"price",
"quantity",
"revenue"
],
"productPath": ".resultsItem"
},
"cartAdd": {
"trigger": ".lw-cart-add-trigger"
},
"click": {},
"fields": {
"productId": ":scope .resultId @text",
"title": ":scope .itemTitle @text"
},
"purchase": {
"trigger": ".lw-purchase-trigger"
},
"query": {
"facets": {
"keyValueDelimiter": ":",
"queryParams": "fq",
"valueExtractorRegex": "/\\\"(.*)\\\"/",
"valuesDelimiter": ","
},
"queryParam": "q"
}
}
```
When users find a product they like, they click on it.
Capture that interaction. Map the product ID from the HTML to the `click` and `fields` fields in the Signals Beacon configuration.
The `click` field inherits values from the `fields` field. It’s recommended to use the `fields` field directly.
```html HTML highlight={3-4} theme={"dark"}
```
```json Click Config highlight={14-18} theme={"dark"}
{
"attributes": {
"idQueryParam": "pid",
"numberTypeFields": [
"price",
"quantity",
"revenue"
],
"productPath": ".resultsItem"
},
"cartAdd": {
"trigger": ".lw-cart-add-trigger"
},
"click": {},
"fields": {
"productId": ":scope .resultId @text",
"title": ":scope .itemTitle @text"
},
"purchase": {
"trigger": ".lw-purchase-trigger"
},
"query": {
"facets": {
"keyValueDelimiter": ":",
"queryParams": "fq",
"valueExtractorRegex": "/\\\"(.*)\\\"/",
"valuesDelimiter": ","
},
"queryParam": "q"
}
}
```
When users add a product to their shopping cart, that action shows strong interest and intent to buy.
Capture this interaction by mapping the click from your HTML to the `cartAdd` field in the Signals Beacon configuration. Tracking this helps you understand what products matter most and where users may drop off before purchasing.
```html HTML highlight={7} theme={"dark"}
12345
Wireless Noise-Canceling Headphones
199.99
```
```json Cart Add Config highlight={11-13} theme={"dark"}
{
"attributes": {
"idQueryParam": "pid",
"numberTypeFields": [
"price",
"quantity",
"revenue"
],
"productPath": ".resultsItem"
},
"cartAdd": {
"trigger": ".lw-cart-add-trigger"
},
"click": {},
"fields": {
"productId": ":scope .resultId @text",
"title": ":scope .itemTitle @text"
},
"purchase": {
"trigger": ".lw-purchase-trigger"
},
"query": {
"facets": {
"keyValueDelimiter": ":",
"queryParams": "fq",
"valueExtractorRegex": "/\\\"(.*)\\\"/",
"valuesDelimiter": ","
},
"queryParam": "q"
}
}
```
One of the most important actions that drives your business is a user purchase. When users complete the checkout process, it confirms their decision to buy.
Make sure to capture this event. Map the relevant HTML to the purchase field in the Signals Beacon configuration.
```html HTML highlight={9} theme={"dark"}
12345
Wireless Noise-Canceling Headphones
1
199.99
199.99
```
```json Purchase Config highlight={19-21} theme={"dark"}
{
"attributes": {
"idQueryParam": "pid",
"numberTypeFields": [
"price",
"quantity",
"revenue"
],
"productPath": ".resultsItem"
},
"cartAdd": {
"trigger": ".lw-cart-add-trigger"
},
"click": {},
"fields": {
"productId": ":scope .resultId @text",
"title": ":scope .itemTitle @text"
},
"purchase": {
"trigger": ".lw-purchase-trigger"
},
"query": {
"facets": {
"keyValueDelimiter": ":",
"queryParams": "fq",
"valueExtractorRegex": "/\\\"(.*)\\\"/",
"valuesDelimiter": ","
},
"queryParam": "q"
}
}
```
When you're ready to start collecting user insights in Lucidworks Analytics using the Signals Beacon, activate it in the **Integrations** screen. Add your site to the list of **Allowed origins**.
Once added, the Signals Beacon starts sending user insights to Lucidworks Analytics. Your **Signals analytics** dashboard will begin to show key metrics.
# Beacon configuration interface
Source: https://doc.lucidworks.com/docs/lw-platform/lw-analytics/signals/beacon-configuration-interface
[localhost link]: http://localhost:3000/docs/lw-platform/lw-analytics/signals/beacon-configuration-interface
[mintlify link]: https://doc.lucidworks.com/docs/lw-platform/lw-analytics/signals/beacon-configuration-interface
[old doc.lw link]: https://doc.lucidworks.com/lw-platform/analytics/k6vyeq
This topic details the `BeaconConfig` interface, including related interfaces, types, and default values.
For more information about the script to paste into the `` tag of your website’s HTML file and the JSON configuration, see [Signals Beacon](/docs/lw-platform/lw-analytics/signals/signals-beacon).
## BeaconConfig
This object details the primary configuration interface for the signals beacon.
The beacon supports both Retail signals and legacy site search signals.
This attribute specifies a global configuration that will be inherited by each explicit action configuration for the beacon signals.
Configuration for query signals.
Configuration for click signals.
Configuration for add-to-cart signals.
Configuration for purchase complete signals.
The main section for extracting product field values from your site.
This specifies how beacon signals should be handled when the main product page is searched.
The configuration for beacon signals sent when a product is clicked.
The configuration for beacon signals sent when a product is added to the shopping cart or bag.
The configuration for beacon signals sent when the user proceeds to the checkout phase on your site.
## BeaconFieldConfig
This is a type alias for field configuration. The value can be a string, or a `BeaconFieldRichConfig` object.
## BeaconFieldRichConfig
This object details the interface to fine tune field details.
This is the field value path.
This string is excluded from the raw value.
This property provides additional typing information that allows for proper transformation of the value.
## BeaconAttributes
This object details the interface for the global configuration inherited by each action configuration.
The URL query parameter used to get the product ID from the product page.
The local storage key for the consumer page storing the cart UUID.
The CSS selector for targeting product elements. This is the global path inherited by each signal action.
Optional inactivity threshold in milliseconds that ends a session and starts a new one. The default is 2400000 ms (40 minutes).
This specifies the fields that are treated as number type fields.
This provides a more advanced way to obtain product details using serialized JSON in an element attribute.
## BeaconSignalProps
This object details the interface that provides common properties for beacon signals.
The custom selector to target product elements specific to particular retail actions. The action values are query, click, cartAdd, and purchase.
The main section for extracting product field values from your site.
## BeaconQueryProps
This object details the interface for configuring beacon signals sent during a product search.
The URL query parameter the beacon uses to monitor query updates.
The CSS selector for targeting multiple page elements related to product query activities. The beacon analyzes the comma-separated elements, monitors when the 'Enter' key is pressed on the input element as well as clicks on the remaining elements, then sends query signals accordingly.
In addition, the beacon supports the ability to track the 'Enter' key along with clicks on the search button for scenarios where there is no query parameter if the `trigger` property is used. For example, the main search flow could use the query parameter `q` while the search within the search uses the beacon properties to record both the input and the search button clicks. An example configuration is:
```"query": { theme={"dark"}
"queryParam": "q",
"trigger": ".search-box input, .search-box .btn"
}
```
This specifies the delay, in milliseconds, for tracking element updates in the query.
## BeaconClickProps
This object details the interface for configuring beacon signals when the user clicks a product.
The custom selector to target product elements specific to particular retail actions. The action values are query, click, cartAdd, and purchase.
The main section for extracting product field values from your site.
## BeaconCartAddProps
This object details the interface for configuring beacon signals sent when a product is added to the shopping cart or bag.
The custom selector to target product elements specific to particular retail actions. The action values are query, click, cartAdd, and purchase.
The main section for extracting product field values from your site.
The CSS selector for targeting multiple page elements related to trigger add-to-cart actions.
## BeaconPurchaseProps
This object details the interface for configuring beacon signals sent when the user proceeds to the checkout phase of the transaction.
The custom selector to target product elements specific to particular retail actions. The action values are query, click, cartAdd, and purchase.
The main section for extracting product field values from your site.
The CSS selector for targeting multiple page elements related to trigger purchase or checkout actions.
The path to the element that displays the total sum of the items in the cart.
# Signals Beacon FAQ
Source: https://doc.lucidworks.com/docs/lw-platform/lw-analytics/signals/faq
[localhost link]: http://localhost:3000/docs/lw-platform/lw-analytics/signals/faq
[mintlify link]: https://doc.lucidworks.com/docs/lw-platform/lw-analytics/signals/faq
[old doc.lw link]: https://doc.lucidworks.com/lw-platform/analytics/vdi9jl
This page contains frequently-asked questions about the Signals Beacon.
## Does the beacon impact site performance or load times?
The Signals Beacon is designed to be lightweight and efficient.
It uses a lazy loading technique to minimize the impact on page load times and overall site performance.
The beacon scripts are served from Google Cloud Storage, which ensures fast loading with minimal overhead.
## What events does the Signals Beacon track?
The Signals Beacon transmits the URLs of visited pages, actions taken on those pages, and the results of those actions.
For example, when a user performs a search, the beacon captures the list of displayed products and subsequent interactions such as clicks, cart adds, or purchases.
The following signals are collected:
* **Query signals.** When a user performs a search, a signal is recorded.
* **Click signals.** When a user clicks on a product, a signal is recorded. One example of a click signal is when a user clicks a search result, which is a strong indicator the result is relevant.
* **Cart-add signals.** When a user clicks to add an item to the cart, a signal is recorded.
* **Purchase complete signals.** When a user completes a purchase and is taken to the order confirmation page, a signal is recorded.
* **Facet usage signals.** When a user interacts with the facets associated with the results, a signal is recorded.
## How do I customize the beacon to track specific user actions?
By default, the Signals Beacon tracks visitor, session, query, and cart IDs.
You can customize which IDs the beacon tracks by [editing the JSON configuration](/docs/lw-platform/lw-analytics/signals/signals-beacon#install-the-beacon).
## Can I send signals directly to Analytics Studio without using the Signals Beacon?
Yes, by using the [Signals API](/docs/lw-platform/lw-analytics/signals/signals-api).
## Does the Signals Beacon track users between devices?
No, the user is tracked in the browser’s local storage and is tracked anonymously (no personally identified information, like name or address).
## If I have multiple websites, do I need to create multiple beacons?
Yes, you need to create a separate Signals Beacon for each website.
Each website’s beacon has its own Signals Store and dashboard.
## How does the beacon handle user privacy and data security?
The Signals Beacon does not use cookies and only captures anonymized session data to respect user privacy.
IP addresses are anonymized by dropping the last octet, ensuring no precise user location or personal data is stored.
For more information, see [Lucidworks Signals Beacon Privacy and Security policies](/docs/policies/lw-signals-beacon) and [GDPR Compliance policy](/docs/policies/lw-signals-beacon-gdpr-compliance)
## Can the Signals Beacon replace indexing signals in Fusion?
You can use the Signals Beacon in addition to or instead of indexing signals in Fusion.
## Can I use the Signals Beacon in a self-hosted, non-SaaS deployment?
Yes, the Signals Beacon is part of a standalone SaaS offering that captures and sends signals to the Lucidworks Signals Store.
## Can I use the Signals Beacon with dynamic CSS selectors?
Yes.
If your site uses dynamically-generated CSS selectors, you can configure the field paths for the Signals Beacon using data attributes or the `textContent` special selector.
See [Field path configuration](/docs/lw-platform/lw-analytics/signals/signals-beacon#special-selectors) for details.
## Does bot traffic affect my signals data?
No, the Signals Beacon automatically identifies and blocks a list of Google’s crawler IP addresses to ensure your signals data is accurate. This prevents bot traffic from triggering the Signals Beacon on your website and skewing your analytics.
# Signals
Source: https://doc.lucidworks.com/docs/lw-platform/lw-analytics/signals/overview
[old doc.lw link]: https://doc.lucidworks.com/lw-platform/analytics/0le96f
[localhost link]: http://localhost:3000/docs/lw-platform/lw-analytics/signals/overview
[mintlify link]: https://doc.lucidworks.com/docs/lw-platform/lw-analytics/signals/overview
The signals features in Analytics Studio include the [signals beacon](/docs/lw-platform/lw-analytics/signals/signals-beacon), a lightweight script tag that gathers query, click, cart-add, purchase complete, and facet usage signals from customer interactions on your website and the [Signals Store](/docs/lw-platform/lw-analytics/signals/signals-store) for the designated workspace where signals are stored.
Instead of using the signals beacon, you can manually configure and maintain code in your backend codebase by using the [Signals API](/docs/lw-platform/lw-analytics/signals/signals-api) to send signals to Analytics Studio. This method is useful if you need to send signals from your backend, rather than a website or frontend application, or if your website does not support JavaScript.
Lucidworks adheres to strict data privacy and security policies, including General Data Protection Regulation (GDPR) compliance, when collecting and storing signals. Security features include user IP address anonymization, exclusion of personally-identifiable information, and secure storage. The policies also provide guidelines and best practices to help clients implement compliant configurations. To review the detailed policies, see [Lucidworks Signals Beacon Privacy and Security policies](/docs/policies/lw-signals-beacon) and [GDPR Compliance policy](/docs/policies/lw-signals-beacon-gdpr-compliance).
The following signals are collected:
* **Query signals.** When a user performs a search, a signal is recorded.
* **Click signals.** When a user clicks on a product, a signal is recorded. One example of a click signal is when a user clicks a search result, which is a strong indicator the result is relevant.
* **Cart-add signals.** When a user clicks to add an item to the cart, a signal is recorded.
* **Purchase complete signals.** When a user completes a purchase and is taken to the order confirmation page, a signal is recorded.
* **Facet usage signals.** When a user interacts with the facets associated with the results, a signal is recorded.
If present in the signal, the information collected by the signals beacon and housed in the Signals Store for each signal is:
* Product ID
* Title
* Image URI
* Price
## Overview of signals process
This section provides an overview of how to implement the use of signals in Analytics Studio. More information is detailed in additional topics.
1. Analyze the queries, clicks, cart-adds, and purchases on your site to determine areas that could be strengthened to enhance your company’s goals.
2. Create the [Signals Store](/docs/lw-platform/lw-analytics/signals/signals-store) for the designated workspace. For example, you may have a testing workspace and a production workspace. The Signals Store for the testing workspace may contain data that would not be available in your production site.
3. Add the [signals beacon](/docs/lw-platform/lw-analytics/signals/signals-beacon) to the `` section of your site. This includes not only header information, but specific field identifiers for the signals based on your site analysis.
When the signals beacon is added, signals are:
* Gathered by the signals beacon
* Stored in the Signals Store
* Sent to [Analytics Studio metrics](/docs/lw-platform/lw-analytics/analytics-screen) that provide valuable customer behavior insights and can be used to improve your business
# Signal Usage Metrics
Source: https://doc.lucidworks.com/docs/lw-platform/lw-analytics/signals/signal-usage
The Signal Usage screen displays global signal usage metrics for a specified date range.
Here you can see the total volume of signals from all of your signals buckets, distinguished by signal type.
You can use this screen to confirm that signals are flowing into the system so that you can use them in Analytics Studio, Commerce Studio, or anywhere signals are displayed. This graph lets you identify issues with signal intake and take prompt action to adapt settings and configuration to prevent issues that might be caused by missing signals or incomplete data.
Click your organization type for information about the benefits of using the Signal Usage screen:
These metrics reflect user behavior patterns that help you improve product offerings and campaigns that enhance the customer search experience and increase conversions. For example, a significant increase during a promotional campaign helps confirm the success of the event, and lets you adjust the campaign to increase conversions.
The Signal Usage screen helps your organization monitor data flow and identify ingestion and other issues quickly, which enhances the ability to maintain operational efficiency. For example, if a significant decrease in signals occurs, you can troubleshoot automated workflows and other areas to determine the cause.
Signal usage metrics provide information about relevance, ranking, and discoverability of site content. For example, a decrease in signals can indicate users may not be finding the information they need. A review of relevance settings and potential content gaps could help resolve any issues.
To access the Signal Usage screen, open the megamenu and click **Signals**, then click the **Signal Usage** tab.
Click **Get Started** below for a quick demo that shows you how to navigate to the Signal Usage screen and explore your signal data.
# Platform Signals API
Source: https://doc.lucidworks.com/docs/lw-platform/lw-analytics/signals/signals-api
[old doc.lw link]: https://doc.lucidworks.com/lw-platform/analytics/uist05
[localhost link]: http://localhost:3000/docs/lw-platform/lw-analytics/signals/signals-api
[mintlify link]: https://doc.lucidworks.com/docs/lw-platform/lw-analytics/signals/signals-api
The Platform Signals API and the embed token feature can be used instead of the [signals beacon](/docs/lw-platform/lw-analytics/signals/signals-beacon) to collect query, click, cart-add, and purchase complete signals. When the signals are collected, you can use the insights and analytics in [Analytics Studio](/docs/lw-platform/lw-analytics/overview) to help refine your site and improve customer interactions. Visit signals are not currently used and are scheduled to be deprecated.
While the Javascript signals beacon functionality is versatile and configurable, your organization may need to use the Platform Signals API and embed features instead. For example, your organization may use a mobile app that cannot accept Javascript, have security configuration differences, or already use another framework for sending signals.
To use the Platform Signals API and embeds to collect signals, your engineering team must add the appropriate information to the backend codebase on your site. Examples of the fields that need to be added include anonymous visitor ID and session ID values for each customer interaction. The [Platform Signals API specification](/api-reference/get-signals/query-signals) details each signal request.
For detailed API specifications in Swagger/OpenAPI format, see [Platform APIs](/api-reference/get-signals/query-signals).
## UI components required to implement the Platform Signals API
Two values you need to use the Platform Signals API are the Signals Store ID and embed token. This section provides the steps in the necessary order to create and obtain those values from the UI.
### 1. Create a Signals Store
1. Navigate to the megamenu and click **Signals**.
2. Click **Create New**.
3. In the **Name** field, enter a unique user-defined for the Signals Store.
4. In the **Region** field, select the geographical area for the Signals Store.
This selection cannot be changed. If you need a Signals Store in a different region, you need to create another Signals Store with the other region specified.
5. Click **Create**.
### 2. Generate and copy the embed token to your codebase
To generate the embed token, navigate to the megamenu and click **Signals**.
1. Click the Signals Store where you want to configure the signals beacon and then click **Integrations > Signals Using Embed Token**.
2. If you haven’t configured embeds for this Signals Store, the "No embed token found" message displays.
3. Click **Generate**. It may take a few moments to generate the token.
The generated embed token for the Signals Store is permanent and cannot be changed.
4. In the Embed Token section, click **Copy** and then paste the token into your codebase. To view the token, click **Show** or click **Hide** to prevent it from displaying.
The embed token also displays in asterisks in the Authorization header line of all of the cURL examples. When you copy and then paste the cURL example, the value of the embed token displays instead of the asterisks. For example:
```bash wrap theme={"dark"}
--header 'Authorization: Embed EMBED_TOKEN'
```
### 3. Obtain the Signals Store ID
To obtain the Signals Store ID, navigate to the megamenu and click **Signals**.
1. Click the Signals Store where you want to configure the signals beacon and then click **Integrations > Signals Using Embed Token**.
2. The Signals Store ID displays in first line of all of the cURL examples. Copy that ID to paste into your codebase. For example:
```bash wrap theme={"dark"}
curl 'https://SIGNAL_STORE_ID.applications.lucidworks.com/retail/signals/query'
```
### 4. Copy or create code to collect signals
Lucidworks provides cURL command examples for the query, click, cart-add, and purchase complete signals that you can copy into your backend codebase and modify based on the language you use, such as Java, C#, Python, or Nodejs.
For detailed API specifications in Swagger/OpenAPI format, see [Platform APIs](/api-reference/get-signals/query-signals).
To copy cURL command examples you can paste into your site codebase, navigate to the megamenu and click **Signals**.
1. Click the Signals Store where you want to configure the signals beacon and then click **Integrations > Signals Using Embed Token**.
2. Click **Copy** to copy the cURL command example for the signal you want to add to your codebase.
3. Paste the command in your codebase and modify based on your organizational needs. If you choose not to use the examples Lucidworks provides, enter code into your codebase manually.
## Example user flow and signal events
This section provides a typical user interaction flow for browsing and purchasing a product by describing the required API signals and UUID generation necessary for event tracking.
You can program your site to generate UUIDs using any standards-compliant method. Each signal depends on proper UUID generation and tracking. Consistent IDs are necessary for signal integrity and accurate metrics for analytics.
To successfully create signals and ensure complete signal tracking, your site must generate and persist the following UUIDs:
* `visitorId` and `sessionId` are required in every signal (query, click, cartadd, and purchase complete) to identify the user and session.
* `queryId` is required in every signal (query, click, cartadd, and purchase complete) to identify the query, including refinements like facet selection.
* `searchTerms` is required in the query signal to return results.
* `cartId` is required in the cartadd and purchase complete signals for cart activity and must be replaced with a different UUID for each new user cartadd signal.
* `orderId` is optional, and is used on the purchase\_complete signal and identifies an actual purchase as opposed to an attempted purchase.
As part of the code you add to your site, you need to implement UUID generation and determine how to store these values and all other parameters. For example, `visitorId` is typically stored in `localStorage` to persist across sessions, and `sessionId` is usually stored in `sessionStorage` to reset between sessions.
Other parameters are detailed in the [Platform Signals API specification.](/api-reference/get-signals/query-signals)
### Sample user interaction flow
The following example provides typical interactions when a customer searches for and purchases a refrigerator.
### 1. Site entry and ID generation
When a user visits your site, the site must generate the `visitorId` and `sessionId` universal unique identifiers (UUIDs).
### 2. Search query and initial query signal
When the user searches for “silver french door bottom freezer refrigerator,” your site must generate a new `queryId` UUID and send a query signal.
```json wrap expandable Generate queryID theme={"dark"}
curl --request POST \
--url https://SIGNAL_STORE_ID.applications.lucidworks.com/retail/signals/query \
--header 'Content-Type: application/json' \
--data '[
{
"timestamp": 1744305461169,
"visitorId": "abcc8e25-d30f-2212-9f60-9b45a71453cf",
"sessionId": "41a50e2-8e32-4674-5429-e1450dfdbb14",
"queryId": "2a97fc33-3335-3369-c2dc-456b775eb90f",
"ipAddress": "111.222.333",
"searchTerms": "silver french door bottom freezer refrigerator",
"productDetails": [
{
"product": {
"productId": "1234567",
"currencyCode": "USD",
"price": 2592,
"title": "Silver french door bottom freezer 27.4 cu ft refrigerator",
"imageuri": "https://example.com/images/refrigerators/silver/bottomfreezer/1234567.jpg"
},
"quantity": 1
}
],
"sourceId": "string",
"uri": "https://example.com/search?q=silver+french+door+bottom+freezer+refrigerator"
}
]'
```
### 3. Product click and click signal
When the user clicks a product, your site must generate a click signal that uses the same `queryId` generated in the query request that returned the results on which the user clicks.
```json wrap expandable Click signal theme={"dark"}
curl --request POST \
--url https://SIGNAL_STORE_ID.applications.lucidworks.com/retail/signals/click \
--header 'Content-Type: application/json' \
--data '[
{
"timestamp": 1744305461169,
"visitorId": "abcc8e25-d30f-2212-9f60-9b45a71453cf",
"sessionId": "41a50e2-8e32-4674-5429-e1450dfdbb14",
"queryId": "2a97fc33-3335-3369-c2dc-456b775eb90f",
"ipAddress": "111.222.333",
"productDetails": [
{
"product": {
"productId": "1234567",
"currencyCode": "USD",
"price": 2592,
"title": "Silver french door bottom freezer 27.4 cu ft refrigerator",
"imageuri": "https://example.com/images/refrigerators/silver/bottomfreezer/1234567.jpg"
},
"quantity": 1
}
],
"sourceId": "string",
"uri": "https://example.com/refrigerators/silver/bottomfreezer",
"selectedPosition": 1
}
]'
```
The `selectedPosition` field is critical for calculating Average Click Position analytics. Use one-based indexing where the first item in the search results is 1, the second is 2, and so on. Do not use zero-based indexing.
### 4. Cart addition and cart-add signal
When the user adds the product to their cart, your site must generate a `cartId` and send a cart-add signal.
```json expandable Cart add signal theme={"dark"}
curl --request POST \
--url https://SIGNAL_STORE_ID.applications.lucidworks.com/retail/signals/cartadd \
--header 'Content-Type: application/json' \
--data '[
{
"timestamp": 1744305461169,
"visitorId": "abcc8e25-d30f-2212-9f60-9b45a71453cf",
"sessionId": "41a50e2-8e32-4674-5429-e1450dfdbb14",
"queryId": "2a97fc33-3335-3369-c2dc-456b775eb90f",
"ipAddress": "111.222.333",
"productDetails": [
{
"product": {
"productId": "1234567",
"currencyCode": "USD",
"price": 2592,
"title": "Silver french door bottom freezer 27.4 cu ft refrigerator",
"imageuri": "https://example.com/images/refrigerators/silver/bottomfreezer/1234567.jpg"
},
"quantity": 1
}
],
"sourceId": "string",
"uri": "https://example.com/refrigerators/silver/bottomfreezer",
"cartId": "7f626d8c-4176-4f55-b17d-e89531e92ef7"
}
]'
```
### 5. Purchase complete and purchase signal
When the user completes the purchase, your site can generate an optional `orderId` along with the purchase complete signal using the `cartId` from the cart-add signal. After the purchase, the site must replace that `cartId` with a new UUID for the next cart-add signal.
```json expandable Purchase complete signal theme={"dark"}
curl --request POST \
--url https://SIGNAL_STORE_ID.applications.lucidworks.com/retail/signals/purchase \
--header 'Content-Type: application/json' \
--data '[
{
"timestamp": 1744305461169,
"visitorId": "abcc8e25-d30f-2212-9f60-9b45a71453cf",
"sessionId": "41a50e2-8e32-4674-5429-e1450dfdbb14",
"queryId": "2a97fc33-3335-3369-c2dc-456b775eb90f",
"ipAddress": "111.222.333",
"productDetails": [
{
"product": {
"productId": "1234567",
"currencyCode": "USD",
"price": 2592,
"title": "Silver french door bottom freezer 27.4 cu ft refrigerator",
"imageuri": "https://example.com/images/refrigerators/silver/bottomfreezer/1234567.jpg"
},
"quantity": 1
}
],
"sourceId": "string",
"uri": "https://example.com/refrigerators/silver/bottomfreezer",
"cartId": "7f626d8c-4176-4f55-b17d-e89531e92ef7",
"orderId": "6a678f11d-2265-4d01-a4ed-6e9875bfbda2",
"revenue": 2592
}
]'
```
## Additional signal information
### Facet refinement and query signal generation
When the user applies a facet, for example, the query parameter of "?facet=appliances", the signals beacon considers it to be a new query. Your site must determine if you want to generate a new `queryId` UUID and new query signal, or if you handle that interaction in a different way.
```json expandable Facet signal theme={"dark"}
curl --request POST \
--url https://SIGNAL_STORE_ID.applications.lucidworks.com/retail/signals/query \
--header 'Content-Type: application/json' \
--data '[
{
"timestamp": 1744305500000,
"visitorId": "abcc8e25-d30f-2212-9f60-9b45a71453cf",
"sessionId": "41a50e2-8e32-4674-5429-e1450dfdbb14",
"queryId": "3be88d22-1c74-44fd-b3a1-6d22078dfed4",
"ipAddress": "111.222.333",
"searchTerms": "silver french door bottom freezer refrigerator",
"productDetails": [
{
"product": {
"productId": "1234567",
"currencyCode": "USD",
"price": 2592,
"title": "Silver french door bottom freezer 27.4 cu ft refrigerator",
"imageuri": "https://example.com/images/refrigerators/silver/bottomfreezer/1234567.jpg"
},
"quantity": 1
}
],
"sourceId": "string",
"uri": "https://example.com/refrigerators/silver/bottomfreezer?facet=appliances"
}
]'
```
# Signals Beacon
Source: https://doc.lucidworks.com/docs/lw-platform/lw-analytics/signals/signals-beacon
[old doc.lw link]: https://doc.lucidworks.com/lw-platform/analytics/k649vy
[localhost link]: http://localhost:3000/docs/lw-platform/lw-analytics/signals/signals-beacon
[mintlify link]: https://doc.lucidworks.com/docs/lw-platform/lw-analytics/signals/signals-beacon
The signals beacon is an embeddable tracker that captures signals corresponding to user actions, like searches and purchases.
With no coding and minimal configuration, you can quickly start sending signals to Lucidworks Platform, including all of the data needed to power your signals analytics.
At a high level, the tracking flow begins when a user visits the site and performs one or more searches. These searches generate query identifiers that associate user actions with specific queries. As the user interacts with search results and shows interest in a product, additional signals such as clicks and cart events are captured. If the user completes a purchase, the flow concludes with a purchase signal that links the transaction back to the original query. This process enables end-to-end tracking of user behavior from search to purchase.
In this scenario, the signals beacon uses several primary tracking mechanisms:
* **Visitor ID**: A randomly generated identifier stored in local browser storage. It is unique to both the visitor and the device. It persists across sessions.
* **Session ID**: A randomly generated identifier stored in session storage. It expires after 40 minutes of inactivity or when the browser session ends. This duration is configurable using `sessionTimeoutThreshold`. When the user returns, a new session ID is generated, but the visitor ID remains the same.
* **Query ID**: A randomly generated identifier created when a search query is submitted. It is temporarily stored in memory and is included with signal events related to that query, such as clicks or purchases.
* **Cart ID**: A unique identifier created when a product is added to the cart. It links cart interactions and purchases to the originating query. After a purchase, a new cart ID is generated to track subsequent transactions independently.
The signal event flow below outlines a simplified sequence of common events. Additional signal types and behaviors are not included in this summary:
1. When a user visits the site, a visitor ID and session ID are generated.
2. When a query is submitted, a query ID is generated and a query signal is sent. This signal includes the visitor ID and session ID.
3. For each new query, a new query ID is generated and another query signal is sent.
4. When the user clicks on a product, a click signal is sent. This signal includes the associated query ID.
5. When the user adds a product to their cart, a cart ID is generated and a cart-add signal is sent.
6. When the user purchases a product, a purchase signal is sent. This signal includes the current cart ID, which allows the purchase to be traced back to the original query. After the signal is sent, the cart ID is replaced with a new one to track future transactions independently.
See the following image for a visual representation of this flow:
## How it works
The beacon sends signals to the:
* [Signals store](/docs/lw-platform/lw-analytics/signals/signals-store) for storage and aggregation
* [Analytics Studio service](/docs/lw-platform/lw-analytics/overview), which provides insights about actions taken by users on your site
The following signals are collected:
* **Query signals.** When a user performs a search, a signal is recorded.
* **Click signals.** When a user clicks on a product, a signal is recorded. One example of a click signal is when a user clicks a search result, which is a strong indicator the result is relevant.
* **Cart-add signals.** When a user clicks to add an item to the cart, a signal is recorded.
* **Purchase complete signals.** When a user completes a purchase and is taken to the order confirmation page, a signal is recorded.
* **Facet usage signals.** When a user interacts with the facets associated with the results, a signal is recorded.
For information about how the signals beacon is related to other signals and analytics functions, see the [Overview of signals process](/docs/lw-platform/lw-analytics/signals/overview).
## Prerequisites
You must be a workspace owner to configure the signals beacon.
Before you begin, complete the following tasks:
* Analyze your site to confirm compatibility with the signals beacon.
The signals beacon is compatible with most websites, but you may need to adjust the configuration to ensure it works as needed.
* Your site should use URL query parameters (`?q=foo`), not other methods of sending the query parameters. If it is not an option to use query parameters on your site, you can use the [Signals API](/docs/lw-platform/lw-analytics/signals/signals-api) to send signals.
- These fields should be surfaced in the page source, either as attributes or text nodes:
* Product ID
* Product title
* Product image URL
* Product price
* Product URL
* Identify any other fields that you want the signals beacon to capture.
You’ll need their CSS paths to configure the beacon.
Again, these can be either attributes or text nodes, as in these examples:
```html theme={"dark"}
12345
```
A field’s path might be different on different pages of your site.
The signals beacon supports multiple paths per field to accommodate this.
* Create the [Signals Store](/docs/lw-platform/lw-analytics/signals/signals-store) where the signals will be stored.
## Enable the beacon
To install the beacon, navigate to the megamenu and click **Signals**.
1. Click the Signals Store where you want to configure the signals beacon and then click **Integrations**.
2. If the beacon has never been configured for this app, you’ll see "No embed token found" on the Signals Using Beacon screen.
In that case, click the **Generate** button.
It can take a few moments to generate the token.
3. In the **Allowed Origins** field, enter the domains where you want to use the signals beacon. The format is `://[:]`. For example, `https://domain.YOUR_COMPANY_NAME.com`.
4. Set the **Signals Beacon** toggle to **On**.
5. Click **Copy** to copy the script and insert that script into the `` tag of your website’s HTML file.
This also enables the `window.logBeaconInfo()` function which you can use with your browser’s developer tools to inspect the fields captured by the beacon so that you can validate and refine your configuration.
See [Development and troubleshooting](#development-and-troubleshooting) below.
6. Edit the JSON configuration to process the signals you need for analytics.
For more details, see:
* [Example JSON configuration](#example-json-configuration)
* [Example signals script](#example-signals-script)
* [Field path configuration](#field-path-configuration)
* [Beacon Configuration Interface](/docs/lw-platform/lw-analytics/signals/beacon-configuration-interface) for detailed configuration property information
7. Click **Apply**. The configuration is applied to your website and it begins to send signals instantly based on that configuration.
You can edit the configuration at any time. When you edit the JSON configuration and click **Apply**, the changes are applied in real time and signals are sent and processed using that configuration until it is modified.
After the script is implemented and the signals are sent, you can access consolidated metrics.
The [Analytics Studio metrics](/docs/lw-platform/lw-analytics/analytics-screen) provide valuable customer behavior insights, which can be used to improve your business.
### Example JSON configuration
Use the following structure as an example to configure the beacon.
```json lines expandable wrap highlight={13,16,21} JSON configuration theme={"dark"}
{
"attributes": {
"productPath": ".lw-product-item",
"sessionTimeoutThreshold": 600000,
"cartId": "lwRetailCartId",
"idQueryParam": "pid",
"numberTypeFields": ["price", "quantity", "revenue"]
},
"fields": {
"productId": ":scope @id",
"imageuri": ":scope img @src",
"price": ":scope .search-entity-price span @text",
"title": ":scope .search-entity-title @text, :scope .search-generic-title @text",
"uri": ".config-url @href"
},
"query": {
"queryParam": "q",
"delayForTrackQueryUpdate": 1000
},
"click": {},
"cartAdd": {
"trigger": ".lw-cart-add-trigger"
},
"purchase": {
"trigger": ".lw-purchase-trigger"
}
}
```
* **Line 13**. You can use multiple comma-separated paths in cases where a field has different paths on different pages of your site. See the [Field path configuration](#field-path-configuration) section for details and examples.
* **Line 16.** Everything below this line is your signal configuration, where you list the signals you want to track and the fields you want to extract from them.
* **Line 21.** By default, these attributes inherit from the top-level attributes `[productPath|fields]`.
For information about:
* Attributes and fields used to extract product details from page elements, see the [Field path configuration](#field-path-configuration) section.
* The BeaconConfig interface, which includes detailed information about configuration properties, types, and default values, see [Beacon configuration interface](/docs/lw-platform/lw-analytics/signals/beacon-configuration-interface).
### Example signals script
The following example contains possible fields and default values, as well as comments to help you determine other options and values.
```json lines wrap expandable highlight={2-5,7,9,14-15,17,34,44} Signals script theme={"dark"}
{
"attributes": {
"data": "data-lw-product-data",
"productPath": ".lw-product-item",
"cartId": "lwRetailCartId",
"numberTypeFields": [price, quantity, revenue]
},
"fields": {
"productId": ":scope @id",
"title": ".title @text",
"imageuri": ".config-image @src",
"uri": ".url-wrapper a @href"
},
"query": {
"queryParam": "query",
"delayForTrackQueryUpdate": 1000,
"productPath": ".lw-product-item",
"fields": {
"productId": ":scope @id",
"title": ".title @text",
"imageuri": ".config-image @src",
"uri": ".url-wrapper a @href"
}
},
"click": {
"productPath": ".lw-product-item",
"fields": {
"productId": ":scope @id",
"title": ".title @text",
"imageuri": ".config-image @src",
"uri": ".url-wrapper a @href"
}
},
"cartAdd": {
"trigger": ".lw-cart-add-trigger",
"productPath": ".lw-product-item",
"fields": {
"productId": ":scope @id",
"title": ".title @text",
"imageuri": ".config-image @src",
"uri": ".url-wrapper a @href"
}
},
"purchase": {
"trigger": ".lw-purchase-trigger",
"productPath": ".lw-cart-item",
"fields": {
"productId": ":scope @cart-item-id",
"title": ".title @text",
"imageuri": ".config-image @src",
"uri": ".url-wrapper a @href"
}
}
}
```
* **Line 2**. Attributes provide a more advanced way to obtain product details using serialized JSON in an element attribute.
* **Line 3**. The CSS selector to target product elements. This is the global path inherited by each signal action if no custom path is provided at `[query/click/cartAdd/purchase].productPath`. The default is `.lw-product-item`.
* **Line 4**. The local storage key for the consumer page storing the cart UUID. The `cartId` value defaults to `lwRetailCartId`. The beacon obtains the value from `localStorage.lwRetailCartId` when attaching cart info to `/retail/signals/cartadd ` and `/retail/signals/purchase`.
* **Line 5**. This list specifies the fields treated as number type fields. The defaults can be `[price, quantity, revenue]`.
* **Line 7**. This is the main section to extract product field value from your site. For more information, see the Beacon element path section in this topic, which provides detailed configuration about extracting product information to ensure accurate and consistent data collection.
* **Line 9**. The `productId` field with a value of `scope@id` is the only required field to use the Lucidworks Retail Beacon API.
* **Line 14**. Specifies the URL parameter the beacon uses to monitor query updates. For example, with the default param `query`, if the current product page is filtered with the keyword `hat`, the URL parameter is http\://example.com?query=hat. The default value is `query`.
* **Line 15**. Specifies the delay, in milliseconds, for tracking element updates in the query. The default value is `1000`.
* **Line 17**. Custom overwrite of the fields configuration at the root level.
* **Line 34**. The CSS selector that targets page elements for triggering add-to-cart actions. This has full CSS support for targeting multiple elements with comma-separated selectors. For example, `button.add-to-bag, #AddToBagWithPrice`.
* **Line 44**. The CSS selector that targets page elements for triggering purchase or checkout actions. This has full CSS support for targeting multiple elements with comma-separated selectors. For example, `button.add-to-bag, #AddToBagWithPrice`.
### Field path configuration
The signals beacon extracts product details from page elements using CSS selectors and attributes.
You can capture any field that’s surfaced in the page’s DOM by adding it to your JSON configuration, under `"fields"`.
This section shows you how to configure your field paths, with examples.
The format is: `[Element CSS Selector] @[Value Attribute]`, as in the example below:
```json Configuration theme={"dark"}
"url": ".details .url @href"
```
```html HTML structure theme={"dark"}
...
```
```bash Data extracted theme={"dark"}
http://example.com/prd-123
```
### Special selectors
These special selectors and attributes are available:
* The `@text` pseudo-attribute captures the inner text node, as in these examples:
```json Configuration theme={"dark"}
"title": ".title @text"
```
```html HTML structure theme={"dark"}
Fancy Hat
```
```bash Data extracted theme={"dark"}
Fancy Hat
```
```json Configuration theme={"dark"}
"price": ".details .price @text"
```
```html HTML structure theme={"dark"}
...
```
```bash Data extracted theme={"dark"}
100
```
* The `:scope` CSS class targets product details defined on the root `productPath` element, as in this example:
```json Configuration theme={"dark"}
{
"attributes": {
"productPath": ".main-page-product",
},
"fields": {
"productId": ":scope @sku",
...
}
}
```
```html HTML structure theme={"dark"}
```
```bash Data extracted theme={"dark"}
prd-123
```
* The `textContent` selector targets elements by their text, as follows:
* Exact match: `[textContent='Add to cart!']`
* Contains: `[textContent*='cart']`
* Starts with: `[textContent^='add']`
* Ends with: `[textContent$='cart!']`
For example:
```json theme={"dark"}
"trigger": "button[textContent='Add to Cart'], .add-to-cart-container button"
```
* You can also use data attributes as selectors, as in this example where `data-testid` is a data attribute:
```json theme={"dark"}
"trigger": ".headerSearchBox, #hdrSrch, input[data-testid=global-search]"
```
### Page type configuration
The `pageTypes` configuration enables you to classify queries by page type based on URL patterns.
**Configuration:**
```json theme={"dark"}
"pageTypes": {
"search": ["*/search*"],
"category": ["*/category/*"],
"product": ["*/product/*"]
}
```
**Type:** `{ [pageType: string]: string[] }`
A mapping of page types to their corresponding URL patterns.
The `pageTypes` configuration enables annotation of query signals via an optional `pageType` property. By default, when no `pageType` is specified, query signals are treated as originating from the search page.
When a `pageType` is provided, its value is used for page-type matching. If a query does not match any configured patterns, its `pageType` is set to "OTHER".
Queries with a `pageType` other than null or "SEARCH" are excluded from the null query report. This is particularly useful for scenarios where queries originate from or redirect to non-search pages (e.g., category pages).
Explicitly configuring the "search" page type is recommended. Doing so ensures that any query signals from URLs that don't match a configured pattern are classified as "OTHER" and excluded from the null query report.
## Configure the beacon with Agent Studio or UI Studio
To use the signals beacon on the same page as Agent Studio or UI Studio, configure the beacon script with additional attributes.
Agent Studio and UI Studio both use the `` web component, which requires specific beacon configuration to work correctly.
Add the following attributes to the beacon script that you downloaded at **Analytics Studio** > **Integrations**:
* `signal-store-api-url`: API URL for your signals store.
* `signal-store-embed-token`: Embed token for the signals store.
* `beacon="{}"`: Initializes the beacon configuration.
* `initiator="AS"`: Identifies the script as being used with Agent Studio or UI Studio.
The following example shows the beacon script configured with those additional attributes:
```html {6-9} theme={"dark"}
```
Without these attributes, the beacon and `lw-template` components may conflict and fail to initialize.
## Configure the beacon for multiple pages of search results
You can configure the signals beacon if your site separates query results by tabs. For example, one tab for content and another tab for product information.
In the following example, each tab’s content path is separated by a comma under `"productPath"` with the `"fields"` property defining what is tracked for each tab using `:scope`.
```json Configuration theme={"dark"}
{
"attributes": {
"numberTypeFields": [
"price",
"quantity",
"revenue"
],
"productPath": ".cmp-content-search-result-list-item, .cmp-productlist__item"
},
"click": {
"trigger": "a, a h3, .button"
},
"fields": {
"imageuri": ":scope .cmp-content-search-result-list-item__img @src, :scope img @src",
"productId": ":scope .cmp-content-search-result-list-item_link @href, :scope .cmp-text .anchor @text",
"title": ":scope .cmp-content-search-result-list-item__details-name @text, :scope .cmp-text p @text",
"uri": ":scope .cmp-content-search-result-list-item_link @href"
},
"query": {
"queryParam": "search_query"
}
}
```
## Programmatic hooks
If your facets are stored somewhere that is inaccessible to the beacon, you can hook into beacon events to transform or suppress signals.
The `BeaconSignalHandlers` mechanism provides a way to modify, enhance, or suppress the signals that the beacon originally constructed before they are sent.
A signal handler can be defined for each of the signal types (`query`, `click`, `cartAdd`, and `purchase`).
Handlers should be called in the site’s code after the beacon is loaded. These functions only need to be set once.
The beacon configuration does not need to change if the intent of using this mechanism is to enhance query signals with facet information that is not available through the URL.
### Signal handler behavior
The signal handler can be used for the following:
* **Modify signals:** return the modified payload object to send an enhanced signal.
* **Suppress signals:** return false to prevent the signal from being sent.
* **Error handling**: if a runtime error occurs in the callback function, the original unmodified signal will be sent.
If the signal handler returns `undefined` or `null`, the unmodified signal will be sent.
The arguments to the `window.addBeaconListener` function should include these:
1. The signal type.
2. Callback function with one argument (signal payload).
### Examples of signal payload enhancement
Enhancing a `click` payload:
```js theme={"dark"}
window.addBeaconListener('click', (payload) => {
// Modify the payload as needed
payload.productDetails[0].product.productId = '123';
// Return the modified payload to send the signal
return payload;
});
```
Example of a `query` payload:
```js theme={"dark"}
window.addBeaconListener('query', (payload) => {
// Modify the payload as needed
payload.selected_facets = [
{
name: 'color',
values: ['blue', 'white'],
},
];
// Return the modified payload to send the signal
return payload;
});
```
### Example of conditional signal suppression
```js theme={"dark"}
window.addBeaconListener('query', (payload) => {
// Do not send query signals that contain a blocked search term
if (payload.searchTerms === 'blocked query') {
return false;
}
});
```
## Content Security Policy (CSP) requirements
If your site uses a Content Security Policy (CSP), you must add the following directives to allow the signals beacon to load and send data:
* **`script-src`**: Add `https://storage.googleapis.com` because the beacon script is served from Google Cloud Storage.
* **`connect-src`**: Add the Signal Store API host and, if using Agent Studio, the App Studio API host.
The following example shows the minimum required CSP directives:
```http theme={"dark"}
Content-Security-Policy:
script-src 'self' https://storage.googleapis.com;
connect-src 'self' YOUR_SIGNAL_STORE_API_URL YOUR_APP_STUDIO_API_URL;
```
Replace `YOUR_SIGNAL_STORE_API_URL` with the value in the `signal-store-api-url` attribute of your beacon script, and `YOUR_APP_STUDIO_API_URL` with the value in the `app-studio-api-url` attribute when using Agent Studio.
## Development and troubleshooting
The signals beacon includes the built-in `window.logBeaconInfo()` debugging utility that logs verbose information if the debug flag is enabled.
To enable the debug flag in your local storage, set `localStorage.enableBeaconDev = true`.
# Signals Store
Source: https://doc.lucidworks.com/docs/lw-platform/lw-analytics/signals/signals-store
[old doc.lw link]: https://doc.lucidworks.com/lw-platform/analytics/88pjp2
[localhost link]: http://localhost:3000/docs/lw-platform/lw-analytics/signals/signals-store
[mintlify link]: https://doc.lucidworks.com/docs/lw-platform/lw-analytics/signals/signals-store
The Signals Store is the repository in your Lucidworks Platform workspace that holds query, click, cart-add, purchase complete, and facet usage signals gathered from customer interactions on your site.
## Create a Signal Store
One of the tasks to use signals is to create a repository to store signals.
1. Navigate to the megamenu and click **Signals**.
2. Click **Create New**.
3. In the **Name** field, enter a unique user-defined for the Signals Store.
4. In the **Region** field, select the geographical area for the Signals Store.
This selection cannot be changed. If you need a Signals Store in a different region, you need to create another Signals Store with the other region specified.
5. Click **Create**. After the Signals Store is created, you can configure the [signals beacon](/docs/lw-platform/lw-analytics/signals/signals-beacon).
## Signal fields stored
If present in the signal, the information collected by the signals beacon and housed in the Signals Store for each signal is:
* Product ID
* Title
* Image URI
* Price
## Technical components
The properties of each Signals Store include:
* User-defined name.
* Universally unique identifier (`uuid`) of the Signals Store.
The `uuid` will be use to generate a unique Signals Store endpoint using the format of `.signals.lucidworks.com`.
For more information about signals, see [signals](/docs/lw-platform/lw-analytics/signals/overview).
# Core Settings
Source: https://doc.lucidworks.com/docs/lw-platform/lw-core/overview
[localhost link]: http://localhost:3000/docs/lw-platform/lw-core/overview
[mintlify link]: https://doc.lucidworks.com/docs/lw-platform/lw-core/overview
[old doc.lw link]: https://doc.lucidworks.com/lw-platform/core/fz448o
Core Settings empowers you to build fast, intuitive, and configurable search experiences without the need for complex coding or infrastructure.
With Core Settings, you can define and optimize the search experience with just a few clicks.
Each Site comes equipped with essential services to drive intelligent, responsive search, using an interactive UI for setup and tuning.
Use the preset defaults to quickly get your search experience up and running.
For customization, Core Settings offers controls to adjust performance and relevance.
The built-in development environment view gives full visibility into how changes impact search results before going live, so you can test and launch with confidence.
To get started, reach out to Lucidworks to request a new site, and we’ll create it for you.
## Use cases
Core Settings enable tailored search experiences for specific audiences across multiple websites.
A multinational ecommerce company can configure search to localize currency, language, and product availability based on regional shipping permissions.
Another example is when you have different ecommerce and support websites.
The ecommerce website can be set up to search the catalog and datasheets, with an embedded product-finding agent.
The support website can be set up to search the knowledge base and include an embedded troubleshooting agent.
Sites with multiple query profiles can manage different search configurations for Relevance and Boost settings.
Each query profile represents a distinct search experience with its own query pipeline and collection.
# Quickstart
Source: https://doc.lucidworks.com/docs/lw-platform/lw-core/quickstart
Build intuitive and configurable searches in 3 steps.
[localhost link]: http://localhost:3000/docs/lw-platform/lw-core/quickstart
[mintlify link]: https://doc.lucidworks.com/docs/lw-platform/lw-core/quickstart
This quickstart guide helps you set up and optimize your customer's search and browse experience to return the most relevant content for their queries.
Core settings default values are set based on informed industry standards when you first launch the application. From there, you can tune and evaluate your configuration to best fit your users.
It is essential that you consistently review results and settings in Commerce Studio and Analytics Studio and modify the Core Settings fields to optimize search results for your customers.
Use Neural Hybrid Search (NHS) to take advantage of advanced searching capabilities that combine lexical search and semantic search.
Click and drag the Neural Hybrid Search Weight slider to set the level of lexical versus semantic search. User behavior and record types will impact the weight setting.
To weight exact word entries higher, drag the slider toward **Lexical**. If your products use specific names, SKUs, or other fields such as model number, lexical search is more likely to generate the most relevant results. Lexical search is also more effective for identifying detailed information in specifications and other documents.
To weight results that interpret the meaning of the query, drag the slider more toward **Semantic**. Semantic search is more helpful if your users tend to enter different terminology for products or enter phrases or complete questions in their queries.
To determine the effectiveness of this setting, enter multiple searches and review where the query entry displays in the results. You can drag the slider in either direction and test again to determine the optimal setting.
Two settings determine the number of records returned:
* **Vector Similarity Threshold**: This threshold sets the minimum of how closely a document must match your query's meaning to be included in results. Enter a decimal value between `0` and `1`.
If you enter a value higher than `1`, an error displays and a rounded down value of `1` defaults in the field. If you enter a negative value less than `0`, an error displays and a rounded up value of `0` defaults in the field.
* A higher minimum value returns only results that strongly match the query's meaning. Assessing the quality of possible matches to return strong results may increase latency (processing response time).
* A lower minimum value returns more results because results do not have to match the query's meaning as closely.
* Results that are not semantically close enough to the query's meaning are filtered out before results are re-ranked.
* **Vector Result Count**: This value specifies how many candidate results the vector (semantic) search retrieves before the results are merged and re-ranked with the lexical results. Enter an integer between `0` and `2000`.
If you enter a value higher than `2000`, an error displays and a rounded down value of `2000` defaults in the field. If you enter a negative value less than `0`, an error displays and a rounded up value of `0` defaults in the field.
* A higher value assesses more semantic candidate results, which improves relevant results, but may increase latency (processing response time).
* A lower value assesses fewer semantic candidate results, which may decrease relevant results, but also takes less processing time.
These fields map to the `Min Return Vector Similarity` and `Number of Vector Results` fields in the Neural Hybrid Query stage settings in Fusion and Lucidworks Search.
For guidance about choosing starting values and tuning these settings for your use case, see the [Neural Hybrid Search section of the Relevance topic](/docs/lw-platform/lw-core/sites-relevance#neural-hybrid-search).
The [rules](/docs/lw-platform/lw-cs/cstudio-rules) and [query rewrites](/docs/lw-platform/lw-cs/cstudio-query-rewrite) created in [Commerce Studio](/docs/lw-platform/lw-cs/overview) determine which results display. If results are not as expected, Lucidworks recommends you review and modify the rules and rewrites in Commerce Studio and then adjust the slider.
Use the Sort options to display the search results ordered either by relevance or date. For more information, see [Relevance](/docs/lw-platform/lw-core/sites-relevance).
These options let you balance the order in which specific results display. You can adjust the display order of products and information by popularity or recency. For more information, see [Boost](/docs/lw-platform/lw-core/sites-boost).
Click and drag the Popularity Boost slider to boost real-time user information from signals. If your products and information are more stable and long-term, the popularity boost provides consistent results for user queries.
Click and drag the Recency Boost slider if you want to boost recently added or recently modified items or information. For example, if newer versions of products or more recent technical or compliance specifications are essential, the recency boost is more important.
You can prioritize certain fields such as SKU or title, if you assign and include them in the Attribute Priority section.
Preview the results to determine if the popular versus recent items display as expected, and make changes as needed. Reviewing results and then changing as needed is an ongoing process to obtain optimal results.
Lucidworks recommends that you consistently assess your site's analytics and determine where metrics could be improved. [Analytics Studio](/docs/lw-platform/lw-analytics/analytics-screen#types-of-metrics) provides signals graphic overviews for multiple metrics.
Important measures such as conversion rate, search conversion rate, and click-through rate help you determine the success of campaigns and other actions. And underperforming products and zero result search rate identify items that may need attention.
Modify the Core Settings fields and Commerce Studio features as needed to improve the search experiences of your customers and enhance your company's strategic goals.
On an ongoing basis, consistently review analytics and change the settings as needed.
# Boost
Source: https://doc.lucidworks.com/docs/lw-platform/lw-core/sites-boost
Core Settings
[localhost link]: http://localhost:3000/docs/lw-platform/lw-core/sites-boost
[mintlify link]: https://doc.lucidworks.com/docs/lw-platform/lw-core/sites-boost
[old doc.lw link]: https://doc.lucidworks.com/lw-platform/core/mcmjd1
Boosting improves the visibility of items in search results. You can control how items are boosted based on their popularity or newness, plus how specific fields in your data affect boosting.
## Query profiles
When your site has multiple query profiles, you can select which profile you want to configure using the query profile selector at the top of the Boost page:
Query profiles enable you to manage multiple search experiences within a single site.
Each query profile represents a distinct search configuration, including a query pipeline and collection.
The query profile selector only appears when your site has multiple query profiles configured.
If you have a single query profile, the selector is hidden and all settings apply to that profile automatically.
You can create and manage query profiles in [Lucidworks Search](/docs/lucidworks-search/05-move-data-out/query-profile/overview) or [self-hosted Fusion](/docs/5/fusion/getting-data-out/query-basics/query-pipelines/query-profiles), or by contacting Lucidworks.
If two query profiles point to the same query pipeline, changes made to one query profile's Core Settings will impact the other.
## Popularity Boost
Increase search result ranking using signals obtained from users, such as how many users clicked on an item, added it to their cart, or purchased it.
Using signals requires adding the [signals beacon](/docs/lw-platform/lw-analytics/signals/signals-beacon) to a website or using the [Signals API](/docs/lw-platform/lw-analytics/signals/signals-api) to send signals without the beacon.
Adding signals gives you a deeper understanding of user behavior and is useful when deciding how to configure and rank your search results.
Popularity Boost uses a discrete slider where you set the degree of how much you want the combined behavior of users to impact the search results.
The discrete slider has five settings that range from no boost to full boost.
When you increase how much to boost with signals, you give more weight to those signals when ranking search results.
More weight means items that users have interacted with more frequently appear higher.
This can improve crowd-based relevance, show trending products, and increase conversion rates.
There are, however, potential risks of setting the weight too high.
For example, products with lower visibility may be pushed down even further.
Giving a higher weight to signals can also decrease the impact of curated business rules if not balanced.
A best practice is to start with moderate boosting, then evaluate the impact and adjust as needed.
This protects important or promoted content you set using business rules or manual boosting.
## Recency Boost
You can increase search result ranking based on whether a document was recently added or modified.
There are cases when you want a newer item or page to be prioritized above older content as the more recently updated items are often what users are interested in.
The Recency Boost slider is where you set the degree of how important it is to show the newest content higher in the search results.
A best practice is to start with the moderate setting, then test and tune the weight as needed.
If the slider is set too high, you run the risk of surfacing less relevant but recent content over older, more helpful items.
If the slider is set too low, users looking for newer content might take longer to find it, causing frustration with outdated or irrelevant top results.
## Attribute Priority
You can decide if individual fields should be given priority for user queries.
Search attributes can be included or excluded using the dropdown options.
To see how these fields affect search results, include or exclude the fields relevant to your data and run some queries to analyze their impact.
The fields can also be arranged to assign their importance, so the field at the top of the list is given a greater significance than lower fields.
Matches on the higher fields are boosted more than matches on the lower ones, and matches on excluded fields don’t affect boosting at all.
This is useful when you want to prioritize queries that match the title of an item versus items that have similar words in their descriptions.
For example, a query for "handles" would show door handles higher than a cleaning product whose description says it "handles tough stains."
# Relevance
Source: https://doc.lucidworks.com/docs/lw-platform/lw-core/sites-relevance
Core Settings
[localhost link]: http://localhost:3000/docs/lw-platform/lw-core/sites-relevance
[mintlify link]: https://doc.lucidworks.com/docs/lw-platform/lw-core/sites-relevance
[old doc.lw link]: https://doc.lucidworks.com/lw-platform/core/juqsws
Relevance optimizes how well the search results match the user’s intent and query.
Increasing relevance means happier users through better findability of products and documents.
Relevant search results are important for:
* E-commerce sites, where relevant results drive conversions and revenue.
* Support sections of sites, such as help pages, where relevance reduces customer frustration and complaints.
* Enterprise search, where relevance boosts productivity and knowledge discovery by helping employees find what they need to perform their work or solve a problem.
## Query profiles
When your site has multiple query profiles, you can select which profile you want to configure using the query profile selector at the top of the Relevance page:
Query profiles enable you to manage multiple search experiences within a single site.
Each query profile represents a distinct search configuration, including a query pipeline and collection.
The query profile selector only appears when your site has multiple query profiles configured.
If you have a single query profile, the selector is hidden and all settings apply to that profile automatically.
You can create and manage query profiles in [Lucidworks Search](/docs/lucidworks-search/05-move-data-out/query-profile/overview) or [self-hosted Fusion](/docs/5/fusion/getting-data-out/query-basics/query-pipelines/query-profiles), or by contacting Lucidworks.
If two query profiles point to the same query pipeline, changes made to one query profile's Core Settings will impact the other.
## Neural Hybrid Search
Lucidworks Core Settings includes the option of using Neural Hybrid Search (NHS) to take advantage of advanced searching capabilities that combine lexical search and semantic search, enabling exact and partial keyword matching, phrase detection, and search results that understand the natural language users might enter in the search box. To improve the relevance of results, you can also set values to assess how closely results match and number of results returned.
Use the **Weight** slider to decide the relative importance of lexical and semantic search for your search results. As you increase the weighting of one type of search, the weight of the other type of search decreases.
Depending on the type of content you have, you can adjust the slider to accommodate the information you want the user to see.
For example, a legal, medical, or financial website might have industry-specific words that benefit from a more lexical approach, while an e-commerce website might want to show users similar products and will benefit from giving a greater weighting to semantic search.
NHS uses both of these to optimize the search experience.
**Fine tune the results**
To improve results, evaluate the specific needs of your use case to determine the appropriate balance between lexical and semantic search.
Begin testing by starting in the middle with 50% lexical and 50% semantic and perform a series of queries.
If the results do not match your expectations, then adjust the slider.
In a B2C context such as a grocery store chain, if a user searches for "ice cream" and the results prioritize "bag of ice" and "sour cream," the semantic weight should be increased. This helps return contextually relevant items like "waffle cones" and "chocolate syrup" that align with the shopper’s intent.
In contrast, on a financial services site where a user is researching "bonds," overly broad results like CDs, mutual funds, or ETFs may dilute relevance. Increasing the lexical weight ensures the top results remain focused on bonds.
Adjust the weighting strategy to match user intent and the goals of your business.
Adjust the weights for your specific use case, and validate that the search results become more relevant for the users.
Two settings determine the number of records returned:
* **Vector Similarity Threshold**: This threshold sets the minimum of how closely a document must match your query's meaning to be included in results. Enter a decimal value between `0` and `1`.
* A higher minimum value returns only results that strongly match the query's meaning. Assessing possible matches to return strong results may increase latency (processing response time).
* A lower minimum value returns more results because results do not have to match the query's meaning as closely.
* Results that are not semantically close enough to the query's meaning are filtered out before results are re-ranked.
* **Vector Result Count**: This value specifies how many candidate results the vector (semantic) search retrieves before the results are merged and re-ranked with the lexical results. Enter an integer between `0` and `2000`.
* A higher value assesses more semantic candidate results, which improves relevant results, but may increase latency (processing response time).
* A lower value assesses fewer semantic candidate results, which may decrease relevant results, but also takes less processing time.
These fields map to the `Min Return Vector Similarity` and `Number of Vector Results` fields in the Neural Hybrid Query stage settings.
**Recommended starting values by use case**
Use these starting values as a baseline, then adjust based on observed results.
B2B catalog search involves dense product catalogs with technical terminology.
Users expect both precision (correct part or product) and coverage (all relevant variants).
| Setting | Starting value |
| - | - |
| Vector Result Count | 75–150 |
| Vector Similarity Threshold | 0.60–0.70 |
This balanced configuration provides a candidate pool wide enough to capture product variants and related SKUs,
while filtering results with weak semantic relevance.
**Signs you should adjust:**
* If searches return too many unrelated items, raise the threshold.
* If variant products are missing from results, lower the threshold or increase the result count.
E-commerce search benefits from high precision.
Shoppers expect results that closely match their query, and irrelevant results reduce conversion.
| Setting | Starting value |
| - | - |
| Vector Result Count | 50–100 |
| Vector Similarity Threshold | 0.65–0.75 |
The higher threshold filters weakly related products before re-ranking.
A moderate result count keeps latency low while giving the re-ranker enough candidates to work with.
**Signs you should adjust:**
* If facet-filtered result sets are unexpectedly sparse, lower the threshold toward `0.60` or increase the result count.
* If results feel unrelated to the query, raise the threshold toward `0.75–0.80`.
Knowledge management and enterprise document search prioritize recall.
Users need to find the right document even when their wording differs from the document's language.
A threshold that is too strict hides relevant content.
| Setting | Starting value |
| - | - |
| Vector Result Count | 100–200 |
| Vector Similarity Threshold | 0.50–0.60 |
The larger candidate pool and lower threshold keep vector search broad enough to surface documents
that match query intent even when wording diverges significantly.
This is especially important for long-tail queries and specialized terminology.
**Signs you should adjust:**
* If results feel noisy or unrelated, raise the threshold in increments of `0.05` and re-evaluate.
* If users report missing documents, lower the threshold or increase the result count.
**Tuning these settings**
Adjust one setting at a time so you can identify what is causing changes in results.
1. **Start permissive.** Set Vector Result Count to `100–150` and Vector Similarity Threshold to `0.50`.
This gives you a broad candidate pool and avoids prematurely hiding relevant results.
2. **Adjust the threshold first.** The similarity threshold has the most direct impact on result quality.
Raise it gradually in increments of `0.05`, querying representative terms after each change.
3. **Tighten the result count last.** Once the threshold produces clean results, reduce Vector Result Count to improve response time.
Watch for result quality changes as you lower it — if results degrade, the count is too low for your collection size.
The underlying vector search algorithm (HNSW) is approximate.
A higher Vector Result Count gives the algorithm more candidates to find the best matches,
which is why starting high and narrowing down produces better outcomes than starting low.
**What to watch for**
* **Zero or very few results**: A threshold that is too high can exclude all vector candidates for some queries, causing NHS to fall back to lexical-only results.
A threshold of `0.80` or higher is rarely appropriate except for tightly controlled, well-structured catalogs.
If users report missing results, lower the threshold by `0.05–0.10` and test again.
* **Noisy or unrelated results**: If results don't feel related to the query, the threshold is likely too low.
Also check whether Vector Result Count is higher than needed — a large candidate pool with a low threshold gives re-ranking more noise to contend with.
* **Slow query response times**: Vector Result Count directly affects how many candidates are retrieved.
If query times are slow, reduce the result count in increments of `25`.
Note that if your collection spans multiple Solr shards, the effective candidate pool is Vector Result Count × shard count.
For advanced pipeline-level configuration of these parameters, see the links applicable to your product:
* Self-hosted Fusion clients, see [Choosing, implementing, and troubleshooting NHS stages](/docs/5/fusion/hybrid-search/hybrid-stage-differences) and [Semantic vector search test guidelines](/docs/5/fusion/hybrid-search/testing-svs).
* Lucidworks Search clients, see [Choosing, implementing, and troubleshooting NHS stages](/docs/lucidworks-search/11-vector-search/hybrid-stage-differences) and [Semantic vector search test guidelines](/docs/lucidworks-search/11-vector-search/testing-svs).
## Sort
Sort is how search results are ordered and can be set according to relevance, which is the degree to which each item matches a user’s query based on search settings.
You can change the default sorting order that is used for results by arranging the tiles in the Sort section.
This lets you sort first by relevance, then by date for items with identical relevance, or sort first by date, then by relevance for items with identical dates.
# Typeahead
Source: https://doc.lucidworks.com/docs/lw-platform/lw-core/sites-typeahead
Core Settings
[localhost link]: http://localhost:3000/docs/lw-platform/lw-core/sites-typeahead
[mintlify link]: https://doc.lucidworks.com/docs/lw-platform/lw-core/sites-typeahead
[old doc.lw link]: https://doc.lucidworks.com/lw-platform/core/1gkhuh
The Typeahead feature in Lucidworks Platform Core Settings provides real-time query and phrase suggestions as a user types into a search bar. This function helps users find relevant results faster by predicting what they are trying to search.
As the user enters text into the search box, the system predicts possible queries and displays instant suggestions in a dropdown menu. The suggestions refine dynamically when the user enters more characters. When the user clicks on the suggestion, the product or document displays.
Typeahead results returned by the API are sorted alphabetically. Sort options are not currently available.
Learn how to configure and manage Typeahead suggestions in Core Settings to help users find relevant results faster.
## How suggestions are created
Suggestions are created in two different ways:
* **Custom terms**. You can enter typeahead suggestions for products and documents. In addition to suggestions you want to include and display when a user searches, you can also enter suggestions to exclude that are not relevant. Excluding suggestions that are not relevant can enhance your user's search experience as much as including suggestions that steer the user to the products or knowledge they seek. A maximum of 4,000 custom terms (included and excluded) can be entered.
* **Suggested terms**. Every hour, Lucidworks Platform analyzes your customers' most frequent queries using signals from the [Signals Beacon](/docs/lw-platform/lw-analytics/signals/signals-beacon) or [Signals API](/docs/lw-platform/lw-analytics/signals/signals-api). The top 1,000 terms from the last 7 days are automatically added to your Included Typeahead Suggestions list.
Click your use case below to see examples of how typeahead suggestions can enhance the search experience:
For a plumbing company, examples of Included typeahead terms are:
* black bathroom faucet
* gold bathroom faucet
* return policy
* clearance
Examples of Excluded typeahead terms are:
* shower curtain
* bath towels
* shower slippers
For an auto parts manufacturer, examples of Included typeahead terms are:
* brake kits
* carburetor calibration manual
* OEM factory service manuals (FSMs)
Examples of Excluded typeahead terms are:
* single unit
* retail price
* body shops
For human resource knowledge management, examples of Included typeahead terms are:
* benefits guide
* company holidays
* sick leave policy
Examples of Excluded typeahead terms are:
* employee address list
* handbook
* policy
## View suggestions
The Typeahead screen displays tabs that contain Included and Excluded suggestions.
You can perform the following actions to review or configure how the suggestions display:
* To view suggestions that contain certain characters, enter those characters in the **Search term** field and press **Enter**.
* To display suggestions based on persistence status, user who last updated the suggestion, or the date the suggestion was created, click **Filters** and select or enter the criteria.
A suggestion marked as "persistent" means that terms generated hourly do not overwrite that suggested term.
All custom typeahead terms are persistent, and must be manually excluded or deleted to be removed from the Included table. To ensure that signal-based Typeahead terms are not replaced during the hourly refresh, toggle `Persistent` to `on`.
The suggestions you enter manually in the user interface are called "custom" terms, with a value of the user signed on in the **Last Edited (Updated) By** field.
The suggestions gleaned from customer queries are called "suggested" terms, with a value of **Lucidworks** in the **Last Edited (Updated) By** field.
* To set the options and columns that display on the Typehead screen, click **Table Settings** and select the toggle to turn the option on or off.
## Configure suggestions
This section details the steps to configure and manage typeahead suggestions.
### Configure Included suggestions
To configure a single Included typeahead suggestion, complete the following actions:
* To add an entry, access the Typeahead screen and click **Included > +Add**. Enter the term and press **Enter**. Then, click **Publish** to activate the changes on your site.
* To edit an existing entry, click the entry, enter the changes, and then, click **Publish** to activate the changes on your site.
All custom typeahead terms are persistent, and must be manually excluded or deleted to be removed from the Included table. To ensure that signal-based Typeahead terms are not replaced during the hourly refresh, toggle `Persistent` to `on`.
**Other actions to manage individual suggestions**
To discard edits to an existing entry, select the entry and click **Discard**.
You can delete a custom term you entered, or a suggested term generated from signals.
To delete an entry, hold the pointer over the entry and click the trash icon.
When you confirm the deletion, the entry displays in a soft delete status.
To finalize the deletion, click **Publish**.
If you do not click **Publish**, the soft delete status is removed , the entry is restored to the list.
To move a custom term you entered or a suggested term generated from signals from the **Included** list to the **Excluded** list, hold the pointer over the entry and click the **Exclude** icon.
If you move a suggested term to the Excluded list, the system will not generate that suggestion again, regardless of how many signals contain that term.
When you confirm the exclusion, the entry display in a soft exclude status.
To finalize the move to the Excluded list, click **Publish**.
If you do not click **Publish**, the soft exclude status is removed when you refresh the screen, and the entry is restored to the Included list.
To edit more than one Included typeahead suggestion at a time, complete the following steps:
In the Typeahead screen, hold the pointer over each entry you want to select and click the checkbox. To deselect all of the entries you have selected, click **Deselect**.
To select all of the entries, hold the pointer over the **Terms** header and select the checkbox. All of the suggestions are selected.
The following sections detail how to exclude, delete, or deselect the selected entries.
To move the selected entries from the **Included** list to the **Excluded** list, click **Exclude**.
If you move a suggested term to the Excluded list, the system will not generate that suggestion again, regardless of how many signals contain that term.
When you confirm the exclusion, the entries display in a soft exclude status.
To finalize the move to the Excluded list, click **Publish**.
If you do not click **Publish**, the soft exclude status is removed when you refresh the screen, and the entries are restored to the Included list.
To delete the selected entries, click **Delete**.
When you confirm the deletion, the entries display in a soft delete status.
To finalize the deletion, click **Publish**.
If you do not click **Publish**, the soft delete status is removed when you refresh the screen, and the entries are restored to the Included list.
### Configure Excluded suggestions
Excluding a term blocks the term and related terms from being suggested. For example, if you exclude "dog", the terms "dog house" and "dog food" are also excluded, but "doghouse" is not excluded.
You might want to exclude certain keywords from typeahead for several reasons:
* Frequently searched terms, such as "sale," might dominate typeahead if not excluded, reducing variety in suggestions.
* Filter out generic terms such as "info" or "product" that provide little user intent and might not help guide the user to meaningful content.
* Exclude typeahead suggestions that do not support business goals, such as low-value queries or terms that distract from high-intent, conversion-focused results.
To manage Excluded typeahead suggestions:
The only actions you can perform on the Excluded list are to add or delete entries.
You cannot move excluded items to the Included list. To change a term from Excluded to Included, you must delete it from the Excluded list and then add it to the Included list.
To configure a single Excluded typeahead suggestion, complete the following steps:
In the Typeahead screen, click **Excluded > +Add**. Enter the term and press **Enter**. Then click **Publish** to activate that term for use on your site.
**Other actions to manage individual suggestions**
To discard edits to an existing entry, select the entry and click **Discard**.
To delete an entry, hold the pointer over the entry and click the trash icon.
When you confirm the deletion, the entry displays in a soft delete status.
To finalize the deletion, click **Publish**.
If you do not click **Publish**, the soft delete status is removed when you refresh the screen, and the entry is restored to the list.
To edit more than one Excluded typeahead suggestion at a time, complete the following steps:
In the Typeahead screen, hold the pointer over each entry you want to select and click the checkbox. To deselect all of the entries you have selected, click **Deselect**.
To select all of the entries, hold the pointer over the **Terms** header and select the checkbox. All of the suggestions are selected.
The following section details how to delete the selected entries.
To delete multiple entries, hold the pointer over each entry, click the checkbox, and then click **Delete**.
When you confirm the deletion, the entries display in a soft delete status.
To finalize the deletion, click **Publish**.
If you do not click **Publish**, the soft delete status is removed when you refresh the screen, and the entries are restored to the list.
# Typeahead Service API
Source: https://doc.lucidworks.com/docs/lw-platform/lw-core/typeahead-api
The Lucidworks Typeahead Service API returns a list of suggestions created in Lucidworks Platform Core Settings. Using the unique `TYPEAHEAD_ID` for your Typeahead configuration, you can retrieve a list of suggestions based on the characters included in the request.
For detailed API specifications in Swagger/OpenAPI format, see [Platform APIs](/api-reference/get-typeahead-suggestions).
## Prerequisites
To use this API, you need:
* Typeahead suggestions entered or written into Lucidworks Platform Core Settings. A total of 5,000 suggestions are allowed in the following distribution:
* Your organization can enter a maximum of 4,000 custom terms (included and excluded) in the Core Settings Typeahead user interface.
* Every hour, a maximum of 1,000 of the terms entered most often in customer searches are added to the Included typeahead suggestions list.
* The unique `TYPEAHEAD_ID` for your typeahead implementation. Contact your Lucidworks Platform representative for this information.
## Unique values for the typeahead use case
Some query parameters unique to the `typeahead` use case can be used in the URL of the `GET` request of the Core Settings Typeahead Service API, including `prefix` and `limit`. Refer to the [API spec](/api-reference/get-typeahead-suggestions) for more information.
### Example GET request with `prefix` query parameter
```bash wrap Request theme={"dark"}
curl --request GET \
--url https://{TYPEAHEAD_ID}.typeahead.lucidworks.com/typeahead/suggestions?prefix=El \
--header 'Accept: application/json' \
--header 'Authorization: Embed EMBED_TOKEN' \
--header 'Content-type: application/json' \
--data ''
```
#### Response
The API call returns suggestions that begins with the letters `El`.
```json wrap theme={"dark"}
[
"Elastic",
"Electric",
"Electrical",
"Electronics",
"Elegant",
"Element",
"Elevator",
"Elite",
"Elixir",
"Ellipse"
]
```
## Result sorting
Typeahead results are sorted alphabetically. Sort options are not currently available with the API.
If the request does not include a query parameter after the `?` in the URL of the request, the complete list of suggestions is returned.
# AI Ranking Insights
Source: https://doc.lucidworks.com/docs/lw-platform/lw-cs/cs-ai-ranking-insights
AI Ranking Insights is a Commerce Studio feature that displays a clear, business-friendly explanation about why a product or content is ranked in a given position.
This feature uses a large language model (LLM) to interpret primary ranking drivers, such as text relevance, rules, boosts, rewrites, and AI-driven signals like Neural Hybrid Search (NHS), in Fusion's debug response.
## AI Ranking Insights example
To display the information, hold the pointer over a search result and select **Click Details**. In the Product Details section, click **AI Ranking Insights**.
The AI ranking insights feature identifies key relevancy influencers including merchandising rules, matching fields, and matching logic, and then returns the reasoning in a readable format so merchandisers and search managers can quickly identify the key factors influencing a search result's relevancy.
In this example, the AI ranking insight explains that the selected search result is ranked first because of a pin rule (listed in the **Execute** section of the Editor screen), and that the rule places the item above all other items. Merchandisers and other personnel would use this information to modify the factors of this and other search results, if needed, to rank other results higher than the bathroom faucet handle.
## Benefits
Merchandisers, search managers, and other personnel can use this concise explanation to:
* More quickly refine and optimize relevance without requiring expertise in Fusion configuration or search engineering.
* Reduce the requirement for technical expertise to decipher raw scores, weights, and pipeline calculations.
* Reduce reliance on engineering resources to troubleshoot relevance issues.
* Reduce the risk of time-consuming, trial-and-error rule and other configuration changes.
## How it works
This is the flow to display AI Ranking Insights for a product:
1. In Commerce Studio Editor, a user searches for a product. For example, bathroom faucet.
2. Lucidworks Platform captures the corresponding Fusion debug response in the Query Workbench.
3. The debug response, aware of the query context, rules, rewrites, and AI signals, is sent to an LLM.
4. The LLM returns a clear, concise explanation that takes the primary ranking factors into account:
* Interaction of related rules, boosts, and relevance
* Why the product displays in relation to the other products
## Technical requirements
Because AI Ranking Insights reads the Fusion debug response, you must configure the query pipeline to include debug and explain data in every Commerce Studio query.
Commerce Studio cannot pass query parameters per request, so this must be set at the pipeline level using following stage for the product you have deployed:
* [Fusion Additional Query Parameters stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/set-query-params-stage#query-pipeline-stage-condition-examples)
* [Lucidworks Search Additional Query Parameters](/docs/lucidworks-search/09-developer-documentation/config-specs/query-pipeline-stages/additional-query-parameters#query-pipeline-stage-condition-examples)
To enable debug output for Commerce Studio:
1. In the Fusion UI, open the query pipeline used by your Commerce Studio application.
2. In the stage's **Condition** field, add the following expression to scope debug output exclusively to Commerce Studio queries:
```js theme={"dark"}
request.hasParam("lw.em.queryType")
```
This ensures explain data is only added for Commerce Studio requests and not returned to other consumers of the same pipeline.
3. Add an **Additional Query Parameters** stage with the following parameter:
| Name | Value | Policy |
| - | - | - |
| `debug` | `true` | `default` |
Use `default` policy so the parameter is added only when not already set by the request.
Use `replace` if you need to force debug output regardless of incoming request parameters.
4. Save the changes.
**Debug response**
The debug response includes Solr's `explain` data, which shows how each document's score was calculated — including contributions from lexical scoring, vector similarity, boosts, and merchandising rules.
This is the data the LLM reads to generate the AI Ranking Insights explanation.
For details about reading Solr explain output, including how to find it in Query Workbench and how to interpret NHS scoring, see the information for your product:
* [Fusion Use Solr explain](/docs/5/fusion/hybrid-search/hybrid-stage-differences#use-solr-explain)
* [Lucidworks Search Use Solr explain](/docs/lucidworks-search/11-vector-search/hybrid-stage-differences#use-solr-explain)
# Migration guide
Source: https://doc.lucidworks.com/docs/lw-platform/lw-cs/cs-migrate
This guide explains the client tasks required to migrate from Predictive Merchandiser to Commerce Studio in Lucidworks Platform.
See [Commerce Studio versus Predictive Merchandiser](/docs/lw-platform/lw-cs/cstudio-versus-pm) for a feature comparison.
While Lucidworks will complete the majority of the migration, you will need to perform certain pre-migration and post-migration tasks.
The migration is one-way. The migration can be reverted back to Predictive Merchandiser, but any changes made in Commerce Studio are not reverted back into Predictive Merchandiser.
## Prepare for the migration
Complete these client tasks before Lucidworks begins the migration.
### Determine which rules and rewrites will be migrated
* **Unpublished** rules or rewrites have never been published.
* **Staged** rules or rewrites have been published at least once, but then were changed and those changes are not yet published. It also includes rules and rewrites that were deleted, but the deletion has not yet been published.
#### Unpublished rules
If you want unpublished rules migrated, then inform Lucidworks to set `excludeUnpublishedRules` to `false` during the migration. This is the default option. If you do not want unpublished rules to be migrated, tell Lucidworks to set `excludeUnpublishedRules` to `true`.
#### Unpublished rewrites
If you want unpublished rewrites migrated, then inform Lucidworks to set `excludeUnpublishedRewrites` to `false` during the migration. This is the default option. If you do not want unpublished rules to be migrated, tell Lucidworks to set `excludeUnpublishedRewrites` to `true`.
#### Publish only approved content
By default, migrated rules and rewrites are published in Commerce Studio if they are published in Predictive Merchandiser.
If you want only approved rewrites and rules to be migrated as published, then inform Lucidworks to set `publishApprovedOnly` to `true` during the migration.
#### Skip rules or rewrites
If you want to skip all rules, then inform Lucidworks to set `skipRules` or `skipRewrites` to `true` during the migration.
#### Staged rules and rewrites
Staged rules and rewrites are skipped in the migration, so you must publish or revert all staged changes to rules and rewrites.
You can only view staged changes in Predictive Merchandiser that belong to your user.
To check for staged changes that belong to other users in your organization, do the following:
1. Copy the `precheck.sh` script below to check for staged (unpublished) rules and rewrites:
```bash expandable theme={"dark"}
#!/bin/bash
set -e
# Function to perform curl requests with error handling and retries
perform_curl_request() {
local url=$1
local max_retries=3
local retry_count=0
local wait_time=5
local curl_opts="-s -w '\n%{http_code}' -H 'Accept: application/json' -u '$FUSION_USERNAME:$FUSION_PASSWORD'"
while [ $retry_count -le $max_retries ]; do
# Execute curl and capture response with status code
local response=$(eval "curl $curl_opts '$url'")
local status_code=$(echo "$response" | tail -n1)
local body=$(echo "$response" | sed '$d')
# Check if status code is in 2xx range
if [[ $status_code =~ ^2[0-9][0-9]$ ]]; then
echo "$body"
return 0
else
# Report the error
echo "Error: HTTP request failed with status code $status_code" >&2
echo "URL: $url" >&2
# If it's a 4xx error, don't retry
if [[ $status_code =~ ^4[0-9][0-9]$ ]]; then
echo "Client error. Not retrying." >&2
exit 1
fi
# For other errors (5xx), retry if we haven't reached max retries
if [ $retry_count -lt $max_retries ]; then
retry_count=$((retry_count + 1))
echo "Retrying in $wait_time seconds... (Attempt $retry_count of $max_retries)" >&2
sleep $wait_time
else
echo "Maximum retries reached. Giving up." >&2
exit 1
fi
fi
done
exit 1
}
# Function to validate JSON blob rules
validate_json_blob_rule() {
local doc="$1"
local doc_id="$2"
local json_blob_s=$(echo "$doc" | jq -r '.json_blob_s // empty')
# Guard: Exit early if no json_blob_s field
[ -z "$json_blob_s" ] && return 0
# Decode base64 and validate JSON structure
local decoded_json=$(echo "$json_blob_s" | base64 -d 2>/dev/null || echo "")
# Guard: Exit early if decoding failed
[ -z "$decoded_json" ] && return 0
# Check if the decoded JSON is valid
if ! echo "$decoded_json" | jq empty 2>/dev/null; then
# Invalid JSON in blob
bad_json_blob_rule_ids+=("$doc_id")
echo "found JSON blob rule with invalid JSON: $doc_id"
# Add this entry to our JSON structure for bad JSON blobs
local content_entry=$(jq -n --arg id "$doc_id" \
--argjson content "$doc" \
--arg decoded_content "$decoded_json" \
'{
"id": $id,
"ruleContent": $content,
"decodedJsonBlob": $decoded_content,
"error": "Invalid JSON syntax"
}')
# Add this entry to the second problem (index 1) in our JSON structure
json_output=$(echo "$json_output" | jq --argjson entry "$content_entry" '.problems[1].content += [$entry]')
return 0
fi
# Valid JSON - now check if it's multiple objects
# A simple heuristic: if we can't parse it as a single JSON value, it's likely multiple objects
if echo "$decoded_json" | jq -e 'type' >/dev/null 2>&1; then
# Guard: Exit early if it's a valid single JSON object
return 0
fi
# This means jq couldn't parse it as a single value, likely multiple objects
bad_json_blob_rule_ids+=("$doc_id")
echo "found JSON blob rule with multiple objects: $doc_id"
# Add this entry to our JSON structure for bad JSON blobs
local content_entry=$(jq -n --arg id "$doc_id" \
--argjson content "$doc" \
--arg decoded_content "$decoded_json" \
'{
"id": $id,
"ruleContent": $content,
"decodedJsonBlob": $decoded_content
}')
# Add this entry to the second problem (index 1) in our JSON structure
json_output=$(echo "$json_output" | jq --argjson entry "$content_entry" '.problems[1].content += [$entry]')
}
# Function to check for staged published rules
check_staged_published_rule() {
local doc="$1"
local doc_id="$2"
local deployed="$3"
# Guard: Exit early if rule is deployed (not staged)
[ "$deployed" != "false" ] && return 0
# Check if the doc exists in the live rewrites collection
local live_url="$FUSION_BASE_URL/api/apps/$FUSION_APP_ID/query-rewrite/search?q=id:$doc_id&isLive=true&doc_type=rule&fl&rows=1&start=0"
echo "fetching live rule via $live_url"
local live_response=$(perform_curl_request "$live_url")
# Check if we have any rules in the live response array
local live_docs_count=$(echo "$live_response" | jq -r '.response.docs | length')
# Guard: Exit early if doc doesn't exist in live collection
[ "$live_docs_count" -eq 0 ] && return 0
# Found a published rule with staged changes
staged_published_rule_ids+=("$doc_id")
echo "found published rule with staged changes: $doc_id"
# Extract the live doc from the response
local live_doc=$(echo "$live_response" | jq '.response.docs[0]')
# Create a new entry for this rule with both staging and live content
local content_entry=$(jq -n --arg id "$doc_id" \
--argjson staging "$doc" \
--argjson live "$live_doc" \
'{
"id": $id,
"stagingContent": $staging,
"liveContent": $live
}')
# Add this entry to our JSON structure
json_output=$(echo "$json_output" | jq --argjson entry "$content_entry" '.problems[0].content += [$entry]')
}
# Function to display help message
show_help() {
echo "Usage: $0 [output_file]"
echo "Example: $0 admin password https://fusion.example.com MyFusionApp precheck_results.json"
echo "This script performs pre-checks for Fusion content before migration."
echo "Arguments:"
echo " fusion_username Fusion username"
echo " fusion_password Fusion password"
echo " fusion_base_url Base URL of the Fusion instance"
echo " fusion_app_id Application ID of the Fusion app to check"
echo " output_file (Optional) Output file to save results (default: precheck_results.json)"
}
# Check if required arguments are provided
if [ "$#" -lt 4 ]; then
show_help
exit 1
fi
FUSION_USERNAME=$1
FUSION_PASSWORD=$2
FUSION_BASE_URL=$3
FUSION_APP_ID=$4
OUTPUT_FILE=${5:-precheck_results.json}
# Input validation
if [ -z "$FUSION_USERNAME" ]; then
echo "Error: fusion_username is required"
show_help
exit 1
fi
if [ -z "$FUSION_PASSWORD" ]; then
echo "Error: fusion_password is required"
show_help
exit 1
fi
if [ -z "$FUSION_BASE_URL" ]; then
echo "Error: fusion_base_url is required"
show_help
exit 1
fi
if [ -z "$FUSION_APP_ID" ]; then
echo "Error: fusion_app_id ID is required"
show_help
exit 1
fi
# Check if jq is installed
if ! command -v jq &> /dev/null; then
echo "Error: jq command line tool is required but not installed. Please install jq and try again: https://jqlang.org/"
exit 1
fi
# Check if output file exists
if [ -f "$OUTPUT_FILE" ]; then
read -p "File $OUTPUT_FILE already exists. Overwrite? (y/n): " -n 1 -r
echo
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
echo "Operation cancelled."
exit 1
fi
fi
# Ensure the output file is writeable
if ! touch "$OUTPUT_FILE" &> /dev/null; then
echo "Error: Output file $OUTPUT_FILE is not writeable. Check output path / permissions and try again."
exit 1
fi
# Initialize JSON structure
json_output='{
"problems": [
{
"problem": "Published rule with staged changes will be skipped during migration",
"fix": "Publish or revert the changes, or manually re-create the skipped content after migrating",
"content": []
},
{
"problem": "JSON blob rule contains multiple JSON objects instead of a single object",
"fix": "Update the JSON blob rule to contain only a single JSON object at the top level",
"content": []
}
]
}'
# Initialize array to store staged published rule IDs
declare -a staged_published_rule_ids
# Initialize array to store bad JSON blob rule IDs
declare -a bad_json_blob_rule_ids
# Fetch query rewrite instances with pagination
start=0
rows=10
processed_docs=0
while true; do
page_url="$FUSION_BASE_URL/api/apps/$FUSION_APP_ID/query-rewrite/instances?sort=id+asc&doc_type=rule&rows=$rows&editSessionId=*&start=$start"
echo "fetching next page of rules via $page_url"
response=$(perform_curl_request "$page_url")
# Check if we got an empty array, which means we're done paginating
array_length=$(echo "$response" | jq -r '. | length')
if [ "$array_length" -eq 0 ]; then
break
fi
echo "Retrieved $array_length rules at offset $start"
# Process each doc in the array
while read -r doc; do
doc_id=$(echo "$doc" | jq -r '.id')
deployed=$(echo "$doc" | jq -r '.deployed')
doc_type=$(echo "$doc" | jq -r '.type // empty')
# Check for bad JSON blob rules
if [ "$doc_type" = "json_blob" ]; then
validate_json_blob_rule "$doc" "$doc_id"
fi
# Check for staged published rules
check_staged_published_rule "$doc" "$doc_id" "$deployed"
processed_docs=$((processed_docs + 1))
done < <(echo "$response" | jq -c '.[]')
# Move to the next page
start=$((start + rows))
done
# Write the final JSON output to file
echo "$json_output" | jq '.' > "$OUTPUT_FILE"
# Output results
total_staged_issues=0
total_json_blob_issues=0
if [ -n "${staged_published_rule_ids[*]:-}" ]; then
total_staged_issues=${#staged_published_rule_ids[@]}
fi
if [ -n "${bad_json_blob_rule_ids[*]:-}" ]; then
total_json_blob_issues=${#bad_json_blob_rule_ids[@]}
fi
total_issues=$((total_staged_issues + total_json_blob_issues))
# Guard: Exit early if no issues found
if [ $total_issues -eq 0 ]; then
echo "All pre-checks passed. No issues found."
echo "Processed $processed_docs rules."
exit 0
fi
# Report issues found
echo "The following pre-checks failed:"
# Report staged published rule issues
if [ $total_staged_issues -gt 0 ]; then
echo " Found published rules with staged changes. Publish or revert the changes to fix this issue."
# Display up to 10 IDs
display_count=$([ $total_staged_issues -gt 10 ] && echo 10 || echo $total_staged_issues)
for ((i=0; i
| Placeholder | Description |
| - | - |
| `FUSION_USERNAME` | Your Fusion username. |
| `FUSION_PASSWORD` | The password for your Fusion user. |
| `FUSION_BASE_URL` | The URL of your Fusion instance including `https://`. For example, `https://FUSION_HOST.com`. |
| `FUSION_APP_ID` | The application ID of the Fusion app that contains your Predictive Merchandiser. This is the value of the `id` field, and may or may not match the `name` of the Fusion app. To obtain the ID, sign in to your Fusion instance. For example, `FUSION_INSTANCE.com`. Then also open a different browser window and enter `https://FUSION_INSTANCE.com/api/apps`. The `id` and `name` display for each valid Fusion app. The value in the `id` field is the value that needs to be used in this field. |
If you receive an error stating `bash: ./precheck.sh: Permission denied`: 1. Make the script
executable: `chmod +x precheck.sh`. 2. Run the script again.
3. Check the output to see if any issues were detected. If there are no issues, the output returns `All pre-checks passed. No issues found.` Continue to [retrieve list of objects in Predictive Merchandiser](#retrieve-lists-of-objects-in-predictive-merchandiser). If there are issues, the output returns a list of issues like the following:
```
The following pre-checks failed:
Found published rules with staged changes. Publish or revert the changes to fix this issue.
- 2fRtM1GEsU
- E1YkUTX9r6
- Eb2A6WARag
- FNS7Whciwx
- LlIsMfK4kq
- OidpIUrVv3
- P39TDQ2X1K
- RYbY2XPMOI
- ZHvW2LiQpe
- aCseNzc4Xc
... and 6 more.
See precheck_results.json for the full list of invalid content
```
4. Review `precheck_results.json` and inspect each reported rule. If you created the rules, sign into Predictive Merchandiser and publish or revert the staged changes. If the unpublished rules were created by a different user, you cannot publish or revert the rules in the Predictive Merchandiser UI. There are a few options for resolving unpublished rules owned by another user:
* After you complete the pre-migration tasks, tell Lucidworks to proceed with the migration. These rules are skipped and you can re-create them in Commerce Studio.
* Use the Fusion [Query Rewrite](/api-reference/query-rewrite-api/get-the-query-rewrite-status) and [Custom Rule](/api-reference/custom-rule-type-api/get-a-custom-rule-type) APIs to manually publish or revert the unpublished rules.
* Use the Fusion [Query Rewrite](/api-reference/query-rewrite-api/get-the-query-rewrite-status) and [Custom Rule](/api-reference/custom-rule-type-api/get-a-custom-rule-type) APIs, or Solr, to manually change the edit session ID of the content in Solr to your current user. Then, manually fix the content in the Predictive Merchandiser UI.
* Have the user that owns the content in Predictive Merchandiser sign in and either publish or revert the changes.
5. After resolving issues, re-run the script. If there are no issues, continue to [retrieve list of objects in Predictive Merchandiser](#retrieve-lists-of-objects-in-predictive-merchandiser).
You can run the precheck.sh script at any time, even after a migration if desired. It makes no
changes to contents and works even if Commerce Studio is enabled.
After preparing Predictive Merchandiser for migration, Lucidworks recommends that you export lists of rules, rewrites, templates, and zones from your Predictive Merchandiser instance to compare against Commerce Studio later. This will help serve as a reference when performing your post-migration validation checks.
1. Export a list of your rules and rewrites:
```bash theme={"dark"}
curl --request GET \
--u USER:PASSWORD \
--url 'https://FUSION_HOST/api/apps/APP_NAME/query-rewrite/instances' \
--header 'accept: application/json' \
> exported-rules.json
```
2. Create an inventory of your rules, noting how many of each type and tag appear in the list.
3. Export a list of your zones:
```bash theme={"dark"}
curl --request GET \
-u USER:PASSWORD \
--url 'https://FUSION_HOST/api/templating/zones?context=app:APP_NAME' \
--header 'Accept: application/json' \
> zones.json
```
4. Export a list of your templates:
```bash theme={"dark"}
curl --request GET \
-u USER:PASSWORD \
--url 'https://FUSION_HOST/api/templating/templates?context=app:APP_NAME' \
--header 'Accept: application/json' \
> templates.json
```
Notify Lucidworks to begin the migration, but keep in mind that once the migration begins, you will no longer be able to access Predictive Merchandiser.
Lucidworks will set up the Commerce Studio integration and instance, and migrate your data from Predictive Merchandiser to Commerce Studio.
Do not proceed with subsequent procedures until Lucidworks confirms the migration is successful.
## Post-migration tasks
Complete these client tasks when Lucidworks notifies you the migration is complete.
Lucidworks will run the validation checks in this section, but you must also validate that all Predictive Merchandiser objects appear in Commerce Studio.
Because you can no longer access Predictive Merchandiser, you must consult the [exported JSON](#retrieve-lists-of-objects-in-predictive-merchandiser) you generated in **Retrieve lists of objects in Predictive Merchandiser** for validation checks.
1. Sign in to [Lucidworks Platform](https://platform.lucidworks.com) as a workspace owner and navigate to **Commerce Studio**.
2. Navigate to **Rules**, then compare your rule count in Commerce Studio against the JSON you [exported from Predictive Merchandiser](#retrieve-lists-of-objects-in-predictive-merchandiser).
3. Navigate to **Rewrite**, then validate that your query rewrites in Commerce Studio match the exported JSON.
4. Navigate to **Settings**, then validate your templates have been correctly migrated.
5. Open each page, then validate your Predictive Merchandiser zones have become sections in Commerce Studio, and they are associated with the correct page.
6. Navigate to the **Editor** and check that rules and rewrites are applied as expected. Some rules and rewrites only fire under specific conditions:
1. For rules that fire under specific templates, change your page to match the required template.
2. For rules that fire only when specific tags are applied, verify the tags are displayed on the Rules page for the rule.
7. Perform additional rules and rewrites validations as needed, using the exported rules JSON as a reference. For example, check that the direction of synonym rewrites matches the expected direction.
If you encounter any issues, contact your Lucidworks representative.
Web traffic continues to use rules and rewrites from Predictive Merchandiser. Changes made in Commerce Studio do not affect live traffic. Contact Lucidworks when you complete the validation tasks.
## Revert to Predictive Merchandiser
If you want to revert back to using Predictive Merchandiser, inform Lucidworks. Then Lucidworks must delete the Commerce Studio instance.
If the Commerce Studio instance is deleted, changes you made to Commerce Studio after the initial migration from Predictive Merchandiser are *not* populated back into Predictive Merchandiser.
When the Commerce Studio instance is deleted, the system should automatically revert the Fusion app back to Predictive Merchandiser. This restores access to Predictive Merchandiser and routes live traffic back to its rules and rewrites, if traffic had previously been switched to Commerce Studio.
## Frequently asked questions
**Can I revert to Predictive Merchandiser after switching to Commerce Studio?**
Yes. However, any changes made in Commerce Studio after switching will *not* be reverted back into Predictive Merchandiser.
**Does switching to Commerce Studio immediately affect live web traffic?**
No. After a successful migration, web traffic continues to use rules and rewrites from Predictive Merchandiser.
**What happens if I forget to publish changes in Predictive Merchandiser before migrating?**
If you fail to publish pending rules and rewrites before starting the migration, the migration may fail or result in incomplete or inaccurate data. All changes must be published before proceeding.
## Troubleshooting
This section provides troubleshooting steps for resolving issues with the Commerce Studio [Editor](/docs/lw-platform/lw-cs/cstudio-editor) when integrated with Fusion. Use this guidance to debug unexpected behavior that may be caused by Fusion query pipeline or query profile configuration.
This content is intended for users with experience in Fusion and search engineering.
### Symptoms
You may be experiencing a configuration issue if you observe any of the following:
* No results or facets returned in the Editor
* Error messages when performing searches
* Rules not firing or not appearing after creation
* Rewrites not triggering or not affecting live traffic
* Mismatch between facets and results list
* “Targeted documents” column in the rules table is empty or incorrect
* No field value suggestions when configuring rule conditions
* Missing stage data in the Ranking Factors UI
### General troubleshooting steps
1. Open your browser’s developer tools and go to the Network tab.
1. Retry the failing action.
2. Look for requests with a non-200 status or check `render` endpoint responses for `errorMessage` fields.
3. Use the `fusionRequestUrl` to replicate the request in Fusion Query Workbench.
2. Verify that the query pipeline behaves as expected in Fusion Query Workbench.
1. Confirm there are no errors.
2. Verify expected results are returned.
3. In the **JSON view**, check for `fusion.applicable_rules` or `fusion.tagger` values.
3. Confirm that rules and rewrites are saved to the correct Solr collections.
1. In Solr Admin, check `APP_NAME_query_rewrite_staging_em` and `APP_NAME_query_rewrite_em`.
2. Use the rule ID to query and confirm the entry exists.
1) Commerce Studio exclusively manages `APP_NAME_query_rewrite_staging_em` and `APP_NAME_query_rewrite_em`. Do not manually add, edit, or delete documents in these collections.
2) Commerce Studio reflects rules stored in its own platform database. Deleting documents from these Fusion Solr collections does not delete the corresponding rules in Commerce Studio. The collections will re-sync the next time Commerce Studio writes rules.
3) Custom tooling built against these collections is not supported.
4. Ensure the staging collection has only one replica.
1. If multiple replicas exist, set `shards.preference=replica.leader` in the **Apply Rules** and **Text Tagger** stage parameters.
5. Verify that the pipeline supports the `fl` parameter.
1. Do not overwrite `fl`. Add required fields only when needed.
2. Do not use `fl=*`.
### Required pipeline configuration
Your Fusion query pipeline must include the following stages in this order:
1. **Text Tagger stage** with a blank value in the **Tagger Collection** field
2. **Apply Rules stage** with a blank value in the **Collection** field
3. **Solr Query stage**
4. **Modify Response with Rules stage**
5. **Response Diagnostics stage**
In addition, the pipeline must:
* Use the `edismax` query parser (`defType=edismax`)
* Preserve the following query parameters: `tags`, `lw.rules.simulate`, `lw.app.emStatus`, `context`, `lw.tagger.debug`, `lw.rules.debug`, `lw.em.staging`
* Preserve required response fields: `response.docs`, `response.numFound`, `fusion.tagger`, `fusion.applicable_rules`, `fusion.applicable_stages`, `response.facet_counts`, `responseHeader.params`
### Feature-specific checks
#### Targeted document lookup
* The Fusion app and query profile must have the same name.
* The query profile must support `q=**:**` and standard `fq` parameters.
* Avoid using grouping, collapse/expand, or custom logic that may interfere with standard filter behavior.
#### Field value suggestions
* The Solr collection must match the Fusion app name.
* The collection must support the `/terms` request handler with standard parameters.
* Suggestions will fail if `/terms` is missing or misconfigured.
#### Rewrites and rules interaction
* If the **Text Tagger** stage appears before the **Apply Rules** stage, rewrites apply first.
* If the **Apply Rules** stage appears before the **Text Tagger** stage, only the original terms are used for rule evaluation.
* Rules must target the rewritten query term, not the original term.
#### Grouping via collapse/expand
* The query pipeline must set an `fq` with the collapse field specified, indicating what field identifies members of a group. For example, set `fq={!collapse field=product_id_s}` when `expand=true` is passed as a query param.
* Set the following query parameters:
```bash theme={"dark"}
expand=true
expand.rows=5
```
* Relevant response fields include the following:
```bash theme={"dark"}
expanded
response.docs
```
#### Query Workbench debugging tips
* Use the correct query profile and associated pipeline.
* Set the following query parameters:
```bash theme={"dark"}
lw.rules.debug=1
lw.tagger.debug=1
lw.em.staging=1=_query_rewrite_staging_em
lw.app.emStatus=CONNECTED
queryProfileID=
tags=
lw.rules.target_segment=
```
* Use **View As: JSON** to verify `fusion.applicable_rules` and the final query string.
* Use custom JavaScript stages to inspect `ctx` or log intermediate values.
### Known configuration issues
* Hardcoded `tags` in pipelines may prevent rules from matching if users forget to apply corresponding tags in the Editor.
* Query pipelines that use grouping may break document targeting. Only collapse/expand-based grouping is supported. Grouping via Solr `group` params or blockjoin query parser is not supported.
* If a boost stage and a rule both adjust relevancy, only one may take effect, depending on rule priority and creation date.
# Rules and Rewrites API
Source: https://doc.lucidworks.com/docs/lw-platform/lw-cs/cs-rules-rewrites-api
The Rules and Rewrites API for Commerce Studio enables you to create and manage merchandising [rules](/docs/lw-platform/lw-cs/cstudio-rules) and [query rewrites](/docs/lw-platform/lw-cs/cstudio-query-rewrite) programmatically.
This API supports enterprise automation workflows, cross-environment synchronization, and integration with internal tools.
For complete details about all of its endpoints, methods, and parameters, see the [Rules and Rewrites API reference](/api-reference/rule-management/list-rules).
## When to use this API
The Rules and Rewrites API enables the following use cases:
* **Automate rule deployment**: Create and update rules and rewrites through CI/CD (continuous integration/continuous deployment) pipelines or automated workflows.
* **Synchronize across environments**: Maintain consistent rules and rewrites across development, QA (quality assurance), and production environments.
* **Manage rules at scale**: Efficiently handle large rule sets that are impractical to manage through the UI (user interface) alone.
* **Fallback with version control**: Fetch rules and rewrites, then commit them to your source control repository at intervals so you can easily revert to earlier versions if needed.
* **Integrate with AI agents**: Use AI to automate creation and updates of rules and rewrites.
For visual rule creation and management, use the Commerce Studio UI.
The API is designed for programmatic workflows where automation and integration are required.
## Key concepts
### Rule and rewrite identifiers
Every rule and rewrite has two ID fields that you can use interchangeably in API operations:
* **`id`**: Automatically generated UUID (Universally Unique Identifier) assigned by the platform when you create a rule or rewrite.
This identifier is unique within the Commerce Studio instance.
* **`externalId`**: Optional user-defined identifier that you can assign when creating a rule or rewrite.
This field enables you to maintain your own naming conventions and synchronize rules across multiple environments.
Use external IDs to maintain stable references across environments.
For example, assign an external ID like `promo-summer-2026` to ensure the same rule can be identified and updated consistently in development, QA, and production.
External IDs must be unique within a Commerce Studio instance.
The API prevents duplicate external IDs to maintain rule integrity.
### Authentication scopes
The API uses three separate authentication scopes:
* `commercestudio.rules`: Required for [rule](/docs/lw-platform/lw-cs/cstudio-rules) operations (create, read, update, delete rules)
* `commercestudio.rewrites`: Required for [query rewrite](/docs/lw-platform/lw-cs/cstudio-query-rewrite) operations (create, read, update, delete rewrites)
* `commercestudio.bulkoperations`: Required for [bulk operations](#bulk-operations) (publish, unpublish, edit, delete, and migrate operations on multiple rules)
Request separate access tokens for each scope based on the operations you need to perform.
### Rate limiting
Platform rate limits apply to all API requests to protect system stability.
If you encounter rate limit errors, implement exponential backoff in your integration.
## Use the Rules and Rewrites API
To use the API, you need a Lucidworks service account.
Then you can gather your credentials and begin making API calls.
For complete details about API endpoints, methods, and parameters, see the [Commerce Studio Rules and Rewrites API reference](/api-reference/rule-management/list-rules).
If you don't already have a Lucidworks service account of type "API", complete the following steps:
In Lucidworks Platform, go to **Settings** > **Service Accounts**.
Click **Create New Service Account**.
In the New Service Account window, be sure to select **API** as the service account type.
Enter a name and optionally a description to help you remember what this account is for.
Click **Create New Service Account**.
The new account appears in the list.
Click the account you created to find the **Client ID** and **Client Secret** you need to access the API.
You'll need the following information to access the API:
* **Commerce Studio ID**: Your Commerce Studio instance identifier
* **Client ID**: From your service account
* **Client Secret**: From your service account
Your Commerce Studio ID is part of your Commerce Studio URL.
For example, if your Commerce Studio URL is `https://COMMERCE_STUDIO_ID.experiencemanager.lucidworks.com`, your Commerce Studio ID is `COMMERCE_STUDIO_ID`.
In Lucidworks Platform, go to **Settings** > **Service Accounts** and click the service account you created.
Copy the **Client ID** and **Client Secret**.
Before making API requests, you need to authenticate and obtain an access token.
Send a POST request to the authentication endpoint:
```bash theme={"dark"}
curl --request POST \
--url 'https://identity.lucidworks.com/oauth2/ausao8uveaPmyhv0v357/v1/token?scope=API_SCOPE&grant_type=client_credentials' \
--header 'Accept: application/json' \
--header 'Authorization: Basic [CLIENT_ID:CLIENT_SECRET_BASE64]' \
--header 'Cache-Control: no-cache' \
--header 'Content-Type: application/x-www-form-urlencoded'
```
Replace `API_SCOPE` with one or more of these (as a space-separated list):
* `commercestudio.rules` for rule operations
* `commercestudio.rewrites` for query rewrite operations
* `commercestudio.bulkoperations` for bulk operations
The `Authorization` header uses basic authentication with your base64-encoded client ID and client secret.
You can encode your credentials using:
```bash theme={"dark"}
echo -n "CLIENT_ID:CLIENT_SECRET" | base64
```
Some HTTP clients perform the base64 encoding automatically when you provide the client ID as the username and client secret as the password.
A successful response returns:
```json theme={"dark"}
{
"token_type": "Bearer",
"expires_in": 3600,
"access_token": "YOUR_ACCESS_TOKEN",
"scope": "commercestudio.rules"
}
```
Use the access token in the `Authorization` header of your API requests:
```bash theme={"dark"}
curl --request GET \
--url 'https://{COMMERCE_STUDIO_ID}.experiencemanager.lucidworks.com/em-rules' \
--header 'Authorization: Bearer YOUR_ACCESS_TOKEN'
```
Replace `{COMMERCE_STUDIO_ID}` with your Commerce Studio instance ID.
Access tokens expire after 60 minutes.
Once you request a new token, the previous one is deleted *even if it has not yet expired*.
All subsequent API calls must use the new token.
## Bulk operations
Bulk operations enable you to perform actions on multiple rules simultaneously.
This is useful for scenarios like these:
* Publishing or unpublishing large sets of rules at once
* Applying consistent edits across multiple rules
* Deleting obsolete rules in batches
* Migrating rule sets between Commerce Studio instances
Bulk operations require an authentication token with the `commercestudio.bulkoperations` scope.
See [Authentication scopes](#authentication-scopes).
### Supported bulk operations
The API supports these bulk operation types:
* `publishRule`: Make multiple rules active simultaneously.
* `unpublishRule`: Deactivate multiple rules at once.
* `editRule`: Apply the same updates to multiple rules.
* `deleteRule`: Remove multiple rules in one operation.
* `migrateRule`: Copy rules to another Commerce Studio instance.
### Example requests
Each bulk operation is a POST request to `/bulk-operations`.
The request body's `type` field determines which operation runs, and you can target rules by ID, external ID, or a `specification` filter (or a combination of all three), as shown in these examples:
Publish all draft rules tagged `holiday-2026`.
```json theme={"dark"}
{
"type": "publishRule",
"specification": {
"tags": ["holiday-2026"],
"publishState": "DRAFT"
}
}
```
Unpublish a specific set of rules by ID.
```json theme={"dark"}
{
"type": "unpublishRule",
"ruleIds": [
"3fa85f64-5717-4562-b3fc-2c963f66afa6",
"7c9e6679-7425-40de-944b-e07fc1f90ae7"
]
}
```
Raise the precedence of every rule tagged `holiday-2026`.
This example sets `stageChanges` to `false`, so the patch applies immediately, including to already-published rules.
Set `stageChanges` to `true` instead to apply the patch to staging content only, leaving published rules live and unaffected until the changes are published.
```json theme={"dark"}
{
"type": "editRule",
"specification": {
"tags": ["holiday-2026"]
},
"patch": {
"precedence": 5
},
"stageChanges": false
}
```
Delete draft rules last modified before a cutoff date, excluding one rule you want to keep.
```json theme={"dark"}
{
"type": "deleteRule",
"specification": {
"publishState": "DRAFT",
"lastModifiedEnd": "2026-01-01T00:00:00Z"
},
"excludeRuleIds": [
"3fa85f64-5717-4562-b3fc-2c963f66afa6"
]
}
```
Migrate a rule to another Commerce Studio instance and publish it there immediately.
```json theme={"dark"}
{
"type": "migrateRule",
"ruleExternalIds": [
"promo-summer-2026"
],
"targetExperienceManagerInstanceId": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
"published": true
}
```
### How bulk operations work
Bulk operations execute asynchronously.
When you create a bulk operation, the API returns immediately with an operation ID.
You can then poll the operation status to track progress.
Send a POST request to `/bulk-operations` with the operation type and list of rule IDs or external IDs.
The API returns a bulk operation ID and initial status.
Poll the bulk operation status using GET `/bulk-operations/{id}`.
The response includes the following information:
* Overall operation status:
* `IN_PROGRESS`: Operation is currently executing.
* `COMPLETED`: All rules processed successfully.
* `FAILED`: Operation failed before processing any rules.
* `COMPLETED_WITH_ERRORS`: Some rules succeeded, others failed.
* Individual status for each rule
* Error messages for any failed rules
Rules included in an active bulk operation cannot be modified until the operation completes.
The `pendingInBulkOperation` field indicates when a rule is locked by a bulk operation.
When the operation reaches a terminal status (`COMPLETED`, `FAILED`, or `COMPLETED_WITH_ERRORS`), review the results.
For operations that completed with errors, the response identifies which rules succeeded and which failed.
For detailed information about bulk operation endpoints and parameters, see the [Bulk Operations API reference](/api-reference/bulk-operations/list-bulk-operations).
### Best practices
Follow these guidelines when using bulk operations:
* **Start with small batches**: Test bulk operations on a small set of rules before scaling up.
* **Monitor operation status**: Don't assume success—always check the final status and handle partial failures.
* **Handle errors gracefully**: When operations complete with errors, review which rules failed and retry them individually if needed.
* **Avoid modifying locked rules**: Check the `pendingInBulkOperation` flag before attempting to update rules.
* **Use external IDs for stability**: External IDs make it easier to retry operations and maintain consistency across environments.
For detailed information about bulk operation endpoints and parameters, see the [Bulk Operations API reference](/api-reference/bulk-operations/list-bulk-operations).
# Editor
Source: https://doc.lucidworks.com/docs/lw-platform/lw-cs/cstudio-editor
The Commerce Studio Editor lets you configure search ranking rules, manage product boosting and blocking, set facets, and preview search results in real time to optimize product visibility.
To access it, go to your site in Commerce Studio and click **Commerce Studio** > **Editor**.
If the Editor doesn't display items in the expected format, you might need to configure Commerce Studio's [settings](/docs/lw-platform/lw-cs/cstudio-settings).
## Use the Conditions Manager
At the top of the Editor you'll find the Conditions Manager.
This is where you set the conditions for any rules you create in Commerce Studio, such as the query, timeframe, category, and more.
The full page control displays in the Conditions Manager. To collapse the Conditions Manager, click the arrow below the Conditions Manager.
### Select a search profile to preview
The Preview dropdown shows you the available search profiles.
Query profiles represent different predefined search results that you can display on different pages of your customer-facing site, such as new items, seasonal items, sale items, and so on.
You might also have experimental query pipelines for testing different configurations.
### Use search conditions
When **Search** is selected in the Conditions Manager, you can enter queries and see the results in the live preview.
You can enter multiple queries that specify the qualifier for the search terms:
* **is**: The search query must match the defined term exactly. For example, if the rule is set to "\[is] black shoes", it only triggers when the user search is exactly "black shoes."
* **contains**: The search query must include one or more of the defined terms anywhere within the query. Each word is evaluated independently. For example, if the rule is set to "\[contains] black shoes", it triggers for any query containing "black" (such as "black boots") and any query containing "shoes" (such as "running shoes"). The words do not need to appear together or in a specific order.
* **contains phrase**: The search query must include the defined phrase in the exact word order entered. For example, if the rule is set to "\[contains phrase] black shoes", it triggers for "black shoes" or "men's black shoes," but not for "shoes black" or "black running shoes."
* **starts with**: The search query must begin with the defined word or phrase. For example, if the rule is set to "\[starts with] winter", it triggers for "winter jackets" or "winter boots," but not for "jackets for winter." Multi-word phrases respect word order, so "\[starts with] winter jackets" triggers for "winter jackets sale" but not for "jackets winter sale."
* **ends with**: The search query must end with the defined word or phrase. For example, if the rule is set to "\[ends with] manual", it triggers for "installation manual" or "product manual," but not for "manual installation guide." Multi-word phrases respect word order, so "\[ends with] near me" triggers for "pizza near me" but not for "near pizza me."
If you enter multiple queries, you can click to switch between them in the live preview.
You can also filter results by tags, fields, or date range.
### Use category conditions
When you select **Category** in the Conditions Manager, you can type to search for categories, then select one or more to display in the live preview. Just like search conditions, you can enter more than one category, then click to switch between them in the live preview.
## Configure the item card display
The fields displayed in the Editor for any search profile are configured on the [Settings](/docs/lw-platform/lw-cs/cstudio-settings) screen.
Once those are configured, you can click **Preview Settings** in the Editor to toggle whether each field is displayed or hidden, including these:
* Item fields, such as the title, description, image, and so on.
* Variants such as color or image variants
* Metric fields such as impressions, add-to-cart (ATC), conversion rate (CVR), revenue per visit (RPV), and more.
## View and modify rules
In the Execute panel on the left side of the Editor, you can view the rules that apply to the current query or category.
### Rule state
Published rules are indicated by an up arrow in the **State** column. Draft rules are displayed with a pencil icon in the **State** column.
### Preview rule
The **Preview** feature lets you test to see the search results if a rule is removed from a query without actually deleting the rule.
You can click the **Preview** toggle `on` to execute that rule in the query, or click **Preview** `off` to exclude it from the query. The query results display based on the rules included in the query.
Your live site is *never* affected by setting the **Preview** toggle to `off`.
### Rule details
When you click a specific rule, the Rule Details section displays. You can view rule information such as name, creation information, and the action such as boost or bury.
In addition, you can scroll to see other information such as the history of changes to the rule, conflicts with other rules, and items linked to the rule.
You can edit, delete, or duplicate the rule. If the rule status is `Draft`, you can click **Publish** to make the rule active on your site. If the rule status is `Published`, you can click **Unpublish** to deactivate the rule, or click **Edit** to open the rule for editing, then click **Save as Draft** to save your changes as a draft attached to the published rule — the live version continues to serve your site until you publish the draft. For more information, see [Manage rules](/docs/lw-platform/lw-cs/cstudio-rules#manage-rules).
## Create new rules
When you create a new rule in the Editor, the rule is triggered by the current query or category in the **Conditions Manager**.
For example, when "party supplies" is the current query and you create and publish a rule, the rule is triggered only when end users search for "party supplies".
Likewise, when "Rugs" is the current category, a new rule is triggered only when end users view the "Rugs" category on your site.
There are several ways to create new rules in the Editor:
* Click **Create Rule** in the Execute panel to open the Visual Builder and select a rule type.
* Hold the pointer over any facet and click to create a facet rule.
* Hold the pointer over any item card and click the controls to create a rule that applies that item.
* Select the checkboxes for multiple items, then click the controls to create a rule that applies to that set of items.
You can also create rules directly on the [Rules](/docs/lw-platform/lw-cs/cstudio-rules) screen using the Form Builder, which provides a form-based interface without the live preview.
## View product details
If you **Click for details**, the Product details panel displays and provides the following information: product name, image, position (ranking in results), triggered rules for the item, and the item analytics selected to view.
## Ranking insights
This section displays the ranking factors for the results.
Factors include:
* **Sort**: Ranking signals such as relevance score
* **Query processing**: Any rules that fired when creating the result set, such as phrase detections
* **Analytics**: Selected performance metrics such as total revenue, total searches, total purchases, and average click position
For more information about these and other metrics in Analytics Studio, see [types of metrics](/docs/lw-platform/lw-analytics/analytics-screen#types-of-metrics).
## Impressions metrics
Product impressions measure the number of times a product displays in search results in response to user queries. The product is counted each time it displays, even if users do not interact further with the product.
Impressions metrics can be displayed in [**Top Performing Products**](/docs/lw-platform/lw-analytics/analytics-screen#param-top-performing-products) in Analytics Studio and Commerce Studio Editor on the search results cards.
For information about how to use impressions metrics to enhance product visibility and conversions, see [Impressions metrics usage](#impressions-metrics-usage).
If you select [**Top Performing Products**](/docs/lw-platform/lw-analytics/analytics-screen#param-top-performing-products) in Analytics Studio, the dashboard displays a column for the number of impressions per product.
To enable impressions to display on product cards in Commerce Studio Editor, complete the following steps:
1. Navigate to Commerce Studio Settings and scroll to the [Inline Analytics section](/docs/lw-platform/lw-cs/cstudio-settings#inline-analytics). Click the toggle `on` for Inline Analytics. This setting is required to display the Metrics option in Commerce Studio Editor Preview Settings. If the toggle is `off`, you cannot display any metrics on search results cards in Commerce Studio Editor.
* **Metrics Duration.** The timeframe selected in the Metrics Duration field sets the default. But you can override this duration by selecting a different timeframe in Commerce Studio Editor Preview Settings Metrics.
* **Document ID Field.** Enter the name of the search result document field used as the document ID when retrieving metrics from the Signals Store. The default is `id`.
For detailed information, see [Commerce Studio inline analytics settings](/docs/lw-platform/lw-cs/cstudio-settings#inline-analytics).
2. Navigate to the Commerce Studio Editor screen and click **Preview Settings**. In the **Metrics** section, click the toggle to `on` for each metric you want to display on search results cards in Commerce Studio Editor.
The search results cards reflect all of the metrics selected. Here is an example with Impressions, Sales, and CVR metrics.
### Impressions metrics usage
Merchandisers can use impressions metrics information to help optimize and measure the effectiveness of customer search experiences. Ensuring the appropriate products display is essential to visibility and increased conversions.
This section provides information about how merchandisers can use product impressions metrics to identify opportunities and take action in Commerce Studio.
#### Understand the context for engagement metrics
For example, a product with 1,000 impressions and 50 clicks has a 5% click-through rate, while a product with 50 impressions and 50 clicks has a 100% click-through rate. Without impression data, these scenarios appear identical.
If a product has thousands of impressions but few clicks, consider taking the following actions to improve the potential for conversions.
* Review product title, description, and image quality.
* Check if the product is appearing for relevant search queries.
* Consider adjusting product positioning or boosting rules.
#### Identify product visibility gaps
For example, products with high conversion rates but low impressions may be underexposed and represent missed revenue opportunities.
If a product has a high click-through rate and conversion rate but limited impressions, consider taking the following actions to improve product visibility and enhance the potential for even more conversions.
* Boost the product in relevant search queries to increase visibility.
* Add merchandising rules to feature the product more prominently.
* Ensure the product is properly categorized and tagged.
#### Measure the impact of merchandising efforts
For example, impressions metrics track how boosting rules, pinning, and other merchandising actions affect product visibility over time.
If a previously visible product shows decreasing impression trends, consider taking the following actions to improve product visibility and enhance the potential for conversions.
* Investigate if new products are displacing that product in search results.
* Review relevancy rules and ensure the product matches user search intent.
* Check inventory status and product availability.
#### Expand the number of products with high impression-to-conversion ratios
If most of the impressions are concentrated on a few products while others receive minimal exposure, consider taking the following actions to improve impressions for a greater number of products to enhance the potential for conversions.
* Diversify search results to expose a broader product range.
* Implement merchandising rules to rotate featured products.
* Balance automated relevance with strategic product visibility.
# Rewrites
Source: https://doc.lucidworks.com/docs/lw-platform/lw-cs/cstudio-query-rewrite
You can configure query rewrite rules to help users find more relevant results. Create rules to account for underperforming or misspelled search terms, identify phrases to boost items, map synonyms to alternative search terms, and remove words from queries that limit the results returned.
To learn more about the different rewrite rules, see [Types](#types). To start creating and publishing query rewrites, refer to the [Manage rewrites](#manage-rewrites) section.
## Query rewrites vs. rules
Query rewrites improve the accuracy and relevance of search results by modifying the query. In contrast, [rules](/docs/lw-platform/lw-cs/cstudio-rules) manipulate the ranking and presentation of search results.
Query rewrites focus on the user’s intent. They take effect before the query is submitted and results are returned.
## Types
There are several different types of rewrite rules to choose from depending on your use case.
### Underperforming Queries
An underperforming rewrite, also known as a tail query rewrite, replaces a poorly-defined search term with an improved one. This function adds fields that are used as facets and tags to clarify intent, which improves the relevance of results.
* B2B organizations use this feature to change vague or incomplete user entries and add structured information that clarifies user intent and enhances relevant results. For example, if a user enters a query using general terms, a query rewrite can add business-specific terminology to improve the results returned.
* B2C organizations use this feature to interpret user entries that may be general or ambiguous and add fields that align with items in the product catalog. For example, you can create a tail query rewrite that replaces the query `iphone case yellow` with `iphone case yellow +color:"yellow"`, which uses the color field in your search facets.
* Knowledge management organizations use this feature to tag incomplete or imprecise user queries with existing content to improve discoverability of site information. For example, if the user enters `set password`, the tags may include `set password +doc_type:"login credentials"`.
### Misspelling Detections
A misspelling query rewrite detects and corrects common spelling mistakes.
When a customer enters a search term containing a known misspelling, the incorrect spelling is replaced with the spelling correction.
* B2B organizations enter corrections for industry-specific terminology, jargon, or technical terminology. For example, if your organization provides auto parts, a user might enter `brake petal`, so the correction would be `brake pedal`.
* B2C organizations enter corrections to common misspellings of items in the product catalog. For example, if your customers frequently misspell or mistype the word `sweater` as `sweatre`, you can define an entry to correct it.
* Knowledge management organizations enter terms from their content that users frequently misspell. For example, if medication clinical trials are part of the data, the user enters `metfrmin side effects`, so the correction your site enters would be `metformin`.
### Phrase Detections
A phrase rewrite identifies phrases used in search terms so that products with matching phrases are boosted in the search results.
This is helpful when users do not use quotation marks to identify phrases in their search terms.
For B2B, B2C, and knowledge management organizations, this function adds quotation marks around the phrase the user enters to match the entire phrase and not the individual words of the phrase. For example, without phrase detection, a search for the words `remote control` would show results for both `remote` and `control`.
With phrase detection, this search would correctly boost results for `"remote control"`.
### Synonym Detections
You can specify synonyms for a query so that alternative words with the same meaning are automatically used in the search query.
When a customer enters a search term with a synonym match, the alternative words are used instead of, or in addition to, the original search term.
B2B, B2C and knowledge management organizations enter synonyms for products and content specific to their site. For example, a B2C organization might enter synonyms such as `pullover` and `jumper` for the search term `sweater`.
Synonym query rewrites can be one-directional or two-directional.
* One-directional synonyms replace the original query with the synonym.
Use a one-directional synonym if you are confident the synonym is best for your product catalog.\
For example, a running specialty store may use a one-directional synonym to rewrite the term `sneakers` to `running shoes`.
* Two-directional synonyms expand the query to include the original term and specified synonyms.
Use a two-directional synonym when terms are used interchangeably (`sweater` and `pullover`) to increase the volume of results.
### Word Removal
Use the remove words rewrite to simplify search queries and increase results. B2B, B2C, and knowledge management organizations determine words to remove that will improve query results for their site.
For example, a B2C organization may rewrite the query `best wireless headphones` to remove the word `best`, which restricts the number of results returned. The improved query, `wireless headphones`, returns more results.
**FAQ: When should I use a remove words query rewrite instead of an underperforming rewrite?**
Use a remove words query rewrite to broaden a query to include more results. Use an underperforming query rewrite to simplify a complex query (head) or expand a short query (tail).
## Manage rewrites
To get started working with query rewrites, navigate to Commerce Studio and click **Rewrite** to display the Query Rewrites screen.
### Create a query rewrite
The general steps to create a query rewrite are as follows:
1. Locate the tab specific to the query rewrite type.
2. Click **Add new**.
3. Enter the required information for your rewrite type as explained below and then click **Save**.
To view the steps for a specific rewrite type, click the title tab.
1. In the **Tail Query** field, enter the text for an underperforming query. For example, `latest shoes`.
2. In the **Improved Query** field, enter new text for the query. For example, `new shoes`. To add another query, click **+ Add** and enter the text in the new field.
3. Optional: In the **Description** field, enter user-defined text to describe the query. For example, `improve to display new shoes`.
1. In the **Misspelled Query** field, enter a common incorrect spelling. For example, `intertainment`.
2. Click **Suggested Corrections** and enter corrected text. For example, `entertainment`. To add another correction, click **+ Add** and enter the text in the new field.
3. Optional: In the **Description** field, enter user-defined text to describe the misspelled text. For example, `correcting for entertainment`.
1. In the **Phrase** field, enter text to search. For example, `new shoes with low heels`.
2. Optional: In the **Description** field, enter user-defined text to describe the phrase. For example, `phrase for new, low-heeled shoes`.
1. In the **Search Query** field, enter text that will have synonyms. For example `shoes`.
2. In the **Synonym** field, enter a synonym for the **Search Query** value. For example, `sneakers`, `hightops`, or `running`. To add another synonym, click **+ Add** and enter the text in the new field.
3. Select a **Synonym Matching Method**:
* **One-directional**. This is also referred to as `ONEWAY`.
* **Two-directional**. This is also referred to as `SYMMETRIC`.
4. Optional: In the **Description** field, enter user-defined text to describe the synonym. For example, `synonyms for shoes that are sneakers`.
1. In the **Words to Remove** field, enter text to remove from queries to improve results. For example, `retro`.
2. Click **Trigger Phrases** and enter a phrase that would contain the words to remove. For example, `retro sneakers`. To add another phrase, click **+ Add** and enter the text in the new field.
3. Optional: In the **Description** field, enter user-defined text to describe the word removal. For example, `remove retro to get all sneakers`.
### Edit a query rewrite
To edit a query rewrite, hold the pointer over the entry and click the pencil icon. Enter the changes for the specific query rewrite and click **Save**.
### Publish a query rewrite
To publish a query rewrite, click the **Publish** toggle to the **Yes** position. To unpublish, click the toggle again.
Published query rewrites start working immediately.
To publish multiple query rewrites, select the checkbox for each of the rewrites to publish and then click the up arrow. The toggle switches for each of those entries changes to **Yes**.
To unpublish multiple query rewrites, select the checkbox for each of the rewrites to unpublish and then click the down arrow. The toggle switches for each of those entries changes to **No**.
### Delete a query rewrite
1. Select the checkbox of the query rewrite to delete.
2. Click the trash can icon.
3. Confirm your choice in the dialog box that appears.
To delete multiple rewrites, select the checkbox for each of the rewrites to delete and then click the trash can icon.
Query rewrites are deleted permanently. There is no way to undo this action.
## Effective workflows for teams
Your team’s Data Analyst monitors Analytics Studio to identify low performing queries, recommends recommend which queries should be rewritten, and helps verify query rewrites are effective.
Apart from the insights Analytics Studio provides, your team may start with existing industry knowledge. For example, some clothing brands prefer the term "pant" over "pants." Depending on your catalog, your team’s Product Manager may create a synonym query rewrite to treat the query `pants` the same as the query `pant`.
Your Customer Experience Manager creates query rewrites based on knowledge of your customers' intent. For example, the CX Manager rewrites the query term `cheap` to the query terms `discount` or `sale`. These rewritten queries trigger rules that are used in a sales event.
# Rules
Source: https://doc.lucidworks.com/docs/lw-platform/lw-cs/cstudio-rules
Rules affect the order in which products appear in your search results.
Rules let you pin, boost, bury, and block specific items in your catalog.
You can also use rules to display custom banners, redirect customers to a different page on your site, set facets, and more.
Learn how to use every rule type in Commerce Studio to improve search results and merchandising. This course provides detailed guidance on boost, pin, bury, block, facets, banners, redirects, and advanced configurations.
Click your organization type to see examples of how to use rules to benefit your business goals:
B2C organizations use rules to promote the following to achieve business goals:
* Products for marketing campaigns
* New product launches
* Products with inventory levels the organization wants to reduce
* Any other business objective, such as increasing conversions
For example, a boost rule could be applied to promote summer sandals and clothing for an end-of-summer promotion.
B2B organizations use rules to promote the most relevant items, services, and documents based on customer queries, and provide similar products and specifications based on customer intent. For example, if your organization sells computer memory components to distributors, the results include installation instructions and specifications that distributors can confirm and provide to their customers.
Knowledge management organizations use rules to retrieve and display the most relevant documents based on user queries. For example, an organization that provides scientific study results for medication clinical trials may pin the most recent results, while burying results for previous studies.
## Rules vs. query rewrites
Both rules and query rewrites can be configured to help return the best search results to your users.
Query rewrites allow you to alter and interpret search terms in multiple ways so relevant results are delivered.
Rules changes the order in which search results are displayed, allowing you to promote certain items over others.
Although query rewrites are configured in the [Rewrites](/docs/lw-platform/lw-cs/cstudio-query-rewrite) screen, you will see the effects of those rules on the Editor screen.
## How rules work
Rules contain [conditions](#conditions) and [actions](#actions).
When a condition or a combination of conditions is met, the rule performs an action.
## Conditions
You can trigger rules on a single condition or a combination of conditions.
The date and time range for the rule. Click the field to select a date and time, or enter manually. For example, `05/15/2026 12:00`.
The field value to which the rule applies. For example, you can specify the `Color` field and set it to `Color: Black`. and multiple values for a given field to create more specific rules. For example, if you set the `Color` field to `Color: Black` and specify the `Brand` field to include multiple brands, then the rule triggers only if the Color and Brand field conditions are satisfied.
The query that triggers the rule. You must select a **Query Matching Method** to determine when a rule triggers.
* **is**: The search query must match the defined term exactly. For example, if the rule is set to "\[is] black shoes", it only triggers when the user search is exactly "black shoes."
* **contains**: The search query must include one or more of the defined terms anywhere within the query. Each word is evaluated independently. For example, if the rule is set to "\[contains] black shoes", it triggers for any query containing "black" (such as "black boots") and any query containing "shoes" (such as "running shoes"). The words do not need to appear together or in a specific order.
* **contains phrase**: The search query must include the defined phrase in the exact word order entered. For example, if the rule is set to "\[contains phrase] black shoes", it triggers for "black shoes" or "men's black shoes," but not for "shoes black" or "black running shoes."
* **starts with**: The search query must begin with the defined word or phrase. For example, if the rule is set to "\[starts with] winter", it triggers for "winter jackets" or "winter boots," but not for "jackets for winter." Multi-word phrases respect word order, so "\[starts with] winter jackets" triggers for "winter jackets sale" but not for "jackets winter sale."
* **ends with**: The search query must end with the defined word or phrase. For example, if the rule is set to "\[ends with] manual", it triggers for "installation manual" or "product manual," but not for "manual installation guide." Multi-word phrases respect word order, so "\[ends with] near me" triggers for "pizza near me" but not for "near pizza me."
Trigger this rule on a specified query profile. This value must match an existing Fusion query profile, which points to a specific configuration.
You must select a **Query Matching Method** to determine when a rule triggers:
* **is**: The search query must match the defined term exactly. For example, if the rule is set to "\[is] black shoes", it only triggers when the user search is exactly "black shoes".
* **contains**: The search query must include one or more of the defined terms anywhere within the query. Each word is evaluated independently. For example, if the rule is set to "\[contains] black shoes", it triggers for any query containing "black" (such as "black boots") and any query containing "shoes" (such as "running shoes"). The words do not need to appear together or in a specific order.
* **contains phrase**: The search query must include the defined phrase in the exact word order entered. For example, if the rule is set to "\[contains phrase] black shoes", it triggers for "black shoes" or "mens black shoes," but not for "shoes black" or "black running shoes".
* **starts with**: The search query must begin with the defined word or phrase. For example, if the rule is set to "\[starts with] winter", it triggers for "winter jackets" or "winter boots," but not for "jackets for winter". Multi-word phrases respect word order, so "\[starts with] winter jackets" triggers for "winter jackets sale" but not for "jackets winter sale".
* **ends with**: The search query must end with the defined word or phrase. For example, if the rule is set to "\[ends with] manual", it triggers for "installation manual" or "product manual," but not for "manual installation guide". Multi-word phrases respect word order, so "\[ends with] near me" triggers for "pizza near me" but not for "near pizza me".
Learn how to use the starts with and ends with query matching methods in Commerce Studio rules to trigger rules based on how a query begins or ends.
## Actions
You can only apply one action per rule. Commerce Studio alerts you if you create multiple rules with conflicting actions. For example, a conflict occurs if you create rules that boost, block, and bury the same item.
### Rerank
#### Boost
Increases the ranking of one or more products so they move up in the search results.
##### Boost by attributes
Boosts items in the search results that match selected attributes, based on the weight given the attribute in a slider. This is the recommended type of boost because it adapts to changing catalogs and reflects a broader merchandising strategy.
This demo describes how to create a boost rule by specifying attributes in the Commerce Studio Rules screen (Form Builder).
If you click **Create Rule** and then select **Visual Builder**, the Commerce Studio Editor screen displays and you can create a rule from that screen.
You can boost items based on multiple attributes, including multiple values in one category.
* **Within a single attribute**: When you select multiple values for one attribute (for example, `Color: Black, Gray`), items matching any of those values are included using **OR logic**, so items matching either Black or Gray are boosted.
* **Across multiple attributes**: You can combine different attribute categories using **AND logic** for more granular targeting. For example, if you set `Color: Black, Gray` AND `Count: 40, 50`, the rule boosts only black or gray items that come in packs of 40 or 50. Items must match at least one value from each attribute category.
Existing attribute rules continue to use OR logic between attributes, ensuring backward compatibility for older rules.
The **Boost Param** dropdown sets how the boost is applied to matching products.
Enter your query expression in the **Boost Query** field.
| Boost Param | Boost Query example | Effect on products |
| - | - | - |
| `bq` | `category_s:"Small Business"^5` | Additive. Products matching the query receive an extra relevance boost on top of their normal ranking. The number after `^` sets how much weight to add. |
| `boost` | `if(termfreq(brand_s,'Acme'),10,1)` | Multiplicative. Products where the condition is true have their score multiplied by the specified number (here, 10×). Products where the condition is false are multiplied by 1 (unchanged). |
This rule type also supports these comparison operators for numerical fields:
* **Equals** matches the specified value exactly (text or numeric).
* **Greater than** matches numeric values greater than the specified number.
* **Less than** matches numeric values less than the specified number.
* **Greater than or equal to** matches numeric values greater than or equal to the specified number.
* **Less than or equal to** matches numeric values less than or equal to the specified number.
* **Between** matches numeric values within the specified range (inclusive).
* **Not equal to** excludes values that match the specified value exactly.
Numeric comparison operators and boosting by AND attributes require Fusion 5.17.1 or later.
##### Boost items
Boosts specific items to the top of the results. These items must be identified individually. This approach is more precise but less flexible than boost by attributes.
#### Pin items
Pins products to an exact location within search results. This allows precise control over product placement. For example, you can ensure certain products always appear next to each other.
Another example is that you can pin a product to a specific position that reliably appears on page 2 after pagination. This capability requires the applicable Fusion patch to be installed. For more information, contact your Lucidworks support representative.
#### Bury
Moves selected items to the bottom of search results. Unlike the Block action, products are not removed entirely from the search results.
##### Bury by attributes
Buries items in the search results that match selected attributes, based on the weight given the attribute in a slider. This is the recommended type of bury for the same reasons as boost by attributes.
For example, you can use this to de-emphasize a particular brand in your results without removing it entirely.
The steps to bury by attributes are the same as those described in the [Boost by attributes](#boost-by-attributes) demo, except you would select the **Bury** in the **Action** field.
You can bury items based on multiple attributes, including multiple values in one category.
* **Within a single attribute**: When you select multiple values for one attribute (for example, `Color: Black, Gray`), items matching any of those values are included using **OR logic**, so items matching either Black or Gray are buried.
* **Across multiple attributes**: You can combine different attribute categories using **AND logic** for more granular targeting. For example, if you set `Color: Black, Gray` AND `Count: 40, 50`, the rule buries only black or gray items that come in packs of 40 or 50. Items must match at least one value from each attribute category.
Existing attribute rules continue to use OR logic between attributes, ensuring backward compatibility for older rules.
This rule type also supports these comparison operators for numerical fields:
* **Equals** matches the specified value exactly (text or numeric).
* **Greater than** matches numeric values greater than the specified number.
* **Less than** matches numeric values less than the specified number.
* **Greater than or equal to** matches numeric values greater than or equal to the specified number.
* **Less than or equal to** matches numeric values less than or equal to the specified number.
* **Between** matches numeric values within the specified range (inclusive).
* **Not equal to** excludes values that match the specified value exactly.
Numeric comparison operators and burying by AND attributes require Fusion 5.17.1 or later.
##### Bury items
Buries specific items to the bottom of the results. These items must be identified individually.
Like boost items, this approach is more precise but less flexible than bury by attributes.
#### Block
Removes selected items from search results entirely. Blocked items don't appear for matching searches.
##### Block by attributes
Applies block based on selected attributes. Products matching the specified attributes are completely removed from search results.
You can block items based on multiple attributes, including multiple values in one category.
* **Within a single attribute**: When you select multiple values for one attribute (for example, `Color: Black, Gray`), items matching any of those values are included using **OR logic**, so items matching either Black or Gray are blocked.
* **Across multiple attributes**: You can combine different attribute categories using **AND logic** for more granular targeting. For example, if you set `Color: Black, Gray` AND `Count: 40, 50`, the rule blocks only black or gray items that come in packs of 40 or 50. Items must match at least one value from each attribute category.
Existing attribute rules continue to use OR logic between attributes, ensuring backward compatibility for older rules.
This rule type also supports these comparison operators for numerical fields:
* **Equals** matches the specified value exactly (text or numeric).
* **Greater than** matches numeric values greater than the specified number.
* **Less than** matches numeric values less than the specified number.
* **Greater than or equal to** matches numeric values greater than or equal to the specified number.
* **Less than or equal to** matches numeric values less than or equal to the specified number.
* **Between** matches numeric values within the specified range (inclusive).
* **Not equal to** excludes values that match the specified value exactly.
Numeric comparison operators and blocking by AND attributes require Fusion 5.17.1 or later.
##### Block items
Applies block to specific products that must be identified individually. These products will be completely removed from search results.
### Facets
#### Manage facets
Customizes which facet categories appear in the query response, their display order, and their individual configuration settings. For example, if your site sells clothing, you can configure facet categories for brand, color, price, size, and other attributes.
Each facet category can be extensively customized to control how values are displayed and how users interact with them.
Specifies how facet values are generated. **Term** generates discrete values like brand names or colors. **Gap Range** generates numeric ranges like price brackets or size ranges.
Controls how users can select facet values. **Single-select** allows users to choose only one value at a time. **Multi-select** allows users to choose multiple values simultaneously.
Determines how facet values are sorted. **Index** sorts values in alphabetical or numeric order. **Count** sorts values by the number of matching products.
Sets the minimum number of matching products required for a facet value to be displayed. Values with fewer matches will be hidden from the facet list.
Promotes specific facet values higher in the list, making them appear more prominently to users.
Moves specific facet values lower in the list, de-emphasizing them without removing them entirely.
Removes specific facet values from display entirely, preventing them from appearing in the facet list.
### Advanced
#### Add banner
Associates an image URL with a search rule and includes it in the search results JSON response. You can specify the image URL and optionally define a page zone identifier. Your frontend application can then consume this data to dynamically display promotional banners or images alongside the corresponding search results.
The URL of the image to display as a banner.
An identifier specifying which zone or area of the page the banner should appear in.
```json theme={"dark"}
"banner": [{
"url": "https://example.com/assets/holiday-banner.png",
"zone": "top-wide"
}]
```
#### Filter items
Applies [filter queries](https://solr.apache.org/guide/common-query-parameters.html) (`fq`) to the query and changes the results so a pre-selected set of content displays.
#### Set JSON configuration
Allows you to define a custom JSON blob that will be included in the search results response. You can input any valid JSON structure, which your frontend application can then interpret and use for custom functionality. This provides flexibility to pass arbitrary data alongside search results for frontend consumption.
An optional identifier used primarily for categorizing or identifying the blob.
The custom JSON blob to include in the search results response.
```json theme={"dark"}
"jsonBlob": {
"example-blob": [{
"event": "Black Friday Sale 2026",
"startDate": "11/27/2026 00:00",
"endDate": "11/27/2026 23:59",
"description": "Our biggest sale of the year! Save up to 70% on select items during our Black Friday event.",
"banner": {
"url": "https://example.com/assets/black-friday-2026-banner.png",
"zone": "top-wide"
},
"promotionCode": "BF2026"
}]
},
```
#### Redirect
Includes a redirect URL in the search results JSON response. Your frontend application can consume this data to redirect users to a specific destination, such as dedicated landing pages or custom pages for queries that return zero results.
The URL or path to redirect users to.
```json theme={"dark"}
"redirect": ["https://example.com/new-location"]
```
#### Set response value
Sends an arbitrary value to the frontend or pipeline to trigger another action. This action can be combined with other actions within the same rule. For example, an e-commerce website may use the response value action to display an advertisement.
#### Set parameters
Sets a [parameter name and value](https://solr.apache.org/guide/solr/latest/query-guide/common-query-parameters.html) to a search query. For example, the sort order for the results, or a filter that affects results.
## Advanced Grouping
Advanced grouping is a conditional, rule-based approach to product [variant grouping](/docs/lw-platform/lw-cs/cstudio-settings#variants) that enables different grouping behaviors based on rules.
You can implement advanced grouping using the Set Parameters rule type.
When you create a Set Parameters rule that includes the required grouping parameters, Commerce Studio automatically detects and enables grouping for queries where the rule conditions are met. This conditional behavior means grouping can activate for some searches but not others, based on your rule configuration.
Advanced grouping enables scenarios like these:
* Apply color-based grouping for apparel queries but ID-based grouping for electronics
* Group by size for furniture searches but by style for home decor
* Enable grouping only for specific categories or query types
* Switch grouping fields based on seasonal campaigns or promotions
### Enable advanced grouping
To enable advanced grouping, your Set Parameters rule must include the following:
Controls how many variants to display per group.
Must be set to `true` to enable expansion.
Contains the Solr `collapse` syntax that defines the grouping field.
Here's an example of the syntax for the `fq` parameter:
```
{!collapse tag=collapse field=FIELD_NAME nullPolicy=expand}
```
It includes the following parameters:
This must be set to `collapse`.
The field to group by (for example, `id`, `color_s`, `product_family_s`).
Determines behavior for items without a value in the grouping field (typically `expand`).
### Example configurations
These example configurations illustrate how to configure advanced grouping for different scenarios:
```
{!collapse tag=collapse field=id nullPolicy=expand}
```
```
{!collapse tag=collapse field=color_s nullPolicy=expand}
```
```
{!collapse tag=collapse field=Collapse_Field_s nullPolicy=expand}
```
## Create rules
Commerce Studio offers two ways to create rules: the Visual Builder and the Form Builder.
### Visual Builder
The Visual Builder lets you create rules while previewing live search results in the [Editor](/docs/lw-platform/lw-cs/cstudio-editor). Use this approach when you want to see how a rule affects item ranking in context as you configure it.
More information about creating rules in the Editor, including options to create rules for individual items, is in the [Commerce Studio Editor](/docs/lw-platform/lw-cs/cstudio-editor) topic.
### Form Builder
The Form Builder lets you create rules directly on the Rules screen without the live preview. Use this approach when you know exactly what rule you want to create and do not need to see the search results in context.
To create a rule using the Form Builder:
On the Rules screen, click **Create Rule** and select **Form Builder**.
Configure the rule fields in the panel that appears.
A name for the rule. You can edit this at any time.
Determines the order in which rules are applied when multiple rules match the same conditions. Lower numbers take higher priority.
The conditions that trigger the rule. These correspond to the condition types described in [Conditions](#conditions).
The action the rule performs when its conditions are met. Select an action type from the dropdown. The form updates to show the configuration fields for that action type. For details on each action type, see [Actions](#actions).
Optional text describing the purpose of the rule.
Click **Save as Draft** to save the rule without publishing it, or click **Publish** to make it immediately active on your site.
To edit a rule that is already published, see [Edit a published rule](#edit-a-published-rule).
## Manage rules
The Rules screen lists all rules in your Commerce Studio environment and is where you publish, edit, filter, and perform bulk actions on them.
### Rule states
Each rule on the Rules screen is in one of the following states:
* **Published.** The rule is active on your site.
* **Draft.** The rule is saved but is not active on your site.
* **Published with draft edits.** A published rule has unpublished edits saved as a draft. The published version remains active on your site, and the draft appears as a second row beneath it so you can see and act on both.
### Edit a published rule
When you edit a published rule, the published version continues to serve your site until you explicitly publish the changes. Your edits are shown in the Editor as a draft so you can preview their impact without affecting customers.
When you finish editing, click **Publish** to replace the published version with your edits and apply them to your site immediately. Alternatively, click **Save as Draft** to save your edits as a draft attached to the published rule. The published version remains live, and the draft appears beneath it on the Rules screen.
A published rule can have only one draft edit at a time. If a draft already exists, opening the rule for editing adds to the existing draft.
### Manage a published rule with draft edits
When a published rule has draft edits, both versions appear as two rows on the Rules screen. The actions available depend on which row you act on.
On the **published row**:
* **Edit** is disabled. Edits must be made to the draft version.
* **Delete** removes both the published rule and the draft edits.
* **Duplicate** is disabled. The draft edits must be published or deleted before the rule can be duplicated.
* **Unpublish** unpublishes the rule and deletes the draft edits.
On the **draft row**:
* **Edit** opens the draft for further editing.
* **Delete** discards the draft and restores the rule to the published version.
* **Publish** replaces the published version with the draft and applies it to your site.
### Bulk actions
When you select multiple rules from the Rules screen, the bulk action options let you update all of the selected rules at one time.
Watch this demo to learn how to select multiple Commerce Studio rules and update them all at once.
The following bulk options are available based on the rules selected.
1. After you select the rules to publish, click **Publish** from the bulk action option bar. The confirmation window displays, and indicates any rules that conflict with the rules you want to publish.
2. To proceed, click **Publish**.
1. After you select the rules to unpublish, click **Unpublish** from the bulk action option bar. The confirmation window displays and indicates the number of rules to unpublish.
2. To proceed, click **Unpublish**.
1. After you select the rules to modify, click **Edit** from the bulk action option bar.
2. Click the **Tags** field to select from the list. If you click **+** you can add another tag.
3. Click the **Query Profiles** field to select from the list. If you click **+**, you can add another query profile.
4. Select either **Save as Draft** or **Publish changes** for the rules you are editing.
If you select to change a rule that has existing draft edits, this edit overwrites those existing edits.
5. Click **Apply** to save the changes to the selected rules.
This option lets you select multiple rules to migrate from one Commerce Studio environment to a different Commerce Studio environment. For example, you might want to migrate rules in your Development environment to your Production environment.
Before you use this option, the following issues need to be considered.
* The bulk migration doesn't let you filter which rules are migrated by author, label, campaign, or tag.
* The bulk migration doesn't detect if a rule already exists in the target environment that might conflict with a rule being migrated.
* It is vital to preview what will change before committing to the migration.
1. After you select the rules you want to migrate from one Commerce Studio environment to a different Commerce Studio environment, click **Migrate** from the bulk action option bar.
2. Click the **Target Environment** field to select from the list.
3. Select either to migrate the selected rules and designate the rules with the **Draft** or **Published** status.
4. Click **Migrate** to complete the migration of those rules.
5. Access the target Commerce Studio environment and verify the rules migrated correctly.
This option lets you select rules and export them to a CSV file for processing outside of Commerce Studio.
After you select the rules to export, click **Export** from the bulk action option bar. The rules are written to a CSV file that is saved to your computer.
After you select rules you want to remove, click **Delete** from the bulk action option bar. Click **Delete** to confirm the removal of the rules and any draft edits on those rules.
The selected rules and **any draft edits** on the rules are deleted using this process. Verify that is the action you want to take before confirming the deletion.
To deselect all of the rules currently selected on the screen, click **Deselect** from the bulk action bar.
## Examples to improve search results using rules
Commerce Studio allows you to apply targeted actions when a search query contains specific terms. These actions do not require code changes and support the following business goals to:
* Improve relevance
* Guide users to the content they are seeking
This section provides examples of how to use rules to improve search results.
B2B and B2C organizations can use rules to improve product discovery and merchandising outcomes by performing the following actions. These examples describe some actions for a bookstore website.
* **Promote high-priority products.** Boost books from a popular series so they appear higher in search results when users search by author name.
* **Remove outdated or unavailable items.** Block discontinued titles to ensure only relevant, purchasable books appear in results.
* **Highlight specific products.** Boost selected new or overstocked titles to increase visibility and drive sales.
* **Refine results by intent.** Filter results to show only children’s and young adult books when users search for children’s content.
* **Improve ranking within related groups.** Boost items within a series so the most popular titles appear first when users search for that series.
* **Target branded searches.** Use "\[starts with]" to boost promotional campaigns when users begin their query with a specific brand name, without affecting queries where the brand appears later in the query.
* **Support location-based queries.** Use "\[ends with] near me" to apply location-specific filtering or boosting logic for geo-modified searches.
* **Manage available facets.** Display genre facets on the homepage and top author facets on search results pages to help users refine results.
* **Apply advanced configurations.** Use a `Set Banner` rule to display a promotional banner when users search for mystery titles.
B2B, B2C, and Knowledge management organizations can use rules to improve user search experiences for documents on their site by performing the following actions.
For example, organizations could set rules to enhance the discovery of internal policies and procedures for employees, and different rules for searches about external, customer-facing policies, procedures, and product specifications.
These examples describe some actions that can be applied to knowledge and support content to reduce friction and guide users quickly.
* **Surface critical content.** Pin a gift card information document to the top of results for searches including the term, "gift".
* **Guide users to key destinations.** Redirect users to a "Contact Us" page when they search for terms such as "email" or "contact".
* **Target support documentation.** Use "\[ends with] manual" to boost user manuals and product documentation, or use "\[ends with] pdf" to prioritize downloadable resources.
* **De-emphasize outdated content.** Bury notices related to discontinued programs so they appear lower in results.
* **Prioritize important documents.** Redirect to the privacy policy page when a user searches for "privacy policy".
* **Manage grouped content visibility.** Bury older product support documents within a group while pinning new program information to the top for related searches.
# Settings
Source: https://doc.lucidworks.com/docs/lw-platform/lw-cs/cstudio-settings
In Commerce Studio's settings, you can configure what the [Editor](/docs/lw-platform/lw-cs/cstudio-editor) displays for each search profile.
Some of these settings require the names of data fields in Fusion, but you can type to search for the fields you need.
In your site on Lucidworks Platform, navigate to **Commerce Studio** > **Settings** to access these settings.
Start by selecting a search profile to configure.
Search profiles represent different predefined search results that you can display on different pages of your customer-facing site, such as new items, seasonal items, sale items, and so on.
You might also have experimental search profiles for testing different configurations.
The configuration you create here determines how the item cards are displayed in the Editor when the same search profile is selected.
## Configure Search Results
In the **Search Results** section, you can configure the fields displayed for each result item in the Editor.
Once you configure these settings, you can turn different display fields on or off while you're working in the Editor using its [Preview Settings](/docs/lw-platform/lw-cs/cstudio-editor#configure-the-item-card-display).
### Layout
The **Layout** section lets you select to display search results in a vertical list or a grid pattern.
Select the option you want, click **Save**, and then verify the list displays on the Editor screen.
This demo explains the Layout options and how each displays search results.
### Fields Style
Under **Fields Style**, you can select and format the fields that the preview cards display in the Editor.
* To add a field, click the button.
* To change a field, click the field and type to search for another one.
* To change a field's display style, click the dropdown and select the style that best matches the field's data type.
### Variants
When you enable Variants, you can select the fields that contain data about different variants of your products. To enable product grouping in Commerce Studio, follow the procedure for your configuration.
If you are using Fusion, configure grouping in your query pipeline. Then, configure grouping in Commerce Studio.
In Fusion, open the query pipeline for which you want to configure grouping.
The query profile must pass a `fq`, `expand`, and `expand.rows` query parameters.
1. Add a new **Additional Query Parameters** pipeline stage or edit an existing one.
2. Add the following query parameters:
`fq`
`{!collapse field=FIELD_NAME}`
`default`
`expand`
`true`
`default`
`expand.rows`
The number of variant swatches displayed on each product card in Commerce Studio. For example, `5` displays 5 variant swatches on the product card. All variants are included in the response regardless of this value. This parameter must be set in Fusion or in the same Set Parameter Rule as the collapse parameter.
`default`
3. Save the pipeline stage and pipeline configuration.
4. Verify that your query pipeline passes the query parameters you set up in previous steps. In the query workbench, open the **View As** selector and select JSON. Navigate to `responseHeader.params` in the response.
Setting a parameter requires proper Solr syntax. If you receive errors in Commerce Studio after completing these steps, verify that you set up these parameters correctly.
Go to **Commerce Studio** > **Settings** and configure these options:
The Fusion field containing visual data (image path or color) used for variant swatches on the product card. This information is used for visual display purposes only.
Specifies whether the variant field name contains an `image` or `color`, determining which type of swatch Commerce Studio displays. This information is used for visual display purposes only.
The field containing IDs of individual products collapsed into a single group in the Editor. This information is used for visual display of variant swatches and for creating rules to rerank products within the group.
These configurations determine how products with variants are visualized in the Editor.
Save your changes, then visit the Editor to verify that items are correctly grouped.
If you are using Core Settings, enable product grouping in the Lucidworks Platform UI.
Go to **Core Settings** > **Relevance** and turn on **Product Grouping**:
Go to **Commerce Studio** > **Settings** and configure these options:
The Fusion field containing visual data (image or color) used for variant swatches on the product card. For visual display purposes only.
Specifies whether the variant field name contains an `image` or `color`, determining which type of swatch Commerce Studio displays. For visual display purposes only.
The field containing IDs of individual products collapsed into a single group in the Editor. This information is used for visual display of variant swatches and for creating rules to rerank products within the group.
Save your changes, then visit the Editor to verify that items are correctly grouped.
For conditional, rule-based grouping that can vary based on queries, categories, or other conditions, see [advanced grouping](/docs/lw-platform/lw-cs/cstudio-rules#advanced-grouping) in the Rules documentation.
### Inline Analytics
When you configure and use the signals beacon and Analytics Studio to collect and display signal information, you can also enable Inline Analytics in Commerce Studio Settings, which lets you select the metrics to display in Commerce Studio Editor search results cards.
To enable metrics to display in Commerce Studio, complete the following steps:
1. Navigate to Inline Analytics and click the toggle `on`.
IMPORTANT: This setting is required to display the Metrics option in Commerce Studio Editor Preview Settings. If the toggle is `off`, you cannot display any metrics on search results cards in Commerce Studio Editor.
2. Select a default option in the **Metrics Duration** field. This is the default duration for the Commerce Studio Editor Preview Settings Metrics section. Options include Last Hour, Last 24 Hours, Last 7 Days, Last 30 Days, and Last Year.
The timeframe selected in the Metrics Duration field is the default. But you can override this duration by selecting a different timeframe in Commerce Studio Editor Preview Settings Metrics.
3. In the **Document ID Field**, enter the name of the search result document field used as the document ID when retrieving metrics from the Signals Store. The default is `id`.
This is the Solr field that corresponds to the product identifier that your signals beacon logs for each item. The default value of `id` matches the Solr `uniqueKey` for most catalogs. However, use the value your catalog uses. Another example of the value used is `sku_s`.
This value is used to calculate the search result's Impressions, ATC,
Orders, Sales, RPV, and CVR data on the result card.
If the Document ID Field doesn't match what your signals beacon records as the product identifier, the metrics cannot be calculated and a `--` placeholder displays on the result card.
4. Optionally, if you want to build a preview of how metrics display on the search result card before you deploy it in Commerce Studio Editor, you can select the metrics checkboxes for Impressions, Add-To-Cart (ATC), Orders, Sales, Revenue Per Visit (RPV), and Conversion Rate (CVR).
In this example, all of the checkboxes except Impressions are checked, and the Settings Preview Result Card indicates how that displays:
IMPORTANT: When you select these options, they only display on the Preview result card in Commerce Studio Settings. To select metrics to display on search result cards in Commerce Studio Editor, you must navigate to Commerce Studio Editor > Preview Settings > Metrics and select metrics options. For detailed information, see [Commerce Studio Editor Impressions metrics](/docs/lw-platform/lw-cs/cstudio-editor#impressions-metrics).
## Configure categories
In the Editor, you can create and manage rules that apply within specific categories of items in your catalog.
To enable category-level search optimization in the Editor, you need to set the category field here.
Type to search for the data field that contains your site's categories, such as `category_s`.
When you select a field, this section displays example values found in that field so you can verify that the field you selected contains the data you need.
Once you've selected the category field here, you can [use category conditions](/docs/lw-platform/lw-cs/cstudio-editor#use-category-conditions) in the Editor to access the available categories and optimize the results for each one.
## Configure typeahead
Typeahead, also known as autocomplete or autosuggest, is a feature that provides real-time query and phrase suggestions as you type in the Editor's search bar.
In the Typeahead section, you can configure the fields to use for typeahead for this search profile.
* To add a field, click the button.
* To change a field, click the field and type to search for another one.
* To change a field's display style, click the dropdown and select the style that best matches the field's data type.
The Commerce Studio Settings Typeahead feature is for internal use in Commerce Studio. This is different than the [Lucidworks Platform Core Settings Typeahead feature](/docs/lw-platform/lw-core/sites-typeahead), which is where typeahead options for customer queries is configured.
# Commerce Studio versus Predictive Merchandiser
Source: https://doc.lucidworks.com/docs/lw-platform/lw-cs/cstudio-versus-pm
[old doc.lw link]: https://doc.lucidworks.com/lw-platform/cs/dndnds
[localhost link]: http://localhost:3000/docs/lw-platform/lw-cs/cstudio-versus-pm
[mintlify link]: https://doc.lucidworks.com/docs/lw-platform/lw-cs/cstudio-versus-pm
This topic explains the differences and similarities between Commerce Studio and [Predictive Merchandiser](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/overview) to make [migrating from Predictive Merchandiser to Commerce Studio](/docs/lw-platform/lw-cs/cs-migrate) easy.
Both Predictive Merchandiser and Commerce Studio enable merchandisers, marketers, and product teams to manipulate search results for maximum impact.
Commerce Studio has enhancements that make it easier and more efficient to use, optimizing your team's workflow.
Contact Lucidworks for information about migrating from Predictive Merchandiser to Commerce Studio.
## Feature overview
This table summarizes the differences and similarities between Predictive Merchandiser (PM) and Commerce Studio (CS):
| Feature | PM | CS | Details |
| - | :-: | :-: | - |
| Real-time dynamic preview | | | In Commerce Studio's [Editor](/docs/lw-platform/lw-cs/cstudio-editor), search results are updated instantly while you create and manage rules and query rewrites. |
| Instant point-and-click rule creation | | | In Commerce Studio's Editor, you can hold the pointer over any item and click to create a new rule for it. No need to start a new task first; just point and click. |
| Drag-and-drop rule creation | | | In Commerce Studio's Editor, you can drag and drop items to instantly create rules about their ordering. |
| Integrated analytics | | | Commerce Studio displays metrics inline in the Editor to show you which products are performing or underperforming so you can take action with rules and rewrites. |
| Conditions Manager | | | Commerce Studio's Conditions Manager lets you compare multiple queries and set specific conditions for your rules, such as a time range or tags. |
| Ranking Insights | | | Commerce Studio's Ranking Insights feature shows you all of the ranking method and any rewrites that affect a query's results so you can adjust them as needed to achieve your goals. |
| Execute panel | | | Commerce Studio's Execute panel displays all the of the rules associated with the given conditions (such as query, dates, and so on) right in the Editor, allowing you to view, add, and/or edit rules as needed. |
| Rules Editor | | | In Commerce Studio, you can use either the [Rules UI](/docs/lw-platform/lw-cs/cstudio-rules) or the Editor. |
| Query Rewrites Editor | | | Like Predictive Merchandiser, Commerce Studio provides a simple interface for [query rewrites](/docs/lw-platform/lw-cs/cstudio-query-rewrite). |
| Customer interface simulation | | | Whereas Predictive Merchandiser uses templates and zones, Commerce Studio uses [preview settings](/docs/lw-platform/lw-cs/cstudio-settings) that you can customize for each search profile to simulate the pages on your customer-facing website. |
| Integration with Fusion | | | Both products support [integrating with Fusion](/docs/lw-platform/lw-platform/settings/fusion-integrations/commerce-studio-integration). |
## Intro to Commerce Studio for Predictive Merchandiser clients
Click below for a quick tour of Commerce Studio's Editor that shows you how it improves on Predictive Merchandiser's user experience with smoother, integrated controls:
## Query rewrites
All types of query rewrites are supported in both Predictive Merchandiser and Commerce Studio.
When you [migrate to Commerce Studio](/docs/lw-platform/lw-cs/cs-migrate), all of your query rewrites are migrated too.
The names of the query rewrite types differ slightly between the two products, as shown in this table:
| Predictive Merchandiser rewrites | Commerce Studio rewrites |
| - | - |
| Head/Tail | Underperforming Queries |
| Misspelling | Misspelling Detections |
| Phrase | Phrase Detections |
| Synonym | Synonym Detections |
| Remove Words | Word Removal |
See [Rewrites](/docs/lw-platform/lw-cs/cstudio-query-rewrite) for more details.
## Rule publication
Like query rewrites, all of your existing rules are migrated when you migrate from Predictive Merchandiser to Commerce Studio.
The possible values for a rule's publication status differ slightly between Predictive Merchandiser and Commerce Studio.
This table shows the equivalent rule statuses between the two products:
| Predictive Merchandiser rule status | Commerce Studio rule status |
| - | - |
| Published + Disabled | Draft |
| Published + Enabled | Published |
| Unpublished + Disabled | Draft |
| Unpublished + Enabled | n/a |
See [Rules](/docs/lw-platform/lw-cs/cstudio-rules) for more details.
# Commerce Studio
Source: https://doc.lucidworks.com/docs/lw-platform/lw-cs/overview
[old doc.lw link]: https://doc.lucidworks.com/lw-platform/cs/nd39eb
[localhost link]: http://localhost:3000/docs/lw-platform/lw-cs/overview
[mintlify link]: https://doc.lucidworks.com/docs/lw-platform/lw-cs/overview
Commerce Studio is Lucidworks’ next-generation merchandising studio, featuring an intuitive drag-and-drop, point-and-click editor.
Designed and built for search teams, Commerce Studio lets business users tailor the search and browse experience through powerful rule building, visual merchandising, and more.
To use Commerce Studio, you must complete all of the procedures described in the [Commerce Studio integration with Fusion](/docs/lw-platform/lw-platform/settings/fusion-integrations/commerce-studio-integration) topic.
This article describes features that are only available in certain packages. See [Lucidworks Feature Limited Availability](/docs/policies/feature-availability) for details, or reach out to your Lucidworks representative.
**FAQ: What if I'm using Predictive Merchandiser?**
You can migrate Predictive Merchandiser data to Commerce Studio, including rules, query rewrites and template data. To compare the features of Predictive Merchandiser to Commerce Studio, see [Commerce Studio versus Predictive Merchandiser](/docs/lw-platform/lw-cs/cstudio-versus-pm). To migrate Predictive Merchandiser to Commerce Studio, see the [Migration guide](/docs/lw-platform/lw-cs/cs-migrate).
## Key features
Commerce Studio provides several core features for crafting your search experience.
### Editor
The [Editor](/docs/lw-platform/lw-cs/cstudio-editor) screen lets you visually merchandise results sets and simulate and test outcomes. The options on this screen let you:
* Simulate search and browse experiences including rule execution and facet selection.
* Review results based on rule and query rewrite configurations.
* View ranking insights and analytics.
* Preview field, variant, and metrics settings that display on each item in the catalog, including both standard and advanced grouping configurations.
* View product performance metrics for each item in the result set.
When you sign in to Lucidworks Platform, scroll to the Commerce Studio section, and then click the Commerce Studio instance to manage. The Editor screen is the default display.
For more information, see [Editor](/docs/lw-platform/lw-cs/cstudio-editor).
### Rules
The [Rules](/docs/lw-platform/lw-cs/cstudio-rules) screen lets you easily and intuitively manage your merchandising rules. You can create conditional rules based on queries, durations, and categories.
The rules affect the order products display in search results. The options on this screen let you pin, boost, bury, and block specific items in your catalog. You can also set facets, redirect customers to another page on your site, and more.
The people in your organization who would create and edit rules include the following personnel:
* Customer Experience Manager
* Product Manager
* Merchandiser
For more information about user roles, see [Who is Commerce Studio for?](#who-is-commerce-studio-for)
From the Commerce Studio screen, click **Rules** to display the screen. For example:
For more information, see [Rules](/docs/lw-platform/lw-cs/cstudio-rules).
### Rewrites
The [Rewrites](/docs/lw-platform/lw-cs/cstudio-query-rewrite) screen lets you configure rules to improve the query and generate more relevant results. In this option, you can:
* Replace and improve less relevant search terms in underperforming queries
* Correct misspelled words
* Rewrite phrases to match higher performing phrases
* Specify synonyms
* Remove unnecessary words
The people in your organization who would rewrite queries include the following personnel:
* Director of eCommerce
* Customer Experience Manager
* Product Manager
* Merchandiser
* Digital Marketer
For more information about user roles, see [Who is Commerce Studio for?](#who-is-commerce-studio-for)
From the Commerce Studio screen, click **Rewrites** to display the screen. For example:
For more information, see [Rewrites](/docs/lw-platform/lw-cs/cstudio-query-rewrite).
### Settings
The [Settings](/docs/lw-platform/lw-cs/cstudio-settings) screen lets you customize the Editor's live preview to reflect how your website looks.
The options on this screen let you add fields to display for each search profile and configure typeahead settings to closely match the layout and behavior of your frontend so you can simulate scenarios that test the effectiveness of your rules and query rewrites before you implement them in your live customer environment.
From the Commerce Studio screen, click **Settings** to display the screen.
The people in your organization who would configure the preview settings include the following personnel:
* Director of eCommerce
* Customer Experience Manager
* Product Manager
* Merchandiser
* Digital Marketer
For more information about user roles, see [Who is Commerce Studio for?](#who-is-commerce-studio-for)
The Commerce Studio Settings Typeahead feature is for internal use in Commerce Studio. This is different than the [Lucidworks Platform Core Settings Typeahead feature](/docs/lw-platform/lw-core/sites-typeahead), which is where typeahead options for customer queries is configured.
For more information, see [Settings](/docs/lw-platform/lw-cs/cstudio-settings).
### Technical requirements
| Requirement | Commerce Studio | Analytics Studio |
| - | - | - |
| Fusion 5.9.8 or later\[1] | | \[2] |
| Lucidworks Platform | | |
| Signals data | \[3] | |
| Signals beacon\[4] | \[5] | |
* \[1]: Requires a Fusion or Lucidworks Search environment running Fusion 5.9.8 or later.
* \[2]: Analytics Studio provides search analytics insights and can operate independently of Fusion.
* \[3]: Commerce Studio does not require signals data, but key features like product analytics depend on it.
* \[4]: Future support is planned for alternative signals data sources.
* \[5]: Commerce Studio does not require signals data, but key features like product analytics depend on it.
## Who is Commerce Studio for?
Commerce Studio is designed for business users and ecommerce teams, including directors of ecommerce, customer experience managers, product managers, merchandisers, and digital marketers. The ecommerce teams can use Commerce Studio to build search and browse experiences to drive relevant search results to end users, while leveraging robust review and publishing workflows.
Depending on the size and structure of your organization, one person may cover several of these roles.
To use Commerce Studio, you must be granted **Workspace Owner** permissions in a Lucidworks Platform environment.
### Director of Ecommerce
Use Commerce Studio to help develop and execute the overall ecommerce strategy. This includes all aspects of the online shopping experience, from site performance and customer experience to search optimization and revenue generation.
* Optimize the search and browse experience and identify areas for improvement.
* Develop and implement search optimization strategies to drive organic traffic and achieve revenue goals.
* Analyze customer data and behavior to identify current and emerging trends and preferences.
* Optimize ecommerce operations for efficiency and cost-effectiveness.
### Customer Experience Manager
Use Commerce Studio to ensure customers have a seamless and satisfying search-to-purchase experience.
* Conduct user testing and research to understand customer preferences and pain points.
* Implement changes to the search experience based on customer insights.
* Analyze search data to understand customer intent and preferences.
* Identify and address factors contributing to "no results found" queries (also known as "null results" queries).
### Product Manager
Use Commerce Studio to oversee product assortment and strategy across all digital channels. Ensure that products are optimally positioned to maximize visibility, customer experience, and sales.
* Develop strategies to ensure that products are easily discoverable by customers.
* Strike a balance between promoting trending products and ensuring that evergreen products remain visible.
* Monitor inventory levels and ensure that in-stock products are prominently displayed and out-of-stock products are buried or hidden.
* Consider bundling or cross-selling to promote slow-moving items.
### Merchandiser
Use Commerce Studio to oversee product merchandising across all digital storefronts. Ensure that products are effectively positioned to maximize sales and conversions, drive customer engagement, and maintain brand consistency.
* Develop and implement merchandising strategies to increase product visibility and discoverability on the website.
* Strategically place products on the website to maximize sales and conversions.
* Develop and execute merchandising plans for seasonal promotions.
* Monitor the performance of seasonal promotions and make necessary adjustments to optimize results.
### Digital Marketer
Use Commerce Studio to drive traffic to the site through marketing strategies. Ensure that digital marketing campaigns align with search results and maximize return on marketing campaigns align with search results and maximize return on ad spend.
* Develop marketing campaigns that align with product launches or promotions.
* Optimize product pages and website content for relevant keywords to ensure that promotions and new products appear prominently in search results.
* Monitor and adjust strategies to maximize visibility during promotional periods.
### Data Analyst
Use Commerce Studio to analyze search data and customer behavior. Uncover valuable insights that influence the merchandising strategies of the team.
* Track and analyze key search metrics.
* Identify areas of strength and weakness in search performance.
* Recommend data-driven merchandising strategies.
* Monitor performance over time to adjust strategies and recommendations.
# Authentication API
Source: https://doc.lucidworks.com/docs/lw-platform/lw-platform/authentication-api
[old doc.lw link]: https://doc.lucidworks.com/lw-platform/ai/oa96k5
[localhost link]: http://localhost:3000/docs/lw-platform/lw-platform/authentication-api
[mintlify link]: https://doc.lucidworks.com/docs/lw-platform/lw-platform/authentication-api
The Authentication API authenticates users to use Lucidworks Platform APIs, including the [Lucidworks AI APIs](/docs/lw-platform/lw-ai/lw-ai-apis/overview) and [Commerce Studio Rules and Rewrites API](/docs/lw-platform/lw-cs/cs-rules-rewrites-api).
The single endpoint generates an access token to use in future API requests.
## Token guidelines and behavior
* Lucidworks uses OAuth 2.0 to authenticate your credentials.
* Tokens generated by the Authentication API are [Java Web tokens (JWT) tokens](https://jwt.io/).
* Every token generated is active for 3600 seconds before it expires. After that token expires, API requests attempting to use the token fail and generate an HTTP status `401 Unauthorized` error.
* You can generate a new token at any time. To avoid authorization failures, generate a new token before the current token expires. For example, generate a new token five minutes before the current token expires.
To calculate the token expiration time, Base64 decode the JWT and review the `exp` field, which contains the expiration time in seconds since the Unix epoch.
## Authentication scope
You must specify one or more values for the `scope` of each authentication request.
The values depend on which API you use:
| Scope | Used For |
| - | - |
| `machinelearning.model` | Models API |
| `machinelearning.predict` | Use Case API, Prediction API, Async Prediction API, and Async Chunking API |
| `commercestudio.rules` | Commerce Studio rule operations (create, read, update, delete rules) |
| `commercestudio.rewrites` | Commerce Studio query rewrite operations (create, read, update, delete rewrites) |
Because the Use Case API, Prediction API, Async Prediction API, and Async Chunking API use the `machinelearning.predict` scope, if you request a new token with that scope, it deletes the existing token, so all of the APIs are affected.
You need to manage token regeneration for those APIs to run successfully.
## Fetch the access token
Where you find the client ID and secret depends on which API you're using:
* For [Lucidworks AI APIs](/docs/lw-platform/lw-ai/lw-ai-apis/overview), go to **Models** > **Integrations** and click your integration, then copy the **Client ID** and **Client Secret**. [Learn more about Lucidworks AI integrations](/docs/lw-platform/lw-ai/integrations).
* For [Commerce Studio APIs](/docs/lw-platform/lw-cs/cs-rules-rewrites-api), go to **Settings** > **Service Accounts** and click your service account, then copy the **Client ID** and **Client Secret**. [Learn how to create a service account](/docs/lw-platform/lw-cs/cs-rules-rewrites-api#create-a-service-account).
Send your access token request to the authentication endpoint.
The request uses basic authentication, where your base64-encoded client ID is your username and your base64-encoded client secret is your password.
Some clients perform the encoding automatically; if not, you can use the `base64` command-line utility, like this:
```bash theme={"dark"}
echo -n "CLIENT_ID:CLIENT_SECRET" | base64
```
Compose your request like this:
```bash theme={"dark"}
curl --request POST \
--url 'https://identity.lucidworks.com/oauth2/ausao8uveaPmyhv0v357/v1/token?scope=API_SCOPE&grant_type=client_credentials' \
--header 'Accept: application/json' \
--header 'Authorization: Basic [CLIENT_ID:CLIENT_SECRET_BASE64]' \
--header 'Cache-Control: no-cache' \
--header 'Content-Type: application/x-www-form-urlencoded'
```
The value of the `scope` query parameter varies between APIs, and you need a separate access token for each scope:
* The Models API uses the `machinelearning.model` scope.
* All other APIs use the `machinelearning.predict` scope.
A successful request returns:
```json theme={"dark"}
{
"token_type": "Bearer",
"expires_in": 3600,
"access_token": "ACCESS_TOKEN",
"scope": "API_SCOPE"
}
```
Copy the value of the `access_token` key and use it for API authentication.
API access tokens expire after 60 minutes.
Once you request a new token, the previous one is deleted *even if it has not yet expired*.
All subsequent API calls must use the new token.
# Indexes
Source: https://doc.lucidworks.com/docs/lw-platform/lw-platform/indexes
The Indexes screen in Lucidworks Platform is where you can view your Fusion instances and perform promotions between environments. For Lucidworks Search clients, Lucidworks sets up and configure Fusion integrations for Lucidworks Platform. Self-hosted Fusion clients must set up [Fusion integrations](/docs/lw-platform/lw-platform/settings/fusion-integrations/overview) to view their Fusion instances here.
Navigate to the megamenu and click **Indexes** to display the screen.
## Fusion instances
The Fusion Instances tab displays by default, indicating if the deployment is self-hosted or Lucidworks Search and the environment such as Dev, Staging, or Prod.
Using multiple Fusion environments lets you to try out new features, test configurations, and stage changes before pushing them into production. Making your changes in a Fusion environment for developers allows your team to test any changes thoroughly and make sure everything works as expected before promoting to an environment where more people, including your end users, can view the changes. These environments protect your website from a situation where a simple configuration error may affect your users, security, and operational goals.
### Accessing Lucidworks Search instances
If you're using [Lucidworks Search](/docs/lucidworks-search/overview), you can access Fusion instances using SSO directly from Lucidworks Platform.
Click **Managed** on a Fusion instance card to authenticate automatically.
If you need elevated access to perform administrative tasks in an instance of Lucidworks Search, request role elevation from Lucidworks Support or your Customer Success Manager.
After the role is granted, log in again with Lucidworks Platform to activate your new permissions.
### Accessing Grafana metrics
If you're using Lucidworks Search, you can access Grafana monitoring using SSO from Lucidworks Platform.
Click the three dots on your Fusion instance card and select **View Metrics** to access your [Customer Dashboard](/docs/lucidworks-search/08-administration/monitoring/lucidworks-search/grafana-customer-dashboard-managed).
## Configuration promotion
This feature is available to both self-hosted and Lucidworks Search clients using Fusion 5.9.8 or later.
The Configuration Promotion tool uses a GitHub configuration for every Fusion instance you want to manage.
To use this feature, self-hosted Fusion clients must set up
[ConfigSync](/docs/5/fusion/operations/config-sync/overview) and complete all of the procedures described in the
[Configuration Promotion tool integration with Fusion](/docs/lw-platform/lw-platform/settings/fusion-integrations/configuration-promotion-integration) topic.
### Create and manage promotions
To create and manage promotions, navigate to the Indexes screen and click the **Configuration Promotion** tab.
#### Promotion statuses
The Configuration Promotion table displays an Approval Status for each promotion request. The following statuses indicate the current state of a promotion:
* **Creating Pull Request**: The system is generating the pull request for the promotion.
* **Pending**: The pull request has been created and is awaiting review.
* **Approved**: The promotion has been reviewed and approved for merging.
* **Merged**: The changes have been successfully applied to the target environment.
* **Withdrawn**: The promotion request has been canceled before merging.
* **Broken**: The promotion encountered an error and cannot proceed. Contact your administrator for assistance.
#### Start promotion
For Lucidworks Search clients, promotion requests are reviewed by Lucidworks. You can track the approval status in the Configuration Promotion table.
To promote an instance, click **Start Promotion**.
1. In the **Source** section, select the Fusion instance that contains the changes you want to promote. For example, if you’re promoting changes to a Staging environment, select your Dev environment. A file tree appears in the Source section that represents all the Fusion objects within the selected system.
2. Select the checkboxes for the files you want to promote into the target instance.
Some files are restricted and cannot be selected for promotion.
3. In the **Target** section, select the Fusion instance where the changes will be promoted and then verify your selection.
4. Click **Request Promotion**.
On the Configuration Promotion page, a new promotion appears with an Approval Status of `Creating Pull Request`. When the Approval Status changes to `Pending`, the pull request has been created and is ready to be reviewed.
#### Withdraw promotion
To cancel a promotion request that hasn’t been merged yet, click the request to display the Details screen and then click **Withdraw Promotion**.
# Platform Features
Source: https://doc.lucidworks.com/docs/lw-platform/lw-platform/overview
Lucidworks Platform
[old doc.lw link]: https//doc.lucidworks.com/lw-platform/platform/rcwjfo
[localhost link]: http://localhost:3000/docs/lw-platform/lw-platform/overview
[mintlify link]: https://doc.lucidworks.com/docs/lw-platform/lw-platform/overview
Lucidworks Platform connects everything behind the scenes so your team can focus on what drives value: smarter experiences and better outcomes.
Seamlessly integrated with Fusion, it provides powerful, no-code tools like [Lucidworks AI](/docs/lw-platform/lw-ai/overview), [Commerce Studio](/docs/lw-platform/lw-cs/overview), [Analytics Studio](/docs/lw-platform/lw-analytics/overview), and [Agent Studio](/docs/lw-platform/lw-agent-studio/overview) that let merchandisers, analysts, and AI innovators launch and optimize high-converting experiences with speed and confidence.
Click your use case below to see examples of how you can enhance the search experience using Lucidworks Platform:
* **Commerce Studio:**
Boost conversion rates by giving merchandisers no-code tools to fine-tune product rankings, promotions, and rules based on real-time shopper behavior.
* **Analytics Studio:**
Identify underperforming queries and high-value trends to optimize search and category experiences for maximum revenue impact.
* **Agent Studio:**
Power AI-driven support that delivers faster, more accurate answers by training agents on semantically chunked product and help content.
* **Lucidworks AI:**
Deploy scalable, cloud-hosted AI models that personalize search results, recommendations, and content based on user intent and behavior—without the infrastructure overhead.
* **Commerce Studio:**
Customize product visibility and ranking by buyer type, region, or industry to better serve complex purchase journeys.
* **Analytics Studio:**
Analyze buyer behavior and search trends to improve lead qualification, refine content strategy, and increase deal velocity.
* **Agent Studio:**
Create AI agents that surface relevant specs, compliance data, and product insights from complex content libraries so buyers can verify requirements faster.
* **Lucidworks AI:**
Centralize control of AI model lifecycles for industrial-scale deployments—enabling more accurate relevance, faster updates, and streamlined governance across product catalogs and document repositories.
* **Commerce Studio:**
Organize and prioritize knowledge artifacts so users and partners can access the most relevant information quickly and confidently.
* **Analytics Studio:**
Uncover content gaps and measure knowledge findability to continuously improve documentation and training resources.
* **Agent Studio:**
Deliver context-aware, role-specific answers with AI agents trained on semantically segmented internal knowledge, driving faster onboarding and support.
* **Lucidworks AI:**
Scale intelligent search and question-answering capabilities across vast corpora by managing models, chunking, and vector indexing from one unified cloud interface.
## Platform Home screen
When you access Lucidworks Platform, the Home screen displays:
From the Home screen, you can access all of your sites, plus Lucidworks Platform products your organization has deployed, such as [Lucidworks AI](/docs/lw-platform/lw-ai/overview), [Commerce Studio](/docs/lw-platform/lw-cs/overview), [Analytics Studio](/docs/lw-platform/lw-analytics/overview), and [Agent Studio](/docs/lw-platform/lw-agent-studio/overview).
## Megamenu
The megamenu lets you access and manage global functions.
It's available on every screen in Lucidworks Platform.
Different options display based on Lucidworks Platform features and components your organization deploys.
[Signals](/docs/lw-platform/lw-analytics/signals/overview) populate metrics in Analytics Studio. This option lets you view and manage the configuration for signals that collect and display customer interaction data from your website.
The [Indexes](/docs/lw-platform/lw-platform/indexes) option lets you access and interact with Fusion instances that are registered with Lucidworks Platform. These include Fusion and Lucidworks search instances that provide links to the webpage and additional actions such as [performing promotions between environments](/docs/lw-platform/lw-platform/indexes). For example, after you test the configuration and performance of Lucidworks Platform in the Development (Dev) environment, you can promote it to Production (Prod) for your customers to use.
**Models** is the option to manage integrations with Fusion and deploy pre-trained embedding models, custom embedding models, and generative AI models for [Lucidworks AI](/docs/lw-platform/lw-ai/overview).
Under [Settings](/docs/lw-platform/lw-platform/settings/overview), you can [create and manage user accounts](/docs/lw-platform/lw-platform/settings/lw-users/overview) and [configure integrations between Fusion and Lucidworks Platform and its products](/docs/lw-platform/lw-platform/settings/fusion-integrations/overview).
## Feature compatibility and availability
The table below lists the available Lucidworks Platform offerings with their dependencies and licensing requirements.
| Feature | Dependency |
| - | - |
| Lucidworks AI | Fusion or Lucidworks Search 5.9.5 or later |
| Lucidworks Platform | Fusion or Lucidworks Search 5.9.5 or later |
| Neural Hybrid Search with Lucidworks AI | Fusion or Lucidworks Search 5.9.5 or later |
| Neural Hybrid Search with Seldon | Fusion or Lucidworks Search 5.9.5 or later |
| Enterprise SSO | Fusion or Lucidworks Search 5.9.5 or later |
| Commerce Studio | Fusion or Lucidworks Search 5.9.8 or later |
| Cloud Signal Storage | Lucidworks Search 5.9.6 or later |
| Signals beacon | |
| Analytics Studio | Fusion or Lucidworks Search 5.9.8 or later. Also requires the Signals beacon or Signals API. |
Contact Lucidworks for access to the features you need.
## Getting help
You can open the Help Center anywhere in Lucidworks Platform to access resources from our Knowledge Services and Technical Support teams, plus announcements about the latest products and capabilities.
Ready for your first training module? Start here:
Learn about Lucidworks Platform in this free training course.
# Rate limiting
Source: https://doc.lucidworks.com/docs/lw-platform/lw-platform/rate-limiting
Rate limiting is used to help control network traffic and performance based on configuration of the timing and repetition of requests allowed on the network.
Lucidworks Platform employs rate limiting to:
* Provide a reliable service and optimal network performance for client organizations.
* Protect clients from exceeding contracted consumption quotas which could result in overage charges.
Each of the Lucidworks Platform APIs uses a similar approach:
* Rate limitations are specified in the Lucidworks contract for your organization. Typically, most of the simple operations that do not impact query time are set to a lower rate limit. APIs considered to be in the critical path for query performance use a separate, higher rate limit.
* If the request rate to a particular API exceeds the relevant limit, the system generates the [HTTP 429 Too Many Requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429) error. To resolve the issue, Lucidworks recommends you retry the request by implementing an [exponential backoff pattern](https://en.wikipedia.org/wiki/Exponential_backoff).
For more information, or to ask questions about your account related to rate limiting, contact [Lucidworks Support](https://support.lucidworks.com).
# Commerce Studio integration with Fusion
Source: https://doc.lucidworks.com/docs/lw-platform/lw-platform/settings/fusion-integrations/commerce-studio-integration
This type of Fusion integration lets you connect self-hosted Fusion instances to Lucidworks Commerce Studio.
If you want to migrate from Predictive Merchandiser to Commerce Studio, the information in this topic is used in coordination with the procedures described in [Migration guide from Predictive Merchandiser to Commerce Studio](/docs/lw-platform/lw-cs/cs-migrate).
You must satisfy the [prerequisites](#prerequisites) and complete all of the procedures in the order described in this topic to implement Lucidworks Commerce Studio:
1. [Create a Commerce Studio integration with Fusion](#create-a-commerce-studio-integration-with-fusion)
2. [Create Fusion roles required for Commerce Studio](#create-fusion-roles-required-for-commerce-studio)
3. [Create a Fusion security JWT realm for Commerce Studio](#create-a-fusion-security-jwt-realm-for-commerce-studio)
4. [Create the lwp service account in the JWT realm](#create-the-lwp-service-account-in-the-jwt-realm)
5. [Verify Commerce Studio integration with Fusion](#verify-commerce-studio-integration-with-fusion)
6. [Create a Commerce Studio instance using the UI](#create-a-commerce-studio-instance-using-the-ui)
This topic is specific to integrating Lucidworks Platform with Fusion for the use of Commerce Studio.
* To configure a general integration for Lucidworks Platform with Fusion, follow the procedures in [Fusion integrations](/docs/lw-platform/lw-platform/settings/fusion-integrations/overview).
* To configure an integration that uses the Lucidworks Configuration Promotion tool, follow the procedures in the [Configuration Promotion tool integration with Fusion](/docs/lw-platform/lw-platform/settings/fusion-integrations/configuration-promotion-integration) topic.
* For information about a Fusion integration with Lucidworks AI, see [Lucidworks AI integrations](/docs/lw-platform/lw-ai/integrations).
## Prerequisites
To create an integration, you must have the following prerequisites in place:
* Admin privileges to a self-hosted Fusion environment running Fusion 5.9.8 or later.
* A Fusion application that contains usable data.
* Workspace Owner permissions in a Lucidworks Platform environment.
* Privileges to access and use Lucidworks Commerce Studio in a Lucidworks platform workspace.
**Network access requirements:** Lucidworks Platform must be able to reach the Fusion API URL you provide. Your Fusion instance must also be able to reach the Lucidworks JWK endpoint (`https://api.lucidworks.com/customers/`) to retrieve JSON Web Key Sets during JWT authentication. This endpoint is hosted on a separate IP range from the Lucidworks Platform itself. To obtain the IP ranges to add to your network allowlist, or for help configuring proxy access through the Fusion API gateway, contact Lucidworks support.
## Create a Commerce Studio integration with Fusion
Complete the following procedures to create an integration from Lucidworks Commerce Studio to Fusion.
1. Navigate to the megamenu and click **Settings > Fusion Integrations**.
2. Click **Connect New Fusion**.
3. In the **Name** field, enter a unique name for the integration. For example, `Lucidworks Demo Dev`.
4. In the **API URL** field, enter the URL of your Fusion instance. For example, `https://FUSION_INSTANCE.com`. Do not enter a path after `.com`.
5. In the **Environment** field, select the Fusion instance environment or use the default of **Development**.
6. Optional: Enter one or more descriptive **Tags** that differentiate multiple integrations.
7. Do not enter values in any of the fields in the **GitHub** section. Values in these fields are only required to set up the [Configuration Promotion tool integration with Fusion](/docs/lw-platform/lw-platform/settings/fusion-integrations/configuration-promotion-integration).
8. Click **Add**. Lucidworks Platform sends a `GET /api` request to your Fusion API gateway to validate the connection. If that request cannot reach your Fusion instance, the integration will not be created. If the value in the **API URL** field is:
* Not valid, the integration is not created, and it is not displayed on the Fusion Integrations screen.
* Valid, the Instructions to Connect with Fusion screen displays:
9. Copy both the **JWK Set URL** value and the **Issuer URL** value to a secure location. These values must be used to create the JWT realm and are not displayed again after you click **Close**.
10. Click **Close**.
11. Verify the new integration displays on the Fusion Integrations screen.
To edit the integration, hold the pointer over the integration, click the pencil icon, enter the changes, and click **Save**. To delete the integration, hold the pointer over the integration and click the trash can icon.
## Create Fusion roles required for Commerce Studio
1. Sign in to your Fusion instance as an admin and click the Fusion application to connect with Commerce Studio.
2. Navigate to **System > Access Control > Roles** and click **Add Role**.
3. In the **Name** field, enter `lwp_user`.
4. In the **Permissions** field, enter:
```bash lines highlight={3-4} theme={"dark"}
GET,PUT,DELETE,POST,HEAD:/query/experience-manager/**
GET,POST:/query/*
POST:/query/*/preview
GET,POST,PUT:/query-pipelines/**
GET:/suggestions/**
GET:/solr/*/terms
PUT,POST,GET:/apps/*/query-pipelines/**
GET:/apps/*/query-profiles/**
GET,PUT:/apps/*
POST:/apps/*/query/**
GET:/solr/*/admin/luke
GET:/solr/**
```
Each permission grants the following access:
* `GET,PUT,DELETE,POST,HEAD:/query/experience-manager/**`: Commerce Studio query endpoints (read/write) for Commerce Studio UI and rule management.
* `GET,POST:/query/*`: Query execution endpoints (read/write). `POST` sends query parameters in the request body and does not create or persist any configuration in Fusion.
* `POST:/query/*/preview`: Rule preview endpoint (write) to view rule results without publishing them to production.
* `GET,POST,PUT:/query-pipelines/**`: Query pipeline configuration (read/write) to manage pipeline stages used by Commerce Studio.
* `GET:/suggestions/**`: Autocomplete and typeahead suggestions (read-only).
* `GET:/solr/*/terms`: Solr term analysis (read-only) to view field values and vocabulary.
* `PUT,POST,GET:/apps/*/query-pipelines/**`: Application-scoped query pipeline management (read/write) to manage app-specific pipeline operations.
* `GET:/apps/*/query-profiles/**`: Query profiles (read-only) to view search configuration profiles.
* `GET,PUT:/apps/*`: Fusion application settings (read/write) to view and update app metadata.
* `POST:/apps/*/query/**`: Run queries (write) against a specified app's query profile without modifying the app.
* `GET:/solr/*/admin/luke`: Solr index schema inspection (read-only) used to view available fields.
* `GET:/solr/**`: General Solr read access (read-only).
Beginning with Fusion 5.9.15, all Commerce Studio integrations must have the `POST:/query/*/preview` and `GET,POST,PUT:/query-pipelines/**` permissions assigned to both the `lwp_user` and `lwp_service_account` roles in Fusion.
This role does not require any UI permissions, so you do not have to select any values in the **UI Permissions** field.
5. In the **Description** field, enter role information. For example, "The lwp\_user account is required to access and use Commerce Studio. It is a super user account, but does not include the ability to get, modify, or delete the `HEAD` section of the site."
6. Click **Save**.
7. Click **Add Role**.
8. In the **Name** field, enter `lwp_service_account`.
9. In the **Permissions** field, enter the following permissions. These permissions include most of the lwp\_user account permissions plus the privileges to migrate Fusion Predictive Merchandiser rewrites, rules, templates and zones.
```bash lines highlight={3-4} theme={"dark"}
GET,PUT,DELETE,POST,HEAD:/query/experience-manager/**
GET,POST:/query/*
POST:/query/*/preview
GET,POST,PUT:/query-pipelines/**
GET:/suggestions/**
GET:/solr/*/terms
PUT,POST,GET:/apps/*/query-pipelines/**
GET:/apps/*/query-profiles/**
GET,PUT:/apps/*
GET,DELETE,PATCH,PUT,POST,HEAD:/apps/*/query-rewrite/**
GET,HEAD:/templating/**
GET:/solr/*/admin/luke
GET:/solr/**
```
The `lwp_service_account` role includes most `lwp_user` permissions plus the following additional permissions that are required to migrate Predictive Merchandiser to Commerce Studio:
* `GET,DELETE,PATCH,PUT,POST,HEAD:/apps/*/query-rewrite/**`: Query rewrite rules (read/write) to migrate Predictive Merchandiser rewrites, rules, and zones to Commerce Studio.
* `GET,HEAD:/templating/**`: Templating access (read-only) to access Predictive Merchandiser templates during migration.
Beginning with Fusion 5.9.15, all Commerce Studio integrations must have the `POST:/query/*/preview` and `GET,POST,PUT:/query-pipelines/**` permissions assigned to both the `lwp_user` and `lwp_service_account` roles in Fusion.
This role does not require any UI permissions, so you do not have to select any values in the **UI Permissions** field.
10. In the **Description** field, enter role information. For example, "The lwp\_service\_account contains most of the lwp\_user permissions as well as the privileges to migrate Fusion Predictive Merchandiser components to Lucidworks Commerce Studio."
11. Click **Save**.
The `PUT,POST,GET:/apps/*/query-pipelines/**` permission reflects the requirement to assign all necessary query pipeline permissions to the `lwp_user` and `lwp_service_account` roles. An additional example is for configuring a Lucidworks AI Gateway integration, where you need to assign `POST:/your-account-name/`.
## Create a Fusion security JWT realm for Commerce Studio
1. Sign in to your Fusion instance as an admin and click the application in which you created the `lwp_user` and `lwp_service_account` accounts to use with Commerce Studio.
2. Navigate to **System > Access Control > Security Realms** and click **Add Security Realm**.
3. In the **Name** field, enter a unique connection name. For example, **Lucidworks Commerce Studio JWT realm**.
4. In the **Type** field, select **jwt**.
5. Set the checkboxes in the following configuration.
* Select the **Enabled** checkbox.
* Do not select the **Ephemeral Users** checkbox.
* Select the **Auto-Create Users** checkbox.
6. In the **Roles** field, select `lwp_user`.
7. In the **JWT Issuer** field, paste the **Issuer URL** value generated in the [Create a Commerce Studio integration with Fusion](#create-a-commerce-studio-integration-with-fusion) procedure.
8. In the **JWK Set URL** field, paste the **JWKS URL** value generated in the [Create a Commerce Studio integration with Fusion](#create-a-commerce-studio-integration-with-fusion) procedure.
9. In the **JWK Set Timeout** field, enter a timeout value in milliseconds for retrieving the JWKS.
* This field is required.
* Minimum value: `500` ms (set higher for known network latency).
10. Do not enter values (default is blank) in any of the following fields:
* **Signing Key**
* **Groups Key**
* **Groups Mapping**
* **User ID Attribute**
11. Click **Save**.
Self-hosted Fusion clients can also view the integration `ID`, `Issuer URL`, `JWKS URL`, and other integration details in **Settings > Fusion Integrations** when the existing integration is selected. For general information, see [Lucidworks Platform integration with Fusion](/docs/lw-platform/lw-platform/settings/fusion-integrations/overview). Integrations for Lucidworks Search clients are created and updated by Lucidworks.
## Create the lwp service account in the JWT realm
1. Sign in to your Fusion instance as an admin and click the application to be connected with Commerce Studio.
2. Navigate to **System > Access Control > Users** and click **Add User**.
3. In the **Realm** field, select the JWT realm you created in the [Create a Fusion security JWT realm for Commerce Studio](#create-a-fusion-security-jwt-realm-for-commerce-studio).
4. In the **Username** field, enter `lwp-service-account`.
5. In the **Roles** field, select the `lwp_service_account` role.
6. In the **Timezone** field, select the appropriate timezone for the user account.
7. Click **Create**.
## Verify Commerce Studio integration with Fusion
When the Fusion security JWT realm is created, complete the following steps to verify the Fusion information is displayed.
1. Navigate to the megamenu and click **Settings > Fusion Integrations**.
2. Hold the pointer over the integration and click the pencil icon.
3. Verify the information is accurate.
The information in the Instructions section must match the values in the **Additional URLs JWKS URL** and **Issuer URL** fields.
## Create a Commerce Studio instance using the UI
Do not create a Commerce Studio instance using the UI if you are migrating data from Predictive Merchandiser into that Commerce Studio instance. Instead, you must create the [instance using the API](/docs/lw-platform/lw-cs/cs-migrate).
1. In Lucidworks Platform, navigate to **Commerce Studio** and click **Create New**.
2. In the **Name** field, enter a unique name for the Commerce Studio instance.
3. In the **Region** field, select the geographic region for the Commerce Studio instance. This cannot be changed after the instance is created.
4. In the **Fusion to Connect** field, select the Fusion integration instance that matches the **Name** field created in the [Create a Commerce Studio integration with Fusion](#create-a-commerce-studio-integration-with-fusion) procedure.
5. Optional: In the **Signal Store** field, select the location where the Commerce Studio signals are stored. This value is only required if you want to use Lucidworks Analytics.
6. In the **Fusion Application** field, enter Lucidworks Platform application that connects to your Fusion instance.
7. Click **Next** to confirm settings, and then **Next** to create the instance.
Data from the Fusion app is sent to Commerce Studio so you can create rules and refine the search experience. You can only connect one instance of Commerce Studio to a Fusion app. You cannot create multiple instances of Commerce Studio using the same Fusion app.
1. Commerce Studio exclusively manages `APP_NAME_query_rewrite_staging_em` and `APP_NAME_query_rewrite_em`. Do not manually add, edit, or delete documents in these collections.
2. Commerce Studio reflects rules stored in its own platform database. Deleting documents from these Fusion Solr collections does not delete the corresponding rules in Commerce Studio. The collections will re-sync the next time Commerce Studio writes rules.
3. Custom tooling built against these collections is not supported.
## Troubleshooting
These errors may occur when Commerce Studio communicates with Fusion.
When an error response includes a `fusionErrorMessage` field, use the message contents to identify the cause.
The Commerce Studio instance status in Fusion may have been removed or disabled.
Manually set the status back to `READY` (if in preview mode) or `CONNECTED` (if already live).
The JWT security realm used by Commerce Studio is not configured correctly.
Verify that the JWT realm is configured as described in [Create a Fusion security JWT realm for Commerce Studio](#create-a-fusion-security-jwt-realm-for-commerce-studio).
If the realm configuration is correct, an expired JWT may be reaching Fusion.
Contact [Lucidworks support](https://lucidworks.com/support/).
In some network environments, a JWT realm misconfiguration, particularly when Fusion cannot reach the JWKS endpoint, can surface as a 500 error rather than a 401 or 403.
If you are investigating a 500 error and cluster health appears normal, also check the JWT realm configuration and network egress to the JWKS endpoint.
This typically indicates a problem with the Fusion instance itself.
Check the details of the returned error response to determine the cause:
* **"Could not find a healthy node"** — The Fusion cluster cannot reach a healthy Solr node. Contact your Fusion instance administrator or [Lucidworks support](https://lucidworks.com/support/).
* **"Connection reset", "interrupted", or "timed out"** — The Fusion cluster may be overloaded or have a configuration issue causing slow responses. Contact your Fusion instance administrator.
* **JWT realm / JWKS endpoint unreachable** — If Fusion cannot reach the Lucidworks JWKS endpoint, for example, due to a firewall or proxy blocking egress, it fails to retrieve JWT signing keys and may return a 500 to Commerce Studio instead of a 401 or 403. Check egress rules, proxy configuration, and the JWT realm setup even when the error is a 500. See [network access requirements](#prerequisites) and the JWT realm configuration steps.
# Configuration Promotion tool integration with Fusion
Source: https://doc.lucidworks.com/docs/lw-platform/lw-platform/settings/fusion-integrations/configuration-promotion-integration
This type of Fusion integration lets you connect self-hosted Fusion instances to the Lucidworks Configuration Promotion tool.
## Use cases
Configuration Promotion connects Fusion to a Git-backed promotion flow so you can version, review, and safely move configuration changes to pipelines, schemas, and rules from development to staging to production. It standardizes releases with approvals and audit trails, reducing risk while speeding up delivery.
### B2B use case
Configuration Promotion adds a controlled, auditable path for repeatable, low-risk releases.
For example, support and knowledge teams can reliably promote new index pipelines and security configurations after a pull request is approved.
### B2C use case
For consumer search applications, Configuration Promotion shortens time-to-market by enforcing controlled promotions of relevancy rules, pipelines, and schemas across environments.
For example, merchandising teams can test ranking tweaks in staging and promote them safely to production during a release window, following the same GitHub checks every time.
This topic is specific to integrating the Lucidworks Platform with Fusion for the use of the Configuration Promotion tool.
* To configure a general integration for Lucidworks Platform with Fusion, follow the procedures in [Fusion integrations](/docs/lw-platform/lw-platform/settings/fusion-integrations/overview).
* To configure an integration that uses Lucidworks Commerce Studio, follow the procedures in the [Commerce Studio integration with Fusion](/docs/lw-platform/lw-platform/settings/fusion-integrations/commerce-studio-integration) topic.
* For information about a Fusion integration with Lucidworks AI, see [Lucidworks AI integrations](/docs/lw-platform/lw-ai/integrations).
## Prerequisites
To create an integration, you must have the following prerequisites in place:
* Admin privileges to a self-hosted Fusion environment running Fusion 5.9.8 or later.
* A Fusion application that contains usable data.
* Workspace Owner permissions in a Lucidworks Platform environment.
You must complete all of the procedures in the order described in this topic to implement the Lucidworks Configuration Promotion tool.
1. [Create a GitHub token for configuration promotion](#create-a-github-token-for-configuration-promotion)
2. [Set up GitHub branch protection (optional)](#set-up-github-branch-protection-optional)
3. [Configure the integration](#configure-the-integration)
4. [Create a Fusion security JWT realm](#create-a-fusion-security-jwt-realm)
5. [Verify the Configuration Promotion tool integration with Fusion](#verify-the-configuration-promotion-tool-integration-with-fusion)
## Create a GitHub token for configuration promotion
To set up the Configuration Promotion tool, you must generate a GitHub token.
1. Sign in to GitHub with an account that has access to the integrated systems repository.
2. In GitHub, click your profile icon, and then click **Settings**.
3. Navigate to **Developer Settings > Personal Access Tokens > Tokens (classic)**.
4. Click **Generate new token** and then select **Generate new token (classic)**.
5. In the **Note** field, enter details about the token.
6. In the **Expiration** field, select the period of time the token is active. Lucidworks recommends configuring a long duration in line with your organization’s security practices. You must generate and configure a new token for your Fusion integration every time your token expires.
7. In the **Select scopes** section, select the **repo** checkbox. This ensures the token has all required permissions.
8. Click **Generate token**.
9. Copy the generated token and save it in a secure location.
Your GitHub organization may need to authorize your token.
For more information, see [Managing your personal access tokens](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens).
## Set up GitHub branch protection (optional)
Lucidworks recommends enabling GitHub branch protection for your production (PROD) branch. This protects a branch with requested changes from being promoted until that request has been reviewed.
To configure branch protections, see [About protected branches](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches).
The configuration promotion tool looks at GitHub to see if the branch is ready to merge. The tool can integrate with any GitHub automations that change the merge-ready status of the branch.
## Configure the integration
To create an integration for the Lucidworks Configuration Promotion tool with Fusion, navigate to the megamenu and click **Settings > Fusion Integrations**.
1. Click **Connect New Fusion**.
2. In the **Name** field, enter a unique name for the integration. For example, `Lucidworks Demo Dev`.
3. In the **API URL** field, enter the URL of your Fusion instance. For example, `https://FUSION_INSTANCE.com`. Do not enter a path after `.com`.
4. In the **Environment** field, select the Fusion instance environment or use the default of **Development**.
5. Optional: Enter one or more descriptive **Tags** that differentiate multiple integrations.
6. All fields in the **GitHub** section are required if you’re setting up the Configuration Promotion tool.
1. In the **GitHub Branch URL** field, enter the URL from GitHub for your self-hosted Fusion install. The format is `https://github.com/ORGANIZATION_NAME/REPOSITORY_NAME/tree/BRANCH_NAME`. For example, `https://github.com/company_name/fusion_install/tree/lucidworks_demo_dev`.
2. In the **GitHub Token** field, enter the [personal access token](#create-a-github-token-for-configuration-promotion) granted privileges to the GitHub organization linked to your self-hosted Fusion install.
7. Click the **Enable Fusion Configuration Promotion** field to **ON** to allow files or instances to be promoted from development to a staging or production environment.
8. Click **Add**. If the value in the **API URL** field is:
* Not valid, the integration is not created, and it is not displayed on the Fusion Integrations screen.
* Valid, the Instructions to Connect with Fusion screen displays:
9. Copy both the **JWK Set URL** value and the **Issuer URL** value to a secure location. These values must be used to [Create a Fusion security JWT realm](#create-a-fusion-security-jwt-realm). These values must be used to create the JWT realm and are not displayed again after you click **Close**.
10. Click **Close**.
11. Verify the new integration displays on the Fusion Integrations screen.
To edit the integration, hold the pointer over the integration, click the pencil icon, enter the changes, and click **Save**. To delete the integration, hold the pointer over the integration and click the trash can icon.
## Create a Fusion security JWT realm
To complete required fields in the integration, you must create a Fusion security realm.
1. Sign in to your Fusion instance as an admin and click the Fusion application to connect with the Lucidworks Configuration Promotion tool.
2. Navigate to **System > Access Control > Security Realms** and click **Add Security Realm**.
3. In the **Name** field, enter a unique connection name. For example, **Lucidworks Configuration Promotion tool JWT realm**.
4. In the **Type** field, select **jwt**.
5. Set the checkboxes in the following configuration.
* Select the **Enabled** checkbox.
* Do not select the **Ephemeral Users** checkbox.
* Select the **Auto-Create Users** checkbox.
6. In the **Roles** field, select the roles that this realm provides after successfully authenticating a user by default.
7. In the **JWT Issuer** field, paste the **Issuer URL** value generated in the [Configure the integration](#configure-the-integration) procedure.
8. In the **JWK Set URL** field, paste the **JWKS URL** value generated in the [Configure the integration](#configure-the-integration) procedure.
9. Do not enter values (default is blank) in any of the following fields:
* **Signing Key**
* **Groups Key**
* **Groups Mapping**
* **User ID Attribute**
10. Click **Save**.
## Verify the Configuration Promotion tool integration with Fusion
When the Fusion security JWT realm is created, complete the following steps to verify the Fusion information is displayed.
1. Navigate to the megamenu and click **Settings > Fusion Integrations**.
2. Hold the pointer over the integration and click the pencil icon.
3. Verify the information is accurate.
The information in the Instructions section must match the values in the **Additional URLs JWKS URL** and **Issuer URL** fields.
# Fusion Integrations
Source: https://doc.lucidworks.com/docs/lw-platform/lw-platform/settings/fusion-integrations/overview
The Fusion Integrations features lets you connect self-hosted Fusion instances to Lucidworks Platform features and other software to extend and enhance your workflows.
This topic details how to create an integration between Lucidworks Platform and Fusion. You must satisfy the [prerequisites](#prerequisites) and complete the procedures in the order described in this topic:
1. [Create a Lucidworks Platform integration with Fusion](#create-a-lucidworks-platform-integration-with-fusion)
2. [Create a Fusion security JWT realm for Lucidworks Platform](#create-a-fusion-security-jwt-realm-for-lucidworks-platform)
3. [Verify the integration](#verify-the-integration)
This topic details the steps to configure a general integration for Lucidworks Platform with Fusion.
* To configure an integration that uses the Lucidworks Configuration Promotion tool, follow the procedures in the [Configuration Promotion tool integration with Fusion](/docs/lw-platform/lw-platform/settings/fusion-integrations/configuration-promotion-integration) topic.
* To configure an integration that uses Lucidworks Commerce Studio, follow the procedures in the [Commerce Studio integration with Fusion](/docs/lw-platform/lw-platform/settings/fusion-integrations/commerce-studio-integration) topic.
* For information about a Fusion integration with Lucidworks AI, see [Lucidworks AI integrations](/docs/lw-platform/lw-ai/integrations).
## Prerequisites
To create an integration between Lucidworks Platform and Fusion, you must have the following prerequisites in place:
* Admin privileges to a self-hosted Fusion environment running Fusion 5.9.8 or later.
* A Fusion application that contains usable data.
* Workspace Owner permissions in a Lucidworks Platform environment.
## Create a Lucidworks Platform integration with Fusion
Complete the following steps to create an integration from Lucidworks Platform to Fusion.
1. Navigate to the megamenu and click **Settings > Fusion Integrations**.
2. Click **Connect New Fusion**.
3. In the **Name** field, enter a unique name for the integration. For example, `Lucidworks Demo Dev`.
4. In the **API URL** field, enter the URL of your Fusion instance. For example, `https://FUSION_INSTANCE.com`. Do not enter a path after `.com`.
5. In the **Environment** field, select the Fusion instance environment or use the default of **Development**.
6. Optional: Enter one or more descriptive **Tags** that differentiate multiple integrations.
7. Do not enter values in any of the fields in the **GitHub** section. Values in these fields are only required to set up the [Configuration Promotion tool integration with Fusion](/docs/lw-platform/lw-platform/settings/fusion-integrations/configuration-promotion-integration).
8. Click **Add**. If the value in the **API URL** field is:
* Not valid, the integration is not created, and it is not displayed on the Fusion Integrations screen.
* Valid, the Instructions to Connect with Fusion screen displays:
9. Copy both the **JWK Set URL** value and the **Issuer URL** value to a secure location. These values must be used to create the JWT realm. These values must be used to create the JWT realm and are not displayed again after you click **Close**.
10. Click **Close**.
11. Verify the new integration displays on the Fusion Integrations screen.
To edit the integration, hold the pointer over the integration, click the pencil icon, enter the changes, and click **Save**. To delete the integration, hold the pointer over the integration and click the trash can icon.
## Create a Fusion security JWT realm for Lucidworks Platform
To complete required fields in the integration, you must create a Fusion security realm.
1. Sign in to your Fusion instance as an admin and click the Fusion application to connect with Lucidworks Platform.
2. Navigate to **System > Access Control > Security Realms** and click **Add Security Realm**.
3. In the **Name** field, enter a unique connection name. For example, **Lucidworks Platform JWT realm**.
4. In the **Type** field, select **jwt**.
5. Set the checkboxes in the following configuration.
* Select the **Enabled** checkbox.
* Do not select the **Ephemeral Users** checkbox.
* Select the **Auto-Create Users** checkbox.
6. In the **Roles** field, select the roles that this realm provides after successfully authenticating a user by default.
7. In the **JWT Issuer** field, paste the **Issuer URL** value generated in the [Create a Lucidworks Platform integration with Fusion](#create-a-lucidworks-platform-integration-with-fusion) procedure.
8. In the **JWK Set URL** field, paste the **JWKS URL** value generated in the [Create a Lucidworks Platform integration with Fusion](#create-a-lucidworks-platform-integration-with-fusion) procedure.
9. Do not enter values (default is blank) in any of the following fields:
* **Signing Key**
* **Groups Key**
* **Groups Mapping**
* **User ID Attribute**
10. Click **Save**.
## Verify the integration
When the Fusion security JWT realm is created, complete the following steps to verify the Fusion information is displayed.
1. Navigate to the megamenu and click **Settings > Fusion Integrations**.
2. Hold the pointer over the integration and click the pencil icon.
3. Verify the information is accurate.
The information in the Instructions section must match the values in the **Additional URLs JWKS URL** and **Issuer URL** fields.
# Users and Provisioning
Source: https://doc.lucidworks.com/docs/lw-platform/lw-platform/settings/lw-users/overview
The Users and Provisioning screen lets you [invite workspace owners](#invite-workspace-owners) to use Lucidworks Platform. In addition, you can [create authentication that supports sign-on (SSO)](/docs/lw-platform/lw-platform/settings/lw-users/sso), which restricts user logins to specified email domains.
## Users
The Users screen displays all the users associated with your workspaces.
To view a list of users, navigate to the megamenu and click **Settings > Users & Provisioning**. The name, status, date user was created, and email of each user displays.
First and last name of the user.
Possible values:
* **Active.** User has clicked the **Activate** link in the email and can access Lucidworks Platform as a workspace owner, which can manage the workspace and invite other users.
* **Awaiting Activation.** User is invited, but has not clicked the **Activate** link in the email.
* **Deleted.** User account has been deleted.
* **Deprovisioned.** User account has been deactivated and cannot access Lucidworks Platform.
* **Recovery.** User account was once deprovisioned, but has been recovered for future use.
* **Password-expired.** User must create a new password to reactivate the account.
Date the invitation was created.
User’s email address.
### Invite workspace owners
Only workspace owners for the specified workspace can invite users to that workspace.
1. Navigate to the megamenu and click **Settings > Users & Provisioning**.
2. Click **+ Invite** and enter the user’s information in the **First name**, **Last name**, and **Email** fields. Required fields are notated with an asterisk (\*).
3. To email the user the invitation, click **Send Invite**. To exit without creating a user, click **Cancel**.
4. In the Users list, verify the status of the new user status is **Awaiting Activation**.
When the user accepts the email invitation, the status displays as Active. If the user does not receive the email invitation in their inbox, ask them to look in the spam folder for an email from `no-reply@lucidworks.com`. If the invitation is not in their inbox or spam folder, verify the user's email information. If your troubleshooting efforts are unsuccessful, contact [Lucidworks Support](https://support.lucidworks.com).
### Edit or delete users
To edit a user, click the entry for the user you want to edit. You can also click **View/Edit**. Workspace owners can also click **Delete User** to delete the user.
## Authentication using single-sign on
The Authentication screen lets you configure a single sign-on (SSO) option, which supports the following features:
* Restricts user logins to specified email domains
* Uses your organization’s identity provider authentication method instead of Lucidworks' built-in authentication
Workspace owners can configure SSO via OpenID Connect (OIDC) or Security Assertion Markup Language (SAML).
For more information, see [Single sign-on authentication](/docs/lw-platform/lw-platform/settings/lw-users/sso).
# Single sign-on authentication
Source: https://doc.lucidworks.com/docs/lw-platform/lw-platform/settings/lw-users/sso
Lucidworks Platform supports single sign-on (SSO), which restricts user logins to specified email domains and uses your organization’s identity provider authentication method instead of Lucidworks' built-in authentication.
Workspace owners can configure SSO via [OpenID Connect (OIDC)](#oidc-authentication) or [Security Assertion Markup Language (SAML)](#saml-authentication).
## OIDC Authentication
[OIDC](https://openid.net/developers/how-connect-works/) is an authentication protocol based on the OAuth 2.0 framework, which uses HTTPS endpoints to retrieve user data. It uses JSON web tokens (JWT) to authenticate user identity data (also known as claims).
Tokens are digitally signed and can also be encrypted.
Complete all of the following procedures to configure OIDC authentication.
### Create users in Lucidworks Platform
**Prerequisites**
1. Users must be added to the Lucidworks Platform.
2. Users must exist in your identity provider, and be enabled for the application you are using to connect with Lucidworks Platform.
3. The user’s email address in your identity provider and Lucidworks Platform must match, including case. Some identity providers treat email addresses as case-sensitive, which can prevent a user from accessing Lucidworks Platform if the case doesn't match exactly.
1. Navigate to the megamenu and click **Settings > Users & Provisioning > Users**.
2. Verify all necessary users are listed, active, and their emails match their identity provider user counterparts.
3. If any users are missing, add the appropriate users by completing the steps in [invite workspace owners](/docs/lw-platform/lw-platform/settings/lw-users/overview#invite-workspace-owners).
### Create an OIDC application in your identity provider
1. Create a new OIDC application in your identity provider. Refer to your provider's documentation for detailed steps:
* [Okta](https://help.okta.com/en-us/content/topics/apps/apps_app_integration_wizard_oidc.htm)
* [Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/identity/enterprise-apps/add-application-portal-setup-oidc-sso#configure-oidc-sso-for-custom-non-gallery-applications)
* [Google Workspace](https://support.google.com/cloud/answer/6158849)
2. When creating the application, use the following settings:
* **Sign-in method**: Select **OIDC**.
* **Application type**: Select **Web Application**.
* **Sign-in redirect URIs**: Leave this blank for now. You’ll configure it in a later step.
* **Logout redirect URIs**: Remove all URIs.
* **Controlled access**: Allow everyone in your organization to access the application.
* **Access approval**: Disable any setting that grants users immediate access without admin approval.
3. After creating the application, copy the **Client ID** and the **Client Secret**, and save them in a secure location.
The **Client ID** and the **Client Secret** credentials are critical for OIDC communication and must be kept secure.
4. Assign at least one user and one admin to the application. These users must already exist in Lucidworks Platform. You must assign yourself as one of the users to be able to complete the authentication configuration.
### Create the Identity Provider in Lucidworks Platform
1. Navigate to the megamenu and click **Settings > Users & Provisioning > Authentication > OIDC**.
2. Enter values in the required fields:
* **Email Domains**: Enter the list of domains you want to use as filter options, for example, `lucidworks.com`.
Be careful not to lock yourself out at this step in the process. If you list a domain that you already use to log into Lucidworks Platform, and you don’t complete the rest of the steps listed here, then you will be prompted to log in via SSO the next time you log out, which won’t work. If you are unable to log into Lucidworks Platform, contact Support or your Lucidworks representative.
* Enter the **Client ID** and **Client Secret** provided by your identity provider's OIDC application.
* Enter the **Configuration URL** provided by your identity provider.
To get the full OIDC configuration URL, append `/.well-known/openid-configuration` to your Issuer URL. For example, if your issuer is `https://idp.example.com`, the full configuration URL would be: `https://idp.example.com/.well-known/openid-configuration`
* Enter the **Issuer URL**, which identifies your identity provider's authorization server and is provided by your identity provider's OIDC application.
* Enter the **Authorization URL** for the server that performs user authorization.
* Enter the **Token URL** for the server where the authorization code is sent to obtain a user access token.
* Enter the **User Info URL** for the server where the user profile attributes are stored and can be retrieved after authentication.
* Enter the **JWKS URL**, which is the JSON Web Key Set that verifies signed JWT tokens, provided by your identity provider's OIDC application.
3. Click **Save**.
4. The Identity Provider process may take several minutes to complete. Once finished, the **URLs for IDP Configuration** display on the right side of the screen. Copy the **Redirect URL** and save it in a secure location.
### Configure the redirect URI in your identity provider
1. In your identity provider, navigate to the application you created for Lucidworks SSO.
2. Add the **Redirect URL** generated by Lucidworks Platform to the application's sign-in redirect URIs.
3. Save your changes.
### Test SSO functionality
After configuring both Identity Provider and the application, test the integration by attempting to log into Lucidworks Platform with one of the users assigned to the application. If the configuration is successful, you are prompted to log in with your identity provider's authentication.
## SAML authentication
[SAML](https://en.wikipedia.org/wiki/Security_Assertion_Markup_Language) is an XML-based protocol that uses HTTP to authenticate and authorize user data. In a SAML workflow, Lucidworks Platform is the Relying Party (RP) that requests and receives SAML assertions from the identity provider.
### Create users in Lucidworks Platform
**Prerequisites**
1. Users must be added to the Lucidworks Platform.
2. Users must exist in your identity provider, and be enabled for the application you are using to connect with Lucidworks Platform.
3. The user’s email address in your identity provider and Lucidworks Platform must match, including case. Some identity providers treat email addresses as case-sensitive, which can prevent a user from accessing Lucidworks Platform if the case doesn't match exactly.
1. Navigate to the megamenu and click **Settings > Users & Provisioning > Users**.
2. Verify all necessary users are listed, active, and their emails match their identity provider user counterparts.
3. If any users are missing, add the appropriate users by completing the steps in [invite workspace owners](/docs/lw-platform/lw-platform/settings/lw-users/overview#invite-workspace-owners).
### Create a SAML application in your identity provider
1. Create a new SAML application in your identity provider. Refer to your provider's documentation for detailed steps:
* [Okta](https://help.okta.com/en-us/content/topics/apps/apps_app_integration_wizard_saml.htm)
* [Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/identity/enterprise-apps/add-application-portal-setup-sso)
* [Google Workspace](https://support.google.com/a/answer/6087519)
2. When creating the application, use the following settings:
* **Sign-in method**: Select **SAML 2.0**.
* **Single Sign On URL (ACS URL)**: Enter a placeholder URL for now. You’ll configure it in a later step.
* **Audience URI (SP Entity ID)**: Enter a placeholder URL for now. You’ll configure it in a later step.
* **Name ID Format**: Select the appropriate Name ID format. This is usually `EmailAddress`.
* **Application username**: Choose how your identity provider should map the user’s application username. For example, their identity provider username or email address.
3. After creating the application, locate its metadata, copy the **Metadata URL**, and save it in a secure location.
The **Metadata URL** is critical for SAML communication and must be kept secure.
4. Assign at least one user and one admin to the application. These users must already exist in Lucidworks Platform. You must assign yourself as one of the users to be able to complete the authentication configuration.
### Create the Identity Provider in Lucidworks Platform
1. Navigate to the megamenu and click **Settings > Users & Provisioning > Authentication > SAML**.
2. Fill out required fields:
* **Email Domains**: Enter the list of domains you want to filter by, for example, `\lucidworks.com`.
Be careful not to lock yourself out at this step in the process. If you list a domain that you already use to log into Lucidworks Platform, and you don’t complete the rest of the steps listed here, then you will be prompted to log in via SSO the next time you log out, which won’t work. If you are unable to log into Lucidworks Platform, contact Support or your Lucidworks representative.
* Enter the **Metadata URL** provided by your identity provider's SAML application.
* Enter the **Issuer URL** that specifies which identity provider is used by SAML. This value is generated when the SAML application is created.
* Enter the **Identity Provider SSO URL**, which is the single sign-on endpoint for users, and is provided by your identity provider's SAML application.
* Paste the **Certificate** information, which is provided by your identity provider's SAML application and is used by the identity provider to sign SAML responses.
3. Click **Save**.
4. The Identity Provider process may take several minutes to complete. Once finished, the **URLs for IDP Configuration** display on the right side of the screen:
* **Audience URI**
* **Assertion Consumer Service URL**
5. Copy the URLs and save them in a secure location.
### Configure the application in your identity provider
1. In your identity provider, navigate to the application you created for Lucidworks SSO.
2. Update the application's SAML settings:
* In **Single Sign On URL (ACS URL)** paste the **Assertion Consumer Service URL** generated by Lucidworks Platform.
* In **Audience URI (SP Entity ID)** paste the **Audience URI** generated by Lucidworks Platform.
3. Optional: If your identity provider supports SAML attribute mapping, configure the following mappings to send additional user information to Lucidworks Platform:
| | Name | Name Format | Value |
| - | - | - | - |
| **Email** | `email` | Unspecified | `user.email` |
| **First Name** | `firstName` | Unspecified | `user.firstName` |
| **Last Name** | `lastName` | Unspecified | `user.lastName` |
| **Username** | `username` | Unspecified | `user.login` |
4. Save your changes.
### Test SSO functionality
After configuring both Identity Provider and the application, test the integration by attempting to log into Lucidworks Platform with one of the users assigned to the application. If the configuration is successful, you are prompted to log in with your identity provider's authentication.
# Settings
Source: https://doc.lucidworks.com/docs/lw-platform/lw-platform/settings/overview
The Settings options of Lucidworks Platform lets you:
import from '/snippets/LwTemplate.jsx';
* Create and manage user accounts, including setting roles, permissions, and authentication for users in Lucidworks Platform.
* Configure integrations that connect Fusion to Lucidworks Platform and other software to extend and enhance your workflows.
To access the screen, navigate to the megamenu, click **Settings**.
* To create and manage user accounts and user permissions, click **Users & Provisioning > Users**. For more information, see [Users and Provisioning](/docs/lw-platform/lw-platform/settings/lw-users/overview).
* To configure single sign-on authentication, click **Users & Provisioning > Authentication**. For more information, see [Single sign-on authentication](/docs/lw-platform/lw-platform/settings/lw-users/sso).
* To configure and manage Fusion integrations to Lucidworks Platform, Commerce Studio, and Analytics Studio, click **Fusion Integrations**. For more information, see
* [Lucidworks Platform integrations with Fusion](/docs/lw-platform/lw-platform/settings/fusion-integrations/overview)
* [Commerce Studio integration with Fusion](/docs/lw-platform/lw-platform/settings/fusion-integrations/commerce-studio-integration)
* [Configuration Promotion tool integration with Fusion](/docs/lw-platform/lw-platform/settings/fusion-integrations/configuration-promotion-integration)
For information about integrations between Lucidworks AI and Fusion, see [Lucidworks AI integrations](/docs/lw-platform/lw-ai/integrations).
# Button
Source: https://doc.lucidworks.com/docs/lw-platform/lw-ui-studio/components/button
Interactive or static button component for navigation or actions in UI Studio applications
The Button component provides buttons that users can click or view for UI Studio applications. Use buttons to enable navigation to other pages or trigger actions like CSV data exports. You can style buttons with custom text and colors.
## Basic usage
Add a button to your UI Studio page to create an interactive element for users.
Enter descriptive label text for the button and apply formatting in the Button component Customize section.
## Best practices
Use descriptive labels that clearly indicate the action. For example, **View details** or **Submit form**. Avoid vague labels such as **Click here**.
## Configuration properties
Text displayed on the button. Keep this concise and action-oriented.
The action performed when the user clicks the button.
Navigation button with URL. The `Page Path` field contains the target URL.
Action button for CSV downloads.
URL path for navigation. Supports token syntax: `${TOKEN_NAME}` that is specified in the `Token Name` and `Field Name` fields.
Examples:
* `/home`
* `/users/${userId}`
* `/products/${productId}/details?tab=${activeTab}`
When enabled, the link specified in the `Page Path` field opens in a new browser tab.
Token settings for button navigation.
Unique name for the navigation token.
The field that contains the token used when the user clicks the button.
Typography settings for the text.
Font typeface for the text.
Font weight for the text. Options are Regular, Light, Bold, and Extra Bold.
Font size in pixels.
Recommended ranges:
* Body text: 14-16px.
* Headings: 18-32px.
* Small text: 12-13px.
Color scheme for the button.
Background color of the button.
Text color for the button label. Ensure sufficient contrast with background color.
Outline color of the button.
Border radius in pixels. Controls corner roundness.
* `0` = Square corners
* `4-8` = Subtle rounding
* `20+` = Pill-shaped button
# Facets
Source: https://doc.lucidworks.com/docs/lw-platform/lw-ui-studio/components/facets
Interactive filtering component for refining search results by categories and attributes in UI Studio applications
The Facets component in UI Studio provides interactive filters that allow users to refine search results by selecting categories, attributes, and values. Use facets to narrow down large datasets by criteria like manufacturer, category, price range, or any custom field from your data.
Learn how to configure the Results List with Grid or Document layouts, set up facets with human-legible field names, and enable search-within-results refinement.
## Basic usage
Add a facets component to your UI Studio page to enable filtering capabilities. To build a minimal facets component, you must set a value for the Datastore field and a facet category to display.
If no facets are available for the provided datastore, the component displays an empty state message: "No facets found. Please try adjusting your search criteria."
To add a facet, click the **+** next to the **Facets** selector. Facets display in a vertical list.
## Searchable facet values
Enable search within facet values for categories with many options. When enabled, a search box appears within each facet category, allowing users to quickly find specific values. This is especially useful for facets with many values, for example, cities or product models.
Toggle the **Searchable Facet Values** switch to turn this feature on or off.
## Collapsible facets
Collapsible facets let users show or hide individual facet categories to reduce visual clutter.
When enabled, each facet category displays a chevron icon. Users can click to collapse categories they're not using, creating a cleaner interface. The collapsed state persists during the user's session.
Enabling collapsible facets is recommended when you have many facet categories. Users can focus on the filters they're actively using while hiding others.
Toggle the **Collapsible** switch to turn collapsible facet categories on or off.
## Inclusive and exclusive filtering
Enable advanced filtering with inclusion and exclusion operators. These filters let users include or exclude a facet from their search results and create complex filter combinations such as "show products from Sony or Samsung, but not refurbished items."
When enabled, each facet value displays plus and minus icons:
* **+ (Plus)**: Include results that match this facet value.
* **- (Minus)**: Exclude results that match this facet value.
Use inclusive and exclusive filtering for complex filtering through many facets. For simpler interfaces, turn this setting off to avoid overwhelming users with additional controls.
Toggle the **Inclusive/Exclusive** switch to turn this setting on or off.
## Selected facet pills
Use facet pills to display selected filters as removable pills before the facet categories. These facet pills provide a visual indicator to users about the facets they're actively filtering by.
Users can select individual facet pills to remove that filter from their current search results.
A **Clear All** link also appears, letting users remove all active filters at once.
Toggle the **Selected Facet Pills** switch to turn this feature on or off.
## Configuration properties
Data source that provides facet categories and values. Must be configured in your UI Studio application.
Optional text label displayed as the heading above the facets panel.
Configure individual facet fields to display in the component.
The field name from your datastore to use as a facet.
The label shown to users for this facet category (for example, "Manufacturer", "Color").
When enabled, allows users to select multiple values within this facet category.
When enabled, adds a search box within each facet category for filtering values. This setting is recommended for facets with ten or more values.
When enabled, facet categories can be collapsed and expanded with chevron icons. The collapsed state persists in the user's session.
When enabled, displays **+** and **-** icons next to facet values for inclusive or exclusive filtering. This allows complex filter combinations like "include Sony or Samsung, but exclude refurbished".
When enabled, displays active filters as removable pills at the top with a **Clear All** option. Shows inclusive (+) or exclusive (-) indicators when Inclusive/Exclusive mode is enabled.
Typography settings for the main facets heading or title.
Font family for facets title.
Font weight for facets title.
Standard weight.
Heavy weight.
Font size in pixels for facets title.
Typography settings for facet category titles (for example, "Manufacturer", "Color").
Font family for facet category titles.
Font weight for facet category titles.
Standard weight.
Heavy weight.
Font size in pixels for facet category titles.
Typography settings for facet values (for example, "Sony", "Purple").
Font family for facet values.
Font weight for facet values.
Standard weight.
Heavy weight.
Font size in pixels for facet values.
Typography settings for the search input field within facet categories. Used when **Searchable Facet Values** is enabled.
Font family for search input text.
Font weight for search input text.
Standard weight.
Heavy weight.
Font size in pixels for search input text.
Typography settings for the "Selected Filters:" header text when Selected Facet Pills is enabled.
Font family for selected filters header.
Font weight for selected filters header.
Standard weight.
Heavy weight.
Font size in pixels for selected filters header.
Typography settings for the filter pill text when Selected Facet Pills is enabled.
Font family for filter pill text.
Font weight for filter pill text.
Standard weight.
Heavy weight.
Font size in pixels for filter pill text.
Color scheme for the facets component.
Background color for the facets container.
Color properties for the facets title.
Text color for the main facets title.
Background color for the facets title area.
Border color for the facets title area.
Color properties for the facet field.
Text color for facet category titles. Ensure sufficient contrast with background.
Background color for facet category title rows.
Border color for facet category title rows.
Color properties for the facet value.
Text color for facet values. Ensure sufficient contrast with background.
Background color when hovering over facet values.
Color properties for the facet search input.
Text color for the search input field within facet categories.
Background color for the search input field within facet categories.
Color properties for the selected filters header.
Text color for the "Selected Filters:" header label.
Text color for the "Clear All" link.
Color properties for the selected filter pill.
Text color for filter pill labels.
Background color for filter pills.
Border color for filter pills.
Color for the remove icon (×) on filter pills.
Color properties for the inclusive/exclusive buttons.
Background color for the + and - buttons when Inclusive/Exclusive mode is enabled.
Border color for the + and - buttons when Inclusive/Exclusive mode is enabled.
Background color for active + or - buttons when Inclusive/Exclusive mode is enabled.
Icon color for active + or - buttons when Inclusive/Exclusive mode is enabled.
# Image
Source: https://doc.lucidworks.com/docs/lw-platform/lw-ui-studio/components/image
Display images with optional links and views in UI Studio applications
The Image component displays static or dynamic images in your UI Studio pages. Use images to add visual content, branding, product photos, or clickable banners. The component supports clickthrough links, alternative text, and fallback images.
## Basic usage
Add an image component to your UI Studio page to display visual content.
Provide an image URL to display it on your page. The component supports common image formats including JPG, PNG, GIF, and SVG.
If the primary image fails, the component automatically loads the fallback image if one is configured. If both images fail, the component does not display in order to avoid broken image icons. If there is no image URL, the component does not render.
## Best practices
* Optimize images to reduce file size without sacrificing quality. While Lucidworks does not make a specific recommendation, tools such as TinyPNG, ImageOptim, or your image editor's "Save for Web" feature are available. The recommended maximum file size is 2MB for web display.
* If possible, configure a fallback image that displays if the primary image fails to load.
* Use appropriate image formats for different content types:
* **JPG**: Photos and complex images
* **PNG**: Graphics with transparency or text
* **SVG**: Logos and icons to ensure correct scaling
* **WebP**: Modern format with better compression, if browser support allows
* Always provide meaningful Alt (alternative) text that conveys the purpose and content of the image. Screen readers rely on this to describe images to visually impaired users. Alt text also improves SEO and displays when images fail to load.
* Avoid using images of text for important content. Screen readers cannot read text embedded in images.
## Configuration properties
URL to the image file.
Descriptive text for accessibility and SEO.
Describe the image content and purpose. Avoid generic phrases like "image" or "photo".
Clickthrough URL. When configured, clicking the image navigates to this URL.
Examples:
* `https://www.example.com`
* `/products`
* `https://example.com/offers`
Leave blank for non-clickable images.
Controls how the image fits within its container dimensions.
Scales the image to fit within the container while maintaining aspect ratio. May show letterboxing or pillarboxing (empty space on sides/top/bottom).
Scales the image to cover the entire container while maintaining aspect ratio. May crop portions of the image. This is the default.
Corner roundness in pixels.
* `0` = Square corners (default)
* `4-8` = Subtle rounded corners
* `12-16` = Noticeably rounded corners
* `50+` = Circular (when width equals height)
Optional backup image URL. Displays if the primary image fails to load.
# Results
Source: https://doc.lucidworks.com/docs/lw-platform/lw-ui-studio/components/results
Display search results with sorting, pagination, conditional row coloring, and dynamic row linking
The Results component displays search results from your datastore in a table format. Use this component to present search findings, product listings, data tables, or any collection of records. The component supports sorting, pagination, conditional row coloring, and dynamic page linking.
Learn how to configure the Results List with Grid or Document layouts, set up facets with human-legible field names, and enable search-within-results refinement.
## Basic usage
Add a Results component to your UI Studio page to display search results. The component typically works alongside a [Search Bar component](/docs/lw-platform/lw-ui-studio/components/search-bar) and [Facets component](/docs/lw-platform/lw-ui-studio/components/facets) to display and filter query results, but can also work standalone.
When no search query is active, the component displays all records by default. The Results component updates when users apply filters or perform searches.
The Results component supports a variety of content types and layouts. Common use cases include:
* **Product listings**: A catalog of items for sale, such as an ecommerce store. Create results listings with images, prices, and "Add to Cart" buttons.
* **Search results**: A display of documents or records matching a user query. Create results listings with titles, descriptions, and snippets.
* **Data tables**: A structured view for tabular data users need to compare or filter. Create results listings with sorting and conditional coloring for status.
* **User directory**: A browsable list of users or contacts with role information. Create results listings with profile links and role-based coloring.
* **Document library**: A collection of files organized for browsing and download. Create results listings with file types, dates, and download links.
## Display your results
The table view displays results as a structured data table with columns and rows. This layout is best for structured data and sortable datasets, as the table view is optimized for dense information display.
Click the **+** next to **Columns** to display a column in the results. Columns display in the order that they are added.
To keep column headers visible while you scroll through results, turn on the **Sticky Header** toggle.
## Sort options
Configure the sort criteria for users. Use these options so that users can sort the results by preset criteria, such as price, name, or quantity.
The sort options display in a dropdown menu above the search results. Sorting by relevance is included by default.
The sort options display in the order that you added them. The default Relevance option always appears first in the user-facing dropdown, even though it is listed last in the configuration panel.
## Pagination
Use pagination to configure the number of results per page and how many results the user can choose to display per page.
When pagination is turned off, only the first page of results displays.
When you enable pagination, you can set the default number of results to display per page in the **Default Results Count** field. The default value is 10 results.
You can also configure the number of results to display per page for users to select. Set these values in the **Results Count** fields, listed from lowest to highest.
## Conditional row coloring
Use conditional row coloring to apply background and text color to rows based on field values, making it easy to surface important information at a glance. Use this feature to flag inventory items with low stock, highlight tasks that are past due, distinguish active records from inactive ones, or draw attention to high-priority items without requiring users to sort or filter the table themselves.
Each rule evaluates a field against a condition and a value, then applies the configured colors to any matching row. When you configure multiple rules, the component evaluates them in the order listed and applies the first matching rule. Rows that match no rule use the default table styling.
Because the component evaluates rules top to bottom and applies the first match, you can order rules from most critical to least critical status to ensure proper highlighting.
An ecommerce catalog uses a `stock_status` field to alert buyers and staff about inventory availability. The rule checks whether `stock_status` equals `Out of stock` and applies a red background to signal that the item is unavailable.
That same approach extends to any number of statuses. A second rule can check whether `stock_status` equals `Low stock` and apply a yellow background to indicate limited availability, and a third can check whether `stock_status` equals `In Stock` and apply a green background to indicate healthy inventory levels. You can continue adding rules for any status value your catalog defines.
A task management table uses a `status` field to communicate record state at a glance. One rule checks whether `status` equals `Active` and applies a green background. Another checks whether `status` equals `Inactive` and applies a gray background. A third checks whether `status` equals `Error` and applies a red background with white text.
Use subtle background tints rather than saturated colors so that text remains readable across all rows.
A support queue uses a `priority` field to help agents triage incoming requests. One rule checks whether `priority` equals `High` and applies red text. Another checks whether `priority` equals `Medium` and applies orange text. A third checks whether `priority` equals `Low` and applies green text.
Text color changes alone work well when you want to signal priority without overwhelming the table with heavy background coloring.
## Dynamic page link
Use dynamic page links to navigate users directly from a table row to a detail page, connecting your results table to the rest of your application. Use this feature to build product catalog tables where rows link to product detail pages, user directories where names link to profile pages, or document lists where entries open document viewers.
When **Link Mode** is set to **Action**, the **Display Type** setting controls how the link appears in the table. Choose **Link** when the field value is a natural, recognizable label that users expect to click, such as a username or a document title. Choose **Button** when the field value is an internal ID or technical string you do not want to display, and the action should appear as a labeled button.
To use a redirect button, define the destination URL with the **Page** base path and a **URL Template** that combines the page path with a placeholder for the field value. Wrap the placeholder in curly braces to mark where the field value is injected. The template `/products/PRODUCT_ID` resolves to `/products/ABC123` when the `PRODUCT_ID` field contains `ABC123`. You can position the placeholder anywhere in the template path, such as `/users/{username}/profile` or `/documents/{doc_id}`.
You can configure multiple dynamic links on the same table, assigning one link configuration per column. This lets you create multi-action rows with a mix of link types, such as a "View" redirect button in one column alongside an "Edit" button in another.
A product catalog table links each row to its corresponding product detail page. Set **Column** to `PRODUCT_ID`, **Link Mode** to **Action**, **Display Type** to **Button**, **Button Label** to "View Product". Set the **URL Template** to `/products/PRODUCT_ID`. A row where `PRODUCT_ID` contains `ABC123` navigates the user to `/products/ABC123`.
A user directory makes the username itself a clickable link to the user profile page. Set **Column** to `username`, **Link Mode** to **Inline**, and **URL Template** to `/users/{username}/profile`. The displayed username becomes a link without a separate button.
A document library hides the internal document ID and exposes only a labeled action button. Set **Column** to `doc_id`, **Link Mode** to **Action**, **Display Type** to **Button**, **Button Label** to "Open Document", and **URL Template** to `/documents/{doc_id}/view`. The document ID is not shown to users. Only the button is displayed.
When creating URL templates for Dynamic Page Links, test with sample data to ensure links resolve correctly. Invalid templates result in broken navigation.
## Configuration properties
### Basic configuration
Data source for results. Must be configured in your UI Studio application.
When enabled, highlights search terms within table cells to show matches visually.
When enabled, alternates row background colors for improved readability.
Configure which data fields display as table columns. Add multiple columns using the **+** button.
Field name from your datastore to display as a column.
Label shown in the column header.
How the field content is rendered.
Plain text display.
Image display.
Formatted date display.
Currency-formatted display.
Clickable link display.
When enabled, column headers remain visible while scrolling through table rows.
Typography settings for column headers.
Font family for column header text.
Font weight for column header text.
Standard weight.
Heavy weight.
Font size in pixels for column header text.
Configure available sort criteria for users. The **Relevance** option is included by default and displays last in the sort dropdown.
Available sort options. Add multiple options using the + button.
Data field to sort by.
Sort direction for this option.
Sort from lowest to highest: A-Z, 0-9, oldest to newest.
Sort from highest to lowest: Z-A, 9-0, newest to oldest.
Label shown to users in the sort dropdown.
Examples: "Price: Low to High", "Newest First", "Product Name: A-Z"
Configure pagination controls for navigating through result pages.
When enabled, displays pagination controls for navigating through result pages. Otherwise, only the first page of results displays.
Number of results displayed per page by default.
Number of results to display per page.
Number of results per page.
Add or remove values using the **+** and **-** buttons to customize available page sizes.
Visual appearance settings for the table.
Typography settings for table cell content.
Font family for cell value text.
Font weight for cell value text.
Standard weight.
Heavy weight.
Font size in pixels for cell value text.
Horizontal alignment for table cell content.
Align text to the left edge of the column.
Align text to the center of the column.
Align text to the right edge of the column.
Color scheme for the table.
Background color for the entire table.
### Advanced configuration
Apply styling to rows based on field values.
Column or field to evaluate.
Comparison operator.
Matches rows where the field equals the specified value.
Matches rows where the field does not equal the specified value.
Matches rows where the field contains the specified value.
Matches rows where the field does not contain the specified value.
Matches rows where the field value begins with the specified text.
Matches rows where the field value ends with the specified text.
Value to compare against. The type depends on field and condition.
Row background color when condition is true.
Row text color when condition is true. Ensure sufficient contrast with background.
Clickable links or buttons in table columns that navigate to detail pages.
Field that provides the link data or navigation context.
Determines how the link is integrated into the table.
Link embedded directly in the cell value.
Separate action link or button alongside the cell value.
How the action appears when Link Mode is set to Action.
Appears only when **Link Mode** is set to **Action**.
Displays as a text link.
Displays as a button.
Text displayed for the link.
Appears only when the **Display Type** is **Link** or when **Link Mode** is **Inline**.
Text displayed on the button.
Only shown when **Link Mode** is **Action** and **Display Type** is **Button**.
URL pattern with placeholders for dynamic values.
Format: Use curly braces `{field_name}` as placeholders for field values.
Examples:
* `/details/PRODUCT_ID`
* `/users/{username}/profile`
* `/documents/{doc_id}`
When enabled, treats the column value itself as a complete URL rather than injecting it into the URL Template.
When enabled, opens the link in a new browser tab instead of navigating in the current tab.
# Search Bar
Source: https://doc.lucidworks.com/docs/lw-platform/lw-ui-studio/components/search-bar
Interactive search input with typeahead autocomplete suggestions in UI Studio applications
The Search Bar component in UI Studio provides a text input field for users to search for content. Use search bars to enable keyword searching with optional typeahead autocomplete that suggests results as users type. The component supports customizable placeholder text, styling, and configurable typeahead suggestions.
Pair the search bar component with the [results component](/docs/lw-platform/lw-ui-studio/components/results) and the [facets component](/docs/lw-platform/lw-ui-studio/components/facets) to build a complete search experience.
Learn how to configure the Search Bar component with Typeahead, map it to a Fusion query profile, and customize its placeholder text and styling.
## Basic usage
Add a search bar component to your UI Studio page to enable search capability. To build a minimal search component, you must set values for the Name and Datastore fields.
Users type queries and press Enter to search.
Learn how to link UI Studio components to Fusion datastores for live data, then use preview mode to validate the experience.
## Typeahead
You can enable autocomplete suggestions that appear as users type. When typeahead is enabled, the search bar displays a dropdown of suggested queries as users type. Users can:
* Click a suggestion to search for it.
* Use arrow keys to navigate suggestions.
* Press Enter to search the highlighted suggestion.
* Press Escape to close the dropdown.
To configure typeahead, you must have a configured typeahead datasource. This datasource can be the same as your primary datasource for search. If the endpoint is missing or unreachable, UI Studio automatically deactivates typeahead to prevent errors.
Typeahead suggestions improve discoverability and reduce typos, but they also add complexity. Turn the typeahead setting off for simple search scenarios or when your dataset is small.
Expand the **Typeahead** section to configure autocomplete behavior and appearance.
### Suggestion count
Set the maximum number of typeahead suggestions to display in the **Suggestion Count** field.
Enter a number in the **Suggestion Count** field, for example, 5, 10, or 15.
The default value is 10 suggestions. Keep suggestion counts between 5 and 10 to balance performance and suggestion variety. Too few suggestions limit discovery, while too many suggestions can overwhelm users.
### Search character count
Set the minimum number of characters required before showing suggestions. Enter a number in the **Search Character Count** field, for example, 2, 3, or 4.
Setting a higher number prevents typeahead from triggering on very short queries that would return too many broad suggestions. If you have a dataset with distinct terms, use a lower value.
Higher character counts reduce the number of typeahead API calls, improving performance. However, higher character counts also delay when users see useful suggestions.
## Configuration properties
Component identifier for organization and reference. The name does not display on your user-facing website.
Primary data source for search queries. Must be configured in your UI Studio application.
Data source for autocomplete suggestions. This datastore can be the same as the main datastore or a separate optimized source. If the endpoint is missing or unreachable, typeahead is automatically deactivated to prevent errors.
Hint text displayed when the search bar is empty. Should be concise and descriptive.
Typography settings for placeholder text.
Font family for placeholder text.
Font weight for placeholder text.
Standard weight.
Heavy weight.
Font size in pixels for placeholder text.
Color scheme for the search bar.
Background color for the search input field.
Text color for placeholder text. Should be lighter than entered text for clear distinction.
Autocomplete suggestion settings.
When enabled, displays autocomplete suggestions as users type. Typeahead is enabled by default but requires a configured Typeahead datastore. This datastore can be the same as your search datastore. If no datastore is configured, typeahead is automatically deactivated.
Maximum number of autocomplete suggestions to display.
Recommended range: 5 to 10
Minimum number of characters required before showing typeahead suggestions.
Recommended range: 2 to 5
Background color for the suggestions dropdown.
Background color when hovering over a suggestion.
Background color for keyboard-selected suggestions.
Background color for actively selected items.
Text color for suggestion items.
Font family for typeahead suggestion text.
Font weight for typeahead suggestion text.
Font size in pixels for typeahead suggestion text.
# Text
Source: https://doc.lucidworks.com/docs/lw-platform/lw-ui-studio/components/text
Display and format text content with rich styling options in UI Studio applications
The Text component of UI Studio lets you create formatted text that displays on your UI Studio pages. Use text components to add headings, paragraphs, instructions, labels, or any textual content. The text in the component can be configured to display in various formats such as italic, underline, specific text alignment, and custom styling.
## Basic usage
Add a text component to your UI Studio page to display content.
Enter your text content and apply formatting to create visually appealing, readable content.
## Best practices
* Format similar items consistently. For example, if you bold important terms in one text component, use the same format for all important terms.
* Limit the number of fonts and font sizes on a specific UI Studio page. Use a font type that reflects your organization's design. For font size, select one for all headings, another for body text, and another for footnotes and other small text.
* Avoid using multiple emphasis styles on the same text. For example, choose just bold, but do not underline or italicize the same text as well.
## Configuration properties
### Basic properties
The text content to display. Supports rich text formatting when Font options are applied. Text line length is 50-75 characters for optimal readability, but use 35-50 characters for mobile devices.
Typography settings for the text.
Font typeface for the text.
Font weight for the text. Options are Regular, Light, Bold, and Extra Bold.
Font size in pixels.
Recommended ranges:
* Body text: 14-16px.
* Headings: 18-32px.
* Small text: 12-13px.
When enabled, the text is italicized.
When enabled, the text is underlined.
How the text is aligned in the text field. Options are Left, Center, or Right.
Color settings for the text.
Text color for the component.
### Advanced properties
The amount of whitespace at the top of the text field in pixels.
The amount of whitespace on the left edge of the text field in pixels.
# Compositions
Source: https://doc.lucidworks.com/docs/lw-platform/lw-ui-studio/compositions
A composition is the top-level container you build, version, and publish in UI Studio.
A composition contains one or more pages, each with its own URL. You arrange components on each page's canvas, configure them, and publish when ready. UI Studio tracks every save as a new version so you can view the full history of your work.
Learn how to access the UI Studio home page and manage your existing compositions.
## The compositions list
The Compositions screen shows all compositions in your environment. Each row displays the composition name and the date and time it was last edited. Click any row to open the composition in the editor.
Use the search bar to filter compositions by name or description. Use the **Filters** button to filter by a date range for the Last Edited On column. Click the **Name** or **Last Edited On** column headers to sort the list.
Follow these steps to add a new composition.
Navigate to **UI Studio** in the left navigation panel and click **Builder**.
On the Compositions screen, click **Create New UI**, enter a name and description for the composition, and click **Create**.
The new composition is listed on the Compositions screen. You can now select the composition and build it.
Follow these steps to add components to build a composition.
Navigate to **UI Studio** in the left navigation.
Select a composition from the list.
The new composition opens in the editor.
* Add pages, and then click the main canvas. Click the gear icon of the canvas to open the **Customize Section** panel and set the canvas color and size.
* Drag components from the left panel onto the canvas to build your page layout.
* Click any component to select it and configure it in the **Customize** panel.
* Repeat across pages until your composition is ready, then save your work.
The Quickstart walks through this process end to end, and the component documentation covers every configuration option.
Get up and running with your first UI Studio page in minutes.
Browse available components and learn how to configure them.
A composition contains one or more pages. Each page has a name and a URL path. One page is designated as the home page, which loads when a user visits the root of your published composition. Pages can also contain subpages, each with a URL path nested under the parent page's path.
Display name for the page.
URL path for the page, relative to the composition's root URL.
When true, this page loads when a user visits the root of the published composition. Only one page can be the home page.
Optional child pages nested under this page's URL path. Each subpage has the same configuration options as a top-level page.
Publishing makes your composition available at a public URL in the format `YOUR_COMPANY.ui.lucidworks.com/COMPOSITION_NAME`.
In the composition's editor screen, click **Publish**.
Review and confirm the location to publish. It typically includes your company name, composition name, and page title. Click **Publish**.
The published URL uses the composition name. To change the URL, update the composition name before publishing.
## End-user single sign-on for published compositions
UI Studio supports end-user Single Sign-On (SSO) using Okta OIDC and controls access for end users who visit published search sites.
When enabled, users must authenticate before accessing a published composition at `YOUR_COMPANY.ui.lucidworks.com/COMPOSITION_NAME`. When SSO is not enabled, the published URL is publicly accessible.
To enable UI Studio SSO for your published compositions, contact your Lucidworks representative.
UI Studio SSO is different from Lucidworks Platform SSO, which controls access for administrators who build and manage compositions in the Lucidworks Platform workspace.
## Composition settings
Settings apply at the composition level and affect all pages. Access settings from the editor toolbar.
### Branding
Upload a logo and favicon for your composition. These appear in the browser tab and any branded areas of your pages.
Image file displayed in branded areas of your composition.
Icon file displayed in the browser tab.
### Localization
Configure the language, timezone, and date and time format for your composition. These settings affect how dates and times display across all pages.
Language code for the composition, for example, `en-US`.
Timezone for the composition, for example, `America/Los_Angeles`.
Format string for displaying dates and times, for example, `MM/DD/YYYY h:mm A`.
### Authentication
Configure single sign-on (SSO) for your composition using OIDC.
When true, SSO is required to access the composition.
How long, in hours, a user session remains valid.
How many minutes before session expiry a session is eligible for extension.
Identity provider configuration for the OIDC protocol.
Client identifier registered with your identity provider.
Client secret registered with your identity provider.
Base URI of your identity provider.
URI where users are redirected to authenticate.
URI used to exchange authorization codes for tokens.
URI of your identity provider's JSON Web Key Set.
URI used to retrieve user profile information.
URI used to log users out of the identity provider.
OAuth scopes to request from the identity provider, for example, `openid`, `profile`, `email`.
# UI Studio
Source: https://doc.lucidworks.com/docs/lw-platform/lw-ui-studio/overview
Build interactive search experience pages with drag-and-drop canvas sections, pre-built components, and no-code configuration.
UI Studio lets you create and publish search experience pages without writing code. Arrange pre-built components on visual canvas sections, connect them to your datastore, and deploy directly to your users.
## How UI Studio works
**Compositions are standalone web experiences.** A composition works like a self-contained website, made up of one or more pages, each with its own URL. Arrange components on each page's canvas sections, configure them independently, and publish when ready.
**Components reflect your data.** Components retrieve and display data from a configured datastore. Your datastore changes update automatically.
**Configuration is visual.** Use the Customize panel to adjust component settings, styling, and behavior without editing code.
**Your changes are backed up automatically.** UI Studio saves your composition history as you work. If you need to revert a critical change, contact your Lucidworks representative.
**Publishing generates a public URL.** When you publish, your composition is hosted and immediately available to your users.
## Get started
Get up and running with your first UI Studio page in minutes.
Learn how to link UI Studio components to Fusion datastores for live data, then use preview mode to validate the experience.
## Build your app
Arrange and manage page layouts using composition canvas sections.
## Components
Use pre-built components to build interactive search experiences.
Accept user queries and trigger search across your datastore.
Display search results views with sorting and pagination.
Let users narrow results by filtering on field values.
Add navigation, export, or action buttons to your pages.
Display images with optional links, borders, and fallback support.
Add static or dynamic text content with rich styling options.
# Quickstart
Source: https://doc.lucidworks.com/docs/lw-platform/lw-ui-studio/quickstart
Build a basic search interface in UI Studio Builder with Search Bar, Results, and Facets components.
This guide walks you through building a basic but functional search interface in UI Studio Builder. By the end, you will have a composition with a Search Bar, Results, Facets, Text, Image, and Button component working together as a complete search experience.
For more information, see [Compositions](/docs/lw-platform/lw-ui-studio/compositions).
Navigate to **UI Studio** in the left navigation panel and click **Builder**.
On the Compositions screen, click **Create New UI**.
Enter a name and description for the composition and click **Create**.
The new composition is listed on the Compositions screen. You can now select the composition and build it.
On the Compositions screen, click a composition to open it in the editor.
Click the main canvas and then click the gear icon to open the **Customize** panel on the right. Set a background color to match your company's branding, or adjust the section dimensions. These settings can be refined at any time.
In the left panel of the Editor screen, click **Components**. Available components are listed in the left sidebar, organized into two categories: **Search** and **Content**.
To customize each component, click the component in the Editor screen and select values in the **Customize** panel on the right. Set a background color to match your company's branding, or adjust the section dimensions. These settings can be refined at any time.
Accept user queries and trigger search across your datastore.
In the left sidebar, find the **Search Bar** under the **Search** category. Drag it from the sidebar and hover over the header section in the canvas. The interface indicates where the component will land as you hover. Drop it into position.
The **Customize** panel opens on the right with the Search Bar configuration options. Configure the following:
* **Datastore**: Select the datastore you want to use for search queries.
* **Typeahead Datastore**: Select the datastore for autocomplete suggestions. This can be the same datastore as the one above.
* **Placeholder Text**: Replace the default text with something descriptive for your users, such as `Search our site`.
* **Typeahead**: Expand this section and toggle **Typeahead Enabled** to turn autocomplete on or off.
Learn how to link UI Studio components to Fusion datastores for live data, then use preview mode to validate the experience.
For details on all Search Bar settings, see the [Search Bar](/docs/lw-platform/lw-ui-studio/components/search-bar) component documentation.
Display search results in a table view with sorting and pagination.
Drag the **Results** component from the **Search** category in the left sidebar and drop it into the body area of the canvas.
In the **Customize** panel, set the **Datastore** to the same datastore you selected for the Search Bar. This ensures the Results component displays results from the same search. Adjust the **Pagination** setting from the default to control how many results appear per page.
Advanced compositions may use multiple Search Bars and Results components to support separate search experiences on the same page. For example, a company intranet might have one for document search and another for people search. To accommodate this, add a new section to the canvas by hovering between existing sections, where the option to create a new section appears.
For details on all Results settings, see the [Results](/docs/lw-platform/lw-ui-studio/components/results) component documentation.
Let users narrow results by filtering on field values.
Drag the **Facets** component from the **Search** category and drop it onto the canvas alongside the Results.
In the **Customize** panel, set the **Datastore** to the same datastore as the Search Bar and Results. This ensures the facets filter the same set of results. Configure a few options to shape the filtering experience:
* **Searchable Facet Value**: Toggle this on to add a search box within each facet category. This is helpful when a category has many values.
* **Collapsible**: Toggle this on to let users collapse and expand individual facet categories to save space.
* **Selected Facet Pills**: Toggle this on to show active filters as removable pills at the top of the facets panel.
For details on all Facets settings, see the [Facets](/docs/lw-platform/lw-ui-studio/components/facets) component documentation.
Add text content with rich styling options.
Display images with optional links, borders, and fallback support.
Add navigation, export, or action buttons to your pages.
With the core search components in place, add content components from the **Content** category in the left sidebar: **Text**, **Image**, and **Button**.
* **Text**: Drag a Text component onto the canvas. For example, place it above the Results component as a section heading. The canvas layout adapts flexibly as you place components.
* **Image**: Drag an Image component next to the Search Bar in the header section. In the **Customize** panel, upload your company logo and set an appropriate width and height.
* **Button**: Drag a Button component onto the canvas. In the **Customize** panel, set the **Type** to **Link**, enter a label such as `Visit our site`, and enter your company site URL in the **Page Path** field.
Explore the configuration options for each component. For details, see the [Text](/docs/lw-platform/lw-ui-studio/components/text), [Image](/docs/lw-platform/lw-ui-studio/components/image), and [Button](/docs/lw-platform/lw-ui-studio/components/button) component documentation.
When you are satisfied with your composition, use the toolbar at the top of the builder to save or publish it.
After publishing, preview the composition to see the user experience. Test a search query, apply facet filters, and confirm that the results update as expected.
# Create bulk operation
Source: https://doc.lucidworks.com/api-reference/bulk-operations/create-bulk-operation
/api-reference/saas/cs-rules-rewrites.json post /bulk-operations
Create a new bulk operation to perform actions on multiple rules. Supports publish, unpublish, edit, delete, and migrate operation types. Operations execute asynchronously.
# Delete bulk operation
Source: https://doc.lucidworks.com/api-reference/bulk-operations/delete-bulk-operation
/api-reference/saas/cs-rules-rewrites.json delete /bulk-operations/{id}
Delete a bulk operation record. This removes the operation from the system but does not reverse changes already applied to rules.
# Get bulk operation
Source: https://doc.lucidworks.com/api-reference/bulk-operations/get-bulk-operation
/api-reference/saas/cs-rules-rewrites.json get /bulk-operations/{id}
Retrieve details of a specific bulk operation including current status, individual rule statuses, and error messages for failed rules.
# List bulk operations
Source: https://doc.lucidworks.com/api-reference/bulk-operations/list-bulk-operations
/api-reference/saas/cs-rules-rewrites.json get /bulk-operations
List all bulk operations with optional filtering and pagination. Returns operations of all types including publish, unpublish, edit, delete, and migrate.
# Update bulk operation
Source: https://doc.lucidworks.com/api-reference/bulk-operations/update-bulk-operation
/api-reference/saas/cs-rules-rewrites.json put /bulk-operations/{id}
Update an existing bulk operation. Used to modify operation parameters or retry failed rules.
# FAQ Enrichment use case
Source: https://doc.lucidworks.com/api-reference/create-predictions/faq-enrichment-use-case
/api-reference/saas/machine-learning-platform-async-prediction.json post /ai/async-prediction/faq-enrichment/{MODEL_ID}
The FAQ enrichment use case uses a large language model (LLM) to ingest input text and return a JSON response that contains a list of question and answer pairs.
# Image metadata enrichment use case
Source: https://doc.lucidworks.com/api-reference/create-predictions/image-metadata-enrichment-use-case
/api-reference/saas/machine-learning-platform-async-prediction.json post /ai/async-prediction/image-metadata-enrichment/{MODEL_ID}
In the image metadata enrichment use case, the LLM ingests a text containing either a base64-encoded image or a public HTTP/HTTPS or Google Cloud Storage (`gs://`) image link and routes the image to a multimodal model. A JSON response is returned that contains an `imageMetadata` dictionary with a list of keywords, subcategories, and keyword synonyms.
The POST request obtains and indexes prediction information related to the specified use case, and returns a unique `predictionId` and `status` of the request. The `predictionId` can be used later in the GET request to retrieve the results.
# Keyword extraction use case
Source: https://doc.lucidworks.com/api-reference/create-predictions/keyword-extraction-use-case
/api-reference/saas/machine-learning-platform-async-prediction.json post /ai/async-prediction/keyword-extraction/{MODEL_ID}
In the keyword-extraction use case, the LLM ingests text and returns a JSON response that contains a list of keywords extracted from the text. No options can be configured.
The POST request obtains and indexes prediction information related to the specified use case, and returns a unique `predictionId` and `status` of the request. The `predictionId` can be used later in the GET request to retrieve the results.
# Model predictions by use case
Source: https://doc.lucidworks.com/api-reference/create-predictions/model-predictions-by-use-case
/api-reference/saas/machine-learning-platform-async-prediction.json post /ai/async-prediction/{USE_CASE}/{MODEL_ID}
This is an all-purpose prediction endpoint you can use by specifying a `USE_CASE` and `MODEL_ID`. Upon submission, the API responds with a unique `predictionId` and a `status`. The `predictionId` can be used later in the GET request to retrieve the results.
IMPORTANT: The available use cases are detailed in their own section in this specification.
# Named Entity Recognition (NER) use case
Source: https://doc.lucidworks.com/api-reference/create-predictions/named-entity-recognition-ner-use-case
/api-reference/saas/machine-learning-platform-async-prediction.json post /ai/async-prediction/ner/{MODEL_ID}
In the Named Entity Recognition (NER) use case, the LLM ingests text and entities to extract and return a JSON response that contains a list of entities extracted from the text. No options can be configured.
This use case can be used to extract nouns and proper nouns such as Brand, Date, Company, Places, and Category in order to guide and refine searches.
The POST request obtains and indexes prediction information related to the specified use case, and returns a unique `predictionId` and `status` of the request. The `predictionId` can be used later in the GET request to retrieve the results.
# Passthrough use case
Source: https://doc.lucidworks.com/api-reference/create-predictions/passthrough-use-case
/api-reference/saas/machine-learning-platform-async-prediction.json post /ai/async-prediction/passthrough/{MODEL_ID}
The passthrough use case lets you use the service as a proxy to the large language model (LLM). The service sends text (no additional prompts or other information) to the LLM.
The POST request obtains and indexes prediction information related to the specified use case, and returns a unique `predictionId` and `status` of the request. The `predictionId` can be used later in the GET request to retrieve the results.
# RAG use case
Source: https://doc.lucidworks.com/api-reference/create-predictions/rag-use-case
/api-reference/saas/machine-learning-platform-async-prediction.json post /ai/async-prediction/rag/{MODEL_ID}
The rag use case uses candidate documents that are inserted into a LLM’s context to ground the generated response to those documents instead of generating an answer from details stored in the LLM’s trained weights. This type of search adds guardrails so the LLM can search private data collections.
The RAG search can perform queries against external documents passed in as part of the request.
The POST request obtains and indexes prediction information related to the specified use case, and returns a unique `predictionId` and `status` of the request. The `predictionId` can be used later in the GET request to retrieve the results.
# Standalone query rewriter
Source: https://doc.lucidworks.com/api-reference/create-predictions/standalone-query-rewriter
/api-reference/saas/machine-learning-platform-async-prediction.json post /ai/async-prediction/standalone-query-rewriter/{MODEL_ID}
The standalone query rewriter use case rewrites the text in reference to the context based on the memoryUuid.
The POST request obtains and indexes prediction information related to the specified use case, and returns a unique `predictionId` and `status` of the request. The `predictionId` can be used later in the GET request to retrieve the results.
# Summarization use case
Source: https://doc.lucidworks.com/api-reference/create-predictions/summarization-use-case
/api-reference/saas/machine-learning-platform-async-prediction.json post /ai/async-prediction/summarization/{MODEL_ID}
In the summarization use case, the LLM ingests text and returns a summary of the text as a response.
The context length is 2048 tokens. No options can be configured.
The POST request obtains and indexes prediction information related to the specified use case, and returns a unique `predictionId` and `status` of the request. The `predictionId` can be used later in the GET request to retrieve the results.
# Get prediction results
Source: https://doc.lucidworks.com/api-reference/fetch-predictions/get-prediction-results
/api-reference/saas/machine-learning-platform-async-prediction.json get /ai/async-prediction/{PREDICTION_ID}
Fetch the results of a prediction task by its ID.
# Classification use case
Source: https://doc.lucidworks.com/api-reference/get-predictions/classification-use-case
/api-reference/saas/machine-learning-platform-predict.json post /ai/prediction/classification/{MODEL_ID}
The classification use case lets you use embedding models to compute similarity scores between the incoming text and the labels. It returns the labels ranked in order of most similar to least similar.
The classification use case is compatible with all Lucidworks hosted pre-trained and custom embedding models. The default behavior of embedding models is to always have a score returned.
The `topK` and `similarityCutoff` parameters can be used to achieve behaviors where only the following are returned:
* The single most applicable label
* Labels with similarities that exceed a threshold
* A set number of items
* A set number if it exceeds the threshold
# Custom model prediction
Source: https://doc.lucidworks.com/api-reference/get-predictions/custom-model-prediction
/api-reference/saas/machine-learning-platform-predict.json post /ai/prediction/embedding/{MODEL_ID}
The custom model prediction is an embedding use case. If a custom model is trained and deployed using the Lucidworks AI Models API, the `DEPLOYMENT_ID` in the Models API is the same value as the `MODEL_ID` you enter in the custom model to return a prediction.
# English language model text encoder
Source: https://doc.lucidworks.com/api-reference/get-predictions/english-language-model-text-encoder
/api-reference/saas/machine-learning-platform-predict.json post /ai/prediction/embedding/text-encoder
The English language encoder is an embedding use that takes in plain English text and returns a 768-dimensional vector encoding of that text. This model powers this semantic search.
The API truncates incoming text to approximately 256 words before the model encodes it and returns a vector. An example usage pattern is to encode all the texts and descriptions in a website and then use this encoder on query text, supporting natural language queries such as "1990s children’s fiction".
Each API request includes one batch containing up to 32 text strings.
# FAQ Enrichment use case
Source: https://doc.lucidworks.com/api-reference/get-predictions/faq-enrichment-use-case
/api-reference/saas/machine-learning-platform-predict.json post /ai/prediction/faq-enrichment/{MODEL_ID}
The FAQ enrichment use case uses a large language model (LLM) to ingest input text and return a JSON response that contains a list of question and answer pairs.
# Image metadata enrichment use case
Source: https://doc.lucidworks.com/api-reference/get-predictions/image-metadata-enrichment-use-case
/api-reference/saas/machine-learning-platform-predict.json post /ai/prediction/image-metadata-enrichment/{MODEL_ID}
In the image metadata enrichment use case, the LLM ingests a text containing either a base64-encoded image or a public HTTP/HTTPS or Google Cloud Storage (`gs://`) image link and routes the image to a multimodal model. A JSON response is returned that contains an `imageMetadata` dictionary with a list of keywords, subcategories, and keyword synonyms. The metadata can be generated based on the image, the image title, and the categories.
# Keyword extraction use case
Source: https://doc.lucidworks.com/api-reference/get-predictions/keyword-extraction-use-case
/api-reference/saas/machine-learning-platform-predict.json post /ai/prediction/keyword-extraction/{MODEL_ID}
In the keyword-extraction use case, the LLM ingests text and returns a JSON response that contains a list of keywords extracted from the text. No options can be configured.
# Model predictions by use case
Source: https://doc.lucidworks.com/api-reference/get-predictions/model-predictions-by-use-case
/api-reference/saas/machine-learning-platform-predict.json post /ai/prediction/{USE_CASE}/{MODEL_ID}
This is an all-purpose prediction endpoint you can use by specifying a `USE_CASE` and `MODEL_ID`.
IMPORTANT: The available use cases are detailed in their own section in this specification.
# Named Entity Recognition (NER) use case
Source: https://doc.lucidworks.com/api-reference/get-predictions/named-entity-recognition-ner-use-case
/api-reference/saas/machine-learning-platform-predict.json post /ai/prediction/ner/{MODEL_ID}
In the Named Entity Recognition (NER) use case, the LLM ingests text and entities to extract and return a JSON response that contains a list of entities extracted from the text. No options can be configured.
This use case can be used to extract nouns and proper nouns such as Brand, Date, Company, Places, and Category in order to guide and refine searches.
# Part Number Detection use case
Source: https://doc.lucidworks.com/api-reference/get-predictions/part-number-detection-use-case
/api-reference/saas/machine-learning-platform-predict.json post /ai/prediction/part-number-detection/{CUSTOM_DEPLOYMENT_ID}
The Part Number Detection use case of the LWAI Prediction API detects part numbers in order to help with query routing and classification, and uses the embedding model to ingest text and return a JSON response of `true` if the input is a part number, or `false` if the input is not a part number.
# Passthrough use case
Source: https://doc.lucidworks.com/api-reference/get-predictions/passthrough-use-case
/api-reference/saas/machine-learning-platform-predict.json post /ai/prediction/passthrough/{MODEL_ID}
The passthrough use case lets you use the service as a proxy to the large language model (LLM). The service sends text (no additional prompts or other information) to the LLM and returns a response.
# RAG use case
Source: https://doc.lucidworks.com/api-reference/get-predictions/rag-use-case
/api-reference/saas/machine-learning-platform-predict.json post /ai/prediction/rag/{MODEL_ID}
The rag use case uses candidate documents that are inserted into a LLM’s context to ground the generated response to those documents instead of generating an answer from details stored in the LLM’s trained weights. This type of search adds guardrails so the LLM can search private data collections.
The RAG search can perform queries against external documents passed in as part of the request.
# Standalone query rewriter
Source: https://doc.lucidworks.com/api-reference/get-predictions/standalone-query-rewriter
/api-reference/saas/machine-learning-platform-predict.json post /ai/prediction/standalone-query-rewriter/{MODEL_ID}
The standalone query rewriter use case rewrites the text in reference to the context based on the memoryUuid.
# Summarization use case
Source: https://doc.lucidworks.com/api-reference/get-predictions/summarization-use-case
/api-reference/saas/machine-learning-platform-predict.json post /ai/prediction/summarization/{MODEL_ID}
In the summarization use case, the LLM ingests text and returns a summary of the text as a response.
The context length is 2048 tokens. No options can be configured.
# Cartadd signals
Source: https://doc.lucidworks.com/api-reference/get-signals/cartadd-signals
/api-reference/saas/lw-platform-signals.json post /retail/signals/cartadd
When a user adds an item to the cart for a potential sale, the information is contained in the cartadd signal.
# Click signals
Source: https://doc.lucidworks.com/api-reference/get-signals/click-signals
/api-reference/saas/lw-platform-signals.json post /retail/signals/click
When a user clicks on an item displayed from your site, including results returned from a query, the information is contained in a click signal.
# Purchase complete signals
Source: https://doc.lucidworks.com/api-reference/get-signals/purchase-complete-signals
/api-reference/saas/lw-platform-signals.json post /retail/signals/purchase
When a user completes a purchase and is taken to the order confirmation page, a signal is recorded.
# Query signals
Source: https://doc.lucidworks.com/api-reference/get-signals/query-signals
/api-reference/saas/lw-platform-signals.json post /retail/signals/query
When a user enters text to search your site, a query request is generated and the information is contained in a query signal.
# Tokenization by MODEL_ID
Source: https://doc.lucidworks.com/api-reference/get-tokens/tokenization-by-model_id
/api-reference/saas/machine-learning-platform-tokenization.json post /ai/tokenization/{MODEL_ID}
The tokenization request for the pre-trained and custom embedding use cases and specified embedding `modelId` (model name) sends text to return results in formats supported by embedding models.
# Get typeahead suggestions
Source: https://doc.lucidworks.com/api-reference/get-typeahead-suggestions
/api-reference/saas/lw-platform-typeahead.json get /typeahead/suggestions
The `GET /typeahead/suggestions` request retrieves a list of typeahead suggestions.
A total of 5,000 suggestions are allowed in the following distribution:
1. Your organization can enter a maximum of 4,000 custom terms (included and excluded) in the Core Settings Typeahead user interface.
2. Every hour, a maximum of 1,000 of the terms entered most often in customer searches are added to the Included typeahead suggestions list.
Results are sorted alphabetically. Sort options are not currently available.
The optional `prefix` query parameter filters suggestions returned. For example, `GET /typeahead/suggestions?prefix=El` returns the suggestions that begin with the letters `El`.
The optional `limit` query parameter limits the maximum number of suggestions returned. For example, `GET /typeahead/suggestions?limit=10` returns a maximum of 10 suggestions.
# Delete a custom model deployment
Source: https://doc.lucidworks.com/api-reference/manage-deployments/delete-a-custom-model-deployment
/api-reference/saas/machine-learning-models.json delete /customers/{DEPLOYMENT_ID}/ai/deployments
Delete the specified custom model deployment.
# Deploy a custom model
Source: https://doc.lucidworks.com/api-reference/manage-deployments/deploy-a-custom-model
/api-reference/saas/machine-learning-models.json post /customers/{DEPLOYMENT_ID}/ai/deployments
Deploy a custom model.
# List a model's deployments
Source: https://doc.lucidworks.com/api-reference/manage-deployments/list-a-models-deployments
/api-reference/saas/machine-learning-models.json get /customers/{CUSTOMER_ID}/ai/models/{MODEL_ID}/deployments
Fetch the list of deployments for the specified model.
# List custom model deployments
Source: https://doc.lucidworks.com/api-reference/manage-deployments/list-custom-model-deployments
/api-reference/saas/machine-learning-models.json get /customers/{DEPLOYMENT_ID}/ai/deployments
Fetch the list of custom model deployments. Pre-trained models are not returned in the response because they are deployed in all available regions.
# Create a model
Source: https://doc.lucidworks.com/api-reference/manage-models/create-a-model
/api-reference/saas/machine-learning-models.json post /customers/{CUSTOMER_ID}/ai/models
Create a custom model and start a training job. The custom model cannot be modified after it is created.
# Get a model
Source: https://doc.lucidworks.com/api-reference/manage-models/get-a-model
/api-reference/saas/machine-learning-models.json get /customers/{CUSTOMER_ID}/ai/models/{MODEL_ID}
Get the details about a specific model.
# List all models
Source: https://doc.lucidworks.com/api-reference/manage-models/list-all-models
/api-reference/saas/machine-learning-models.json get /customers/{CUSTOMER_ID}/ai/models
Fetch the list of pre-trained and custom models.
# Passthrough prompt
Source: https://doc.lucidworks.com/api-reference/passthrough-prompt/passthrough-prompt
/api-reference/saas/machine-learning-platform-prompt-passthrough.json post /ai/prompt/passthrough/{MODEL_ID}
The prompt request for the passthrough use case and specified generative AI (GenAI) `modelId` (model name) sends text to return results in formats supported by generative AI (GenAI) models.
# Create a query rewrite
Source: https://doc.lucidworks.com/api-reference/query-rewrite-management/create-a-query-rewrite
/api-reference/saas/cs-rules-rewrites.json post /query-rewrites
Create a query rewrite in Commerce Studio.
# Delete a query rewrite
Source: https://doc.lucidworks.com/api-reference/query-rewrite-management/delete-a-query-rewrite
/api-reference/saas/cs-rules-rewrites.json delete /query-rewrites/{id}
Delete a query rewrite.
# Get a query rewrite
Source: https://doc.lucidworks.com/api-reference/query-rewrite-management/get-a-query-rewrite
/api-reference/saas/cs-rules-rewrites.json get /query-rewrites/{id}
Retrieve a query rewrite.
# List query rewrites
Source: https://doc.lucidworks.com/api-reference/query-rewrite-management/list-query-rewrites
/api-reference/saas/cs-rules-rewrites.json get /query-rewrites
Note that sorting applies to the rule's staging content only. Filtering applies OR logic, so it matches if either the staging or live content matches the filter. Use `textSearchRelevance` sort to sort by search relevance (cannot be combined with other sort parameters).
# Update a query rewrite
Source: https://doc.lucidworks.com/api-reference/query-rewrite-management/update-a-query-rewrite
/api-reference/saas/cs-rules-rewrites.json put /query-rewrites/{id}
Update a query rewrite.
# Request Access Token
Source: https://doc.lucidworks.com/api-reference/request-access-token/request-access-token
/api-reference/saas/authentication.json post /oauth2/{auth}/v1/token
Use this API to generate an access token.
# Get results by chunkingId
Source: https://doc.lucidworks.com/api-reference/request-chunked-content/get-results-by-chunkingid
/api-reference/saas/machine-learning-platform-async-chunking.json get /ai/async-chunking/{CHUNKING_ID}
Fetch chunked content by chunking ID. The ID is returned in the response of the `POST` operation you used to trigger the chunking operation.
# Create a rule
Source: https://doc.lucidworks.com/api-reference/rule-management/create-a-rule
/api-reference/saas/cs-rules-rewrites.json post /em-rules
Create a rule in Commerce Studio.
# Delete a rule
Source: https://doc.lucidworks.com/api-reference/rule-management/delete-a-rule
/api-reference/saas/cs-rules-rewrites.json delete /em-rules/{id}
Delete a rule.
# Get a rule
Source: https://doc.lucidworks.com/api-reference/rule-management/get-a-rule
/api-reference/saas/cs-rules-rewrites.json get /em-rules/{id}
Retrieve a rule.
# List rules
Source: https://doc.lucidworks.com/api-reference/rule-management/list-rules
/api-reference/saas/cs-rules-rewrites.json get /em-rules
List all rules with optional filtering and pagination. Note that sorting applies to the rule's staging content only.
Filtering applies OR logic, matching if either the staging or live content matches the filter.
Use `textSearchRelevance` sort to sort by search relevance (cannot be combined with other sort parameters).
# Update a rule
Source: https://doc.lucidworks.com/api-reference/rule-management/update-a-rule
/api-reference/saas/cs-rules-rewrites.json put /em-rules/{id}
Update a rule.
# Chunk text using the specified chunker
Source: https://doc.lucidworks.com/api-reference/split-content-into-chunks/chunk-text-using-the-specified-chunker
/api-reference/saas/machine-learning-platform-async-chunking.json post /ai/async-chunking/{CHUNKER}/{MODEL_ID}
This is an all-purpose chunking endpoint you can use by specifying the `chunker` and `modelId` to use.
The `POST` request submits text to be split into chunks based on the `CHUNKER` (chunking strategy) and `MODEL_ID` sent in the request.
The API responds with the following information:
* `chunkingId` that is a unique UUID for the submitted chunking task, and can be used later to retrieve the results.
* `status` that indicates the current state of the chunking task.
IMPORTANT: The available use cases by `chunker` are detailed in their own section in this specification.
# Chunk using dynamic-sentence
Source: https://doc.lucidworks.com/api-reference/split-content-into-chunks/chunk-using-dynamic-sentence
/api-reference/saas/machine-learning-platform-async-chunking.json post /ai/async-chunking/dynamic-sentence/{MODEL_ID}
The dynamic-sentence chunker (chunking strategy) splits the provided text into sentences. Sentences are joined until they reach the `maxChunkSize`. If `overlapSize` is provided, adjacent chunks will both have overlapping sentences on the sides.
# Chunk using semantic chunker
Source: https://doc.lucidworks.com/api-reference/split-content-into-chunks/chunk-using-semantic-chunker
/api-reference/saas/machine-learning-platform-async-chunking.json post /ai/async-chunking/semantic/{MODEL_ID}
The semantic chunker (chunking strategy) creates chunks based on semantic similarity.
Using the model defined in the URL request, the semantic chunker splits text into sentences, encodes the sentences, and then compares the sentence to the building chunk to determine if they are similar enough to group together.
After merging two semantically-similar sentences into a pre-chunk, the semantic chunker needs to encode it to get its vector to compare with the next sentence vector.
This chunker is the slowest of all of the chunkers even if you set the `approximate` field to true.
# Split text on a regex
Source: https://doc.lucidworks.com/api-reference/split-content-into-chunks/split-text-on-a-regex
/api-reference/saas/machine-learning-platform-async-chunking.json post /ai/async-chunking/regex-splitter/{MODEL_ID}
The regex-splitter chunker (chunking strategy) splits the submitted text based on the specified regex (regular expression), according to the conventions employed by the `re` python package. For more information about the `re` operations, see https://docs.python.org/3/library/re.html.
# Split text on newlines
Source: https://doc.lucidworks.com/api-reference/split-content-into-chunks/split-text-on-newlines
/api-reference/saas/machine-learning-platform-async-chunking.json post /ai/async-chunking/dynamic-newline/{MODEL_ID}
The dynamic-newline chunker (chunking strategy) splits the provided text on all new line characters. Then all of the split chunks under the `maxChunkSize` limit will be merged.
# Split text on sentences
Source: https://doc.lucidworks.com/api-reference/split-content-into-chunks/split-text-on-sentences
/api-reference/saas/machine-learning-platform-async-chunking.json post /ai/async-chunking/sentence/{MODEL_ID}
The sentence chunker (chunking strategy) splits text on sentences.
# Get use cases
Source: https://doc.lucidworks.com/api-reference/use-cases/get-use-cases
/api-reference/saas/machine-learning-platform-use-case.json get /ai/usecase
Fetch the list of supported use cases with related models.
# Deprecations and Removals
Source: https://doc.lucidworks.com/docs/5/fusion/deprecations-and-removals
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/deprecations-and-removals
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/deprecations-and-removals
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/8855
This document provides a comprehensive list of deprecated and removed features in Fusion, organized by version. It includes details on the deprecation and expected removal of various features, along with notes on alternatives or replacements where applicable. The document also contains links to relevant documentation for further information.
The deprecations section lists features scheduled for removal in future releases, and the removals section lists features already removed in past releases. Each entry includes the feature name, the version in which it was deprecated or removed, and relevant notes or recommended replacements.
The deprecation schedule outlined in this document applies exclusively to Fusion 5. Other major versions of Fusion have separate deprecation schedules not covered by this document.
Fusion Connectors deprecations and removals
Deprecation and removal details for Fusion Connectors are found at [Fusion Connectors Deprecations and Removals](/docs/fusion-connectors/deprecations-and-removals).
## Deprecations
Deprecated features in Fusion 5 are scheduled for removal in a future release. The following table lists each deprecated feature with an expected removal value, either a version number or a date. When a version number is specified, the feature will be removed in that specific release. When a date is specified, it represents the earliest possible day the feature will remain available. Removal will not occur before that date.
**FAQ**
When were these features originally deprecated?
If a feature was originally deprecated in a prior version, the original deprecation version is noted as a footnote in the **Deprecated** column.
| Feature | Deprecated | Expected Removal | Notes |
| - | - | - | - |
| [ALS Recommender Job](/docs/5/fusion/reference/config-ref/jobs/als-recommender/overview): The ALS Recommender job in Fusion AI is used to compute user recommendations or item similarities using a collaborative filtering recommender. | All versions of 5.9.x\[[4](#_footnoteref_4 "View footnote.")] | Version TBD | Use the [BPR Recommender job](/docs/5/fusion/reference/config-ref/jobs/bpr-recommender) instead of the ALS Recommender job. The BPR Recommender job delivers better recommendation accuracy with a shorter runtime. |
| [Apache Tika Parser](/docs/5/fusion/reference/config-ref/parser-stages/apache-tika-parser): The Apache Tika Parser is a versatile tool designed to support the parsing of numerous unstructured document formats. | All versions of 5.9.x\[[3](#_footnoteref_3 "View footnote.")] | Version TBD | Use the [Tika Asynchronous Parser](/docs/5/fusion/getting-data-in/indexing/asynchronous-tika-parsing) instead. Asynchronous Tika parsing performs parsing in the background. This allows Fusion to continue indexing documents while the parser is processing others, resulting in improved indexing performance for large numbers of documents. |
| [App Insights](/docs/5/fusion/getting-data-out/data-analytics/app-insights/overview): A tool within the Fusion workspace that provides real-time, searchable reports and visualizations from signals data. | 5.9.13\[[1](#_footnoteref_1 "View footnote.")] | Version TBD, but no earlier than October 22, 2025 | - |
| [Banana Dashboards](/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/overview): Banana serves as the underlying browser application for Fusion dashboards, enabling the creation of dynamic and interactive dashboards within Fusion. These dashboards support various data visualizations and user interactions. | 5.9.13\[[1](#_footnoteref_1 "View footnote.")] | Version TBD, but no earlier than October 22, 2025 | Deploy Banana Dashboards using a separate Docker deployment option, available in a future release. This approach provides continued support for users who rely on Banana Dashboards while allowing for independent management and updates outside of Fusion’s core deployment. |
| [Data Augmentation Job](/docs/5/fusion/reference/config-ref/jobs/data-augmentation): The Data Augmentation Job is designed to enhance training and testing data for machine learning models by increasing data quantity and introducing textual variations. It performs tasks such as backtranslation, synonym substitution, keystroke misspelling, and split word tasks. | 5.9.13\[[1](#_footnoteref_1 "View footnote.")] | Version TBD, but no earlier than October 22, 2025 | - |
| [Domain-Specific Language (DSL)](/docs/5/fusion/getting-data-out/query-basics/domain-specific-language): The Domain Specific Language (DSL) in Fusion is designed to simplify the complexity of crafting search queries. It allows users to express complex search queries without needing to understand the intricate syntax required by the legacy Solr parameter format. | 5.9.4 | Version TBD | Avoid using the Domain-Specific Language (DSL) feature, as it may cause performance degradation. Instead, use the [DSL to Legacy Parameters](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/dsl-to-legacy-query-stage) query pipeline stage to convert DSL requests to the legacy Solr format while maintaining compatibility. |
| [Field Parser Index Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/field-parser-index-stage): The Field Parser index pipeline stage in Fusion is designed to parse content embedded within fields of documents. This stage operates separately from the parsers that handle whole documents. | All versions of 5.9.x\[[3](#_footnoteref_3 "View footnote.")] | Version TBD | Use the [Tika Asynchronous Parser](/docs/5/fusion/getting-data-in/indexing/asynchronous-tika-parsing) instead. Asynchronous Tika parsing performs parsing in the background. This allows Fusion to continue indexing documents while the parser is processing others, resulting in improved indexing performance for large numbers of documents. |
| [Head/Tail Analysis](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis): Perform head/tail analysis of queries from collections of raw or aggregated signals, to identify underperforming queries and the reasons. | 5.9.15 | TBD | Instead, use [Neural Hybrid Search](/docs/5/fusion/hybrid-search/overview) for superior relevance compared to this legacy machine learning method. |
| [Hybrid Query (5.9.9 and earlier)](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/hybrid-search-query): A stage to perform hybrid lexical-semantic search that combines BM25-type lexical search with KNN dense vector search via Solr. | 5.9.13 | TBD | Use [Neural Hybrid Query Stage (5.9.10 and later)](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/neural-hybrid-query) instead to benefit from updates and improvements to the search capabilities of the stage. |
| [Logistic Regression Classifier Training Jobs](/docs/5/fusion/reference/config-ref/jobs/logistic-regression-classifier-training): This job trains a logistic regression model with regularization to classify text into different categories. | All versions of 5.9.x\[[4](#_footnoteref_4 "View footnote.")] | Version TBD | Replace Logistic Regression Classifier Training Jobs with the [Classification job](/docs/5/fusion/reference/config-ref/jobs/classification). This alternative provides expanded configuration options and improved logging capabilities. |
| [Milvus Ensemble Query Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-ensemble-query-stage): The Milvus Ensemble Query stage is used to enhance search results by incorporating vector-based similarity scoring. | 5.9.5 | TBD or no later than May 4, 2025 | Replace the Milvus Ensemble Query Stage with [Seldon](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/vectorize-query-via-seldon-query-stage) or [Lucidworks AI vector query stages](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/lwai-vectorize-query). These alternatives improve vector search integration and support within Fusion’s evolving AI and machine learning capabilities. |
| [Milvus Query Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-query-stage): The Milvus Query stage performs vectors similarity search in Milvus, an open source vector similarity search engine integrated into Fusion to streamline its deep learning capabilities and reduce the workload on Solr. | 5.9.5 | TBD or no later than May 4, 2025 | Replace the Milvus Ensemble Query Stage with [Seldon](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/vectorize-query-via-seldon-query-stage) or [Lucidworks AI vector query stages](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/lwai-vectorize-query). These options enhance query efficiency and provide broader support for machine learning-driven search. |
| [Milvus Response Update Query Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-response-update-stage): The Milvus Response Update stage is designed to update response documents with vectors similarity and ensemble scores. | 5.9.5 | TBD or no later than May 4, 2025 | Replace the Milvus Ensemble Query Stage with [Seldon](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/vectorize-query-via-seldon-query-stage) or [Lucidworks AI vector query stages](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/lwai-vectorize-query). These alternatives improve performance when updating response documents with vector similarity data. |
| [Parsers Indexing CRUD API](/api-reference/parsers-crud-api/list-all-parsers): The Parsers API provides CRUD operations for parsers, allowing users to create, read, update, and delete parsers. | 5.9.11\[[1](#_footnoteref_1 "View footnote.")] | Version TBD, but no earlier than September 4, 2025 | A new API introduced in Fusion 5.12.0, Async Parsing API, will replace the Parsers Indexing CRUD API and will be backported to Fusion 5.9.x. This API provides improved functionality and aligns with Fusion’s updated architecture, ensuring consistency across versions. |
| [Phrase Extraction](/docs/5/fusion/reference/config-ref/jobs/phrase-extraction): Identify multi-word phrases in signals. | 5.9.15 | TBD | Instead, use [Neural Hybrid Search](/docs/5/fusion/hybrid-search/overview) for superior relevance compared to this legacy machine learning method. |
| [Query-to-Query Collaborative Similarity Job](/docs/5/fusion/reference/config-ref/jobs/query-to-query-collaborative-similarity): This job uses SparkML’s Alternating Least Squares (ALS) to analyze past queries and find similarities between them. It helps recommend related queries or suggest relevant items based on previous searches. | All versions of 5.9.x\[[4](#_footnoteref_4 "View footnote.")] | Version TBD | Switch to [Query-to-Query Session-Based Similarity jobs](/docs/5/fusion/reference/config-ref/jobs/query-to-query-session-based-similarity) instead of the Query-to-Query Collaborative Similarity Job. The new method improves performance and increases the coverage of query similarity calculations. |
| [Random Forest Classifier Training Jobs](/docs/5/fusion/reference/config-ref/jobs/random-forest-classifier-training): This job trains a machine learning model using a random forest algorithm to classify text into different categories. | All versions of 5.9.x\[[4](#_footnoteref_4 "View footnote.")] | Version TBD | Use the [Classification job](/docs/5/fusion/reference/config-ref/jobs/classification) instead of Random Forest Classifier Training Jobs. This alternative provides enhanced configurability and better logging for improved model training. |
| [Security Trimming Query Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/security-trimming-query-stage): The Security Trimming query pipeline stage in Fusion is designed to restrict query resultsby matching security ACL metadata, ensuring that users only see results they are authorized to access. | 5.9.0 | Version TBD | Replace the Security Trimming Query Stage with the [Graph Security Trimming Query Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/security-trimming-graph-query-stage). The new method uses a single filter query across all data sources. |
| [Smart Answers Coldstart Training job](/docs/5/fusion/reference/config-ref/jobs/smart-answers-coldstart-training): The Smart Answers Coldstart Training job in Fusion is designed to help train models when there is no historical training data available. | 5.9.13\[[1](#_footnoteref_1 "View footnote.")] | Version TBD, but no earlier than October 22, 2025 | Use pre-trained models or the supervised training job instead of the Smart Answers Coldstart Training job. Pre-trained models eliminate the need for manual training when historical data is unavailable, while supervised training jobs offer greater flexibility in model customization. |
| [Support for Nashorn Javascript engine](/docs/5/fusion/getting-data-in/indexing/index-pipelines/custom-javascript-index-stages#the-javascript-engine-used-by-fusion): Fusion uses the Nashorn engine JavaScript engine for the JavaScript index and query stages. | 5.9.8 | Version TBD, but no earlier than July 7, 2025 | Use the OpenJDK Nashorn JavaScript engine instead of the deprecated Nashorn JavaScript engine in Fusion. This ensures continued JavaScript execution compatibility in pipeline configurations. You can select the engine from a dropdown in the pipeline views or in the workbenches. |
| [Synonym Detection](/docs/5/fusion/reference/config-ref/jobs/synonym-detection): Used to generate pairs of synonyms and pairs of similar queries. Two words are considered potential synonyms when they are used in a similar context in similar queries. | 5.9.15 | TBD | Instead, use [Neural Hybrid Search](/docs/5/fusion/hybrid-search/overview) for superior relevance compared to this legacy machine learning method. |
| [Time-based partitioning](/api-reference/collections/get-collection-partitions-status): Time-based partitioning in Fusion collections allows mapping to multiple Solr collections or partitions based on specific time ranges. | All versions of 5.9.x\[[4](#_footnoteref_4 "View footnote.")] | Version TBD | - |
| [Token and Phrase Spell Correction](/docs/5/fusion/reference/config-ref/jobs/token-and-phrase-spell-correction): Detect misspellings in queries or documents using the numbers of occurrences of words and phrases. | 5.9.15 | TBD | Instead, use [Neural Hybrid Search](/docs/5/fusion/hybrid-search/overview) for superior relevance compared to this legacy machine learning method. |
| [Word2Vec Model Training Jobs](/docs/5/fusion/reference/config-ref/jobs/word2vec-model-training): The Word2Vec model training job trains a shallow neural model to generate vector embeddings for text data and stores the results in a specified output collection. It supports configurable parameters for input data, model tuning, featurization, and output settings. | 5.9.11\[[4](#_footnoteref_4 "View footnote.")] | Version TBD, but no earlier than September 4, 2025 | - |
| Connectors fetcher property `AccessControlFetcher`: Connectors that support security filtering previously used separate fetchers for content and access control. One fetcher type is now used for both content and security fetching. `AccessControlFetcher` has been deprecated. | All versions of 5.9.x\[[5](#_footnoteref_5 "View footnote.")] | Version TBD | Fetcher implementations that use `AccessControlFetcher` should instead use `ContentFetcher`. |
| Messaging Stage Configs | All versions of 5.9.x\[[5](#_footnoteref_5 "View footnote.")] | Version TBD | - |
| MLeap deployments of SpaCy and SparkNLP | 5.9.10\[[4](#_footnoteref_4 "View footnote.")] | 5.9.12 | Use **Develop and Deploy a Machine Learning Model** instead. |
| Webapps Service: The Webapps service provides an embedded instance of App Studio within each Fusion instance, simplifying the deployment process. | 5.9.10\[[2](#_footnoteref_2 "View footnote.")] | Version TBD, but no earlier than August 20, 2025 | Deploy App Studio Enterprise using the **Deploy App Studio Enterprise to a Fusion 5 Cluster (GKE)** instead of relying on the Webapps service. This method improves scalability and provides a more robust deployment approach for enterprise environments. |
## Removals
This section lists features that have been removed in past releases of Fusion. Each entry includes the feature name, the version in which it was removed, and any relevant notes or replacement recommendations.
| Feature | Removed | Notes |
| - | - | - |
| **Fusion 5.11** | | |
| MLeap | 5.11.0 | Use **Develop and Deploy a Machine Learning Model** instead. This method provides a more flexible and modern approach to deploying machine learning models. |
| Subscriptions UI | 5.11.0 | - |
| **Fusion 5.10** | | |
| Forked Apache Tika Parser | 5.10.0 | The Forked Apache Tika Parser was replaced by the [Tika Asynchronous Parser](/docs/5/fusion/getting-data-in/indexing/asynchronous-tika-parsing). The asynchronous parser improves performance by handling document parsing more efficiently and scaling better for enterprise workloads. |
| **Fusion 5.9** | | |
| Analytics Catalog Query Stage | 5.9.15 | - |
| [Tika Server Parser](/docs/5/fusion/reference/config-ref/parser-stages/apache-tika-server) | 5.9.12 | Use the [Tika Asynchronous Parser](/docs/5/fusion/getting-data-in/indexing/asynchronous-tika-parsing) instead. Asynchronous Tika parsing performs parsing in the background. This allows Fusion to continue indexing documents while the parser is processing others, resulting in improved indexing performance for large numbers of documents. |
| MLeap in Machine Learning models | 5.9.12 | - |
| **Fusion 5.7** | | |
| NLP Annotator Index Stage | 5.7.0 | To implement similar functionality, see the **Develop and Deploy a Machine Learning Model** guide, which provides an adaptable example. |
| NLP Annotator Query Stage | 5.7.0 | To implement similar functionality, see the **Develop and Deploy a Machine Learning Model** guide, which provides an adaptable example. |
| OpenNLP NER Extraction Index Stage | 5.7.0 | To implement similar functionality, see the **Develop and Deploy a Machine Learning Model** guide, which provides an adaptable example. |
| **Fusion 5.6** | | |
| Fusion SQL | 5.6.1 | - |
| Apache Pulsar | 5.6.0 | Apache Pulsar was removed in Fusion 5.6.0 and replaced with Kafka. Kafka offers better scalability, reliability, and industry support for message streaming. |
| Log Viewer & DevOps Center UI panel | 5.6.0 | These features were removed in Fusion 5.6.0 as they depended on Apache Pulsar, which has been replaced by Kafka. Users should transition to Kafka-based logging and monitoring solutions. |
| Subscriptions API | 5.6.0 | - |
| Send to Message Bus Index Stage | 5.6.0 | - |
| **Fusion 5.5** | | |
| Jupyter | 5.5.2 | - |
| Superset | 5.5.2 | - |
***
1. Originally deprecated in Fusion 5.12.0.
2. Originally deprecated in Fusion 5.11.0.
3. Originally deprecated in Fusion 5.8.0.
4. Originally deprecated in Fusion 5.2.0.
5. Originally deprecated in Fusion 5.1.2.
# Introduction to Fusion
Source: https://doc.lucidworks.com/docs/5/fusion/intro/overview
[localhost link]: http://localhost:3000/docs/5/fusion/intro/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/intro/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/3210
Fusion combines the Apache Solr open source search engine with the distributed power of Apache Spark for artificial intelligence. Highly scalable, Fusion indexes and stores data for real-time discovery.
The learning path for **Fusion Basics** focuses on the basic structure, essential components, and foundational concepts you’ll need to understand Fusion.
# Self-hosted Fusion
Source: https://doc.lucidworks.com/docs/5/fusion/overview
[localhost link]: http://localhost:3000/docs/5/fusion/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/6764
Lucidworks Fusion 5 lets customers easily deploy AI-powered data discovery and search applications in a modern, containerized, cloud-native architecture. Data scientists interact with those applications by:
* Leveraging existing machine learning models and workflows
* Using popular tools (Python ML, TensorFlow, scikit-learn, and spaCy) to quickly create and deploy new models
Fusion combines the Apache Solr open source search engine with the distributed power of Apache Spark for artificial intelligence. Highly scalable, Fusion indexes and stores data for real-time discovery.
* Index billions of records of any type, from any data source
* Process thousands of queries per second from thousands of concurrent users
* Conduct full-text search using standard SQL capabilities and powerful analytics
## Key concepts
Fusion’s ecosystem allows you to manage and access your data in an intuitive fashion.
### Apache Solr
Solr is the fast open source search platform built on Apache Lucene™ that provides scalable indexing and search, as well as faceting, hit highlighting, and advanced analysis/tokenization capabilities. Solr and Lucene are managed by the [Apache Software Foundation](http://www.apache.org/).
For more information, see the [Solr Reference Guide](/docs/5/fusion/reference/solr-reference-guide) for your Fusion release.
### Apache Spark
[Apache Spark](http://spark.apache.org/) is an open source cluster-computing framework that serves as a fast and general execution engine for large-scale data processing jobs that can be decomposed into stepwise tasks, which are distributed across a cluster of networked computers.
Spark improves on previous MapReduce implementations by using resilient distributed datasets (RDDs), a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner.
See [Spark Operations](/docs/5/fusion/intro/fusion-stack/spark/overview) for more information.
### Connectors
Connectors are the out-of-the-box components for pulling your data into Fusion. Lucidworks provides a wide variety of connectors, each specialized for a particular data type. When you add a [datasource](/docs/fusion-connectors/concepts/datasources) to a [collection](/docs/5/fusion/getting-data-in/indexing/collections/overview), you specify the connector to use for ingesting data.
Connectors are distributed separately from Fusion. For complete information, see [Fusion Connectors](/docs/fusion-connectors/overview).
Fusion offers dozens of connectors so you can access your data from a large variety of sources.
To learn more about Fusion connectors, see [connectors concepts](/docs/5/fusion/getting-data-in/indexing/connectors) or the [connectors section](/docs/fusion-connectors/overview).
### Pipelines
Pipelines dictate how data flows through Fusion and becomes accessible by a search application. Fusion has two types of pipelines: index pipelines and query pipelines.
[Index pipelines](/docs/5/fusion/getting-data-in/indexing/overview) ingest data, indexes it, and stores it in a format that is optimized for searching.
[Query pipelines](/docs/5/fusion/getting-data-out/query-basics/query-pipelines/overview) filter, transform, and augment Solr queries and responses to return all and only the most relevant search results.
## How-to information
Want to start right away? See our 4-part **Getting Started with Fusion** tutorial below for detailed instructions.
## Create an app
Create a Movie Search app. An app is a set of Fusion objects that performs a specific searching task (such as searching for movies).
1. In the Fusion launcher, click **Create new app**.
2. In the **App Name** field, enter `Movie Search`.
3. In the **App Description** field, enter `App to search for movies`.
4. Click **Create App**.
The Movie Search app now appears in the Fusion launcher:
## Next steps
You now have the MovieLens dataset from which you will use a CSV file that contains data about movies. And you have a Fusion app that you will transform into a movie search app.
In Part 2, you will use Index Workbench to get the MovieLens dataset into Fusion.
Fusion’s Index Workbench provides the tools to configure datasources, parsers, and index pipelines. It lets you *preview* the results of indexing before you load your data into the actual index.
Index Workbench first sets up the necessary data extraction configuration, and then retrieves a small number of documents as sample data. You can use the sample documents to test and refine your index pipeline. All processing is *simulated* processing of the test data. No actual data ingestion takes place.
After you have a complete configuration, Index Workbench saves this as a Fusion datasource.
To load your data into Fusion, use the Fusion Datasource tool to run the resulting configuration.
Part 2 takes you through configuring a datasource using Index Workbench. In Part 3, you will load the data into Fusion and view it using Query Workbench.
## Before you begin
To proceed with this part of the tutorial, you must first complete Part 1, which gives you a Fusion app.
## Download the MovieLens dataset
1. [Download the dataset](https://files.grouplens.org/datasets/movielens/ml-latest-small.zip).
This is a MovieLens dataset created by the [Grouplens](https://grouplens.org/) research lab.
2. Unpack the `ml-latest-small.zip` file.
Fusion can parse `.zip` files, but for simplicity we will index just one file from the archive (`movies.csv`).
The `movies.csv` file contains a list of 9,125 movie titles, plus a header row. Here is a truncated listing:
```csv theme={"dark"}
movieId,title,genres
1,Toy Story (1995),Adventure|Animation|Children|Comedy|Fantasy
2,Jumanji (1995),Adventure|Children|Fantasy
3,Grumpier Old Men (1995),Comedy|Romance
4,Waiting to Exhale (1995),Comedy|Drama|Romance
5,Father of the Bride Part II (1995),Comedy
6,Heat (1995),Action|Crime|Thriller
7,Sabrina (1995),Comedy|Romance
8,Tom and Huck (1995),Adventure|Children
9,Sudden Death (1995),Action
10,GoldenEye (1995),Action|Adventure|Thriller
```
## Open the Movie Search app
1. If the Fusion UI is not already open, then open it.
2. Enter the password for the user `admin`, and then click **Log in**.
The Fusion launcher appears. You see the Movie Search app you created in Part 1:
3. In the Fusion launcher, click the **Movie Search** app.
The Fusion workspace appears. It has controls along the left and top sides.
4. In the upper left, hover over Apps . You can see that Movie Search is the currently selected app.
Also, the user collection `Movie_Search` is selected in the collection picker. This is the default collection for the Movie Search app, and where Fusion will place index data.
## Configure the datasource
A collection includes one or more datasources. A datasource is a configuration that manages the import, parsing, and indexing of data into a collection. You will use Index Workbench to configure a datasource for the movie data.
1. In the collection picker, verify that the collection **Movie\_Search** is selected.
2. Open Index Workbench. Navigate to **Indexing** > **Index Workbench**.
Initially, no data preview appears because no datasource has been configured. When you configure a datasource, Fusion samples the data and displays a preview of how it would be formatted in the index using the default parsing and index pipeline configurations.
3. In the upper right, click **New**.
4. Select **Or, upload a file**.
5. Click **Choose File**.
6. Navigate to the `movies.csv` file, select it, and then click **Open**.
7. Click **Add New Datasource**.
The Datasource (File Upload) configuration panel appears, with the default datasource ID `movies_csv-Movie_Search` and the default file ID `movies.csv`. These default values are fine.
8. Enter the **Description** `Movies CSV file`.
9. Click **Apply**.
Index Workbench reads up to 20 documents into memory from the `movies.csv` file, and then displays a preview of how they would be indexed.
You have finished configuring the datasource. At the bottom of the page, click **Cancel**.
In the lower right, you can select the number of documents to preview.
## Analyze the default output
1. Notice that Fusion made some assumptions about your original fields:
* `genres` became `genres_t` (the `text_general` field type) and `genres_s` (the `string` field type). String fields are useful for faceting and sorting, while text fields are for full-text search. At this point, Fusion does not know whether you intend to use this field for faceting and sorting, for full-text search, or for both.
* `title` became `title_t` and `title_s` for the same reason.
* `movieId` became `movieId_t` and `movieId_s` for the same reason. This might seem odd, because the original field contains numbers. But, at this stage, Fusion creates `text_general` and `string` fields. To use the contents of this field as an integer, you would map the field to an integer field.
You also see fields that begin with `_lw`. These fields contain data that Fusion creates for its own housekeeping. You can ignore them.
These fields are created by the [Solr Dynamic Field Name Mapping stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/solr-dynamic-field-name-mapping-index-stage) in the default index pipeline. This stage attempts to automatically detect field types, and renames fields accordingly. For this tutorial, you will manually configure the fields instead.
2. Turn off the **Solr Dynamic Field Name Mapping** stage by clicking the green circle next to it.
Your data’s original fields reappear: `genres`, `movieId`, and `title`.
## Configure the index pipeline
First you will configure the field mappings in the index pipeline so each field has the correct data type. Then you will split the `genres` field into multiple values so each value can be used as a facet in Part 3 of this tutorial.
### Configure field mappings
Configure field mappings to control the field types of Fusion documents. Fusion uses field name suffixes to determine field types. When a field name has no suffix, Fusion stores it as a string field and treats it as an unanalyzed whole. For precise analysis and search, most fields need suffixes to indicate their specific types. You will see how this relates to the fields in the dataset.
1. In the list of index pipeline stages on the left, click **Field Mapping** to open the Field Mapping stage configuration panel.
2. In **Field Translations**, click **Add** to create a new field mapping rule.
3. Under **Source Field**, enter `genres`.
4. Under **Target Field**, enter `genres_ss`.
The field suffix `_ss` means that this field is a multi-valued string field.
Fusion currently interprets this field as having a single value. You can see that the field actually contains a pipe-delimited array of values. You will fix this after you finish configuring field mappings.
5. Under **Operation**, select **move**.
The move operation means that the resulting document no longer has a `genres` field; it only has `genres_ss`.
6. Click **Apply**.
Applying the new configuration re-runs the simulation and updates the contents of the preview panel. Notice the change in the field name from `genres` to `genres_ss`:
| | |
| - | - |
| Before | After |
| | |
7. Click **Add** to add more field mapping rules as follows:
* The `movieId` field is a unique document identifier. It should be *copied* into the document’s `id` field.
* The `title` should be searchable as a text field, so you *move* it to the field `title_txt`.
Your field mappings should look like this:
8. Click **Apply**.
After you have specified these explicit field mapping rules, you can browse the resulting documents
in the preview panel to check your work.
| | |
| - | - |
| Before | After |
| | |
9. In the upper right, click **Save**. This saves your modified index pipeline. Get in the habit of saving your work as you work.
Now your document ID is more useful, and your movie titles are full-text searchable.
Because the input documents in this tutorial are simple documents with a fixed number of known fields, it is easy to configure the Field Mapping stage to ensure the correct document structure for Fusion. When documents have large numbers of fields, the [Solr Dynamic Field Mapping stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/solr-dynamic-field-name-mapping-index-stage) can reduce the work required to configure the index pipeline.
### Split a multi-value field
The `genres_ss` field has been parsed as a single-value field, but you can see that it is really a pipe-delimited array of values. To split this field into its constituent values, you will add a [Regex Field Extraction stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/solr-dynamic-field-name-mapping-index-stage) to your index pipeline. This stage uses regular expressions to extract data from specific fields. It can append or overwrite existing fields with the extracted data, or use the data to populate new fields.
1. Click **Add a stage**.
2. Scroll down and select **Regex Field Extraction** (under Field Transformation).
The Regex Field Extraction stage configuration panel appears.
3. Under **Regex Rules**, click **Add** .
4. On the new line, hover over the `[...]` under **Source Fields**, and then click **Edit** .
The Source Fields window opens.
5. Click **Add** .
6. Enter `genres_ss`, and then click **Apply**.
7. Under **Target Field**, enter `genres_ss`.
8. In the **Write Mode** field, select **overwrite**.
9. In the **Regex Pattern** field, enter this expression:
```
[^|\s][^\|]*[^|\s]*
```
You might need to scroll horizontally to see this field.
The first bracketed term in the regex matches any character that is not a vertical bar or a space.
The second term matches any character that is not a vertical bar, zero or more times.
The last term matches any character that is not a vertical bar, zero or more times.
10. In the **Return If No Match** field, select `input_string`.
11. Click **Apply**.
Initially, your data does not change.
12. In the list of index pipeline stages, drag the **Regex Field Extraction** stage down so that it comes after the Field Mapping stage:
Now the preview shows multiple values for the `genres_ss` field:
| | |
| - | - |
| Before | After |
| | |
If the preview panel does not update automatically, select a different number of documents to view using the dropdown in the bottom right of the screen. This forces the preview to update.
13. To view the values of the `genres_ss` field, expand it and `values` under it by clicking the right triangle :
These field values are useful for faceting, which you will explore in Part 3 of this tutorial.
14. In the upper right, click **Save**. This saves your modified index pipeline.
### Create a new field from part of an existing one
Notice that the `title_txt` field also contains the year in which the movie was released. Instead of including the year in your full-text search field, it would be more useful as a separate field that you can use for faceting. This is another job for the [Regex Field Extraction stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/solr-dynamic-field-name-mapping-index-stage).
1. In the list of index pipeline stages, click **Regex Field Extraction**.
2. In the Regex Field Extraction configuration panel, under Regex Rules, click **Add** .
3. On the new line, hover over the `[...]` under **Source Fields**, and then click **Edit** .
The Source Fields window appears.
4. Click **Add** .
5. Enter `title_txt`, and then click **Apply**.
6. Under Target Field, enter `year_i`.
The `_i` suffix indicates an integer point field (specifically, that the field is a dynamic field with a `pint` field type). Fusion will create this new field whenever the regular expression matches the contents of the source field.
When you use the Regex Field Extraction stage to create a new field, the value of **Write Mode** makes no difference.
7. In the **Regex Pattern** field, enter this expression to match the digits inside the parentheses at the end of the `title_txt` value:
```
\(([0-9]+)\)$
```
8. In the **Regex Capture Group** field, enter `1`. This lets the index pipeline stage transfer the year into the `year_i` field.
Scroll all the way to the right to see this field.
9. Click **Apply**.
Now the preview includes the new `year_i` field:
| | |
| - | - |
| Before | After |
| | |
10. In the upper right, click **Save**. This saves your modified index pipeline.
### Trim a field's value
The `title_txt` field still includes the year of the film’s release, which you have extracted into its own field, `year_i`. Let us trim that information from the `title_txt` values so that only the title text remains.
1. In the list of index pipeline stages, click **Regex Field Extraction**.
2. In the Regex Field Extraction configuration panel, under Regex Rules, click Add .
3. On the new line, hover over **Source Fields**, and then click Edit .
The Source Fields window appears.
4. Click **Add** .
5. Enter `title_txt`, and then click **Apply**.
6. Under Target Field, enter `title_txt`.
7. In the **Write Mode** field, select `overwrite`.
8. In the **Regex Pattern** field, enter this expression to match the digits inside the parentheses at the end of the `title_txt` value:
```
^(.+)\s\(([0-9]+)\)$
```
9. In the **Regex Capture Group** field, enter `1`.
10. Click **Apply**.
Now the preview pane shows the `title_txt` field with only the title string:
| | |
| - | - |
| Before | After |
| | |
11. In the upper right, click **Save**. This saves your modified index pipeline.
## Run the datasource job
Now you have a correctly-configured index pipeline appropriate to your data. You are ready to index the data.
1. In the upper left, click **Start job**.
This launches a datasource job that imports and indexes the complete contents of your `movies.csv` file, using the configuration you just saved.
Your datasource job is finished when the Index Workbench displays `Status: success` in the upper left. If the status does not change, go back to the launcher and relaunch your app.
## Close panels you no longer need open
Fusion opens panels beside already open panels. Close all of the panels that are open by clicking **Close** .
## Reindexing
Documents are associated with a collection through the name of the datasource. The datasource name is stored as a value in the `_lw_data_source_s` field. For various reasons, you may wish to remove all documents associated with a datasource from a collection before using CrawlDB to add relevant documents back to the collection. This process is known as **reindexing**.
To accomplish this, navigate to **Indexing** > **Datasources**, select the datasource name, and then click **Clear Datasource**. This will remove all documents with the selected datasource name in the `_lw_data_source_s` field. After the documents are removed from the collection, you can repeat \[the steps above"/> to reindex the data.
Use caution when creating or changing a datasource name. If you create a new datasource name that is identical to an existing datasource name, all document associations will be shared between the datasource names. Changing the name to match an existing name will have the same result.
## What is next
Now you have 9,125 movie listings from the MovieLens database in Fusion’s index, customized to indicate the data type for each field. You also split a multi-valued field so that its values can be treated individually, created a new field to contain partial contents of a different field, and trimmed that content from the original field.
Let us compare the initial indexing of your data with the indexing after field mappings and extractions:
| Before | After |
| - | - |
| | |
In Part 3, you will use Query Workbench to get search results from your collection and configure the query pipeline that customizes those results. You will add faceting using the `genres_ss` and `year_i` fields so that users can easily filter their search results.
In Part 2, you used Index Workbench to get data into Fusion by previewing the dataset, configuring the index pipeline, and then indexing the data.
In Part 3 (this part), you will explore Query Workbench and learn how to configure Fusion’s output (search results), including faceting. Facets are the ubiquitous, dynamic lists of categories or features offered as filters within a search results page.
## Before you begin
To proceed with this part of the tutorial, you must first complete Part 1 and Part 2, which give you an indexed dataset for Query Workbench to read.
The dataset has three fields that users of your search application might find relevant:
* `genres_ss`. A list of one or more genre labels
* `title_txt`. The name of the movie
* `year_i`. The movie’s year of release
The field suffixes indicate the type of data stored in each field:
* Fields with the suffix `_ss` (multi-valued string fields) contain one or more strings values.
String fields require an exact match between the query string and the
string value stored in that field.
* Fields with the suffix `_txt` (text fields) contain text.
Text fields allow for free text search over the field contents.\
For example, because the movie titles are stored in a text field, a search on
the word "Star" will match movies titled "Star", "A Star is Born", all movies in the
Star Wars and Star Trek franchises, as well as "Dark Star", "Lone Star", and "Star Kid".
* Fields with the suffix `_i` (point integer fields) contain integer values.
Numeric fields allow range matches as well as exact matches, and point integer fields allow efficient comparisons between the field’s values and the search criteria.
The different field types allow for different kinds of searches. Configuration of the query pipeline determines how fields are searched.
## Explore search results
Query Workbench lets you interactively configure a query pipeline while previewing the search results it produces. A query pipeline converts a free text query submitted to your search application into a structured query for Solr.
1. Log in to Fusion.
2. Click the **Movie Search** app.
The Fusion workspace appears.
3. Open Query Workbench. Navigate to **Querying** > **Query Workbench**.
### Perform a search
The default search is the wildcard search (`\*:*`), which returns all documents in the collection. You will enter a different [search query](/docs/5/fusion/getting-data-out/query-basics/query-language-cheat-sheet) to get started with facet configuration.
1. In the search box, enter the query string `star`, and then press Enter or click **Search** .
This search returns all movies that have the word "star" in the title.
The output is configured by the default query pipeline, which has the same name as the collection (4.x) or app (5.x). In this case, the name is `movie-search`.
To see more of the default output, you can perform other searches if you like.
1. In the search box, enter the query string `\*:*` to return all documents, and then press Enter or click **Search** .
If you wish to refine your results, you can change your index pipeline configuration and reindex your data.
### Explore the role of query pipeline stages
A default query pipeline consists of the stages below.
Some of these support Fusion AI features for advanced relevancy tuning, which you can read about separately.
In this tutorial, we will configure a couple of the basic stages:
* **Boost with Signals.** Use signals data to boost relevant documents.
* **Query Fields.** Specify the set of fields over which to search.
We will configure this stage later in this tutorial.
* **Field Facet.** Specify the fields to use for faceting.
We will configure this stage, too, for basic faceting and range faceting.
* **Text Tagger.** Look for known phrases, synonyms, misspellings, and so on, that can be used to improve the query with [query rewriting](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting).
* **Apply Rules.** Modify the query using [business rules](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/overview), if any are triggered.
* **Solr Query.** Perform the query and return the results.
This is the only stage that is *always required* to perform a query and receive results.
* **Modify Response with Rules.** Modify Solr’s response using business rules, if any are triggered.
1. Turn off the Solr Query stage by clicking on the green circle on the left. The circle will change to white and Solr Query will dim to indicate the Solr Query stage is off.
All search results disappear from the preview pane because Fusion does not send a query to Solr.
2. Turn on the Solr Query stage and turn all other stages off.
Now the search results look much like they did before. At this point, the disabled stages do not affect the output because they are not yet configured.
3. Turn all stages on.
## Configure faceting
Facets are the ubiquitous, dynamic lists of categories or features offered as filters within a search results page. Facets provide a simple way for users to explore and filter their search results without having to construct complicated queries. You configure facets as a part of configuring a query pipeline.
The data you indexed in Part 1 has two fields that are natural choices for faceting: `genres_ss` and `year_i`. For example, a user could search for science fiction of the 1950s in just a few clicks.
### Configure basic faceting for genres
The `genres_ss` field is ready for faceting as-is.
1. Click **Add a field facet** and select the `genres_ss` field.
2. Click **Sci-Fi** to select movies that have the value `Sci-Fi` for `genres_ss`:
Genres are not in any specific order. In configuration for the Field Facet stage, you can choose a value of Sort for the facet field - `index` (alphabetical ascending order) or `count` (number of documents). Or you can add field facets by configuring the Field Facet stage.
3. Under the field facet **genres\_ss**, click **Clear all**.
### Configure range faceting for years
If you were to just configure faceting for the `year_i` field as you did above for the `genres_ss` field, you would get one facet per year, which is not very useful.
But the `year_i` field will be more usable if you configure *range faceting*. Range faceting is a way of grouping values together so that the user can select a value range instead of one specific value. For example, range facets are commonly used with pricing ($50-$100) or ratings (4 stars or higher). Here, we group years by decade.
Range faceting requires sending an additional query parameter to Fusion’s Solr core. You can configure this with the [Additional Query Parameters stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/set-query-params-stage). In this case, you will use several of [Solr’s range facet query parameters](https://cwiki.apache.org/confluence/display/solr/Faceting#Faceting-RangeFaceting).
Use the Additional Query Parameters stage to configure range faceting for the `year_i` field:
1. Click **Add a stage**.
2. Scroll down under Advanced and select **Additional Query Parameters**.
The Additional Query Parameters configuration panel appears.
3. Under **Parameters and Values**, add the following parameter names and values:
| | |
| - | - |
| Parameter Name | Parameter Value |
| `facet.range` | `year_i` |
| `facet.range.start` | `1900` |
| `facet.range.end` | `2020` |
| `facet.range.gap` | `10` |
| `facet.range.include` | `outer` |
In this case, you do not need to modify the **Update Policy** field; the default value of `append` is fine.
4. Click **Apply**, and then **Cancel** (which just closes the configuration panel for the Additional Query Parameters stage).
The year facets are now grouped by decade:
Facets are not the only way for users to find items by year. In your user application, you can let users search for specific values in the `year_i` field, for example, by using a text field or dropdown list.
## Configure query fields
In this section, you will see why it is useful to specify which fields Fusion should use to match a query.
1. Search for "2001".
The results are not what an end user might expect:
"2001: A Space Odyssey" is not the top search result.
2. Under one of the movies listed, click **show fields**.
Here is the reason: your search query matches the `id` field, but users do not care about this field.
You will use the [Query Fields](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/search-fields-query-stage) stage to specify the fields that users really care about.
3. Below the name for the movie you selected, click **hide fields**.
4. Click the **Query Fields** stage of the query pipeline.
The Query Fields configuration panel appears.
5. Under **Search Fields**, click **Add** .
6. Enter `title_txt`.
7. Click **Add** again.
8. Enter `year_i`.
9. Click **Apply**, and then **Cancel** (which just closes the configuration panel for the Query Fields stage).
Now movies with '2001' in their title rise to the top of your search results, followed by films made in the year 2001:
## Save the query pipeline configuration
1. In the upper right, click **Save**.
The **Save Pipeline** window appears. By default, you will overwrite the default query pipeline for this datasource.
2. Click **Save pipeline**.
## What is next
With just two facet fields combined with keyword search, this prototype is already beginning to feel like a real search application.
In Part 4, you will enable signals, generate some signal data, aggregate it, and search it to see what it looks like. Signals can be used for recommendations or boosting.
Signals are events that can be aggregated and used for automatic boosting or recommendations, which are ways of making search results more relevant.
As an example of boosting, the most popular search results for certain queries can be boosted so that they appear first (or at least nearer the top) when other users make similar queries.
Similarly for recommendations, click events or purchase events can be collected as signals and used to display "Customers who viewed this also viewed" or "Best-selling holiday items".
In a production environment, users' actions generate signals. For the purposes of this tutorial, you will use Query Workbench to generate click signals.
The learning path for **Refining Search Results** focuses on the Fusion features that help you maximize the relevancy of your search results.
## Before you begin
To proceed with this part of the tutorial, you must first complete Part 1, Part 2, and Part 3, which give you an indexed dataset that is configured for faceted search.
If the Fusion UI is not already open, then open it.
1. In a browser window, open `localhost:8764`.
2. Enter the password for the user `admin`, and then click **Login**.
The Fusion launcher appears.
3. Click the **Movie Search** app.
The Fusion workspace appears.
## Format display fields
To help you understand the continuity in the next steps, first make sure that some relevant fields are displayed.
1. Open Query Workbench.
Navigate to **Querying** > **Query Workbench**.
2. At the top right of the page, select **Display Fields**.
3. In the **Name** field, click the row to reveal a dropdown of possible values.
4. Select `title_txt`. You can filter the list of possible values to help you.
5. In the **Description** field, select `id`.
6. Close the Display Fields window.
1. Click **Display Fields** to close the Display Fields window.
## Enable synthetic signals
You need some signal data beyond the few signals you generated while completing Part 3. Because this is a prototype app, users are not generating signals. Instead, you will enable synthetic signals in Query Workbench.
1. At the bottom of the Query Workbench page, click **Format Results**.
2. Select **Show signal generators** *and* **Send click signals**.
3. Click **Save**.
4. Hover over one of the search results.
Now when you hover over a search result, Query Workbench displays controls that include a **Simulate** button next to a field that lets you specify the number of signals to simulate:
## Generate signals
With synthetic signals enabled, you will generate a simple set of signal data that you can use to generate meaningful recommendations.
For this tutorial, you will generate signals that you can use to boost your favorite sci-fi titles so that they appear first.
1. Search for `star wars`.
The top results are not your favorite titles:
2. In **Format Results**, **Set results per page** to 20. This should allow you to scroll for this next part instead of explicitly searching for the titles.
Next you will generate signals that you can use to boost certain titles. Signals are tied to the search query, so your boosted titles will appear first in the search results only when users search for `star wars`.
3. Hover over "Star Wars: Episode IV - A New Hope".
4. Set the number of signals to 4000 and click **Simulate**.
5. Hover over "Star Wars: Episode V - The Empire Strikes Back".
6. Set the number of signals to 3000 and click **Simulate**.
7. Hover over "Star Wars: Episode VI - Return of the Jedi".
8. Set the number of signals to 2000 and click **Simulate**.
With synthetic signals enabled, you can also send a single signal by clicking the underlined movie title.
9. In the upper right, click **Save**.
The **Save Pipeline** window appears. By default, you will overwrite the existing query pipeline for the selected collection (in this case, `Movie_Search_signals`).
10. Click **Save pipeline**.
## Explore the raw signals
Whenever you create a collection, two corresponding collections are also created automatically: `COLLECTION_NAME_signals` for raw signals and `COLLECTION_NAME_signals_aggr` for aggregated signals (in this case, `Movie_Search_signals` and `Movie_Search_signals_aggr`). Just as you did with your primary collection, you can use Query Workbench to explore the data in the `_signals` collection.
1. In the collection picker in the upper left, select **Movie\_Search\_signals**.
2. Open Query Workbench. Navigate to **Querying** > **Query Workbench**.
Your signal data appears.
Your signals collection contains several types of signals, such as click signals and search result signals. Let us investigate the click signals.
3. Search for `type:click`.
4. For any of the results, click **show fields**.
* The `count_i` field shows the number of click signals you generated for this event. For example, given the corresponding `doc_id` for Star Wars: Episode IV - A New Hope, the `count_i` equals 4000.
In Fusion 4.0:
\*\*\* The contents of the `doc_id_s` field in the `Movie_Search_signals` collection is the same as the contents of the `id` field in your `Movies_Search` collection, that is, the ID of the document that you clicked in Query Workbench, or for which you specified a number of clicks, and then clicked *Simulate*\*.
\*\* The `query_orig_s` field in the `Movie_Search_signals` collection contains the original query string that produced this search result.
5. Click **hide fields**.
### Optional: Format signals fields
You can configure your search view to display fields that are meaningful for your investigation.
For example, you can display the document ID and the number of click signals.
1. Click **Display Fields**.
2. For the **Name** field, select `doc_id`.
3. For the **Description** field, select `count_i`.
## Explore the aggregated signal data
Aggregation jobs are created automatically when you create an app. However, you need to run the aggregation job manually:
1. Navigate to Collections > **Jobs**.
2. Select `Movie_Search_click_signals_aggregation` from the job list.
3. Click **Run**.
4. Click **Start**.
After you run your job (it might take a minute or two), open the Query Workbench and check whether the aggregated data has arrived in the `Movie_Search_signals_aggr` collection.
1. In the collections picker in the upper left, select **Movie\_Search\_signals\_aggr**.
2. Open Query Workbench. Navigate to **Querying** > **Query Workbench**.
Your aggregated signal data should appear. If not, wait a minute and then reload your browser, or click **Search** in Query Workbench.
3. Click **Display Fields**.
4. For the **Name** field, select `doc_id`.
5. For the **Description** field, select `aggr_count_i`.
6. For the result 1210, click **show fields**.
The fields for aggregated signals are very similar to the fields for raw signals, with additional fields to describe the aggregation:
* `aggr_count_i`. Number of signals that have been aggregated (in this case, 3000)
* `aggr_id_s`. Name of the aggregation job
* `aggr_job_id_s`. Job ID
* `aggr_type_s`. Aggregation type
7. In the upper right, click **Save**.
The **Save Pipeline** window appears. By default, you will overwrite the existing query pipeline for the selected collection (in this case, `Movie_Search`).
8. Click **Save pipeline**.
## View the search results with and without default boosting
1. In the collections picker in the upper left, select **Movie\_Search**.
2. Open Query Workbench. Navigate to **Querying** > **Query Workbench**.
3. Search for `star wars`.
Now, "Star Wars: Episode IV - A New Hope" is the first search result, followed by Episode V and then VI. These search results are automatically boosted by the default configuration of the
[Boost with Signals query pipeline stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommendation-boosting-query-stage),
which boosts on the `id` field.
The Boost with Signals stage requires a Fusion AI license. Your Fusion trial license enables Fusion AI features.
4. Click **Compare**. Another preview panel opens. Now the working pipeline is on the right and a static snapshot of that same pipeline is on the left.
In this view, you can compare results from one query pipeline side by side with another query pipeline.
In this case, you will compare results for the same pipeline (`Movie_Search`) with and without the Boost with Signals stage enabled.
5. Turn off the Boost with Signals stage.
Now the search results on the right appear as they did before you generated synthetic click signals. Rank-change indicators indicate which results moved up or down as a result of turning off boosting.
6. Turn on the Boost with Signals stage again to restore the boosted results.
7. Close the comparison preview panel by clicking the close icon.
Interested in using Fusion 5 with Kubernetes? See [Kubernetes concepts](/docs/5/fusion/intro/kubernetes) and **Deploy Fusion 5 on Azure Kubernetes Service (AKS)**. We also have guides for deploying Fusion on **Deploy Fusion 5 on Amazon Elastic Kubernetes Service (EKS)** and **Deploy Fusion 5 on Google Kubernetes Engine (GKE)**.
Fusion supports deployment on Azure Kubernetes Service (AKS). This topic explains how to deploy a Fusion cluster on AKS using the [`setup_f5_aks.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_aks.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository.
The `setup_f5_aks.sh` script is the basic foundation for getting started and proof-of-concept purposes. For information about custom values in a production-ready environment, see Custom values YAML file.
## Prerequisites
This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes.
### Release Name and Namespace
Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) namespace to install Fusion into.
Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*.
However, please *do not* install more than one Fusion release in the same namespace.
All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_
Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default.
### Install Helm
Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster.
Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform.
On MacOS, you can do:
```bash theme={"dark"}
brew install kubernetes-helm
```
If you already have helm installed, make sure you're using the latest version:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/)
The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`.
#### Helm User Permissions
If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory.
When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.:
```bash theme={"dark"}
alias k=kubectl
```
To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into:
```bash theme={"dark"}
k create namespace fusion-namespace
```
Apply the `role.yaml` and `cluster-role.yaml` files to that namespace
```bash theme={"dark"}
k apply -f cluster-role.yaml
k config set-context --current --namespace=$NAMESPACE
k apply -f role.yaml
```
Then bind the rolebinding and clusterolebinding to the install user:
```bash wrap theme={"dark"}
k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user
k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user
```
You will then be able to run the `helm install` command as the ``
### Clone fusion-cloud-native from GitHub
You should clone this repo from github as you'll need to run the scripts on your local workstation:
```bash theme={"dark"}
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts.
```bash theme={"dark"}
cd fusion-cloud-native
git pull
```
Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip).
Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory.
The [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_aks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_aks.sh) script provided in this repo is strictly optional.
The script is mainly to help those new to Kubernetes and/or Fusion get started quickly.
If you're already familiar with K8s, Helm, and AKS, then you use Helm directly to install Fusion into an existing cluster or one you create yourself using the process described `<>`.
If you're new to Azure, then please visit [https://azure.microsoft.com/en-us/free/search/](https://azure.microsoft.com/en-us/free/search/) to set up an account.
### Set up the AKS CLI tools
Before launching an AKS cluster, you need to install and configure `kubectl` and `az` using the links provided below:
Required AKS Command-line Tools:
* `kubectl`: [https://kubernetes.io/docs/tasks/tools/install-kubectl/](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
* `az`: [https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest)
To confirm your account access and command-line tools are set up correctly, run the `az login` command (`az login –help` to see available options).
### Azure Prerequisites
To launch a cluster in AKS (or pretty much do anything with Azure) you need to setup a Resource Group. Resource Groups are a way of organizing and managing related resources in Azure.
For more information about resource groups, see [https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview#resource-groups](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview#resource-groups).
You also need to choose a location where you want to spin up your AKS cluster, such as `westus2`. For a list of locations you can choose, see [https://azure.microsoft.com/en-us/global-infrastructure/locations/](https://azure.microsoft.com/en-us/global-infrastructure/locations/).
Use the Azure console in your browser to create a resource group, or simply do:
```bash theme={"dark"}
az group create -g $AZURE_RESOURCE_GROUP -l $AZURE_LOCATION
```
To recap, you should have the following requirements in place:
* Azure Account set up.
* `azure-cli` (`az`) command-line tools installed.
* `az` login working.
* Created an Azure Resource Group and selected a location to launch the cluster.
### Set up Fusion on AKS
Download and run the [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_aks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_aks.sh) to install Fusion 5.x in a AKS cluster. To create a new cluster and install Fusion, simply do:
```bash theme={"dark"}
./setup_f5_aks.sh -c -p
```
If you don't want the script to create a cluster, then you need to create a cluster before running the script and simply pass the name of the existing cluster using the `-c` parameter.
Use the `--help` option to see full script usage.
By default, our script installs Fusion into the default namespace; think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster in separate namespaces. However, please do not install more than one Fusion release in the same namespace.
You can override the namespace using the `-n` option. In addition, our script uses f5 for the Helm release name; you can customize this using the `-r` option. Helm uses the release name you provide to track a specific instance of an installation, allowing you to perform updates and rollback changes for that specific release only.
You can also pass the `--preview` option to the script, which enables soon-to-be-released features for AKS, such as deploying a multi-zone cluster across 3 availability zones for higher availability guarantees. For more information about the Availability Zone feature, see [https://docs.microsoft.com/en-us/azure/aks/availability-zones](https://docs.microsoft.com/en-us/azure/aks/availability-zones).
It takes a while for AKS to spin up the new cluster. The cluster will have three Standard\_D4\_v3 nodes which have 4 CPU cores and 16 GB of memory. Behind the scenes, our script calls the `az aks create` command.
If using Helm V2, the `setup_f5_aks.sh` script installs Helm's `tiller` component into your AKS cluster with the cluster admin role. If you don't want this, then please upgrade to Helm v3.
After running the `setup_f5_aks.sh` script, proceed to `<>`.
#### AKS Ingress
The `setup_f5_aks.sh` script exposes the Fusion proxy service on an external IP over HTTP. This is done for demo or getting started purposes. However, you're strongly encouraged to configure a K8s Ingress with TLS termination in front of the proxy service.
Use the `-t` and `-h ` options to have our script create an Ingress with a TLS certificate issued by Let's Encrypt.
#### Upgrades and Ingress
IMPORTANT: If you used the `-t -h ` options when installing your cluster, our script created an additional values yaml file named `tls-values.yaml`.
To make things easier for you when upgrading, you should add the settings from this file into your main custom values yaml file. For example:
```yaml theme={"dark"}
api-gateway:
service:
type: "NodePort"
ingress:
enabled: true
host: ""
tls:
enabled: true
annotations:
"networking.gke.io/managed-certificates": "-managed-certificate"
"kubernetes.io/ingress.class": "gce"
```
This way, you don't have to remember to pass the additional `tls-values.yaml` file when upgrading.
## Verifying the Fusion Installation
In this section, we provide some tips on how to verify the Fusion installation.
Check if the Fusion Admin UI is available at `\https://:6764/admin/`.
Let's review some useful kubectl commands.
### Enhance the K8s Command-line Experience
Here is a list of tools we found useful for improving your command-line experience with Kubernetes:
* krew (kubectl plugin mgr): [https://github.com/kubernetes-sigs/krew/](https://github.com/kubernetes-sigs/krew/)
* kube-ps1 (show current context on command line prompt): [https://github.com/jonmosco/kube-ps1](https://github.com/jonmosco/kube-ps1)
* kubectx / kubens (switch between clusters / namespaces): [https://github.com/ahmetb/kubectx](https://github.com/ahmetb/kubectx)
### Useful kubectl commands
kubectl reference: [https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands)
Set the namespace for `kubectl` if not using the default:
```bash theme={"dark"}
kubectl config set-context --current --namespace=
```
**This saves you from having to pass `-n` with every command.**
Get a list of running pods: `k get pods`
Get logs for a pod using a label: `k logs –l app.kubernetes.io/component=query-pipeline`
Get pod deployment spec and details: `k get pods -o yaml`
Get details about a pod events: `k describe po `
Port forward to a specific pod: `k port-forward 8983:8983`
SSH into a pod: `k exec -it -- /bin/bash`
CPU/Memory usage report for pods: `k top pods`
Forcefully kill a pod: `k delete po --force --grace-period 0`
Scale up (or down) a deployment: `k scale deployment.v1.apps/ --replicas=N`
Get a list of pod versions: `k get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq`
### Check Fusion Pods and Services
Once the install script completes, you can check that all pods and services are available using:
```bash theme={"dark"}
kubectl get pods
```
If all goes well, you should see a list of pods similar to:
```bash wrap theme={"dark"}
NAME READY STATUS RESTARTS AGE
seldon-controller-manager-6675874894-qxwrv 1/1 Running 0 8m45s
f5-admin-ui-74d794f4f8-m5jms 1/1 Running 0 8m45s
f5-ambassador-fd6b9b5dc-7ghf6 1/1 Running 0 8m43s
f5-api-gateway-6b9998b9c-tmchk 1/1 Running 0 8m45s
f5-auth-ui-7565564b4c-rdc74 1/1 Running 0 8m42s
f5-classic-rest-service-0 1/1 Running 3 8m44s
f5-devops-ui-77bb867ffb-fbzxd 1/1 Running 0 8m42s
f5-fusion-admin-78b8f8fc7f-4d7l8 1/1 Running 0 8m42s
f5-fusion-indexing-599c8d448-xzsvm 1/1 Running 0 8m44s
f5-insights-665fd9f6fc-g5psw 1/1 Running 0 8m43s
f5-job-launcher-84dd4c5c96-p8528 1/1 Running 0 8m44s
f5-job-rest-server-6d44d964b8-xtnxw 1/1 Running 0 8m45s
f5-logstash-0 1/1 Running 0 8m45s
f5-ml-model-service-6987dc94c9-9ppp8 2/2 Running 1 8m45s
f5-monitoring-grafana-5d499dbb58-pzw72 1/1 Running 0 10m
f5-monitoring-prometheus-kube-state-metrics-54d6678dv9h7h 1/1 Running 0 10m
f5-monitoring-prometheus-pushgateway-7d65c65b85-vwrwf 1/1 Running 0 10m
f5-monitoring-prometheus-server-0 2/2 Running 0 10m
f5-pm-ui-86cbc5bb65-nd2n8 1/1 Running 0 8m44s
f5-pulsar-bookkeeper-0 1/1 Running 0 8m45s
f5-pulsar-broker-b56cc776f-56msx 1/1 Running 0 8m45s
f5-query-pipeline-5d75d7d5f4-l2mdf 1/1 Running 0 8m43s
f5-connectors-7bb6cfc65f-7wfs2 1/1 Running 0 8m42s
f5-connectors-backend-987fdc648-dldwv 1/1 Running 0 8m45s
f5-rules-ui-6b9d55b78f-9hzzj 1/1 Running 0 8m43s
f5-solr-0 1/1 Running 0 8m44s
f5-solr-exporter-c4687c785-jsm7x 1/1 Running 0 8m45s
f5-ui-6cdbcc68c6-rj9cq 1/1 Running 0 8m45s
f5-webapps-6d6bb9bfd-hm4qx 1/1 Running 0 8m45s
f5-workflow-controller-7b66679fb7-sjbvp 1/1 Running 0 8m44s
f5-zookeeper-0 1/1 Running 0 8m45s
```
The number of pods per deployment / statefulset will vary based on your cluster size and replicaCount settings in your custom values YAML file.
Also, don't worry if you see some pods having been restarted as that just means they were too slow to come up and Kubernetes killed and restarted them.
You do want to see at least one pod running for every service. If a pod is not running after waiting a sufficient amount of time,
use `kubectl logs ` to see the logs for that pod; to see the logs for previous versions of a pod, use: `kubectl logs -p`.
You can also look at the actions Kubernetes performed on the pod using `kubectl describe po `.
To see a list of Fusion services, do:
```bash theme={"dark"}
kubectl get svc
```
For an overview of the various Fusion 5 microservices, see: [Fusion microservices](/docs/5/fusion/reference/microservices).
Once you're ready to build a Fusion cluster for production, please see see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
### Upgrading with Zero Downtime
One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. Fusion 5 allows customers to upgrade from Fusion 5.x.y to a later 5.x.z version on a live cluster with zero downtime or disruption of service.
When Kubernetes performs a rolling update to an individual microservice, there will be a mix of old and new services in the cluster concurrently (only briefly in most cases) and requests from other services will be routed to both versions. Consequently, Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same 5.x line of releases. We also ensure stored configuration remains compatible in the same 5.x release line.
Lucidworks releases minor updates to individual services frequently, so our customers can pull in those upgrades using Helm at their discretion.
To upgrade your cluster at any time, use the `--upgrade` option with our setup scripts in this repo.
The scripts in this repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.
To see what would be upgraded, you can pass the `--dry-run` option to the script.
### Grafana Dashboards
Get the initial Grafana password from a K8s secret by doing:
```bash wrap theme={"dark"}
kubectl get secret --namespace "${NAMESPACE}" ${RELEASE}-monitoring-grafana \
-o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```
With Grafana, you can either setup a temporary port-forward to a Grafana pod or expose Grafana on an external IP using a K8s LoadBalancer.
To define a LoadBalancer, do (replace \$ with your Helm release label):
```bash wrap theme={"dark"}
kubectl expose deployment ${RELEASE}-monitoring-grafana --type=LoadBalancer --name=grafana --port=3000 --target-port=3000
```
You can use `kubectl get services --namespace ` to determine when the load balancer is setup and its IP address. Direct your browser to `http://:3000` and enter the username `admin@localhost` and the password that was returned in the previous step.
This will log you into the application. It is recommended that you create another administrative user with a more desirable password.
The dashboards and datasoure will be setup for you in grafana, simply navigate to `Dashboards` -> `Manage` to view the available dashboards
## More deployment options
* How to deploy Fusion 5 in Google Kubernetes Engine
* How to deploy Fusion 5 in Amazon Elastic Kubernetes Service
* How to deploy Fusion on 5 other Kubernetes platforms
## Frequently Asked Questions
**Can the stateful database services, for example, MySQL, be supported by an Azure PaaS service?**
This option is not supported. In theory, it may be possible to implement this function.
**Is it possible to use cross-zone storage solutions rather than volumes, such as Microsoft Azure file storage, for stateful services?**
While in theory it may be possible to implement this functionality, the configuration has not been tested by Lucidworks and is not supported.
## Learn more
The course for **Deploying Fusion 5** focuses on the prerequisite software needed to deploy Fusion, the necessary setup steps, and the physical act of deployment.
Fusion supports deployment on Amazon Elastic Kubernetes Service (EKS). This topic explains how to deploy a Fusion cluster on EKS using the [`setup_f5_eks.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_eks.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository.
In addition, this topic provides information about how to [configure IAM roles for the service account](#configure-iam-roles-for-the-service-account).
## Prerequisites
This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes.
### Release Name and Namespace
Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) Kubernetes namespace to install Fusion into.
Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*.
However, please *do not* install more than one Fusion release in the same namespace.
All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_
Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default.
### Install Helm
Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster.
Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform.
On MacOS, you can do:
```bash theme={"dark"}
brew install kubernetes-helm
```
If you already have helm installed, make sure you're using the latest version:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/)
The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`.
#### Helm User Permissions
If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory.
When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.:
```bash theme={"dark"}
alias k=kubectl
```
To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into:
```bash theme={"dark"}
k create namespace fusion-namespace
```
Apply the `role.yaml` and `cluster-role.yaml` files to that namespace
```bash theme={"dark"}
k apply -f cluster-role.yaml
k config set-context --current --namespace=$NAMESPACE
k apply -f role.yaml
```
Then bind the rolebinding and clusterolebinding to the install user:
```bash wrap theme={"dark"}
k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user
k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user
```
You will then be able to run the `helm install` command as the ``
### Clone fusion-cloud-native from GitHub
You should clone this repo from github as you'll need to run the scripts on your local workstation:
```bash theme={"dark"}
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts.
```bash theme={"dark"}
cd fusion-cloud-native
git pull
```
Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip).
Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory.
The [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_eks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_eks.sh) `setup_f5_eks.sh` script provided in this repo is strictly optional.
The script is mainly to help those new to Kubernetes and/or Fusion get started quickly.
If you're already familiar with K8s, Helm, and EKS, then you use Helm directly to install Fusion into an existing cluster or one you create yourself using the process described `<>`.
If you're new to Amazon Web Services (AWS), then please visit the Amazon Web Services [https://aws.amazon.com/getting-started/](https://aws.amazon.com/getting-started/) to set up an account.
If you're new to Kubernetes and EKS, then we recommend going through Amazon's [https://eksworkshop.com/introduction/](https://eksworkshop.com/introduction/) before proceeding with Fusion.
### Set up the AWS CLI tools
Before launching an EKS cluster, you need to install and configure `kubectl`, `aws`, `eksctl`, `aws-iam-authenticator` using the links provided below:
Required AWS Command-line Tools:
* kubectl: [https://kubernetes.io/docs/tasks/tools/install-kubectl/](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
* aws: [https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html)
* eksctl: [https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html)
* aws-iam-authenticator: [https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html](https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html)
Run `aws configure` to configure a profile for authenticating to AWS. You'll use the profile name you configure in this step, which defaults to `default`, as the `-p` argument to the `setup_f5_eks.sh` script in the next section.
When working in Ubuntu, avoid using the eksctl snap version. Alternative sources can have different versions that could cause command failures. Also, always make sure you are using the latest version for each one of the required tools.
### Set up Fusion on EKS
To create a cluster in EKS the following IAM policies are required:
* AmazonEC2FullAccess
* AWSCloudFormationFullAccess
EKS Permissions:
```bash theme={"dark"}
eks:CreateCluster
eks:DeleteCluster
eks:DescribeCluster
eks:DescribeUpdate
eks:ListClusters
eks:ListUpdates
eks:UpdateClusterVersion
```
VPC Permissions:
```bash theme={"dark"}
ec2:AssociateSubnetCidrBlock
ec2:AssociateVpcCidrBlock
ec2:AttachInternetGateway
ec2:CreateInternetGateway
ec2:CreateSubnet
ec2:CreateVpc
ec2:CreateVpcEndpoint
ec2:DeleteInternetGateway
ec2:DeleteSubnet
ec2:DeleteVpc
ec2:DeleteVpcEndpoints
ec2:DescribeSubnets
ec2:DescribeVpcAttribute
ec2:DescribeVpcs
ec2:DetachInternetGateway
ec2:DisassociateSubnetCidrBlock
ec2:DisassociateVpcCidrBlock
ec2:ModifySubnetAttribute
ec2:ModifyVpcAttribute
ec2:ModifyVpcEndpoint
```
.IAM Permissions
```bash theme={"dark"}
iam:AddRoleToInstanceProfile
iam:AttachRolePolicy
iam:CreateInstanceProfile
iam:CreatePolicy
iam:CreatePolicyVersion
iam:CreateRole
iam:DeleteInstanceProfile
iam:DeletePolicy
iam:DeletePolicyVersion
iam:DeleteRole
iam:DeleteRolePolicy
iam:DetachRolePolicy
iam:GetInstanceProfile
iam:GetPolicy
iam:GetPolicyVersion
iam:GetRole
iam:GetRolePolicy
iam:ListInstanceProfiles
iam:ListInstanceProfilesForRole
iam:PassRole
iam:PutRolePolicy
iam:RemoveRoleFromInstanceProfile
iam:TagRole
iam:UntagRole
```
Download and run the [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_eks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_eks.sh) `setup_f5_eks.sh` script to install Fusion 5.x in an EKS cluster.
This script does not support multiple node pools and should not be used for production clusters.
* To create a new cluster and install Fusion, run the following command:
```
./setup_f5_eks.sh -c my-eks-cluster -p profile-name -n fusion-namespace --create demo
```
* Replace `my-eks-cluster`, `profile-name`, and `fusion-namespace` with your cluster, profile, and namespace values.
* Pass the `--create` option with either `demo` or `multi_az`.
* To use an existing cluster and install Fusion, run the following command:
```
./setup_f5_eks.sh -c cluster-name -p profile-name
```
* Replace `cluster-name` with the name of the cluster you already created.
* Replace `profile-name` with the name of your profile.
The profile is automatically set to `default` if you ran the AWS configure command without giving the profile a name.
Use the `--help` option to see full script usage.
If using Helm V2, the `setup_f5_eks.sh` script installs Helm's `tiller` component into your EKS cluster with the cluster admin role. If you don't want this, then please upgrade to Helm v3.
The `setup_f5_eks.sh` script creates a service account that provides S3 read-only permissions to the created pods.
After running the `setup_f5_eks.sh` script, proceed to the `<>` section below.
#### EKS cluster overview
The EKS cluster is created using `eksctl` ([https://eksctl.io/](https://eksctl.io/)). By default it will setup the following resources in your AWS account:
* A dedicated VPC for the EKS cluster in the specified region with CIDR: `192.168.0.0/16`
* 3 Public and 3 Private subnets within the created VPC, each with a `/19` CIDR range, along with the corresponding route tables.
* A NAT gateway in each Public subnet
* An Auto Scaling Group of the instance type specified by the script, which defaults to `m5.2xlarge`, with 3 instances spanning the public subnets.
See [https://eksctl.io/usage/vpc-networking/](https://eksctl.io/usage/vpc-networking/) for more information on the networking setup.
#### EKS Ingress
The `setup_f5_eks.sh` script exposes the Fusion proxy service on an external DNS name provided by an ELB over HTTP. This is done for demo or getting started purposes. However, you're strongly encouraged to configure a K8s Ingress with TLS termination in front of the proxy service.
See: [https://aws.amazon.com/premiumsupport/knowledge-center/terminate-https-traffic-eks-acm/](https://aws.amazon.com/premiumsupport/knowledge-center/terminate-https-traffic-eks-acm/)
Our EKS script creates a classic ELB for exposing fusion proxy service. In case you need to change this behavior and use [https://github.com/kubernetes-sigs/aws-load-balancer-controller](https://github.com/kubernetes-sigs/aws-load-balancer-controller) instead you can use the following parameters when running the `setup_f5_eks.sh` script:
```bash theme={"dark"}
--deploy-alb # Tells the script to deploy an ALB
```
By default the `kube-system` namespace is being used for installing the `aws-load-balancer-controller` because pods `priorityClassName` is set to `system-cluster-critical`.
In case you need to deploy an internal ALB you can use the `--internal-alb` option. This will create the nodes in the internal subnets. Fusion will be reachable from an AWS instance located in any of the external subnets on the same VPC. To use an ALB also an ingress with a DNS name is required, you can use the `-h` option to create an ingress with the required DNS name.
Finally, use Route 53 or your DNS provider for creating an A ALIAS DNS record for your DNS name pointing to the ingress ADRESS. You can get the address listing the ingress using the command `kubectl get ing`.
## Verifying the Fusion Installation
In this section, we provide some tips on how to verify the Fusion installation.
Check if the Fusion Admin UI is available at `\https://:6764/admin/`.
Let's review some useful kubectl commands.
### Enhance the K8s Command-line Experience
Here is a list of tools we found useful for improving your command-line experience with Kubernetes:
* krew (kubectl plugin mgr): [https://github.com/kubernetes-sigs/krew/](https://github.com/kubernetes-sigs/krew/)
* kube-ps1 (show current context on command line prompt): [https://github.com/jonmosco/kube-ps1](https://github.com/jonmosco/kube-ps1)
* kubectx / kubens (switch between clusters / namespaces): [https://github.com/ahmetb/kubectx](https://github.com/ahmetb/kubectx)
### Useful kubectl commands
kubectl reference: [https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands)
Set the namespace for `kubectl` if not using the default:
```bash theme={"dark"}
kubectl config set-context --current --namespace=
```
**This saves you from having to pass `-n` with every command.**
Get a list of running pods: `k get pods`
Get logs for a pod using a label: `k logs –l app.kubernetes.io/component=query-pipeline`
Get pod deployment spec and details: `k get pods -o yaml`
Get details about a pod events: `k describe po `
Port forward to a specific pod: `k port-forward 8983:8983`
SSH into a pod: `k exec -it -- /bin/bash`
CPU/Memory usage report for pods: `k top pods`
Forcefully kill a pod: `k delete po --force --grace-period 0`
Scale up (or down) a deployment: `k scale deployment.v1.apps/ --replicas=N`
Get a list of pod versions: `k get po -o jsonpath='{..image}'
tr -s '[[:space:]]' '\n'
sort
uniq`
### Check Fusion Pods and Services
Once the install script completes, you can check that all pods and services are available using:
```bash theme={"dark"}
kubectl get pods
```
If all goes well, you should see a list of pods similar to:
```bash wrap theme={"dark"}
NAME READY STATUS RESTARTS AGE
seldon-controller-manager-6675874894-qxwrv 1/1 Running 0 8m45s
f5-admin-ui-74d794f4f8-m5jms 1/1 Running 0 8m45s
f5-ambassador-fd6b9b5dc-7ghf6 1/1 Running 0 8m43s
f5-api-gateway-6b9998b9c-tmchk 1/1 Running 0 8m45s
f5-auth-ui-7565564b4c-rdc74 1/1 Running 0 8m42s
f5-classic-rest-service-0 1/1 Running 3 8m44s
f5-devops-ui-77bb867ffb-fbzxd 1/1 Running 0 8m42s
f5-fusion-admin-78b8f8fc7f-4d7l8 1/1 Running 0 8m42s
f5-fusion-indexing-599c8d448-xzsvm 1/1 Running 0 8m44s
f5-insights-665fd9f6fc-g5psw 1/1 Running 0 8m43s
f5-job-launcher-84dd4c5c96-p8528 1/1 Running 0 8m44s
f5-job-rest-server-6d44d964b8-xtnxw 1/1 Running 0 8m45s
f5-logstash-0 1/1 Running 0 8m45s
f5-ml-model-service-6987dc94c9-9ppp8 2/2 Running 1 8m45s
f5-monitoring-grafana-5d499dbb58-pzw72 1/1 Running 0 10m
f5-monitoring-prometheus-kube-state-metrics-54d6678dv9h7h 1/1 Running 0 10m
f5-monitoring-prometheus-pushgateway-7d65c65b85-vwrwf 1/1 Running 0 10m
f5-monitoring-prometheus-server-0 2/2 Running 0 10m
f5-pm-ui-86cbc5bb65-nd2n8 1/1 Running 0 8m44s
f5-pulsar-bookkeeper-0 1/1 Running 0 8m45s
f5-pulsar-broker-b56cc776f-56msx 1/1 Running 0 8m45s
f5-query-pipeline-5d75d7d5f4-l2mdf 1/1 Running 0 8m43s
f5-connectors-7bb6cfc65f-7wfs2 1/1 Running 0 8m42s
f5-connectors-backend-987fdc648-dldwv 1/1 Running 0 8m45s
f5-rules-ui-6b9d55b78f-9hzzj 1/1 Running 0 8m43s
f5-solr-0 1/1 Running 0 8m44s
f5-solr-exporter-c4687c785-jsm7x 1/1 Running 0 8m45s
f5-ui-6cdbcc68c6-rj9cq 1/1 Running 0 8m45s
f5-webapps-6d6bb9bfd-hm4qx 1/1 Running 0 8m45s
f5-workflow-controller-7b66679fb7-sjbvp 1/1 Running 0 8m44s
f5-zookeeper-0 1/1 Running 0 8m45s
```
The number of pods per deployment / statefulset will vary based on your cluster size and replicaCount settings in your custom values YAML file.
Also, don't worry if you see some pods having been restarted as that just means they were too slow to come up and Kubernetes killed and restarted them.
You do want to see at least one pod running for every service. If a pod is not running after waiting a sufficient amount of time,
use `kubectl logs ` to see the logs for that pod; to see the logs for previous versions of a pod, use: `kubectl logs -p`.
You can also look at the actions Kubernetes performed on the pod using `kubectl describe po `.
To see a list of Fusion services, do:
```bash theme={"dark"}
kubectl get svc
```
For an overview of the various Fusion 5 microservices, see: [Fusion microservices](/docs/5/fusion/reference/microservices).
Once you're ready to build a Fusion cluster for production, please see see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
### Upgrading with Zero Downtime
One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. Fusion 5 allows customers to upgrade from Fusion 5.x.y to a later 5.x.z version on a live cluster with zero downtime or disruption of service.
When Kubernetes performs a rolling update to an individual microservice, there will be a mix of old and new services in the cluster concurrently (only briefly in most cases) and requests from other services will be routed to both versions. Consequently, Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same 5.x line of releases. We also ensure stored configuration remains compatible in the same 5.x release line.
Lucidworks releases minor updates to individual services frequently, so our customers can pull in those upgrades using Helm at their discretion.
To upgrade your cluster at any time, use the `--upgrade` option with our setup scripts in this repo.
The scripts in this repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.
To see what would be upgraded, you can pass the `--dry-run` option to the script.
### Grafana Dashboards
Get the initial Grafana password from a K8s secret by doing:
```bash wrap theme={"dark"}
kubectl get secret --namespace "${NAMESPACE}" ${RELEASE}-monitoring-grafana \
-o jsonpath="{.data.admin-password}"
base64 --decode ; echo
```
With Grafana, you can either setup a temporary port-forward to a Grafana pod or expose Grafana on an external IP using a K8s LoadBalancer.
To define a LoadBalancer, do (replace \$ with your Helm release label):
```bash wrap theme={"dark"}
kubectl expose deployment ${RELEASE}-monitoring-grafana --type=LoadBalancer --name=grafana --port=3000 --target-port=3000
```
You can use `kubectl get services --namespace ` to determine when the load balancer is setup and its IP address. Direct your browser to `http://:3000` and enter the username `admin@localhost` and the password that was returned in the previous step.
This will log you into the application. It is recommended that you create another administrative user with a more desirable password.
The dashboards and datasoure will be setup for you in grafana, simply navigate to `Dashboards` -> `Manage` to view the available dashboards
## Configure IAM roles for the service account
Configuring IAM roles lets you utilize the Amazon Web Services Security Token Service (AWS STS) for short-term authentication credentials to access services like the Amazon S3 simple storage service.
To configure IAM roles, your user account must be granted `admin` permissions or `IAM:FullAccess`. Complete the following steps:
1. To create the OpenID Connect (OIDC) provider, run the following command:
```bash wrap theme={"dark"}
eksctl utils associate-iam-oidc-provider --cluster cluster_name --approve
```
2. To create an IAM role for the service account associated with the plugin-pod, run the following command:
```bash wrap theme={"dark"}
eksctl create iamserviceaccount --name f5-connector-plugin --namespace default --cluster cluster_name --attach-policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess --approve --override-existing-serviceaccounts
```
This command:
* Creates an IAM role and attaches the target policy.
* Updates the existing Kubernetes `f5-connector-plugin` service account and annotates it with the IAM role.
* Uses the existing `policy/AmazonS3ReadOnlyAccess` policy.
If the IAM role was already created without the command, and you want to associate the service account, run the following command:
```bash wrap theme={"dark"}
kubectl annotate serviceaccount -n default f5-connector-plugin eks.amazonaws.com/role-arn=arn:aws:iam::411271863668:role/FUS_ROLE --overwrite=true
```
To utilize this feature, create a data source with the settings in **S3 Authentication Settings > AWS Instance Credentials Authentication Settings**. For detailed installation information, see [AWS S3 Pro connector](/docs/fusion-connectors/connectors/amazon-aws-s3-pro).
For more information, see:
* [Welcome to the AWS Security Token Service API Reference](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html)
* [Temporary security credentials in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html)
* [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html)
* [What is Amazon EKS?](https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html)
* [Fine-grained IAM roles for service accounts](https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts)
## More deployment options
* How to deploy Fusion 5 in Google Kubernetes Engine
* How to deploy Fusion 5 in Azure Kubernetes Service
* How to deploy Fusion 5 on other Kubernetes platforms
## Learn more
The course for **Deploying Fusion 5** focuses on the prerequisite software needed to deploy Fusion, the necessary setup steps, and the physical act of deployment.
Fusion supports deployment on Google Kubernetes Engine (GKE). This topic explains how to deploy a Fusion cluster on GKE using the [`setup_f5_gke.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_gke.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository.
## Prerequisites
This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes.
### Release Name and Namespace
Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) Kubernetes namespace to install Fusion into.
Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*.
However, please *do not* install more than one Fusion release in the same namespace.
All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_
Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default.
### Install Helm
Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster.
Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform.
On MacOS, you can do:
```bash theme={"dark"}
brew install kubernetes-helm
```
If you already have helm installed, make sure you're using the latest version:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/)
The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`.
#### Helm User Permissions
If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory.
When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.:
```bash theme={"dark"}
alias k=kubectl
```
To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into:
```bash theme={"dark"}
k create namespace fusion-namespace
```
Apply the `role.yaml` and `cluster-role.yaml` files to that namespace
```bash theme={"dark"}
k apply -f cluster-role.yaml
k config set-context --current --namespace=$NAMESPACE
k apply -f role.yaml
```
Then bind the rolebinding and clusterolebinding to the install user:
```bash wrap theme={"dark"}
k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user
k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user
```
You will then be able to run the `helm install` command as the ``
### Clone fusion-cloud-native from GitHub
You should clone this repo from github as you'll need to run the scripts on your local workstation:
```bash theme={"dark"}
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts.
```bash theme={"dark"}
cd fusion-cloud-native
git pull
```
Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip).
Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory.
## Prerequisites
This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes.
### Release Name and Namespace
Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) Kubernetes namespace to install Fusion into.
Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*.
However, please *do not* install more than one Fusion release in the same namespace.
All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_
Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default.
### Install Helm
Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster.
Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform.
On MacOS, you can do:
```bash theme={"dark"}
brew install kubernetes-helm
```
If you already have helm installed, make sure you're using the latest version:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/)
The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`.
#### Helm User Permissions
If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory.
When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.:
```bash theme={"dark"}
alias k=kubectl
```
To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into:
```bash theme={"dark"}
k create namespace fusion-namespace
```
Apply the `role.yaml` and `cluster-role.yaml` files to that namespace
```bash theme={"dark"}
k apply -f cluster-role.yaml
k config set-context --current --namespace=$NAMESPACE
k apply -f role.yaml
```
Then bind the rolebinding and clusterolebinding to the install user:
```bash wrap theme={"dark"}
k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user
k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user
```
You will then be able to run the `helm install` command as the ``
### Clone fusion-cloud-native from GitHub
You should clone this repo from github as you'll need to run the scripts on your local workstation:
```bash theme={"dark"}
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts.
```bash theme={"dark"}
cd fusion-cloud-native
git pull
```
Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip).
Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory.
The [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_gke.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_gke.sh) `setup_f5_gke.sh` script provided in this repo is strictly optional.
The script is mainly to help those new to Kubernetes and/or Fusion get started quickly.
If you're already familiar with K8s, Helm, and GKE, then you can skip the script and just use Helm directly to install Fusion into an existing cluster or one you create yourself using the process described `<>`.
### Set up the Google Cloud SDK (one time only)
If you've already installed the `gcloud` command-line tools, you can skip to `<>`.
These steps set up your local Google Cloud SDK environment so that you're ready to use the command-line tools to manage your Fusion deployment.
Usually, you only need to perform these setup steps once. After that, you're ready to create a cluster.
For a nice getting started tutorial for GKE, see: [https://cloud.google.com/kubernetes-engine/docs/deploy-app-cluster](https://cloud.google.com/kubernetes-engine/docs/deploy-app-cluster)
How to set up the Google Cloud SDK:
1. [https://console.cloud.google.com/apis/library/container.googleapis.com?q=kubernetes%20engine](https://console.cloud.google.com/apis/library/container.googleapis.com?q=kubernetes%20engine)
2. Log in to Google Cloud: `gcloud auth login`
3. Set up the Google Cloud SDK:
1. `gcloud config set compute/zone `
If you are working with regional clusters instead of zone clusters, use `gcloud config set compute/region ` instead.
2. `gcloud config set core/account `
3. *New GKE projects only:* `gcloud projects create `
If you have already created a project, for example in [https://console.cloud.google.com/](https://console.cloud.google.com/), then skip to the next step.
4. `gcloud config set project `
Make sure you install the Kubernetes command-line tool `kubectl` using:
```bash theme={"dark"}
gcloud components install kubectl
gcloud components update
```
### Create a single-node demo cluster
Run the [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_gke.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_gke.sh) `setup_f5_gke.sh` script to install Fusion 5.x in a GKE cluster. To create a new, single-node *demo* cluster and install Fusion, simply do:
```bash theme={"dark"}
./setup_f5_gke.sh -c -p --create demo
```
Use the `--help` option to see script usage. If you want the script to create a cluster for you, then you need to pass the `--create` option with either `demo` or `multi_az`. If you don't want the script to create a cluster, then you need to create a cluster before running the script and simply pass the name of the existing cluster using the `-c` parameter.
If you pass `--create demo` to the script, then we create a single node GKE cluster (defaults to using `n1-standard-8` node type). The minimum node type you'll need for a 1 node cluster is an `n1-standard-8` (on GKE) which has 8 CPU and 30 GB of memory. This is cutting it very close in terms of resources as you also need to host all of the Kubernetes system pods on this same node. Obviously, this works for kicking the tires on Fusion 5.1 but is not sufficient for production workloads.
You can change the instance type using the `-i` parameter; see: [https://cloud.google.com/compute/docs/regions-zones/#available](https://cloud.google.com/compute/docs/regions-zones/#available) for a list of which machine types are available in your desired region.
If not provided the script generates a custom values file named `gke___fusion_values.yaml` which you can use to customize the Fusion chart.\_\_
\#WARNING# If using Helm V2, the `setup_f5_gke.sh` script installs Helm's `tiller` component into your GKE cluster with the cluster admin role. If you don't want this, then please upgrade to Helm v3.
If you see an error similar to the following, then wait a few seconds and try running the `setup_f5_gke.sh` script again with the same arguments as this is usually a transient issue:
```bash wrap theme={"dark"}
Error: could not get apiVersions from Kubernetes: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
```
After running the `setup_f5_gke.sh` script, proceed to the `<>` section below.
When you're ready to deploy Fusion to a production-like environment, see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
### Create a three-node regional cluster to withstand a zone outage
With a three-node regional cluster, nodes are deployed across three separate availability zones.
```bash wrap theme={"dark"}
./setup_f5_gke.sh -c -p -n --region --create multi_az
```
* `` value should be the name of a non-existent cluster; the script will create the new cluster.
* `` must match the name of an existing project in GKE. Run `gcloud config get-value project` to get this value.
* `` Kubernetes namespace to install Fusion into, defaults to `default` with release `f5`
* `` value should be the name of a GKE region, defaults to `us-west1`. Run `gcloud config get-value compute/zone` to get this value.
In this configuration, Kubernetes deploys a ZooKeeper and Solr pod on each of the three nodes, which allows the cluster to retain ZK quorum and remain operational after losing one node, such as during an outage in one availability zone.
When running in a multi-zone cluster, each Solr node has the `solr_zone` system property set to the zone it is running in, such as `-Dsolr_zone=us-west1-a`.
After running the `setup_f5_gke.sh` script, proceed to the `<>` section below.
When you're ready to deploy Fusion to a production-like environment, see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
#### GKE Ingress and TLS
The Fusion proxy service provides authentication and serves as an API gateway for accessing all other Fusion services.
It's typical to use an Ingress for TLS termination in front of the proxy service.
The `setup_f5_gke.sh` supports creating an Ingress with a TLS cert for a domain you own by passing: `-t -h `
After the script runs, you need to create an A record in GCP's DNS service to map your domain name to the Ingress IP. Once this occurs, our script setup uses [https://letsencrypt.org/](https://letsencrypt.org/) to issue a TLS cert for your Ingress.
To see the status of the Let's Encrypt issued certificate, do:
```bash theme={"dark"}
kubectl get managedcertificates -n -o yaml
```
Please refer to the Kubernetes documentation on configuring an Ingress for GKE: [https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer](https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer)
The GCP Ingress defaults to a 30 second timeout, which can lead to false negatives for long running requests such as importing apps. To configure the timeout for the backend in kubernetes:
Create a BackendConfig object in your namespace:
```yaml theme={"dark"}
---
apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
name: backend_config_name
spec:
timeoutSec: 120
connectionDraining:
drainingTimeoutSec: 60
```
Then make sure that the following entries are in the right place in your values.yaml file:
```yaml theme={"dark"}
api-gateway:
service:
annotations:
beta.cloud.google.com/backend-config: '{"ports": {"6764":"backend_config_name"}}'
```
and upgrade your release to apply the configuration changes
#### Ingresses and externalTrafficPolicy
When running a fusion cluster behind an externally controlled LoadBalancer it can be advantageous
to configure the `externalTrafficPolicy` of the `proxy` service to `Local`. This preserves the client
source IP and avoids a second hop for LoadBalancer and NodePort type services, but risks potentially
imbalanced traffic spreading. Although when running in a cluster with a dedicated pool for spark jobs
that can scale up and down freely it can prevent unwanted request failures. This behaviour can be
altered with the `api-gateway.service.externalTrafficPolicy` value, which is set to `Local` if the example values
file is used.
**You must use `externalTrafficPolicy`=`Local` for the Trusted HTTP Realm to work correctly.**
If you are already using a custom 'values.yaml' file, create an entry for `externalTrafficPolicy` under `api-gateway` service.
```yaml theme={"dark"}
api-gateway:
service:
externalTrafficPolicy: Local
```
#### Considerations when using the nginx ingress controller
If you are using the `nginx` ingress controller to fulfil your ingress definitions there are a couple
of options that are recommended to be set in the configmap:
```bash wrap theme={"dark"}
enable-underscores-in-headers: "true" # Fusion can return some headers that have underscores, these have to be explicitly enabled in nginx
proxy-body-size: "0" # By default nginx places a maximum size on request bodies, either increase as needed or disable by setting to 0
proxy-read-timeout: "300" # Increases the timeout for potential slow queries.
```
#### Custom values
There are some example values files that can be used as a starting point for
resources, affinity and replica count configuration in the `example-values` folder.
These can be passed to the install script using the `--values` option, for example:
```bash wrap theme={"dark"}
./setup_f5_gke.sh -c -p -r -n \
--values example-values/affinity.yaml --values example-values/resources.yaml --values example-values/replicas.yaml
```
The `--values` option can be passed multiple times, if the same configuration property is contained within multiple `values` files then the values from the latest file passed as a `--values` option are used.
##### Connectors custom values
If you are using Fusion 5.9 or later, you can specify resources and replica count per connector. This allows you to set different resource limits for each connector. If you do not set custom values for a connector, that connector uses the default values.
Set each connector's resource values in the `connector-plugin` section under `pluginValues`. The `pluginValues` section is a list of plugins and its resources. The following sample shows an example.
```yaml theme={"dark"}
pluginValues:
- id: "plugin-id" <1>
resources: <2>
limits:
cpu: "2"
memory: "3Gi"
requests:
cpu: "250m"
memory: "2Gi"
replicaCount: 1 <3>
```
* `<1>` The plugin ID. The plugin ID must match the plugin ID on the plugin ZIP file. without the `lucidworks.` prefix. For example, if the plugin ID on the plugin ZIP file is `lucidworks.sharepoint-optimized`, the plugin ID is `sharepoint-optimized`.
* `<2>` The resources settings. You may specify the limits, the requests, and the CPU and memory for each.
* `<3>` The number of replicas per connector. This value is 1 by default.
After editing the `connector-plugin` section, you must reinstall the affected connector.
#### Upgrades and Ingress
If you used the `-t -h ` options when installing your cluster, our script created an additional values yaml file named `tls-values.yaml`.
To make things easier for you when upgrading, you should add the settings from this file into your main custom values yaml file, e.g.:
```yaml theme={"dark"}
api-gateway:
service:
type: "NodePort"
ingress:
enabled: true
host: ""
tls:
enabled: true
annotations:
"networking.gke.io/managed-certificates": "-managed-certificate"
"kubernetes.io/ingress.class": "gce"
```
This way you don't have to remember to pass the additional `tls-values.yaml` file when upgrading.
## Verifying the Fusion Installation
In this section, we provide some tips on how to verify the Fusion installation.
Check if the Fusion Admin UI is available at `\https://:6764/admin/`.
Let's review some useful kubectl commands.
### Enhance the K8s Command-line Experience
Here is a list of tools we found useful for improving your command-line experience with Kubernetes:
* krew (kubectl plugin mgr): [https://github.com/kubernetes-sigs/krew/](https://github.com/kubernetes-sigs/krew/)
* kube-ps1 (show current context on command line prompt): [https://github.com/jonmosco/kube-ps1](https://github.com/jonmosco/kube-ps1)
* kubectx / kubens (switch between clusters / namespaces): [https://github.com/ahmetb/kubectx](https://github.com/ahmetb/kubectx)
### Useful kubectl commands
kubectl reference: [https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands)
Set the namespace for `kubectl` if not using the default:
```bash theme={"dark"}
kubectl config set-context --current --namespace=
```
**This saves you from having to pass `-n` with every command.**
Get a list of running pods: `k get pods`
Get logs for a pod using a label: `k logs –l app.kubernetes.io/component=query-pipeline`
Get pod deployment spec and details: `k get pods -o yaml`
Get details about a pod events: `k describe po `
Port forward to a specific pod: `k port-forward 8983:8983`
SSH into a pod: `k exec -it -- /bin/bash`
CPU/Memory usage report for pods: `k top pods`
Forcefully kill a pod: `k delete po --force --grace-period 0`
Scale up (or down) a deployment: `k scale deployment.v1.apps/ --replicas=N`
Get a list of pod versions: `k get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq`
### Check Fusion Pods and Services
Once the install script completes, you can check that all pods and services are available using:
```bash theme={"dark"}
kubectl get pods
```
If all goes well, you should see a list of pods similar to:
```bash wrap theme={"dark"}
NAME READY STATUS RESTARTS AGE
seldon-controller-manager-6675874894-qxwrv 1/1 Running 0 8m45s
f5-admin-ui-74d794f4f8-m5jms 1/1 Running 0 8m45s
f5-ambassador-fd6b9b5dc-7ghf6 1/1 Running 0 8m43s
f5-api-gateway-6b9998b9c-tmchk 1/1 Running 0 8m45s
f5-auth-ui-7565564b4c-rdc74 1/1 Running 0 8m42s
f5-classic-rest-service-0 1/1 Running 3 8m44s
f5-devops-ui-77bb867ffb-fbzxd 1/1 Running 0 8m42s
f5-fusion-admin-78b8f8fc7f-4d7l8 1/1 Running 0 8m42s
f5-fusion-indexing-599c8d448-xzsvm 1/1 Running 0 8m44s
f5-insights-665fd9f6fc-g5psw 1/1 Running 0 8m43s
f5-job-launcher-84dd4c5c96-p8528 1/1 Running 0 8m44s
f5-job-rest-server-6d44d964b8-xtnxw 1/1 Running 0 8m45s
f5-logstash-0 1/1 Running 0 8m45s
f5-ml-model-service-6987dc94c9-9ppp8 2/2 Running 1 8m45s
f5-monitoring-grafana-5d499dbb58-pzw72 1/1 Running 0 10m
f5-monitoring-prometheus-kube-state-metrics-54d6678dv9h7h 1/1 Running 0 10m
f5-monitoring-prometheus-pushgateway-7d65c65b85-vwrwf 1/1 Running 0 10m
f5-monitoring-prometheus-server-0 2/2 Running 0 10m
f5-pm-ui-86cbc5bb65-nd2n8 1/1 Running 0 8m44s
f5-pulsar-bookkeeper-0 1/1 Running 0 8m45s
f5-pulsar-broker-b56cc776f-56msx 1/1 Running 0 8m45s
f5-query-pipeline-5d75d7d5f4-l2mdf 1/1 Running 0 8m43s
f5-connectors-7bb6cfc65f-7wfs2 1/1 Running 0 8m42s
f5-connectors-backend-987fdc648-dldwv 1/1 Running 0 8m45s
f5-rules-ui-6b9d55b78f-9hzzj 1/1 Running 0 8m43s
f5-solr-0 1/1 Running 0 8m44s
f5-solr-exporter-c4687c785-jsm7x 1/1 Running 0 8m45s
f5-ui-6cdbcc68c6-rj9cq 1/1 Running 0 8m45s
f5-webapps-6d6bb9bfd-hm4qx 1/1 Running 0 8m45s
f5-workflow-controller-7b66679fb7-sjbvp 1/1 Running 0 8m44s
f5-zookeeper-0 1/1 Running 0 8m45s
```
The number of pods per deployment / statefulset will vary based on your cluster size and replicaCount settings in your custom values YAML file.
Also, don't worry if you see some pods having been restarted as that just means they were too slow to come up and Kubernetes killed and restarted them.
You do want to see at least one pod running for every service. If a pod is not running after waiting a sufficient amount of time,
use `kubectl logs ` to see the logs for that pod; to see the logs for previous versions of a pod, use: `kubectl logs -p`.
You can also look at the actions Kubernetes performed on the pod using `kubectl describe po `.
To see a list of Fusion services, do:
```bash theme={"dark"}
kubectl get svc
```
For an overview of the various Fusion 5 microservices, see: [Fusion microservices](/docs/5/fusion/reference/microservices).
Once you're ready to build a Fusion cluster for production, please see see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
### Upgrading with Zero Downtime
One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. Fusion 5 allows customers to upgrade from Fusion 5.x.y to a later 5.x.z version on a live cluster with zero downtime or disruption of service.
When Kubernetes performs a rolling update to an individual microservice, there will be a mix of old and new services in the cluster concurrently (only briefly in most cases) and requests from other services will be routed to both versions. Consequently, Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same 5.x line of releases. We also ensure stored configuration remains compatible in the same 5.x release line.
Lucidworks releases minor updates to individual services frequently, so our customers can pull in those upgrades using Helm at their discretion.
To upgrade your cluster at any time, use the `--upgrade` option with our setup scripts in this repo.
The scripts in this repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.
To see what would be upgraded, you can pass the `--dry-run` option to the script.
### Grafana Dashboards
Get the initial Grafana password from a K8s secret by doing:
```bash wrap theme={"dark"}
kubectl get secret --namespace "${NAMESPACE}" ${RELEASE}-monitoring-grafana \
-o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```
With Grafana, you can either setup a temporary port-forward to a Grafana pod or expose Grafana on an external IP using a K8s LoadBalancer.
To define a LoadBalancer, do (replace \$ with your Helm release label):
```bash wrap theme={"dark"}
kubectl expose deployment ${RELEASE}-monitoring-grafana --type=LoadBalancer --name=grafana --port=3000 --target-port=3000
```
You can use `kubectl get services --namespace ` to determine when the load balancer is setup and its IP address. Direct your browser to `http://:3000` and enter the username `admin@localhost` and the password that was returned in the previous step.
This will log you into the application. It is recommended that you create another administrative user with a more desirable password.
The dashboards and datasoure will be setup for you in grafana, simply navigate to `Dashboards` -> `Manage` to view the available dashboards
## More deployment options
* How to deploy Fusion 5 on Amazon Elastic Kubernetes Service
* How to deploy Fusion 5 on Azure Kubernetes Service
* How to deploy Fusion on 5 other Kubernetes platforms
## Learn more
The course for **Deploying Fusion 5** focuses on the prerequisite software needed to deploy Fusion, the necessary setup steps, and the physical act of deployment.
Looking to upgrade your Fusion instance? See **Fusion 5 Upgrades**.
This guide describes how to perform Fusion 5 upgrades.
Before upgrading, be aware of changes by checking for [Deprecations and Removals](/docs/5/fusion/deprecations-and-removals) between versions.
Lucidworks recommends upgrading to the next minor version only. For example, you should upgrade from Fusion 5.6.1 to Fusion 5.7.1 before upgrading to Fusion 5.8.0.
The [general upgrade process](#general-upgrade-process) is described in this article. However, the specific upgrade procedures may vary depending on your upgrade path. For the most accurate instructions, please refer to the upgrade article specific to your upgrade.
## General upgrade process
Fusion natively supports deployments on supported Kubernetes platforms, including AKS, EKS, and GKE.
Fusion includes an upgrade script for AKS, EKS, and GKE. This script is not generated for other Kubernetes deployments.
Upgrades differ from platform to platform. See below for more information about upgrading on your platform of choice.
Whenever you upgrade Fusion, you must also update your [remote connectors](/docs/fusion-connectors/developers/remote-v2-connectors), if you are running any.
You can download the latest files at [V2 Connectors Downloads](/docs/fusion-connectors/downloads/v2-connectors-downloads).
### Natively supported deployment upgrades
| Deployment type | Platform |
| - | - |
| **Azure Kubernetes Service (AKS)** | `aks` |
| **Amazon Elastic Kubernetes Service (EKS)** | `eks` |
| **Google Kubernetes Engine (GKE)** | `gke` |
Fusion includes upgrade scripts for natively supported deployment types. To upgrade:
1. Open the `___upgrade_fusion.sh` upgrade script file for editing.
2. Update the `CHART_VERSION` to your target Fusion version, and save your changes.
3. Run the `___upgrade_fusion.sh` script. The `` value is the same as your namespace, unless you overrode the default value using the `-r` option.
After running the upgrade, use `kubectl get pods` to see the changes applied to your cluster. It may take several minutes to perform the upgrade, as new Docker images are pulled from DockerHub. To see the versions of running pods, do:
```
kubectl get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq
```
### Other Kubernetes deployment upgrades
To update an existing installation, do:
```
RELEASE=f5
NAMESPACE=default
helm repo update
helm upgrade ${RELEASE} "lucidworks/fusion" --namespace "${NAMESPACE}" --values "${MY_VALUES}"
```
Except for ZooKeeper, all K8s deployments and statefulsets use a `RollingUpdate` update policy:
```yaml theme={"dark"}
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
```
ZooKeeper instances use `OnDelete` to avoid changing critical stateful pods in the Fusion deployment. To apply changes to Zookeeper after performing the upgrade (uncommon), you need to manually delete the pods. For example:
```
kubectl delete pod f5-zookeeper-0
```
Delete one pod at a time. Verify the new pod is healthy and serving traffic, before deleting the next healthy pod.
You can also set the `updateStrategy` under the `zookeeper` section in your `"${MY_VALUES}"` file:
```yaml theme={"dark"}
solr:
...
zookeeper:
updateStrategy:
type: "RollingUpdate"
```
#### Upgrades with Helm v3
One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. For example, Fusion 5 allows customers to upgrade from Fusion 5.1.0 to a later 5.x.y version on a live cluster with zero downtime or disruption of service.
When Kubernetes performs a rolling update to an individual microservice, there is a mix of old and new services in the cluster. Requests from other services route to both versions.
Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same minor release version (5.x). We also ensure that the stored configuration remains compatible in the same minor release version.
Lucidworks releases minor updates to individual services frequently. Pull in those upgrades using Helm at your discretion.
**How to upgrade Fusion**
1. Clone the [**fusion-cloud-native** repo](https://github.com/lucidworks/fusion-cloud-native), if you haven’t already.
2. Locate the `setup_f5_.sh` script that matches your Kubernetes platform.
3. Run the script with the `--upgrade` option.
To see what would be upgraded, pass the `--dry-run` option to the script.
The scripts in the **fusion-cloud-native** repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.
#### Helm upgrade script
Once you deploy a working cluster, use the upgrade script created by the `customize_fusion_values.sh` script. The upgrade script hard-codes the parameters and eases the need to remember which parameters to pass to the script. This is helpful when working with multiple K8s clusters. Make sure you check the script into version control alongside your custom values YAML files.
Whenever you change the custom values YAML files for your cluster, you need to run the upgrade script to apply the changes. The script calls `helm upgrade` with the correct parameters and `--values` options.
If you run `helm upgrade` without passing the custom values YAML files, the deployment will revert to using chart defaults, which you never want to do.The script assumes your `kubeconfig` is pointing to the correct cluster and you’re using Heml v3. If not, the upgrade fails. Select the correct `kubeconfig` before running the script.
## Important reference information
Our reference section includes information on [Fusion’s API](/docs/5/fusion/dev-portal/rest-apis), [index pipelines stages](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/overview), [query pipelines stages](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/overview), [connections](/docs/fusion-connectors/connectors/overview), and more.
See [Reference](/docs/5/fusion/reference/overview) for complete reference information.
## LucidAcademy
Lucidworks offers free training to help you get started with Fusion. Visit the [LucidAcademy](https://academy.lucidworks.com) and select classes such as **Working With Data** and **Query Fine-Tuning** to learn how Fusion ingests, indexes, and queries data.
## Key Improvements over Fusion 4
Fusion 5 offers the following upgrade benefits over Fusion 4:
* [Predictive Merchandiser](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/overview): AI-powered visual tool including abilities to pin, boost, bury, or block products, rewrite and fix misspellings, and interpret synonyms.
* Advanced [Machine Learning](/docs/5/fusion/intro/machine-learning/overview) models and jobs.
* [Semantic Vector Search](/docs/5/fusion/intro/machine-learning/vector-search) to improve product discovery by influencing recall, relevancy, and precision.
* [Signals](/docs/5/fusion/getting-data-out/query-enhancement/signals/overview) for automatic self-tuning relevancy.
* Goes beyond search for personalized content discovery: Browse pages, Recommendations, Chatbots, AB Testing, Analytics, and more.
# Fusion 5.17.0
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.17/5.17.0-release-notes
Released on April 15, 2026, this is a [minor release](/docs/policies/lifecycle-policies/lw-semantic-version-support-lifecycle#minor-release-support-policy) that focuses on AI-driven search integration through MCP, event-driven autoscaling with KEDA, and enhanced Dynamic Index hierarchical support.
Upgrading to the latest version of Fusion offers several key benefits:
* **Access to latest features:** Stay current with the latest features and functionality to ensure compatibility and optimal performance.
* **Simplified process:** Fusion 5.9.5 and later use an in-place upgrade strategy, making upgrades easier than ever.
* **Extended support:** Upgrading keeps you up-to-date with the latest supported Kubernetes versions, as outlined in the [Lucidworks Semantic Version Support Lifecycle](/docs/policies/lifecycle-policies/lw-semantic-version-support-lifecycle) policy.
For supported Kubernetes versions and key component versions, see [Platform support and component versions](#platform-support-and-component-versions).
## Key highlights
### Lucidworks MCP for agentic search
Fusion 5.17 introduces support for the Model Context Protocol (MCP), an open standard that enables AI assistants to search your Fusion data.
With the new MCP server, you can connect Claude Code to your Fusion instance using a standardized protocol.
The MCP server integrates with Fusion's existing security controls, query pipelines, and multi-tenant architecture, allowing different users and use cases to access different data subsets through query profiles.
Electronic component distributors and industrial suppliers can search technical specifications using queries like "Find ceramic capacitors rated for 50V with 10% tolerance" to locate precise parts across complex catalogs.
Retail customers can ask "Find wireless headphones under \$200 with noise cancellation, sorted by rating" to discover products conversationally without navigating faceted search interfaces.
Employees search enterprise documentation with queries like "Find JWT authentication guides from the past six months" and receive filtered results with highlighted snippets while field-level security protects sensitive content.
For complete configuration options, multi-tenant setups, and troubleshooting, see [Lucidworks MCP](/docs/5/fusion/getting-data-out/mcp).
### Kubernetes event-driven autoscaling
Fusion 5.17 adds support for [KEDA (Kubernetes Event-Driven Autoscaling)](https://keda.sh/) as an alternative autoscaling approach for Fusion workloads.
Unlike traditional HPA-based scaling that relies mainly on CPU and memory utilization, KEDA enables you to scale based on events, schedules, and business metrics.
This allows you to align infrastructure capacity more closely with real demand, improving operational efficiency and reducing unnecessary resource consumption.
KEDA provides these advantages for managing Fusion workloads:
* **Event-driven autoscaling**
Scale Fusion workloads in response to external signals such as Prometheus metrics, queue depth, or pipeline execution load.
* **Scheduled scaling**
Automatically scale workloads up or down based on predictable traffic patterns, such as market hours or peak usage windows.
* **Scale-to-zero capability**
Reduce infrastructure costs by allowing certain workloads to scale down to zero during off-hours or periods of inactivity.
* **Improved operational efficiency**
Align infrastructure capacity with real business demand instead of relying solely on CPU or memory thresholds.
In this release, these Fusion services support KEDA:
* `api-gateway`
* `query-pipeline`
* `fusion-indexing`
For complete details and setup instructions, see [Autoscaling with KEDA](/docs/5/fusion/operations/keda).
### Connectors improvements
#### Skipping validation is now supported in V2 and Pro connectors
All V2 and Pro connectors now support a new core property that allows you to skip validation when needed. In previous versions of Fusion, long-running validation checks could cause connector configuration timeouts in the UI, preventing you from saving connector configurations even when the settings were correct. You can now configure connectors to skip validation by setting the appropriate property in the datasource configuration.
Enable to skip configuration validation when it takes too long and causes a timeout issue.
This gives you more control over the connector configuration process and prevents unnecessary delays when you're confident in your settings.
#### The Selenium version for the Web connector has been upgraded
The Selenium Hub used by the Web connector has been upgraded from version 4.20.0 to version 4.38.0. The previous version was over a year old and contained security vulnerabilities. Users can expect more reliable web crawling operations, enhanced security, and improved browser compatibility with Chrome, Firefox, and Edge.
### Facet and JSON Facet stage improvements
Both the Facet and JSON Facet stages have been enhanced with several new capabilities that give you more control over facet presentation and matching:
* **Boost, bury, and suppress facet values**
The Facet and JSON Facet stages now support boost, bury, and suppress options for individual facet values, similar to the capabilities previously available only in facet rules. You can now promote important facet values to the top of the list (boost), move less relevant values to the bottom (bury), or hide specific values entirely (suppress) directly within the stage configuration. This provides greater flexibility for controlling facet display without requiring separate rules.
* **Facet field ordering**
You can now control the order in which facet fields appear in query responses for both standard facets and JSON facets. This ordering capability is available in both facet stages and facet rules, allowing you to define a specific display sequence for Field and Range facets that matches your business requirements or user interface needs.
### Monitoring improvements with kafka-exporter and jmx-exporter
Fusion 5.17 introduces Docker images for `kafka-exporter` and `jmx-exporter`, providing enhanced monitoring capabilities for Kafka and JMX metrics. These exporters enable you to collect detailed metrics from Kafka brokers and Java applications, making it easier to monitor Fusion component health and performance through Prometheus and other observability tools. The `kafka-exporter` provides visibility into Kafka consumer lag, topic metrics, and broker performance, while the `jmx-exporter` exposes JVM and application-level metrics for Fusion services.
### Additional model parameters supported in Ray and Seldon vectorization stages
The [Ray / Seldon Vectorize Field](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/vectorize-field-via-seldon-index-stage) and [Ray/Seldon Vectorize Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/vectorize-query-via-seldon-query-stage) stages now support passing additional custom parameters to Ray and Seldon models beyond the basic text input. You can now include parameters such as tracking IDs, custom endpoints, input/output keys, and other model-specific configuration options in your vectorization requests.
Additional model parameters to pass to the model. These will be included in the model request. Each item in the array contains a key-value pair.
The name of the model parameter to pass in the request.
The value for the model parameter.
This enhancement provides greater flexibility for logging, monitoring, and customizing model behavior to meet specific deployment requirements. The feature is backward compatible with existing configurations and works with both Ray and Seldon model deployments.
### ConfigSync annotations
ConfigSync now supports [annotations](/docs/5/fusion/operations/config-sync/metrics) that overlay ConfigSync event information on Grafana time-series graphs to highlight when specific events occurred. Annotations provide context by overlaying event information directly onto your graphs, making it easier to correlate Fusion system behavior with ConfigSync activities.
```promql Git files changed theme={"dark"}
increase(configsync_git_file_changes_total[1m]) > 0
```
```promql Fusion updated theme={"dark"}
increase(configsync_fusion_updates_total[1m]) > 0
```
```promql User commits theme={"dark"}
increase(configsync_git_commits_total{commit_type="explicit"}[1m]) > 0
```
### Commerce Studio collection backup support in ConfigSync
ConfigSync now supports automatic backup of Commerce Studio collections. When Commerce Studio is enabled for an application, it creates a specialized set of collections with the `_em` suffix (for example, `app_name_query_rewrite_em`).
These collections are now automatically included in ConfigSync backups alongside standard query rewrite collections, ensuring that Commerce Studio configurations are preserved and can be synchronized across environments.
## API Updates
Fusion 5.17 removes several deprecated endpoints in favor of more standardized alternatives. These changes align Fusion's API surface with modern best practices and improve consistency across services.
The following endpoints have been removed:
* `PUT /collections/{collection}/schema/dynamicfields/{fieldName}` has been removed. Instead, use `POST /solr/{collection}/schema` with a `replace-dynamic-field` command in the request body:
```json theme={"dark"}
{
"replace-dynamic-field": {
"name": "*_s",
"type": "string",
"stored": true
}
}
```
* `PUT /collections/{collection}/schema/fields/{fieldName}` has been removed. Instead, use `POST /solr/{collection}/schema` with a `replace-field` command in the request body:
```json theme={"dark"}
{
"replace-field": {
"name": "field_name",
"type": "string",
"stored": true
}
}
```
### Application API enhancements for Solr collection configuration
The Fusion Application Create API now provides advanced options for controlling default Solr shard and replica strategies when creating applications programmatically. You can now specify the number and types of replicas (NRT, TLOG, or PULL) for the collections that will be created with your application. The API also provides visibility into all collections that will be created for the application, giving you better control over resource allocation and replication strategies. This enhancement is particularly useful for customers who need to customize Solr topology to match specific performance, availability, or resource constraints when deploying Fusion applications through automation.
### OpenAPI documentation improvements
Several Fusion services have been updated to use OpenAPI specifications and resolve Swagger documentation issues. The fusion-indexing, fusion-api-gateway, and fusion-chassis services now provide improved API documentation with proper OpenAPI compliance, making it easier to understand available endpoints, request/response formats, and integration patterns. These updates enhance the developer experience when working with Fusion APIs and ensure that API documentation accurately reflects the actual service capabilities.
## Bug fixes
### PIN rules: pinned items outside the result window are now excluded rather than promoted to position 1
When a document was pinned to a position greater than the query's `rows` value, Fusion incorrectly promoted it to position 1 instead of omitting it.
PIN rule processing is now pagination-aware: pins that fall outside the current page are ignored.
| Scenario | Before | After |
| - | - | - |
| Pin at position 15, rows=10 | Promoted to position 1 | Excluded from results |
| Pin at position 15, rows=30 | Appears at position 15 | Appears at position 15 |
### LWAI Gateway SSL protocol mismatch fixed
In Fusion 5.9.16, self-hosted deployments using TLS with the LWAI Gateway service experienced HTTP `503 Service Unavailable` errors when users attempted to configure LWAI Vectorize Field or Query stages. This occurred because the API Gateway attempted HTTPS connections to the LWAI Gateway service, which was only listening on HTTP, causing an `Unsupported or unrecognized SSL message` error.
The protocol mismatch has been resolved, and self-hosted TLS deployments can now properly configure LWAI vectorization stages.
### Chunking RAG bridge stage now escapes document IDs properly
In previous versions of Fusion, the Chunking RAG Bridge stage failed with a Solr syntax error when processing documents with IDs containing special characters such as colons:
```json wrap focus={3-4,6} theme={"dark"}
{
"timestamp" : "2026-04-09T14:50:03.947759",
"service" : "query",
"error" : "client",
"path" : "/query-pipelines/Lucidworks_Docs_Site_Chunking_memoryUuid/collections/chunking/select",
"message" : "org.apache.solr.search.SyntaxError: Cannot parse '(id:https://doc.lucidworks.com/docs/lw-platform/lw-cs/cstudio-editor* OR id:https://doc.lucidworks.com/docs/lw-platform/lw-platform/overview* OR ...': Encountered \" \":\" \": \"\" at line 1, column 9.\nWas expecting one of:\n ...\n ...\n ...\n \"+\" ...\n \"-\" ...\n ...\n \"(\" ...\n \")\" ...\n \"*\" ...\n \"^\" ...\n ...\n ...\n ...\n ...\n ...\n ...\n \"[\" ...\n \"{\" ...\n ...\n \"filter(\" ...\n ...\n"
}
```
This pattern is commonly seen in web-crawled content where the document ID is a URL such as `https://example.com`. The stage now properly escapes document IDs in filter queries, allowing it to work correctly with web-crawled content and other sources that use special characters in document IDs.
### Pipeline cache initialization fixed to reduce ZooKeeper load
In previous versions of Fusion, query and index pipeline services generated excessive ZooKeeper read traffic due to an uninitialized cache. The cache was created but never initialized, causing every pipeline lookup to bypass the cache and query ZooKeeper directly. This resulted in high ZooKeeper traffic that scaled with query volume and the number of pipeline pods in the cluster.
The cache initialization has been fixed, significantly reducing ZooKeeper traffic and improving overall cluster performance.
### Solr Indexer stage no longer fails with `NullPointerException` when using Unmapped Fields Mapping
In previous versions of Fusion starting with 5.9.10, the Solr Indexer stage could fail with a `NullPointerException` when the **Unmapped Fields Mapping** configuration was enabled with a DELETE operation. This occurred when a field configured for deletion in the **Unmapped Fields Mapping** settings caused the validator to incorrectly return null when processing other unrelated fields that had multiple values being mapped to single-value fields. Documents would fail to index entirely when this condition was triggered.
The field validation logic has been fixed to properly handle DELETE operations in the **Unmapped Fields Mapping** configuration, ensuring that unmapped field operations only affect their intended target fields and do not interfere with the validation of other fields in the document.
### Ray vectorization now handles non-ASCII characters correctly
In previous versions of Fusion, Ray vectorization failed when processing text containing non-ASCII characters such as trademark symbols, copyright symbols, or other special characters. Query pipelines failed with a `Model execution error: CANCELLED: Failed to read message` error, while index pipelines silently omitted vector fields from documents without any error message.
This has been fixed by properly setting `charset=UTF-8` in the `Content-Type` header for all Ray model requests. Query and index pipelines using Ray-based vectorization stages now correctly process text containing special characters, ensuring that documents with these characters are properly vectorized.
### Filter queries now trigger rules when fq contains OR for the same field
In previous versions of Fusion, rules were only triggered when filter queries (`fq`) contained single values for a field, such as `fq=categories_s:hep-th`. Rules failed to trigger when multiple values for the same field were combined using OR logic, such as `fq=categories_s:(hep-ph OR hep-th)`. This prevented rules from applying consistently across different query patterns.
The rules engine has been enhanced to support OR-based filter values for the same field, so rules now trigger correctly whether you use single filter values or combine multiple values with OR clauses.
### Vectorize query stage performance improved
In previous versions of Fusion, the vectorize query stage could experience significant latency due to inefficient OAuth token handling. The OAuth token initialization was being performed lazily on each request rather than being cached properly.
This has been fixed by optimizing the OAuth token initialization and caching strategy, significantly reducing query latency. Additional logging has been added to improve troubleshooting and observability for vectorization stages.
### Helm chart whitespace syntax issues fixed
In previous versions of Fusion starting with 5.9.14, Helm charts contained whitespace formatting issues that could cause YAML syntax errors during deployment. These whitespace problems affected chart parsing and could prevent successful installations or upgrades.
The Helm charts have been corrected to ensure proper YAML formatting and eliminate syntax problems related to whitespace.
### Admin UI logging loop fixed
Previously, the Admin UI could enter an infinite logging loop when the `/api/ui-logs` endpoint failed. When this endpoint returned an error, the Admin UI would attempt to log that error, which would send another request to the same failing endpoint, creating a cascading loop.
The Admin UI now detects and prevents logging errors that originate from the ui-logs endpoint itself, breaking the infinite loop. Additionally, log queues are now properly cleared even when logging fails, and error details are output to the console for debugging purposes.
### JavaScript stage security enhancement
A security patch has been created to disable the Nashorn JavaScript engine for query and index pipeline JavaScript stages. This enhancement restricts JavaScript execution to only the more secure GraalVM engine, providing enhanced security for environments with strict security requirements.
This feature is particularly useful for customers undergoing penetration testing or who require highly secure JavaScript execution environments.
You can enable it by adding this to your Fusion Helm chart:
```yaml theme={"dark"}
featureFlags:
javascriptIsolated: true
```
### Parsing and indexing fixes
Several datasource indexing and parsing bugs have been fixed that previously caused data inconsistencies, missing documents, or incorrect field extraction.
* When Fusion parsed a document into multiple documents (such as splitting a large file into sections), it would index an extra `container-doc` document representing the parent document alongside the parsed child documents. This behavior caused confusion for users who would see unexpected extra documents in their index.
The connector now skips indexing the `container-doc` while still maintaining proper deletion handling for parsed documents.
* When Fusion parsed XML files with a root path configuration of `/` (representing the entire XML file as a single document), the `absolute_resource_name` field incorrectly included a `#0` suffix. For example, `https://example.com/file.xml` would be stored as `https://example.com/file.xml#0`.
This suffix has been removed, and the `absolute_resource_name` field now correctly reflects the original XML file path without any suffix.
### LWAI Vectorize stage validation errors improved
In previous versions of Fusion, providing invalid or misplaced parameters in the LWAI Vectorize stage configuration (such as placing a parameter meant for `modelConfig` into `useCaseConfig` instead) resulted in an unhelpful `NullPointerException`.
The stage now properly validates configuration parameters and returns an HTTP `400 Bad Request` with a descriptive error message that indicates which parameter is invalid, the list of valid parameters for each configuration section, and contextual suggestions. This improves the user experience and reduces troubleshooting time.
### Updating a datasource now displays default values
In previous versions of Fusion, default values for datasource configuration fields were only displayed when creating a new datasource. When editing an existing datasource, fields that had no value set by the user would appear empty instead of showing their default values. This made it difficult to understand the actual datasource configuration being used and could lead to confusion about whether default values were being applied.
Default values now display properly in edit mode for all fields that have not changed the default value, ensuring consistency between the create and edit experiences.
### Next job run time displays properly
In previous versions of Fusion, the next scheduled run time for jobs did not appear in the Jobs UI unless the browser was manually refreshed.
This has been fixed, and the job schedule information now displays correctly and updates automatically without needing to manually refresh your browser.
## Known issues
### Streaming mode indexing fails with unquoted character errors
When indexing content in streaming mode, the Jackson library upgrade (2.13.5 → 2.19.1) introduced stricter validation.
The current implementation might fail with the long requests which use a `Transfer-Encoding: chunked` header.
Below is an example of a failure message, though other messages are also possible:
```
Illegal unquoted character ((CTRL-CHAR, code 13)): has to be escaped using backslash to be included in name
```
As a workaround, divide your requests into batches and send each batch as a single request with a `Content-Length` header.
## Platform support and component versions
### Kubernetes platform support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.30, 1.31, 1.32, 1.33, 1.34, 1.35
* **Microsoft Azure Kubernetes Service (AKS):** 1.30, 1.31, 1.32, 1.33, 1.34, 1.35
* **Amazon Elastic Kubernetes Service (EKS):** 1.30, 1.31, 1.32, 1.33, 1.34, 1.35
Support is also offered for Rancher Kubernetes Engine (RKE and RKE2) and OpenShift 4 versions that are based on Kubernetes 1.30, 1.31, 1.32, 1.33, 1.34, 1.35; note that RKE2 may require some Helm chart modification. OpenStack and customized Kubernetes installations are *not* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-semantic-version-support-lifecycle#kubernetes-support).
### Component versions
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| Solr | fusion-solr 5.17.0 *(based on Solr 9.6.1)* |
| ZooKeeper | 3.9.1 |
| Spark | 3.4.1 |
| Ingress Controllers | Nginx, Ambassador (Envoy), GKE Ingress Controller |
| Ray | ray\[serve] 2.46.0 |
| Helm | 3.4.1 |
More information about support dates can be found at [Lucidworks Semantic Version Support Lifecycle](/docs/policies/lifecycle-policies/lw-semantic-version-support-lifecycle).
# Fusion 5.17.1
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.17/5.17.1-release-notes
Released on May 28, 2026, Fusion 5.17.1 is a [patch release](/docs/policies/lifecycle-policies/lw-semantic-version-support-lifecycle#patch-release-support-policy) that delivers a security update and bug fixes for Commerce rules and pipeline compatibility.
Upgrading to the latest version of Fusion offers several key benefits:
* **Access to latest features:** Stay current with the latest features and capabilities to ensure compatibility and optimal performance.
* **Simplified process:** Fusion 5.9.5 and later use an in-place upgrade strategy, making upgrades easier than ever.
* **Extended support:** Upgrading keeps you up-to-date with the latest supported Kubernetes versions, as outlined in the [Lucidworks Semantic Version Support Lifecycle](/docs/policies/lifecycle-policies/lw-semantic-version-support-lifecycle) policy.
For supported Kubernetes versions and key component versions, see [Platform support and component versions](#platform-support-and-component-versions).
## Bug fixes
### Fusion now supports numeric operators for attribute-based rules
In previous versions of Fusion, configuring numeric field conditions for Boost by Attribute, Bury by Attribute, and Block by Attribute rules required writing Solr queries directly.
Fusion now supports numeric comparison operators for attribute-based rules, eliminating the need for manual query construction.
This update also includes fixes to the underlying JSON structure to support multiple block groups separated by OR logic.
### MCP Server now starts correctly when TLS is enabled
In previous versions of Fusion, the MCP Server failed to start when TLS was enabled due to missing SSL keystore password configuration, resulting in `BadPaddingException` errors at startup.
The MCP Server configuration now includes the required SSL properties (`server.ssl.key-store-password` and `server.ssl.trust-store-password`) when TLS is enabled, allowing the server to properly decrypt the keystore and start successfully.
### Apply Rules stage now correctly combines pipeline tags and URL tags
In previous versions of Fusion, the Apply Rules stage overwrote pipeline-configured tags with user-provided URL tags when both were present, causing rules scoped to pipeline tags to stop firing.
For example, if a pipeline had `tags=pipelineTag1` and a request also included `tags=urlTag`, only rules for `urlTag` would fire.
Rules for `pipelineTag1` were silently ignored, even though the response headers correctly showed both tags.
The Apply Rules stage now combines pipeline-configured tags and URL tags so that rules for all active tags fire as expected.
### `kubectl cp` restored after Java upgrade
A regression introduced by a previous Java base image upgrade caused `kubectl cp` commands to fail when copying files from Fusion service pods. The failure occurred because the updated base image was missing the `libpcre2-8.so.0` shared library required by the `tar` utility used internally by `kubectl cp`:
```text theme={"dark"}
tar: error while loading shared libraries: libpcre2-8.so.0: cannot open shared object file: No such file or directory
command terminated with exit code 127
```
The base image now includes the missing library, and `kubectl cp` works correctly across all Fusion service pods.
### RAG Bridge Stage no longer returns incorrect vector format error for byte vector fields after upgrade
After upgrading to Fusion 5.17.0, the RAG Bridge Stage returned an "incorrect vector format" error when processing byte vector fields, preventing RAG-based pipelines from functioning correctly.
Fusion 5.17.1 resolves this regression, and the RAG Bridge Stage now correctly processes byte vector fields.
### Security update: CVE-2026-33870 patched
Fusion 5.17.1 includes security patches for CVE-2026-33870 in the following services:
* apps-manager
* job-config
* lwai-gateway
### Prometheus metrics restored for low time scale resolution
In previous versions of Fusion, Prometheus metrics lacked resolution when viewed at low time scales because smaller time bucket intervals were missing from the metrics configuration.
Fusion 5.17.1 restores the smaller bucket intervals, and Prometheus metrics now display with correct granularity at low time scales.
## Known issues
### Streaming mode indexing fails with unquoted character errors
When indexing content in streaming mode, the Jackson library upgrade (2.13.5 → 2.19.1) introduced stricter validation. The current implementation might fail with the long requests which use a `Transfer-Encoding: chunked` header. Below is an example of a failure message, though other messages are also possible:
```
Illegal unquoted character ((CTRL-CHAR, code 13)): has to be escaped using backslash to be included in name
```
As a workaround, divide your requests into batches and send each batch as a single request with a `Content-Length` header.
## Platform support and component versions
### Kubernetes platform support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.30, 1.31, 1.32, 1.33, 1.34, 1.35
* **Microsoft Azure Kubernetes Service (AKS):** 1.30, 1.31, 1.32, 1.33, 1.34, 1.35
* **Amazon Elastic Kubernetes Service (EKS):** 1.30, 1.31, 1.32, 1.33, 1.34, 1.35
Support is also offered for Rancher Kubernetes Engine (RKE and RKE2) and OpenShift 4 versions based on Kubernetes 1.30, 1.31, 1.32, 1.33, 1.34, 1.35.
Note that RKE2 may require some Helm chart modification.
OpenStack and customized Kubernetes installations *aren't* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-semantic-version-support-lifecycle#kubernetes-support).
### Component versions
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| Solr | fusion-solr 5.17.1 *(based on Solr 9.6.1)* |
| ZooKeeper | 3.9.1 |
| Spark | 3.4.1 |
| Ingress Controllers | Nginx, Ambassador (Envoy), GKE Ingress Controller |
| Ray | ray\[serve] 2.46.0 |
| Helm | 4.1.1 |
For more information about support dates, see [Lucidworks Semantic Version Support Lifecycle](/docs/policies/lifecycle-policies/lw-semantic-version-support-lifecycle).
# Fusion 5.17.2
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.17/5.17.2-release-notes
Released on June 12, 2026, Fusion 5.17.2 is a [patch release](/docs/policies/lifecycle-policies/lw-semantic-version-support-lifecycle#patch-release-support-policy) that delivers critical security updates to address vulnerabilities in the Netty framework that handles network communication.
**Upgrade strongly recommended:** While Lucidworks is not aware of any active exploitation, this update addresses security vulnerabilities that could potentially be exploited.
We strongly recommend upgrading to ensure continued security and platform stability.
Upgrading to the latest version of Fusion offers several key benefits:
* **Access to latest features:** Stay current with the latest features and capabilities to ensure compatibility and optimal performance.
* **Simplified process:** Fusion 5.9.5 and later use an in-place upgrade strategy, making upgrades easier than ever.
* **Extended support:** Upgrading keeps you up-to-date with the latest supported Kubernetes versions, as outlined in the [Lucidworks Semantic Version Support Lifecycle](/docs/policies/lifecycle-policies/lw-semantic-version-support-lifecycle) policy.
For supported Kubernetes versions and key component versions, see [Platform support and component versions](#platform-support-and-component-versions).
## Security updates
### Security vulnerabilities in Netty resolved
[CVE-2026-44249](https://nvd.nist.gov/vuln/detail/CVE-2026-44249)[CVE-2026-42577](https://nvd.nist.gov/vuln/detail/CVE-2026-42577)[CVE-2026-42579](https://nvd.nist.gov/vuln/detail/CVE-2026-42579)[CVE-2026-42581](https://nvd.nist.gov/vuln/detail/CVE-2026-42581)[CVE-2026-42583](https://nvd.nist.gov/vuln/detail/CVE-2026-42583)[CVE-2026-42584](https://nvd.nist.gov/vuln/detail/CVE-2026-42584)[CVE-2026-42585](https://nvd.nist.gov/vuln/detail/CVE-2026-42585)[CVE-2026-42587](https://nvd.nist.gov/vuln/detail/CVE-2026-42587)[CVE-2026-45416](https://nvd.nist.gov/vuln/detail/CVE-2026-45416)[CVE-2026-45674](https://nvd.nist.gov/vuln/detail/CVE-2026-45674)[CVE-2026-47691](https://nvd.nist.gov/vuln/detail/CVE-2026-47691)
Prior to this release, Netty versions before 4.1.135.Final contained multiple security vulnerabilities affecting HTTP request processing, DNS resolution, and network connection handling, including request smuggling vulnerabilities.
These vulnerabilities could potentially be exploited to bypass security controls, hijack user sessions, or compromise data integrity.
These vulnerabilities are now resolved.
Fusion 5.17.2 updates Netty to version 4.1.135.Final, which addresses these security issues across multiple Netty components including HTTP/HTTP2 codecs, DNS resolver, transport layer, and connection handlers.
Upgrading to Fusion 5.17.2 protects your deployment against these vulnerabilities.
## Known issues
### Streaming mode indexing fails with unquoted character errors
When indexing content in streaming mode, the Jackson library upgrade (2.13.5 → 2.19.1) introduced stricter validation.
The current implementation might fail with long requests that use a `Transfer-Encoding: chunked` header.
Below is an example of a failure message, though other messages are also possible:
```
Illegal unquoted character ((CTRL-CHAR, code 13)): has to be escaped using backslash to be included in name
```
As a workaround, divide your requests into batches and send each batch as a single request with a `Content-Length` header.
## Platform support and component versions
### Kubernetes platform support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.30, 1.31, 1.32, 1.33, 1.34, 1.35
* **Microsoft Azure Kubernetes Service (AKS):** 1.30, 1.31, 1.32, 1.33, 1.34, 1.35
* **Amazon Elastic Kubernetes Service (EKS):** 1.30, 1.31, 1.32, 1.33, 1.34, 1.35
Support is also offered for Rancher Kubernetes Engine (RKE and RKE2) and OpenShift 4 versions based on Kubernetes 1.30, 1.31, 1.32, 1.33, 1.34, 1.35.
Note that RKE2 may require some Helm chart modification.
OpenStack and customized Kubernetes installations *aren't* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-semantic-version-support-lifecycle#kubernetes-support).
### Component versions
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| Solr | fusion-solr 5.17.2 *(based on Solr 9.6.1)* |
| ZooKeeper | 3.9.1 |
| Spark | 3.4.1 |
| Ingress Controllers | Nginx, Ambassador (Envoy), GKE Ingress Controller |
| Ray | ray\[serve] 2.46.0 |
| Helm | 4.1.1 |
For more information about support dates, see [Lucidworks Semantic Version Support Lifecycle](/docs/policies/lifecycle-policies/lw-semantic-version-support-lifecycle).
# Fusion 5.18.0
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.18/5.18.0-release-notes
Released on July 7, 2026, this is a [minor release](/docs/policies/lifecycle-policies/lw-semantic-version-support-lifecycle#minor-release-support-policy) that focuses on AI enhancements, connector reliability improvements, and operational stability.
Upgrading to the latest version of Fusion 5 offers several key benefits:
* **Access to latest features:** Stay current with the latest features and functionality to ensure compatibility and optimal performance.
* **Simplified process:** Fusion 5.9.X and later use an in-place upgrade strategy.
* **Extended support:** Upgrading keeps you up-to-date with the latest supported Kubernetes versions, as outlined in the [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-fusion-product-lifecycle) policy.
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
For supported Kubernetes versions and key component versions, see [Platform support and component versions](#platform-support-and-component-versions).
## Key highlights
### Secrets management
Fusion now provides [Secrets management using Fusion Vault](/docs/5/fusion/operations/security/external-vault) that interfaces with [Kubernetes External Secrets Operator (ESO)](https://external-secrets.io/latest/) to reference and use credentials stored in external databases, services, and APIs.
External secret providers include Kubernetes, HashiCorp Vault, Amazon Web Services Secrets Manager, Google Cloud Platform (GCP) Secret Manager, and Azure Key Vault.
Install and configure ESO to interface between the vault provider and Fusion, and then configure Fusion pipelines, jobs, and connectors with variable references for the secrets.
Fusion contains the variable references, but does not store the secret values, ensuring usernames, passwords, certificates, and API keys are kept secure.
Credential values never display in Fusion configurations.
These examples illustrate the datasources, index and query pipeline stages, and job settings in Fusion.
Your managed secret is found in the datasource configuration in **Crawl Authentication Properties > Basic Authentication > Host**.
If selected, **Use Managed Secret** applies to the name, password, and secret reference.
Your managed secret is found in **Advanced > Scripts Secrets**.
When you update a credential in your vault, all related pipelines change automatically without manual pipeline updates or service restarts.
### OpenTelemetry tracing support
OpenTelemetry (OTel) provides distributed tracing capabilities for Fusion that enable engineering and operations teams to identify performance bottlenecks and errors across microservices in real time.
This comprehensive, request-level observability across Fusion's microservices helps teams trace individual queries separately from monitoring systems such as Prometheus without overwhelming them.
The API Gateway handles incoming queries and routes them through the Query Service to Solr.
The services that support OpenTelemetry are API Gateway, Query, and Indexing.
#### Distributed trace flow
This section illustrates and describes the OpenTelemetry distributed trace flow.
Fusion's distributed tracing uses the API Gateway as the trace root with selective sampling for Query Service requests, while the Query Service uses parent-based sampling to ensure complete trace coverage.
OpenTelemetry collectors run locally on each server to minimize network overhead, bundling trace data before sending it to Grafana Tempo.
The tracing system integrates with the existing Lucidworks observability stack (Grafana, Loki, Prometheus) without requiring code changes to Solr, which uses environment variable configuration.
For more information, see [OpenTelemetry Distributed Tracing](/docs/5/fusion/operations/monitoring-and-reporting/opentelemetry-implementation-fusion).
### Lucidworks MCP enhancements
Fusion's Model Context Protocol (MCP) implementation includes improved configuration capabilities, enhanced authentication options, and monitoring in this release.
Enhancements include:
* **OAuth authentication with external identity providers**: Fusion now supports OAuth 2.1 authentication using external identity providers for both HTTP and STDIO modes.
This enables enterprise single sign-on workflows for MCP access and is the recommended authentication method for Claude Desktop applications.
See [Set up authentication](/docs/5/fusion/getting-data-out/mcp#set-up-authentication) for configuration details.
* **UI-based search experience configuration**: For simpler setup, you can configure MCP search experiences directly through the Fusion [Query Profile](/docs/5/fusion/getting-data-out/query-basics/query-pipelines/query-profiles) panel instead of editing a configuration file.
* **Multi-experience search**: The MCP server can now use multiple search experiences in a single request, allowing Claude to gather information from multiple data sources in one query.
In 5.17.x, the server selected only the single best-matching search experience per request.
This enables AI assistants to retrieve information from multiple data sources in one query.
* **Enhanced document descriptions**: The MCP server now checks the `body_t` field as an additional source for document descriptions returned to AI assistants, providing richer document summaries.
* **MCP usage monitoring documentation**: The existing API gateway metrics that track MCP server usage (available since Fusion 5.17) are now documented with setup instructions and a ready-to-use Grafana dashboard.
The metrics provide request rates, latency percentiles, error rates, and traffic breakdowns by status code.
Note that STDIO-based MCP usage (such as earlier Claude Desktop integrations) does not flow through the `/mcp` endpoint and is not captured in these metrics.
See [MCP Server Monitoring](/docs/5/fusion/operations/monitoring-and-reporting/mcp-monitoring) for setup instructions and dashboard configuration.
For more information about MCP configuration and setup, see [Lucidworks MCP](/docs/5/fusion/getting-data-out/mcp).
### Mutual Transport Layer Security support
Fusion 5.18 introduces Mutual Transport Layer Security (mTLS) for self-hosted Fusion and Lucidworks Search clients on 5.9.15 and later.
mTLS is a network-layer authentication feature that extends standard TLS encryption by requiring both the client and server to present valid certificates during the TLS connection process.
While standard TLS encrypts traffic and allows clients to verify the server's identity, mTLS adds bidirectional authentication where the server also verifies the client's identity before allowing the connection.
mTLS is implemented at the Google Cloud External Application Load Balancer level and ensures that only trusted, certificate-authenticated clients can access APIs and UI endpoints.
This feature complements but does not replace application-layer authentication methods like SSO and OIDC, and provides an additional security layer that satisfies zero-trust architecture requirements.
Performance impact is minimal with only sub-millisecond latency added during initial connection, and the feature supports multiple CA certificates as trust anchors for flexible multi-environment access control.
This diagram illustrates an example authentication process.
For complete details, see [Mutual Transport Layer Security](/docs/5/fusion/operations/security/mtls-fusion).
#### Use cases
mTLS applies to B2B, B2C, and knowledge management scenarios.
Organizations can restrict Lucidworks Search API access to only authenticated services within their infrastructure, requiring valid client certificates signed by their certificate authority.
For multi-environment deployments, mTLS enables different certificate requirements across development, staging, and production environments.
Development environments can use self-signed certificates while production requires certificates from enterprise public key infrastructure (PKI).
The feature is particularly critical for organizations operating under zero-trust security frameworks that need to follow these guidelines:
* Adhere to strict compliance requirements such as SOC2, ISO 27001, or HIPAA
* Enforce network-layer security controls beyond application authentication
All client certificate usage is logged in Google Cloud logging, which provides comprehensive audit trails for regulatory compliance.
For more information, see [Lucidworks Search Mutual Transport Layer Security](/docs/5/fusion/operations/security/mtls-fusion).
### Ray environment variables
[Ray deploy jobs](/docs/5/fusion/reference/config-ref/jobs/create-ray-model-deployment) now support Ray environment variables, enabling better configuration and control of Ray-based machine learning workloads within Fusion's deployment infrastructure.
This enables configuration and control of Ray-based machine learning workloads, such as setting memory limits, adjusting worker parallelism, and configuring logging levels for Ray tasks within Fusion's deployment infrastructure.
### Connectors
#### Stray content deletion
All V2 and Pro [connectors](/docs/fusion-connectors/overview) now include a configurable circuit breaker that prevents accidental mass deletion of indexed content.
Previously, stray content deletion could silently remove all of a datasource's content due to misconfigurations or source system issues, with no safety mechanism to detect anomalous deletion patterns.
You can set a threshold in a connector's settings for the maximum percentage of indexed items that may be deleted as stray in a crawl.
If the total stray items exceeds this percentage, the job fails, and the circuit breaker blocks the deletion to prevent accidental data loss.
Failure details are visible in the job history.
To enable tracking each document, child documents of an embedded parser such as entries in a ZIP file or lines in a CSV file are now tracked by a signature so these documents also participate correctly in stray content deletion and circuit breaker accounting, instead of being silently reindexed or deleted.
Stray content deletion and the circuit breaker are enabled by default. You can edit these settings in each connector's core properties.
When enabled, items not re-encountered in the current crawl are deleted from the content collection.
When enabled, stray deletion is blocked if the percentage of items to be deleted exceeds the configured threshold.
Disable this setting only if unconditional stray deletion is required regardless of volume.
Maximum percentage of indexed items that may be deleted as stray in a crawl.
If the total stray items exceeds this percentage, the circuit breaker blocks the deletion to prevent accidental data loss.
Accepts values from 0 to 100.
The circuit breaker setting eliminates expensive recovery operations, prevents search downtime, and maintains search quality even when source systems experience temporary issues.
#### Sidecar collections for graph security trimming
Connectors that use [graph security trimming](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/security-trimming-graph-query-stage) can now route access-control documents to a dedicated sidecar collection separate from the content collection.
This enables improved performance and cleaner separation of documents for deployments using graph ACLs.
By default, ACL documents continue to land in the content collection, maintaining backward compatibility.
To use a sidecar collection, navigate to **Graph security filtering configuration** in your datasource and set the **ACL Collection ID** datasource parameter to the name of your dedicated ACL collection. After configuring your datasource, navigate to the Query Workbench and configure the graph security trimming query stage to use this sidecar collection.
Connectors that support sidecar collections include SharePoint Optimized, LDAP, and any V2 or Pro connector that emits graph ACLs.
Name of the dedicated sidecar collection for storing ACL records.
If not specified, ACL records are stored in the content collection (default behavior).
#### Document signatures
V2 and Pro connectors now evaluate document signatures to prevent reindexing unchanged content.
Each document, including parsed child documents, receives a `_lw_signature_s` field containing a fingerprint built from the document's fields, metadata, and ACLs.
During recrawls, documents with unchanged signatures skip reindexing, reducing memory overhead.
## API changes
### Connectors SDK upgrade
The Fusion Connectors SDK has been upgraded to version 4.2.5.
This release includes protobuf schema improvements, enhanced plugin resolution, and updated dependencies for custom connector development.
For details, see [Connectors SDK 4.2.5 Javadocs](https://lucidworks.github.io/connectors-sdk-javadocs/4.2.5/).
### Connector configuration endpoints removed
The Connectors API has removed the following internal configuration management endpoints:
* `DELETE /configs`
* `DELETE /configs/{id}`
* `GET /configs`
* `GET /configs/{id}`
* `POST /configs`
* `PUT /configs/{id}`
Applications using these connector configuration endpoints must migrate to alternative configuration management approaches.
### Templating API enhancements
Twenty-one templating endpoints received parameter additions and type corrections. These changes appear to be non-breaking enhancements that add missing parameters and fix type definitions in the OpenAPI specification. Review the [Fusion Templating API](/api-reference/render-controller/get-template-information-using-your-app-name) specification for complete details.
## Bug fixes
### Connectors
#### PhaseState no longer causes cached failures in fusion-connectors
In previous versions of Fusion, PhaseState operations resulted in cached failures that persisted inappropriately in the fusion-connectors service.
Fusion now properly manages PhaseState caching to prevent spurious failure states.
#### RepositoryPluginsService cache failures resolved
In previous versions of Fusion, the RepositoryPluginsService experienced cached failures that prevented proper plugin loading and initialization in fusion-connectors.
Fusion now correctly manages the plugin service cache lifecycle.
#### Stray content deletion now works correctly when parsers emit marker documents
In previous versions of Fusion, stray content deletion failed to remove orphaned child documents when a parser emitted an error or comment marker document before any records, causing orphaned documents to accumulate in the content collection across crawls.
Fusion now properly removes orphaned child documents to prevent document accumulation.
#### Connector plugin pods now allocated increased resources
In previous versions of Fusion, connector plugin pods operated with insufficient resource allocations, causing performance degradation and potential out-of-memory issues.
Fusion now allocates appropriate CPU and memory resources to connector plugin pods for reliable operation under load.
### AI and machine learning
#### LWAI stages now continue pipeline execution on failure
In previous versions of Fusion, LWAI stage failures caused entire pipelines to fail, preventing subsequent stages from executing.
Fusion now logs LWAI stage failures while allowing the pipeline to continue execution, ensuring that temporary AI service issues do not block document processing.
#### Ray vectorization now handles non-ASCII text correctly
In previous versions of Fusion, Ray vectorization failed for non-ASCII text due to missing UTF-8 encoding configuration.
Fusion now properly encodes text using UTF-8 before vectorization, ensuring reliable processing of international and special characters.
#### Machine Learning stage now works correctly with Neural Hybrid Query and Apply Rules
In previous versions of Fusion, using the Machine Learning stage together with the Neural Hybrid Query stage and the Apply Rules stage caused query pipeline failures.
Fusion now supports all three stages working together correctly in query pipelines.
#### lwai-gateway now returns correct error code on authentication failure
In previous versions of Fusion, the lwai-gateway service incorrectly returned HTTP 401 (Unauthorized) when it failed to authenticate itself with backend services, misleading clients about the nature of the error.
Fusion now returns HTTP 500 (Internal Server Error) to accurately indicate server-side authentication failures.
### Jobs and infrastructure
#### Jobs now display correct status
In previous versions of Fusion, jobs occasionally displayed incorrect status information in the UI.
Fusion now correctly reports job status in all cases.
#### job-launcher-spark Role now includes correct permissions
In previous versions of Fusion, the job-launcher-spark Kubernetes role was missing the `deletecollection` verb and referenced an incorrect PVC resource name, preventing proper cleanup operations.
Fusion now includes the correct RBAC permissions and resource references for Spark job management.
#### EventBus memory leaks eliminated
In previous versions of Fusion, Google Guava's EventBus caused memory leaks in singleton components, particularly affecting the DefaultDataProcessor and RPC service.
Fusion now migrates all remaining EventBus usage to Spring's event system, eliminating memory leaks and improving long-term stability.
#### Selenium driver updated in rpc-service
In previous versions of Fusion, the rpc-service used an outdated Selenium driver version with known issues and security vulnerabilities.
Fusion now uses the latest Selenium driver, providing improved compatibility and security.
#### Merge Async Results thread safety improved
In previous versions of Fusion, the DefaultContext class in the CDW 5.9.16 merge async results implementation was not thread-safe, causing ConcurrentExecutionException errors under concurrent load.
Fusion now resolves thread safety issues in asynchronous result merging.
#### apps-manager now starts correctly with SSL enabled
In previous versions of Fusion, the apps-manager service failed to start when SSL/TLS was enabled.
Fusion now properly initializes SSL configuration, allowing apps-manager to start successfully in TLS-enabled environments.
### User interface
#### Datasource configuration UI checkbox labels now align correctly
In previous versions of Fusion, checkbox labels in the datasource configuration UI appeared below or misaligned from their checkboxes, creating a confusing user experience.
Fusion now renders checkbox labels in the correct position next to their controls.
#### Blob binary data uploads now work correctly through UI
In previous versions of Fusion, binary blob data became corrupted when uploaded through the Admin UI.
Fusion now preserves binary data integrity during upload operations, ensuring files are stored and retrieved without corruption.
#### ACL label description clarified
In previous versions of Fusion, the ACL configuration label description was unclear, causing confusion during security configuration.
Fusion now provides clearer ACL label descriptions in the UI and documentation.
### Search and query
#### Bury by attribute now uses correct query parameter
In previous versions of Fusion, the Experience Manager API's bury by attribute functionality incorrectly used the `bq` (boost query) parameter instead of the `boost` parameter.
Fusion now correctly applies negative boosts using the `boost` parameter, ensuring buried items are properly demoted in Commerce Studio's search results.
#### Pinned documents now respect query parameters
In previous versions of Fusion, pinned documents returned by query rules did not inherit query parameters from the main query pipeline, such as the field list (`fl`) filter configured in the Query Fields stage.
Non-pinned documents correctly applied these parameters, but pinned documents outside the initial row count returned all fields regardless of pipeline configuration.
Fusion now propagates query parameters to the secondary lookup used to retrieve pinned documents, ensuring consistent field filtering across all results.
## Known issues
### JavaScript and Active Directory stages defer Vault validation
The Active Directory Security Trimming Stage and JavaScript pipeline stages (GraalVM, Nashorn, OpenJDK Nashorn) do not validate Vault-backed credentials during stage initialization.
The Active Directory stage recreates the LDAP resolver on every request using freshly resolved Vault credentials. Misconfigured or unavailable secrets only surface when the first query arrives, rather than being caught during pipeline deployment or updates.
JavaScript stages currently access Vault secrets through a polling mechanism that checks for updates on a fixed interval, rather than through a native integration with the Fusion Vault API. This means a JavaScript stage configured with a required secret that is unavailable in Vault can initialize successfully and only fail at request time. Secret values updated in the vault are not guaranteed to be reflected in the stage without a service restart.
**Impact:** Query pipelines with misconfigured Vault credentials appear to deploy successfully but fail when processing the first query.
**Workaround:** Manually verify all required Vault secrets are available and correctly configured before deploying or updating query pipelines that use the Active Directory Security Trimming Stage or JavaScript stages with Vault-backed credentials.
A future release will introduce Vault API integration for JavaScript stages, including seamless secret value updates without requiring service restarts and improved secret refresh capabilities.
This integration will change how secrets are accessed in JavaScript stage scripts.
If you use secrets management with JavaScript query stages, review and update your stage configurations before upgrading once this change is released.
## Deprecations
In Fusion 5.18.0, all V1 connectors are deprecated. This means they are no longer being actively developed and will be removed in Fusion 6.
* **Some V1 connectors already have a direct replacement available.** You can find the replacement connector on the [Fusion Connectors Deprecations and Removals page](/docs/fusion-connectors/deprecations-and-removals).
* **For all other V1 connectors, a replacement is still under development.** The documentation is updated when the replacement is available.
If you are using a V1 connector, you must migrate to the replacement connector or a supported alternative before upgrading to Fusion 6. Migrate to the replacement connector as soon as possible to avoid any disruption to your workflows.
## Platform support and component versions
### Kubernetes support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.31, 1.32, 1.33, 1.34, 1.35, 1.36
* **Microsoft Azure Kubernetes Service (AKS):** 1.31, 1.32, 1.33, 1.34, 1.35, 1.36
* **Amazon Elastic Kubernetes Service (EKS):** 1.31, 1.32, 1.33, 1.34, 1.35, 1.36
Lucidworks also supports Rancher Kubernetes Engine (RKE and RKE2) and OpenShift 4 versions that are based on Kubernetes 1.31, 1.32, 1.33, 1.34, 1.35, 1.36.
RKE2 may require Helm chart modification.
OpenStack and customized Kubernetes installations are *not* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-fusion-product-lifecycle#kubernetes-support).
### Components
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| Solr | fusion-solr 5.18.0 *(based on Solr 9.6.1)* |
| ZooKeeper | 3.9.1 |
| Spark | 3.4.1 |
| Ingress Controllers | Nginx, Ambassador (Envoy), GKE Ingress Controller |
| Ray | ray\[serve] 2.46.0 |
| Helm | 4.1.1 |
See [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-fusion-product-lifecycle) for more information about support dates.
# Fusion 5.9.0
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9/5.9.0-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.9.0-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9.0-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/kpv8sh
Released on June 29, 2023, this [long-term release](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#long-term-support-release-policy) includes the new Consumption Dashboard feature, adding Tika Asynchronous Parsing support for remote V2 connectors, and new version requirements for the LDAP ALCs V2 connector and the SharePoint Optimized V2 connector.
To learn more, skip to the [release notes](#new-features).
## Platform Support and Component Versions
### Kubernetes platform support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.25 and 1.26
* **Microsoft Azure Kubernetes Service (AKS):** 1.25 and 1.26
* **Amazon Elastic Kubernetes Service (EKS):** 1.25 and 1.26
Support is also offered for Rancher Kubernetes Engine (RKE) and OpenShift 4 versions that are based on Kubernetes 1.25 and 1.26. OpenStack and customized Kubernetes installations are *not* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support).
### Component versions
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| **Solr** | fusion-solr 5.9.0 *(based on Solr 9.1.1)* |
| **ZooKeeper** | 3.7.1 |
| **Spark** | 3.2.2 |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
**More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).**
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
Upgrades from Fusion 5.8.0
Fusion 5.8.1 fixed an issue with the Fusion 5.8.0 Helm chart that prevented horizontal pod autoscaling.
If you are upgrading from Fusion 5.8.0 to any later version and use horizontal pod autoscaling, you must complete the procedure detailed in the [Fusion 5.8.1 release notes](/docs/5/fusion/release-notes/more/5.8.1-release-notes).
## New Features
* The new [Consumption Dashboard](/docs/5/fusion/operations/monitoring-and-reporting/consumption) tracks your request and document usage against the allotted consumption in your Fusion license. To access the dashboard, log into Fusion and go to **Analytics > Consumption**.
The course for **The Consumption Dashboard** focuses on the purpose and function of this dashboard as well as how to navigate it using the Fusion Admin UI.
## Improvements
### Fusion
* Fusion now supports Kubernetes 1.25 and 1.26. This applies to GKE, AKS, and EKS. It also applies to Rancher (RKE) and OpenShift 4 versions that are compatible with Kubernetes 1.25 and 1.26. Refer to Kubernetes documentation for versions [1.25](https://kubernetes.io/blog/2022/08/04/upcoming-changes-in-kubernetes-1-25/) and [1.26](https://kubernetes.io/blog/2022/11/18/upcoming-changes-in-kubernetes-1-26/) to decide which version to use for your deployment.
* Added a new Admin button to the Rules Editor screen. Click on it to log out of Fusion.
* Improved Spark Solr’s performance to ingest and transport data, as well as setting the most efficient sort for data export.
### Fusion Connectors
* Remote V2 connectors now support Tika Asynchronous Parsing. You can **Use Tika Asynchronous Parsing** on [remote V2 connectors](/docs/fusion-connectors/developers/remote-v2-connectors) to separate document crawling from document parsing, which is useful for large sets of complex documents.
This document describes how to set up your application to use Tika asynchronous parsing.
Unlike synchronous Tika parsing, which uses a parser stage, asynchronous Tika parsing is configured in the datasource and index pipeline. For more information, see [Asynchronous Tika Parsing](/docs/5/fusion/getting-data-in/indexing/asynchronous-tika-parsing).
**Field names change with asynchronous Tika parsing.**
In contrast to synchronous parsing, asynchronous Tika parsing prepends `parser_` to fields added to a document. System fields, which start with `\_lw_`, are not prepended with `parser_`. If you are migrating to asynchronous Tika parsing, and your search application configuration relies on specific field names, update your search application to use the new fields.
## Configure the connectors datasource
1. Navigate to your datasource.
2. Enable the **Advanced** view.
3. Enable the **Async Parsing** option.
**Fusion 5.9.11 and later uses your parser configuration when using asynchronous parsing.**
The asynchronous parsing service performs Tika parsing using Apache Tika Server. In Fusion 5.8 through 5.9.10, other parsers, such as HTML and JSON, are not supported by the asynchronous parsing service. By enabling asynchronous parsing, the parser configuration linked to your datasource is ignored. In Fusion 5.9.11 and later, other parsers, such as HTML and JSON, are supported by the asynchronous parsing service. By enabling asynchronous parsing, the parser configuration linked to your datasource is used.
4. Save the datasource configuration.
## Configure the parser stage
You must do this step in Fusion 5.9.11 and later.
1. Navigate to **Parsers**.
2. Select the parser, or create a new parser.
3. From the **Add a parser stage** menu, select **Apache Tika Container Parser**.
4. (Optional) Enter a label for this stage. This label changes the names from Apache Tika Container Parser to the value you enter in this field.
5. If the Apache Tika Container Parser stage is not already the first stage, drag and drop the stage to the top of the stage list so it is the first stage that runs.
## Configure the index pipeline
1. Go to the **Index Pipeline** screen.
2. Add the **Solr Partial Update Indexer** stage.
3. Turn off the **Reject Update if Solr Document is not Present** option and turn on the **Process All Pipeline Doc Fields** option:
4. Include an extra update field in the stage configuration using any update type and field name. In this example, an incremental field `docs_counter_i` with an increment value of `1` is added:
5. Enable the **Allow reserved fields** option:
6. Click **Save**.
7. Turn off or remove the **Solr Indexer stage**, and move the **Solr Partial Update Indexer stage** to be the last stage in the pipeline.
Asynchronous Tika parsing setup is now complete. Run the datasource indexing job and monitor the results.
* Updated the connectors-sdk library version to Java 11.
* Updated Fusion to prevent automatically updating V2 connector plug-ins. Intentional updates allow you to verify related functions and operations.
* Updated Fusion to specify pod resources for connector plug-in type to ensure optimal capability and performance.
## Other changes
### Babel Street (formerly Basistech) Compatibility
* Fusion 5.9.0 and later can reference a new Solr 9.0/9.1 image that comes bundled with Babel Street JAR files.
### Fusion Connectors
* Fusion 5.9.0 introduces new version requirements for the LDAP ACLs V2 and SharePoint Optimized V2 connectors. The [LDAP ACLs V2 connector](/docs/fusion-connectors/connectors/ad-acl-ldap) must use version 1.5.0 or later, and the [SharePoint Optimized V2 connector](/docs/fusion-connectors/connectors/sharepoint-v2-optimized) must use version 1.6.0 or later.
## Bug Fixes
### Fusion
* Fixed an issue where the parsing endpoint in the indexing service did not parse a DOC file or other Microsoft Word documents. The Apache POI was updated with the `log4j` API.
### Fusion Connectors
* Fixed an issue where clearing the `crawldb` for the Access Control List (ACL) datasource also deletes the ACLs.
* Fixed an issue where the Box.com V2 connector did not start successfully when a Fusion branch was deployed.
## Known issues
* New Kerberos security realms cannot be configured successfully in this version of Fusion.
# Fusion 5.9.1
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9/5.9.1-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.9.1-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9.1-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/ht2sax
Released on September 5, 2023, this [maintenance release](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#maintenance-release-support-policy) includes corrections to remote connectors, the JDBC V2 connector, and LDAP ACL documents.
To learn more, skip to the [release notes](#bug-fixes).
## Platform Support and Component Versions
### Kubernetes platform support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.25 and 1.26
* **Microsoft Azure Kubernetes Service (AKS):** 1.25 and 1.26
* **Amazon Elastic Kubernetes Service (EKS):** 1.25 and 1.26
Support is also offered for Rancher Kubernetes Engine (RKE) and OpenShift 4 versions that are based on Kubernetes 1.25 and 1.26. OpenStack and customized Kubernetes installations are *not* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support).
### Component versions
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| **Solr** | fusion-solr 5.9.1 *(based on Solr 9.1.1)* |
| **ZooKeeper** | 3.7.1 |
| **Spark** | 3.2.2 |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
## Bug Fixes
### Fusion Connectors
* Fixed an issue to make gRPC max outbound message size configurable to prevent remote connectors from failing on large field sizes.
* Fixed an issue where a restart was required when the remote JDBC V2 connector timed out and did not restart correctly.
* Fixed an issue where the JDBC V2 connector failed to index data.
* Fixed an issue where the Connectors SDK delete-by-query that contains a NOT-EQUAL phrase did not delete the appropriate documents.
* Fixed an issue where the outbound access control list (ACL) object contained a reference to itself in the `outbound_ss` field by default. The plug-in that creates the ACL should enter that reference explicitly only when required.
* Fixed an issue where the connectors-backend service did not add the following fields by default to the LDAP ACL documents: `_lw_data_source_type_s`, `_lw_batch_id_s`, `_lw_data_source_collection_s`, and `_lw_block_start_time_l`. These fields are required by the LDAP plug-in to keep the list of ACL objects up-to-date.
## Known issues
* New Kerberos security realms cannot be configured successfully in this version of Fusion.
# Fusion 5.9.10
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9/5.9.10-release-notes
February 20, 2025
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.9.10-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9.10-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/kf2gbd
Fusion 5.9.10 is a [maintenance release](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#maintenance-release-support-policy) that introduces enhanced capabilities for Neural Hybrid Search, that improve search relevance for ecommerce by intelligently blending semantic and lexical signals and optimizing result collapsing.
Support for Kubernetes 1.31 boosts container security and operational reliability, while the Spark 3.4.1 upgrade accelerates data processing and analytics.
Additional security updates and the new Apps Manager API strengthen system protection and give better visibility into licensing and usage.
Collectively, these improvements drive more accurate search, faster data workflows, and a more secure, stable platform—delivering clear business value across search, infrastructure, and data operations.
**Security patch available for api-gateway: Netty request smuggling vulnerabilities**
A patch is available for the `api-gateway` service to address critical Netty request smuggling vulnerabilities (CVE-2026-42581, CVE-2026-42585, CVE-2026-42587). These vulnerabilities allow attackers to smuggle HTTP requests through the gateway, potentially bypassing security controls.
The `api-gateway` service requires the Netty security patch.
Follow these steps to apply the patched image:
1. Open your Fusion Helm values file.
2. Add or update the `api-gateway` image configuration:
```yaml theme={"dark"}
api-gateway:
image:
repository: lucidworks
name: api-gateway
tag: 5.9.10-SUST-1634-patch
imagePullPolicy: IfNotPresent
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace, `RELEASE_NAME` with your Helm release name, and `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the `api-gateway` pods to restart and verify they are using the patched image.
**Urgent action required by November 26, 2025**
A patch is required by November 26, 2025 for all self-hosted Fusion deployments running on Amazon Elastic Kubernetes Service (EKS). Certain Java versions used by Fusion components reach end of life on this date. Failure to apply the patch will result in compatibility issues.
The following Fusion services require the `cgroupv2` patch:
| Service | Affected Fusion versions | Patch tag |
| - | - | - |
| `insights` | 5.9.4 to 5.9.15 | `lucidworks/insights:5.9-cgroupv2-patch` |
| `spark-solr-etl` | 5.9.4 to 5.9.11 | `lucidworks/spark-solr-etl:5.9-cgroupv2-patch` |
| `keytool-utils` | 5.9.4 to 5.9.10 | `lucidworks/keytool-utils:5.9-cgroupv2-patch` |
Follow these steps to apply the patched images:
1. Open your Fusion Helm values file. For Fusion Cloud Native deployments, use the values file for your current deployment. For Helm deployments, use the values file you used to create the deployment.
2. For each service listed in the following table that applies to your Fusion version, add or update the image configuration:
Expand the following code snippet for the complete image configuration list.
```yaml expandable theme={"dark"}
global:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
sql-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
reverse-search:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
solr:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
kafka:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
ml-model-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-admin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-indexing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
query-pipeline:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
async-parsing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
admin-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
api-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
auth-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
classic-rest-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-resources:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-config:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
insights:
image:
imagePullPolicy: Always
name: insights
repository: lucidworks
tag: 5.9-cgroupv2-patch
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-launcher:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-rest-server:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connector-plugin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors-backend:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
rules-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
pm-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
lwai-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
webapps:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
apps-manager:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
templating:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
tikaserver:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
seldon-core-operator:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo-common-workflows:
image:
imagePullPolicy: Always
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
utilitiesTag: 5.9.11
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: 5.9-cgroupv2-patch
argo-common-workflows:
image:
imagePullPolicy: IfNotPresent
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: insights:5.9-cgroupv2-patch
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run the following command:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace. Replace `RELEASE_NAME` with your Helm release name. Replace `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the affected service pods to restart and verify they are using the patched images.
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
For supported Kubernetes versions and key component versions, see [Platform support and component versions](#component-versions).
## What’s new
### Improved relevance with Neural Hybrid Search
Fusion 5.9.10 introduces the Neural Hybrid Query stage, enhancing Neural Hybrid Search (NHS) by refining how semantic and lexical scores are combined in search ranking.
This new stage works with Lucidworks AI to ensure that result collapsing and final ranking use a unified scoring approach, leading to more precise and consistent search results—especially in ecommerce, where selecting the most relevant SKU for a product is critical.
Key benefits include:
* **Smarter relevance ranking:** Combines semantic and lexical signals earlier for more accurate ordering.
* **Optimized result collapsing:** Ensures the best representative item is selected before final ranking.
* **Broad applicability:** Enhances search performance in ecommerce and other scenarios that use result collapsing.
The [Neural Hybrid Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/neural-hybrid-query) differs from the [Hybrid Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/hybrid-search-query) in a few ways. The new stage contains the following new fields:
* **Lexical Query Squash Factor** lets you input a value that squashes the lexical query scores from `0..inf` to `0..1`. This setting helps prevent the lexical query from dominating the final score.
* **Compute Vector Similarity for Lexical-Only Matches** computes vector similarity scores for documents found in lexical search results, but not in the initial vector search results. This setting can rescue orphaned nodes by finding docs which match lexically but are not in the vector results, and compute the vector similarity score for those.
This update provides a more intelligent hybrid search experience, improving relevance across a range of search applications.
### Support for Kubernetes 1.31
Fusion 5.9.10 introduces support for Kubernetes 1.31, bringing enhanced security, improved resource management, and better networking reliability.
This update strengthens container security, improves how custom resources are managed and filtered, and enhances the reliability of `kubectl` operations like `exec` and `port-forward`, especially in complex network environments.
By upgrading to Fusion 5.9.10, you can take full advantage of Kubernetes 1.31’s advancements for stronger security, streamlined resource handling, and improved system stability.
### Expanded support for read-only file system
Fusion 5.9.11 completes support for the read-only root file system feature across all Fusion services, strengthening protection against unauthorized changes and improving security posture in hardened environments.
All services can now be configured to run in read-only mode using Helm values, with writable mounts available for components that require write access.
To learn more, including which services have this feature enabled by default, see
**Enable Read-Only Root File System**.
This topic describes how to configure a read-only root file system for a Fusion deployment.
Read-only mode safeguards your file system against unauthorized changes, such as by malicious software or other attacks.
Enabling this feature entails configuring the `readOnlyRootFilesystem` attribute in the Fusion Helm chart.
This feature is available starting in Fusion 5.9.9 and in all subsequent Fusion 5.9 releases.
Beginning with Fusion 5.9.10, all of [Fusion’s services](/docs/5/fusion/reference/microservices) are designed to work with a read-only root file system, but some external services may require write access to the file system.
In that case, you can configure a read-only root file system and mount a separate writable file system for these services.
The examples below show how to configure a read-only root file system for various Fusion services.
## Solr
```yaml theme={"dark"}
fusion:
solr:
containerSecurityContext:
readOnlyRootFilesystem: true
```
**With optional services:**
```yaml theme={"dark"}
fusion:
solr:
exporter:
enabled: true
securityContext:
readOnlyRootFilesystem: true
containerSecurityContext:
readOnlyRootFilesystem: true
```
## Zookeeper
```yaml theme={"dark"}
fusion:
zookeeper:
containerSecurityContext:
readOnlyRootFilesystem: true
```
**With optional services:**
```yaml theme={"dark"}
fusion:
zookeeper:
containerSecurityContext:
readOnlyRootFilesystem: true
exporters:
jmx:
enabled: true
zookeeper:
enabled: true
jobs:
chroots:
enabled: true
config:
create:
- /root-read-only
```
## Kafka
```yaml theme={"dark"}
fusion:
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
```
**With optional services:**
```yaml expandable theme={"dark"}
fusion:
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
volumePermissions:
enabled: true
containerSecurityContext:
readOnlyRootFilesystem: true
metrics:
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
enabled: true
jmx:
enabled: true
containerSecurityContext:
enabled: true
readOnlyRootFilesystem: true
externalAccess:
enabled: true
autoDiscovery:
enabled: true
containerSecurityContext:
enabled: true
readOnlyRootFilesystem: true
provisioning:
enabled: true
topics:
- name: test
partitions: 1
replicationFactor: 1
## https://kafka.apache.org/documentation/#topicconfigs
config:
max.message.bytes: 64000
flush.messages: 1
containerSecurityContext:
readOnlyRootFilesystem: true
enabled: true
```
## ML Model Service
```yaml expandable theme={"dark"}
fusion:
ml-model-service:
ambassador:
containerSecurityContext:
readOnlyRootFilesystem: true
milvus:
mysql:
containerSecurityContext:
readOnlyRootFilesystem: true
securityContext:
readOnlyRootFilesystem: true
With optional services:
fusion:
ml-model-service:
ambassador:
containerSecurityContext:
readOnlyRootFilesystem: true
prometheusExporter:
enabled: true
securityContext:
readOnlyRootFilesystem: true
milvus:
mysql:
metrics:
enabled: true
securityContext:
readOnlyRootFilesystem: true
containerSecurityContext:
readOnlyRootFilesystem: true
cluster:
enabled: false
mishards:
securityContext:
readOnlyRootFilesystem: true
admin:
enabled: true
securityContext:
readOnlyRootFilesystem: true
securityContext:
readOnlyRootFilesystem: true
```
## Argo
```yaml theme={"dark"}
fusion:
argo:
minio:
containerSecurityContext:
readOnlyRootFilesystem: true
extraVolumes:
- name: config-dir
emptyDir: {}
- name: certs-dir
emptyDir: {}
extraVolumeMounts:
- name: config-dir
mountPath: /.minio
- name: certs-dir
mountPath: /etc/minio/certs
server:
securityContext:
readOnlyRootFilesystem: true
controller:
securityContext:
readOnlyRootFilesystem: true
mainContainer:
securityContext:
readOnlyRootFilesystem: true
executor:
securityContext:
readOnlyRootFilesystem: true
```
## Seldon Core Operator
```yaml theme={"dark"}
fusion:
seldon-core-operator:
containersSecurityContext:
readOnlyRootFilesystem: true
```
## Argo Common Workflows
```yaml theme={"dark"}
fusion:
argo-common-workflows:
containerSecurityContext:
readOnlyRootFilesystem: true
```
## Question Answering
```yaml theme={"dark"}
fusion:
question-answering:
containersSecurityContext:
readOnlyRootFilesystem: true
```
## Classification
```yaml theme={"dark"}
fusion:
classification:
containersSecurityContext:
readOnlyRootFilesystem: true
```
## All-in-one values.yaml example
```yaml expandable theme={"dark"}
fusion:
solr:
containerSecurityContext:
readOnlyRootFilesystem: true
solr-managed:
containerSecurityContext:
readOnlyRootFilesystem: true
zookeeper:
containerSecurityContext:
readOnlyRootFilesystem: true
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
ml-model-service:
ambassador:
containerSecurityContext:
readOnlyRootFilesystem: true
milvus:
mysql:
containerSecurityContext:
readOnlyRootFilesystem: true
securityContext:
readOnlyRootFilesystem: true
argo:
minio:
containerSecurityContext:
readOnlyRootFilesystem: true
extraVolumes:
- name: config-dir
emptyDir: {}
- name: certs-dir
emptyDir: {}
extraVolumeMounts:
- name: config-dir
mountPath: /.minio
- name: certs-dir
mountPath: /etc/minio/certs
server:
securityContext:
readOnlyRootFilesystem: true
controller:
securityContext:
readOnlyRootFilesystem: true
mainContainer:
securityContext:
readOnlyRootFilesystem: true
executor:
securityContext:
readOnlyRootFilesystem: true
seldon-core-operator:
containersSecurityContext:
readOnlyRootFilesystem: true
argo-common-workflows:
containerSecurityContext:
readOnlyRootFilesystem: true
question-answering:
containersSecurityContext:
readOnlyRootFilesystem: true
classification:
containersSecurityContext:
readOnlyRootFilesystem: true
```
## All-in-one values.yaml with optional services example
```yaml expandable theme={"dark"}
fusion:
argo:
minio:
persistence:
enabled: false
containerSecurityContext:
readOnlyRootFilesystem: true
extraVolumes:
- name: config-dir
emptyDir: {}
- name: certs-dir
emptyDir: {}
extraVolumeMounts:
- name: config-dir
mountPath: /.minio
- name: certs-dir
mountPath: /etc/minio/certs
server:
securityContext:
readOnlyRootFilesystem: true
controller:
securityContext:
readOnlyRootFilesystem: true
mainContainer:
securityContext:
readOnlyRootFilesystem: true
executor:
securityContext:
readOnlyRootFilesystem: true
solr:
exporter:
enabled: true
securityContext:
readOnlyRootFilesystem: true
containerSecurityContext:
readOnlyRootFilesystem: true
# tls:
# enabled: true
solr-managed:
exporter:
enabled: true
securityContext:
readOnlyRootFilesystem: true
enableExternalFiles: true
enabledStorage:
- gcs
processRaw:
image:
repository: fusion-dev-docker.ci-artifactory.lucidworks.com
cloudRoot: gs://lw-managed-fusion-data/tmp
gcs:
secret: gcs-key
secretFieldName: key.json
securityContext:
readOnlyRootFilesystem: true
containerSecurityContext:
readOnlyRootFilesystem: true
# tls:
# enabled: true
zookeeper:
containerSecurityContext:
readOnlyRootFilesystem: true
exporters:
jmx:
enabled: true
zookeeper:
enabled: true
jobs:
chroots:
# enabled: true
config:
create:
- /root-read-only
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
volumePermissions:
enabled: true
image:
repository: "fusion-dev-docker.ci-artifactory.lucidworks.com/os-shell"
containerSecurityContext:
readOnlyRootFilesystem: true
metrics:
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
enabled: true
jmx:
enabled: true
containerSecurityContext:
enabled: true
readOnlyRootFilesystem: true
externalAccess:
enabled: true
autoDiscovery:
enabled: true
containerSecurityContext:
enabled: true
readOnlyRootFilesystem: true
provisioning:
enabled: true
topics:
- name: test
partitions: 1
replicationFactor: 1
## https://kafka.apache.org/documentation/#topicconfigs
config:
max.message.bytes: 64000
flush.messages: 1
containerSecurityContext:
readOnlyRootFilesystem: true
enabled: true
ml-model-service:
enabled: true
ambassador:
containerSecurityContext:
readOnlyRootFilesystem: true
prometheusExporter:
enabled: true
securityContext:
readOnlyRootFilesystem: true
milvus:
mysql:
metrics:
enabled: true
securityContext:
readOnlyRootFilesystem: true
containerSecurityContext:
readOnlyRootFilesystem: true
cluster:
enabled: false
mishards:
securityContext:
readOnlyRootFilesystem: true
admin:
enabled: true
securityContext:
readOnlyRootFilesystem: true
securityContext:
readOnlyRootFilesystem: true
seldon-core-operator:
containersSecurityContext:
readOnlyRootFilesystem: true
argo-common-workflows:
containerSecurityContext:
readOnlyRootFilesystem: true
question-answering:
containersSecurityContext:
readOnlyRootFilesystem: true
classification:
containersSecurityContext:
readOnlyRootFilesystem: true
```
## Supported services
The table below lists the services that support a read-only root file system, the ones that have it enabled by default, and the Fusion release in which support was added:
| Chart Name | Pod Name | Container Name | Supported | Default enabled | Supported version |
| - | - | - | - | - | - |
| admin-ui | admin-ui | admin-ui | ✅ | ✅ | 5.9.9+ |
| api-gateway | api-gateway | init/api-gateway | ✅ | ✅ | 5.9.9+ |
| api-gateway | api-gateway | api-gateway | ✅ | ✅ | 5.9.9+ |
| api-gateway | api-gateway | generate-jks | ✅ | ✅ | 5.9.9+ |
| apps-manager | apps-manager | apps-manager | ✅ | ✅ | 5.9.9+ |
| argo | argo-server | argo-server | ✅ | ❌ | 5.9.10+ |
| argo | argo-executor | executor | ✅ | ❌ | 5.9.10+ |
| argo | argo-mainContainer | mainContainer | ✅ | ❌ | 5.9.10+ |
| argo | argo-controller | controller | ✅ | ✅ | 5.9.10+ |
| argo/minio | minio | minio | ✅ | ❌ | 5.9.10+ |
| argo/minio | minio | minio | ✅ | ❌ | 5.9.10+ |
| argo/minio | make-bucket-job | minio-mc | ✅ | ❌ | 5.9.10+ |
| argo-common-workflows | delete-model | init/main/wait | ✅ | ❌ | 5.9.10+ |
| argo-common-workflows | deploy-model | init/main/wait | ✅ | ❌ | 5.9.10+ |
| argo-common-workflows | milvus-maintenance | init/main/wait | ✅ | ❌ | 5.9.10+ |
| argo-common-workflows | upload-model-to-cloud | init/main/wait | ✅ | ❌ | 5.9.10+ |
| async-parsing | async-parsing | tika-server | ✅ | ✅ | 5.9.9+ |
| async-parsing | async-parsing | async-parsing | ✅ | ✅ | 5.9.9+ |
| auth-ui | auth-ui | auth-ui | ✅ | ✅ | 5.9.9+ |
| classic-rest-service | classic-rest-service | init/import-certs | ✅ | ✅ | 5.9.10+ |
| classic-rest-service | classic-rest-service | classic-rest-service | ✅ | ✅ | 5.9.9+ |
| classification | argo/classification | init/wait/main | ✅ | ❌ | 5.9.10+ |
| connector-plugin | connector-plugin | init/import-certs | ✅ | ❌ | 5.9.10+ |
| connector-plugin | connector-plugin | connector-plugin | ✅ | ✅ | 5.9.9+ |
| connectors | connectors | connectors | ✅ | ✅ | 5.9.9+ |
| connectors-backend | CRD | | ✅ | ✅ | 5.9.9+ |
| connectors-backend | connectors-backend | connectors-backend | ✅ | ✅ | 5.9.9+ |
| fusion-admin | fusion-admin | admin | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-admin | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-api-gateway | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-indexing | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-kafka | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-logstash | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-pulsar | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | setup-keystore-and-properties | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-zk | ✅ | ✅ | 5.9.9+ |
| fusion-config-sync | fusion-config-sync | fusion-config-sync | ✅ | ✅ | 5.9.9+ |
| fusion-data-augmentation | argo/data-augmentation/volume-fix | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/init-workspace | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/write-job-configs | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/write-io-configs | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/add-zkhost | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/pull-data-training-and-metadata | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/pull-data-training-and-metadata-cloud | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/volume-fix2 | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/synonym-list | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/download-synonym-dictionary | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/keystroke-list | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/download-keystroke-blob | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/augment | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/push-augmented-data | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/push-augmented-data-cloud | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-indexing | fusion-indexing | fusion-indexing | ✅ | ✅ | 5.9.9+ |
| fusion-resources | fusion-resources-secret-hook | setup-keystore-and-properties | ✅ | ✅ | 5.9.10+ |
| insights | insights | insights | ✅ | ✅ | 5.9.9+ |
| job-launcher | job-launcher | job-launcher | ✅ | ✅ | 5.9.9+ |
| job-launcher | job-launcher | kubectl-runner | ✅ | ✅ | 5.9.9+ |
| job-launcher | job-launcher-spark-cleanup | kubectl-runner | ✅ | ✅ | 5.9.9+ |
| job-launcher | spark-kubernetes-driver | spark-kubernetes-driver | ✅ | ❌ | 5.9.10+ |
| job-launcher | spark-kubernetes-executor | spark-kubernetes-executor | ✅ | ❌ | 5.9.10+ |
| job-rest-server | job-rest-server | job-rest-server | ✅ | ✅ | 5.9.9+ |
| kafka | kafka-metrics | kafka-exporter-archived | ✅ | ❌ | 5.9.10+ |
| kafka | kafka-provisioning | init/wait-for-available-kafka | ✅ | ❌ | 5.9.10+ |
| kafka | kafka-provisioning | kafka-provisioning | ✅ | ❌ | 5.9.10+ |
| kafka | kafka | kafka | ✅ | ❌ | 5.9.10+ |
| kafka | kafka | jmx-exporter | ✅ | ❌ | 5.9.10+ |
| kafka | kafka | init/check-zk | ✅ | ✅ | 5.9.9+ |
| kafka | kafka | init/auto-discovery | ✅ | ❌ | 5.9.10+ |
| kafka | kafka | init/volume-permissions | ✅ | ❌ | 5.9.10+ |
| lwai-gateway | lwai-gateway | lwai-gateway | ✅ | ✅ | 5.9.9+ |
| ml-model-service | ml-model-service | java-service | ✅ | ✅ | 5.9.9+ |
| ml-model-service | ml-model-service-namespace-hook | kubectl-runner | ✅ | ✅ | 5.9.9+ |
| ml-model-service/ambassador | ambassador | ambassador | ✅ | ❌ | 5.9.10+ |
| ml-model-service/ambassador | ambassador | prometheus-exporter | ✅ | ❌ | 5.9.10+ |
| ml-model-service/milvus | milvus-writable | milvus | ✅ | ❌ | 5.9.10+ |
| ml-model-service/milvus | milvus-writable | init/wait-for-mysql | ✅ | ✅ | 5.9.10+ |
| ml-model-service/milvus | milvus-writable | init/create-for-share-storage | ✅ | ❌ | 5.9.10+ |
| ml-model-service/milvus | milvus-admin | admin | ✅ | ❌ | 5.9.10+ |
| ml-model-service/milvus | milvus-admin | init/wait-for-milvus | ✅ | ✅ | 5.9.10+ |
| ml-model-service/milvus | milvus-mishards | init/wait-for-mysql | ✅ | ✅ | 5.9.10+ |
| ml-model-service/milvus | milvus-mishards | init/wait-for-mysql | ✅ | ✅ | 5.9.10+ |
| ml-model-service/milvus | milvus-mishards | mishards | ✅ | ❌ | 5.9.10+ |
| ml-model-service/milvus/mysql | mysql | init/remove-lost-found | ✅ | ✅ | 5.9.10+ |
| ml-model-service/milvus/mysql | mysql | mysql | ✅ | ❌ | 5.9.10+ |
| pm-ui | pm-ui | pm-ui | ✅ | ✅ | 5.9.9+ |
| query-pipeline | query-pipeline | query-pipeline | ✅ | ✅ | 5.9.9+ |
| question-answering | argo/qna-coldstart/init-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/write-job-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/add-zkHost | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/pull-data | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/pull-data-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/train | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/list-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/post-model | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/apply-seldon-deployment | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/init-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/write-job-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/add-zkhost | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/pull-qa-data-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/pull-qa-data | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/train | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/train-with-texts | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/list-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/post-model | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/apply-seldon-deployment | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/init-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/write-job-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/add-zkhost | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/list-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/pull-eval-data | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/pull-eval-data-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/evaluate | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/push-eval-results-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/push-eval-results | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/init-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/write-job-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/add-zkhost | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/pull-data-training-and-metadata | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/pull-data-training-and-metadata-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/train-with-metadata | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/train-without-metadata | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/push-recommendations | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/push-recommendations-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/add-default-exclude-query | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/init-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/copy-model | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/write-job-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/add-zkhost | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/pull-data | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/pull-data-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/train | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/push-content | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/push-content-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/add-default-exclude-query | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/delete-old-content-recommendations | init/wait/main | ✅ | ❌ | 5.9.10+ |
| reverse-search | reverse-search | init/set-reverse-search-zone | ✅ | ✅ | 5.9.10+ |
| reverse-search | reverse-search | init/check-zk | ✅ | ✅ | 5.9.9+ |
| reverse-search | reverse-search | init/enable-tls-in-reverse-search | ✅ | ✅ | 5.9.10+ |
| reverse-search | reverse-search | reverse-search | ✅ | ❌ | 5.9.10+ |
| rules-ui | rules-ui | rules-ui | ✅ | ✅ | 5.9.9+ |
| seldon-core-operator | seldon-controller-manager | manager | ✅ | ❌ | 5.9.10+ |
| seldon-core-operator | seldon-spartakus-volunteer | seldon-spartakus-volunteer | ✅ | ❌ | 5.9.9+ |
| seldon-core-operator | crd/SeldonDeployment | | ✅/❌ | ❌ | 5.9.9+ |
| solr | solr | init/set-solr-zone | ✅ | ✅ | 5.9.10+ |
| solr | solr | init/enable-tls-in-solr | ✅ | ✅ | 5.9.10+ |
| solr | configset-bootstrap | configset-bootstrap | ✅ | ✅ | 5.9.10+ |
| solr | solr | solr | ✅ | ❌ | 5.9.10+ |
| solr | solr-exporter | exporter | ✅ | ❌ | 5.9.10+ |
| solr | solr-exporter | init/solr-init | ✅ | ✅ | 5.9.10+ |
| solr-managed | configset-bootstrap | configset-bootstrap | ✅ | ✅ | 5.9.10+ |
| solr-managed | exporter | exporter | ✅ | ❌ | 5.9.10+ |
| solr-managed | exporter | init/solr-init | ✅ | ✅ | 5.9.10+ |
| solr-managed | solr | init/set-solr-zone | ✅ | ✅ | 5.9.10+ |
| solr-managed | solr | init/enable-tls-in-solr | ✅ | ✅ | 5.9.10+ |
| solr-managed | solr | solr | ✅ | ❌ | 5.9.10+ |
| solr-managed | solr | process-raw | ✅ | ❌ | 5.9.10+ |
| solr-backup-runner | solr-backup-runner-backup | solr-backups | ✅ | ❌ | 5.9.10+ |
| solr-backup-runner | solr-backup-runner-prune | solr-prune | ✅ | ❌ | 5.9.10+ |
| templating | templating | templating | ✅ | ✅ | 5.9.9+ |
| webapps | webapps | webapps | ✅ | ✅ | 5.9.9+ |
| zookeeper | zookeeper | zookeeper | ✅ | ❌ | 5.9.10+ |
| zookeeper | zookeeper | jmx-exporter | ✅ | ❌ | 5.9.10+ |
| zookeeper | zookeeper | zookeeper-exporter | ✅ | ❌ | 5.9.10+ |
| zookeeper | zookeeper-chroots | main | ✅ | ❌ | 5.9.10+ |
### Faster, more efficient data processing with Spark 3.4.1
Fusion 5.9.10 upgrades Apache Spark to 3.4.1, bringing faster query execution, improved data transformation efficiency, and greater stability for distributed workloads.
This enhancement optimizes indexing, refines SQL query handling, and ensures smoother analytics workflows, enabling you to process large-scale data with greater speed and precision.
For more details, see the [Spark 3.4.1 release notes](https://spark.apache.org/releases/spark-release-3-4-1.html).
The Apache Spark 3.4.1 upgrade impacts jobs that use Python 3.7 behavior or compatibility, which may have automatically updated to Python 3.10.x and no longer function correctly. Update your code to ensure compatibility with Python 3.10.x and then test your Spark jobs in a staging environment before deploying to production.
### Enhanced security and stability
Fusion 5.9.10 introduces a new wave of security enhancements, ensuring a more resilient and up-to-date platform. This release includes critical updates across core services, including admin, frameworks, apps manager, classic connectors, and query indexing, reinforcing protection across the stack. Additionally, we’ve updated the `bitnami-shell` base image and upgraded `key-tools` to v3.0.2, further strengthening security and compliance.
These enhancements help maintain a robust and secure Fusion environment, keeping your data and infrastructure protected while optimizing performance for mission-critical workloads.
### Apps Manager API
The new [Apps Manager API](/api-reference/license/get-the-fusion-license) gives information about your Fusion license, entitlements, and usage.
The quick learning for **Apps Manager API** focuses on the purpose and functions of the Apps Manager API.
## Bug fixes
* `Solr-exporter` pods no longer get stuck in an `ImagePullBackOff` state, ensuring they pull the correct image and start reliably.
* The `job-launcher` and `job-rest-server` services now start correctly in SSL mode, resolving an issue where missing dependencies caused failures during initialization.
* Fusion now returns all matching search rules and rewrites in Commerce Studio instead of just the first ten, ensuring complete rule retrieval and better compatibility between the two systems.
* Prometheus stage execution histograms and counters now include stage labels, making it easier to organize dashboards and interpret stage metrics without relying on auto-generated IDs.
* Resolved an issue in Fusion 5.9.4 where v2 connectors failed to start in certain self-hosted EKS environments, preventing timeouts and ensuring successful job execution.
* Increased the request buffer size in `lwai-gateway` from 250 KB to 5 MB, allowing large messages to be processed without failures.
## Known issues
* Saving large query pipelines may cause OOM failures under high load.
In Fusion versions 5.9.10 through 5.9.13, saving a large query pipeline during high query volume can result in thread lock, excessive memory use, and eventual OOM errors in the Query service.\
This issue is fixed in Fusion 5.9.14.
* Deploying Fusion with TLS flags enabled fails in ArgoCD due to Helm chart rendering limitations.\
When using ArgoCD to deploy Fusion 5.9.10 or 5.9.11 with TLS options enabled, Helm chart rendering fails due to the use of the `lookup` function, which is unsupported by ArgoCD. This prevents ArgoCD from generating manifests, blocking deployment workflows that rely on TLS configuration.\
As a workaround, deploy Fusion without enabling TLS in ArgoCD-managed environments, or perform the deployment using Helm directly.\
This issue is fixed in Fusion 5.9.12.
* An issue prevents segment-based rule filtering from working correctly in Commerce Studio. This issue is fixed in Fusion 5.9.12.
## Deprecations and removals
For full details on deprecations and removals, see [Deprecations and Removals](/docs/5/fusion/deprecations-and-removals).
### Bitnami removal
Fusion 5.9.10 will be re-released with the same functionality but updated image references.
In the meantime, Lucidworks will self-host the required images while we work to replace Bitnami images with internally built open-source alternatives.
If you are a self-hosted Fusion customer, *you must upgrade before August 28* to ensure continued access to container images and prevent deployment issues.
You can reinstall your current version of Fusion or upgrade to Fusion 5.9.14, which includes the updated Helm chart and prepares your environment for long-term compatibility.
See [Prevent image pull failures due to Bitnami deprecation in Fusion 5.9.5 to 5.9.13](https://support.lucidworks.com/hc/en-us/articles/33966125467799-Prevent-image-pull-failures-due-to-Bitnami-deprecation-in-Fusion-5-9-5-to-5-9-13) for more information on how to prevent image pull failures.
### Webapps service
Fusion has deprecated the Webapps service.
In previous versions of Fusion, you could use this service to deploy an App Studio WAR file into Fusion.
As of Fusion 5.9.10, you should instead **Deploy App Studio Enterprise to a Fusion 5 Cluster (GKE)**.
This article explains how to deploy [App Studio Enterprise](/docs/5/fusion/dev-portal/appkit/concepts/app-studio-enterprise/overview) (ASE) to an existing Fusion 5 cluster in Google Kubernetes Engine (GKE) without using the Webapps service.
Before completing this guide, ensure that your `fusion.conf` file points to the IP or URI and port of the proxy service.
Run the App Studio Enterprise application locally and verify functioning security and search features with the cluster you are deploying to.
## Prepare the package
1. Package your app into a single JAR file:
```
./app-studio package
```
2. App Studio Enterprise includes a `dockerfile`. Create the App Studio Enterprise Docker image:
```
docker build PATH -t APP_NAME
```
Set or replace `APP_NAME` with the name of your application. Replace `PATH` with the path to build from.
3. You can test the Docker image locally with the following command:
```
docker run -it -p LOCAL_PORT:8080 APP_NAME
```
Set or replace `LOCAL_PORT` with the port on your local machine that can access the app. Replace `APP_NAME` with the ASE application name.
## Publish the image
You can publish the Docker image anywhere that supports Docker images. This section explains how to publish a Docker image to Google Kubernetes Engine.
1. Tag your container for the registry:
```
docker tag APP_NAME gcr.io/PROJECT_NAME/APP_NAME
```
2. Push your Docker image to the Google Container Registry:
```
docker push gcr.io/PROJECT_NAME/APP_NAME
```
3. Verify the image:
```
gcloud compute instances list
```
## Deploy the app to a cluster
After publishing your ASE Docker image, deploy the image to a cluster. Your existing Fusion 5 cluster is a good choice.
1. Switch the context to your Fusion 5 cluster:
```
gcloud container clusters get-credentials CLUSTER_NAME
```
Replace `CLUSTER_NAME` with your existing Fusion 5 cluster’s name.
2. Create a deployment in your cluster using the image you published:
```
kubectl create deployment APP_NAME --image=gcr.io/PROJECT_NAME/APP_NAME:latest
```
## Create an Ingress resource
After deploying the app, create an Ingress resource for your ASE instance. If you require more Ingress rules, your Ingress resource will look different. Learn more about [Ingress resources](https://kubernetes.io/docs/concepts/services-networking/ingress/).
1. Use the following command to create a minimal Ingress resource:
```
cat <
### Forked Apache Tika Parser
The Forked Apache Tika parser stage has been removed. Use [asynchronous Tika parsing](/docs/5/fusion/getting-data-in/indexing/asynchronous-tika-parsing) instead.
## Upgrade and platform support details
Upgrading to the latest version of Fusion 5.9 offers several key benefits:
* **Access to latest features**: Stay current with the latest features and functionality to ensure compatibility and optimal performance.
* **Simplified process**: Fusion 5.9.5 and later use an in-place upgrade strategy, making upgrades easier than ever.
* **Extended support**: Upgrading keeps you up-to-date with the latest supported Kubernetes versions, as outlined in the [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle) policy.
Check out the **Fusion 5 Upgrades** for details.
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
### Kubernetes platform support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.28, 1.29, 1.30, 1.31
* **Microsoft Azure Kubernetes Service (AKS):** 1.28, 1.29, 1.30, 1.31
* **Amazon Elastic Kubernetes Service (EKS):** 1.28, 1.29, 1.30, 1.31
Support is also offered for Rancher Kubernetes Engine (RKE) and OpenShift 4 versions that are based on Kubernetes 1.28, 1.29, 1.30, 1.31. OpenStack and customized Kubernetes installations are *not* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support).
### Component versions
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| **Solr** | fusion-solr 5.9.10 *(based on Solr 9.6.1)* |
| **ZooKeeper** | 3.9.1 |
| **Spark** | 3.4.1 |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
**More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).**
# Fusion 5.9.11
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9/5.9.11-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.9.11-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9.11-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/z72wlm
Released on March 20, 2025, this [maintenance release](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#maintenance-release-support-policy) provides KNN-based performance enhancements for Neural Hybrid Search, Java 17 support for connectors, a new asynchronous parsing service, the Apps Manager API, and critical security updates.
Upgrading to the latest version of Fusion 5.9 offers several key benefits:
* **Access to latest features:** Stay current with the latest features and functionality to ensure compatibility and optimal performance.
* **Simplified process:** Fusion 5.9.5 and later use an in-place upgrade strategy, making upgrades easier than ever.
* **Extended support:** Upgrading keeps you up-to-date with the latest supported Kubernetes versions, as outlined in the [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle) policy.
**Security patch available for api-gateway: Netty request smuggling vulnerabilities**
A patch is available for the `api-gateway` service to address critical Netty request smuggling vulnerabilities (CVE-2026-42581, CVE-2026-42585, CVE-2026-42587). These vulnerabilities allow attackers to smuggle HTTP requests through the gateway, potentially bypassing security controls.
The `api-gateway` service requires the Netty security patch.
Follow these steps to apply the patched image:
1. Open your Fusion Helm values file.
2. Add or update the `api-gateway` image configuration:
```yaml theme={"dark"}
api-gateway:
image:
repository: lucidworks
name: api-gateway
tag: 5.9.11-SUST-1634-patch
imagePullPolicy: IfNotPresent
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace, `RELEASE_NAME` with your Helm release name, and `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the `api-gateway` pods to restart and verify they are using the patched image.
**Urgent action required by November 26, 2025**
A patch is required by November 26, 2025 for all self-hosted Fusion deployments running on Amazon Elastic Kubernetes Service (EKS). Certain Java versions used by Fusion components reach end of life on this date. Failure to apply the patch will result in compatibility issues.
The following Fusion services require the `cgroupv2` patch:
| Service | Affected Fusion versions | Patch tag |
| - | - | - |
| `insights` | 5.9.4 to 5.9.15 | `lucidworks/insights:5.9-cgroupv2-patch` |
| `spark-solr-etl` | 5.9.4 to 5.9.11 | `lucidworks/spark-solr-etl:5.9-cgroupv2-patch` |
| `keytool-utils` | 5.9.4 to 5.9.10 | `lucidworks/keytool-utils:5.9-cgroupv2-patch` |
Follow these steps to apply the patched images:
1. Open your Fusion Helm values file. For Fusion Cloud Native deployments, use the values file for your current deployment. For Helm deployments, use the values file you used to create the deployment.
2. For each service listed in the following table that applies to your Fusion version, add or update the image configuration:
Expand the following code snippet for the complete image configuration list.
```yaml expandable theme={"dark"}
global:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
sql-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
reverse-search:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
solr:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
kafka:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
ml-model-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-admin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-indexing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
query-pipeline:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
async-parsing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
admin-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
api-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
auth-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
classic-rest-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-resources:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-config:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
insights:
image:
imagePullPolicy: Always
name: insights
repository: lucidworks
tag: 5.9-cgroupv2-patch
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-launcher:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-rest-server:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connector-plugin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors-backend:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
rules-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
pm-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
lwai-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
webapps:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
apps-manager:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
templating:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
tikaserver:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
seldon-core-operator:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo-common-workflows:
image:
imagePullPolicy: Always
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
utilitiesTag: 5.9.11
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: 5.9-cgroupv2-patch
argo-common-workflows:
image:
imagePullPolicy: IfNotPresent
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: insights:5.9-cgroupv2-patch
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run the following command:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace. Replace `RELEASE_NAME` with your Helm release name. Replace `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the affected service pods to restart and verify they are using the patched images.
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
For supported Kubernetes versions and key component versions, see [Platform support and component versions](#platform-support-and-component-versions).
## Key highlights
### KNN performance enhancements
The [Neural Hybrid Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/neural-hybrid-query) now uses K-Nearest Neighbors (KNN) instead of vector similarity (vecSim).
KNN is a more efficient method for finding the most relevant results, leading to faster and more accurate searches that help users find the most relevant information faster.
New configuration options give you finer control over the speed and accuracy of neural hybrid queries:
* `vectorDepth` (Number of Vector Results) sets the number of vector results to return from the vector portion of the hybrid query.
Increasing `vectorDepth` retrieves more vector results but may increase query time.
Lowering it speeds up search but may reduce result diversity.
* `vecPreFilterBoolean` (Block pre-filtering) indicates whether to prevent pre-filtering.
Pre-filtering can improve performance, while blocking it can yield more accurate facet counts and search results.
### Java 17 for connectors
Connectors have been updated to use Java 17. If you use remote connectors, you must upgrade to JVM 17. See **Configure Remote V2 Connectors** for more information.
If you need to index data from behind a firewall, you can configure a V2 connector to run remotely on-premises using TLS-enabled gRPC.
## Prerequisites
Before you can set up an on-prem V2 connector, you must configure the egress from your network to allow HTTP/2 communication into the Fusion cloud. You can use a [forward proxy server](#egress-and-proxy-server-configuration) to act as an intermediary between the connector and Fusion.
The following is required to run V2 connectors remotely:
* The [plugin zip file and the connector-plugin-standalone JAR](https://plugins.lucidworks.com/).
* A configured connector backend gRPC endpoint.
* Username and password of a user with a `remote-connectors` or `admin` role.
* If the host where the remote connector is running is not configured to trust the server’s TLS certificate, you must configure the file path of the trust certificate collection.
If your version of Fusion doesn’t have the `remote-connectors` role by default, you can create one. No API or UI permissions are required for the role.
## Connector compatibility
Only V2 connectors are able to run remotely on-premises.
You also need the remote connector client JAR file that matches your Fusion version.
You can download the latest files at [V2 Connectors Downloads](/docs/fusion-connectors/downloads/v2-connectors-downloads).
Whenever you upgrade Fusion, you must also update your remote connectors to match the new version of Fusion.
The gRPC connector backend is not supported in Fusion environments deployed on AWS.
## System requirements
The following is required for the on-prem host of the remote connector:
* (Fusion 5.9.0-5.9.10) JVM version 11
* (Fusion 5.9.11) JVM version 17
* Minimum of 2 CPUs
* 4GB Memory
Note that memory requirements depend on the number and size of ingested documents.
## Enable backend ingress
In your `values.yaml` file, configure this section as needed:
```yaml theme={"dark"}
ingress:
enabled: false
pathtype: "Prefix"
path: "/"
#host: "ingress.example.com"
ingressClassName: "nginx" # Fusion 5.9.6 only
tls:
enabled: false
certificateArn: ""
# Enable the annotations field to override the default annotations
#annotations: ""
```
* Set `enabled` to `true` to enable the backend ingress.
* Set `pathtype` to `Prefix` or `Exact`.
* Set `path` to the path where the backend will be available.
* Set `host` to the host where the backend will be available.
* In Fusion 5.9.6 *only*, you can set `ingressClassName` to one of the following:
* `nginx` for Nginx Ingress Controller
* `alb` for AWS Application Load Balancer (ALB)
* Configure TLS and certificates according to your CA’s procedures and policies.
TLS must be enabled in order to use AWS ALB for ingress.
## Connector configuration example
```yaml theme={"dark"}
kafka-bridge:
target: mynamespace-connectors-backend.lucidworkstest.com:443 # mandatory
plain-text: false # optional, false by default.
proxy-server: # optional - needed when a forward proxy server is used to provide outbound access to the standalone connector
host: host
port: some-port
user: user # optional
password: password # optional
trust: # optional - needed when the client's system doesn't trust the server's certificate
cert-collection-filepath: path1
proxy: # mandatory fusion-proxy
user: admin
password: password123
url: https://fusiontest.com/ # needed only when the connector plugin requires blob store access
plugin: # mandatory
path: ./fs.zip
type: #optional - the suffix is added to the connector id
suffix: remote
```
### Minimal example
```yaml theme={"dark"}
kafka-bridge:
target: mynamespace-connectors-backend.lucidworkstest.com:443
proxy:
user: admin
password: "password123"
plugin:
path: ./testplugin.zip
```
### Logback XML configuration file example
```xml theme={"dark"}
%d - %-5p [%t:%C{3.}@%L] - %m{nolookups}%nutf8${LOGDIR:-.}/connector.log${LOGDIR:-.}/connector-%d{yyyy-MM-dd}.%i.log.gz50MB10GB%d - %-5p [%t:%C{3.}@%L] - %m{nolookups}%nutf8
```
## Run the remote connector
```java theme={"dark"}
java [-Dlogging.config=[LOGBACK_XML_FILE]] \
-jar connector-plugin-client-standalone.jar [YAML_CONFIG_FILE]
```
The `logging.config` property is optional. If not set, logging messages are sent to the console.
## Test communication
You can run the connector in communication testing mode. This mode tests the communication with the backend without running the plugin, reports the result, and exits.
```java theme={"dark"}
java -Dstandalone.connector.connectivity.test=true -jar connector-plugin-client-standalone.jar [YAML_CONFIG_FILE]
```
## Encryption
In a deployment, communication to the connector’s backend server is encrypted using TLS. You should only run this configuration without TLS in a testing scenario. To disable TLS, set `plain-text` to `true`.
## Egress and proxy server configuration
One of the methods you can use to allow outbound communication from behind a firewall is a proxy server. You can configure a proxy server to allow certain communication traffic while blocking unauthorized communication. If you use a proxy server at the site where the connector is running, you must configure the following properties:
* **Host.** The hosts where the proxy server is running.
* **Port.** The port the proxy server is listening to for communication requests.
* **Credentials.** Optional proxy server user and password.
When you configure egress, it is important to disable any connection or activity timeouts because the connector uses long running gRPC calls.
## Password encryption
If you use a login name and password in your configuration, run the following utility to encrypt the password:
1. Enter a user name and password in the connector configuration YAML.
2. Run the standalone JAR with this property:
```java theme={"dark"}
-Dstandalone.connector.encrypt.password=true
```
3. Retrieve the encrypted passwords from the log that is created.
4. Replace the clear password in the configuration YAML with the encrypted password.
## Connector restart (5.7 and earlier)
The connector will shut down automatically whenever the connection to the server is disrupted, to prevent it from getting into a bad state. Communication disruption can happen, for example, when the server running in the `connectors-backend` pod shuts down and is replaced by a new pod. Once the connector shuts down, connector configuration and job execution are disabled. To prevent that from happening, you should restart the connector as soon as possible.
You can use Linux scripts and utilities to restart the connector automatically, such as [Monit](https://mmonit.com/monit/).
## Recoverable bridge (5.8 and later)
If communication to the remote connector is disrupted, the connector will try to recover communication and gRPC calls. By default, six attempts will be made to recover each gRPC call. The number of attempts can be configured with the `max-grpc-retries` bridge parameters.
## Job expiration duration (5.9.5 only)
The timeout value for irresponsive backend jobs can be configured with the `job-expiration-duration-seconds` parameter. The default value is `120` seconds.
## Use the remote connector
Once the connector is running, it is available in the Datasources dropdown. If the standalone connector terminates, it disappears from the list of available connectors. Once it is re-run, it is available again and configured connector instances will not get lost.
## Enable asynchronous parsing (5.9 and later)
To separate document crawling from document parsing, enable Tika Asynchronous Parsing on remote V2 connectors.
### Asynchronous parsing service
An asynchronous parsing service for connectors has been added.
While traditional synchronous parsing can create delays in indexing when handling large documents or high data volumes, asynchronous parsing processes files in the background, allowing indexing to continue without waiting for each document to be fully parsed.
This new service brings more efficient data processing, improved search freshness, scalability without added complexity, and better support for diverse data types, supporting HTML, JSON, and other formats.
A new parsing stage, [Apache Tika Container](/docs/5/fusion/reference/config-ref/parser-stages/apache-tika-container), has been added to route asynchronous parsing through the new service.
This stage is now required when using asynchronous parsing for connectors.
To use asynchronous parsing for connectors, be sure that ***Async Parsing*** is checked in the datasource and the Apache Tika Container parser stage is enabled in the index pipeline.
See **Use Tika Asynchronous Parsing** for detailed steps to set up asynchronous parsing.
This document describes how to set up your application to use Tika asynchronous parsing.
Unlike synchronous Tika parsing, which uses a parser stage, asynchronous Tika parsing is configured in the datasource and index pipeline. For more information, see [Asynchronous Tika Parsing](/docs/5/fusion/getting-data-in/indexing/asynchronous-tika-parsing).
**Field names change with asynchronous Tika parsing.**
In contrast to synchronous parsing, asynchronous Tika parsing prepends `parser_` to fields added to a document. System fields, which start with `\_lw_`, are not prepended with `parser_`. If you are migrating to asynchronous Tika parsing, and your search application configuration relies on specific field names, update your search application to use the new fields.
## Configure the connectors datasource
1. Navigate to your datasource.
2. Enable the **Advanced** view.
3. Enable the **Async Parsing** option.
**Fusion 5.9.11 and later uses your parser configuration when using asynchronous parsing.**
The asynchronous parsing service performs Tika parsing using Apache Tika Server. In Fusion 5.8 through 5.9.10, other parsers, such as HTML and JSON, are not supported by the asynchronous parsing service. By enabling asynchronous parsing, the parser configuration linked to your datasource is ignored. In Fusion 5.9.11 and later, other parsers, such as HTML and JSON, are supported by the asynchronous parsing service. By enabling asynchronous parsing, the parser configuration linked to your datasource is used.
4. Save the datasource configuration.
## Configure the parser stage
You must do this step in Fusion 5.9.11 and later.
1. Navigate to **Parsers**.
2. Select the parser, or create a new parser.
3. From the **Add a parser stage** menu, select **Apache Tika Container Parser**.
4. (Optional) Enter a label for this stage. This label changes the names from Apache Tika Container Parser to the value you enter in this field.
5. If the Apache Tika Container Parser stage is not already the first stage, drag and drop the stage to the top of the stage list so it is the first stage that runs.
## Configure the index pipeline
1. Go to the **Index Pipeline** screen.
2. Add the **Solr Partial Update Indexer** stage.
3. Turn off the **Reject Update if Solr Document is not Present** option and turn on the **Process All Pipeline Doc Fields** option:
4. Include an extra update field in the stage configuration using any update type and field name. In this example, an incremental field `docs_counter_i` with an increment value of `1` is added:
5. Enable the **Allow reserved fields** option:
6. Click **Save**.
7. Turn off or remove the **Solr Indexer stage**, and move the **Solr Partial Update Indexer stage** to be the last stage in the pipeline.
Asynchronous Tika parsing setup is now complete. Run the datasource indexing job and monitor the results.
Other parsers, such as HTML and JSON, are now supported by the asynchronous parsing service. By enabling asynchronous parsing, the parser configuration linked to your datasource is used.
To learn how to start using the new asynchronous parsing service, see the following demonstration:
This update also includes a new [Async Parsing API](/api-reference/async-parsing-api/index-a-document-using-a-profile-id).
The course for **Asynchronous Parsing Service** focuses on how to use asynchronous parsing to index your data more efficiently.
### Job Config service
The new Job Config service provides more accurate and reliable job status reporting.
This service uses asynchronous communication through Kafka for improved efficiency over the previous synchronous calls, with these benefits:
* More accurate job tracking. You get real-time, reliable job status updates, reducing uncertainty about whether a job is running, completed, or failed.
* Faster troubleshooting. With detailed job histories and improved reporting, teams can quickly diagnose issues instead of chasing down incomplete or outdated job statuses.
* Seamless transition. For most users, no action is required—API calls through `api-gateway` automatically reroute. Internal API users need only a simple update to continue tracking jobs accurately.
The following API endpoints have been migrated from the `admin` service to the new `job-config` service:
```bash theme={"dark"}
/api/tasks/{id}
/api/jobs/{resource}/schedule
/api/tasks
/api/jobs/{resource}/actions
/api/tasks/_schema
/api/jobs
/api/jobs/{resource}
/api/jobs/{resource}/history
/api/jobs/_schema
```
For API calls made to the `api-gateway` service, you do not need to make any changes; the endpoints above are automatically rerouted to the new `job-config` service.
See [Job Config API](/api-reference/manage-task-jobs/list-task-jobs) for reference information about these endpoints.
If you are making internal API calls to the `admin` service using any of the endpoints above, you must update your API calls to point to the new `job-config` service.
### Security updates
Lucidworks remains committed to providing a secure and resilient platform. Fusion 5.9.11 includes critical security updates across a number of Fusion services, including the admin, connectors, distributed compute, indexing, job configuration, and query services, ensuring continued protection and reliability for your deployments.
### API path changes
Trailing slashes are no longer supported when making API calls.
For example, this API call works:
```
http://HOSTNAME/api/apps/APP_NAME/query/QUERY_PROFILE?q=hello
```
But this API call does not work and results in an error:
```
http://HOSTNAME/api/apps/APP_NAME/query/QUERY_PROFILE/?q=hello
```
## Bug fixes
* Critical fix for Azure Kubernetes Service (AKS) and Amazon Elastic Kubernetes Service (EKS) compatibility on Kubernetes 1.30+\
By upgrading the Kubernetes Client library to version 6.2.0, this update prevents token refresh failures that previously caused service disruptions. Affected services—including connectors backend, indexing, job rest server, and job launcher services—now operate reliably in OIDC-enabled AKS and EKS environments, strengthening Fusion’s stability on modern Kubernetes deployments.
## Known issues
* Saving large query pipelines may cause OOM failures under high load.\
In Fusion 5.9.x versions through 5.9.13, saving a large query pipeline during high query volume can result in thread lock, excessive memory use, and eventual OOM errors in the Query service.\
This issue is fixed in Fusion 5.9.14.
* Scheduled jobs that perform Solr operations may fail to run or save due to permission-related issues in the `job-config` service.\
In some cases, users without full permissions can create or modify scheduled tasks that execute actions they aren’t authorized to run directly, specifically tasks that invoked Solr using `solr://` URIs.\
These jobs may fail silently or prevent schedule changes from being saved. This issue can also affect the execution of jobs after upgrade. It is fixed in Fusion 5.9.13.
* Deploying Fusion with TLS flags enabled fails in ArgoCD due to Helm chart rendering limitations.\
When using ArgoCD to deploy Fusion 5.9.10 or 5.9.11 with TLS options enabled, Helm chart rendering fails due to the use of the `lookup` function, which is unsupported by ArgoCD. This prevents ArgoCD from generating manifests, blocking deployment workflows that rely on TLS configuration.\
This issue is fixed in Fusion 5.9.12.\
As a workaround, deploy Fusion without enabling TLS in ArgoCD-managed environments, or perform the deployment using Helm directly.
* Incorrect Solr image repository in Helm charts.\
Fusion 5.9.11’s Helm chart incorrectly references an internal Lucidworks Artifactory repository for the Solr image. This can prevent successful deployment in environments without access to internal infrastructure.\
This issue is fixed in Fusion 5.9.12.\
As a workaround, override the Solr image repository in your custom `values.yaml` file.
* Jobs API returns incorrect timestamps for jobs scheduled after 12:00 UTC.\
In Fusion 5.9.11, job timestamps returned by the API may be off by 12 hours if the time is after noon UTC. This affects scheduling accuracy and may cause the UI to misinterpret whether a change has been made.\
This issue is fixed in Fusion 5.9.12.
* Cannot update existing job triggers in the Schedulers view.\
In Fusion 5.9.11, scheduled job triggers cannot be modified due to incorrect timestamp comparison logic in the Admin UI. You must delete and recreate the trigger instead.\
This issue is fixed in Fusion 5.9.12.
* Scheduler may stop working if ZooKeeper becomes briefly unavailable.\
If all `job-config` pods lose connection to ZooKeeper in Fusion 5.9.11, they may fail to re-elect a leader, halting all scheduling until one of the pods is manually restarted.\
This issue is fixed in Fusion 5.9.12.
* Datasource jobs show incorrect "started by" user.\
In Fusion 5.9.11, datasource jobs started in the UI incorrectly show `default-subject` as the initiating user in job history instead of the actual user.\
This issue is fixed in Fusion 5.9.12.
* Schema API incompatibility with nonstandard schema file names.\
In Fusion 5.9.11, new apps save Solr schemas with the name `managed-schema.xml` instead of `managed-schema`, while older apps may include only a `managed-schema` file.\
This mismatch can cause errors when editing or previewing schemas using the Schema API, particularly when the expected file extension is missing.\
For existing config sets, it’s best to copy the contents from `managed-schema` to a new `managed-schema.xml`, then delete `managed-schema`.\
Fusion 5.9.12 resolves this by supporting both file names, ensuring backward compatibility with older apps and consistent behavior for new ones.
* Job scheduling may stop working if all `job-config` pods lose connection to ZooKeeper.\
In rare cases, when a ZooKeeper connection is lost, the leader latch mechanism used by the `job-config` service might fail to recover. If no node can re-establish leadership, the job scheduler stops until at least one `job-config` pod is restarted.\
This issue is fixed in Fusion 5.9.13.
* Scheduled job triggers can’t be updated in the Schedulers view.\
While the Admin UI displays the current trigger time correctly, changes to the time are not saved.\
This is due to a mismatch in how the API formats UTC timestamps, especially for times after 12:00 UTC, which prevents the UI from detecting that a change has been made.\
To work around this, delete the existing scheduler entry and create a new one with the desired time.\
This issue is resolved in Fusion 5.9.13.
* Dependent job scheduling may fail.\
Jobs configured to run based on the success or failure of another job may not trigger as expected. This includes configurations using “on\_success\_or\_failure” and “Start + Interval” options. This issue is resolved in Fusion 5.9.13.
* Aborted jobs in Fusion are listed twice in the Job History.\
This issue is fixed in Fusion 5.9.12.
* When a new datasource is configured in **Indexing > Index Workbench**, the simulated results do not display, and the following error is generated: "Failed to simulate results from a working pipeline." As a result, the Index Workbench cannot simulate results for new datasources, preventing users from configuring the indexing process within the workbench.\
This issue is fixed in Fusion 5.9.12.\
To work around this issue, configure each part of the indexing process separately instead of using the Index Workbench:
* Configure datasources in **Indexing > Datasources**.
* Configure parsers in **Indexing > Parsers**.
* Configure index pipelines in **Indexing > Index Pipelines**.
* In Fusion 5.9.10, the Apache Spark 3.4.1 upgrade impacted jobs that use Python 3.7 behavior or compatibility, which may have automatically updated to Python 3.10.x and no longer function correctly. If you have not yet updated the code for Python 3.10.x, update your code to ensure compatibility with Python 3.10.x and then test your Spark jobs in a staging environment before deploying to production.
* An issue prevents segment-based rule filtering from working correctly in Commerce Studio. This issue is fixed in Fusion 5.9.12.
## Deprecations
For full details on deprecations, see [Deprecations and Removals](/docs/5/fusion/deprecations-and-removals).
* The [Parsers Indexing CRUD API](/api-reference/parsers-crud-api/update-a-parser), which provides CRUD operations for parsers, allowing users to create, read, update, and delete parsers, is deprecated in Fusion 5.9.11. This feature was originally deprecated in Fusion 5.12.0. It will be removed in a future release no later than September 4, 2025.
The [Async Parsing API](/api-reference/parsers-crud-api/list-all-parsers) replaces the Parsers Indexing CRUD API and is available in Fusion 5.9.11. This API provides improved functionality and aligns with Fusion’s updated architecture, ensuring consistency across versions.
* The [Word2Vec Model Training Job](/docs/5/fusion/reference/config-ref/jobs/word2vec-model-training), which trains a shallow neural model to generate vector embeddings for text data, is deprecated in Fusion 5.9.11. It will be removed in a future release no later than September 4, 2025.
Lucidworks offers a wide array of AI solutions, including [Lucidworks AI](/docs/lw-platform/lw-ai/overview). Lucidworks AI provides easy-to-use, AI-powered data discovery and search capabilities including:
Pre-trained embedding models
Custom embedding model training
Seamless integration with Fusion
## Removals
### Bitnami removal
Fusion 5.9.11 will be re-released with the same functionality but updated image references.
In the meantime, Lucidworks will self-host the required images while we work to replace Bitnami images with internally built open-source alternatives.
If you are a self-hosted Fusion customer, *you must upgrade before August 28* to ensure continued access to container images and prevent deployment issues.
You can reinstall your current version of Fusion or upgrade to Fusion 5.9.14, which includes the updated Helm chart and prepares your environment for long-term compatibility.
See [Prevent image pull failures due to Bitnami deprecation in Fusion 5.9.5 to 5.9.13](https://support.lucidworks.com/hc/en-us/articles/33966125467799-Prevent-image-pull-failures-due-to-Bitnami-deprecation-in-Fusion-5-9-5-to-5-9-13) for more information on how to prevent image pull failures.
## Platform support and component versions
### Kubernetes platform support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.28, 1.29, 1.30, 1.31
* **Microsoft Azure Kubernetes Service (AKS):** 1.28, 1.29, 1.30, 1.31
* **Amazon Elastic Kubernetes Service (EKS):** 1.28, 1.29, 1.30, 1.31
Support is also offered for Rancher Kubernetes Engine (RKE and RKE2) and OpenShift 4 versions that are based on Kubernetes 1.28, 1.29, 1.30, 1.31; note that RKE2 may require some Helm chart modification. OpenStack and customized Kubernetes installations are *not* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support).
### Component versions
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| Solr | fusion-solr 5.9.11 *(based on Solr 9.6.1)* |
| ZooKeeper | 3.9.1 |
| Spark | 3.4.1 |
| Ingress Controllers | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).
# Fusion 5.9.12
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9/5.9.12-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.9.12-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9.12-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/lt4m82
Originally released on April 16, 2025 and re-released on May 14, 2025, this [maintenance release](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#maintenance-release-support-policy) provides support for chunking in Lucidworks AI, model hosting with Ray, and important security upgrades and bug fixes.
Upgrading to the latest version of Fusion 5.9 offers several key benefits:
* **Access to latest features:** Stay current with the latest features and functionality to ensure compatibility and optimal performance.
* **Simplified process:** Fusion 5.9.5 and later use an in-place upgrade strategy, making upgrades easier than ever.
* **Extended support:** Upgrading keeps you up-to-date with the latest supported Kubernetes versions, as outlined in the [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle) policy.
**Security patch available for api-gateway: Netty request smuggling vulnerabilities**
A patch is available for the `api-gateway` service to address critical Netty request smuggling vulnerabilities (CVE-2026-42581, CVE-2026-42585, CVE-2026-42587). These vulnerabilities allow attackers to smuggle HTTP requests through the gateway, potentially bypassing security controls.
The `api-gateway` service requires the Netty security patch.
Follow these steps to apply the patched image:
1. Open your Fusion Helm values file.
2. Add or update the `api-gateway` image configuration:
```yaml theme={"dark"}
api-gateway:
image:
repository: lucidworks
name: api-gateway
tag: 5.9.12-SUST-1634-patch
imagePullPolicy: IfNotPresent
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace, `RELEASE_NAME` with your Helm release name, and `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the `api-gateway` pods to restart and verify they are using the patched image.
**Urgent action required by November 26, 2025**
A patch is required by November 26, 2025 for all self-hosted Fusion deployments running on Amazon Elastic Kubernetes Service (EKS). Certain Java versions used by Fusion components reach end of life on this date. Failure to apply the patch will result in compatibility issues.
The following Fusion services require the `cgroupv2` patch:
| Service | Affected Fusion versions | Patch tag |
| - | - | - |
| `insights` | 5.9.4 to 5.9.15 | `lucidworks/insights:5.9-cgroupv2-patch` |
| `spark-solr-etl` | 5.9.4 to 5.9.11 | `lucidworks/spark-solr-etl:5.9-cgroupv2-patch` |
| `keytool-utils` | 5.9.4 to 5.9.10 | `lucidworks/keytool-utils:5.9-cgroupv2-patch` |
Follow these steps to apply the patched images:
1. Open your Fusion Helm values file. For Fusion Cloud Native deployments, use the values file for your current deployment. For Helm deployments, use the values file you used to create the deployment.
2. For each service listed in the following table that applies to your Fusion version, add or update the image configuration:
Expand the following code snippet for the complete image configuration list.
```yaml expandable theme={"dark"}
global:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
sql-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
reverse-search:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
solr:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
kafka:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
ml-model-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-admin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-indexing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
query-pipeline:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
async-parsing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
admin-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
api-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
auth-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
classic-rest-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-resources:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-config:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
insights:
image:
imagePullPolicy: Always
name: insights
repository: lucidworks
tag: 5.9-cgroupv2-patch
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-launcher:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-rest-server:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connector-plugin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors-backend:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
rules-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
pm-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
lwai-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
webapps:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
apps-manager:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
templating:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
tikaserver:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
seldon-core-operator:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo-common-workflows:
image:
imagePullPolicy: Always
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
utilitiesTag: 5.9.11
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: 5.9-cgroupv2-patch
argo-common-workflows:
image:
imagePullPolicy: IfNotPresent
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: insights:5.9-cgroupv2-patch
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run the following command:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace. Replace `RELEASE_NAME` with your Helm release name. Replace `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the affected service pods to restart and verify they are using the patched images.
If you downloaded or installed Fusion 5.9.12 **before May 14, 2025** and are encountering issues, you should pull the images again and re-install Fusion 5.9.12 to pick up the latest bug fixes. This ensures you benefit from critical stability, security, and compatibility improvements included in the re-released version. If you are using self-hosted Fusion and installed 5.9.12 prior to the re-release, see the [self-hosted reinstallation instructions below](#reinstall-5-9-12-for-self-hosted-fusion).
When requesting help from the Lucidworks support team about this release, be sure to specify whether you are running the original Fusion 5.9.12 release or the re-release, so we can provide accurate guidance and faster resolution.
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
Fusion 5.9.5 introduced changes that affect Spark jobs. If you are upgrading from any Fusion version earlier than 5.9.5 to a version later than 5.9.5, Lucidworks recommends first upgrading to Fusion 5.9.5 before proceeding to the target version.
For supported Kubernetes versions and key component versions, see [Platform support and component versions](#platform-support-and-component-versions).
## Reinstall 5.9.12 for self-hosted Fusion
If you are a self-hosted Fusion client and installed 5.9.12 in your environment before the re-release, the standard image pull policy may prevent pulling the re-release version. With the original release, the worker nodes cache the images, and the new version won’t be pulled because they weren’t retagged with a newer version string.
### Instructions
1. Verify the job-launcher pod is using the correct image by comparing its sha256 digest to the expected value of `fusion/lucidworks/job-launcher@sha256:02624cfbc3eb536e4a29353b7bfe0846380a09f98ae3c78da859793686782092`.
1. Run the following command:
```sh wrap theme={"dark"}
kubectl get pod job-launcher-pod -n ns-name -o jsonpath='{.status.containerStatuses[0].imageID}'
```
2. If the digest output is different in any way, the image is not from the re-release.
2. Check the `imagePullPolicy` of your Deployments:
```sh wrap theme={"dark"}
kubectl get deployment -n ns-name -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .spec.template.spec.containers[*]}{.name}={.imagePullPolicy}{" "}{end}{"\n"}{end}'
```
1. If the policy is set to `IfNotPresent`, patch it to `Always`:
```sh wrap theme={"dark"}
kubectl patch deployments -n ns-name --type=json -p='[{"op":"replace","path":"/spec/template/spec/containers/0/imagePullPolicy","value":"Always"}]'
```
2. Restart the Deployments and StatefulSets to apply the changes:
```sh theme={"dark"}
kubectl rollout restart deployment,statefulset -n ns-name
```
3. (Optional) Check the `imagePullPolicy` for StatefulSets specifically to ensure they are set correctly:
```sh wrap theme={"dark"}
kubectl get sts -n ns-name -o=jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .spec.template.spec.containers[*]}{.name}={.imagePullPolicy}{" "}{end}{"\n"}{end}'
```
3. After the rollout, confirm again that the job-launcher pod is using the correct image:
```sh wrap theme={"dark"}
kubectl get pod job-launcher-pod -n ns-name -o jsonpath='{.status.containerStatuses[0].imageID}'
```
1. If the result matches the expected digest of `sha256:02624cfbc3eb536e4a29353b7bfe0846380a09f98ae3c78da859793686782092` then the pod is running the latest re-release version.
## Key highlights
### Improved relevance in large documents with chunking
Fusion 5.9.12 introduces document chunking, a major advancement for search and generative AI quality and performance. Chunking breaks large documents into smaller, meaningful segments—called chunks—that are stored and indexed using Solr’s block join capabilities. Each chunk captures either lexical content (for keyword-based search) or semantic vectors (for neural search), enabling Fusion to retrieve the most relevant part of a document, rather than treating the document as a single block.
This improves:
* **Search relevance**: Users get results that point to the most relevant sections within large documents, not just documents that match overall.
* **Neural search precision**: Vector chunks improve hybrid scoring by aligning semantic relevance with specific lexical content.
* **Scalability and maintainability**: Updates or deletions are applied at the chunk level, ensuring consistency and avoiding stale or orphaned content.
* **Faceted search and UX**: Results can be grouped and ranked more accurately, especially in use cases where dense documents contain multiple topics.
Document chunking is particularly valuable in knowledge management and technical content domains, where retrieving the right paragraph can be more important than retrieving the right document.
* A new [LWAI Chunker index pipeline stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/lucidworks-ai-chunker-index-stage) uses one of the available chunking strategies (chunkers) for the specified LW AI model to provide optimized storage and retrieval. The chunkers asynchronously split the provided text in various ways such as by sentence, new line, semantics, and regular expression (regex) syntax.
* A new [Chunking Neural Hybrid Query pipeline stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/chunking-neural-hybrid-query) now detects chunked documents and retrieves the most relevant lexical and vector segments for hybrid search.
* Updates and deletions now ensure consistent chunk synchronization to prevent orphaned data.
* This feature includes a new [Lucidworks AI Async Chunking API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-async-chunking-api).
Click **Get Started** below to see how to enable chunking in Fusion:
Contact your Lucidworks account manager to confirm that your license includes this feature.
### Model hosting with Ray
Fusion 5.9.12 introduces support for model hosting with Ray, replacing the previous Seldon-based approach. Ray offers a more scalable and efficient architecture for serving machine learning models, with native support for distributed inference, autoscaling, and streamlined deployment. This transition simplifies Fusion’s AI infrastructure, enhances performance, and aligns with modern MLOps practices to make deploying and managing models faster, more reliable, and easier to monitor.
Ray is present by default but still needs to be configured to run and perform tasks.
Istio is only required for using Ray if you require in-cluster TLS.
* The Seldon vectorize stages have been renamed to [Ray/Seldon Vectorize Field](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/vectorize-field-via-seldon-index-stage) and [Ray/Seldon Vectorize Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/vectorize-query-via-seldon-query-stage).
* There are two new jobs for deploying models with Ray:
* [Create Ray Deployment](/docs/5/fusion/reference/config-ref/jobs/create-ray-model-deployment).
* [Delete Ray Deployment](/docs/5/fusion/reference/config-ref/jobs/delete-ray-model-deployment).
If you previously deployed a model with Seldon, you can deploy the same model with Ray.
Just follow the instructions in **Develop and deploy a machine learning model with Ray**, and deploy the model with a different name to avoid conflicts.
This tutorial walks you through deploying your own model to Fusion with Ray.
This feature is only available in Fusion 5.9.x for versions 5.9.12 and later.
## Prerequisites
* A Fusion instance with an app and indexed data.
* An understanding of Python and the ability to write Python code.
* [Docker](https://docs.docker.com/get-docker/) installed locally, plus a private or public Docker repository.
* Ray installed locally: `pip install ray[serve]` using the version of ray\[serve] found in the release notes for your version of Fusion.
* Code editor; you can use any editor, but Visual Studio Code is used in this example.
* Model: [intfloat/e5-small-v2](https://huggingface.co/intfloat/e5-small-v2)
* Docker image: [e5-small-v2-ray](https://hub.docker.com/r/jstrmec/e5-small-v2-ray/)
## Tips
* Always test your Python code locally before uploading to Docker and then Fusion.
This simplifies troubleshooting significantly.
* Once you’ve created your Docker you can also test locally by doing `docker run` with a specified port, like 9000, which you can then `curl` to confirm functionality in Fusion.
See the testing example below.
* If you previously deployed a model with Seldon, you can deploy the same model with Ray after making a few changes to your Docker image as explained in this topic.
To avoid conflicts, deploy the model with a different name.
When you have verified that the Ray model is working after deployment with Ray, you can delete the Seldon model using the [Delete Seldon Core Model Deployment job](/docs/5/fusion/reference/config-ref/jobs/delete-seldon-core-model-deployment).
* If you run into an issue with the model not deploying and you’re using the 'real' example, there is a very good chance you haven’t allocated enough memory or CPU in your job spec or in the Ray-Argo config.
It’s easy to increase the resources. To edit the ConfigMap, run `kubectl edit configmap argo-deploy-ray-model-workflow -n ` and then find the `ray-head` container in the artisanal escaped YAML and change the memory limit.
Exercise caution when editing because it can break the YAML.
Just delete and replace a single character at a time without changing any formatting.
* For additional guidance, see the [testing locally e5-model example](https://colab.research.google.com/drive/1ef-NTJBtP8p77aAa8Pp9SPXtQyozJEas?usp=sharing).
The course for **Intro to Machine Learning in Fusion** focuses on using machine learning to infer the goals of customers and users in order to deliver a more sophisticated search experience.
## Local testing example
1. Docker command:
```bash theme={"dark"}
docker run -p 127.0.0.1:9000:9000 DOCKER_IMAGE
```
2. Curl to hit Docker:
```bash theme={"dark"}
curl -i -X POST http://127.0.0.1:8000 -H 'Content-Type: application/json' -d '{"text": "The quick brown fox jumps over the lazy dog."}'
```
3. Curl model in Fusion:
```bash theme={"dark"}
curl -u $FUSION_USER:$FUSION_PASSWORD -X POST -H 'Content-Type: application/json' -d '{"text": "i love fusion"}' https://FUSION_HOST.com:6764/api/ai/ml-models/MODEL_NAME/prediction
```
4. See all your deployed models:
```bash theme={"dark"}
curl -u USERNAME:PASSWORD http://FUSION_HOST:FUSION_PORT/api/ai/ml-models
```
5. Check the Ray UI to see Replica State, Resources, and Logs.\
If you are getting an internal model error, the best way to see what is going on is to query via port-forwarding the model.\
The `MODEL_DEPLOYMENT` in the command below can be found with `kubectl get svc -n NAMESPACE`. It will have the same name as set in the model name in the **Create Ray Model Deployment** job.
```bash theme={"dark"}
kubectl -n NAMESPACE port-forward svc/MODEL_DEPLOYMENT-head-svc 8000:8000
```
Once port-forwarding is successful, you can use the below cURL command to see the issue.
At that point your worker logs should show helpful error messages.
```bash theme={"dark"}
curl --location 'http://127.0.0.1:8000/' \
--header 'charset: utf-8' \
--header 'Content-Type: application/json' \
--data '{"text": "i love fusion"}'
```
## Download the model
This tutorial uses the [`e5-small-v2`](https://huggingface.co/intfloat/e5-small-v2) model from Hugging Face, but any pre-trained model from [https://huggingface.co](https://huggingface.co) will work with this tutorial.
If you want to use your own model instead, you can do so, but your model must have been trained and then saved though a function similar to the PyTorch’s `torch.save(model, PATH)` function.
See [Saving and Loading Models](https://pytorch.org/tutorials/beginner/saving_loading_models.html) in the PyTorch documentation.
## Format a Python class
The next step is to format a Python class which will be invoked by Fusion to get the results from your model.
The skeleton below represents the format that you should follow.
See also [Getting Started](https://docs.ray.io/en/latest/serve/getting_started.html) in the Ray Serve documentation.
```python wrap expandable theme={"dark"}
from ray import serve
from starlette.requests import Request
# These defaults are for the ray serve deployment
# when running simply from docker. The 'Create Ray Model Deployment'
# job can override these replicas and resources if needed.
@serve.deployment(num_replicas=1, ray_actor_options={"num_cpus": 1})
class Deployment(object):
def __init__(self):
"""
Add any initialization parameters. Generally this is where you would load
your model. This method will be called once when the deployment is created.
"""
print("Initializing")
self.model = load_model() #faux code
# This can be named as any method which takes a dictionary as input and returns a dictionary
# as output. In this example, we are using the encode method to encode the
# input text into a vector.
def encode(self, input_dict: Dict[str, Any]) -> Dict[str, Any]:
"""
This method will be called when the deployment is queried. It will receive
the input data and should return the output data.
"""
text = input_dict["text"]
embeddings = self.model.encode #faux code
return { "vector": embeddings } # To use the 'Ray / Seldon Vectorize Field' stage, the output key should be `vector`, if using the 'Machine Learning' stage you must ensure the output key matches the output key in the 'Machine Learning' stage
async def __call__(self, http_request: Request) -> Dict[str, Any]:
input_dict: Dict[str, Any] = await http_request.json()
return self.encode(input_dict=input_dict) # This will be the function you defined above, in this case encode
app = Deployment.bind()
```
A real instance of this class with the `e5-small-v2` model is as follows:
This code pulls from Hugging Face. To have the model load in the image without pulling from Hugging Face or other external sources, download the model weights into a folder name and change the model name to the folder name preceded by `./`.
```python wrap expandable theme={"dark"}
import json
import sys
from time import time
from typing import Any, Dict
import torch
import torch.nn.functional as F
from ray import serve
from starlette.requests import Request
from starlette.responses import JSONResponse
from torch import Tensor
from transformers import AutoModel, AutoTokenizer
HUB_MODEL_NAME = "intfloat/e5-small-v2"
@serve.deployment(num_replicas=1, ray_actor_options={"num_cpus": 1})
class Deployment(object):
def __init__(self):
from loguru import logger
self.logger = logger
# Initializing logger
self.logger.remove()
self.logger.add(sys.stdout, level="INFO", serialize=False, colorize=True)
# Initializing model
self.logger.info("Loading model...")
self.tokenizer = AutoTokenizer.from_pretrained(HUB_MODEL_NAME)
self.model = AutoModel.from_pretrained(HUB_MODEL_NAME)
self.model.eval()
self.logger.info("Model initialization finished!")
def encode(self, input_dict: Dict[str, Any]) -> Dict[str, Any]:
_start_time = time()
# Extracting text from input
text = input_dict["text"]
# Tokenization
tokenized_texts = self.tokenizer(
text,
max_length=512,
padding=True,
truncation=True,
return_tensors="pt",
)
# Encoding
with torch.inference_mode():
# Forward pass of the model
outputs = self.model(**tokenized_texts)
# Average pooling the last hidden states
embeddings = self.average_pool(
outputs.last_hidden_state, tokenized_texts["attention_mask"]
)
# Normalizing embeddings
embeddings = F.normalize(embeddings, p=2, dim=1)
# Converting into output format
output_dict = {"vector": embeddings.squeeze().tolist()}
prediction_time = (time() - _start_time) * 1000
self.logger.info(f"Time taken to make a prediction: {prediction_time:.0f}ms")
return output_dict
async def __call__(self, http_request: Request) -> Dict[str, Any]:
try:
input_dict: Dict[str, Any] = await http_request.json()
except UnicodeDecodeError:
body_bytes = await http_request.body()
try:
decoded = body_bytes.decode("utf-8", errors="replace")
input_dict = json.loads(decoded)
except json.JSONDecodeError:
return JSONResponse({"error": "Invalid JSON"}, status_code=400)
return self.encode(input_dict=input_dict)
@staticmethod
def average_pool(last_hidden_states: Tensor, attention_mask: Tensor) -> Tensor:
last_hidden = last_hidden_states.masked_fill(
~attention_mask[..., None].bool(), 0.0
)
return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]
app = Deployment.bind()
```
In the preceding code, logging has been added for debugging purposes.
The preceding code example contains the following functions:
* `__call__`: This function is non-negotiable.
* `init`: The `init` function is where models, tokenizers, vectorizers, and the like should be set to self for invoking.\
It is recommended that you include your model’s trained parameters directly into the Docker container rather than reaching out to external storage inside `init`.
* `encode`: The `encode` function is where the field or query that is passed to the model from Fusion is processed.\
Alternatively, you can process it all in the `__call__` function, but it is cleaner not to.\
The `encode` function can handle any text processing needed for the model to accept input invoked in its `model.predict()` or equivalent function which gets the expected model result.
If the output needs additional manipulation, that should be done before the result is returned.
For embedding models, the return value must have the shape of (1, DIM), where DIM (vector dimension) is a consistent integer, to enable Fusion to handle the vector encoding into Ray.
The class name must be `Deployment()` and the name of this file must be `deployment.py`.
In the preceding example, the Python file is named `deployment.py` and the class name is `Deployment()`.
## Create a Dockerfile
The next step is to create a Dockerfile. The Dockerfile should follow this general outline; read the comments for additional details:
```dockerfile theme={"dark"}
#It is important that python version is 3.x-slim
FROM python:3.10-slim
# Install dependencies
RUN apt-get update && apt-get install -y wget
# Create working app directory
RUN mkdir -p /app
WORKDIR /app
# Copy the requirements file and install the dependencies
COPY requirements.txt /app
RUN pip install -r requirements.txt --no-cache-dir
# Copy source code
COPY deployment.py /app
# Expose serving port for HTTP communication with Fusion
EXPOSE 8000
# The end of the command follows module:application and the below value should be set in the RAY DEPLOYMENT IMPORT PATH field in 'Create Ray Model Deployment' job
CMD exec serve run deployment:app
```
## Create a requirements file
The `requirements.txt` file is a list of installs for the `Dockerfile` to run to ensure the Docker container has the right resources to run the model.
For the `e5-small-v2` model, the requirements are as follows:
```text theme={"dark"}
torch -f https://download.pytorch.org/whl/torch_stable.html # Make sure that we download CPU version of PyTorch
transformers
loguru
ray[serve]==2.42.1
```
Any recent ray\[serve] version should work, but the tested value and known supported version is 2.42.1.
In general, if an item was used in an `import` statement in your Python file, it should be included in the requirements file.
To populate the requirements, use the following command in the terminal, inside the directory that contains your code:
```bash theme={"dark"}
pip freeze > requirements.txt
```
## Build and push the Docker image
After creating the `MODEL_NAME.py`, `Dockerfile`, and `requirements.txt` files, you need to run a few Docker commands.
Run the following commands in order:
```bash theme={"dark"}
DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build . -t [DOCKERHUB-USERNAME]/[REPOSITORY]:[VERSION-TAG]
```
```bash theme={"dark"}
docker push [DOCKERHUB USERNAME]/[REPOSITORY]:[VERSION-TAG]
```
Using the example model, the terminal commands would be as follows:
```bash theme={"dark"}
DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build . -t jstrmec/e5-small-v2-ray:0.1
```
```bash theme={"dark"}
docker push jstrmec/e5-small-v2-ray:0.1
```
This repository is public and you can visit it here: [e5-small-v2-ray](https://hub.docker.com/r/jstrmec/e5-small-v2-ray/)
## Deploy the model in Fusion
Now you can go to Fusion to deploy your model.
When deploying your Ray model, you have two options for handling traffic:
* Use a single deployment. Deploy one model job that handles both indexing and query traffic. This is simpler to manage and requires only one deployment.
* Use separate deployments for indexing and querying. Deploy two separate model jobs: one dedicated to indexing and another for query traffic. This approach eliminates the risk of indexing workloads impacting query response times, providing better performance isolation and independent scaling control.
Choose separate deployments if query performance is critical and you want to ensure indexing operations don't compete with user-facing queries for resources.
To use separate deployments, follow the deployment steps to create two Ray model deployment jobs with different job IDs (for example, `EXAMPLE_MODEL_INDEX` and `EXAMPLE_MODEL_QUERY`). Use the index-specific model in your index pipeline stages and the query-specific model in your query pipeline stages. To keep both deployments in sync, ensure both jobs use the exact same model name, Ray deployment import path, Docker repository, and image name.
1. In Fusion, navigate to **Collections** > **Jobs**.
2. Add a job by clicking the **Add+** Button and selecting **Create Ray Model Deployment**.
3. Fill in each of the text fields:
| Parameter | Description |
| - | - |
| Job ID | A string used by the Fusion API to reference the job after its creation. |
| Model name | A name for the deployed model. This is used to generate the deployment name in Ray. It is also the name that you reference as a `model-id` when making predictions with the ML Service. |
| Model min replicas | The minimum number of load-balanced replicas of the model to deploy. |
| Model max replicas | The maximum number of load-balanced replicas of the model to deploy. Specify multiple replicas for a higher-volume intake. |
| Model CPU limit | The number of CPUs to allocate to a single model replica. |
| Model memory limit | The maximum amount of memory to allocate to a single model replica. |
| Ray Deployment Import Path | The path to your top-level Ray Serve deployment (or the same path passed to `serve run`). For example, `deployment:app` |
| Docker Repository | The public or private repository where the Docker image is located. If you’re using Docker Hub, fill in the Docker Hub username here. |
| Image name | The name of the image. For example, `e5-small-v2-ray:0.1`. |
| Kubernetes secret | If you’re using a private repository, supply the name of the Kubernetes secret used for access. |
4. Click **Advanced** to view and configure advanced details:
| | |
| - | - |
| Parameter | Description |
| Additional parameters. | This section lets you enter parameter `name:parameter` value options to be injected into the training JSON map at runtime. The values are inserted as they are entered, so you must surround string values with `"`. This is the sparkConfig field in the configuration file. |
| Write Options. | This section lets you enter parameter `name:parameter` value options to use when writing output to Solr or other sources. This is the writeOptions field in the configuration file. |
| Read Options. | This section lets you enter parameter `name:parameter` value options to use when reading input from Solr or other sources. This is the readOptions field in the configuration file. |
5. Click **Save**, then **Run** and **Start**.
When the job finishes successfully, you can proceed to the next section.
Now that the model is in Fusion, you can use it in the Machine Learning or Ray / Seldon Vectorize index and query stages.
## Configure the Fusion pipelines
Your real-world pipeline configuration depends on your use case and model, but for our example we will configure the index pipeline and then the query pipeline.
**Configure the index pipeline**
1. Create a new index pipeline or load an existing one for editing.
2. Click **Add a Stage** and then **Machine Learning**.
3. In the new stage, fill in these fields:
* The model ID
* The model input
* The model output
4. Save the stage in the pipeline and index your data with it.
**Configure the query pipeline**
1. Create a new query pipeline or load an existing one for editing.
2. Click **Add a Stage** and then **Machine Learning**
3. In the new stage, fill in these fields:
* The model ID
* The model input
* The model output
4. Save the stage and then run a query by typing a search term.
5. To verify the Ray results are correct, use the **Compare+** button to see another pipeline without the model implementation and compare the number of results.
You have now successfully uploaded a Ray model to Fusion and deployed it.
When you have verified that the Ray model is working after deployment with Ray, you can delete the Seldon model using the [Delete Seldon Core Model Deployment job](/docs/5/fusion/reference/config-ref/jobs/delete-seldon-core-model-deployment).
### AI and machine learning features
Fusion’s machine learning services now run on Python 3.10 and Java 11, bringing improved performance, security, and compatibility with the latest libraries.
These upgrades enhance model execution speed, memory efficiency, and long-term support, ensuring Fusion’s machine learning capabilities remain optimized for your evolving AI workloads.
No configuration changes are required to take advantage of these improvements.
### Improved prefiltering support for Neural Hybrid Search (NHS)
Fusion 5.9.12 introduces early improvements for a more robust prefiltering strategy with Neural Hybrid Search, including support for chunked document queries.
Be aware that some prefiltering capabilities, such as access to the `preFilterKey` context property and the `VectorPreFilter` helper class are not available in 5.9.12.
These features were included in Fusion 5.9.13.
In 5.9.12, prefiltering may require custom pipeline logic or workarounds and may not behave the same as in earlier hybrid query implementations. For full prefiltering support, including the `VectorPreFilter` context object and documented JavaScript usage, we recommend upgrading to Fusion 5.9.13 or later, where the functionality is fully available.
Prefiltering is a technique that can improve performance and accuracy by filtering documents before applying the algorithm, reducing the number of documents that need to be processed.
You can enable prefiltering in any NHS query pipeline stage; see the configuration reference topics for details:
* [Chunking Neural Hybrid Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/chunking-neural-hybrid-query)
* [Neural Hybrid Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/neural-hybrid-query)
## Bug fixes
* Fixed a Helm chart rendering issue that blocked ArgoCD deployments with TLS flags enabled.\
In previous releases, using TLS flags during deployment caused Helm chart rendering to fail in ArgoCD due to the use of the unsupported `lookup` function. Fusion 5.9.12 resolves this issue by modifying the chart templates for compatibility with ArgoCD, restoring support for TLS-enabled deployments in GitOps workflows.
* Fixed incorrect image repository for Solr in Helm charts.\
In Fusion 5.9.11, the Helm chart specified an internal Lucidworks Artifactory repository for the Solr image.\
This has been corrected in 5.9.12 so the Solr image repository is either empty or points to `lucidworks/fusion-solr`, aligning with other components and simplifying deployment for external environments.
* Added support for configuring the Spark version used by Fusion.\
Fusion 5.9.12 now lets you switch from the default Spark 3.4.1 version(introduced in Fusion 5.9.10) to the earlier Spark 3.2.2 version used in Fusion 5.9.9.\
This flexibility helps maintain compatibility with legacy Python (3.7.3) and Scala environments, especially for apps that depend on specific Spark runtime behaviors.
When Spark 3.4.1 is enabled, custom Python jobs require Python 3.10.\
To enable Spark 3.2.2 instead, change the `fusion-spark` image in the configmap of the `job-launcher`, like this:
```yaml theme={"dark"}
driver:
container:
image: example.com/dir/fusion-spark-3.2.2
executor:
container:
image: example.com/dir/fusion-spark-3.2.2
```
The Spark 3.4.1 runtime is incompatible with Azure Blob Storage when accessed via the deprecated `wasbs://` protocol due to a Jetty version conflict; run Spark 3.2.2 instead if your jobs rely on `wasbs://`.
In the long term, we recommend migrating to the `abfs://` protocol for Azure Blob Storage access, which is fully supported in Spark 3.4.1.
* Fixed incorrect `started-by` values for datasource jobs in the job history.\
In previous versions, datasource jobs started from the Fusion UI were incorrectly shown as started by `default-subject` instead of the actual user.\
Fusion now correctly records and displays the initiating user in the job history, restoring accurate audit information for datasource operations.
* Fixed a schema loading issue that prevented older apps from working with the Schema API.\
Fusion now correctly handles both `managed-schema` and `managed-schema.xml` files when reading Solr config sets, ensuring backward compatibility with apps created before the move to template-based config sets.\
This prevents Schema API failures caused by unhandled exceptions during schema file lookup.
* Scheduled jobs now correctly trigger dependent jobs.\
In Fusion 5.9.12, we fixed an issue that prevented scheduled jobs from triggering other jobs based on their success or failure status.\
This includes jobs configured to run “on\_success\_or\_failure” or using the “Start + Interval” option.\
With this fix, dependent jobs now execute as expected, restoring reliable job chaining and scheduling workflows.
* Fixed an issue that prevented updates to existing scheduled job triggers in the Schedulers view.\
This bug was caused by inconsistencies in how the API returned UTC timestamps, particularly for times after 12:00 UTC. The Admin UI now correctly detects changes and allows updates to trigger times without requiring the entry to be deleted and recreated.
* Fixed multiple scheduling issues in the `job-config` service to improve reliability and user experience.\
This release includes several important fixes to ensure that scheduled jobs function correctly across upgrades and user actions:
* Improved leader election recovery: Ensures the scheduler service recovers if all `job-config` pods lose connection to ZooKeeper.
* Corrected permission checks: Prevents scheduling failures caused by mismatches between user and service account permissions.
* Fixed job history display for system jobs: Resolved an issue where delete-old-system-logs and delete-old-job-history jobs did not appear in the UI after upgrade, despite existing in the backend.
* Restored schedule creation in certain apps: Fixed a bug where clicking Save when adding a schedule in the Run dialog appeared to succeed but failed to persist the schedule in specific app states.
* Fixed a simulation failure in the Index Workbench when configuring new datasources.\
Fusion 5.9.12 resolves an issue where Index Workbench failed to simulate results after configuring a new datasource, displaying the error “Failed to simulate results from a working pipeline.”
This fix restores full functionality to the Index Workbench, allowing you to preview and configure indexing workflows in one place without switching between multiple views.
* Fixed a bug that caused aborted jobs to appear twice in the job history.\
Previously, when you manually aborted a job, it was recorded twice in the job history.\
This duplication has been resolved, and each aborted job now appears only once in the history log.
* Fixed an issue that prevented segment-based rule filtering from working correctly in Commerce Studio. Fusion now honors the `lw.rules.target_segment` parameter, ensuring only matching rules are triggered and improving rule targeting and safety.
* This release eliminates extra warning messages in the API Gateway related to undetermined service ports. Previously, the gateway logged repeated warnings about missing `primary-port-name` labels, even though this did not impact functionality. This fix reduces unnecessary log noise and improves the clarity of your logs.
* Fixed in the Fusion 5.9.15 release, this patch was also applied to Fusion 5.9.12:
Added a custom module that serializes Nashorn `undefined` values as JSON `null`, preventing serialization errors when such values appear in Java objects. When using the OpenJDK version of Nashorn, an exception was thrown when the context contained keys with undefined JavaScript values. This exception occurred because the class for the undefined values was not visible outside of the OpenJDK Nashorn package. A custom object mapper is added for the class so serialization issues are now resolved when handling `undefined` in JavaScript pipeline to run reliably under Java 17.
## Known issues
* Job-config service may appear “down” in UI even when running correctly.\
In Fusion versions 5.9.12 through 5.9.13, the job-config service may falsely report as “down” in the Fusion UI, particularly during startup or in TLS-enabled deployments.
This issue is fixed in Fusion 5.9.14.
For the `job-config` service, the **Next Run** value is updated *only after the current job completes*. Because the same job cannot run multiple times simultaneously, if another run is scheduled before the current job completes, that run is not executed.
* Jobs and V2 datasources may fail when Fusion collections are remapped to different Solr collections.\
In Fusion versions 5.9.12 through 5.9.13, strict validation in the `job-config` service causes “Collection not found” errors when jobs or V2 datasources target Fusion collections that point to differently named Solr collections.\
This issue is fixed in Fusion 5.9.14.\
As a workaround, use V1 datasources or avoid using REST call jobs on remapped collections.
* Saving large query pipelines may cause OOM failures under high load.\
In Fusion versions 5.9.12 through 5.9.13, saving a large query pipeline during high query volume can result in thread lock, excessive memory use, and eventual OOM errors in the Query service.\
This issue is fixed in Fusion 5.9.14.
* Some S3 and Web datasource jobs cannot be stopped in Fusion 5.9.12.\
Clicking the Stop button has no effect in some cases where the backend job is no longer being tracked. This causes the `job-config` service to ignore the job and prevents the system from updating the job status. A workaround is to issue a `POST {"action": "start"}` to the appropriate job actions endpoint, which aborts the stuck job.\
This issue is fixed in Fusion 5.9.14.
* Spark jobs may disappear from the job list after pod deletion\
In Fusion 5.9.12 and 5.9.13, Spark jobs may vanish from the job list if a Spark driver pod is deleted. This behavior can cause confusion and require a job-launcher restart to restore job visibility.\
This issue is fixed in Fusion 5.9.12.
* Jobs for Web V2 connectors may fail to start after an earlier failure.\
A bug in the connectors-backend service can prevent jobs from running if a previous crawl attempt was interrupted—for example, if the connector pod was scaled down mid-job.\
A subsequent crawl attempt may fail with the error `The state should never be null`, even after clearing the datasource configuration.\
This issue is fixed in Fusion 5.9.13.
* The `fusion-spark-3.2.2` image included in Fusion 5.9.12 contains a Fabric8 token refresh bug.\
This issue affects Spark jobs running in Kubernetes environments that rely on token-based authentication.\
Due to an outdated Fabric8 client library, the image fails to refresh Kubernetes tokens correctly, which can cause authentication errors in long-running or distributed Spark jobs.\
This issue is fixed in Fusion 5.9.13.
* The `job-config` service may report a `DOWN` status at `/actuator/health` even when it is fully operational.\
This issue can occur after a prolonged ZooKeeper outage when TLS is enabled.\
Despite the service being healthy and showing `UP` status on readiness and liveness checks, the `/job-config/actuator/health` endpoint may still report `DOWN`, potentially triggering false alarms or unnecessary restarts.\
This issue is fixed in Fusion 5.9.13.
* When upgrading to Fusion 5.9.12, add the following to your `values.yaml` file to avoid a known issue that prevents the `kuberay-operator` pod from launching successfully:
```yaml theme={"dark"}
kuberay-operator:
crd:
create: true
```
* Web connector may fail to index due to corrupted job state\
Fusion 5.9.12 may fail to index with the Webv2 connector (v2.0.1) due to a corrupted job state in the `connectors-backend` service.\
Affected jobs log the error `The state should never be null`, and common remediation steps like deleting the datasource or reinstalling the connector plugin may not resolve the issue.\
The issue is fixed in Fusion 5.9.13.
* Saving new datasource schedules may fail silently.\
In some Fusion 5.9.12 environments, clicking **Save** when adding a schedule from the datasource “Run” dialog does not persist the schedule or show an error message, particularly in apps created before the upgrade.\
As a workaround, use a new app or manually verify that the job configuration was saved.\
This issue is fixed in Fusion 5.9.13.
* The Apache Tika Container parser in Fusion 5.9.12 does not index the body field for SharePoint connector documents, such as DOCS, PPTX, and XLSX files. Documents are split into multiple parts, so the body content only ends up in child documents, while the parent document lacks the body field.
This issue is fixed in Fusion 5.9.15.
* Fusion 5.9.12 introduces updated built-in Solr configsets used by Fusion collections. The managed schema filename is `managed-schema.xml`, with updated configset names such as `_fusion_default`, `_fusion_query_rewrite`, `_fusion_signals`, `_fusion_user_prefs`, and `_fusion_blobs`.
Collections created prior to this change may still reference `managed-schema` without the `.xml` extension. While these collections serve queries normally after upgrade, Fusion UI features and APIs that access Solr configuration files may fail to load schema or configset data.
If you encounter schema or configuration access errors after upgrading, verify that affected collections use `managed-schema.xml` in their ZooKeeper configsets. Lucidworks recommends removing the legacy `managed-schema` file once the XML version is present to avoid ambiguity.
## Removals
For full details on removals, see [Deprecations and Removals](/docs/5/fusion/deprecations-and-removals).
### Bitnami removal
Fusion 5.9.12 will be re-released with the same functionality but updated image references.
In the meantime, Lucidworks will self-host the required images while we work to replace Bitnami images with internally built open-source alternatives.
If you are a self-hosted Fusion customer, *you must upgrade before August 28* to ensure continued access to container images and prevent deployment issues.
You can reinstall your current version of Fusion or upgrade to Fusion 5.9.14, which includes the updated Helm chart and prepares your environment for long-term compatibility.
See [Prevent image pull failures due to Bitnami deprecation in Fusion 5.9.5 to 5.9.13](https://support.lucidworks.com/hc/en-us/articles/33966125467799-Prevent-image-pull-failures-due-to-Bitnami-deprecation-in-Fusion-5-9-5-to-5-9-13) for more information on how to prevent image pull failures.
### Tika Server Parser removal
The Tika Server Parser is removed in this release.
Use the [Tika Asynchronous Parser](/docs/5/fusion/getting-data-in/indexing/asynchronous-tika-parsing) instead. Asynchronous Tika parsing performs parsing in the background. This allows Fusion to continue indexing documents while the parser is processing others, resulting in improved indexing performance for large numbers of documents.
### MLeap removal
MLeap is removed from the `ml-model` service. MLeap was deprecated in Fusion 5.2.0 and was no longer used by Fusion.
## Platform support and component versions
### Kubernetes platform support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.29, 1.30, 1.31
* **Microsoft Azure Kubernetes Service (AKS):** 1.29, 1.30, 1.31
* **Amazon Elastic Kubernetes Service (EKS):** 1.29, 1.30, 1.31
Support is also offered for Rancher Kubernetes Engine (RKE and RKE2) and OpenShift 4 versions that are based on Kubernetes 1.29, 1.30, 1.31; note that RKE2 may require some Helm chart modification. OpenStack and customized Kubernetes installations are *not* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support).
### Component versions
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| Solr | fusion-solr 5.9.12 *(based on Solr 9.6.1)* |
| ZooKeeper | 3.9.1 |
| Spark | 3.4.1 |
| Ingress Controllers | Nginx, Ambassador (Envoy), GKE Ingress Controller |
| Ray | ray\[serve] 2.42.1 |
More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).
# Fusion 5.9.13
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9/5.9.13-release-notes
June 17, 2025
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.9.13-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9.13-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/4dtlz0
Fusion 5.9.13 is a
[maintenance release](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#maintenance-release-support-policy) that introduces
advanced SKU grouping with Solr collapse, custom certificates for indexing and querying services, and compatibility with Kubernetes 1.32.
Fusion 5.9.13 also improves authentication resilience with a configurable JWT timeout, and resolves key scheduling and security bugs to ensure greater stability and compliance in enterprise environments.
**Security patch available for api-gateway: Netty request smuggling vulnerabilities**
A patch is available for the `api-gateway` service to address critical Netty request smuggling vulnerabilities (CVE-2026-42581, CVE-2026-42585, CVE-2026-42587). These vulnerabilities allow attackers to smuggle HTTP requests through the gateway, potentially bypassing security controls.
The `api-gateway` service requires the Netty security patch.
Follow these steps to apply the patched image:
1. Open your Fusion Helm values file.
2. Add or update the `api-gateway` image configuration:
```yaml theme={"dark"}
api-gateway:
image:
repository: lucidworks
name: api-gateway
tag: 5.9.13-SUST-1634-patch
imagePullPolicy: IfNotPresent
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace, `RELEASE_NAME` with your Helm release name, and `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the `api-gateway` pods to restart and verify they are using the patched image.
**Urgent action required by November 26, 2025**
A patch is required by November 26, 2025 for all self-hosted Fusion deployments running on Amazon Elastic Kubernetes Service (EKS). Certain Java versions used by Fusion components reach end of life on this date. Failure to apply the patch will result in compatibility issues.
The following Fusion services require the `cgroupv2` patch:
| Service | Affected Fusion versions | Patch tag |
| - | - | - |
| `insights` | 5.9.4 to 5.9.15 | `lucidworks/insights:5.9-cgroupv2-patch` |
| `spark-solr-etl` | 5.9.4 to 5.9.11 | `lucidworks/spark-solr-etl:5.9-cgroupv2-patch` |
| `keytool-utils` | 5.9.4 to 5.9.10 | `lucidworks/keytool-utils:5.9-cgroupv2-patch` |
Follow these steps to apply the patched images:
1. Open your Fusion Helm values file. For Fusion Cloud Native deployments, use the values file for your current deployment. For Helm deployments, use the values file you used to create the deployment.
2. For each service listed in the following table that applies to your Fusion version, add or update the image configuration:
Expand the following code snippet for the complete image configuration list.
```yaml expandable theme={"dark"}
global:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
sql-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
reverse-search:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
solr:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
kafka:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
ml-model-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-admin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-indexing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
query-pipeline:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
async-parsing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
admin-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
api-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
auth-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
classic-rest-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-resources:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-config:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
insights:
image:
imagePullPolicy: Always
name: insights
repository: lucidworks
tag: 5.9-cgroupv2-patch
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-launcher:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-rest-server:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connector-plugin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors-backend:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
rules-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
pm-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
lwai-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
webapps:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
apps-manager:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
templating:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
tikaserver:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
seldon-core-operator:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo-common-workflows:
image:
imagePullPolicy: Always
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
utilitiesTag: 5.9.11
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: 5.9-cgroupv2-patch
argo-common-workflows:
image:
imagePullPolicy: IfNotPresent
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: insights:5.9-cgroupv2-patch
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run the following command:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace. Replace `RELEASE_NAME` with your Helm release name. Replace `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the affected service pods to restart and verify they are using the patched images.
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
Fusion 5.9.5 introduced changes that affect Spark jobs. If you are upgrading from any Fusion version earlier than 5.9.5 to a version later than 5.9.5, Lucidworks recommends first upgrading to Fusion 5.9.5 before proceeding to the target version.
For supported Kubernetes versions and key component versions, see [Platform support and component versions](#platform-support-and-component-versions).
## What’s new
### Expanded support for collapsed search results
Now Fusion gives you access to all of the available Solr settings for collapsing search results, giving you finer control over how Fusion groups variations of each item into a single search result.
You can use collapse to improve conversion rates and customer satisfaction by streamlining search results, reducing cognitive load, and surfacing the most relevant product variations first.
For example, you can use a `product_id` field as the collapse field to group all versions or SKUs of a product into a single search result.
You can also control how Fusion selects the variation that represents the collapsed group; the default is the one most relevant to the user’s query.
For example, a user who searches for "red shoes" sees all of the red variations of shoes first, with the option to drill down and see all the variations.
Additional capabilities include:
* **Faceting compatibility**: Facets can reflect counts based on collapsed groups instead of individual SKUs.
* **Sorting options**: Choose how the representative SKU is selected using sort fields like `sales_rank` or `popularity_score`.
* **Expand support**: Optional expansion of collapsed groups allows users to see all SKUs for a product on demand.
* **Commerce Studio integration**: Merchandising actions such as pinning, boosting, and burying now apply to the entire product group, not just individual SKUs.
* **Query Workbench support**: You can preview collapsed and expanded result sets directly in Query Workbench for easy validation.
This update eliminates the need for custom collapse implementations and makes SKU/product rollup behavior a first-class capability in Fusion.
For complete details about the new configuration options, see the [Query Fields stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/search-fields-query-stage) configuration reference.
### Kubernetes 1.32 support for better security and long-term compatibility
Fusion 5.9.13 introduces full compatibility with Kubernetes version 1.32, ensuring seamless deployment and operation on the latest Kubernetes platforms.
This update allows you to take advantage of the latest stability, performance, and security improvements in Kubernetes, including better control over sidecar container behavior and improvements to admission webhooks and scheduling logic.
By supporting Kubernetes 1.32, Fusion stays aligned with cloud provider upgrades and helps future-proof your infrastructure, especially on managed services like AKS, EKS, and GKE.
### Improved JWT authentication resilience with configurable timeout
Fusion now allows you to configure the `jwkSetTimeout` variable in the JWT Realm settings, enabling better control over how long Fusion waits for a response when retrieving a JSON Web Key (JWK) set.
This improves authentication reliability in environments where key providers may respond slowly.
By increasing the default 500 ms timeout as needed (for example, to 2000 ms), you can reduce the risk of failed authentication due to network latency or external service delays.
You can configure this in the Fusion UI under the **System > Access Control > Security Realms** tab.
Alternatively, you can set this in the `security.initial-realm-configs` spring boot properties:
```yml theme={"dark"}
security:
initial-realm-configs:
realmType: jwt
enabled: true
name: jwt_okta
config:
autoCreateUsers: true
jwtIssuer: https://HOSTNAME/oauth2/default
jwkSetUri: https://HOSTNAME/oauth2/default/v1/keys
jwkSetTimeout: 2000
roleNames:
- developer
```
### Configurable vector quantization method in LWAI pipeline stages
Fusion 5.9.13 adds vector quantization in certain Lucidworks AI (LWAI) pipeline stages, making it easier to reduce memory usage and accelerate vector search without sacrificing quality.
Quantization converts high-precision float vectors into compact 8-bit integer vectors, significantly lowering storage and compute costs.
Now you can choose between `min-max` or `max-scale` quantization methods directly in the pipeline configuration interface for the LWAI vectorization stages:
* [LWAI Vectorize Field Index stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/lwai-vectorize-field)
* [LWAI Vectorize Query Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/lwai-vectorize-query)
To select the quantization method, go to **Model Config** in the LWAI pipeline stage configuration and enter the `vectorQuantizationMethod` parameter with the value for the desired method:
### Custom certificates for indexing and querying services
Fusion 5.9.13 introduces the ability add custom certificates for indexing and querying services, making it easier to align to your organization’s specific security requirements. This feature allows a Helm chart update to support persistent custom certificates and adding them into truststores during pod startup.
To add a custom certificate, create a new YAML file for your custom certificates and edit it to include your indexing or querying certificates. You must use different YAML files in order to use different certificates for indexing and querying services. See **Deploy Fusion at Scale** for full instructions, including the Helm chart update.
Before you begin, see [Fusion Server Deployment](/docs/5/fusion/operations/deployment) to understand the architecture and requirements.
This article explains how to plan and execute a Fusion deployment at the scale required for staging or production.
While the `setup_f5_*.sh` scripts are handy for getting started and proof-of-concept purposes, this article covers the planning process for building a production-ready environment.
The course for **Preparing for Fusion Implementation** focuses on the key elements for a successful implementation, defining your business requirements, preparing clean data, and involving the right personnel.
## Prerequisites
You must meet the following prerequisites before you can customize your Fusion cluster:
* A local copy of the [fusion-cloud-native repository](https://github.com/lucidworks/fusion-cloud-native). This must be up-to-date with the latest master branch.
* Any cloud provider-specific **command line tools**, such as `gcloud` or `aws`, and `kubectl`.\
See the platform-specific instructions linked above, or check with your cloud provider.
* Helm v3
* To install on a Mac:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
* For other operating systems, download from [Helm Releases](https://github.com/helm/helm/releases).
* Verify your installation:
```bash theme={"dark"}
helm version --short
v3.0.0+ge29ce2a
```
* Kubernetes namespace
* Collect the following information about your Kubernetes environment:
* **CLUSTER**: Cluster name (passed to our setup scripts using the `-c` arg)
* **NAMESPACE**: Kubernetes namespace where to install Fusion; a namespace should only contain lowercase letters (a-z), digits (0-9), or dash. No periods or underscores allowed.
* *(optional)* Clarify your organization’s DockerHub policy. The Fusion Helm chart points to public Docker images on DockerHub. Your organization may not allow Kubernetes to pull images directly from DockerHub or may require extra security scanning before loading images into production clusters.\
Consult your Kubernetes and Docker admin team to find how to get the Fusion images loaded into a registry that’s accessible to your cluster. You can update the image for each service using the [custom values YAML file](#custom-values-yaml-file).
**Kubernetes namespace tips**
* Fusion 5 service discovery requires all services for the same release be deployed in the same namespace. Moreover, you should only run one instance of Fusion in a namespace. If you need multiple instances of Fusion running in the same Kubernetes cluster, then you need to deploy them in separate namespaces.
* If your organization requires CPU / Memory quotas for namespaces, you can start with a minimum of 12 CPU and 45GB of RAM (such as 3 x n1-standard-4 on GKE), but you will need to increase the quotas once you start load testing Fusion with production workloads and real datasets.
* Fusion requires at least 3 ZooKeeper nodes and 2 Solr nodes to achieve high availability.
## Custom values YAML file
1. Clone the `fusion-cloud-native` repository: `git clone https://github.com/lucidworks/fusion-cloud-native`
2. Run the `customize_fusion_values.sh` script.
```bash theme={"dark"}
./customize_fusion_values.sh --provider -c -n \
--num-solr 3 \
--solr-disk-gb 100 \
--node-pool \
--prometheus true \
--with-resource-limits \
--with-affinity-rules
```
Pass the `--help` parameter to see script usage details.
The script creates the following files:
| File | Description |
| - | - |
| `___fusion_values.yaml` | Main custom values YAML used to override Helm chart defaults for Fusion microservices. |
| `___monitoring_values.yaml` | Custom values yaml used to configure Prometheus and Grafana. |
| `___fusion_resources.yaml` | Resource requests and limits for all Microservices. |
| `___fusion_affinity.yaml` | Pod affinity rules to ensure multiple replicas for a single service are evenly distributed across zones and nodes. |
| `___upgrade_fusion.sh` | Script used to install and/or upgrade Fusion using the aforementioned custom values YAML files. |
For an explanation of these placeholder values, see [Configuration Values](#custom-values-yaml-file) below.
3. Add the new files to version control. You will make changes to it over time as you fine-tune your Fusion installation. You will also need it to perform upgrades. If you try to upgrade your Fusion installation and don’t provide the custom values YAML, your deployment will revert to chart defaults.\
Review the `___fusion_values.yaml` file to familiarize yourself with its structure and contents. Notice it contains a separate section for each of the Fusion microservices. The example configuration of the `query-pipeline` service below illustrates some important concepts about the custom values YAML file.
```yaml highlight={1,2,3,5,6} theme={"dark"}
query-pipeline:
enabled: true
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
javaToolOptions: "..."
pod:
annotations:
prometheus.io/port: "8787"
prometheus.io/scrape: "true"
prometheus.io/path: "/actuator/prometheus"
```
4. Service-specific setting overrides under the top-level heading
5. Every Fusion service has an implicit enabled flag that defaults to true, set to false to remove this service from your cluster
6. Node selector identifies the label find nodes to schedule pods on
7. Used to pass JVM options to the service
8. Pod annotations to allow Prometheus to scrape metrics from the service
Once we go through all of the configuration topics in this topic, you’ll have a well-configured custom values YAML file for your Fusion 5 installation. You’ll then use this file during the Helm v3 installation at the end of this topic.
### Deployment-specific values
The script creates a custom values YAML file using the naming convention: `___fusion_values.yaml`. For example, `gke_search_f5_fusion_values.yaml`.
| Parameter | Description |
| - | - |
| `` | The K8s platform you’re running on, such as `gke`. |
| `` | The name of your cluster. |
| `` | The K8s namespace where you want to install Fusion. |
| `` | Specifies a `nodeSelector` label to find nodes to schedule Fusion pods on. |
Providing the correct `--node-pool ` label is very important. Using the wrong value will cause your pods to be stuck in the `pending` state. If you’re not sure about the correct value for your cluster, pass ’'\` to let Kubernetes decide which nodes to schedule Fusion pods on.
Default `nodeSelector` labels are provider-specific. The `fusion-cloud-native` scripts use the following defaults for GKE and EKS:
| Provider | Default node selector |
| - | - |
| GKE | cloud.google.com/gke-nodepool: default-pool |
| EKS | alpha.eksctl.io/nodegroup-name: standard-workers |
If you are deploying Fusion 5.9.12, add the following to your `values.yaml` file to avoid a known issue that prevents the `kuberay-operator` pod from launching successfully: `yaml kuberay-operator: crd: create: true `
### Flags
The script provides flags for additional configuration:
| Flag | Description |
| - | - |
| `--node-pool` | Add a Fusion specific label to your nodes. |
| `--with-resource-limits` | Configure resource requests/limits. |
| `--with-replicas` | Configure replica counts. |
| `--with-affinity-rules` | Configure pod affinity rules for Fusion services. |
Use `--node-pool` to add a Fusion specific label to your nodes by doing:
```bash theme={"dark"}
kubectl label fusion_node_type=
```
Then, pass `--node-pool 'fusion_node_type: '`.
## Configure Solr sizing
When you’re ready to build a production-ready setup for Fusion 5, you need to customize the Fusion Helm chart to ensure Fusion is well-configured for production workloads.
You’ll be able to scale the number of nodes for Solr up and down after building the cluster, but you need to establish the initial size of the nodes (memory and CPU) and the size and type of disks you need.
See the example config below to learn which parameters to change in the custom values YAML file.
```yaml expandable theme={"dark"}
solr:
resources: # Set resource limits for Solr to help K8s pod scheduling;
limits: # these limits are not just for the Solr process in the pod,
cpu: "7700m" # so allow ample memory for loading index files into the OS cache (mmap)
memory: "26Gi"
requests:
cpu: "7000m"
memory: "25Gi"
logLevel: WARN
nodeSelector:
fusion_node_type: search # Run this Solr StatefulSet in the "search" node pool
exporter:
enabled: true # Enable the Solr metrics exporter (for Prometheus) and
# schedule on the default node pool (system partition)
podAnnotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9983"
prometheus.io/path: "/metrics"
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
image:
tag: 8.4.1
updateStrategy:
type: "RollingUpdate"
javaMem: "-Xmx3g -Dfusion_node_type=system" # Configure memory settings for Solr
solrGcTune: "-XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UseStringDeduplication -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=150 -XX:+UseLargePages -XX:+AlwaysPreTouch"
volumeClaimTemplates:
storageSize: "100Gi" # Size of the Solr disk
replicaCount: 6 # Number of Solr pods to run in this StatefulSet
zookeeper:
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
replicaCount: 3 # Number of Zookeepers
persistence:
size: 20Gi
resources: {}
env:
ZK_HEAP_SIZE: 1G
ZOO_AUTOPURGE_PURGEINTERVAL: 1
```
To be clear, you can tune GC settings and number of replicas after the cluster is built. But changing the size of the persistent volumes is more complicated so you should try to pick a good size initially.
### Configure storage class for Solr pods (optional)
If you wish to run with a storage class other than the default you can create a storage class for your Solr pods before you install. For example, to create regional disks in GCP you can create a file called `storageClass.yaml` with the following contents:
```yaml theme={"dark"}
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: solr-gke-storage-regional
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-standard
replication-type: regional-pd
zones: us-west1-b, us-west1-c
```
and then provision into your cluster by calling:
```bash theme={"dark"}
kubectl apply -f storageClass.yaml
```
to then have Solr use the storage class by adding the following to the custom values YAML:
```yaml theme={"dark"}
solr:
volumeClaimTemplates:
storageClassName: solr-gke-storage-regional
storageSize: 250Gi
```
We’re not advocating that you must use regional disks for Solr storage, as that would be redundant with Solr replication. We’re just using this as an example of how to configure a custom storage class for Solr disks if you see the need. For instance, you could use regional disks without Solr replication for write-heavy type collections.
## Configure multiple node pools
Lucidworks recommends isolating search workloads from analytics workloads using multiple node pools. The included scripts do not do this for you; this is a manual process.
See the example script for GKE, see [create\_gke\_cluster\_node\_pools.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/additional_environments/create_gke_cluster_node_pools.sh).
In the custom values YAML file, you can add additional Solr StatefulSets by adding their names to the list under the `nodePools` property. If any property for that statefulset needs to be changed from the default set of values, then it can be set directly on the object representing the node pool, any properties that are omitted are defaulted to the base value. See the following example (additional whitespace added for display purposes only):
```yaml expandable highlight={3,4,9,17,22,31} theme={"dark"}
solr:
nodePools:
- name: ""
- name: "analytics"
javaMem: "-Xmx6g"
replicaCount: 6
storageSize: "100Gi"
nodeSelector:
fusion_node_type: analytics
resources:
requests:
cpu: 2
memory: 12Gi
limits:
cpu: 3
memory: 12Gi
- name: "search"
javaMem: "-Xms11g -Xmx11g"
replicaCount: 12
storageSize: "50Gi"
nodeSelector:
fusion_node_type: search
resources:
limits:
cpu: "7700m"
memory: "26Gi"
requests:
cpu: "7000m"
memory: "25Gi"
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
...
```
1. The empty string `""` is the suffix for the default partition.
2. Overrides the settings for the **analytics** Solr pods.
3. Assigns the **analytics** Solr pods to the node pool and attaches the label `fusion_node_type=analytics`. You can use the `fusion_node_type` property in Solr auto-scaling policies to govern replica placement during collection creation.
4. Overrides the settings for the **search** Solr pods.
5. Assigns the **search** Solr pods to the node pool and attaches the label `fusion_node_type=search`.
6. Sets the default settings for all Solr pods, if not specifically overridden in the `nodePools` section above.
Do not edit the `nodePools` value `""`.
In the example above, the **analytics** partition `replicaCount`, or number of Solr pods, is six. The **search** partition `replicaCount` is twelve.
Each nodePool is automatically be assigned the -Dfusion\_node\_type property of ``, ``, or ``. This value matches the name of the nodePool. For example, `-Dfusion_node_type=`.
The Solr pods have a `fusion_node_type` system property, as shown below:
## Solr auto-scaling policy
Use [replica placement plugins](https://solr.apache.org/guide/solr/latest/configuration-guide/replica-placement-plugins.html) to control how replicas are placed in Solr.
## Pod network policy
A Kubernetes network policy governs how groups of pods communicate with each other and other network endpoints. With Fusion, all incoming traffic flows through the API Gateway service. All Fusion services in the same namespace expect an internal JWT, which is supplied by the Gateway, as part of the request. As a result, Fusion services enforce a basic level of API security and don’t need an additional network policy to protect them from other pods in the cluster.
To install the network policy for Fusion services, pass `--set global.networkPolicyEnabled=true` when installing the Fusion Helm chart.
## On-premises private Docker registries
For on-premises Kubernetes deployments, your organization may not allow Kubernetes to pull [Fusion’s Docker images from DockerHub](https://hub.docker.com/u/lucidworks/). See the instructions below for details on using a private Docker registry with Fusion. These are general instructions that may need to be adapted to work within your organization’s security policies:
1. Transfer the public images from DockerHub to your private Docker registry.
2. Establish a workstation that has access to [DockerHub](https://hub.docker.com). This workstation must connect to your internal Docker registry, most likely via VPN connection. In this example, the workstation is referred to as `envoy`.
3. Install Docker on `envoy`. You need at least 100GB of free disk for Docker.
4. Pull all of the images from DockerHub to `envoy`’s local registry. For example, to pull the query pipeline image, run `docker pull lucidworks/query-pipeline:5.9.0`. See `docker pull --help` for more information about pulling Docker images.
5. Establish a connection from `envoy` to the private Docker registry, most likely via a VPN connection. In this example, the private Docker registry is referred to as ``.
6. Push the images from `envoy`’s Docker registry to the private registry. This will take a long time.
1. You’ll need to re-tag all images for the internal registry. For example, to tag the query-pipeline image, run:
```bash theme={"dark"}
docker tag lucidworks/query-pipeline:5.9.0 /query-pipeline:5.9.0
```
2. Push each image to the internal repo:
```bash theme={"dark"}
docker push /query-pipeline:5.9.0
```
7. Install the Docker registry secret in Kubernetes. Create the Docker registry secret in the Kubernetes namespace where you want to install Fusion:
```bash theme={"dark"}
SECRET_NAME=
REPO=
kubectl create secret docker-registry "${SECRET_NAME}" \
--namespace "${NAMESPACE}" \
--docker-server="${REPO}" \
--docker-username=${REPO_USER} \
--docker-password=${REPO_PASS} \
--docker-email=${REPO_USER}
```
For details, see the Kubernetes article [Pull an Image from a Private Registry](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/).
8. Update the custom values YAML for your cluster to point to your private registry and secret to allow Kubernetes to pull images. For example:
```yaml theme={"dark"}
query-pipeline:
image:
imagePullSecrets:
- name:
repository:
```
Repeat the process for all Fusion services.
### Customize Helm Chart
Every Fusion service allows you to override the `imagePullSecrets` setting using custom values YAML. However, other 3rd party services--including Zookeeper, Pulsar, Prometheus, and Grafana--don’t allow you to supply the pull secret using the custom values YAML.
To patch the default service account for your namespace and add the pull secret, run the following:
```bash theme={"dark"}
kubectl patch sa default -n $NAMESPACE \
-p '"imagePullSecrets": [{"name": "" }]'
```
In Windows using PowerShell or another CLI, you might have to escape the double quotes with a backslash (`\`) or reverse the order of single and double quotes:
```bash theme={"dark"}
kubectl patch sa default -n $NAMESPACE \
-p "'imagePullSecrets': [{'name': ''}]"
```
Replace `` with the name of the secret you created in the steps above.
This allows the default service account to pull images from the private registry without specifying the pull secret on the resources directly.
## Add additional trusted certificate(s) to Fusion’s indexing and querying services *(optional)*
You can add custom trusted certificates to support Fusion’s indexing and querying services. You may want to use custom trusted certificates if, for example, you have specific security requirements for data handling or need to support an existing infrastructure and its security needs. This method involves updating your Helm chart.
If you want to add custom trusted certificates for both the indexing and querying services, follow these instructions twice: once for the indexing service, and once for the querying service. To add different certificates to the indexing and querying services, create one YAML file with the indexing service certificates and one YAML file for the querying service certificates before following these instructions.
You may use the same YAML file if you want to use the same certificates for both services.
To add custom trusted certificates:
1. Create a new YAML file for your custom trusted certificates. The `customcerts.yaml` file is the example file in these instructions.
2. Add the custom certificate(s) in the YAML file created in the previous step. For example:
```yaml theme={"dark"}
trustedCertificates:
enabled: true
files:
some.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE-----
other.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE---------
```
3. Update the indexing or querying service by running the following Helm command. Replace `EXAMPLE-VALUES-FILE.yaml` with your previous values file.
```bash theme={"dark"}
helm upgrade --install --namespace ${EXAMPLE-NAMESPACE} ${HELM-RELEASE} ${HELM-CHART-PATH} --values EXAMPLE-VALUES-FILE.yaml --values customcerts.yaml
```
4. Verify the indexing or querying pod has a new `init-container` with the name `import-certs`.
## Add additional trusted certificate(s) for connectors to allow crawling of web resources with SSL/TLS enabled *(optional)*
To crawl a datasource which for some reason is using a self-signed certificate, add arbitrary certificates to connectors. For example:
```yaml wrap expandable theme={"dark"}
classic-rest-service:
trustedCertificates:
enabled: true
files:
some.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE-----
other.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE---------
connector-plugin:
trustedCertificates:
enabled: true
files:
some.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE-----
other.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE---------
```
### Generating the certificate on linux command line
Use the following command to generate a `.crt` file in `$fusion_home/apps/jetty/connectors/etc/yourcertname.crt`:
```bash theme={"dark"}
openssl s_client -servername remote.server.net -connect remote.server.net:443 $fusion_home/apps/jetty/connectors/etc/yourcertname.crt
```
### Generating the certificate using Firefox web browser
1. Navigate to the SharePoint host.
2. Click the in the address bar, then click the icon.
3. Next, navigate to **More Information** > **View Certificate** > **Export**.\
Save the file to the following folder:
`$fusion_home/apps/jetty/connectors/etc/yourcertname.crt`
### Generating the certificate using Chrome web browser
1. Navigate to **Chrome menu** > **More Tools** > **Developer Tools** > **Security Tab**.
This will display the **Security overview**.
2. Click the **View certificate** button.
3. Save the file to the following folder:
`$fusion_home/apps/jetty/connectors/etc/yourcertname.crt`
### Generating the certificate using powershell
Use the following script to generate a `.crt`` file in `\$fusion\_home\apps\jetty\connectors\etc\yourcertname.crt\`\`:
```bash theme={"dark"}
$fusion_home = c:\your\fusion\install\directory
$webRequest = [Net.WebRequest]::Create("https://your-hostname")
try { $webRequest.GetResponse() } catch {}
$cert = $webRequest.ServicePoint.Certificate
$bytes = $cert.Export([Security.Cryptography.X509Certificates.X509ContentType]::Cert)
set-content -value $bytes -encoding byte -path "$fusion_home\apps\jetty\connectors\etc\yourcertname.binary.crt"
certutil -encode "$fusion_home\apps\jetty\connectors\etc\yourcertname.binary.crt" "$fusion_home\apps\jetty\connectors\etc\yourcertname.crt"
rm "$fusion_home\apps\jetty\connectors\etc\yourcertname.binary.crt" -f
```
## Install Fusion 5 on Kubernetes
At this point, you’re ready to install Fusion 5 using the custom values YAML files and upgrade script. If you used the `customize_fusion_values.sh` script, run it using BASH:
```bash theme={"dark"}
./gke_search_f5_upgrade_fusion.sh
```
Once the installation is complete, verify your Fusion installation is running correctly.
## Monitoring Fusion with Prometheus and Grafana
Lucidworks recommends using Prometheus and Grafana for monitoring the performance and health of your Fusion cluster. Your operations team may already have these services installed. If not, install them into the Fusion namespace.
The [Custom values YAML file shown above](#custom-values-yaml-file) activates the Solr metrics exporter service and adds pod annotations so Prometheus can scrape metrics from Fusion services.
1. Run the `customize_fusion_values.sh` script with the `--prometheus true` option. This creates an extra custom values YAML file for installing Prometheus and Grafana, `___monitoring_values.yaml`. For example: `gke_search_f5_monitoring_values.yaml`.
2. Commit the YAML file to version control.
3. Review its contents to ensure that the settings suit your needs. For example, decide how long you want to keep metrics. The default is 36 hours.\
See the [Prometheus documentation](https://github.com/helm/charts/tree/master/stable/prometheus) and [Grafana documentation](https://github.com/helm/charts/tree/master/stable/grafana) for details.
4. Run the `install_prom.sh` script to install Prometheus & Grafana in your namespace. Include the provider, cluster name, namespace, and helm release as in the example below:
```bash theme={"dark"}
./install_prom.sh --provider gke -c search -n f5 -r 5-5-1
```
Pass the `--help` parameter to see script usage details.
The Grafana dashboards from [monitoring/grafana](https://github.com/lucidworks/fusion-cloud-native/tree/master/monitoring/grafana) are installed automatically by the `install_prom.sh` script.
### Support for pre-filtering in the Chunking Neural Hybrid Query stage
For parity with the [Neural Hybrid Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/neural-hybrid-query), the [Chunking Neural Hybrid Query Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/chunking-neural-hybrid-query) now supports pre-filtering.
Pre-filtering can improve performance by reducing the number of chunks that need to be processed.
However, in some cases it can also lead to less accurate facet counts and search results.
Pre-filtering is blocked by default.
You can enable it by unchecking the **Block pre-filtering** checkbox in the Chunking Neural Hybrid Query stage configuration.
### Default Solr data commit parameter changed to autoCommit
Solr's `commitWithin` parameter was removed as the default at the collection level. The default parameter is now `autoCommit`.
Solr offers three methods to determine when newly-indexed documents are made visible in searches. The methods are manual commits, auto commit settings, and the per-update `commitWithin` parameter.
Fusion relies on Solr's commit mechanisms, but adds a Fusion-managed `commitWithin` policy per collection. Fusion's typical commit configuration includes the following parameters:
* The `autoCommit` setting (Solr hard commit) is Fusion's default parameter for Solr data commits, is inherited from the collection's `solrconfig.xml`, and defaults to `15` seconds. This saves the data, but does not force the search results to refresh immediately. With this setting, search performance is not slowed, but the new data may not show in search results until the next refresh. This property can be used instead of the `commitWithin` property.
* The `autoSoftCommit` setting (Solr soft commit) does not save the data, but makes the data visible to searches almost immediately. If the system crashes, that new data is lost because it has not been saved. The default is false, and search visibility is managed using the `commitWithin` setting.
* Solr's `commitWithin` parameter, configured at the collection level with a default of `10000` milliseconds, ensures data is committed and available for searching within 10 seconds. The default for signals collections is `1000` milliseconds. Fusion uses `commitWithin` to avoid relying on specific Solr-side configurations. The `com.lucidworks.apollo.solr.commitWithin` global property defines the default `commitWithin` for all documents added through Fusion. When you create a new collection in Fusion, its per-collection `commitWithin` is initialized from this global default.
For more information, see [Collection configuration properties](/docs/5/fusion/getting-data-in/indexing/collections/overview#collection-configuration-properties) and [various operations in the Collections API](/api-reference/collections/list-all-collections).
## Bug fixes
* Corrected the health reporting behavior for the `job-config` service after ZooKeeper disruptions.\
Fusion now ensures the `/actuator/health` endpoint correctly reflects the actual status of the `job-config` service, even after temporary ZooKeeper unavailability.\
This prevents false `DOWN` reports that could affect monitoring or automated recovery systems.
* Updated the `fusion-spark-3.2.2` image to resolve a Fabric8 token refresh bug.\
The Fabric8 Kubernetes client in this Spark image has been patched to fix a bug that prevented token refresh under OIDC authentication.\
This ensures that Spark jobs using `fusion-spark-3.2.2` run reliably in Kubernetes environments that require token-based authentication.
* Fixed a bug that prevented Web V2 connector jobs from restarting after failure.\
In previous versions, if a job was interrupted (such as by scaling down the connector pod), the connectors-backend service could enter a corrupted state, preventing future runs of the same job with errors like `The state should never be null`.\
Fusion now properly resets internal job state, ensuring that failed jobs can be restarted reliably.
* Fixed Web connector indexing failure caused by corrupted job state.\
Fusion 5.9.13 restores indexing functionality for the Webv2 connector (v2.0.1) by resolving an issue that caused a corrupted job state in the `connectors-backend` service.
Jobs that previously failed with `The state should never be null` can now complete successfully.
* Fixed an issue that prevented schedule changes from persisting for some datasources.\
In Fusion 5.9.12, clicking **Save** after configuring a new schedule for a datasource in the “Run” dialog could fail silently in certain apps, leaving the schedule unsaved with no warning to the user.\
This was due to a `job-config` handling issue that affected pre-existing app configurations.\
Fusion 5.9.13 resolves this issue so that new schedules are reliably saved and acknowledged as expected.
* Fixed permission handling in the `job-config` service to ensure scheduled jobs run as expected.\
Fusion now correctly handles permission checks when creating or modifying scheduled jobs, preventing failures caused by mismatches between user and service account permissions.\
This resolves issues where job could not be scheduled or executed following upgrades.
* Helm charts now support Kubernetes secrets for TLS keystore passwords.\
Fusion 5.9.13 updates the Helm charts to eliminate the use of plaintext passwords for TLS keystores. You can now securely manage the keystorePassword using a Kubernetes secret, aligning with hardened OpenShift and enterprise security policies.
* Upgraded the Spring framework in the `web-apps` service to improve security and ensure compatibility with token authentication behavior on modern Kubernetes platforms.
## Known issues
* Job-config service may appear “down” in UI even when running correctly\
In Fusion 5.9.13, the job-config service may falsely report as “down” in the Fusion UI, particularly during startup or in TLS-enabled deployments.\
This issue is fixed in Fusion 5.9.14.
For the `job-config` service, the **Next Run** value is updated *only after the current job completes*. Because the same job cannot run multiple times simultaneously, if another run is scheduled before the current job completes, that run is not executed.
* Jobs and V2 datasources may fail when Fusion collections are remapped to different Solr collections.\
In Fusion 5.9.13, strict validation in the `job-config` service causes “Collection not found” errors when jobs or V2 datasources target Fusion collections that point to differently named Solr collections.
This issue is fixed in Fusion 5.9.14.
As a workaround, use V1 datasources or avoid using REST call jobs on remapped collections.
* Saving large query pipelines may cause OOM failures under high load.\
In Fusion 5.9.13, saving a large query pipeline during high query volume can result in thread lock, excessive memory use, and eventual OOM errors in the Query service.\
This issue is fixed in Fusion 5.9.14.
* Some S3 and Web datasource jobs cannot be stopped in Fusion 5.9.13.\
Clicking the Stop button has no effect in some cases where the backend job is no longer being tracked. This causes the `job-config` service to ignore the job and prevents the system from updating the job status. A workaround is to issue a `POST {"action": "start"}` to the appropriate job actions endpoint, which aborts the stuck job.\
This issue is fixed in Fusion 5.9.14.
* Spark jobs may disappear from the job list after pod deletion\
In Fusion 5.9.12 and 5.9.13, Spark jobs may vanish from the job list if a Spark driver pod is deleted. This behavior can cause confusion and require a job-launcher restart to restore job visibility.\
This issue is fixed in Fusion 5.9.13.
## Deprecations and removals
For full details, see [Deprecations and Removals](/docs/5/fusion/deprecations-and-removals).
### Bitnami removal
Fusion 5.9.13 will be re-released with the same functionality but updated image references.
In the meantime, Lucidworks will self-host the required images while we work to replace Bitnami images with internally built open-source alternatives.
If you are a self-hosted Fusion customer, *you must upgrade before August 28* to ensure continued access to container images and prevent deployment issues.
You can reinstall your current version of Fusion or upgrade to Fusion 5.9.14, which includes the updated Helm chart and prepares your environment for long-term compatibility.
See [Prevent image pull failures due to Bitnami deprecation in Fusion 5.9.5 to 5.9.13](https://support.lucidworks.com/hc/en-us/articles/33966125467799-Prevent-image-pull-failures-due-to-Bitnami-deprecation-in-Fusion-5-9-5-to-5-9-13) for more information on how to prevent image pull failures.
### Hybrid Query pipeline stage
The [Hybrid Query pipeline stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/hybrid-search-query) is now deprecated.
Instead, use the [Neural Hybrid Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/neural-hybrid-query), which combines lexical and vector search and includes improvements such as K-Nearest Neighbors (KNN), chunking, and more.
### Container image for `kafka-exporter`
Bitnami stopped supporting `kafka-exporter` and renamed the [docker repository](https://hub.docker.com/r/bitnamilegacy/kafka-exporter-archived) from `bitnami/kafka-exporter` to `bitnami/kafka-exporter-archived`.
If you were using `kafka-exporter`, then you need to update the repository name in your `values.yaml` file:
```yaml theme={"dark"}
kafka.metrics.kafka.image.repository=bitnami/kafka-exporter-archived
```
Changing the repository name will allow you to continue using Kafka metrics evaluation with Prometheus/Grafana.
### Removed deprecated `X-XSS-Protection` header from session API responses
Fusion 5.9.13 removes the deprecated `X-XSS-Protection` HTTP response header from the session API.
This header is no longer supported by modern browsers and has no effect on security behavior.
Its removal helps avoid confusion during security audits and aligns with current web security standards.
## Platform support and component versions
### Kubernetes platform support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.29, 1.30, 1.31, 1.32
* **Microsoft Azure Kubernetes Service (AKS):** 1.29, 1.30, 1.31, 1.32
* **Amazon Elastic Kubernetes Service (EKS):** 1.29, 1.30, 1.31, 1.32
Support is also offered for Rancher Kubernetes Engine (RKE and RKE2) and OpenShift 4 versions that are based on Kubernetes 1.29, 1.30, 1.31, 1.32; note that RKE2 may require some Helm chart modification. OpenStack and customized Kubernetes installations are *not* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support).
### Component versions
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| Solr | fusion-solr 5.9.13 *(based on Solr 9.6.1)* |
| ZooKeeper | 3.9.1 |
| Spark | 3.4.1 |
| Ingress Controllers | Nginx, Ambassador (Envoy), GKE Ingress Controller |
| Ray | ray\[serve] 2.42.1 |
More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).
# Fusion 5.9.14
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9/5.9.14-release-notes
Fusion 5.9.14 is a
[maintenance release](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#maintenance-release-support-policy) that
adds official support for Kubernetes 1.33,
introduces the new Local Chunker index stage for Ray-hosted model support,
expands observability and async processing for pipelines, and
improves operational tooling with structured JSON logs.
This release also includes numerous internal security updates and system hardening improvements.
Upgrading to the latest version of Fusion 5.9 offers several key benefits:
* **Access to latest features:** Stay current with the latest features and functionality to ensure compatibility and optimal performance.
* **Simplified process:** Fusion 5.9.5 and later use an in-place upgrade strategy, making upgrades easier than ever.
* **Extended support:** Upgrading keeps you up-to-date with the latest supported Kubernetes versions, as outlined in the [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle) policy.
**Security patch available for api-gateway: Netty request smuggling vulnerabilities**
A patch is available for the `api-gateway` service to address critical Netty request smuggling vulnerabilities (CVE-2026-42581, CVE-2026-42585, CVE-2026-42587). These vulnerabilities allow attackers to smuggle HTTP requests through the gateway, potentially bypassing security controls.
The `api-gateway` service requires the Netty security patch.
Follow these steps to apply the patched image:
1. Open your Fusion Helm values file.
2. Add or update the `api-gateway` image configuration:
```yaml theme={"dark"}
api-gateway:
image:
repository: lucidworks
name: api-gateway
tag: 5.9.14-SUST-1634-patch
imagePullPolicy: IfNotPresent
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace, `RELEASE_NAME` with your Helm release name, and `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the `api-gateway` pods to restart and verify they are using the patched image.
**Urgent action required by November 26, 2025**
A patch is required by November 26, 2025 for all self-hosted Fusion deployments running on Amazon Elastic Kubernetes Service (EKS). Certain Java versions used by Fusion components reach end of life on this date. Failure to apply the patch will result in compatibility issues.
The following Fusion services require the `cgroupv2` patch:
| Service | Affected Fusion versions | Patch tag |
| - | - | - |
| `insights` | 5.9.4 to 5.9.15 | `lucidworks/insights:5.9-cgroupv2-patch` |
| `spark-solr-etl` | 5.9.4 to 5.9.11 | `lucidworks/spark-solr-etl:5.9-cgroupv2-patch` |
| `keytool-utils` | 5.9.4 to 5.9.10 | `lucidworks/keytool-utils:5.9-cgroupv2-patch` |
Follow these steps to apply the patched images:
1. Open your Fusion Helm values file. For Fusion Cloud Native deployments, use the values file for your current deployment. For Helm deployments, use the values file you used to create the deployment.
2. For each service listed in the following table that applies to your Fusion version, add or update the image configuration:
Expand the following code snippet for the complete image configuration list.
```yaml expandable theme={"dark"}
global:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
sql-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
reverse-search:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
solr:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
kafka:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
ml-model-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-admin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-indexing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
query-pipeline:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
async-parsing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
admin-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
api-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
auth-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
classic-rest-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-resources:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-config:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
insights:
image:
imagePullPolicy: Always
name: insights
repository: lucidworks
tag: 5.9-cgroupv2-patch
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-launcher:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-rest-server:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connector-plugin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors-backend:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
rules-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
pm-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
lwai-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
webapps:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
apps-manager:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
templating:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
tikaserver:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
seldon-core-operator:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo-common-workflows:
image:
imagePullPolicy: Always
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
utilitiesTag: 5.9.11
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: 5.9-cgroupv2-patch
argo-common-workflows:
image:
imagePullPolicy: IfNotPresent
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: insights:5.9-cgroupv2-patch
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run the following command:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace. Replace `RELEASE_NAME` with your Helm release name. Replace `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the affected service pods to restart and verify they are using the patched images.
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
Fusion 5.9.5 introduced changes that affect Spark jobs. If you are upgrading from any Fusion version earlier than 5.9.5 to a version later than 5.9.5, Lucidworks recommends first upgrading to Fusion 5.9.5 before proceeding to the target version.
For supported Kubernetes versions and key component versions, see [Platform support and component versions](#platform-support-and-component-versions).
## Key highlights
### Fusion now supports Kubernetes 1.33 and kubectl 1.33
Fusion 5.9.14 adds official support for Kubernetes 1.33, including an upgrade to `kubectl` 1.33.
This ensures compatibility with the latest Kubernetes platform features, security patches, and performance improvements.
By aligning with Kubernetes 1.33:
* Platform compatibility is expanded, allowing you to deploy or upgrade Fusion in modern, compliant Kubernetes environments.
* Security posture is improved, as support for current Kubernetes versions ensures you benefit from upstream fixes and hardened defaults.
* Operational reliability is enhanced, reducing friction during upgrades and simplifying lifecycle management.
This update future-proofs your deployment infrastructure and allows DevOps teams to adopt the latest Kubernetes tooling without workarounds.
### New Local Chunker index stage
Fusion 5.9.14 adds a new [Local Chunker index pipeline stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/local-chunker-index-stage) that performs document chunking using models hosted in your own [Ray](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/local-chunker-index-stage#develop-and-deploy-a-chunking-machine-learning-model-with-ray) infrastructure as an alternative to SaaS-based Lucidworks AI.
This makes it easier to deploy RAG pipelines in fully self-hosted environments.
This stage gives you control over chunk processing and storage, with the option to include or exclude chunk text depending on your deployment preferences.
By default, it uses the vector context variable as input, aligning with the output of Ray-hosted models.
### Asynchronous processing for LWAI Vectorize Query and LWAI Prediction pipeline stages
To help reduce query latency and improve overall performance, the [LWAI Vectorize Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/lwai-vectorize-query) and [LWAI Prediction](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/lucidworks-ai-prediction-query-stage) stages now support asynchronous execution.
This enhancement allows more efficient parallel processing in pipelines that use multiple AI-powered stages.
As with other async-enabled stages, you can aggregate results using the [Merge Async Results](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/merge-async-results-query-stage) stage.
### Standardized component logging using structured JSON format
Fusion services can now log in structured JSON format instead of plaintext (Log4j-style) output. This improves compatibility with monitoring tools and log aggregation systems by making logs easier to parse, filter, and analyze.
JSON logging is supported for the following services:
* `admin`
* `apps-manager`
* `connectors`
* `connectors-backend`
* `connectors-classic`
* `distributed-compute` (`job-launcher`, `job-rest-server`)
* `indexing`
* `job-config`
* `ml-model-service` (including `kuberay-operator`, `seldon-webhook-service`)
* `proxy` / `api-gateway`
* `query`
* `solr`
* `templating`
* `webapps`
JSON logging is off by default.
To enable it, you can set `jsonOutput: true` globally or for specific services in the `values.yaml` configuration file.
This update does not affect log formats in the following services:
* `admin-ui`
* `auth-ui`
* `insights`
* `pm-ui`
* `rules-ui`
* `argo`
* `kafka`
* `spark`
* `zookeeper`
### Improved visibility for Solr ingestion errors
Fusion 5.9.14 enhances logging for Solr indexing. These improvements make it easier to identify and address problematic documents without losing valid ones:
* The system logs detailed information about each document that fails to index, including the cause of the error.
* Logs use a structured, machine-readable format, making them easier to parse and monitor.
* Indexing continues for valid documents even when some fail, preventing unnecessary data loss.
These changes help you troubleshoot ingestion issues faster, trace failures to specific documents, and keep indexing pipelines running smoothly in self-hosted Fusion environments.
### Security improvements
Fusion 5.9.14 includes a broad set of internal security fixes and hardening updates. These updates improve the overall security posture of the platform by addressing potential vulnerabilities and ensuring better alignment with secure development best practices.
This ongoing work helps protect your deployments from evolving threats and ensures Fusion continues to meet enterprise-grade security standards.
## Bug fixes
* Fixed an issue where certain datasource jobs could not be stopped.
Fusion 5.9.14 resolves this issue and restores proper job control behavior.
* Fixed excessive debug logging in Spark driver pods.
Fusion 5.9.14 reduces logging verbosity by default.
* Fixed pipeline errors in Commerce Studio caused by empty or invalid collection names.
Fusion 5.9.14 resolves this issue by correcting the thread handling logic, validating collection name inputs, and raising logging levels to warn when invalid configurations are detected, helping you troubleshoot more effectively and avoid unnecessary pipeline failures.
* Fixed an issue where Spark jobs would disappear from Fusion after pod deletion.
Fusion 5.9.14 ensures Spark jobs remain visible and manageable even after pod restarts.
* Fixed out-of-memory errors when saving large query pipelines under high load.
Fusion 5.9.14 resolves this issue by disabling unnecessary JMX monitoring in script engine pools and improving thread synchronization, ensuring pipeline changes no longer disrupt high-RPS environments.
* Fixed job and datasource errors when Fusion collections pointed to remapped Solr collections.
Previously, jobs and V2 datasources failed with "Collection not found" errors when a Fusion collection pointed to a Solr collection with a different name.
Fusion 5.9.14 removes this restriction, allowing jobs and datasources to work with valid Fusion collections regardless of Solr collection naming, restoring expected behavior and improving compatibility with remapped environments.
* Fixed incorrect "down" status reporting in the `job-config` service.
In some cases, the `job-config` service could incorrectly report a "down" status in the Fusion UI, even though the service was healthy and running.
Fusion 5.9.14 corrects this reporting error so the UI reflects the true status of the service, making it easier to monitor system health and avoid unnecessary debugging.
For the `job-config` service, the **Next Run** value is updated *only after the current job completes*. Because the same job cannot run multiple times simultaneously, if another run is scheduled before the current job completes, that run is not executed.
## Known issues
In some asynchronous pipeline stages, the "Asynchronous Execution Config" checkbox is misaligned such that the checkbox is not directly clickable.
To work around this issue, click on the checkbox title, "Asynchronous Execution Config".
The checkbox alignment is fixed in Fusion 5.9.15.
## Removals
### Bitnami removal
Starting August 28, 2025, Bitnami will be removed from Fusion's Helm charts due to changes in how they host images.
Fusion 5.9.14 includes an updated Helm chart with new image references.
Upgrading to this release ensures continued access to container images and prevents deployment issues.
See [Docker Images by Fusion Version](/docs/5/fusion/reference/docker-images-versions) for the complete list of container images.
## Platform support and component versions
### Kubernetes platform support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.29, 1.30, 1.31, 1.32, 1.33
* **Microsoft Azure Kubernetes Service (AKS):** 1.29, 1.30, 1.31, 1.32, 1.33
* **Amazon Elastic Kubernetes Service (EKS):** 1.29, 1.30, 1.31, 1.32, 1.33
Support is also offered for Rancher Kubernetes Engine (RKE and RKE2) and OpenShift 4 versions that are based on Kubernetes 1.29, 1.30, 1.31, 1.32, 1.33; note that RKE2 may require some Helm chart modification. OpenStack and customized Kubernetes installations are *not* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support).
### Component versions
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| Solr | fusion-solr 5.9.14 *(based on Solr 9.6.1)* |
| ZooKeeper | 3.9.1 |
| Spark | 3.4.1 |
| Ingress Controllers | Nginx, Ambassador (Envoy), GKE Ingress Controller |
| Ray | ray\[serve] 2.46.0 |
More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).
# Fusion 5.9.15
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9/5.9.15-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.9.15-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9.15-release-notes
Released on October 16, 2025, Fusion 5.9.15 is a [maintenance release](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#maintenance-release-support-policy) that includes scalability enhancements to LWAI stages, accuracy improvements in the Consumption Dashboard entitlement reporting, expanded monitoring metrics for asynchronous query stages, security fixes, and updated UI behavior.
Upgrading to the latest version of Fusion 5.9 offers several key benefits:
* **Access to latest features:** Stay current with the latest features and functionality to ensure compatibility and optimal performance.
* **Simplified process:** Fusion 5.9.5 and later use an in-place upgrade strategy, making upgrades easier than ever.
* **Extended support:** Upgrading keeps you up-to-date with the latest supported Kubernetes versions, as outlined in the [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle) policy.
**Security patch available for api-gateway: Netty request smuggling vulnerabilities**
A patch is available for the `api-gateway` service to address critical Netty request smuggling vulnerabilities (CVE-2026-42581, CVE-2026-42585, CVE-2026-42587). These vulnerabilities allow attackers to smuggle HTTP requests through the gateway, potentially bypassing security controls.
The `api-gateway` service requires the Netty security patch.
Follow these steps to apply the patched image:
1. Open your Fusion Helm values file.
2. Add or update the `api-gateway` image configuration:
```yaml theme={"dark"}
api-gateway:
image:
repository: lucidworks
name: api-gateway
tag: 5.9.15-SUST-1634-patch
imagePullPolicy: IfNotPresent
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace, `RELEASE_NAME` with your Helm release name, and `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the `api-gateway` pods to restart and verify they are using the patched image.
**Urgent action required by November 26, 2025**
A patch is required by November 26, 2025 for all self-hosted Fusion deployments running on Amazon Elastic Kubernetes Service (EKS). Certain Java versions used by Fusion components reach end of life on this date. Failure to apply the patch will result in compatibility issues.
The following Fusion services require the `cgroupv2` patch:
| Service | Affected Fusion versions | Patch tag |
| - | - | - |
| `insights` | 5.9.4 to 5.9.15 | `lucidworks/insights:5.9-cgroupv2-patch` |
| `spark-solr-etl` | 5.9.4 to 5.9.11 | `lucidworks/spark-solr-etl:5.9-cgroupv2-patch` |
| `keytool-utils` | 5.9.4 to 5.9.10 | `lucidworks/keytool-utils:5.9-cgroupv2-patch` |
Follow these steps to apply the patched images:
1. Open your Fusion Helm values file. For Fusion Cloud Native deployments, use the values file for your current deployment. For Helm deployments, use the values file you used to create the deployment.
2. For each service listed in the following table that applies to your Fusion version, add or update the image configuration:
Expand the following code snippet for the complete image configuration list.
```yaml expandable theme={"dark"}
global:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
sql-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
reverse-search:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
solr:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
kafka:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
ml-model-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-admin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-indexing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
query-pipeline:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
async-parsing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
admin-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
api-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
auth-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
classic-rest-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-resources:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-config:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
insights:
image:
imagePullPolicy: Always
name: insights
repository: lucidworks
tag: 5.9-cgroupv2-patch
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-launcher:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-rest-server:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connector-plugin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors-backend:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
rules-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
pm-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
lwai-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
webapps:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
apps-manager:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
templating:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
tikaserver:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
seldon-core-operator:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo-common-workflows:
image:
imagePullPolicy: Always
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
utilitiesTag: 5.9.11
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: 5.9-cgroupv2-patch
argo-common-workflows:
image:
imagePullPolicy: IfNotPresent
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: insights:5.9-cgroupv2-patch
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run the following command:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace. Replace `RELEASE_NAME` with your Helm release name. Replace `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the affected service pods to restart and verify they are using the patched images.
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
Fusion 5.9.5 introduced changes that affect Spark jobs. If you are upgrading from any Fusion version earlier than 5.9.5 to a version later than 5.9.5, Lucidworks recommends first upgrading to Fusion 5.9.5 before proceeding to the target version.
Fusion 5.9.15 upgrades the Stage SDK to v2.0.0.
Any custom plugins compiled with SDK v1.0.0, such as `query-sample-plugin-stage-0.0.1.zip` and `index-sample-plugin-stage-0.0.1.zip`, will not load after the upgrade.
Pipelines that reference those stages will be hidden until you install SDK v2.0.0 replacements.
Take these steps before you begin the upgrade process.
1. Rebuild your plugins against Stage SDK v2.0.0.
* Example outputs:
* `2-0-0-query-sample-plugin-stage-0.0.2.zip`
* `2-0-0-index-sample-plugin-stage-0.0.2.zip`
Do not remove v1.0.0 plugins before installing their 2.0.0 replacements because pipelines still reference the old classes stored in ZooKeeper.
2. Use Java 11 to recompile the plugins.
3. Update your `gradle.properties` file to use v2.0.0 of the Stage SDK. For the Query Stage SDK, use:
```bash theme={"dark"}
queryStagePluginSDKVersion=2.0.0
```
For the Index Stage SDK, use:
```bash theme={"dark"}
indexStagePluginSDKVersion=2.0.0
```
4. Verify that the `queryStagePluginSDKVersion` or `indexStagePluginSDKVersion` variable is used in the `build.gradle` task that builds the SDK stage. The following example uses the Query Stage SDK:
```bash theme={"dark"}
attributes 'Plugin-SDK-Version': "${queryStagePluginSDKVersion}"
```
See the [Query Stage SDK sample stage](https://github.com/lucidworks/query-stage-sdk/blob/master/examples/sample-plugin-stage/build.gradle) for an example.
5. Audit your pipelines and remove or replace any stages that no longer exist or are changed in the new plugins. If a pipeline still references a removed stage, it will continue to fail deserialization after upgrade.
Pipelines using custom stages will be hidden between the moment you upgrade and the moment you install the new plugins.
For supported Kubernetes versions and key component versions, see [Platform support and component versions](#platform-support-and-component-versions).
## Key highlights
### Consumption Dashboard update
The [Consumption Dashboard](/docs/5/fusion/operations/monitoring-and-reporting/consumption) uses graphs to visually track request and document usage against the allotted consumption for a Fusion license. The hover interaction for **requests per pipeline** graphs now displays a sortable list of pipelines to make it easier to analyze pipeline-level consumption.
Pipelines in a **requests per pipeline** graph can be sorted in the following ways:
* **Value descending**: sort pipelines according to number of requests from most to least.
* **Value ascending**: sort pipelines according to number of requests from least to most.
* **Alphabetically ascending**: sort by pipeline name from A to Z.
* **Alphabetically descending**: sort by pipeline name from Z to A.
### Gain deeper insight into asynchronous query performance
Monitoring for asynchronous [query stages](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/overview) is expanded to provide performance insights for async workloads, allowing for better optimization and more efficient troubleshooting.
Structured logs and Grafana dashboards for each async stage let you view metrics, including the following:
* Execution time and trends for how long a job or request takes from start to finish.
* Success rates for percentage of requests and jobs that complete without error.
* Number of retries for how many retry attempts were made after initial failures.
* Queue wait times for the time a job or request spends waiting to be processed.
* Resource usage for consumption of compute/storage/network by workload or tenant.
Prometheus metrics are used across async stages for additional visibility into query behavior, such as query performance comparisons across different filters and per-node monitoring to identify which Fusion node handled a specific query.
Logs can be filtered by:
* Application
* Pipeline
* Profile
* Stage
* Execution status
* Custom time ranges
This visibility makes it easier to detect performance bottlenecks, identify root causes of failures, and tune pipeline behavior across applications and query profiles.
### New Retrieval-Augmented Generation (RAG) query stage
Lucidworks continues expanding RAG capabilities that improve the quality and accuracy of LLM-powered retrieval workflows.
Documents that require chunking tend to be large and cover multiple topics.
Queries return an answer that links to the document, but the answer does not show exactly *where* to find that information in the document.
A [Chunking RAG Bridge Query Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/chunking-rag-bridge) is introduced to evolve RAG from document-level retrieval to chunk-level precision.
With this new stage, each cited source that accompanies a RAG answer also provides a link to the relevant document along with a snippet containing information about where to find the content in the document.
### New Batch Vectorize index stage
Fusion 5.9.15 introduces an [LWAI Batch Vectorize Index Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/lwai-batch-vectorize-index-stage) for AI pipelines.
This works the same as the [LWAI Vectorize Field stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/lwai-vectorize-field) but delivers faster indexing throughput because it processes your documents in batches before sending those documents to the next stage in your index pipeline. The LWAI Batch Vectorize stage can improve document indexing performance up to ten times the rate of the LWAI Vectorize Field stage.
The LWAI Vectorize stage processes documents one at a time.
With the LWAI Batch Vectorization Index stage, Fusion can send multiple documents per API call, reducing API overhead and improving scalability.
This stage includes new properties.
A batch is forwarded to the API as soon as it is full or when the wait time elapses, whichever comes first.
Maximum number of documents per batch (up to 32).
Maximum time to wait in milliseconds since the first document is buffered before proceeding.
### Restrictive mode for query pipelines
Fusion 5.9.15 introduces restrictive mode, a new way to safeguard query pipelines against unauthorized or risky changes.
When you enable restrictive mode, Fusion enforces stricter query parameter validation and sanitization, blocking unsafe inputs and limiting potentially expensive operations.
This feature helps you improve security and prevent costly mistakes.
See [Restrictive Mode](/docs/5/fusion/getting-data-out/query-basics/query-pipelines/restrictive-mode) for complete details about how restrictive mode works and how to enable it.
### Fusion stage SDKs updated to support JDK 11
The [Index Stage](/docs/5/fusion/dev-portal/index-stage-sdk) and [Query Stage](/docs/5/fusion/dev-portal/query-stage-sdk/overview) SDKs in self-hosted Fusion now support JDK 11.
These updates apply only to self-hosted Fusion and are not supported in Lucidworks Search.
Adding support for JDK 11 does the following:
* Increases runtime success.
* Improves dependency management to build plugins against stable versions.
* Ensures long-term compatibility of critical operational pipelines for plugin developers.
* Enables support for existing search functionality when planning upgrades.
### Fusion Admin UI updates
Several updates to the Fusion Admin UI improve safety and usability:
* A description is added to the JSON Facets stage, clarifying its purpose.
* UI labels are added for Ray model deployment jobs, grouping them under “Model Deployment Jobs” for clarity and discoverability.
* The **Asynchronous Execution Config** checkbox for the AI pipeline stages is now properly aligned with its label for better visual consistency and user experience.
* An optional red banner can be added across the bottom of all production environments as a visual reminder that you are in a production environment. The banner is static and non-dismissible, reducing the risk of accidental misconfiguration.
To enable the banner, add the following to your custom values YAML files.
```yaml theme={"dark"}
admin-ui:
banner:
enabled: true
```
If you would like to customize the text that shows in the UI, you can add parameters to your custom values YAML file.
```yaml theme={"dark"}
admin-ui:
banner:
enabled: true
text: "CUSTOM_BANNER_MESSAGE"
backgroundColor: "#ff4444"
textColor: "#ffffff"
fontSize: "14px"
fontWeight: "bold"
position: "bottom"
```
Configuration parameters:
* `text` (required): the message displayed in the banner
* `backgroundColor`: background color; default is red #ff4444
* `textColor`: text color; default is white #ffffff
* `fontSize`: font size; default is 14px
* `fontWeight`: font weight; default is bold
* `position`: banner position - bottom or top; default is bottom
### Naming conventions for AI-generated fields
Fields generated by Lucidworks AI now include a prefix of `lwai_` when indexed to Solr, making it easier to identify those fields when excluding them or using them for other tasks, such as rules, queries, or reports. Any field names previously generated are still supported and no changes are needed to continue using them.
### Additional Selenium support for the Web V2 connector
For users of the [Web V2 connector](/docs/fusion-connectors/connectors/web-v2) v2.2.1 and later, Fusion 5.9.15 improves support for Selenium Grid. The Selenium service is now installed automatically with the Web V2 connector and removed automatically when the connector is removed, resulting in faster setup and ease of use.
### Smarter hybrid search
Resiliency is improved for searches using the [Neural Hybrid Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/neural-hybrid-query).
When a hybrid query depends on vectorization, failures in the upstream vectorization stage previously resulted in incomplete or unstable search behavior.
Now Fusion automatically switches to lexical-only search if vectorization is unavailable to ensure uninterrupted results.
Clear indicators and improved logging make it easier to understand when the fallback occurs:
* If no vector is produced, you will see the message: `Input vector value is blank, skipping vector processing`.
* If a vector exists but is empty, you will see the message: `Input vector value is empty, {}, skipping vector processing`.
### Ability added for Prediction API
The Prediction API now supports [image metadata enrichment](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/image-metadata-enrichment). This API use case generates keywords and categories for images. This information can then be used as metadata for SEO, image-based product discovery, and more.
### Enhanced previews for integrated Lucidworks Platform products
A new, system-level API was added to Fusion 5.9.15 that allows Commerce Studio and other Lucidworks Platform products to simulate configuration changes before they are published. This change supports the following improvements:
* **Improved safety:** Test and validate changes without affecting live search traffic, reducing the risk of errors or downtime.
* **Faster iteration:** Experiment and refine configurations in real time, improving development speed and decision-making.
* **Higher confidence in publishing:** Preview query behavior before deployment to increase trust in changes and reduce rollback events.
Beginning with Fusion 5.9.15, all Commerce Studio integrations must have the `POST:/query/*/preview` and `GET,POST,PUT:/query-pipelines/**` permissions assigned to both the `lwp_user` and `lwp_service_account` roles in Fusion. Verify that your roles meet the following requirements:
```bash lines highlight={3-4} lwp_user theme={"dark"}
GET,PUT,DELETE,POST,HEAD:/query/experience-manager/**
GET,POST:/query/*
POST:/query/*/preview
GET,POST,PUT:/query-pipelines/**
GET:/suggestions/**
GET:/solr/*/terms
PUT,POST,GET:/apps/*/query-pipelines/**
GET:/apps/*/query-profiles/**
GET,PUT:/apps/*
GET:/solr/*/admin/luke
GET:/solr/**
```
```bash lines highlight={3-4} lwp_service_account theme={"dark"}
GET,PUT,DELETE,POST,HEAD:/query/experience-manager/**
GET,POST:/query/*
POST:/query/*/preview
GET,POST,PUT:/query-pipelines/**
GET:/suggestions/**
GET:/solr/*/terms
PUT,POST,GET:/apps/*/query-pipelines/**
GET:/apps/*/query-profiles/**
GET,PUT:/apps/*
GET,DELETE,PATCH,PUT,POST,HEAD:/apps/*/query-rewrite/**
GET,HEAD:/templating/**
GET:/solr/*/admin/luke
GET:/solr/**
```
## Bug fixes
* Added a custom module that serializes Nashorn `undefined` values as JSON `null`, preventing serialization errors when such values appear in Java objects. When using the OpenJDK version of Nashorn, an exception was thrown when the context contained keys with undefined JavaScript values. This exception occurred because the class for the undefined values was not visible outside of the OpenJDK Nashorn package. A custom object mapper is added for the class so serialization issues are now resolved when handling `undefined` in JavaScript pipeline to run reliably under Java 17.
* Time zone conversion issues are fixed in the Consumption Dashboard that caused counts to span extra days. The time selector was creating an overlap due to UTC conversion that included counts for both the selected day and the following day. As the API only accepts dates and not times, the time selector was removed to ensure accurate daily totals for entitlement reporting.
* Addressed an issue where jobs are stuck in a running state during an upgrade. Allowing all crawl jobs to complete before performing Fusion upgrades is necessary to maintain data integrity and prevent invalid job states. Now requests to stop jobs during an upgrade process are enhanced to check for invalid states, attempt automatic fixes, and provide detailed status messages.
* Adjusted license checks when apps-manager is unavailable for consistent and predictable license validation behavior. In previous releases, if apps-manager was unavailable or license capabilities could not be verified, the license check could not complete. The addition of broader exception catching when retrieving license capabilities and improved error logging for service unavailability ensures predictable capability behavior while connectivity is restored.
* Fixed a permissions flow in the Fusion indexing path used by V2 connectors. REST calls made from index pipelines using V2 connectors were running without the required permissions. This returned insufficient permissions errors and skipped mapped fields. REST calls in index pipelines now execute as expected when using the required privileges.
* Fixed an error where the Tika asynchronous parser in Fusion 5.9.12 did not index the body field for SharePoint connector documents, such as DOCS, PPTX, and XLSX files. Documents were split into multiple parts, so the body content only ended up in child documents, while the parent document lacked the body field. Now the first parsed part of the document is sent to Fusion using the parent document ID so the body field merges into the expected parent document. This restores expected content extraction and preserves child parts for images and other embeds.
## Known issues
Upgrading to Fusion 5.9.15 moves stage SDKs from v1.0.0 to v2.0.0.
When upgrading, plugins compiled with SDK v1.0.0 do not load.
Pipelines referencing those stages remain hidden until you replace the plugins.
* Before upgrading, rebuild any custom index or query plugins from SDK v1.0.0 using SDK v2.0.0. Also, verify pipelines do not reference stages removed or renamed in the new plugins.
* After upgrading, upload the new plugin ZIP files to the blob store and wait for the installation to complete. At this point, the pipelines will reappear.
See [Upgrade notes for Stage SDK plugins](#upgrade-notes-for-stage-sdk-plugins) above for more details.
## Deprecations
For full details on deprecations, see [Deprecations and Removals](/docs/5/fusion/deprecations-and-removals).
### Deprecated jobs
The following jobs are deprecated in this release and will be removed in a future release:
* [Head/Tail Analysis](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis)
* [Phrase Extraction](/docs/5/fusion/reference/config-ref/jobs/phrase-extraction)
* [Synonym Detection](/docs/5/fusion/reference/config-ref/jobs/synonym-detection)
* [Token and Phrase Spell Correction](/docs/5/fusion/reference/config-ref/jobs/token-and-phrase-spell-correction)
Lucidworks recommends migrating to [Neural Hybrid Search](/docs/5/fusion/hybrid-search/overview), which achieves superior relevance compared to these legacy machine learning methods.
Rules previously generated by these jobs and promoted to the `_query_rewrite` collection will remain until you delete them.
## Platform support and component versions
### Kubernetes platform support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.29, 1.30, 1.31, 1.32, 1.33
* **Microsoft Azure Kubernetes Service (AKS):** 1.29, 1.30, 1.31, 1.32, 1.33
* **Amazon Elastic Kubernetes Service (EKS):** 1.29, 1.30, 1.31, 1.32, 1.33
Support is also offered for Rancher Kubernetes Engine (RKE and RKE2) and OpenShift 4 versions that are based on Kubernetes 1.29, 1.30, 1.31, 1.32, 1.33; note that RKE2 may require some Helm chart modification. OpenStack and customized Kubernetes installations are *not* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support).
### Component versions
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| Solr | fusion-solr 5.9.15 *(based on Solr 9.6.1)* |
| ZooKeeper | 3.9.1 |
| Spark | 3.4.1 |
| Ingress Controllers | Nginx, Ambassador (Envoy), GKE Ingress Controller |
| Ray | ray\[serve] 2.46.0 |
More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).
# Fusion 5.9.16
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9/5.9.16-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.9.16-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9.16-release-notes
Released on February 24, 2026, Fusion 5.9.16 is a [maintenance release](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#maintenance-release-support-policy) that focuses on smoother upgrades and more dependable production behavior across connectors, query pipelines, indexing, and security.
Upgrading to the latest version of Fusion 5.9 offers several key benefits:
* **Access to latest features:** Stay current with the latest features and functionality to ensure compatibility and optimal performance.
* **Simplified process:** Fusion 5.9.5 and later use an in-place upgrade strategy, making upgrades easier than ever.
* **Extended support:** Upgrading keeps you up-to-date with the latest supported Kubernetes versions, as outlined in the [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle) policy.
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
Fusion 5.9.5 introduced changes that affect Spark jobs. If you are upgrading from any Fusion version earlier than 5.9.5 to a version later than 5.9.5, Lucidworks recommends first upgrading to Fusion 5.9.5 before proceeding to the target version.
If you use Seldon models or Smart Answers, which depends on Seldon, and override `JAVA_TOOL_OPTIONS` or `javaToolOptions` in your Helm values for `ml-model-service`, `fusion-query`, or `fusion-indexing`, the required protobuf flag is lost when you upgrade to Fusion 5.9.16. Without this flag, Seldon model serving and Smart Answers will fail.
To avoid this:
1. Locate where you set `JAVA_TOOL_OPTIONS` or `javaToolOptions` in your Helm values.
2. Add `-Dcom.google.protobuf.use_unsafe_pre22_gencode` to your Java options.
3. Validate Seldon models or Smart Answers after upgrade.
**Security patch available for api-gateway: Netty request smuggling vulnerabilities**
A patch is available for the `api-gateway` service to address critical Netty request smuggling vulnerabilities (CVE-2026-42581, CVE-2026-42585, CVE-2026-42587). These vulnerabilities allow attackers to smuggle HTTP requests through the gateway, potentially bypassing security controls.
The `api-gateway` service requires the Netty security patch.
Follow these steps to apply the patched image:
1. Open your Fusion Helm values file.
2. Add or update the `api-gateway` image configuration:
```yaml theme={"dark"}
api-gateway:
image:
repository: lucidworks
name: api-gateway
tag: 5.9.16-SUST-1634-patch
imagePullPolicy: IfNotPresent
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace, `RELEASE_NAME` with your Helm release name, and `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the `api-gateway` pods to restart and verify they are using the patched image.
For supported Kubernetes versions and key component versions, see [Platform support and component versions](#platform-support-and-component-versions).
Fusion 5.9.16 upgrades the SolrJ library from 8.11.x to 9.4.1. This upgrade changes how Solr resolves naming conflicts between physical collections and aliases. Previously, alias resolution was checked first, so queries against an alias name correctly expanded to the alias's underlying collections. After this upgrade, physical collections take priority over aliases of the same name. If a physical collection and a Solr alias share the same name, queries will be routed to the physical collection rather than the alias.
Before upgrading to 5.9.16, check whether any physical collections in your environment share a name with a Solr alias. If a conflict exists, rename the physical collection to remove the ambiguity and ensure correct query routing after upgrading.
## Key highlights
### Fusion Spell Check query stage
The [Spell Check query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/spell-check-query-stage) lets you invoke a spell checking function that returns relevant spelling suggestions when queries contain misspellings or typing errors.
If a query doesn't return the minimum number of results you specify, a spell checking function searches dictionaries, indexed data, and files to discover close matches to the terms entered in the query. Using this stage provides the following benefits:
* Enhances your customer's search experience by suggesting relevant terms even if they mistype query terms.
* Improves search relevance by discovering possible matches to terms entered in queries.
* Minimizes zero search result experiences by returning potential matches to mistyped search terms.
You can configure the Spell Check stage to retrieve suggestions based on the results from the following stages:
* Solr Query stage, which is more effective for lexical searches.
* Neural Hybrid Query stage or Chunking Neural Hybrid Query stage, which is more effective for semantic searches.
This example shows the Spell Check Query pipeline configured to use the **Result Count Threshold**, which uses the results from the Solr Query stage.
If you configure both the **Result Count Threshold** field and the **Cutoff margin (NHS)** field in the **same** query pipeline, **only** the **Result Count Threshold** field value that uses Solr Query stage results is used. The **Cutoff margin (NHS)** setting is ignored.
If you want to configure Spell Check query stages based on the results for both a Solr Query stage and a neural hybrid query stage, the best practice is to set up these pipelines:
* A query pipeline with a Spell Check stage using the Solr Query stage results.
* A different query pipeline with a Spell Check stage that uses the lexical score from either the Neural Hybrid Query stage or Chunking Neural Hybrid Query stage.
After the stage is configured, you can test by entering misspelled search terms in Query Workbench, and reviewing the suggestions returned by the spell check function. For example, if you enter `electronc`, the query does not return any results. The Spell Check stage triggers the spell checking functions, and possible matches display in the **Did you mean?** section.
Spell check suggestions are returned in the query response JSON within the `fusion.spellcheck_stage_suggestions` array:
```json highlight={3} wrap Spell check suggestions theme={"dark"}
{
"fusion": {
"spellcheck_stage_suggestions": ["electronic", "electronics", "electric", "election", "electro"]
}
}
```
Because the spell checking function returns suggestions from your indexed data, you can test terms specific to your organization that customers may commonly mistype. Testing in Query Workbench lets you verify the suggestions and results your customers will see in their search experience.
### Consumption Dashboard update
The consumption dashboard is updated to make entitlement discovery easier and reporting more accurate.
A new **Hide Temporary Pipelines** checkbox streamlines filtering by removing temporary pipelines from view with a single click. Previously, excluding temporary pipelines required manually deselecting each one, which could be cumbersome when dealing with dozens of pipelines.
Consumption reporting now properly excludes signals and signals aggregations collections from entitlements metrics. These collections were previously included incorrectly, which could inflate indexed record counts and complicate entitlement tracking.
### Customizable annotations and labels for Kubernetes resources
Fusion now supports customizable annotations and labels for all Kubernetes resources through values files, eliminating the need for direct chart template modifications and simplifying upgrades for enterprise deployments.
Previously, if you needed to apply organization-specific annotations and labels to Fusion Kubernetes resources, you had to modify chart templates directly. This approach increased upgrade complexity, introduced maintenance overhead, and created risk during version migrations.
With this enhancement, you can now configure custom annotations and labels through either global values that apply to all services or service-specific values that apply only to individual charts. Common use cases include adding cost allocation labels, ownership labels, monitoring annotations, or service mesh annotations such as controlling Istio sidecar injection.
Global values apply to all Fusion services:
```yaml theme={"dark"}
global:
labels:
owner: search-platform
annotations:
cost-center: fin-1234
```
Service-specific values apply only to that chart:
```yaml theme={"dark"}
fusion-indexing:
labels:
team: platform-search
annotations:
monitoring: enabled
```
The chart automatically merges and renders these values into Kubernetes resource metadata. This improves upgrade supportability by keeping your configurations separate from chart templates, reducing the risk of conflicts during version updates and eliminating the need to reapply and review template modifications on every upgrade.
### Timeout improvements for query pipeline stability and control
Fusion now includes multiple improvements that make query timeouts more consistent and production-friendly, especially under heavy load.
#### Enforcing an end-to-end query pipeline timeout using a request header
Previously, the gateway could time out using the default of 15 seconds while the query pipeline continued running in the background. This decreased stability during load spikes. With this change, Fusion checks the remaining time after each pipeline stage and cancels execution when the timeout is exceeded.
#### Fail pipelines when Solr returns partial results
When Solr exceeds a `timeAllowed` query limit, it can return `HTTP 200` with partial results that can break downstream pipeline stages. Fusion now detects these partial results and fails the pipeline by default to avoid producing incomplete or misleading responses. A new Solr Query stage option of **Process partial results** can be enabled to continue processing when partial results are acceptable.
#### More consistent Solr query limit enforcement across deployments
Fusion improves default query limit enforcements such as `timeAllowed` and `cpuAllowed` and aligns image and configset behavior so limits are applied consistently across deployments. This includes updated defaults with property overrides, a handler with higher limits for analytics workloads, and configuration updates to ensure query limits remain enforced.
### Query pipeline performance and routing improvements
#### Smoother performance under burst traffic
Pipeline creation is meant to be cached after the first request. Previously, if multiple requests arrived before the first initialization completed, each request could trigger its own pipeline initialization. This caused slowdowns and extra resource pressure.
Fusion now synchronizes pipeline initialization so only the first request does the work while concurrent requests wait, reducing request spikes and improving stability under load.
#### More predictable subquery latency
Solr subqueries previously lacked a built-in way to steer requests toward the replica types best suited for low-latency query traffic. Without explicit routing, subqueries could land on replica types, such as TLOG, that can introduce latency spikes and complicate troubleshooting.
Fusion now adds a `preferredReplicaType` option to the Solr Subquery stage, making replica steering easier to configure and more consistently applied—improving tail latency and keeping query routing aligned with operational expectations.
### Collections API now honors replica parameters
Previously, `nrtReplicas`, `tlogReplicas`, and `pullReplicas` could be ignored in Collections API requests, resulting in collections created with defaults instead of the intended architecture. Fusion now passes these parameters through correctly so automated collection creation matches the requested shard and replica strategy.
### Admin UI, APIs, and observability improvements
Fusion 5.9.16 includes several updates that improve UI usability and provide better diagnostics and monitoring.
#### Blob viewer loads faster and avoids unnecessary downloads
The Admin UI Blob viewer previously downloaded full blob content by default, which could cause large downloads and errors, especially for JSON blobs. Fusion now fetches blob metadata by default and only downloads full content when editing is requested, improving responsiveness and reliability.
#### Admin UI search enhanced for reliability and clarity
The Admin UI search has been improved to focus exclusively on Fusion application resources and navigation. This enhancement eliminates unreliable background indexing jobs and provides clearer search results, making it easier to find and manage your collections, pipelines, and other application components.
#### Reduced gateway audit log noise
OPTIONS requests are common in CORS flows and often not meaningful for audit purposes.
```text expandable theme={"dark"}
INFO 2025-10-20T18:43:56.544684097Z 2025-10-20T18:43:56.543Z - INFO [reactor-http-epoll-1:c.l.c.g.c.AuditListenerConfiguration@86] - ACCESS anonymous 192.0.2.1 OPTIONS /api/apps/www_EXAMPLE_com/query/www_EXAMPLE_com_navigation 200 0 -
INFO 2025-10-20T18:43:56.546076826Z 2025-10-20T18:43:56.544Z - INFO [reactor-http-epoll-2:c.l.c.g.c.AuditListenerConfiguration@86] - ACCESS anonymous 192.0.2.2 OPTIONS /api/apps/www_EXAMPLE_com/query/www_EXAMPLE_com_megamenu 200 0 -
INFO 2025-10-20T18:43:56.546141917Z 2025-10-20T18:43:56.545Z - INFO [reactor-http-epoll-2:c.l.c.g.c.AuditListenerConfiguration@86] - ACCESS anonymous 192.0.2.3 OPTIONS /api/apps/www_EXAMPLE_com/query/www_EXAMPLE_com_typeahead 200 0 -
INFO 2025-10-20T18:43:56.546448948Z 2025-10-20T18:43:56.544Z - INFO [reactor-http-epoll-3:c.l.c.g.c.AuditListenerConfiguration@86] - ACCESS anonymous 192.0.2.4 OPTIONS /api/apps/www_EXAMPLE_com/query/www_EXAMPLE_com_navigation 200 0 -
INFO 2025-10-20T18:43:56.553131217Z 2025-10-20T18:43:56.552Z - INFO [reactor-http-epoll-4:c.l.c.g.c.AuditListenerConfiguration@86] - ACCESS anonymous 192.0.2.5 OPTIONS /api/apps/www_EXAMPLE_com/query/www_EXAMPLE_com_megamenu 200 0 -
INFO 2025-10-20T18:43:56.554958418Z 2025-10-20T18:43:56.554Z - INFO [reactor-http-epoll-2:c.l.c.g.c.AuditListenerConfiguration@86] - ACCESS anonymous 192.0.2.6 OPTIONS /api/apps/www_EXAMPLE_com/query/www_EXAMPLE_com_megamenu 200 0 -
INFO 2025-10-20T18:43:56.555879448Z 2025-10-20T18:43:56.555Z - INFO [reactor-http-epoll-4:c.l.c.g.c.AuditListenerConfiguration@86] - ACCESS anonymous 192.0.2.7 OPTIONS /api/apps/www_EXAMPLE_com/query/www_EXAMPLE_com_search 200 0 -
INFO 2025-10-20T18:43:56.555968984Z 2025-10-20T18:43:56.555Z - INFO [reactor-http-epoll-4:c.l.c.g.c.AuditListenerConfiguration@86] - ACCESS anonymous 192.0.2.8 OPTIONS /api/apps/www_EXAMPLE_com/query/www_EXAMPLE_com_typeahead 200 0 -
INFO 2025-10-20T18:43:56.556998106Z 2025-10-20T18:43:56.556Z - INFO [reactor-http-epoll-3:c.l.c.g.c.AuditListenerConfiguration@86] - ACCESS anonymous 192.0.2.9 OPTIONS /api/apps/www_EXAMPLE_com/query/www_EXAMPLE_com_megamenu 200 0 -
INFO 2025-10-20T18:43:56.564406315Z 2025-10-20T18:43:56.563Z - INFO [reactor-http-epoll-3:c.l.c.g.c.AuditListenerConfiguration@86] - ACCESS anonymous 192.0.2.10 OPTIONS /api/apps/www_EXAMPLE_com/query/www_EXAMPLE_com_typeahead 200 0 -
INFO 2025-10-20T18:43:56.565558537Z 2025-10-20T18:43:56.565Z - INFO [reactor-http-epoll-1:c.l.c.g.c.AuditListenerConfiguration@86] - ACCESS anonymous 192.0.2.11 OPTIONS /api/apps/www_EXAMPLE_com/query/www_EXAMPLE_com_typeahead 200 0 -
INFO 2025-10-20T18:43:56.565757271Z 2025-10-20T18:43:56.565Z - INFO [reactor-http-epoll-2:c.l.c.g.c.AuditListenerConfiguration@86] - ACCESS anonymous 192.0.2.12 OPTIONS /api/apps/www_EXAMPLE_com/query/www_EXAMPLE_com_megamenu 200 0 -
```
Fusion now excludes OPTIONS requests from gateway audit logging, resulting in cleaner logs and easier investigations.
### Solr backup runner volume mount path change
In Fusion 5.9.16, the `solr-backup-runner` backup volume mount path changed from `/mnt/solr-backups` to `/var/solr/data/solr-backups`. This path aligns with Solr's security manager, which restricts file system access to authorized directories within `/var/solr/data`. The previous path was not within Solr's security policy and caused permission errors that prevented backups from completing successfully.
The `solr-backup-runner` component is disabled by default. This change only affects deployments where it has been explicitly enabled.
If you have `solr-backup-runner` enabled and are upgrading from a previous version, update the following to avoid backup failures after upgrading:
1. Update your Persistent Volume (PV) and Persistent Volume Claim (PVC) to reference the new backup path: `/var/solr/data/solr-backups`.
2. Update any Helm values or backup-related configuration that reference the old path `/mnt/solr-backups`.
3. Verify that backups complete successfully after upgrading.
### Operational stability and build reliability
Fusion 5.9.16 includes several improvements that strengthen platform resilience and improve reliability for deployments and integrations.
#### Improved recovery after temporary ZooKeeper outages
When ZooKeeper became unavailable briefly and then recovered, it could trigger downstream service issues, such as blob service behavior or job visibility problems, and often required manual restarts. Fusion now recovers more cleanly after temporary ZooKeeper unavailability, improving resilience during transient infrastructure events and reducing manual intervention.
#### Supported Docker images for monitoring exporters
The Bitnami images used for `jmx-exporter` and `kafka-exporter` were deprecated and relied on older base images. Fusion now provides replacement Docker images for both exporters, improving long-term supportability and reducing risk from upstream deprecations.
#### ETL indexer can automatically trigger Spark jobs after indexing
Many indexing workflows require post-index processing such as enrichment, aggregation, or downstream job execution. Previously, running follow-on Spark jobs often required custom coordination. Fusion 5.9.16 streamlines automation by allowing the ETL indexer to automatically kick off a Spark job after an indexing run.
#### API gateway no longer depends on Kubernetes API availability
During GCP control plane upgrades or Kubernetes API maintenance events, `api-gateway` and `query-pipeline` pods could become unavailable due to readiness and liveness probe failures. These failures prevented pods from recovering during the maintenance window, causing service disruptions.
Fusion now refines the availability checks, reducing the risk of service outages during routine cluster maintenance.
## API Updates
Indexing API:
* Added subscription indexing endpoints for asynchronous document indexing via Kafka: `GET /subscription_indexing/status` and `POST /subscription_indexing/{id}/collections/{collection}/index`.
* Added optional `throwOnError` parameter to `/index/{id}` and `/index/{id}/debug` endpoints. When enabled, indexing operations fail immediately if an error is encountered rather than continuing with partial success.
See [Subscriptions API](/api-reference/subscriptions-api/get-subscription-indexing-status) for complete details.
Query API:
* Removed deprecated endpoint: `/query-stages/lwai-chunking-strategies`.
## Bug fixes
### TLS support added for the `config-sync` service
Fusion deployments can use TLS with the `config-sync` service. Previously, this service could not operate in TLS-enabled Fusion clusters due to hardcoded HTTP URLs for inter-service communication and missing TLS configuration for Solr and ZooKeeper connections.
The `config-sync` service adds HTTPS protocol switching by using `application-tls.yml`, configures ZooKeeper TLS on port `2281` with JVM properties, and configures Solr TLS using Java SSLContext through the truststore.
All sync operations work correctly, including publish, subscribe, and pub/sub.
For more information, see [ConfigSync](/docs/5/fusion/operations/config-sync/overview).
### Parsing and indexing improvements
These fixes improve ingestion reliability, reduce indexing interruptions, and make scaling behavior more predictable in production.
* **Async parsing works correctly even when other parsers are deactivated.**\
Async parsing could fail to run when additional parsers were present in the configuration, even if those parsers were explicitly deactivated. Fusion now ignores deactivated parsers during selection, ensuring async parsing executes reliably regardless of configuration layout.
- **Increased reliability of body extraction for large documents in async parsing.**\
For large, text-heavy files, the async Tika parser could fail to extract body content due to string length constraints. In some cases this happened even when truncation limits were configured, resulting in missing body fields. Fusion now improves async parsing behavior for large documents so body extraction is more reliable and configured limits are respected more consistently.
* **Fixed file descriptor leak in async parsing.**\
A resource leak during filesystem operations could accumulate over time and trigger "Too many open files" errors, interrupting indexing and requiring repeated pod restarts to recover. Fusion now properly closes resources during async parsing file operations, improving long-running stability and reducing ingestion disruptions.
- **Batch partial updates continue even when some document IDs are missing.**\
In incremental indexing workflows, partial update batches could stop processing early if a submitted document ID didn’t exist, preventing valid updates later in the same batch from being applied. Fusion now rejects missing IDs as intended while continuing to process the rest of the batch, improving throughput and reducing the need for manual retries.
* **Partial updates work more reliably with common vector configurations.**\
Using LWAI with partial updates sometimes caused failures when applying vector fields across common formats and configurations. Fusion now improves partial-update indexing workflows to better support these vector setups, reducing failures and making vector enrichment/indexing more dependable.
- **`fusion-query` and `fusion-indexing` have faster startup under HPA.**\
`fusion-query` and `fusion-indexing` could experience a 15-second startup delay due to default polling behavior, slowing scale-up responsiveness under Horizontal Pod Autoscaling. Fusion now reduces unnecessary startup delay by improving service availability polling behavior, enabling faster scaling and better responsiveness during burst traffic.
### Connectors
These fixes improve connector reliability during migrations, increase crawl correctness, and make connector APIs more stable and secure.
* **Web V1 connector now properly indexes documents when using JavaScript evaluation.**\
In Fusion 5.9.14 and 5.9.15, the Web V1 connector referenced two versions of the Selenium library, which caused runtime incompatibilities. Fusion now resolves this Selenium class-path conflict, and V1 JavaScript evaluation works as expected, reducing crawl disruptions.
- **Web V2 connector now dedupes correctly using canonical URLs.**\
The Web V2 connector did not reliably deduplicate pages by canonical URL due to incorrect canonical redirect limit logic, preventing the canonical-handling code path from running as intended. Fusion now corrects canonical URL handling so canonical-based deduplication works properly, reducing duplicate content in the index and improving crawl efficiency.
* **Connector schema responses are now consistently ordered.**\
Connector schema output could appear in inconsistent property order due to loss of insertion order during V2 transformations. Fusion now preserves stable property ordering in connector schema responses, resulting in more predictable API output, cleaner diffs, and easier downstream automation and validation.
### SDKs and services
These fixes improve upgrade safety for custom extensions, reduce service startup disruption, and restore expected runtime behavior for encrypted configurations.
* **Index Stage SDK plugins load more reliably in newer versions.**\
Some Index Stage SDK plugins could fail to load and would not appear in the Fusion UI, blocking plugin installation and upgrades. In particular, certain common Java constructs could cause the plugin framework to fail to generate a valid stages definition, resulting in errors like "no stages definition." Fusion now improves Index Stage SDK plugin loading so stages using these constructs are discovered correctly and no longer block plugin installation.
- **Query Stage SDK plugins must use `CloudLegacySolrClient` for Solr connections.**\
The SolrJ upgrade from 8.11.x to 9.4.1 in Fusion 5.9.16 changes `CloudSolrClient` to use HTTP2 by default. Because HTTP2 connections are not supported by pipeline services in Fusion 5, custom query stages must use `CloudLegacySolrClient` instead of `CloudSolrClient` or `CloudHttp2SolrClient` for Solr connections.
* **`ml-model-service` starts more safely when legacy model types are present.**\
Deployments could fail or crash-loop when unsupported legacy model types, such as OpenNLP or MLeap, were encountered, or when TLS/ingress configurations did not match expected trust material. These failures could disrupt upgrades and cause avoidable downtime. Fusion now handles unsupported legacy model types more safely (instead of failing the entire service), reducing startup disruption and improving operational stability when migrating legacy models to supported approaches.
- **PBL jobs can decrypt configuration secrets again.**\
After upgrades, PBL jobs could fail because encrypted configuration values were no longer being decrypted at runtime. Logs indicated that no `StringDecryptor` implementation was available, preventing secrets from being resolved. Fusion now wires `StringDecryptor` into the job runtime through the Spring application context so encrypted configuration values decrypt correctly, restoring expected job behavior and preventing post-upgrade failures tied to secrets handling.
## Platform support and component versions
### Kubernetes platform support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.30, 1.31, 1.32, 1.33, 1.34, 1.35
* **Microsoft Azure Kubernetes Service (AKS):** 1.30, 1.31, 1.32, 1.33, 1.34, 1.35
* **Amazon Elastic Kubernetes Service (EKS):** 1.30, 1.31, 1.32, 1.33, 1.34, 1.35
Support is also offered for Rancher Kubernetes Engine (RKE and RKE2) and OpenShift 4 versions that are based on Kubernetes 1.30, 1.31, 1.32, 1.33, 1.34, 1.35; note that RKE2 may require some Helm chart modification. OpenStack and customized Kubernetes installations are *not* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support).
### Component versions
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| Solr | fusion-solr 5.9.16 *(based on Solr 9.6.1)* |
| ZooKeeper | 3.9.1 |
| Spark | 3.4.1 |
| Ingress Controllers | Nginx, Ambassador (Envoy), GKE Ingress Controller |
| Ray | ray\[serve] 2.46.0 |
| Helm | 3.4.1 |
More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).
# Fusion 5.9.2
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9/5.9.2-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.9.2-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9.2-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/39hg5n
Released on December 13, 2023, this [maintenance release](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#maintenance-release-support-policy) includes an update to Kubernetes 1.27 and improvements to query service replica scaling and the Web V1 (classic) connector library.
To learn more, skip to the [release notes](#improvements).
**Security patch available for api-gateway: Netty request smuggling vulnerabilities**
A patch is available for the `api-gateway` service to address critical Netty request smuggling vulnerabilities (CVE-2026-42581, CVE-2026-42585, CVE-2026-42587). These vulnerabilities allow attackers to smuggle HTTP requests through the gateway, potentially bypassing security controls.
The `api-gateway` service requires the Netty security patch.
Follow these steps to apply the patched image:
1. Open your Fusion Helm values file.
2. Add or update the `api-gateway` image configuration:
```yaml theme={"dark"}
api-gateway:
image:
repository: lucidworks
name: api-gateway
tag: 5.9.9-SUST-1634-patch
imagePullPolicy: IfNotPresent
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace, `RELEASE_NAME` with your Helm release name, and `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the `api-gateway` pods to restart and verify they are using the patched image.
## Platform Support and Component Versions
### Kubernetes platform support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.27
* **Microsoft Azure Kubernetes Service (AKS):** 1.27
* **Amazon Elastic Kubernetes Service (EKS):** 1.27
Support is also offered for Rancher Kubernetes Engine (RKE) and OpenShift 4 versions that are based on Kubernetes 1.27. OpenStack and customized Kubernetes installations are *not* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support).
### Component versions
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| **Solr** | fusion-solr 5.9.1 *(based on Solr 9.1.1)* |
| **ZooKeeper** | 3.7.1 |
| **Spark** | 3.2.2 |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
## Improvements
* Fusion now supports Kubernetes 1.27. This applies to GKE, AKS, and EKS. It also applies to Rancher (RKE) and OpenShift 4 versions that are compatible with Kubernetes 1.27. Refer to Kubernetes documentation for version [1.27](https://kubernetes.io/blog/2023/03/17/upcoming-changes-in-kubernetes-v1-27/).
* Query service replica scaling has been improved.
## Fusion Connectors
* Updated the Web V1 (classic) connector library to resolve issues for certain JavaScript-enabled web pages. While Fusion was not vulnerable to the Common Vulnerabilities and Exposures (CVE) 2023-4863, the connector updates allowed Lucidworks to also acknowledge and resolve that publicly-reported CVE.
## Bug Fixes
* The connectors-backend service no longer fails to start when deploying to OpenShift 4.12.
* Fixed an issue where the Transport Layer Security (TLS) for Fusion Microservices could not be enabled and used.
## Known issues
* New Kerberos security realms cannot be configured successfully in this version of Fusion.
# Fusion 5.9.3
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9/5.9.3-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.9.3-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9.3-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/lqgbgi
Released on February 26, 2024, this [maintenance release](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#maintenance-release-support-policy) includes an update to Kubernetes 1.28.
To learn more, skip to the [release notes](#improvements).
**Security patch available for api-gateway: Netty request smuggling vulnerabilities**
A patch is available for the `api-gateway` service to address critical Netty request smuggling vulnerabilities (CVE-2026-42581, CVE-2026-42585, CVE-2026-42587). These vulnerabilities allow attackers to smuggle HTTP requests through the gateway, potentially bypassing security controls.
The `api-gateway` service requires the Netty security patch.
Follow these steps to apply the patched image:
1. Open your Fusion Helm values file.
2. Add or update the `api-gateway` image configuration:
```yaml theme={"dark"}
api-gateway:
image:
repository: lucidworks
name: api-gateway
tag: 5.9.9-SUST-1634-patch
imagePullPolicy: IfNotPresent
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace, `RELEASE_NAME` with your Helm release name, and `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the `api-gateway` pods to restart and verify they are using the patched image.
## Platform Support and Component Versions
### Kubernetes platform support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.28
* **Microsoft Azure Kubernetes Service (AKS):** 1.28
* **Amazon Elastic Kubernetes Service (EKS):** 1.28
Support is also offered for Rancher Kubernetes Engine (RKE) and OpenShift 4 versions that are based on Kubernetes 1.28. OpenStack and customized Kubernetes installations are *not* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support).
### Component versions
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| **Solr** | fusion-solr 5.9.3 *(based on Solr 9.1.1)* |
| **ZooKeeper** | 3.7.1 |
| **Spark** | 3.2.2 |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
## Improvements
* Fusion now supports Kubernetes 1.28. This applies to GKE, AKS, and EKS. It also applies to Rancher (RKE) and OpenShift 4 versions that are compatible with Kubernetes 1.28. Refer to Kubernetes documentation at [Kubernetes v1.28](https://kubernetes.io/blog/2023/08/15/kubernetes-v1-28-release/) for more information.
## Bug fixes
* Fixed an issue affecting remote connector datasources where they could enter a state of persistent inability to index following a `No Plugin Activity` timeout error.
* Fixed an issue with the async-parser where large files exceeding the configured memory buffer were repeatedly retried for processing, even though they would ultimately fail. This behavior could lead to unnecessary resource consumption and logging noise. Files that exceed the size configuration are no longer retried.
* Fixed an issue with the async-parser where documents containing empty values were erroneously emitted for indexing, even though the indexing process couldn’t handle them. This behavior led to indexing failures and unnecessary resource consumption.
* Fixed an issue with the Rules UI where very large rule sets could trigger URI errors, preventing the UI from loading or functioning correctly. This required manual deletion of rules to restore functionality.
* Fixed an issue where previously denied and published rules reappeared as new suggestions after running detection jobs.
## Known issues
* New Kerberos security realms cannot be configured successfully in this version of Fusion.
# Fusion 5.9.4
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9/5.9.4-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.9.4-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9.4-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/kgbb0r
Released on June 25, 2024, this [maintenance release](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#maintenance-release-support-policy) includes improvements to Jira and Confluence connectors for GDPR mode support, and access control lists (ACLs) to the Graph Security Trimming query pipeline.
To learn more, skip to the [release notes](#improvements).
**Security patch available for api-gateway: Netty request smuggling vulnerabilities**
A patch is available for the `api-gateway` service to address critical Netty request smuggling vulnerabilities (CVE-2026-42581, CVE-2026-42585, CVE-2026-42587). These vulnerabilities allow attackers to smuggle HTTP requests through the gateway, potentially bypassing security controls.
The `api-gateway` service requires the Netty security patch.
Follow these steps to apply the patched image:
1. Open your Fusion Helm values file.
2. Add or update the `api-gateway` image configuration:
```yaml theme={"dark"}
api-gateway:
image:
repository: lucidworks
name: api-gateway
tag: 5.9.9-SUST-1634-patch
imagePullPolicy: IfNotPresent
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace, `RELEASE_NAME` with your Helm release name, and `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the `api-gateway` pods to restart and verify they are using the patched image.
**Urgent action required by November 26, 2025**
A patch is required by November 26, 2025 for all self-hosted Fusion deployments running on Amazon Elastic Kubernetes Service (EKS). Certain Java versions used by Fusion components reach end of life on this date. Failure to apply the patch will result in compatibility issues.
The following Fusion services require the `cgroupv2` patch:
| Service | Affected Fusion versions | Patch tag |
| - | - | - |
| `insights` | 5.9.4 to 5.9.15 | `lucidworks/insights:5.9-cgroupv2-patch` |
| `spark-solr-etl` | 5.9.4 to 5.9.11 | `lucidworks/spark-solr-etl:5.9-cgroupv2-patch` |
| `keytool-utils` | 5.9.4 to 5.9.10 | `lucidworks/keytool-utils:5.9-cgroupv2-patch` |
Follow these steps to apply the patched images:
1. Open your Fusion Helm values file. For Fusion Cloud Native deployments, use the values file for your current deployment. For Helm deployments, use the values file you used to create the deployment.
2. For each service listed in the following table that applies to your Fusion version, add or update the image configuration:
Expand the following code snippet for the complete image configuration list.
```yaml expandable theme={"dark"}
global:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
sql-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
reverse-search:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
solr:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
kafka:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
ml-model-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-admin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-indexing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
query-pipeline:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
async-parsing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
admin-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
api-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
auth-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
classic-rest-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-resources:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-config:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
insights:
image:
imagePullPolicy: Always
name: insights
repository: lucidworks
tag: 5.9-cgroupv2-patch
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-launcher:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-rest-server:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connector-plugin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors-backend:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
rules-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
pm-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
lwai-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
webapps:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
apps-manager:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
templating:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
tikaserver:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
seldon-core-operator:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo-common-workflows:
image:
imagePullPolicy: Always
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
utilitiesTag: 5.9.11
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: 5.9-cgroupv2-patch
argo-common-workflows:
image:
imagePullPolicy: IfNotPresent
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: insights:5.9-cgroupv2-patch
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run the following command:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace. Replace `RELEASE_NAME` with your Helm release name. Replace `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the affected service pods to restart and verify they are using the patched images.
## Platform Support and Component Versions
### Kubernetes platform support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.29
* **Microsoft Azure Kubernetes Service (AKS):** 1.29
* **Amazon Elastic Kubernetes Service (EKS):** 1.29
Support is also offered for Rancher Kubernetes Engine (RKE) and OpenShift 4 versions that are based on Kubernetes 1.29. OpenStack and customized Kubernetes installations are *not* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support).
### Component versions
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| **Solr** | fusion-solr 5.9.4 *(based on Solr 9.1.1)* |
| **ZooKeeper** | 3.7.1 |
| **Spark** | 3.2.2 |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
**More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).**
**Looking to upgrade?**
Fusion 5.9.5 introduced changes that affect Spark jobs. If you are upgrading from any Fusion version earlier than 5.9.5 to a version later than 5.9.5, Lucidworks recommends first upgrading to Fusion 5.9.5 before proceeding to the target version.
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
## Improvements
* Fusion now supports Kubernetes 1.29. This applies to GKE, AKS, and EKS. It also applies to Rancher (RKE) and OpenShift 4 versions that are compatible with Kubernetes 1.29. Refer to Kubernetes documentation at [Kubernetes v1.29](https://kubernetes.io/blog/2023/12/13/kubernetes-v1-29-release/) for more information.
* The Jira and Confluence classic connectors now support GDPR mode.
* The Graph Security Trimming query pipeline stage now supports two new types of access control lists (ACLs) in addition to the main ACL represented by the `_lw_acl_ss` field:
* A "deny" ACL list (`_lw_deny_acl_ss`).
* A "share" ACL list (`_lw_share_acl_ss`) for connectors such as SMB that support it.
* The Graph Security Trimming query pipeline stage has built the **Join Method**, **ACL Solr Collection**, and **Join Field** properties directly into the stage. These fields have been removed from the stage, and you no longer have to configure them.
## Bug fixes
* Fixed an issue that caused counters from discarded `query-service` instances to be propagated to subsequent days, leading to inaccurate numbers on the Consumption Dashboard.
* Fixed an issue that caused an NRT replica to be created even when `nrtReplicas` is set to `0`.
* Fixed an issue that caused Kafka poll timeouts and commit errors, primarily impacting the SharePoint Optimized V2 connector.
* This release fixes issues with several V1 connectors:
* Fixed an issue that prevented the Jira Cloud V1 connector from handling `429` errors correctly.
* Fixed an issue that prevented the Jira V1 connector from indexing attachments.
* Fixed an issue that prevented the Salesforce V1 connector from indexing data and applying security trimming.
* Fixed an issue that caused datasource jobs to time out prematurely when data takes some time to process.
* Fixed an issue that sometimes led to connector crawls becoming stuck in an infinite loop.
## Deprecations
For full details on deprecations, see [Deprecations and Removals](/docs/5/fusion/deprecations-and-removals).
* The Fusion Search [Domain-Specific Language (DSL)](/docs/5/fusion/getting-data-out/query-basics/domain-specific-language) is deprecated in this release. Be aware that it can cause significant performance degradation in queries.
## Known issues
* New Kerberos security realms cannot be configured successfully in this version of Fusion.
# Fusion 5.9.5
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9/5.9.5-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.9.5-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9.5-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/5wtdd9
Released on November 4, 2024, this [maintenance release](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#maintenance-release-support-policy) includes the new Neural Hybrid Search capability, as well as upgrades to Solr, Kubernetes, Zookeeper, and some bug fixes.
To learn more, skip to the [release notes](#new-features).
**Security patch available for api-gateway: Netty request smuggling vulnerabilities**
A patch is available for the `api-gateway` service to address critical Netty request smuggling vulnerabilities (CVE-2026-42581, CVE-2026-42585, CVE-2026-42587). These vulnerabilities allow attackers to smuggle HTTP requests through the gateway, potentially bypassing security controls.
The `api-gateway` service requires the Netty security patch.
Follow these steps to apply the patched image:
1. Open your Fusion Helm values file.
2. Add or update the `api-gateway` image configuration:
```yaml theme={"dark"}
api-gateway:
image:
repository: lucidworks
name: api-gateway
tag: 5.9.9-SUST-1634-patch
imagePullPolicy: IfNotPresent
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace, `RELEASE_NAME` with your Helm release name, and `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the `api-gateway` pods to restart and verify they are using the patched image.
**Urgent action required by November 26, 2025**
A patch is required by November 26, 2025 for all self-hosted Fusion deployments running on Amazon Elastic Kubernetes Service (EKS). Certain Java versions used by Fusion components reach end of life on this date. Failure to apply the patch will result in compatibility issues.
The following Fusion services require the `cgroupv2` patch:
| Service | Affected Fusion versions | Patch tag |
| - | - | - |
| `insights` | 5.9.4 to 5.9.15 | `lucidworks/insights:5.9-cgroupv2-patch` |
| `spark-solr-etl` | 5.9.4 to 5.9.11 | `lucidworks/spark-solr-etl:5.9-cgroupv2-patch` |
| `keytool-utils` | 5.9.4 to 5.9.10 | `lucidworks/keytool-utils:5.9-cgroupv2-patch` |
Follow these steps to apply the patched images:
1. Open your Fusion Helm values file. For Fusion Cloud Native deployments, use the values file for your current deployment. For Helm deployments, use the values file you used to create the deployment.
2. For each service listed in the following table that applies to your Fusion version, add or update the image configuration:
Expand the following code snippet for the complete image configuration list.
```yaml expandable theme={"dark"}
global:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
sql-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
reverse-search:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
solr:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
kafka:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
ml-model-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-admin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-indexing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
query-pipeline:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
async-parsing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
admin-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
api-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
auth-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
classic-rest-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-resources:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-config:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
insights:
image:
imagePullPolicy: Always
name: insights
repository: lucidworks
tag: 5.9-cgroupv2-patch
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-launcher:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-rest-server:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connector-plugin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors-backend:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
rules-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
pm-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
lwai-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
webapps:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
apps-manager:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
templating:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
tikaserver:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
seldon-core-operator:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo-common-workflows:
image:
imagePullPolicy: Always
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
utilitiesTag: 5.9.11
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: 5.9-cgroupv2-patch
argo-common-workflows:
image:
imagePullPolicy: IfNotPresent
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: insights:5.9-cgroupv2-patch
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run the following command:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace. Replace `RELEASE_NAME` with your Helm release name. Replace `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the affected service pods to restart and verify they are using the patched images.
## Platform Support and Component Versions
### Kubernetes platform support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.28, 1.29, 1.30
* **Microsoft Azure Kubernetes Service (AKS):** 1.28, 1.29, 1.30
* **Amazon Elastic Kubernetes Service (EKS):** 1.28, 1.29, 1.30
Support is also offered for Rancher Kubernetes Engine (RKE) and OpenShift 4 versions that are based on Kubernetes 1.28, 1.29, 1.30. OpenStack and customized Kubernetes installations are *not* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support).
### Component versions
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| **Solr** | fusion-solr 5.9.5 *(based on Solr 9.6.1)* |
| **ZooKeeper** | 3.9.1 |
| **Spark** | 3.2.2 |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
**More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).**
Looking to upgrade?
Upgrading to Fusion 5.9.5 has special considerations, due to changes introduced with Solr 9.6.1 and Lucene 9.10.0. Refer to the **Fusion 5 Upgrade from 5.9.x** for specific details and potential mitigation strategies.
Fusion 5.9.5 introduced changes that affect Spark jobs. If you are upgrading from any Fusion version earlier than 5.9.5 to a version later than 5.9.5, Lucidworks recommends first upgrading to Fusion 5.9.5 before proceeding to the target version.
This article includes instructions for upgrading Fusion from one version to another. In some cases, the instructions do not vary. Other upgrades require special instructions. Start by checking upgrade details for your target version before continuing to the [General upgrade process](#general-upgrade-process).
Whenever you upgrade Fusion, you must also update your [remote connectors](/docs/fusion-connectors/developers/remote-v2-connectors), if you are running any.
You can download the latest files at [V2 Connectors Downloads](/docs/fusion-connectors/downloads/v2-connectors-downloads).
Fusion values change between releases. Check the [example values](https://github.com/lucidworks/fusion-cloud-native/blob/master/customize_fusion_values.yaml.example) and update values as needed.
## Upgrades from 5.9.x
### to 5.9.y
Upgrading from 5.9.x to 5.9.y involves using the [General upgrade process](#general-upgrade-process).
If you are using horizontal pod autoscaling, follow the [steps in the Fusion 5.8.1 release notes](/docs/5/fusion/release-notes/more/5.8.1-release-notes). If you have already done this as part of a previous upgrade, you can skip this process.
### to 5.9.12
When upgrading to Fusion 5.9.12, add the following to your `values.yaml` file to avoid a known issue that prevents the `kuberay-operator` pod from launching successfully:
```yaml theme={"dark"}
kuberay-operator:
crd:
create: true
```
### to 5.9.5
Upgrading to Fusion 5.9.5 has special considerations, due to changes introduced with Solr 9.6.1 and Lucene 9.10.0. All upgrades are susceptible to issues from these changes. Follow the upgrade procedures closely to avoid issues with the upgrade.
See the following sections for an overview of the issues, or [skip to the upgrade process](#upgrade-process).
#### Solr 9.6.1 changes
Prior to Fusion 5.9.5, Fusion utilized Solr 9.1.1 or earlier. Due to changes in Solr 9.3, some Solr configuration and collection configurations are no longer compatible. As Fusion 5.9.5 leverages Solr 9.6.1, it’s imperative to address these compatibility issues during the upgrade process.
To address Solr and collection configuration issues, a Docker utility called `fm-upgrade-apps-to-solr-9`, also known as the Fusion migration script, is included in the Fusion 5.9.5 release. This utility performs the following tasks:
* Removes the unused configuration, ``, from `solrconfig.xml`. Solr no longer supports this configuration.
* Removes the query response writer of class `solr.XSLTResponseWriter`.
* Comments out processors of type `solr.StatelessScriptUpdateProcessorFactory`.
* Removes `` element from request handler.
* Changes cache class attribute of elements `"filterCache"`, `"cache"`, `"documentCache"`, `"queryResultCache"` to `solr.search.CaffeineCache`.
* Removes `keepShortTerm` attribute from filter of class `solr.NGramFilterFactory`.
* Updates collection configurations, as needed.
#### Lucene 9.10.0 changes
A Lucene update to 9.10.0 in Fusion 5.9.5 may cause issues with certain collections in Solr. The change to the FST posting format codec (from `Lucene90PostingsWriterDoc` to `Lucene99PostingsWriterDoc`) in Lucene is incompatible with Solr in Fusion. As a result, Solr will not open a new searcher for collections using the `FST50` postings format.
To identify collections potentially affected by the Lucene codec change, examine the field definitions within your Solr schema. Look for fields that specify the `postingsFormat` attribute with a value of `FST50`. Collections containing such fields may experience compatibility issues. For example:
```xml theme={"dark"}
```
The following log excerpt demonstrates a typical error message encountered when an upgrade is impacted by the codec change:
```java theme={"dark"}
Caused by: org.apache.lucene.index.CorruptIndexException: codec mismatch: actual codec=Lucene90PostingsWriterDoc vs expected codec=Lucene99PostingsWriterDoc (resource=ByteBufferIndexInput(path="/var/solr/data/acme_query_rewrite_staging_shard1_replica_t9/data/index/_cn_FST50_0.doc"))
at org.apache.lucene.codecs.CodecUtil.checkHeaderNoMagic(CodecUtil.java:205) ~[?:?]
at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:194) ~[?:?]
at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:254) ~[?:?]
at org.apache.lucene.codecs.lucene99.Lucene99PostingsReader.(Lucene99PostingsReader.java:80) ~[?:?]
at org.apache.lucene.codecs.memory.FSTPostingsFormat.fieldsProducer(FSTPostingsFormat.java:60) ~[?:?]
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.(PerFieldPostingsFormat.java:330) ~[?:?]
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:392) ~[?:?]
at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:119) ~[?:?]
at org.apache.lucene.index.SegmentReader.(SegmentReader.java:96) ~[?:?]
at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:178) ~[?:?]
at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:220) ~[?:?]
at org.apache.lucene.index.IndexWriter.lambda$getReader$0(IndexWriter.java:542) ~[?:?]
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:138) ~[?:?]
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:604) ~[?:?]
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:112) ~[?:?]
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:91) ~[?:?]
at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:38) ~[?:?]
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2399) ~[?:?]
```
To account for the `postingsFormat="FST50"` codec issue, a Docker utility called `run-the-fm-upgrade-query-rewrite-docker-utility` is provided alongside the Fusion 5.9.5 release. You can pull this image from Docker using `docker pull lucidworks/run-the-fm-upgrade-query-rewrite-docker-utility:2.x`.
This utility performs two actions: `prepare` and `restore`. Use the `prepare` action before the Fusion 5.9.5 upgrade begins. At a high level, the `prepare` action performs the following actions:
1. Removes `postingsFormat="FST50"` from all collections in the environment.
2. Re-indexes documents to new, temporary collections.
3. Compares the original collections to the new, temporary collections to ensure data integrity.
Use the `restore` action after the Fusion 5.9.5 upgrade finishes, which must include the Solr 9.6.1 upgrade. The `restore` action performs the following actions:
1. Restores `postingsFormat="FST50"` to all collections in the environment that were changed with the `prepare` action.
2. Re-indexes documents to new, permanent collections. These collections match the original collections that were in place prior to the `prepare` action.
3. Compares the restored collections to the temporary collections to ensure data integrity.
#### Upgrade process
This section provides a high-level overview of the steps involved in upgrading to Fusion 5.9.5. Follow each step in the order given:
1. [Create a full backup of all Fusion collections](#back-up-your-solr-collections). These backups are intended as an emergency failsafe only.
2. [Run the `fm-upgrade-apps-to-solr-9` Docker utility](#run-the-fm-upgrade-apps-to-solr-9-docker-utility). This updates the Solr configuration and collections for compatibility with Solr 9.6.1.
3. [Run the `run-the-fm-upgrade-query-rewrite-docker-utility` Docker utility](#run-the-fm-upgrade-query-rewrite-docker-utility). Use the `prepare` action to address potential collection compatibility issues with Lucene 9.10.0 codecs.
4. [Upgrade your Fusion environment to version 5.9.5](#general-upgrade-process). Use the upgrade scripts or your own upgrade process.
5. [Re-run the `run-the-fm-upgrade-query-rewrite-docker-utility` Docker utility](#re-run-the-fm-upgrade-query-rewrite-docker-utility). Use the `restore` action to restore collections to their original state.
6. [Validate the upgrade was successful](#validate-the-upgrade). In addition to your usual validations, there are some extra things to check.
To mitigate potential upgrade issues, adhere to the following procedures.
#### Back up your Solr collections
Back up all Solr collections in each environment before continuing with the upgrade. For this upgrade, backups are intended as an emergency failsafe.
Performing a rollback after encountering the issue described is a difficult, time-consuming process and is not recommended.
#### Run the `fm-upgrade-apps-to-solr-9` Docker utility
Use the `fm-upgrade-apps-to-solr-9` Docker utility to mitigate issues related to the change from Solr 9.1.1 and earlier to Solr 9.6.1. To begin, run the `fm-upgrade-apps-to-solr-9` Docker utility using the `DRY_RUN` environmental variable:
```bash theme={"dark"}
docker run --rm -v $(pwd):/upgrade-work -e ZK_HOST=zk:2181 -e DRY_RUN=1 lucidworks/fm-upgrade-apps-to-solr-9:1.2.0
```
The `DRY_RUN` variable prints the changes that would occur to the console without performing those actions. Review the changes thoroughly.
If the changes look correct, run the `fm-upgrade-apps-to-solr-9` Docker utility again without using the `DRY_RUN` environmental variable. The updated config files are saved to the `/upgrade-work/updated-configs` directory. The utility also creates backups for all configs in the `/upgrade-work/backup-configs`.
The `fm-upgrade-apps-to-solr-9` Docker utility has another environmental variable, `REVERT`, that allows you to revert any changes you made. To revert your changes, run:
```bash theme={"dark"}
docker run --rm -v $(pwd):/upgrade-work -e ZK_HOST=zk:2181 -e REVERT=1 lucidworks/fm-upgrade-apps-to-solr-9:1.2.0
```
#### Run the `fm-upgrade-query-rewrite` Docker utility
Next, mitigate codec issues related to the Lucene 9.10.0 update. Run the `run-the-fm-upgrade-query-rewrite-docker-utility` Docker utility `prepare` action:
```bash theme={"dark"}
kubectl run \
--image="lucidworks/fm-upgrade-query-rewrite:2.x" \
--restart=Never \
--env="HELM_RELEASE=FUSION_NAMESPACE" \
--env="TARGET_SOLR_VERSION=9.6.1" \
--env="ACTION=prepare" prepare-595-upgrade
--namespace=FUSION_NAMESPACE
```
Change `FUSION_NAMESPACE` to the name of your application namespace for the Fusion installation. You can find this value using `helm list` against your Fusion namespace. Locate the release using the `fusion` chart, and find the value in the `name` column. Typically, the release name is the same as your namespace name.
Including `--namespace=FUSION_NAMESPACE` lets the update pod runs in the correct application namespace.
The `prepare` action removes `postingsFormat="FST50"` from all collections in the environment before re-indexing data to temporary collections. When the `prepare-595-upgrade` pod shows the status `Completed`, the process is finished.
#### Upgrade your Fusion environment
Upgrade Fusion to version 5.9.5. Before beginning, ensure the Fusion admin is running and all collections are healthy. Then, [complete the General upgrade process](#general-upgrade-process) before returning to the [next step in the process](#re-run-the-fm-upgrade-query-rewrite-docker-utility).
Alternatively, your organization may use a custom upgrade process. In either case, ensure you have successfully upgraded to Solr 9.6.1 as part of the Fusion upgrade.
**Do not** make changes to the `signals` collection with the Rules Editor during the upgrade process. For production clusters, upgrade during a maintenance window.
#### Re-run the `fm-upgrade-query-rewrite` Docker utility
Use the `run-the-fm-upgrade-query-rewrite-docker-utility` utility’s `restore` action to restore the data from the temporary collections created by the `prepare` action. Before you begin, verify all collections appended with `_temp_fix` are online and healthy.
```bash theme={"dark"}
kubectl run \
--image="lucidworks/fm-upgrade-query-rewrite:2.x" \
--restart=Never \
--env="HELM_RELEASE=FUSION_NAMESPACE" \
--env="TARGET_SOLR_VERSION=9.6.1" \
--env="ACTION=restore" restore-595-upgrade
--namespace=FUSION_NAMESPACE
```
Change `FUSION_NAMESPACE` to the name of your application namespace for the Fusion installation. You can find this value using `helm list` against your Fusion namespace. Locate the release using the `fusion` chart, and find the value in the `name` column. Typically, the release name is the same as your namespace name.
Including `--namespace=FUSION_NAMESPACE` lets the update pod runs in the correct application namespace.
When the `restore-595-upgrade` pod shows the status `Completed`, the process is finished.
For a complete summary of what this action does, refer to [Upgrade Utility](#run-the-fm-upgrade-query-rewrite-docker-utility).
#### Validate the upgrade
In addition to your typical validation process, ensure Solr collections are healthy:
1. Log into Fusion as the admin.
2. Access the Solr Admin UI at `https://FUSION_HOST:FUSION_PORT/api/solrAdmin/default/#/`.
3. Watch for error messages. For example, the following message reports errors for the query rewrite staging collections Acme1, Acme2, Acme3, and Acme4:
4. Navigate to the **Cloud graph** screen at `https://FUSION_HOST:FUSION_PORT/api/solrAdmin/default/#/~cloud?view=graph`.
5. Review each collection for errors.
After you have completed validations, delete the temporary `prepare` and `restore` pods that were created by the upgrade utility:
```bash theme={"dark"}
kubectl delete po prepare-595-upgrade
kubectl delete po restore-595-upgrade
```
#### Resolving post-upgrade issues
If you followed the previous instructions, and you are still experiencing issues with the codec, you need to re-feed the affected data. Contact [Lucidworks Support](https://support.lucidworks.com/hc/en-us) for further support.
### to 5.10.y
Upgrading from 5.9.x to 5.10.y involves using the [General upgrade process](#general-upgrade-process).
## General upgrade process
Fusion natively supports deployments on supported Kubernetes platforms, including AKS, EKS, and GKE.
Fusion includes an upgrade script for AKS, EKS, and GKE. This script is not generated for other Kubernetes deployments.
Upgrades differ from platform to platform. See below for more information about upgrading on your platform of choice.
Whenever you upgrade Fusion, you must also update your [remote connectors](/docs/fusion-connectors/developers/remote-v2-connectors), if you are running any.
You can download the latest files at [V2 Connectors Downloads](/docs/fusion-connectors/downloads/v2-connectors-downloads).
### Natively supported deployment upgrades
| Deployment type | Platform |
| - | - |
| **Azure Kubernetes Service (AKS)** | `aks` |
| **Amazon Elastic Kubernetes Service (EKS)** | `eks` |
| **Google Kubernetes Engine (GKE)** | `gke` |
Fusion includes upgrade scripts for natively supported deployment types. To upgrade:
1. Open the `___upgrade_fusion.sh` upgrade script file for editing.
2. Update the `CHART_VERSION` to your target Fusion version, and save your changes.
3. Run the `___upgrade_fusion.sh` script. The `` value is the same as your namespace, unless you overrode the default value using the `-r` option.
After running the upgrade, use `kubectl get pods` to see the changes applied to your cluster. It may take several minutes to perform the upgrade, as new Docker images are pulled from DockerHub. To see the versions of running pods, do:
```
kubectl get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq
```
## New Features
### Neural Hybrid Search
Fusion 5.9.5 introduces [Neural Hybrid Search](/docs/5/fusion/hybrid-search/overview), a capability that combines lexical and semantic vector search. This feature includes:
* A new index pipeline to vectorize fields with [Lucidworks AI](/docs/lw-platform/lw-ai/overview). See **Configure the LWAI Vectorize pipeline**.
* A new query pipeline to set up Neural Hybrid Search with Lucidworks AI. See **Configure the LWAI Neural Hybrid Search pipeline**.
* Query and index stages for vectorizing text using Lucidworks AI. See [LWAI Vectorize Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/lwai-vectorize-query) and [LWAI Vectorize Field stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/lwai-vectorize-field).
* Query and index stages for vectorizing text with Seldon. See [Seldon Vectorize Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/vectorize-query-via-seldon-query-stage) and [Seldon Vectorize Field stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/vectorize-field-via-seldon-index-stage).
* A new query stage for hybrid search that works with Lucidworks AI or Seldon. See [Hybrid Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/hybrid-search-query).
* A new service, `lwai-gateway`, provides a secure, authenticated connection between Fusion and your Lucidworks AI-hosted models.\
See [Lucidworks AI Gateway](/docs/lw-platform/lw-ai/lw-ai-gateway) for details.
* Solr config changes to support dense vector dynamic fields.
* A custom Solr plugin containing a new `vectorSimilarity` QParser that will not be available in Apache Solr until 9.7.
The course for **Neural Hybrid Search** focuses on how neural hybrid search combines lexical and semantic search to improve the relevance and accuracy of results.
#### Configure use case for embedding
In the LWAI Vectorize Field stage, you can specify the use case for your embedding model.
To learn how to configure your embedding use case, see the following demonstration:
#### Fine tune lexical and semantic settings
The Hybrid Query stage is highly customizable. You can lower the **Min Return Vector Similarity** threshold for vector results to include more semantic results. For example, a lower threshold would return "From Dusk Till Dawn" when querying `night` against a movie dataset. A higher threshold prioritizes high scoring results and in this case only returns movie names with `night` in the title.
To learn how to configure the Hybrid Query stage, see the following demonstration:
#### Vector dimension size
There is no limitation on vector dimension sizes. If you’re setting up vector search and Neural Hybrid Search with an embedding model with large dimensions, simply configure your managed-schema to support the appropriate dimension. See **Configure Neural Hybrid Search**.
Neural Hybrid Search combines lexical-semantic search with semantic vector search.
To use semantic vector search in Fusion, you need to configure Neural Hybrid Search.
Then you can choose the balance between lexical and semantic vector search that works best for your use case.
Before you begin, see [Neural Hybrid Search](/docs/5/fusion/hybrid-search/overview) for conceptual information that can help you understand how to configure this feature.
This feature is currently only available to clients who have contracted with Lucidworks for features related to Neural Hybrid Search and Lucidworks AI.This feature is available starting in Fusion 5.9.5 and in all subsequent Fusion 5.9 releases.
Lucidworks recommends setting up Neural Hybrid Search with Lucidworks AI, but you can instead use Ray or Seldon vector search. If using Lucidworks AI, you may use the default LWAI Neural Hybrid Search pipeline.
## Configure vector search
This section explains how to configure vector search using Lucidworks AI, but you can also configure it using Ray or Seldon.
Before you set up the Lucidworks AI index and query stages, make sure you have set up your Lucidworks AI Gateway integration.
### Configure the LWAI Vectorize Field index stage
To vectorize the index pipeline fields:
1. Sign in to Fusion and click **Indexing > Index Pipelines**.
2. Click the pipeline you want to use.
3. Click **Add a new pipeline stage**.
4. In the AI section, click **LWAI Vectorize Field**.
5. In the **Label** field, enter a unique identifier for this stage.
6. In the **Condition** field, enter a script that results in true or false, which determines if the stage should process.
7. In the **Account Name** field, select the Lucidworks AI API account name defined in the Lucidworks AI Gateway service.
If you do not see your account name, check that your Lucidworks AI Gateway integration is correctly configured.
8. In the **Model** field, select the Lucidworks AI model to use for encoding.
If you do not see any models names and you are a non-admin Fusion user, check that you have these permissions: `PUT,POST,GET:/LWAI-ACCOUNT-NAME/**`
Your Fusion account name must match the name of the account that you selected in the **Account Name** dropdown.
For more information about models, see:
* [Pre-trained embedding models](/docs/lw-platform/lw-ai/lw-ai-pre-trained-embedding-models)
* [Custom embedding model training](/docs/lw-platform/lw-ai/lw-ai-custom-embedding-model-training/overview)
9. In the **Source** field, enter the name of the string field where the value should be submitted to the model for encoding. If the field is blank or does not exist, this stage is not processed. Template expressions are supported.
10. In the **Destination** field, enter the name of the field where the vector value from the model response is saved.
If a value is entered in this field, the following information is added to the document:
* `{Destination Field}` is the vector field.
* `{Destination Field}_b` is the boolean value if the vector has been indexed.
11. In the **Use Case Configuration** section, click the **+** sign to enter the parameter name and value to send to Lucidworks AI. The `useCaseConfig` parameter that is common to embedding use cases is `dataType`, but each use case may have other parameters. The value for the query stage is `query`.
12. Optionally, you can use the **Model Configuration** section for any additional parameters you want to send to Lucidworks AI.
Several `modelConfig` parameters are common to generative AI use cases.
For more information, see [Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/overview).
13. Select the **Fail on Error** checkbox to generate an exception if an error occurs while generating a prediction for a document.
14. Click **Save**.
15. Index data using the new pipeline. Verify the vector field is indexed by confirming the field is present in documents.
For reference information, see [Lucidworks AI Vectorize Field](/docs/lw-platform/lw-ai/lw-ai-stages/vectorize-field-via-lucidworks-ai-index-stage).
### Configure the LWAI Vectorize Query stage
To vectorize the query in the query pipeline:
1. Sign in to Fusion and click **Querying > Query Pipelines**.
2. Select the pipeline you want to use.
3. Click **Add a new pipeline stage**.
4. Click **LWAI Vectorize Query**.
5. In the **Label** field, enter a unique identifier for this stage.
6. In the **Condition** field, enter a script that results in true or false, which determines if the stage should process.
7. Select **Asynchronous Execution Config** if you want to run this stage asynchronously. If this field is enabled, complete the following fields:
1. Select **Enable Async Execution**. Fusion automatically assigns an **Async ID** value to this stage. Change this to a more memorable string that describes the asynchronous stages you are merging, such as `signals` or `access_control`.
2. Copy the **Async ID** value.
For detailed information, see [Asynchronous query pipeline processing](/docs/5/fusion/getting-data-out/query-basics/query-pipelines/overview).
8. In the **Account Name** field, select the name of the Lucidworks AI account.
9. In the **Model** field, select the Lucidworks AI model to use for encoding.
If you do not see any model names and you are a non-admin Fusion user, verify with a Fusion administrator that your user account has these permissions: `PUT,POST,GET:/LWAI-ACCOUNT-NAME/**`
Your Fusion account name must match the name of the account that you selected in the **Account Name** dropdown.
For more information about models, see:
* [Pre-trained embedding models](/docs/lw-platform/lw-ai/lw-ai-pre-trained-embedding-models)
* [Custom embedding model training](/docs/lw-platform/lw-ai/lw-ai-custom-embedding-model-training/overview)
10. In the **Query Input** field, enter the location from which the query is retrieved.
11. In the **Output context variable** field, enter the name of the variable where the vector value from the response is saved.
12. In the **Use Case Configuration** section, click the **+** sign to enter the parameter name and value to send to Lucidworks AI. The `useCaseConfig` parameter that is common to embedding use cases is `dataType`, but each use case may have other parameters. The value for the query stage is `query`.
13. Optionally, you can use the **Model Configuration** section for any additional parameters you want to send to Lucidworks AI.
Several `modelConfig` parameters are common to generative AI use cases.
For more information, see [Prediction API](/docs/lw-platform/lw-ai/lw-ai-apis/lw-ai-prediction-api/overview).
14. Select the **Fail on Error** checkbox to generate an exception if an error occurs during this stage.
15. Click **Save**.
The **Top K** setting is 100 by default, but a value as high as 1000 provides better recall if you have fewer than one million indexed documents.
You can raise it even higher, but keep in mind that higher recall also causes higher latency.\
When raising this value, we recommend also setting a higher **Min Return Vector Similarity** value, in the 0.7-0.85 range.
This query stage must be placed *before* the **[Solr Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/solr-query-stage)**.
**Using additional pipeline stages**
For optimal vector search functionality, use the following stages in the order specified: either the LWAI Vectorize Query or Ray/Seldon Vectorize Field, Query Fields (if present), Neural Hybrid Query, and Solr Query. Other stages can be used, but must be placed in the correct processing order in relation to these stages.
## Modify Solr managed-schema (5.9.4 and earlier)
This step is required if you’re migrating a collection from a version of Fusion that does not support Neural Hybrid Search. If creating a new collection in Fusion 5.9.5 and later, you can continue to [Configure Hybrid Query stage](#configure-neural-hybrid-queries).
1. Go to **System** > **Solr Config** and then click **managed-schema** to edit it.
2. Comment out `` and add `` below it. This will concatenate and index all `*_t fields`.
3. Add the following code block to the **managed-schema** file:
```xml theme={"dark"}
```
This example uses 512 vector dimension. If your model uses a different dimension, modify the code block to match your model. For example, `_1024v`. There is no limitation on supported vector dimensions.
## Configure neural hybrid queries
In Fusion 5.9.10 and later, you use the Neural Hybrid Query stage to configure neural hybrid queries.
In Fusion 5.9.9 and earlier, you use the Hybrid Query stage.
### Configure Neural Hybrid Query stage (5.9.10 and later)
1. In the same query pipeline where you configured vector search, click **Add a new pipeline stage**, then select **Neural Hybrid Query**.
2) In the **Label** field, enter a unique identifier for this stage or leave blank to use the default value.
3) In the **Condition** field, enter a script that results in true or false, which determines if the stage should process, or leave blank.
4) In the **Lexical Query Input** field, enter the location from which the lexical query is retrieved. For example, **\**. Template expressions are supported.
5) In the **Lexical Query Weight** field, enter the relative weight of the lexical query. For example, **0.3**. If this value is **0**, no re-ranking will be applied using the lexical query scores.
6) In the **Lexical Query Squash Factor** field, enter a value that will be used to squash the lexical query score.
The squash factor controls how much difference there is between the top-scoring documents and the rest.
It helps ensure that documents with slightly lower scores still have a chance to show up near the top.
For this value, Lucidworks recommends entering the inverse of the lexical maximum score across all queries for the given collection.
7) In the **Vector Query Field**, enter the name of the Solr field for k-nearest neighbor (KNN) vector search.
8) In the **Vector Input** field, enter the location from which the vector is retrieved. Template expressions are supported. For example, a value of `` evaluates the context variable resulting from a previous stage, such as the [LWAI Vectorize Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/lwai-vectorize-query) stage.
9) In the **Vector Query Weight** field, enter the relative weight of the vector query. For example, **0.7**.
10) In the **Min Return Vector Similarity** field, enter the minimum vector similarity value to qualify as a match from the Vector portion of the hybrid query.
11) In the **Min Traversal Vector Similarity** field, enter the minimum vector similarity value to use when walking through the graph during the Vector portion of the hybrid query.
12) When enabled, the **Compute Vector Similarity for Lexical-Only Matches** setting computes vector similarity scores for documents in lexical search results but not in the initial vector search results. Select the checkbox to enable this setting.
13) If you want to use pre-filtering:
1. Uncheck **Block pre-filtering**.
In the Javascript context (`ctx`), the `preFilterKey` object becomes available.
2. Add a [Javascript stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/javascript-query-stage) *after* the Neural Hybrid Query stage and use it to configure your pre-filter.
The `preFilter` object adds both the top-level `fq` and `preFilter` to the parameters for the vector query.
You do not need to manually add the top level `fq` in the javascript stage.
See the example below:
```js theme={"dark"}
var QueryRequestAndResponse = Java.type('com.lucidworks.apollo.pipeline.query.QueryRequestAndResponse');
if(ctx.hasProperty("preFilterKey")) {
var preFilter = ctx.getProperty("preFilterKey");
var wrapper = QueryRequestAndResponse.create(request,response,0)
preFilter.addFilter(wrapper, 'id:* OR foo_s:bar');
}
```
14) Click **Save**.
Make sure the **Hybrid Query** stage is ordered before the **Solr Query** stage.
Be aware that the Neural Hybrid Query stage uses new query parsers, so if you are *not* setting up a new collection, the following must be added to `solrconfig.xml` within the `` tag:
```xml theme={"dark"}
```
### Configure Hybrid Query stage (5.9.9 and earlier)
If you’re setting up Neural Hybrid Search in Fusion 5.9.9 and earlier, use the Hybrid Query stage.
If you’re using Fusion 5.9.10 or later, use the [Neural Hybrid Query stage](#configure-neural-hybrid-queries).
1. In the same query pipeline where you configured vector search, click **Add a new pipeline stage**, then select **Hybrid Query**.
2) In the **Label** field, enter a unique identifier for this stage or leave blank to use the default value.
3) In the **Condition** field, enter a script that results in true or false, which determines if the stage should process, or leave blank.
4) In the **Lexical Query Input** field, enter the location from which the lexical query is retrieved. For example, **\**. Template expressions are supported.
5) In the **Lexical Query Weight** field, enter the relative weight of the lexical query. For example, **0.3**. If this value is **0**, no re-ranking will be applied using the lexical query scores.
6) In the **Number of Lexical Results** field, enter the number of lexical search results to include in re-ranking. For example, **1000**. A value is **0** is ignored.
7) In the **Vector Query Field**, enter the name of the Solr field for k-nearest neighbor (KNN) vector search.
8) In the **Vector Input** field, enter the location from which the vector is retrieved. Template expressions are supported. For example, a value of `` evaluates the context variable resulting from a previous stage, such as the [LWAI Vectorize Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/lwai-vectorize-query) stage.
9) In the **Vector Query Weight** field, enter the relative weight of the vector query. For example, **0.7**.
10) Select the **Use KNN Query** checkbox to use the **knn** query parser and configure its options. This option cannot be selected if **Use VecSim Query** checkbox is selected. In addition, **Use KNN Query** is used if neither **Use KNN Query** or **Use VecSim Query** is selected.
1. If the **Use KNN Query checkbox** is selected, enter a value in the **Number of Vector Results** field. For example, **1000**.
11) Select the **Use VecSim Query** checkbox to use the **vecSim** query parser and configure its options. This option cannot be selected if **Use KNN Query** checkbox is selected.
If the **Use VecSim Query** checkbox is selected, enter values in the following fields:
* **Min Return Vector Similarity**. Enter the minimum vector similarity value to qualify as a match from the Vector portion of the hybrid query.
* **Min Traversal Vector Similarity**. Enter the minimum vector similarity value to use when walking through the graph during the Vector portion of the hybrid query. The value must be lower than, or equal to, the value in the Min Return Vector Similarity field.
12. In the **Minimum Vector Similarity Filter**, enter the value for a minimum similarity threshold for filtering documents. This option applies to all documents, regardless of other score boosting such as rules or signals.
13. Click **Save**.
Make sure the **Hybrid Query** stage is ordered before the **Solr Query** stage.
## Perform hybrid searches
After setting up the stages, you can perform hybrid searches via the [`knn` query parser](https://solr.apache.org/guide/solr/latest/query-guide/dense-vector-search.html#knn-query-parser) as you would with Solr.
Specify the search vector and include it in the query.
For example, change the `q` parameter to a `knn` query parser string.
You can also preview the results in the [Query Workbench](/docs/5/fusion/getting-data-out/query-basics/query-workbench).
Try a few different queries, and adjust the weights and parameters in the Hybrid Query stage to find the best balance between lexical and semantic vector search for your use case.
You can also disable and re-enable the Neural Hybrid Query stage to compare results with and without it.
`XDenseVectorField` is not supported in Fusion 5.9.5 and above. Instead, use `DenseVectorField`.
## Troubleshoot inconsistent results
Neural Hybrid Search leverages Solr semantic vector search, which has known behaviors which can be inconsistent at query time.
These behaviors include score fluctuations with re-querying, documents showing and disappearing on re-querying, and (when SVS is configured without Hybrid stages) completely unfindable documents.
This section outlines possible reasons for inconsistent behavior and resolutions steps.
### NRT replicas and HNSW graph challenges
Lucidworks recommends using PULL and TLOG replicas. These replica types copy the index of the leader replica, which results in the same HNSW graph on every replica. When querying, the HNSW approximation query will be consistent given a static index.
In contrast, NRT replicas have their own index, so they will also have their own HNWS graph. HNSW is an Approximate Nearest Neighbor (ANN) algorithm, so it will not return exactly the same results for differently constructed graphs. This means that queries performed can and will return different results per HNWS graph (number of NRT replicas in a shard) which can lead to noticeable result shifts. When using NRT replicas, the shifts can be made less noticeable by increasing the `topK` parameter. Variation will still occur, but it should be lower in the documents. Another way to mitigate shifts is to use Neural Hybrid Search with a vector similarity cutoff.
For more information, refer to [Solr Types of Replicas](https://solr.apache.org/guide/solr/latest/deployment-guide/solrcloud-shards-indexing.html#types-of-replicas).
In the case of Neural Hybrid Search, lexical BM25 & TF-IDF score differences that can occur with NRT replicas because of index differences for deleted documents, can also affect combined [Hybrid score](/docs/5/fusion/hybrid-search/overview).
If you choose to use NRT replicas then it is possible that any lexical and/or semantic vectors variations can and will be exacerbated.
### Orphaning (Disconnected Nodes)
Solr’s implementation of dense vector search depends on the Lucene implementation of HNSW ANN.
The Lucene implementation has a known issue where, in some collections, nodes in the HNSW graph become unreachable via graph traversal, essentially becoming disconnected or “orphaned.”
#### Identify orphaning
Run the following command to identify orphaning:
```bash theme={"dark"}
curl -sS -u 'USERNAME:PASSWORD' 'https://FUSION_HOST:FUSION_PORT/api/solrAdmin/default/COLLECTION_NAME/select'\
--form-string 'fl=id,vecSim:$vecSim' \
--form-string 'rows=1' \
--form-string 'q=(*:* -{!knn f=VECTOR_FIELD topK=999999 v=$vec})' \
--form-string 'vecSim=vectorSimilarity(VECTOR_FIELD,$vec)' \
--form-string 'vec=COMPATIBLE_VECTOR'
```
If the collection doesn’t have a vector for every document, include a filter so only the documents that have vectors are included. Filter on the boolean vector, as in this example: `--form-string 'fq=VECTOR_FIELD_b:true' \`
Construct a KNN exclusion query where topK is higher than the number of vectors in your collection
If the number of vectors in your collection exceeds 999,999 then increase the value to be at least equal to that value.
If any are documents returned, there are orphans, and the `ids` you see are the orphans.
Proceed to [Resolving orphans](#resolving-orphans).
If no documents are returned, there are likely no orphans.
You can try a few varying vectors to be certain.
#### Resolving orphans
To resolve orphans, do the following:
1. Increase the HNSW Solr schema parameters `hnswBeamWidth` and `hnswMaxConnections` per the [Suggested values](#suggested-values) below.
2. Save the schema.
3. Clear the index.
4. Re-index your collection.
##### Suggested values
| Orphaning rate | `hnswBeamWidth` | `hnswMaxConnections` |
| - | - | - |
| 5% or less | 300 | 64 |
| 5% - 25% | 500 | 100 |
| 25% or more | 3200 | 512 |
## Improvements
* Fusion now supports Kubernetes 1.30 for GKE. Refer to Kubernetes documentation at [Kubernetes v1.30](https://kubernetes.io/blog/2024/04/17/kubernetes-v1-30-release/) for more information.
* Solr has been upgraded to 9.6.1.
* Zookeeper has been upgraded to 3.9.1.
* The default value for `kafka.logRetentionBytes` is increased to 5 GB. This improvement helps prevent failed datasource jobs due to full disk space. Refer to **Troubleshoot failed datasource jobs**.
When indexing large files, or large quantities of files, you may encounter issues such as datasource jobs failing or documents not making it into Fusion.
## Overview
When data flows into Fusion, it passes through a Kafka topic first.
When the number of documents being created by a connector is large, or when the connector is pulling data into the Kafka topic faster than it can be indexed, the topic fills up and the datasource job fails.
For example, if your connector is inputting a large CSV file where every row is imported as a separate Solr document, the indexing processing can time out before the document is fully ingested.
## Identify the cause
If you experience failed datasource jobs or notice your connector isn’t grabbing all the documents it should, check the logs for the Kafka pod.
Look for a message containing the phrases `resetting offset` and `is out of range`, which indicate data has been dropped.
```bash theme={"dark"}
2024-05-28T11:49:40.812Z - INFO [pool-140-thread-3:org.apache.kafka.clients.consumer.internals.Fetcher@1413] - [Consumer clientId=example_Products-irdcsn, groupId=index-pipeline--example_Products--fusion.connectors.datasource-products_S3_Load] Fetch position FetchPosition{offset=6963199, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[fusion5-kafka-0.fusion5-kafka-headless.fusion5.svc.cluster.local:9092 (id: 0 rack: null)], epoch=0}} is out of range for partition fusion.connectors.datasource-products_S3_Load-2, resetting offset
```
## Adjust indexing settings
If you determine that your datasource job is failing due to an issue in Kafka, there are a few options to try.
### Adjust retention parameters
One solution is to increase the Kafka data retention parameters to allow for larger documents.
You can configure these settings in your `values.yaml` file in the Helm chart.
1. The default value for `kafka.logRetentionBytes` is `1073741824` bytes (1 GB).
Try increasing this value to `2147483648` bytes (2 GB) or `3221225472` (3 GB), or larger depending on the size of your documents.
In Fusion 5.9.5, the default value is increased to 5 GB.
You can also set this to `-1` to remove the size limit.
If you do this, be sure to set an appropriate limit for `logRetentionHours` instead.
2. The default value for `kafka.logRetentionHours` is `168` (7 days).
If you increase `kafka.logRetentionBytes` by a significant amount (for example, 20 GB), you might need to decrease this setting to prevent running out of disk space.
However, because older log entries are deleted when either limit is reached, you should set it high enough to ensure the data remains available until it’s no longer needed.
3. In Fusion, go to **Indexing > Datasources** and create a new datasource to trigger a new Kafka topic that incorporates these settings.
### Adjust fetch settings
Another option is to decrease the values for number of fetch threads and request page size in your datasource settings.
1. In Fusion, go to **Indexing > Datasources** and click your datasource.
2. Click the **Advanced** slider to show more settings.
3. Reduce the number of **Fetch Threads**.
4. Reduce the **Request Page Size**.
This setting might not be available in every connector.
* There is a new AI category in the **Add a new pipeline stage** dropdown for Query and Index Pipelines. This category contains the new stages for Neural Hybrid Search, as well as existing machine learning and AI stages.
* The Fusion migration script is updated to align with changes from the Solr upgrade. The migration script:
* Removes the unused configuration, ``, from `solrconfig.xml`. Solr no longer supports this configuration.
* Removes the query response writer of class `solr.XSLTResponseWriter`.
* Comments out processors of type `solr.StatelessScriptUpdateProcessorFactory`.
* Removes `` element from request handler.
* Changes cache class attribute of elements `"filterCache"`, `"cache"`, `"documentCache"`, `"queryResultCache"` to `solr.search.CaffeineCache`.
* Removes `keepShortTerm` attribute from filter of class `solr.NGramFilterFactory`.
* Added the parameter `job-expiration-duration-seconds` for remote connectors that lets you configure the timeout value. Refer to **Configure Remote V2 Connectors**.
If you need to index data from behind a firewall, you can configure a V2 connector to run remotely on-premises using TLS-enabled gRPC.
## Prerequisites
Before you can set up an on-prem V2 connector, you must configure the egress from your network to allow HTTP/2 communication into the Fusion cloud. You can use a [forward proxy server](#egress-and-proxy-server-configuration) to act as an intermediary between the connector and Fusion.
The following is required to run V2 connectors remotely:
* The [plugin zip file and the connector-plugin-standalone JAR](https://plugins.lucidworks.com/).
* A configured connector backend gRPC endpoint.
* Username and password of a user with a `remote-connectors` or `admin` role.
* If the host where the remote connector is running is not configured to trust the server’s TLS certificate, you must configure the file path of the trust certificate collection.
If your version of Fusion doesn’t have the `remote-connectors` role by default, you can create one. No API or UI permissions are required for the role.
## Connector compatibility
Only V2 connectors are able to run remotely on-premises.
You also need the remote connector client JAR file that matches your Fusion version.
You can download the latest files at [V2 Connectors Downloads](/docs/fusion-connectors/downloads/v2-connectors-downloads).
Whenever you upgrade Fusion, you must also update your remote connectors to match the new version of Fusion.
The gRPC connector backend is not supported in Fusion environments deployed on AWS.
## System requirements
The following is required for the on-prem host of the remote connector:
* (Fusion 5.9.0-5.9.10) JVM version 11
* (Fusion 5.9.11) JVM version 17
* Minimum of 2 CPUs
* 4GB Memory
Note that memory requirements depend on the number and size of ingested documents.
## Enable backend ingress
In your `values.yaml` file, configure this section as needed:
```yaml theme={"dark"}
ingress:
enabled: false
pathtype: "Prefix"
path: "/"
#host: "ingress.example.com"
ingressClassName: "nginx" # Fusion 5.9.6 only
tls:
enabled: false
certificateArn: ""
# Enable the annotations field to override the default annotations
#annotations: ""
```
* Set `enabled` to `true` to enable the backend ingress.
* Set `pathtype` to `Prefix` or `Exact`.
* Set `path` to the path where the backend will be available.
* Set `host` to the host where the backend will be available.
* In Fusion 5.9.6 *only*, you can set `ingressClassName` to one of the following:
* `nginx` for Nginx Ingress Controller
* `alb` for AWS Application Load Balancer (ALB)
* Configure TLS and certificates according to your CA’s procedures and policies.
TLS must be enabled in order to use AWS ALB for ingress.
## Connector configuration example
```yaml theme={"dark"}
kafka-bridge:
target: mynamespace-connectors-backend.lucidworkstest.com:443 # mandatory
plain-text: false # optional, false by default.
proxy-server: # optional - needed when a forward proxy server is used to provide outbound access to the standalone connector
host: host
port: some-port
user: user # optional
password: password # optional
trust: # optional - needed when the client's system doesn't trust the server's certificate
cert-collection-filepath: path1
proxy: # mandatory fusion-proxy
user: admin
password: password123
url: https://fusiontest.com/ # needed only when the connector plugin requires blob store access
plugin: # mandatory
path: ./fs.zip
type: #optional - the suffix is added to the connector id
suffix: remote
```
### Minimal example
```yaml theme={"dark"}
kafka-bridge:
target: mynamespace-connectors-backend.lucidworkstest.com:443
proxy:
user: admin
password: "password123"
plugin:
path: ./testplugin.zip
```
### Logback XML configuration file example
```xml theme={"dark"}
%d - %-5p [%t:%C{3.}@%L] - %m{nolookups}%nutf8${LOGDIR:-.}/connector.log${LOGDIR:-.}/connector-%d{yyyy-MM-dd}.%i.log.gz50MB10GB%d - %-5p [%t:%C{3.}@%L] - %m{nolookups}%nutf8
```
## Run the remote connector
```java theme={"dark"}
java [-Dlogging.config=[LOGBACK_XML_FILE]] \
-jar connector-plugin-client-standalone.jar [YAML_CONFIG_FILE]
```
The `logging.config` property is optional. If not set, logging messages are sent to the console.
## Test communication
You can run the connector in communication testing mode. This mode tests the communication with the backend without running the plugin, reports the result, and exits.
```java theme={"dark"}
java -Dstandalone.connector.connectivity.test=true -jar connector-plugin-client-standalone.jar [YAML_CONFIG_FILE]
```
## Encryption
In a deployment, communication to the connector’s backend server is encrypted using TLS. You should only run this configuration without TLS in a testing scenario. To disable TLS, set `plain-text` to `true`.
## Egress and proxy server configuration
One of the methods you can use to allow outbound communication from behind a firewall is a proxy server. You can configure a proxy server to allow certain communication traffic while blocking unauthorized communication. If you use a proxy server at the site where the connector is running, you must configure the following properties:
* **Host.** The hosts where the proxy server is running.
* **Port.** The port the proxy server is listening to for communication requests.
* **Credentials.** Optional proxy server user and password.
When you configure egress, it is important to disable any connection or activity timeouts because the connector uses long running gRPC calls.
## Password encryption
If you use a login name and password in your configuration, run the following utility to encrypt the password:
1. Enter a user name and password in the connector configuration YAML.
2. Run the standalone JAR with this property:
```java theme={"dark"}
-Dstandalone.connector.encrypt.password=true
```
3. Retrieve the encrypted passwords from the log that is created.
4. Replace the clear password in the configuration YAML with the encrypted password.
## Connector restart (5.7 and earlier)
The connector will shut down automatically whenever the connection to the server is disrupted, to prevent it from getting into a bad state. Communication disruption can happen, for example, when the server running in the `connectors-backend` pod shuts down and is replaced by a new pod. Once the connector shuts down, connector configuration and job execution are disabled. To prevent that from happening, you should restart the connector as soon as possible.
You can use Linux scripts and utilities to restart the connector automatically, such as [Monit](https://mmonit.com/monit/).
## Recoverable bridge (5.8 and later)
If communication to the remote connector is disrupted, the connector will try to recover communication and gRPC calls. By default, six attempts will be made to recover each gRPC call. The number of attempts can be configured with the `max-grpc-retries` bridge parameters.
## Job expiration duration (5.9.5 only)
The timeout value for irresponsive backend jobs can be configured with the `job-expiration-duration-seconds` parameter. The default value is `120` seconds.
## Use the remote connector
Once the connector is running, it is available in the Datasources dropdown. If the standalone connector terminates, it disappears from the list of available connectors. Once it is re-run, it is available again and configured connector instances will not get lost.
## Enable asynchronous parsing (5.9 and later)
To separate document crawling from document parsing, enable Tika Asynchronous Parsing on remote V2 connectors.
* Added additional diagnostics between the `connectors-backend` and `fusion-indexing` services.
* Added more detail to the messages that appear in the Fusion UI when a connector job fails.
* Added the `reset` action parameter to the `subscriptions/{id}/refresh?action=some-action` POST API endpoint. Calling `reset` will clear the subscription indexing topic from pending documents. See the [Indexing API](/api-reference/subscriptions-api/refresh-a-subscription).
## Bug fixes
* Fixed an issue that prevented successful configuration of new Kerberos security realms for authentication of external applications.
## Deprecations and removals
For full details on deprecations, see [Deprecations and Removals](/docs/5/fusion/deprecations-and-removals).
### Bitnami removal
Fusion 5.9.5 will be re-released with the same functionality but updated image references.
In the meantime, Lucidworks will self-host the required images while we work to replace Bitnami images with internally built open-source alternatives.
If you are a self-hosted Fusion customer, *you must upgrade before August 28* to ensure continued access to container images and prevent deployment issues.
You can reinstall your current version of Fusion or upgrade to Fusion 5.9.14, which includes the updated Helm chart and prepares your environment for long-term compatibility.
See [Prevent image pull failures due to Bitnami deprecation in Fusion 5.9.5 to 5.9.13](https://support.lucidworks.com/hc/en-us/articles/33966125467799-Prevent-image-pull-failures-due-to-Bitnami-deprecation-in-Fusion-5-9-5-to-5-9-13) for more information on how to prevent image pull failures.
### Milvus deprecation
With the release of Solr supported embeddings and Solr Semantic Vector Search, Lucidworks is deprecating Milvus. The following Milvus query stages are deprecated and will be removed in a future release:
* Milvus Ensemble Query Stage
* Milvus Query Stage
* Milvus Response Update Query Stage
Use [Seldon](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/vectorize-query-via-seldon-query-stage) or [Lucidworks AI](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/lwai-vectorize-query) vector query stages instead.
For more information, see [Deprecations and Removals](/docs/5/fusion/deprecations-and-removals).
# Fusion 5.9.6
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9/5.9.6-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.9.6-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9.6-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/0j9qej
Released on December 2, 2024,
this [maintenance release](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#maintenance-release-support-policy) includes
the new cloud signals feature, support for Open Nashorn, plus bug fixes and other minor improvements.
To learn more, skip to the [release notes](#new-features).
**Security patch available for api-gateway: Netty request smuggling vulnerabilities**
A patch is available for the `api-gateway` service to address critical Netty request smuggling vulnerabilities (CVE-2026-42581, CVE-2026-42585, CVE-2026-42587). These vulnerabilities allow attackers to smuggle HTTP requests through the gateway, potentially bypassing security controls.
The `api-gateway` service requires the Netty security patch.
Follow these steps to apply the patched image:
1. Open your Fusion Helm values file.
2. Add or update the `api-gateway` image configuration:
```yaml theme={"dark"}
api-gateway:
image:
repository: lucidworks
name: api-gateway
tag: 5.9.9-SUST-1634-patch
imagePullPolicy: IfNotPresent
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace, `RELEASE_NAME` with your Helm release name, and `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the `api-gateway` pods to restart and verify they are using the patched image.
**Urgent action required by November 26, 2025**
A patch is required by November 26, 2025 for all self-hosted Fusion deployments running on Amazon Elastic Kubernetes Service (EKS). Certain Java versions used by Fusion components reach end of life on this date. Failure to apply the patch will result in compatibility issues.
The following Fusion services require the `cgroupv2` patch:
| Service | Affected Fusion versions | Patch tag |
| - | - | - |
| `insights` | 5.9.4 to 5.9.15 | `lucidworks/insights:5.9-cgroupv2-patch` |
| `spark-solr-etl` | 5.9.4 to 5.9.11 | `lucidworks/spark-solr-etl:5.9-cgroupv2-patch` |
| `keytool-utils` | 5.9.4 to 5.9.10 | `lucidworks/keytool-utils:5.9-cgroupv2-patch` |
Follow these steps to apply the patched images:
1. Open your Fusion Helm values file. For Fusion Cloud Native deployments, use the values file for your current deployment. For Helm deployments, use the values file you used to create the deployment.
2. For each service listed in the following table that applies to your Fusion version, add or update the image configuration:
Expand the following code snippet for the complete image configuration list.
```yaml expandable theme={"dark"}
global:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
sql-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
reverse-search:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
solr:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
kafka:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
ml-model-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-admin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-indexing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
query-pipeline:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
async-parsing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
admin-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
api-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
auth-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
classic-rest-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-resources:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-config:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
insights:
image:
imagePullPolicy: Always
name: insights
repository: lucidworks
tag: 5.9-cgroupv2-patch
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-launcher:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-rest-server:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connector-plugin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors-backend:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
rules-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
pm-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
lwai-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
webapps:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
apps-manager:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
templating:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
tikaserver:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
seldon-core-operator:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo-common-workflows:
image:
imagePullPolicy: Always
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
utilitiesTag: 5.9.11
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: 5.9-cgroupv2-patch
argo-common-workflows:
image:
imagePullPolicy: IfNotPresent
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: insights:5.9-cgroupv2-patch
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run the following command:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace. Replace `RELEASE_NAME` with your Helm release name. Replace `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the affected service pods to restart and verify they are using the patched images.
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
## Platform Support and Component Versions
### Kubernetes platform support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.28, 1.29, 1.30
* **Microsoft Azure Kubernetes Service (AKS):** 1.28, 1.29, 1.30
* **Amazon Elastic Kubernetes Service (EKS):** 1.28, 1.29, 1.30
Support is also offered for Rancher Kubernetes Engine (RKE) and OpenShift 4 versions that are based on Kubernetes 1.28, 1.29, 1.30. OpenStack and customized Kubernetes installations are *not* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support).
### Component versions
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| **Solr** | fusion-solr 5.9.6 *(based on Solr 9.6.1)* |
| **ZooKeeper** | 3.9.1 |
| **Spark** | 3.2.2 |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).
## New Features
### Support for Open Nashorn
Now your pipeline definitions can include your choice of JavaScript engine, either "Nashorn" or "OpenJDK Nashorn".
You can select the JavaScript engine in the pipeline views.
Your JavaScript pipeline stages are interpreted by the selected engine.
While Nashorn is the default option, it is in the process of being deprecated and will eventually be removed, so it is recommended to use OpenJDK Nashorn when possible.
## Improvements
* Fusion’s Remote V2 Connectors now support the Amazon Web Services (AWS) Application Load Balancer (ALB) for ingress.\
See **Configure Remote V2 Connectors** for configuration details.
If you need to index data from behind a firewall, you can configure a V2 connector to run remotely on-premises using TLS-enabled gRPC.
## Prerequisites
Before you can set up an on-prem V2 connector, you must configure the egress from your network to allow HTTP/2 communication into the Fusion cloud. You can use a [forward proxy server](#egress-and-proxy-server-configuration) to act as an intermediary between the connector and Fusion.
The following is required to run V2 connectors remotely:
* The [plugin zip file and the connector-plugin-standalone JAR](https://plugins.lucidworks.com/).
* A configured connector backend gRPC endpoint.
* Username and password of a user with a `remote-connectors` or `admin` role.
* If the host where the remote connector is running is not configured to trust the server’s TLS certificate, you must configure the file path of the trust certificate collection.
If your version of Fusion doesn’t have the `remote-connectors` role by default, you can create one. No API or UI permissions are required for the role.
## Connector compatibility
Only V2 connectors are able to run remotely on-premises.
You also need the remote connector client JAR file that matches your Fusion version.
You can download the latest files at [V2 Connectors Downloads](/docs/fusion-connectors/downloads/v2-connectors-downloads).
Whenever you upgrade Fusion, you must also update your remote connectors to match the new version of Fusion.
The gRPC connector backend is not supported in Fusion environments deployed on AWS.
## System requirements
The following is required for the on-prem host of the remote connector:
* (Fusion 5.9.0-5.9.10) JVM version 11
* (Fusion 5.9.11) JVM version 17
* Minimum of 2 CPUs
* 4GB Memory
Note that memory requirements depend on the number and size of ingested documents.
## Enable backend ingress
In your `values.yaml` file, configure this section as needed:
```yaml theme={"dark"}
ingress:
enabled: false
pathtype: "Prefix"
path: "/"
#host: "ingress.example.com"
ingressClassName: "nginx" # Fusion 5.9.6 only
tls:
enabled: false
certificateArn: ""
# Enable the annotations field to override the default annotations
#annotations: ""
```
* Set `enabled` to `true` to enable the backend ingress.
* Set `pathtype` to `Prefix` or `Exact`.
* Set `path` to the path where the backend will be available.
* Set `host` to the host where the backend will be available.
* In Fusion 5.9.6 *only*, you can set `ingressClassName` to one of the following:
* `nginx` for Nginx Ingress Controller
* `alb` for AWS Application Load Balancer (ALB)
* Configure TLS and certificates according to your CA’s procedures and policies.
TLS must be enabled in order to use AWS ALB for ingress.
## Connector configuration example
```yaml theme={"dark"}
kafka-bridge:
target: mynamespace-connectors-backend.lucidworkstest.com:443 # mandatory
plain-text: false # optional, false by default.
proxy-server: # optional - needed when a forward proxy server is used to provide outbound access to the standalone connector
host: host
port: some-port
user: user # optional
password: password # optional
trust: # optional - needed when the client's system doesn't trust the server's certificate
cert-collection-filepath: path1
proxy: # mandatory fusion-proxy
user: admin
password: password123
url: https://fusiontest.com/ # needed only when the connector plugin requires blob store access
plugin: # mandatory
path: ./fs.zip
type: #optional - the suffix is added to the connector id
suffix: remote
```
### Minimal example
```yaml theme={"dark"}
kafka-bridge:
target: mynamespace-connectors-backend.lucidworkstest.com:443
proxy:
user: admin
password: "password123"
plugin:
path: ./testplugin.zip
```
### Logback XML configuration file example
```xml theme={"dark"}
%d - %-5p [%t:%C{3.}@%L] - %m{nolookups}%nutf8${LOGDIR:-.}/connector.log${LOGDIR:-.}/connector-%d{yyyy-MM-dd}.%i.log.gz50MB10GB%d - %-5p [%t:%C{3.}@%L] - %m{nolookups}%nutf8
```
## Run the remote connector
```java theme={"dark"}
java [-Dlogging.config=[LOGBACK_XML_FILE]] \
-jar connector-plugin-client-standalone.jar [YAML_CONFIG_FILE]
```
The `logging.config` property is optional. If not set, logging messages are sent to the console.
## Test communication
You can run the connector in communication testing mode. This mode tests the communication with the backend without running the plugin, reports the result, and exits.
```java theme={"dark"}
java -Dstandalone.connector.connectivity.test=true -jar connector-plugin-client-standalone.jar [YAML_CONFIG_FILE]
```
## Encryption
In a deployment, communication to the connector’s backend server is encrypted using TLS. You should only run this configuration without TLS in a testing scenario. To disable TLS, set `plain-text` to `true`.
## Egress and proxy server configuration
One of the methods you can use to allow outbound communication from behind a firewall is a proxy server. You can configure a proxy server to allow certain communication traffic while blocking unauthorized communication. If you use a proxy server at the site where the connector is running, you must configure the following properties:
* **Host.** The hosts where the proxy server is running.
* **Port.** The port the proxy server is listening to for communication requests.
* **Credentials.** Optional proxy server user and password.
When you configure egress, it is important to disable any connection or activity timeouts because the connector uses long running gRPC calls.
## Password encryption
If you use a login name and password in your configuration, run the following utility to encrypt the password:
1. Enter a user name and password in the connector configuration YAML.
2. Run the standalone JAR with this property:
```java theme={"dark"}
-Dstandalone.connector.encrypt.password=true
```
3. Retrieve the encrypted passwords from the log that is created.
4. Replace the clear password in the configuration YAML with the encrypted password.
## Connector restart (5.7 and earlier)
The connector will shut down automatically whenever the connection to the server is disrupted, to prevent it from getting into a bad state. Communication disruption can happen, for example, when the server running in the `connectors-backend` pod shuts down and is replaced by a new pod. Once the connector shuts down, connector configuration and job execution are disabled. To prevent that from happening, you should restart the connector as soon as possible.
You can use Linux scripts and utilities to restart the connector automatically, such as [Monit](https://mmonit.com/monit/).
## Recoverable bridge (5.8 and later)
If communication to the remote connector is disrupted, the connector will try to recover communication and gRPC calls. By default, six attempts will be made to recover each gRPC call. The number of attempts can be configured with the `max-grpc-retries` bridge parameters.
## Job expiration duration (5.9.5 only)
The timeout value for irresponsive backend jobs can be configured with the `job-expiration-duration-seconds` parameter. The default value is `120` seconds.
## Use the remote connector
Once the connector is running, it is available in the Datasources dropdown. If the standalone connector terminates, it disappears from the list of available connectors. Once it is re-run, it is available again and configured connector instances will not get lost.
## Enable asynchronous parsing (5.9 and later)
To separate document crawling from document parsing, enable Tika Asynchronous Parsing on remote V2 connectors.
## Removals
### Bitnami removal
Fusion 5.9.6 will be re-released with the same functionality but updated image references.
In the meantime, Lucidworks will self-host the required images while we work to replace Bitnami images with internally built open-source alternatives.
If you are a self-hosted Fusion customer, *you must upgrade before August 28* to ensure continued access to container images and prevent deployment issues.
You can reinstall your current version of Fusion or upgrade to Fusion 5.9.14, which includes the updated Helm chart and prepares your environment for long-term compatibility.
See [Prevent image pull failures due to Bitnami deprecation in Fusion 5.9.5 to 5.9.13](https://support.lucidworks.com/hc/en-us/articles/33966125467799-Prevent-image-pull-failures-due-to-Bitnami-deprecation-in-Fusion-5-9-5-to-5-9-13) for more information on how to prevent image pull failures.
## Bug fixes
* Fixed an issue that caused the API Gateway service to return a `401` error code for `400` errors.
* Fixed an issue that passed redacted configuration values to Fusion, resulting in invalid configurations.
* Fixed an issue that sometimes caused intermittent errors in Machine Learning stages.
* Fixed an issue where the `sourceType` query parameter was not being conveyed to Lucidworks AI.
* Fixed an issue that prevented the correct display of some UI panels for non-admin users.
* Fixed an issue where the names of some of the configuration checkboxes for the Web connector were truncated.
* Fixed an issue with the Web 1.4.0 connector that, in rare cases, prevented the client secret from being saved.
* Fixed an issue that caused a `Class cannot be created (missing no-arg constructor)` error when a response document contained an undefined value.
# Fusion 5.9.7
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9/5.9.7-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.9.7-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9.7-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/cpx717
Released on December 10, 2024,
this [maintenance release](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#maintenance-release-support-policy) includes new generative AI features and some bug fixes.
To learn more, skip to the [release notes](#new-features).
**Security patch available for api-gateway: Netty request smuggling vulnerabilities**
A patch is available for the `api-gateway` service to address critical Netty request smuggling vulnerabilities (CVE-2026-42581, CVE-2026-42585, CVE-2026-42587). These vulnerabilities allow attackers to smuggle HTTP requests through the gateway, potentially bypassing security controls.
The `api-gateway` service requires the Netty security patch.
Follow these steps to apply the patched image:
1. Open your Fusion Helm values file.
2. Add or update the `api-gateway` image configuration:
```yaml theme={"dark"}
api-gateway:
image:
repository: lucidworks
name: api-gateway
tag: 5.9.9-SUST-1634-patch
imagePullPolicy: IfNotPresent
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace, `RELEASE_NAME` with your Helm release name, and `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the `api-gateway` pods to restart and verify they are using the patched image.
**Urgent action required by November 26, 2025**
A patch is required by November 26, 2025 for all self-hosted Fusion deployments running on Amazon Elastic Kubernetes Service (EKS). Certain Java versions used by Fusion components reach end of life on this date. Failure to apply the patch will result in compatibility issues.
The following Fusion services require the `cgroupv2` patch:
| Service | Affected Fusion versions | Patch tag |
| - | - | - |
| `insights` | 5.9.4 to 5.9.15 | `lucidworks/insights:5.9-cgroupv2-patch` |
| `spark-solr-etl` | 5.9.4 to 5.9.11 | `lucidworks/spark-solr-etl:5.9-cgroupv2-patch` |
| `keytool-utils` | 5.9.4 to 5.9.10 | `lucidworks/keytool-utils:5.9-cgroupv2-patch` |
Follow these steps to apply the patched images:
1. Open your Fusion Helm values file. For Fusion Cloud Native deployments, use the values file for your current deployment. For Helm deployments, use the values file you used to create the deployment.
2. For each service listed in the following table that applies to your Fusion version, add or update the image configuration:
Expand the following code snippet for the complete image configuration list.
```yaml expandable theme={"dark"}
global:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
sql-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
reverse-search:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
solr:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
kafka:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
ml-model-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-admin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-indexing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
query-pipeline:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
async-parsing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
admin-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
api-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
auth-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
classic-rest-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-resources:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-config:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
insights:
image:
imagePullPolicy: Always
name: insights
repository: lucidworks
tag: 5.9-cgroupv2-patch
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-launcher:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-rest-server:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connector-plugin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors-backend:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
rules-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
pm-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
lwai-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
webapps:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
apps-manager:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
templating:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
tikaserver:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
seldon-core-operator:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo-common-workflows:
image:
imagePullPolicy: Always
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
utilitiesTag: 5.9.11
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: 5.9-cgroupv2-patch
argo-common-workflows:
image:
imagePullPolicy: IfNotPresent
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: insights:5.9-cgroupv2-patch
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run the following command:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace. Replace `RELEASE_NAME` with your Helm release name. Replace `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the affected service pods to restart and verify they are using the patched images.
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
## Platform Support and Component Versions
### Kubernetes platform support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.28, 1.29, 1.30
* **Microsoft Azure Kubernetes Service (AKS):** 1.28, 1.29, 1.30
* **Amazon Elastic Kubernetes Service (EKS):** 1.28, 1.29, 1.30
Support is also offered for Rancher Kubernetes Engine (RKE) and OpenShift 4 versions that are based on Kubernetes 1.28, 1.29, 1.30. OpenStack and customized Kubernetes installations are *not* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support).
### Component versions
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| **Solr** | fusion-solr 5.9.7 *(based on Solr 9.6.1)* |
| **ZooKeeper** | 3.9.1 |
| **Spark** | 3.2.2 |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
**More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).**
## New Features
New Lucidworks AI pipeline stages are introduced in this release to enrich your index and search with [Generative AI](/docs/lw-platform/lw-ai/lw-ai-generative-ai) predictions:
* The [LWAI Prediction index stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/lucidworks-ai-prediction-index-stage) allows for asynchronous and synchronous enrichment that add predictions when indexing your data.\
See **Configure the LWAI Prediction index Stage** for more detailed instructions about configuring this stage.
* The [LWAI Prediction query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/lucidworks-ai-prediction-query-stage) fetches synchronous predictions to add to your query response.
See [Generative AI](/docs/lw-platform/lw-ai/lw-ai-generative-ai) for more details about Lucidworks AI’s Generative AI capabilities, including pre-trained models hosted by Lucidworks.
You can also use [LWAI custom trained embedding](/docs/lw-platform/lw-ai/lw-ai-custom-embedding-model-training/overview) and [LWAI hosted pre-trained models](/docs/lw-platform/lw-ai/lw-ai-pre-trained-embedding-models) with these new stages.
A new API endpoint, `/index-pipelines/{id}/async-enrichment/skip-pending`, can be used to clear the queue of outstanding asynchronous prediction index requests if needed.
## Removals
### Bitnami removal
Fusion 5.9.7 will be re-released with the same functionality but updated image references.
In the meantime, Lucidworks will self-host the required images while we work to replace Bitnami images with internally built open-source alternatives.
If you are a self-hosted Fusion customer, *you must upgrade before August 28* to ensure continued access to container images and prevent deployment issues.
You can reinstall your current version of Fusion or upgrade to Fusion 5.9.14, which includes the updated Helm chart and prepares your environment for long-term compatibility.
See [Prevent image pull failures due to Bitnami deprecation in Fusion 5.9.5 to 5.9.13](https://support.lucidworks.com/hc/en-us/articles/33966125467799-Prevent-image-pull-failures-due-to-Bitnami-deprecation-in-Fusion-5-9-5-to-5-9-13) for more information on how to prevent image pull failures.
## Bug fixes
* Fixed an issue where some collections were not displayed in the Collections Manager if the system contained one or more orphaned child collections.\
Note that orphaned child collections are not displayed in the Collections Manager by design, but they are discoverable using the API or the Object Manager.
# Fusion 5.9.8
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9/5.9.8-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.9.8-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9.8-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/en43zu
Released on January 7, 2025, Fusion 5.9.8 brings powerful enhancements to streamline your workflows and prepare your systems for future innovations. This release enables seamless synchronization of Fusion configurations across environments, provides compatibility with upcoming solutions, and resolves key issues to ensure a smoother experience.
Upgrading to the latest version of Fusion 5.9 offers several key benefits:
* **Access to latest features:** Stay current with the latest features and functionality to ensure compatibility and optimal performance.
* **Simplified process:** Fusion 5.9.5 and later use an in-place upgrade strategy, making upgrades easier than ever.
* **Extended support:** Upgrading keeps you up-to-date with the latest supported Kubernetes versions, as outlined in the [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle) policy.
**Security patch available for api-gateway: Netty request smuggling vulnerabilities**
A patch is available for the `api-gateway` service to address critical Netty request smuggling vulnerabilities (CVE-2026-42581, CVE-2026-42585, CVE-2026-42587). These vulnerabilities allow attackers to smuggle HTTP requests through the gateway, potentially bypassing security controls.
The `api-gateway` service requires the Netty security patch.
Follow these steps to apply the patched image:
1. Open your Fusion Helm values file.
2. Add or update the `api-gateway` image configuration:
```yaml theme={"dark"}
api-gateway:
image:
repository: lucidworks
name: api-gateway
tag: 5.9.9-SUST-1634-patch
imagePullPolicy: IfNotPresent
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace, `RELEASE_NAME` with your Helm release name, and `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the `api-gateway` pods to restart and verify they are using the patched image.
**Urgent action required by November 26, 2025**
A patch is required by November 26, 2025 for all self-hosted Fusion deployments running on Amazon Elastic Kubernetes Service (EKS). Certain Java versions used by Fusion components reach end of life on this date. Failure to apply the patch will result in compatibility issues.
The following Fusion services require the `cgroupv2` patch:
| Service | Affected Fusion versions | Patch tag |
| - | - | - |
| `insights` | 5.9.4 to 5.9.15 | `lucidworks/insights:5.9-cgroupv2-patch` |
| `spark-solr-etl` | 5.9.4 to 5.9.11 | `lucidworks/spark-solr-etl:5.9-cgroupv2-patch` |
| `keytool-utils` | 5.9.4 to 5.9.10 | `lucidworks/keytool-utils:5.9-cgroupv2-patch` |
Follow these steps to apply the patched images:
1. Open your Fusion Helm values file. For Fusion Cloud Native deployments, use the values file for your current deployment. For Helm deployments, use the values file you used to create the deployment.
2. For each service listed in the following table that applies to your Fusion version, add or update the image configuration:
Expand the following code snippet for the complete image configuration list.
```yaml expandable theme={"dark"}
global:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
sql-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
reverse-search:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
solr:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
kafka:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
ml-model-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-admin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-indexing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
query-pipeline:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
async-parsing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
admin-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
api-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
auth-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
classic-rest-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-resources:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-config:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
insights:
image:
imagePullPolicy: Always
name: insights
repository: lucidworks
tag: 5.9-cgroupv2-patch
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-launcher:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-rest-server:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connector-plugin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors-backend:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
rules-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
pm-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
lwai-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
webapps:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
apps-manager:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
templating:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
tikaserver:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
seldon-core-operator:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo-common-workflows:
image:
imagePullPolicy: Always
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
utilitiesTag: 5.9.11
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: 5.9-cgroupv2-patch
argo-common-workflows:
image:
imagePullPolicy: IfNotPresent
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: insights:5.9-cgroupv2-patch
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run the following command:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace. Replace `RELEASE_NAME` with your Helm release name. Replace `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the affected service pods to restart and verify they are using the patched images.
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
For supported Kubernetes versions and key component versions, see [Platform support and component versions](#platform-support-and-component-versions).
## Key highlights
### Commerce Studio support
Fusion 5.9.8 is fully compatible with Commerce Studio, Lucidworks' solution for enhancing e-commerce experiences.
Commerce Studio will be generally available at the end of January, 2025. Accompanying that release will be documentation and training materials.
The course for **What is Commerce Studio?** focuses on how to tailor your user’s search and browse experiences through powerful rule building, merchandising, audience segmentation, and more.
### ConfigSync
Automatically keep your Fusion configuration settings consistent and up-to-date across different environments with the new [ConfigSync](/docs/5/fusion/operations/config-sync/overview) feature. By leveraging GitHub’s version control capabilities, ConfigSync eliminates manual synchronization, provides a historical record of your environments, and enables seamless rollbacks and forward changes. Highly customizable, it simplifies environment management while saving time and ensuring consistency.
ConfigSync supports:
* Polling or event-driven synchronization
* Multiple synchronization modes:
* **Publisher mode:** Monitors Fusion for changes and pushes them to a Git repository.
* **Subscriber mode:** Polls a Git repository for changes and applies them to Fusion.
* **Pub/Sub mode:** Combines the features of both Publisher and Subscriber modes.
* Locking the configuration to a specific commit
* Rolling back configurations to a previous state
* Storing and synchronizing Fusion secrets securely in your GitHub repository
* Metrics for monitoring, such as `resource_update_errors_total`, `out_of_sync`, `sync_seconds`, and `sync_seconds_max`
ConfigSync is turned off by default and requires preparation of a GitHub repository. Learn how to turn on ConfigSync in your Fusion 5.9.8 cluster with the [ConfigSync deployment guide](/docs/5/fusion/operations/config-sync/deployment).
The course for **ConfigSync** focuses on how to set up ConfigSync, choose a synchronization mode, and apply filters.
## Bug fixes
* Fixed an issue with Predictive Merchandiser that prevented saving new fields in templates.
* Fixed an issue that prevented the data model field dropdown from displaying in the Data Model Mapping index pipeline stage.
## Known issues
* An issue prevents segment-based rule filtering from working correctly in Commerce Studio. This issue is fixed in Fusion 5.9.12.
* An issue causes the Fusion Helm chart with Transport Layer Security (TLS) enabled to throw an error with Kafka during installation. To avoid this, the `helm install` command used to [install Fusion with TLS](/docs/5/fusion/reference/microservices#transport-layer-security-tls) should include the flags `--set global.zkReplicaCount=3` and `--set kafka.auth.clientProtocol=tls`.
## Deprecations
For full details on deprecations, see [Deprecations and Removals](/docs/5/fusion/deprecations-and-removals).
As part of our ongoing commitment to providing a modern, efficient, and secure product, Lucidworks is officially deprecating support for the Nashorn JavaScript engine in Fusion 5.9.8. Users are advised to use the OpenJDK Nashorn engine, a robust and actively maintained alternative, for all JavaScript pipeline stages.
The deprecation aligns with industry trends and ensures compatibility with modern tools and frameworks. OpenJDK Nashorn provides enhanced performance, improved support, and ongoing updates, making it the ideal replacement.
**FAQ**
How long will Lucidworks continue to support Nashorn?
Nashorn will remain available for at least 6 months from the release of Fusion 5.9.8. All Nashorn-related features and integrations will be removed in a release after this 6-month period.
To prepare for the eventual removal of Nashorn in a future release, Lucidworks strongly recommends beginning the migration of your pipelines to OpenJDK Nashorn:
* **Audit your code.** Identify all areas where Nashorn is used in your applications.
* **Plan your migration.** Transition your JavaScript pipelines from Nashorn to OpenJDK Nashorn. Be sure to thoroughly test compatibility with your configurations to ensure a smooth transition.
* **Seek support.** Lucidworks is available to assist with any migration questions or challenges. Please contact your Client Excellence team if you have any questions.
We appreciate your understanding and cooperation in this transition. Thank you for ensuring the continued evolution and efficiency of your workflows.
You can select the OpenJDK Nashorn engine in the pipeline views or in the workbenches:
## Removals
### Bitnami removal
Fusion 5.9.8 will be re-released with the same functionality but updated image references.
In the meantime, Lucidworks will self-host the required images while we work to replace Bitnami images with internally built open-source alternatives.
If you are a self-hosted Fusion customer, *you must upgrade before August 28* to ensure continued access to container images and prevent deployment issues.
You can reinstall your current version of Fusion or upgrade to Fusion 5.9.14, which includes the updated Helm chart and prepares your environment for long-term compatibility.
See [Prevent image pull failures due to Bitnami deprecation in Fusion 5.9.5 to 5.9.13](https://support.lucidworks.com/hc/en-us/articles/33966125467799-Prevent-image-pull-failures-due-to-Bitnami-deprecation-in-Fusion-5-9-5-to-5-9-13) for more information on how to prevent image pull failures.
## Platform support and component versions
### Kubernetes platform support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.28, 1.29, 1.30
* **Microsoft Azure Kubernetes Service (AKS):** 1.28, 1.29, 1.30
* **Amazon Elastic Kubernetes Service (EKS):** 1.28, 1.29, 1.30
Support is also offered for Rancher Kubernetes Engine (RKE) and OpenShift 4 versions that are based on Kubernetes 1.28, 1.29, 1.30. OpenStack and customized Kubernetes installations are *not* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support).
### Component versions
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| **Solr** | fusion-solr 5.9.8 *(based on Solr 9.6.1)* |
| **ZooKeeper** | 3.9.1 |
| **Spark** | 3.2.2 |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
**More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).**
# Fusion 5.9.9
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9/5.9.9-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.9.9-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.9.9-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/hga40k
Released on January 29, 2025, this [maintenance release](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#maintenance-release-support-policy) delivers essential security enhancements, including the latest updates for Kafka and key improvements for future-ready protection, plus bug fixes.
Upgrading to the latest version of Fusion 5.9 offers several key benefits:
* **Access to latest features:** Stay current with the latest features and functionality to ensure compatibility and optimal performance.
* **Simplified process:** Fusion 5.9.5 and later use an in-place upgrade strategy, making upgrades easier than ever.
* **Extended support:** Upgrading keeps you up-to-date with the latest supported Kubernetes versions, as outlined in the [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle) policy.
For supported Kubernetes versions and key component versions, see [Platform support and component versions](#platform-support-and-component-versions).
**Security patch available for api-gateway: Netty request smuggling vulnerabilities**
A patch is available for the `api-gateway` service to address critical Netty request smuggling vulnerabilities (CVE-2026-42581, CVE-2026-42585, CVE-2026-42587). These vulnerabilities allow attackers to smuggle HTTP requests through the gateway, potentially bypassing security controls.
The `api-gateway` service requires the Netty security patch.
Follow these steps to apply the patched image:
1. Open your Fusion Helm values file.
2. Add or update the `api-gateway` image configuration:
```yaml theme={"dark"}
api-gateway:
image:
repository: lucidworks
name: api-gateway
tag: 5.9.9-SUST-1634-patch
imagePullPolicy: IfNotPresent
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace, `RELEASE_NAME` with your Helm release name, and `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the `api-gateway` pods to restart and verify they are using the patched image.
**Urgent action required by November 26, 2025**
A patch is required by November 26, 2025 for all self-hosted Fusion deployments running on Amazon Elastic Kubernetes Service (EKS). Certain Java versions used by Fusion components reach end of life on this date. Failure to apply the patch will result in compatibility issues.
The following Fusion services require the `cgroupv2` patch:
| Service | Affected Fusion versions | Patch tag |
| - | - | - |
| `insights` | 5.9.4 to 5.9.15 | `lucidworks/insights:5.9-cgroupv2-patch` |
| `spark-solr-etl` | 5.9.4 to 5.9.11 | `lucidworks/spark-solr-etl:5.9-cgroupv2-patch` |
| `keytool-utils` | 5.9.4 to 5.9.10 | `lucidworks/keytool-utils:5.9-cgroupv2-patch` |
Follow these steps to apply the patched images:
1. Open your Fusion Helm values file. For Fusion Cloud Native deployments, use the values file for your current deployment. For Helm deployments, use the values file you used to create the deployment.
2. For each service listed in the following table that applies to your Fusion version, add or update the image configuration:
Expand the following code snippet for the complete image configuration list.
```yaml expandable theme={"dark"}
global:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
sql-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
reverse-search:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
solr:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
kafka:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
zookeeper:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
ml-model-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-admin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-indexing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
query-pipeline:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
async-parsing:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
admin-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
api-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
auth-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
classic-rest-service:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
fusion-resources:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-config:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
insights:
image:
imagePullPolicy: Always
name: insights
repository: lucidworks
tag: 5.9-cgroupv2-patch
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-launcher:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
job-rest-server:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connector-plugin:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
connectors-backend:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
rules-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
pm-ui:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
lwai-gateway:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
webapps:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
apps-manager:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
templating:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
tikaserver:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
seldon-core-operator:
keytoolUtils:
image:
repository: lucidworks
name: "keytool-utils"
tag: "5.9-cgroupv2-patch"
imagePullPolicy: "IfNotPresent"
argo-common-workflows:
image:
imagePullPolicy: Always
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
utilitiesTag: 5.9.11
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: 5.9-cgroupv2-patch
argo-common-workflows:
image:
imagePullPolicy: IfNotPresent
repository: lucidworks
sparkSolrEtlTag: 5.9-cgroupv2-patch
```
```yaml theme={"dark"}
insights:
image:
name: insights
pullPolicy: IfNotPresent
repository: lucidworks
tag: insights:5.9-cgroupv2-patch
```
3. Save the values file.
4. For Fusion Cloud Native deployments, run the `upgrade_fusion.sh` script you used for your current deployment. For Helm deployments, run the following command:
```bash theme={"dark"}
helm upgrade --namespace NAMESPACE RELEASE_NAME PATH_TO_VALUES
```
Replace `NAMESPACE` with your Kubernetes namespace. Replace `RELEASE_NAME` with your Helm release name. Replace `PATH_TO_VALUES` with the path to your updated values file.
5. Wait for the affected service pods to restart and verify they are using the patched images.
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
For supported Kubernetes versions and key component versions, see [Platform support and component versions](#platform-support-and-component-versions).
## Key highlights
### Enhanced security and future-ready safeguards
With the Fusion 5.9.9 release, we’ve upgraded Kafka to version 3.7.0, incorporating the latest security updates to strengthen your platform’s resilience. Additionally, we’ve implemented critical updates for Fusion’s Webapps, Templating, and API Gateway services, ensuring robust protection for your data.
Looking ahead, we’ve introduced enhancements to streamline and expedite the delivery of future security updates, providing you with a safer Fusion environment.
### Secure deployments with immutable root file system
Fusion now supports a read-only root file system to safeguard against unauthorized modifications, protecting your deployment against malicious software and other attacks.
Most services operate efficiently in this mode, and for components that need write access, a separate writable mount is available using specific helm configurations.
Read-only mode is enabled by default for some Fusion services.
See **Enable Read-Only Root File System** for a list of services that support it or have it enabled by default, along with examples of how to configure it in your Fusion deployment.
It should only be enabled for services that do not require CRUD access.
This topic describes how to configure a read-only root file system for a Fusion deployment.
Read-only mode safeguards your file system against unauthorized changes, such as by malicious software or other attacks.
Enabling this feature entails configuring the `readOnlyRootFilesystem` attribute in the Fusion Helm chart.
This feature is available starting in Fusion 5.9.9 and in all subsequent Fusion 5.9 releases.
Beginning with Fusion 5.9.10, all of [Fusion’s services](/docs/5/fusion/reference/microservices) are designed to work with a read-only root file system, but some external services may require write access to the file system.
In that case, you can configure a read-only root file system and mount a separate writable file system for these services.
The examples below show how to configure a read-only root file system for various Fusion services.
## Solr
```yaml theme={"dark"}
fusion:
solr:
containerSecurityContext:
readOnlyRootFilesystem: true
```
**With optional services:**
```yaml theme={"dark"}
fusion:
solr:
exporter:
enabled: true
securityContext:
readOnlyRootFilesystem: true
containerSecurityContext:
readOnlyRootFilesystem: true
```
## Zookeeper
```yaml theme={"dark"}
fusion:
zookeeper:
containerSecurityContext:
readOnlyRootFilesystem: true
```
**With optional services:**
```yaml theme={"dark"}
fusion:
zookeeper:
containerSecurityContext:
readOnlyRootFilesystem: true
exporters:
jmx:
enabled: true
zookeeper:
enabled: true
jobs:
chroots:
enabled: true
config:
create:
- /root-read-only
```
## Kafka
```yaml theme={"dark"}
fusion:
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
```
**With optional services:**
```yaml expandable theme={"dark"}
fusion:
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
volumePermissions:
enabled: true
containerSecurityContext:
readOnlyRootFilesystem: true
metrics:
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
enabled: true
jmx:
enabled: true
containerSecurityContext:
enabled: true
readOnlyRootFilesystem: true
externalAccess:
enabled: true
autoDiscovery:
enabled: true
containerSecurityContext:
enabled: true
readOnlyRootFilesystem: true
provisioning:
enabled: true
topics:
- name: test
partitions: 1
replicationFactor: 1
## https://kafka.apache.org/documentation/#topicconfigs
config:
max.message.bytes: 64000
flush.messages: 1
containerSecurityContext:
readOnlyRootFilesystem: true
enabled: true
```
## ML Model Service
```yaml expandable theme={"dark"}
fusion:
ml-model-service:
ambassador:
containerSecurityContext:
readOnlyRootFilesystem: true
milvus:
mysql:
containerSecurityContext:
readOnlyRootFilesystem: true
securityContext:
readOnlyRootFilesystem: true
With optional services:
fusion:
ml-model-service:
ambassador:
containerSecurityContext:
readOnlyRootFilesystem: true
prometheusExporter:
enabled: true
securityContext:
readOnlyRootFilesystem: true
milvus:
mysql:
metrics:
enabled: true
securityContext:
readOnlyRootFilesystem: true
containerSecurityContext:
readOnlyRootFilesystem: true
cluster:
enabled: false
mishards:
securityContext:
readOnlyRootFilesystem: true
admin:
enabled: true
securityContext:
readOnlyRootFilesystem: true
securityContext:
readOnlyRootFilesystem: true
```
## Argo
```yaml theme={"dark"}
fusion:
argo:
minio:
containerSecurityContext:
readOnlyRootFilesystem: true
extraVolumes:
- name: config-dir
emptyDir: {}
- name: certs-dir
emptyDir: {}
extraVolumeMounts:
- name: config-dir
mountPath: /.minio
- name: certs-dir
mountPath: /etc/minio/certs
server:
securityContext:
readOnlyRootFilesystem: true
controller:
securityContext:
readOnlyRootFilesystem: true
mainContainer:
securityContext:
readOnlyRootFilesystem: true
executor:
securityContext:
readOnlyRootFilesystem: true
```
## Seldon Core Operator
```yaml theme={"dark"}
fusion:
seldon-core-operator:
containersSecurityContext:
readOnlyRootFilesystem: true
```
## Argo Common Workflows
```yaml theme={"dark"}
fusion:
argo-common-workflows:
containerSecurityContext:
readOnlyRootFilesystem: true
```
## Question Answering
```yaml theme={"dark"}
fusion:
question-answering:
containersSecurityContext:
readOnlyRootFilesystem: true
```
## Classification
```yaml theme={"dark"}
fusion:
classification:
containersSecurityContext:
readOnlyRootFilesystem: true
```
## All-in-one values.yaml example
```yaml expandable theme={"dark"}
fusion:
solr:
containerSecurityContext:
readOnlyRootFilesystem: true
solr-managed:
containerSecurityContext:
readOnlyRootFilesystem: true
zookeeper:
containerSecurityContext:
readOnlyRootFilesystem: true
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
ml-model-service:
ambassador:
containerSecurityContext:
readOnlyRootFilesystem: true
milvus:
mysql:
containerSecurityContext:
readOnlyRootFilesystem: true
securityContext:
readOnlyRootFilesystem: true
argo:
minio:
containerSecurityContext:
readOnlyRootFilesystem: true
extraVolumes:
- name: config-dir
emptyDir: {}
- name: certs-dir
emptyDir: {}
extraVolumeMounts:
- name: config-dir
mountPath: /.minio
- name: certs-dir
mountPath: /etc/minio/certs
server:
securityContext:
readOnlyRootFilesystem: true
controller:
securityContext:
readOnlyRootFilesystem: true
mainContainer:
securityContext:
readOnlyRootFilesystem: true
executor:
securityContext:
readOnlyRootFilesystem: true
seldon-core-operator:
containersSecurityContext:
readOnlyRootFilesystem: true
argo-common-workflows:
containerSecurityContext:
readOnlyRootFilesystem: true
question-answering:
containersSecurityContext:
readOnlyRootFilesystem: true
classification:
containersSecurityContext:
readOnlyRootFilesystem: true
```
## All-in-one values.yaml with optional services example
```yaml expandable theme={"dark"}
fusion:
argo:
minio:
persistence:
enabled: false
containerSecurityContext:
readOnlyRootFilesystem: true
extraVolumes:
- name: config-dir
emptyDir: {}
- name: certs-dir
emptyDir: {}
extraVolumeMounts:
- name: config-dir
mountPath: /.minio
- name: certs-dir
mountPath: /etc/minio/certs
server:
securityContext:
readOnlyRootFilesystem: true
controller:
securityContext:
readOnlyRootFilesystem: true
mainContainer:
securityContext:
readOnlyRootFilesystem: true
executor:
securityContext:
readOnlyRootFilesystem: true
solr:
exporter:
enabled: true
securityContext:
readOnlyRootFilesystem: true
containerSecurityContext:
readOnlyRootFilesystem: true
# tls:
# enabled: true
solr-managed:
exporter:
enabled: true
securityContext:
readOnlyRootFilesystem: true
enableExternalFiles: true
enabledStorage:
- gcs
processRaw:
image:
repository: fusion-dev-docker.ci-artifactory.lucidworks.com
cloudRoot: gs://lw-managed-fusion-data/tmp
gcs:
secret: gcs-key
secretFieldName: key.json
securityContext:
readOnlyRootFilesystem: true
containerSecurityContext:
readOnlyRootFilesystem: true
# tls:
# enabled: true
zookeeper:
containerSecurityContext:
readOnlyRootFilesystem: true
exporters:
jmx:
enabled: true
zookeeper:
enabled: true
jobs:
chroots:
# enabled: true
config:
create:
- /root-read-only
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
volumePermissions:
enabled: true
image:
repository: "fusion-dev-docker.ci-artifactory.lucidworks.com/os-shell"
containerSecurityContext:
readOnlyRootFilesystem: true
metrics:
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
enabled: true
jmx:
enabled: true
containerSecurityContext:
enabled: true
readOnlyRootFilesystem: true
externalAccess:
enabled: true
autoDiscovery:
enabled: true
containerSecurityContext:
enabled: true
readOnlyRootFilesystem: true
provisioning:
enabled: true
topics:
- name: test
partitions: 1
replicationFactor: 1
## https://kafka.apache.org/documentation/#topicconfigs
config:
max.message.bytes: 64000
flush.messages: 1
containerSecurityContext:
readOnlyRootFilesystem: true
enabled: true
ml-model-service:
enabled: true
ambassador:
containerSecurityContext:
readOnlyRootFilesystem: true
prometheusExporter:
enabled: true
securityContext:
readOnlyRootFilesystem: true
milvus:
mysql:
metrics:
enabled: true
securityContext:
readOnlyRootFilesystem: true
containerSecurityContext:
readOnlyRootFilesystem: true
cluster:
enabled: false
mishards:
securityContext:
readOnlyRootFilesystem: true
admin:
enabled: true
securityContext:
readOnlyRootFilesystem: true
securityContext:
readOnlyRootFilesystem: true
seldon-core-operator:
containersSecurityContext:
readOnlyRootFilesystem: true
argo-common-workflows:
containerSecurityContext:
readOnlyRootFilesystem: true
question-answering:
containersSecurityContext:
readOnlyRootFilesystem: true
classification:
containersSecurityContext:
readOnlyRootFilesystem: true
```
## Supported services
The table below lists the services that support a read-only root file system, the ones that have it enabled by default, and the Fusion release in which support was added:
| Chart Name | Pod Name | Container Name | Supported | Default enabled | Supported version |
| - | - | - | - | - | - |
| admin-ui | admin-ui | admin-ui | ✅ | ✅ | 5.9.9+ |
| api-gateway | api-gateway | init/api-gateway | ✅ | ✅ | 5.9.9+ |
| api-gateway | api-gateway | api-gateway | ✅ | ✅ | 5.9.9+ |
| api-gateway | api-gateway | generate-jks | ✅ | ✅ | 5.9.9+ |
| apps-manager | apps-manager | apps-manager | ✅ | ✅ | 5.9.9+ |
| argo | argo-server | argo-server | ✅ | ❌ | 5.9.10+ |
| argo | argo-executor | executor | ✅ | ❌ | 5.9.10+ |
| argo | argo-mainContainer | mainContainer | ✅ | ❌ | 5.9.10+ |
| argo | argo-controller | controller | ✅ | ✅ | 5.9.10+ |
| argo/minio | minio | minio | ✅ | ❌ | 5.9.10+ |
| argo/minio | minio | minio | ✅ | ❌ | 5.9.10+ |
| argo/minio | make-bucket-job | minio-mc | ✅ | ❌ | 5.9.10+ |
| argo-common-workflows | delete-model | init/main/wait | ✅ | ❌ | 5.9.10+ |
| argo-common-workflows | deploy-model | init/main/wait | ✅ | ❌ | 5.9.10+ |
| argo-common-workflows | milvus-maintenance | init/main/wait | ✅ | ❌ | 5.9.10+ |
| argo-common-workflows | upload-model-to-cloud | init/main/wait | ✅ | ❌ | 5.9.10+ |
| async-parsing | async-parsing | tika-server | ✅ | ✅ | 5.9.9+ |
| async-parsing | async-parsing | async-parsing | ✅ | ✅ | 5.9.9+ |
| auth-ui | auth-ui | auth-ui | ✅ | ✅ | 5.9.9+ |
| classic-rest-service | classic-rest-service | init/import-certs | ✅ | ✅ | 5.9.10+ |
| classic-rest-service | classic-rest-service | classic-rest-service | ✅ | ✅ | 5.9.9+ |
| classification | argo/classification | init/wait/main | ✅ | ❌ | 5.9.10+ |
| connector-plugin | connector-plugin | init/import-certs | ✅ | ❌ | 5.9.10+ |
| connector-plugin | connector-plugin | connector-plugin | ✅ | ✅ | 5.9.9+ |
| connectors | connectors | connectors | ✅ | ✅ | 5.9.9+ |
| connectors-backend | CRD | | ✅ | ✅ | 5.9.9+ |
| connectors-backend | connectors-backend | connectors-backend | ✅ | ✅ | 5.9.9+ |
| fusion-admin | fusion-admin | admin | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-admin | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-api-gateway | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-indexing | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-kafka | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-logstash | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-pulsar | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | setup-keystore-and-properties | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-zk | ✅ | ✅ | 5.9.9+ |
| fusion-config-sync | fusion-config-sync | fusion-config-sync | ✅ | ✅ | 5.9.9+ |
| fusion-data-augmentation | argo/data-augmentation/volume-fix | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/init-workspace | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/write-job-configs | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/write-io-configs | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/add-zkhost | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/pull-data-training-and-metadata | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/pull-data-training-and-metadata-cloud | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/volume-fix2 | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/synonym-list | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/download-synonym-dictionary | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/keystroke-list | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/download-keystroke-blob | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/augment | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/push-augmented-data | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/push-augmented-data-cloud | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-indexing | fusion-indexing | fusion-indexing | ✅ | ✅ | 5.9.9+ |
| fusion-resources | fusion-resources-secret-hook | setup-keystore-and-properties | ✅ | ✅ | 5.9.10+ |
| insights | insights | insights | ✅ | ✅ | 5.9.9+ |
| job-launcher | job-launcher | job-launcher | ✅ | ✅ | 5.9.9+ |
| job-launcher | job-launcher | kubectl-runner | ✅ | ✅ | 5.9.9+ |
| job-launcher | job-launcher-spark-cleanup | kubectl-runner | ✅ | ✅ | 5.9.9+ |
| job-launcher | spark-kubernetes-driver | spark-kubernetes-driver | ✅ | ❌ | 5.9.10+ |
| job-launcher | spark-kubernetes-executor | spark-kubernetes-executor | ✅ | ❌ | 5.9.10+ |
| job-rest-server | job-rest-server | job-rest-server | ✅ | ✅ | 5.9.9+ |
| kafka | kafka-metrics | kafka-exporter-archived | ✅ | ❌ | 5.9.10+ |
| kafka | kafka-provisioning | init/wait-for-available-kafka | ✅ | ❌ | 5.9.10+ |
| kafka | kafka-provisioning | kafka-provisioning | ✅ | ❌ | 5.9.10+ |
| kafka | kafka | kafka | ✅ | ❌ | 5.9.10+ |
| kafka | kafka | jmx-exporter | ✅ | ❌ | 5.9.10+ |
| kafka | kafka | init/check-zk | ✅ | ✅ | 5.9.9+ |
| kafka | kafka | init/auto-discovery | ✅ | ❌ | 5.9.10+ |
| kafka | kafka | init/volume-permissions | ✅ | ❌ | 5.9.10+ |
| lwai-gateway | lwai-gateway | lwai-gateway | ✅ | ✅ | 5.9.9+ |
| ml-model-service | ml-model-service | java-service | ✅ | ✅ | 5.9.9+ |
| ml-model-service | ml-model-service-namespace-hook | kubectl-runner | ✅ | ✅ | 5.9.9+ |
| ml-model-service/ambassador | ambassador | ambassador | ✅ | ❌ | 5.9.10+ |
| ml-model-service/ambassador | ambassador | prometheus-exporter | ✅ | ❌ | 5.9.10+ |
| ml-model-service/milvus | milvus-writable | milvus | ✅ | ❌ | 5.9.10+ |
| ml-model-service/milvus | milvus-writable | init/wait-for-mysql | ✅ | ✅ | 5.9.10+ |
| ml-model-service/milvus | milvus-writable | init/create-for-share-storage | ✅ | ❌ | 5.9.10+ |
| ml-model-service/milvus | milvus-admin | admin | ✅ | ❌ | 5.9.10+ |
| ml-model-service/milvus | milvus-admin | init/wait-for-milvus | ✅ | ✅ | 5.9.10+ |
| ml-model-service/milvus | milvus-mishards | init/wait-for-mysql | ✅ | ✅ | 5.9.10+ |
| ml-model-service/milvus | milvus-mishards | init/wait-for-mysql | ✅ | ✅ | 5.9.10+ |
| ml-model-service/milvus | milvus-mishards | mishards | ✅ | ❌ | 5.9.10+ |
| ml-model-service/milvus/mysql | mysql | init/remove-lost-found | ✅ | ✅ | 5.9.10+ |
| ml-model-service/milvus/mysql | mysql | mysql | ✅ | ❌ | 5.9.10+ |
| pm-ui | pm-ui | pm-ui | ✅ | ✅ | 5.9.9+ |
| query-pipeline | query-pipeline | query-pipeline | ✅ | ✅ | 5.9.9+ |
| question-answering | argo/qna-coldstart/init-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/write-job-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/add-zkHost | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/pull-data | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/pull-data-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/train | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/list-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/post-model | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/apply-seldon-deployment | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/init-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/write-job-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/add-zkhost | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/pull-qa-data-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/pull-qa-data | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/train | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/train-with-texts | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/list-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/post-model | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/apply-seldon-deployment | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/init-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/write-job-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/add-zkhost | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/list-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/pull-eval-data | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/pull-eval-data-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/evaluate | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/push-eval-results-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/push-eval-results | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/init-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/write-job-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/add-zkhost | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/pull-data-training-and-metadata | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/pull-data-training-and-metadata-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/train-with-metadata | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/train-without-metadata | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/push-recommendations | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/push-recommendations-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/add-default-exclude-query | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/init-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/copy-model | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/write-job-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/add-zkhost | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/pull-data | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/pull-data-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/train | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/push-content | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/push-content-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/add-default-exclude-query | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/delete-old-content-recommendations | init/wait/main | ✅ | ❌ | 5.9.10+ |
| reverse-search | reverse-search | init/set-reverse-search-zone | ✅ | ✅ | 5.9.10+ |
| reverse-search | reverse-search | init/check-zk | ✅ | ✅ | 5.9.9+ |
| reverse-search | reverse-search | init/enable-tls-in-reverse-search | ✅ | ✅ | 5.9.10+ |
| reverse-search | reverse-search | reverse-search | ✅ | ❌ | 5.9.10+ |
| rules-ui | rules-ui | rules-ui | ✅ | ✅ | 5.9.9+ |
| seldon-core-operator | seldon-controller-manager | manager | ✅ | ❌ | 5.9.10+ |
| seldon-core-operator | seldon-spartakus-volunteer | seldon-spartakus-volunteer | ✅ | ❌ | 5.9.9+ |
| seldon-core-operator | crd/SeldonDeployment | | ✅/❌ | ❌ | 5.9.9+ |
| solr | solr | init/set-solr-zone | ✅ | ✅ | 5.9.10+ |
| solr | solr | init/enable-tls-in-solr | ✅ | ✅ | 5.9.10+ |
| solr | configset-bootstrap | configset-bootstrap | ✅ | ✅ | 5.9.10+ |
| solr | solr | solr | ✅ | ❌ | 5.9.10+ |
| solr | solr-exporter | exporter | ✅ | ❌ | 5.9.10+ |
| solr | solr-exporter | init/solr-init | ✅ | ✅ | 5.9.10+ |
| solr-managed | configset-bootstrap | configset-bootstrap | ✅ | ✅ | 5.9.10+ |
| solr-managed | exporter | exporter | ✅ | ❌ | 5.9.10+ |
| solr-managed | exporter | init/solr-init | ✅ | ✅ | 5.9.10+ |
| solr-managed | solr | init/set-solr-zone | ✅ | ✅ | 5.9.10+ |
| solr-managed | solr | init/enable-tls-in-solr | ✅ | ✅ | 5.9.10+ |
| solr-managed | solr | solr | ✅ | ❌ | 5.9.10+ |
| solr-managed | solr | process-raw | ✅ | ❌ | 5.9.10+ |
| solr-backup-runner | solr-backup-runner-backup | solr-backups | ✅ | ❌ | 5.9.10+ |
| solr-backup-runner | solr-backup-runner-prune | solr-prune | ✅ | ❌ | 5.9.10+ |
| templating | templating | templating | ✅ | ✅ | 5.9.9+ |
| webapps | webapps | webapps | ✅ | ✅ | 5.9.9+ |
| zookeeper | zookeeper | zookeeper | ✅ | ❌ | 5.9.10+ |
| zookeeper | zookeeper | jmx-exporter | ✅ | ❌ | 5.9.10+ |
| zookeeper | zookeeper | zookeeper-exporter | ✅ | ❌ | 5.9.10+ |
| zookeeper | zookeeper-chroots | main | ✅ | ❌ | 5.9.10+ |
## Bug fixes
* This release addresses an issue in Fusion 5.9.2 and up, where enabling Transport Layer Security (TLS) during installation resulted in the following error:
```
Error: INSTALLATION FAILED: ... at <.tls.certSecret.name>: nil pointer evaluating interface {}.name
```
With this fix, Fusion can now be successfully installed with TLS enabled.
* This release improves compatibility between Solr and Fusion’s [LWAI Prediction index pipeline stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/lucidworks-ai-prediction-index-stage) to prevent errors about incompatible characters in field names.
* The Fusion API realm can now authenticate when an API key is passed via an `apiKey` parameter.\
Previously, the key could only be passed in the header, as in these examples:
```bash wrap theme={"dark"}
curl -H "x-api-key: API_KEY" "https://FUSION_HOST/api/query/status"
curl -H "apiKey: API_KEY" "https://FUSION_HOST/api/query/status"
```
Now you can also pass the key as a parameter, like this:
```bash theme={"dark"}
curl https://FUSION_HOST/api/query/status?apiKey=API_KEY
```
* The [Configurations](/api-reference/configurations/get-configurations-service-status) and [ZooKeeper Import/Export](/api-reference/zookeeper/get-zookeeper-data) APIs are now accessible only to authorized users from within your Fusion cluster or through Kubernetes port forwarding.
* The API Gateway now includes an optional Kubernetes health check. To enable this check, edit your `values.yaml` file, and add `-Dspring.cloud.kubernetes.loadbalancer.mode=POD` to the API Gateway’s `javaToolOptions`. For example:
```yaml theme={"dark"}
fusion:
api-gateway:
javaToolOptions: "-Dspring.cloud.kubernetes.loadbalancer.mode=POD"
```
When enabled, if the health check fails during pod discovery, Kubernetes will restart the pod.
## Known issues
* Saving large query pipelines may cause OOM failures under high load.\
In Fusion versions 5.9.9 through 5.9.13, saving a large query pipeline during high query volume can result in thread lock, excessive memory use, and eventual OOM errors in the Query service.\
This issue is fixed in Fusion 5.9.14.
* An issue prevents segment-based rule filtering from working correctly in Commerce Studio. This issue is fixed in Fusion 5.9.12.
* After upgrading, the `config-sync` pod sometimes fails due to authorization errors. If this pod is not required by your setup, you can disable it by setting the `enabled` flag to `false` in your Helm chart configuration, saving the configuration, and rerunning the upgrade script.
```yaml theme={"dark"}
fusion-config-sync:
enabled: false
```
* Jobs or pipelines that serialize or deserialize `JSONResponse` objects may fail with an error of `Cannot instantiate class java.util.Collections$SingletonMap with Kryo`.\
The issue was fixed in Fusion 5.9.10 and backported into Fusion 5.9.9.
## Deprecations
For full details on deprecations, see [Deprecations and Removals](/docs/5/fusion/deprecations-and-removals).
As part of our ongoing initiative to replace our V1 (Classic) connectors with our more secure, flexible, and scalable V2 (Plugin) connectors, the following V1 connectors are deprecated in this release:
* [Windows Share SMB 2/3 V1](/docs/fusion-connectors/connectors/v1/windows-share-smb-2-3)
* [File Upload V1](/docs/fusion-connectors/connectors/v1/fileupload)
Learn more about [V1 and V2 connectors](/docs/fusion-connectors/concepts/v1-v2-connectors).
**FAQ**
How long will Lucidworks continue to support these connectors?
These connectors will remain available for at least 6 months from the release of Fusion 5.9.9 and will be removed in a release after this 6-month period.
To prepare for the eventual removal of these connectors in a future release, Lucidworks strongly recommends beginning the migration of your datasources to the following V2 connectors:
* [Windows Share SMB 2/3 V2](/docs/fusion-connectors/connectors/windows-share-smb-2-3-v2)
* [File Upload V2](/docs/fusion-connectors/connectors/fileupload-v2)
## Removals
### Bitnami removal
Fusion 5.9.8 will be re-released with the same functionality but updated image references.
In the meantime, Lucidworks will self-host the required images while we work to replace Bitnami images with internally built open-source alternatives.
If you are a self-hosted Fusion customer, *you must upgrade before August 28* to ensure continued access to container images and prevent deployment issues.
You can reinstall your current version of Fusion or upgrade to Fusion 5.9.14, which includes the updated Helm chart and prepares your environment for long-term compatibility.
See [Prevent image pull failures due to Bitnami deprecation in Fusion 5.9.5 to 5.9.13](https://support.lucidworks.com/hc/en-us/articles/33966125467799-Prevent-image-pull-failures-due-to-Bitnami-deprecation-in-Fusion-5-9-5-to-5-9-13) for more information on how to prevent image pull failures.
## Platform support and component versions
### Kubernetes platform support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.28, 1.29, 1.30
* **Microsoft Azure Kubernetes Service (AKS):** 1.28, 1.29, 1.30
* **Amazon Elastic Kubernetes Service (EKS):** 1.28, 1.29, 1.30
Support is also offered for Rancher Kubernetes Engine (RKE) and OpenShift 4 versions that are based on Kubernetes 1.28, 1.29, 1.30. OpenStack and customized Kubernetes installations are *not* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support).
### Component versions
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| **Solr** | fusion-solr 5.9.9 *(based on Solr 9.6.1)* |
| **ZooKeeper** | 3.9.1 |
| **Spark** | 3.2.2 |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
**More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).**
# Fusion 5.10.0
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/more/5.10.0-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.10.0-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.10.0-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/ce6szc
Released on September 28, 2023, this [rapid release](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#rapid-release-support-policy) includes new features and improvements, including Adobe Experience Manager (AEM) V2 connector, updates to Solr and Consumption Dashboard, and new user roles.
To learn more, skip to the [release notes](#new-features).
## Platform Support and Component Versions
### Kubernetes platform support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.25 and 1.26
* **Microsoft Azure Kubernetes Service (AKS):** 1.25 and 1.26
* **Amazon Elastic Kubernetes Service (EKS):** 1.25 and 1.26
Support is also offered for Rancher Kubernetes Engine (RKE) and OpenShift 4 versions that are based on Kubernetes 1.25 and 1.26. OpenStack and customized Kubernetes installations are *not* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support).
### Component versions
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| **Solr** | fusion-solr 5.10.0 *(based on Solr 9.3.0)* |
| **ZooKeeper** | 3.8.1 |
| **Spark** | 3.4.1 |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
**More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).**
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
## New Features
### Fusion Connectors
* The new Adobe Experience Manager (AEM) V2 connector has been added alongside the Fusion 5.10.0 release. The AEM V2 connector is compatible with Fusion 5.5.1 and later and works with AEM version 6.5. For more information, including configuration details, see [AEM V2 Connector Configuration Reference](/docs/fusion-connectors/connectors/aem).
## Improvements
### Fusion
* Solr has been upgraded to 9.3.
* The consumption dashboard now features a summary section that displays your total entitlements, requests, and records. You can also view your usage over your contract year.
* New user roles for Fusion have been created to enhance security. Each of these roles should be combined with the developer role in order to work.
* The spark-developer role can create, edit, and delete Spark jobs.
* The script-developer role can create, modify, and delete JavaScript code in the querying and indexing pipelines.
* The stage-plugin-developer role can create, edit, and delete custom index and query pipeline stages.
* The GraalVM JavaScript stage has been added to the index and query pipelines. It is available in Nashorn compatibility mode, but full backwards compatibility is not available. The Nashorn JavaScript library is still the default option. To use the GraalVM library, select **GraalVM** from the Javascript engine dropdown in the Fusion UI.\
This is an experimental feature and should not be used in production. Remote calls to other services are not available.
* The [Graph Security Trimming stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/security-trimming-graph-query-stage) has new default values for `aclSolrCollection`, `joinField`, and `joinMethod`. Previously deprecated fields `crossCollection` and `dvWithScore` are removed.
* Lucidworks Search only: The reverse search saved-query stage now removes any `partition_filter` fields from saved query documents that consist of the empty string.
## Bug Fixes
* Fixed a bug where a non-admin user with the proper permissions could not type in the pipeline configuration source code editor.
* Fixed a bug where the Context object used when posting each document in an array to an index pipeline was the same object.
* Fixed a bug where asynchronous parsing was unable to push documents to Kafka when TLS is enabled.
## Deprecations
* MLeap support has been deprecated. MLeap was used for machine learning tasks in Fusion, including SpaCy and SparkNLP deployments and certain ML models. Instead, refer to the **Develop and Deploy a Machine Learning Model** guide.
This tutorial walks you through deploying your own model to Fusion with Seldon Core.
## Prerequisites
* A Fusion instance with an app and indexed data
* An understanding of Python and the ability to write Python code
* [Docker](https://docs.docker.com/get-docker/) installed locally, plus a private or public Docker repository
* Seldon-core installed locally: `pip install seldon-core`
* Code editor; you can use any editor, but Visual Studio Code is used in the example
* Model: [paraphrase-multilingual-MiniLM-L12-v2 from Hugging Face](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2)
* Docker image: [example\_sbert\_model](https://hub.docker.com/r/jstrmec/example_sbert_model/)
## Tips
* Always test your Python code locally before uploading to Docker and then Fusion.
This simplifies troubleshooting significantly.
* Once you’ve created your Docker you can also test locally by doing `docker run` with a specified port, like 9000, which you can then `curl` to confirm functionality in Fusion.
See the testing example below.
The course for **Intro to Machine Learning in Fusion** focuses on using machine learning to infer the goals of customers and users in order to deliver a more sophisticated search experience.
## Local testing example
The examples in this section use the following models:
* [multilingual miniLM model](https://colab.research.google.com/drive/1LeCGl57_Evn_PCL25NJAt1MzP6GYH4j2)
* [e5-model](https://colab.research.google.com/drive/15WOHqqMBng5abUsWJYdElxgVLdAoJdn2)
1. Docker command:
```bash theme={"dark"}
docker run -p 127.0.0.1:9000:9000
```
2. Curl to hit Docker:
```bash theme={"dark"}
curl -X POST -H 'Content-Type: application/json' -d '{"data": { "ndarray": ["Sentence to test"], "names":["text"]} }' https://localhost:9000/api/v1.0/predictions
```
3. Curl model in Fusion:
```bash theme={"dark"}
curl -u $FUSION_USER:$FUSION_PASSWORD -X POST -H 'Content-Type: application/json' -d '{"text": "i love fusion"}' https://.lucidworks.com:6764/api/ai/ml-models//prediction
```
4. See all your deployed models:
```bash theme={"dark"}
curl -u USERNAME:PASSWORD http://FUSION_HOST:FUSION_PORT/api/ai/ml-models
```
## Download the model
This tutorial uses the [paraphrase-multilingual-MiniLM-L12-v2](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2) model from Hugging Face, but any pre-trained model from [https://huggingface.co](https://huggingface.co) will work with this tutorial.
If you want to use your own model instead, you can do so, but your model must have been trained and then saved though a function similar to the PyTorch’s `torch.save(model, PATH)` function.
See [Saving and Loading Models](https://pytorch.org/tutorials/beginner/saving_loading_models.html) in the PyTorch documentation.
## Format a Python class
The next step is to format a Python class which will be invoked by Fusion to get the results from your model.
The skeleton below represents the format that you should follow.
See also [Packaging a Python model for Seldon Core using Docker](https://docs.seldon.ai/seldon-core-1/configuration/wrappers-and-sdks/python-language-wrapper/python_wrapping_docker) in the Seldon Core documentation.
```python wrap expandable theme={"dark"}
class MyModel(object):
"""
Model template. You can load your model parameters in __init__ from a
location accessible at runtime
"""
def __init__(self):
"""
Add any initialization parameters. These will be passed at runtime
from the graph definition parameters defined in your seldondeployment
kubernetes resource manifest.
"""
print("Initializing")
def predict(self,X,features_names,**kwargs):
"""
Return a prediction.
Parameters
----------
X : array-like
feature_names : array of feature names (optional)
"""
print("Predict called - will run identity function")
return X
def class_names(self):
return ["X_name"]
```
A real instance of this class with the Paraphrase Multilingual MiniLM L12 v2 model is as follows:
```python wrap expandable theme={"dark"}
import logging
import os
from transformers import AutoTokenizer, AutoModel
from torch.nn import functional as F
from typing import Iterable
import numpy as np
import torch
log = logging.getLogger()
class mini():
def __init__(self):
self.tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')
self.model= AutoModel.from_pretrained('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')
#Mean Pooling
def mean_pooling(self, model_output, attention_mask):
token_embeddings = model_output[0] #First element of model_output contains all token embeddings
input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
def predict(self, X:np.ndarray, names=None, **kwargs):
# In Fusion there are several variables passed in the numpy array with the Milvus Query stage,
# Encode to Milvus index stage, and Vectorize Seldon index and query stage:
# [pipeline, bool, and text]. Text is what variable will be encoded, so that is what will be set to 'text'
# When using the Machine Learning stage, the input map keys should match what what is in this file.
model_input = dict(zip(names, X))
text = model_input["text"]
with torch.inference_mode(): # Allows torch to run more quickly
# Tokenize sentences
encoded_input = self.tokenizer(text, padding=True, truncation=True, return_tensors='pt')
log.debug('encoded input',str(encoded_input))
model_output = self.model(**encoded_input)
log.debug('model output',str(model_output))
# Perform pooling. In this case, max pooling.
sentence_embeddings = self.mean_pooling(model_output, encoded_input['attention_mask'])
# Normalize embeddings, because Fusion likes it that way.
sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=-1)
# Fixing the shape of the emebbedings to match (1, 384).
final = [sentence_embeddings.squeeze().cpu().detach().numpy().tolist()]
return final
def class_names(self) -> Iterable[str]:
return ["vector"]
```
In the above code, an additional function has been added in the class; this is completely fine to do.
Logging has also been added for debugging purposes.
Two functions are non-negotiable:
* `init`: The `init` function is where models, tokenizers, vectorizers, and the like should be set to self for invoking.\
It is recommended that you include your model’s trained parameters directly into the Docker container rather than reaching out to external storage inside `init`.
* `predict`: The `predict` function processes the field or query that Fusion passes to the model.\
The `predict` function must be able to handle any text processing needed for the model to accept input invoked in its `model.evaluate()`, `model.predict()`, or equivalent function to get the expected model result.
If the output needs additional manipulation, that should be done before the result is returned.\
For embedding models the return value must have the shape of (1, DIM), where DIM (dimension) is a consistent integer, to enable Fusion to handle the vector encoding into Milvus or Solr.
Use the *exact* name of the class when naming this file.
For the example, above the Python file is named `mini.py` and the class name is `mini()`.
## Create a Dockerfile
The next step is to create a Dockerfile. The Dockerfile should follow this general outline; read the comments for additional details:
```dockerfile wrap expandable theme={"dark"}
#It is important that python version is 3.x-slim for seldon-core
FROM python:3.10-slim
# Whatever directory(folder)the python file for your python class, Dockerfile, and
# requirements.txt is in should be copied then denoted as the work directory.
COPY . /app
WORKDIR /app
# The requirements file for the Docker container
COPY requirements.txt requirements.txt
RUN pip install --upgrade pip && pip install --no-cache-dir -r requirements.txt
# Copy source code
COPY . .
# GRPC - Allows Fusion to do a Remote Procedure Call
EXPOSE 5000
# Define environment variable for seldon-core
# !!!MODEL_NAME must be the EXACT same as the python file & python class name!!!
ENV MODEL_NAME=mini
ENV SERVICE_TYPE=MODEL
ENV PERSISTENCE=0
# Changing active directory folder (same one as above on lines 5 & 6) to default user, required for Fusion
RUN chown -R 8888 /app
# Command to wrap python class with seldon-core to allow it to be usable in Fusion
CMD ["sh", "-c", "seldon-core-microservice $MODEL_NAME --service-type $SERVICE_TYPE --persistence $PERSISTENCE"]
# You can use the following if You need shell features like environment variable expansion or
# You need to use shell constructs like pipes, redirects, etc.
# See https://docs.docker.com/reference/dockerfile/#cmd for more details.
# CMD exec seldon-core-microservice $MODEL_NAME --service-type $SERVICE_TYPE --persistence $PERSISTENCE
```
## Create a requirements file
The `requirements.txt` file is a list of installs for the `Dockerfile` to run to ensure the Docker container has the right resources to run the model.\
For the Paraphrase Multilingual MiniLM L12 v2 model, the requirements are as follows:
```text theme={"dark"}
seldon-core
torch
transformers
numpy
```
In general, if an item was used in an `import` statement in your Python file, it should be included in the requirements file.
An easy way to populate the requirements is by using in the following command in the terminal, inside the directory that contains your code:
```bash theme={"dark"}
pip freeze > requirements.txt
```
If you use `pip freeze`, you must manually add `seldon-core` to the requirements file because it is not invoked in the Python file but is required for containerization.
## Build and push the Docker image
After creating the `.py`, `Dockerfile`, and `requirements.txt` files, you need to run a few Docker commands.
Run the commands below in order:
```bash theme={"dark"}
DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build . -t [DOCKERHUB-USERNAME]/[REPOSITORY]:[VERSION-TAG]
```
```bash theme={"dark"}
docker push [DOCKERHUB USERNAME]/[REPOSITORY]:[VERSION-TAG]
```
Using the example model, the terminal commands would be as follows:
```bash theme={"dark"}
DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build . -t jstrmec/example_sbert_model:0.14; docker push jstrmec/example_sbert_model:0.14
```
This repository is public and you can visit it here: [example\_sbert\_model](https://hub.docker.com/r/jstrmec/example_sbert_model/)
## Deploy the model in Fusion
Now you can go to Fusion to deploy your model.
1. In Fusion, navigate to **Collections** > **Jobs**.
2. Add a job by clicking the **Add+** Button and selecting **Create Seldon Core Model Deployment**.
3. Fill in each of the text fields:
| Parameter | Description |
| - | - |
| Job ID | A string used by the Fusion API to reference the job after its creation. |
| Model name | A name for the deployed model. This is used to generate the deployment name in Seldon Core. It is also the name that you reference as a `model-id` when making predictions with the ML Service. |
| Model replicas | The number of load-balanced replicas of the model to deploy; specify multiple replicas for a higher-volume intake. |
| Docker Repository | The public or private repository where the Docker image is located. If you’re using Docker Hub, fill in the Docker Hub username here. |
| Image name | The name of the image with an optional tag. If no tag is given, `latest` is used. |
| Kubernetes secret | If you’re using a private repository, supply the name of the Kubernetes secret used for access. |
| Output columns | A list of column names that the model’s `predict` method returns. |
4. Click **Save**, then **Run** and **Start**.
When the job finishes successfully, you can proceed to the next section.
Now that the model is in Fusion, it can be utilized in either index or query pipelines, depending on the model’s purpose.
In this case the model is a word vectorizer or semantic vector search implementation, so both pipelines must invoke the model.
## Apply an API key to the deployment
These steps are only needed if your model utilizes any kind of secret, such as an API key.
If not, skip this section and proceed to the next.
1. Create and modify a `_sdep.yaml` file.\
In the first line, `kubectl get sdep` gets the details for the currently running Seldon Deployment job and saves those details to a YAML file. `kubectl apply -f open_sdep.yaml` adds the key to the Seldon Deployment job the next time it launches.
```yaml theme={"dark"}
kubectl get sdep -o yaml > _sdep.yaml
# Modify _sdep.yaml to add
- env:
- name: API_KEY
value: "your-api-key-here"
kubectl apply -f _sdep.yaml
```
2. Delete `sdep` before redeploying the model. The currently running Seldon Deployment job does not have the key applied to it. Delete it before redeploying and the new job will have the key.
```bash theme={"dark"}
kubectl delete sdep
```
3. Lastly, you can [encode into Milvus](/docs/5/fusion/intro/fusion-stack/milvus).
## Create a Milvus collection
1. In Fusion, navigate to **Collections** > **Jobs**.
2. Click the **Add+** Button and select **Create Collections in Milvus**.\
This job creates a collection in Milvus for storing the vectors sent to it.
The job is needed because a collection does not automatically spawn at indexing or query time if it does not already exist.
3. Name the job and the collection.
4. Click **Add** on the right side of the job panel.\
The key to creating the collection is the **Dimension** text field; this must exactly match the shape value your output prediction has.\
In our example the shape is (1,384), so 384 will be in the collections **Dimension** field:
The **Metric** field should typically be left at the default of `Inner Product`, but this also depends on use case and model type.
5. Click **Save**, then **Run** and **Start**.
## Configure the Fusion pipelines
Your real-world pipeline configuration depends on your use case and model, but for our example we will configure the index pipeline and then the query pipeline.
**Configure the index pipeline**
1. Create a new index pipeline or load an existing one for editing.
2. Click **Add a Stage** and then **Encode to Milvus**.
3. In the new stage, fill in these fields:
* The name of your model
* The output name you have for your model job
* The field you’d like to encode
* The collection name
4. Save the stage in the pipeline and index your data with it.
**Configure the query pipeline**
1. Create a new query pipeline or load an existing one for editing.
2. Click **Add a Stage** and then **Milvus Query**.
3. Fill in the configuration fields, then save the stage.
4. Add a **Milvus Ensemble Query** stage.\
This stage is necessary to have the Milvus collection scores taken into account in ranking and to weight multiple collections.
The **Milvus Results Context Key** from the Milvus Query Stage is used in this stage to preform math on the Milvus result scores.
One (1) is a typical multiplier for the Milvus results but any number can be used.
5. Save the stage and then run a query by typing a search term.
6. To verify the Milvus results are correct, use the **Compare+** button to see another pipeline without the model implementation and compare the number of results.
You have now successfully uploaded a Seldon Core model to Fusion and deployed it.
## Removals
For full details on removals, see [Deprecations and Removals](/docs/5/fusion/deprecations-and-removals).
* The Forked Apache Tika parser stage has been removed. Use [asynchronous Tika parsing](/docs/5/fusion/getting-data-in/indexing/asynchronous-tika-parsing) instead.
* The Analytics Catalog query stage has been removed.
## Known issues
* New Kerberos security realms cannot be configured successfully in this version of Fusion.
# Fusion 5.11.0
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/more/5.11.0-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.11.0-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.11.0-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/dmayvi
Released on January 3, 2024, this [rapid release](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#rapid-release-support-policy) includes new features and improvements, including the App Manager API that generates information for Consumption Dashboard and updates to Solr and Kubernetes.
To learn more, skip to the [release notes](#new-features).
## Platform Support and Component Versions
### Kubernetes platform support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.27
* **Microsoft Azure Kubernetes Service (AKS):** 1.27
* **Amazon Elastic Kubernetes Service (EKS):** 1.27
Support is also offered for Rancher Kubernetes Engine (RKE) and OpenShift 4 versions that are based on Kubernetes 1.27. OpenStack and customized Kubernetes installations are *not* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support).
## Component versions
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| **Solr** | fusion-solr 5.11.0 *(based on Solr 9.4.0)* |
| **ZooKeeper** | 3.9.0 |
| **Spark** | 3.4.1 |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
**More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).**
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
## New Features
### Fusion
* The [App Manager API](/api-reference/license/get-the-fusion-license) provides programmatic access to information about your Fusion license, entitlements, and usage data. This data is also used to generate usage statistics in the [Consumption Dashboard](/docs/5/fusion/operations/monitoring-and-reporting/consumption).
For example, send a request to the `/apps-manager/entitlements/current` endpoint to retrieve general entitlements usage data:
**Input**
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD https://FUSION_HOST:FUSION_PORT/apps-manager/entitlements/current
```
**Output**
```json theme={"dark"}
{
"contractDuration":{
"years":3,
"months":0,
"days":0
},
"info":{
"start":"2022-12-23",
"end":"2023-12-23",
"requests":10000000,
"records":5000000
},
"usage":{
"requests":2863,
"requestsPercentage":0,
"records":565672,
"recordsPercentage":11
}
}
```
## Improvements
### Fusion
* Solr has been upgraded to 9.4. For details about changes in Solr 9.4, see [Apache Solr Release Notes](https://solr.apache.org/guide/solr/latest/upgrade-notes/major-changes-in-solr-9.html#solr-9-4).
* Fusion now supports Kubernetes 1.27. This applies to GKE, AKS, and EKS. It also applies to Rancher (RKE) and OpenShift 4 versions that are compatible with Kubernetes 1.27. Refer to Kubernetes documentation at [Kubernetes v1.27](https://kubernetes.io/blog/2023/04/11/kubernetes-v1-27-release/) for more information.
* Fusion safeguards against indexing overload with an ingestion flow control system. This system prevents timeouts by:
* Logging all Kafka topics used for indexing to aid in diagnosing performance issues.
* Monitoring the rate of data production and consumption for each Kafka topic across all indexing jobs.
* Automatically throttling producers to ensure consumers can keep up.
* The [Consumption Dashboard](/docs/5/fusion/operations/monitoring-and-reporting/consumption) has been updated to restore three charts.
* The Fusion UI is updated with improved messaging around Fusion license renewal and expiration dates.\
License details
License expiration notice
* The Fusion API Gateway now includes an optional Kubernetes health check. To enable this check, edit your `values.yaml` file, and add `-Dspring.cloud.kubernetes.loadbalancer.mode=POD` to the API Gateway’s `javaToolOptions`. For example:
```yaml theme={"dark"}
fusion:
api-gateway:
javaToolOptions: "-Dspring.cloud.kubernetes.loadbalancer.mode=POD"
```
When enabled, if the health check fails during pod discovery, Kubernetes will restart the pod.
* The fusion-solr Helm chart now accepts user-defined system properties that are set in `values.yaml`. In the `values.yaml` file, populate the `solrEnvExtra` array with entries for each desired variable. Each variable requires a name and either a direct value or a value source (`valueFrom`). These variables will be included in the final configuration in the order you provide them, following any variables dynamically generated by the Helm chart templates. For more information on Kubernetes environment variables, see [Define Dependent Environment Variables](https://kubernetes.io/docs/tasks/inject-data-application/define-interdependent-environment-variables/).
* The Fusion migration script is updated to align with changes from the Solr upgrade. The migration script:
* Removes the unused configuration, ``, from `solrconfig.xml`. Solr no longer supports this configuration.
* Detects if `StatelessScriptUpdateProcessorFactory` is found in `solrconfig.xml`. The migration script will update the configuration, if possible, and provide guidance on actions you must take.
## Fusion connectors
* The default number of fetch threads for V2 connectors is now 5. Previously, the default was 20. This default setting was reduced because some Fusion environments experienced failed datasource crawls as a result of too many requests in a short period of time.
## Bug Fixes
* The [Configurations](/api-reference/configurations/get-configurations-service-status) and [ZooKeeper Import/Export](/api-reference/zookeeper/get-zookeeper-data) APIs are now accessible only to authorized users from within your Fusion cluster or through Kubernetes port forwarding.
* The Fusion API realm can now authenticate when an API key is passed via an `apiKey` query parameter. Previously, this only worked via the `x-api-key` header.
* The ZooKeeper Docker image is updated to use [Logback](https://logback.qos.ch/) for logging instead of log4j. This change fixes a bug that forced all Zookeeper log messages to use the DEBUG level.
* Fixed a bug that sometimes resulted in an HTTP 431 error when using the Fusion admin UI.
* Fixed a bug that sometimes resulted in an HTTP 414 error when using the Rules UI.
## Deprecations
For full details on deprecations, see [Deprecations and Removals](/docs/5/fusion/deprecations-and-removals).
* Fusion has deprecated the Webapps service. In previous versions of Fusion, you could use this service to deploy an App Studio WAR file into Fusion. As of Fusion 5.11.0, you should instead **Deploy App Studio Enterprise to a Fusion 5 Cluster (GKE)**.
This article explains how to deploy [App Studio Enterprise](/docs/5/fusion/dev-portal/appkit/concepts/app-studio-enterprise/overview) (ASE) to an existing Fusion 5 cluster in Google Kubernetes Engine (GKE) without using the Webapps service.
Before completing this guide, ensure that your `fusion.conf` file points to the IP or URI and port of the proxy service.
Run the App Studio Enterprise application locally and verify functioning security and search features with the cluster you are deploying to.
## Prepare the package
1. Package your app into a single JAR file:
```
./app-studio package
```
2. App Studio Enterprise includes a `dockerfile`. Create the App Studio Enterprise Docker image:
```
docker build PATH -t APP_NAME
```
Set or replace `APP_NAME` with the name of your application. Replace `PATH` with the path to build from.
3. You can test the Docker image locally with the following command:
```
docker run -it -p LOCAL_PORT:8080 APP_NAME
```
Set or replace `LOCAL_PORT` with the port on your local machine that can access the app. Replace `APP_NAME` with the ASE application name.
## Publish the image
You can publish the Docker image anywhere that supports Docker images. This section explains how to publish a Docker image to Google Kubernetes Engine.
1. Tag your container for the registry:
```
docker tag APP_NAME gcr.io/PROJECT_NAME/APP_NAME
```
2. Push your Docker image to the Google Container Registry:
```
docker push gcr.io/PROJECT_NAME/APP_NAME
```
3. Verify the image:
```
gcloud compute instances list
```
## Deploy the app to a cluster
After publishing your ASE Docker image, deploy the image to a cluster. Your existing Fusion 5 cluster is a good choice.
1. Switch the context to your Fusion 5 cluster:
```
gcloud container clusters get-credentials CLUSTER_NAME
```
Replace `CLUSTER_NAME` with your existing Fusion 5 cluster’s name.
2. Create a deployment in your cluster using the image you published:
```
kubectl create deployment APP_NAME --image=gcr.io/PROJECT_NAME/APP_NAME:latest
```
## Create an Ingress resource
After deploying the app, create an Ingress resource for your ASE instance. If you require more Ingress rules, your Ingress resource will look different. Learn more about [Ingress resources](https://kubernetes.io/docs/concepts/services-networking/ingress/).
1. Use the following command to create a minimal Ingress resource:
```
cat <
## Removals
For full details on removals, see [Deprecations and Removals](/docs/5/fusion/deprecations-and-removals).
* The previously non-functional Subscriptions UI has been removed. This is related to the removal of the [Subscriptions API](/docs/5/fusion/release-notes/more/5.6.0-release-notes) in Fusion 5.6.0.
* MLeap is removed from the ml-model service. MLeap was deprecated in Fusion 5.2.0 and was no longer used by Fusion.
## Known issues
* New Kerberos security realms cannot be configured successfully in this version of Fusion.
# Fusion 5.12.0
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/more/5.12.0-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.12.0-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.12.0-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/2sq5ks
Released on March 28, 2024, this [rapid release](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#rapid-release-support-policy) includes several new features, including the asynchronous parsing service and the JSON viewer, as well as key improvements to the datasource job history.
To learn more, skip to the [release notes](#new-features).
## Platform Support and Component Versions
### Kubernetes platform support
Lucidworks has tested and validated support for the following Kubernetes platforms and versions:
* **Google Kubernetes Engine (GKE):** 1.28, 1.27, and 1.26
* **Microsoft Azure Kubernetes Service (AKS):** 1.28, 1.27, and 1.26
* **Amazon Elastic Kubernetes Service (EKS):** 1.28, 1.27, and 1.26
Support is also offered for Rancher Kubernetes Engine (RKE) and OpenShift 4 versions that are based on Kubernetes 1.28, 1.27, and 1.26. OpenStack and customized Kubernetes installations are *not* supported.
For more information on Kubernetes version support, see the [Kubernetes support policy](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support).
### Component versions
The following table details the versions of key components that may be critical to deployments and upgrades.
| Component | Version |
| - | - |
| **Solr** | fusion-solr 5.12.0 *(based on Solr 9.4.0)* |
| **ZooKeeper** | 3.9.0 |
| **Spark** | 3.4.1 |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), and GKE Ingress Controller. Istio is not supported. |
For additional component information, see [Docker Images by Fusion Version](/docs/5/fusion/reference/docker-images-versions).
## New Features
### Fusion
#### JSON viewer for Fusion objects
A new JSON viewer has been introduced in Fusion, providing a user-friendly interface to inspect the JSON configuration of key objects like index pipelines, query pipelines, jobs, and datasources. This viewer allows users to:
* **Visualize configuration:** View the complete JSON representation of the selected object’s configuration in a clear and well-formatted manner. For example, a basic datasource configuration resembles the following:
```json theme={"dark"}
{
"id": "mens-products",
"created": "2024-02-12T15:08:29.332Z",
"modified": "2024-02-12T15:08:29.332Z",
"connector": "lucid.fileupload",
"type": "fileupload",
"pipeline": "Documentation",
"parserId": "_system",
"properties": {
"collection": "Documentation",
"fileId": "mens-products.csv",
"mediaType": "application/octet-stream"
}
}
```
* **Facilitate operations:** Easily copy the displayed JSON for use in essential activities such as API calls.
* **Download for reference:** Download the JSON configuration as a file for backup or collaboration.
To learn how to use the JSON viewer, see the following demonstration:
The quick learning for **JSON Viewer** focuses on how to incorporate the JSON viewer into your daily workflow.
#### Asynchronous parsing service
An asynchronous parsing service for connectors, Apache Tika Container, has been added. This stage is now required when using asynchronous parsing for connectors.
To use asynchronous parsing for connectors, be sure that the **Async Parsing** is checked in the datasource and the Apache Tika Container parser stage is enabled in the index pipeline.
Other parsers, such as HTML and JSON, are now supported by the asynchronous parsing service. By enabling asynchronous parsing, the parser configuration linked to your datasource is used.
To learn how to start using the new asynchronous parsing service, see the following demonstration:
### Lucidworks Search
#### Cloud signal storage for Lucidworks Search
Lucidworks Search now offers the flexibility to store signals data in Google Cloud Storage or Amazon S3, providing an alternative to the default Solr storage. Cloud signal storage minimizes the storage burden on your Solr cluster, potentially leading to improved performance and resource optimization.
All signal types, including custom types, are fully supported for cloud storage. Additionally, customization of the default signal schema remains available. Signals data files are automatically compacted over time into larger files, resulting in significant storage space savings and simplified file management.
Cloud signal storage is enabled during initial deployment. For more information, see [Cloud signal storage for Lucidworks Search](/docs/lucidworks-search/07-improve-your-queries/signals/cloud-signal-storage-59).
## Improvements
* Introduced significant improvements to job status reporting within Fusion. A dedicated job function microservice, `job-config`, has been added to control job execution and store job history. This microservice requires no configuration from the client. In addition, to optimize communication efficiency, synchronous calls have been replaced with asynchronous communication through Kafka. As a result of these changes, job reports are now far more accurate and reliable than before.
These changes do not impact Fusion API usage or user workflows.
* Introduced a configurable timeout for datasource jobs, in case the connection to the backend job is lost. Previously, the job was stopped after a fixed duration of 100 seconds.
* Previously query pipeline pods could restart due to scaling or other reasons, leading to 503 errors in the API gateway. This release introduces a configurable retry setting for the api-gateway service that will automatically attempt to reconnect to the query pipeline pods. This addresses the previously observed 503 errors under sustained query load.
* Pre-packaged configurations for commonly used dimensionalities (64, 384, 512, and 768) in vector search operations within Solr are added to Fusion. These configurations leverage a naming convention that incorporates dimensionality into dynamic field names, simplifying setup and management for users. For example:
```xml theme={"dark"}
```
* Introduced automated management of Kafka topic partitions for indexing subscriptions within Fusion. Fusion now intelligently determines the optimal number of partitions, creates the topics with appropriate configurations, and initiates the subscription process.
## Bug Fixes
* Fixed a bug that prevented remote connector datasources from indexing data after encountering a "No Plugin Activity" error. Previously, affected datasources required recreation with a different name despite maintaining identical specifications.
* Fixed a bug that caused a Solr syntax error when attempting to delete an aclDocument using the SDK method `newDeleteGraphAccessControlItem` if the ID of the document contained special characters.
* Fixed a bug that sometimes resulted in a `ConcurrentModificationException` error during high-load testing of query pipelines utilizing Solr Subquery and Merge Async Results stages.
* Addressed a race condition impacting dynamic pricing functionality. Previously, concurrent user requests for the same external file could lead to the second request encountering an error. The fix ensures proper handling of concurrent requests, eliminating the potential for this error.
* Fixed a bug that caused a `ConcurrentModificationException` error during the `.RawPull` replication process within Solr.
* Fixed a bug in the UI login process where attempting to log in with an incorrect password after an expired session resulted in no error message being displayed on the UI.
## Deprecations
For full details on deprecations, see [Deprecations and Removals](/docs/5/fusion/deprecations-and-removals).
### V1 Fusion connectors
In Fusion 5.12.0, all V1 connectors are deprecated. This means they are no longer being actively developed and will be removed in the next major release, Fusion 6.
* **Some V1 connectors already have a direct replacement available.** You can find the replacement connector on the [Fusion Connectors Deprecations and Removals page](/docs/fusion-connectors/deprecations-and-removals).
* **For all other V1 connectors, a replacement is still under development.** The documentation is updated when the replacement is available.
If you are using a V1 connector, you must migrate to the replacement connector or a supported alternative before upgrading to Fusion 6. Migrate to the replacement connector as soon as possible to avoid any disruption to your workflows.
### Banana dashboards
Banana, the open source browser visualization tool that previously powered Fusion’s Banana Dashboards, is deprecated for Fusion. It is scheduled for removal in Fusion 5.13.0. While Banana will no longer be included with Fusion, users who require continued use can leverage a separate Docker deployment option. This alternative deployment method will be available in Fusion 5.13.0.
If you would like to explore the open source project, see [Banana on GitHub](https://github.com/lucidworks/banana).
### Other deprecations
* App Insights and the insights microservice are deprecated and will be removed in a future release.
* The Parsers CRUD API from the indexing service is deprecated and will be removed in a future release. Use `/async-parsing/` instead.
* The Train a Smart Answers cold start model job is deprecated and will be removed in a future release. Use one of the pre-trained models or the supervised training job instead.
* The Data Augmentation job is deprecated and will be removed in a future release.
## Known issues
* New Kerberos security realms cannot be configured successfully in this version of Fusion.
# Fusion 5.5.0
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/more/5.5.0-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.5.0-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.5.0-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/301ya6
**Release date:** March 18, 2022
**Component versions:**
| Component | Version |
| - | - |
| **Solr** | 8.11.1 |
| **ZooKeeper** | 3.6.2 |
| **Spark** | 2.4.5 |
| **Kubernetes** | GKE, AKS, EKS 1.22 Rancher (RKE) and OpenShift 4 compatible with Kubernetes 1.22 OpenStack and customized Kubernetes installs not supported. See [Kubernetes support](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support) for end of support dates. |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
## Improvements
* Fusion 5.5 adds support for Rancher. See **Deploy Fusion 5 on Other Kubernetes Platforms** for more information.
The [`setup_f5_k8s.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_k8s.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository provides deployment support for any Kubernetes platform, including on-premise, private cloud, public cloud, and hybrid platforms.
This script is used by the [`setup_f5_gke.sh`](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_gke.sh), [`setup_f5_eks.sh`](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_eks.sh), and [`setup_f5_aks.sh`](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_aks.sh) scripts, which provide additional platform-specific support for Google Kubernetes Engine (GKE), Amazon Elastic Kubernetes Service (EKS), and Azure Kubernetes Service (AKS).
This topic explains how to deploy a Fusion cluster in Kubernetes using the [`setup_f5_k8s.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_k8s.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository.
If you’re deploying on-premises or using a localized repository, you’ll need to use a private repository for Docker images.
## Prerequisites
This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes.
### Release Name and Namespace
Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) Kubernetes namespace to install Fusion into.
Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*.
However, please *do not* install more than one Fusion release in the same namespace.
All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_
Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default.
### Install Helm
Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster.
Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform.
On MacOS, you can do:
```bash theme={"dark"}
brew install kubernetes-helm
```
If you already have helm installed, make sure you're using the latest version:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/)
The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`.
#### Helm User Permissions
If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory.
When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.:
```bash theme={"dark"}
alias k=kubectl
```
To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into:
```bash theme={"dark"}
k create namespace fusion-namespace
```
Apply the `role.yaml` and `cluster-role.yaml` files to that namespace
```bash theme={"dark"}
k apply -f cluster-role.yaml
k config set-context --current --namespace=$NAMESPACE
k apply -f role.yaml
```
Then bind the rolebinding and clusterolebinding to the install user:
```bash wrap theme={"dark"}
k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user
k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user
```
You will then be able to run the `helm install` command as the ``
### Clone fusion-cloud-native from GitHub
You should clone this repo from github as you'll need to run the scripts on your local workstation:
```bash theme={"dark"}
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts.
```bash theme={"dark"}
cd fusion-cloud-native
git pull
```
Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip).
Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory.
## Deployment
If you're not running on a managed K8s platform like GKE, AKS, or EKS, you can use Helm to install the Fusion chart to an existing Kubernetes cluster.
Fusion version 5.5 now includes support for the Rancher Kubernetes Engine (RKE) platform. Before deploying Fusion to RKE, you must download and install the [latest RKE software](https://rancher.com/docs/rke/latest/en/). After configuring your cluster, you can proceed with the Helm v3 installation.
You must have a working cluster configured before performing the Helm v3 installation.
### Use Helm v3 to Install Fusion
You should upgrade to the latest version of Helm v3 for working with Fusion. If you need to keep Helm V2 for other clusters,
ensure Helm V3 is ahead of Helm V2 in your working shell's PATH before proceeding.
#### Customize Fusion Chart Settings
Fusion aims to be well-configured out-of-the-box, but you can customize any of the built-in settings using a custom values `YAML` file. If you use one of our setup scripts, such as `setup_f5_gke.sh`, then it will create a custom values YAML file for you the first time you run it using [https://github.com/lucidworks/fusion-cloud-native/blob/master/customize\_fusion\_values.yaml.example](https://github.com/lucidworks/fusion-cloud-native/blob/master/customize_fusion_values.yaml.example) as a template.
If you're working with Helm directly and not using one of our setup scripts, then run the [https://github.com/lucidworks/fusion-cloud-native/blob/master/customize\_fusion\_values.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/customize_fusion_values.sh) script to create a custom values YAML file from our [https://github.com/lucidworks/fusion-cloud-native/blob/master/customize\_fusion\_values.yaml.example](https://github.com/lucidworks/fusion-cloud-native/blob/master/customize_fusion_values.yaml.example) template as a starting point:
```bash theme={"dark"}
./customize_fusion_values.sh -c -n \
--provider --num-solr 1 --node-pool ""
```
Pass `--help` for usage details.
In this example:
* `` is the K8s platform you're running on, such as `gke`
* `` is the name of your cluster
* `` is the K8s namespace where you plan to install Fusion
The `--node-pool` option specifies the node selector label for determining which nodes to run Fusion pods. You can pass `"{}"` to let Kubernetes decide which nodes to schedule pods on.
This file is referred to as `${MY_VALUES}` in the commands belo. Replace the filename with the correct filename for your environment. Keep this file handy, as you'll need it to customize Fusion settings and upgrade to a newer version.
Review the settings in the custom values YAML file to ensure the defaults are appropriate for your environment, including the number of Solr and Zookeeper replicas.
Add the Lucidworks Helm repo:
```bash theme={"dark"}
helm repo add lucidworks https://charts.lucidworks.com
```
The `customize_fusion_values.sh` script creates an upgrade script to install/upgrade Fusion into Kubernetes using Helm. Look in the directory where you ran `customize_fusion_values.sh` for a script named like:
`___upgrade_fusion.sh`. Run this script to install Fusion.
## Verifying the Fusion Installation
In this section, we provide some tips on how to verify the Fusion installation.
Check if the Fusion Admin UI is available at `\https://:6764/admin/`.
Let's review some useful kubectl commands.
### Enhance the K8s Command-line Experience
Here is a list of tools we found useful for improving your command-line experience with Kubernetes:
* krew (kubectl plugin mgr): [https://github.com/kubernetes-sigs/krew/](https://github.com/kubernetes-sigs/krew/)
* kube-ps1 (show current context on command line prompt): [https://github.com/jonmosco/kube-ps1](https://github.com/jonmosco/kube-ps1)
* kubectx / kubens (switch between clusters / namespaces): [https://github.com/ahmetb/kubectx](https://github.com/ahmetb/kubectx)
### Useful kubectl commands
kubectl reference: [https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands)
Set the namespace for `kubectl` if not using the default:
```bash theme={"dark"}
kubectl config set-context --current --namespace=
```
**This saves you from having to pass `-n` with every command.**
Get a list of running pods: `k get pods`
Get logs for a pod using a label: `k logs –l app.kubernetes.io/component=query-pipeline`
Get pod deployment spec and details: `k get pods -o yaml`
Get details about a pod events: `k describe po `
Port forward to a specific pod: `k port-forward 8983:8983`
SSH into a pod: `k exec -it -- /bin/bash`
CPU/Memory usage report for pods: `k top pods`
Forcefully kill a pod: `k delete po --force --grace-period 0`
Scale up (or down) a deployment: `k scale deployment.v1.apps/ --replicas=N`
Get a list of pod versions: `k get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq`
### Check Fusion Pods and Services
Once the install script completes, you can check that all pods and services are available using:
```bash theme={"dark"}
kubectl get pods
```
If all goes well, you should see a list of pods similar to:
```bash wrap theme={"dark"}
NAME READY STATUS RESTARTS AGE
seldon-controller-manager-6675874894-qxwrv 1/1 Running 0 8m45s
f5-admin-ui-74d794f4f8-m5jms 1/1 Running 0 8m45s
f5-ambassador-fd6b9b5dc-7ghf6 1/1 Running 0 8m43s
f5-api-gateway-6b9998b9c-tmchk 1/1 Running 0 8m45s
f5-auth-ui-7565564b4c-rdc74 1/1 Running 0 8m42s
f5-classic-rest-service-0 1/1 Running 3 8m44s
f5-devops-ui-77bb867ffb-fbzxd 1/1 Running 0 8m42s
f5-fusion-admin-78b8f8fc7f-4d7l8 1/1 Running 0 8m42s
f5-fusion-indexing-599c8d448-xzsvm 1/1 Running 0 8m44s
f5-insights-665fd9f6fc-g5psw 1/1 Running 0 8m43s
f5-job-launcher-84dd4c5c96-p8528 1/1 Running 0 8m44s
f5-job-rest-server-6d44d964b8-xtnxw 1/1 Running 0 8m45s
f5-logstash-0 1/1 Running 0 8m45s
f5-ml-model-service-6987dc94c9-9ppp8 2/2 Running 1 8m45s
f5-monitoring-grafana-5d499dbb58-pzw72 1/1 Running 0 10m
f5-monitoring-prometheus-kube-state-metrics-54d6678dv9h7h 1/1 Running 0 10m
f5-monitoring-prometheus-pushgateway-7d65c65b85-vwrwf 1/1 Running 0 10m
f5-monitoring-prometheus-server-0 2/2 Running 0 10m
f5-pm-ui-86cbc5bb65-nd2n8 1/1 Running 0 8m44s
f5-pulsar-bookkeeper-0 1/1 Running 0 8m45s
f5-pulsar-broker-b56cc776f-56msx 1/1 Running 0 8m45s
f5-query-pipeline-5d75d7d5f4-l2mdf 1/1 Running 0 8m43s
f5-connectors-7bb6cfc65f-7wfs2 1/1 Running 0 8m42s
f5-connectors-backend-987fdc648-dldwv 1/1 Running 0 8m45s
f5-rules-ui-6b9d55b78f-9hzzj 1/1 Running 0 8m43s
f5-solr-0 1/1 Running 0 8m44s
f5-solr-exporter-c4687c785-jsm7x 1/1 Running 0 8m45s
f5-ui-6cdbcc68c6-rj9cq 1/1 Running 0 8m45s
f5-webapps-6d6bb9bfd-hm4qx 1/1 Running 0 8m45s
f5-workflow-controller-7b66679fb7-sjbvp 1/1 Running 0 8m44s
f5-zookeeper-0 1/1 Running 0 8m45s
```
The number of pods per deployment / statefulset will vary based on your cluster size and replicaCount settings in your custom values YAML file.
Also, don't worry if you see some pods having been restarted as that just means they were too slow to come up and Kubernetes killed and restarted them.
You do want to see at least one pod running for every service. If a pod is not running after waiting a sufficient amount of time,
use `kubectl logs ` to see the logs for that pod; to see the logs for previous versions of a pod, use: `kubectl logs -p`.
You can also look at the actions Kubernetes performed on the pod using `kubectl describe po `.
To see a list of Fusion services, do:
```bash theme={"dark"}
kubectl get svc
```
For an overview of the various Fusion 5 microservices, see: [Fusion microservices](/docs/5/fusion/reference/microservices).
Once you're ready to build a Fusion cluster for production, please see see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
### Upgrading with Zero Downtime
One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. Fusion 5 allows customers to upgrade from Fusion 5.x.y to a later 5.x.z version on a live cluster with zero downtime or disruption of service.
When Kubernetes performs a rolling update to an individual microservice, there will be a mix of old and new services in the cluster concurrently (only briefly in most cases) and requests from other services will be routed to both versions. Consequently, Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same 5.x line of releases. We also ensure stored configuration remains compatible in the same 5.x release line.
Lucidworks releases minor updates to individual services frequently, so our customers can pull in those upgrades using Helm at their discretion.
To upgrade your cluster at any time, use the `--upgrade` option with our setup scripts in this repo.
The scripts in this repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.
To see what would be upgraded, you can pass the `--dry-run` option to the script.
### Grafana Dashboards
Get the initial Grafana password from a K8s secret by doing:
```bash wrap theme={"dark"}
kubectl get secret --namespace "${NAMESPACE}" ${RELEASE}-monitoring-grafana \
-o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```
With Grafana, you can either setup a temporary port-forward to a Grafana pod or expose Grafana on an external IP using a K8s LoadBalancer.
To define a LoadBalancer, do (replace \$ with your Helm release label):
```bash wrap theme={"dark"}
kubectl expose deployment ${RELEASE}-monitoring-grafana --type=LoadBalancer --name=grafana --port=3000 --target-port=3000
```
You can use `kubectl get services --namespace ` to determine when the load balancer is setup and its IP address. Direct your browser to `http://:3000` and enter the username `admin@localhost` and the password that was returned in the previous step.
This will log you into the application. It is recommended that you create another administrative user with a more desirable password.
The dashboards and datasoure will be setup for you in grafana, simply navigate to `Dashboards` -> `Manage` to view the available dashboards
## More deployment options
* How to deploy Fusion 5 in Google Kubernetes Engine
* How to deploy Fusion 5 in Amazon Elastic Kubernetes Service
* How to deploy Fusion 5 in Azure Kubernetes Service
## Learn more
The course for **Deploying Fusion 5** focuses on the prerequisite software needed to deploy Fusion, the necessary setup steps, and the physical act of deployment.
## Security
* Fusion services have been updated with the following versions:
* Log4J service version 2.17.1
* Tika 1.28
* nginix 1.21.5-alpine
## Bug Fixes
### Fusion
* Fixed an issue that caused problems with access and configuration of kubelet options.
* Fixed an issue that potentially allowed Read roles to create/delete Solr operations.
* Fixed an issue that prevented the logout option from being displayed if the "logout" button was clicked while creating or modifying an OIDC/SAML [security realm](/docs/5/fusion/operations/security/access-control/security-realms).
* Fixed an issue that caused the cursor to bypass the password field when using the tab button at login.
* Fixed an issue that prevented [App Insights](/docs/5/fusion/getting-data-out/data-analytics/app-insights/overview) from creating custom reports.
* Fixed an issue that caused invalid language codes to be displayed in the [**Detect Language**](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/language-detection-index-stage) index pipeline stage.
* Fixed an issue that caused the **Schedule a Job** to disappear after saving the job settings.
**How to schedule a job**
1. In the Fusion workspace, navigate to **Collections > Scheduler**.
2. Select the job from the job list.
3. Click **Run**.
4. Click **New Schedule**.
5. Select a job trigger:
* **After Another Job Completes**
Enter the job ID and job result that trigger this one.
* **Cron String**
Enter a Quartz cron expression, using UTC time unless `default.timezone` is set to another time zone in `fusion.cors` (`fusion.properties` in Fusion 4.x). See [the Quartz documentation](http://www.quartz-scheduler.org/api/2.3.0/org/quartz/CronExpression.html) for additional details.
* **Start + Interval**
Enter a start date/time, an interval, and the interval units. Include seconds in the start time.
### Predictive Merchandiser
* Fixed an issue that caused console and facet elements to stretch too far across the screen.
* Fixed an issue that made the **+ Add** button unavailable to associate products with **Merchandising Zero Results**.
1. Enter a search query the generates 0 results:
2. Click the **Add** button:
3. Select products or [product groups](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/product-grouping) that you want to associate with the 0 results query:
4. Click the **Insert** button. The selected products are now displayed when the search query is used, resolving the 0 results query:
* Fixed an issue that caused results to disregard sorting attempts.
* Fixed an issue that caused **Use Hierarchical Facets** values with child facets to not expand when clicked.
## Fusion
To configure hierarchical fields settings:
1. Access the application and click **Relevance > Rules > Templates**.
2. Click the three vertical dots to the right of the application name to display the Hierarchical fields settings section.
3. Complete one of the following:
* To add a new field, click **+ Add**. Enter values in the **Field** and **Delimiter** fields and click **Apply**.
Any symbol can be used as a delimiter.
* To edit an existing field, click **Field** and select the value to edit. Enter the appropriate value in the **Delimiter** field and click **Apply**.
* To delete a field setting, select the value in **Field**, click **X**, and then click **Apply**.
* To cancel any changes before you apply them, click **Cancel**.
* Fixed an issue that prevented facet ordering from working as expected.
* Fixed an issue that caused analytics chart cards to remain blank.
* Fixed an issue that caused facets with no values to produce an error in query logs.
### Connectors
* Various Connector improvements and bug fixes.
## Removals
The following features and functionalities are removed in Fusion 5.5.0:
* The SharePoint V1 connector.
* Usage of the `with` keyword syntax in the [JavaScript Query Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/javascript-query-stage) has been removed, as usage can result in random errors. See [MDN Web Docs](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/with) for more information.
* Child facet values have been removed from [Hierarchical Facet](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/hierarchical-faceting) breadcrumbs in favor of displaying facet values, to avoid confusion.
## Known Issues
**Important**
Users may experience one of the following issues with Predictive Merchandiser within the Fusion 5.5 release:
* The [**Rules**](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor) screen no longer loads in Predictive Merchandiser when importing an app after building from Fusion 5.5.
* After importing an app running an earlier version of Solr, newly-created facets do not appear in the **Use Predictive Merchandiser Facets**.
## Add a Facet
You can create rules that add facets by interacting directly with the Facet List in Interactive Mode. You can add facets that are set unconditionally, or facets that are only set when your customers run a particular query. For example, you might want to set a music genre facet when a customer navigates to the music section of your catalog.
Adding a facet automatically creates a new rule if a default facet rule does not already exist. If a default facet rule does exist, adding a facet updates that rule with the new facet.
### Enable the Facet Zone (Fusion 5.4 and later)
In Fusion 5.4 and later, facets appear in the zone on the left panel of the Predictive Merchandiser, once enabled. Follow the steps below to turn on the facets zone:
1. Click **Start Task**. The button appears.
2. In the left panel, click **+Add** and enter the desired parameters.
3. Click **Done** when you’ve finished.
You can also add facets that are not dependent on rules being triggered. To do this, edit the Facet Zone widget.
### Edit the Facet Zone widget (5.4 and later)
1. In the facet zone, click **+Add**. The **Edit Rule** dialog box appears.
2. Click the **Facet Field** field, and enter the desired facet parameter.
You can add multiple facet parameters by clicking the **+ Add** button:
3. Click the **Save** button to add your facet:
### Add Additional Facets
1. Hover your cursor above or below an existing facet. A **+** button appears:
2. Click the **+** button.
## Move a Facet
1. Hover your cursor over a facet. A menu overlay appears at the top.
2. Click-and-hold the drag bar:
3. Move the facet to the desired location.
4. Release the click.
## Remove a Facet
1. Hover your cursor over a facet. A menu overlay appears at the top.
2. Click the **Remove** button:
## Block a Facet Item
1. Hover your cursor over a facet item. A menu overlay appears.
2. Click the **Block** button:
## Bury a Facet Item
1. Hover your cursor over a facet item. A menu overlay appears.
2. Click the **Bury** button:
## Boost a Facet Item
1. Hover your cursor over a facet item. A menu overlay appears.
2. Click the **Boost** button:
### Reorder Boosted Facet Items
1. Hover over a boosted facet item. A drag bar appears.
2. Click and hold the drag bar:
3. Move the facet item to the desired location.
4. Release the click.
Workaround Steps:
In the `solrconfig.xml`, all cache classes that use `solr.LRUCache` must be changed to `solr.search.CaffeineCache`. See [Cache Implementations](https://solr.apache.org/guide/8_7/query-settings-in-solrconfig.html#cache-implementations) for more information.
* Changing the **Sort by** order of results is not reflected in the results list.
## Known issues
* New Kerberos security realms cannot be configured successfully in this version of Fusion.
# Fusion 5.5.1-1
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/more/5.5.1-1-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.5.1-1-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.5.1-1-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/hsibrb
**Release date:** March 29, 2023
**Component versions:**
| Component | Version |
| - | - |
| **Solr** | 8.11.1 |
| **ZooKeeper** | 3.6.2 |
| **Spark** | 2.4.5 |
| **Kubernetes** | GKE, AKS, EKS 1.24 Rancher (RKE) and OpenShift 4 compatible with Kubernetes 1.24 OpenStack and customized Kubernetes installs not supported. See [Kubernetes support](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support) for end of support dates. |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
**More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).**
A bug in the Netty Recycler component of fusion-spark can cause jobs to terminate early. To work around this issue, add the following parameters to the "Spark Settings" section:
```bash theme={"dark"}
spark.driver.extraJavaOptions=-Dorg.apache.pulsar.shade.io.netty.recycler.maxCapacityPerThread=0
spark.executor.extraJavaOptions=-Dorg.apache.pulsar.shade.io.netty.recycler.maxCapacityPerThread=0
```
For example:
This bug is fixed in [5.5.1-3](/docs/5/fusion/release-notes/more/5.5.1-3-release-notes)
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
Fusion 5.5.1-1 adds support for Kubernetes 1.24 to [Fusion 5.5.1](/docs/5/fusion/release-notes/more/5.5.1-release-notes).
## Known issues
* New Kerberos security realms cannot be configured successfully in this version of Fusion.
# Fusion 5.5.1-2
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/more/5.5.1-2-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.5.1-2-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.5.1-2-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/50cbsq
**Release date:** June 8, 2023
**Component versions:**
| Component | Version |
| - | - |
| **Solr** | 8.11.1 |
| **ZooKeeper** | 3.6.2 |
| **Spark** | 2.4.5 |
| **Kubernetes** | GKE, AKS, EKS 1.25 and 1.26 Rancher (RKE) and OpenShift 4 compatible with Kubernetes 1.25 and 1.26 OpenStack and customized Kubernetes installs not supported. See [Kubernetes support](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support) for end of support dates. |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
**More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).**
A bug in the Netty Recycler component of fusion-spark can cause jobs to terminate early. To work around this issue, add the following parameters to the "Spark Settings" section:
```bash theme={"dark"}
spark.driver.extraJavaOptions=-Dorg.apache.pulsar.shade.io.netty.recycler.maxCapacityPerThread=0
spark.executor.extraJavaOptions=-Dorg.apache.pulsar.shade.io.netty.recycler.maxCapacityPerThread=0
```
For example:
This bug is fixed in [5.5.1-3](/docs/5/fusion/release-notes/more/5.5.1-3-release-notes)
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
## New features
* Fusion 5.5.1-2 adds support for Kubernetes 1.25 and 1.26 to [Fusion 5.5.1](/docs/5/fusion/release-notes/more/5.5.1-release-notes). Support for GKE is limited to Kubernetes 1.25, due to the bug described above.
## Bug fixes
* Fixed a bug with the Tika parser that prevented Fusion from parsing Word documents added to the Blob Store.
## Known issues
* New Kerberos security realms cannot be configured successfully in this version of Fusion.
# Fusion 5.5.1-3
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/more/5.5.1-3-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.5.1-3-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.5.1-3-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/zgxf2l
**Release date:** July 26, 2023
**Component versions:**
| Component | Version |
| - | - |
| **Solr** | 8.11.1 |
| **ZooKeeper** | 3.6.2 |
| **Spark** | 2.4.5 |
| **Kubernetes** | GKE, AKS, EKS 1.25 and 1.26 Rancher (RKE) and OpenShift 4 compatible with Kubernetes 1.25 and 1.26 OpenStack and customized Kubernetes installs not supported. See [Kubernetes support](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support) for end of support dates. |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
**More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).**
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
## Bug fixes
* Fixed an issue where Spark jobs failed when logging was used. You no longer need to use the workaround documented in the [5.5.1-1](/docs/5/fusion/release-notes/more/5.5.1-1-release-notes) and [5.5.1-2](/docs/5/fusion/release-notes/more/5.5.1-2-release-notes) release notes.
* Fixed an issue that prevented QnA Smart Answers jobs from finishing.
* Fixed an issue that prevented deploying Seldon models.
* Fixed an issue that prevented V2 connectors from running index pipelines configured with a Milvus stage.
## Known issues
* New Kerberos security realms cannot be configured successfully in this version of Fusion.
# Fusion 5.5.1
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/more/5.5.1-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.5.1-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.5.1-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/drg5sz
**Release date:** April 13, 2022
**Component versions:**
| Component | Version |
| - | - |
| **Solr** | 8.11.1 |
| **ZooKeeper** | 3.6.2 |
| **Spark** | 2.4.5 |
| **Kubernetes** | GKE, AKS, EKS 1.22 Rancher (RKE) and OpenShift 4 compatible with Kubernetes 1.22 OpenStack and customized Kubernetes installs not supported. See [Kubernetes support](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support) for end of support dates. |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
## New features
* Merchandisers can now curate the metadata content of their pages with the **Page Metadata** feature in Predictive Merchandiser. See [Page Metadata](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/page-metadata) for more information.
## Improvements
### Fusion
* Added titles for Superset and Superset Proxy in the Fusion API **Microservices** dropdown menu.
* Improved logging from client-side retries, to make debugging `retry` jobs easier.
* Streamlined the **System** to remove redundant entries.
### Predictive Merchandiser
* A script has been created to address the [`solrconfig.xml` workaround](/docs/5/fusion/release-notes/more/5.5.0-release-notes) listed in the Fusion 5.5.0 release notes, where users were directed to change their cache classes to `solr.search.CaffeineCache`. See the [Fusion 5 Migration Guide](https://github.com/lucidworks/fusion-cloud-native/tree/master/migrations#migration-from-fusion-5-4-x-to-5-5-x) for more information.
## Bug Fixes
### Fusion
* Fixed an issue that caused the [Tika Parser](/docs/5/fusion/reference/config-ref/parser-stages/apache-tika-parser) to generate "out of memory" errors.
### Predictive Merchandiser
* Fixed an issue that caused images to not display in Predictive Merchandiser results and product details.
* Fixed an issue that caused the interface of the Create a facet console to display incorrectly after adding a new facet field.
* Fixed several UI issues that caused the Publish Rules dropdown to display incorrectly.
## Known issues
* New Kerberos security realms cannot be configured successfully in this version of Fusion.
# Fusion 5.5.2
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/more/5.5.2-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.5.2-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.5.2-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/k63wv1
**Release date:** May 31, 2022
**Component versions:**
| Component | Version |
| - | - |
| **Solr** | 8.11.1 |
| **ZooKeeper** | 3.6.2 |
| **Spark** | 3.2.1 |
| **Kubernetes** | GKE, AKS, EKS 1.22 Rancher (RKE) and OpenShift 4 compatible with Kubernetes 1.22 OpenStack and customized Kubernetes installs not supported. See [Kubernetes support](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support) for end of support dates. |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
## Improvements
* The Spark version has been updated to 3.2.1. See [Spark Overview](https://spark.apache.org/docs/latest/) for more information.
## Bug Fixes
* Fixed an issue that caused the [**click\_signals\_aggregation**](/docs/5/fusion/reference/config-ref/jobs/aggregations/overview) job to fail with a "Driver JVM" error.
* Fixed an issue that caused containers to crash when an Argo-based job failed to list the job status.
## Known Issues
* A bug was discovered in the Spark 3.2.x component that prevents signal aggregation for signals stored in the cloud in Parquet format. For more information about the issue, see the [Spark bug report](https://issues.apache.org/jira/browse/SPARK-38318).
This issue is fixed in Fusion 5.7.1.
## Removals
The following features and functionalities are removed from default deployment in Fusion 5.5.2. See [Deprecations and Removals](/docs/5/fusion/deprecations-and-removals) for more information about suggested replacements:
* Jupyter
* Superset
## Known issues
* New Kerberos security realms cannot be configured successfully in this version of Fusion.
# Fusion 5.6.0
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/more/5.6.0-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.6.0-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.6.0-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/mme9fq
**Release date:** August 10, 2022
This release removes support for Pulsar. Any existing workflows using Pulsar will no longer function when upgrading to 5.6.
**Component versions:**
| Component | Version |
| - | - |
| **Solr** | 8.11.1 |
| **ZooKeeper** | 3.7.1 |
| **Spark** | 3.2.1 |
| **Kubernetes** | GKE, AKS, EKS 1.22 Rancher (RKE) and OpenShift 4 compatible with Kubernetes 1.22 OpenStack and customized Kubernetes installs not supported. See [Kubernetes support](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support) for end of support dates. |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
**More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).**
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
## New Features
### Predictive Merchandiser
Predictive Merchandiser users can now fire templates and rules at every facet level.
Hierarchical template triggers allow you to fire a search configuration template or rules at a parent or child level in a hierarchy, and even at a faceted level.
**Ways to use hierarchical templates:**
* For a sale on "Winter Coats," you can fire a template at the "Outerwear" level or at any of its children.
* For an overstock on a particular clothing size, you could fire a template directly at the “Women’s Outerwear, Size Small” filter.
* When rolling out a new company policy, you could fire a zone on all pages from the homepage down so everyone sees the most important links and documents for the new policy.
See [Predictive Merchandiser Templates](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/templates) for more information. For instructions on how to use Templates, see **Use Predictive Merchandiser Templates**.
This article teaches you how to use the [Templates](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/templates) feature in Predictive Merchandiser, including how to:
* [Create a template](#create-a-new-template)
* [Add zones](#add-a-zone-to-a-template) to the template
* Conditionally [trigger the template](#set-triggering-options)
* [Test](#test-the-template) the template in the Predictive Merchandiser screen
In the instructions below, a new template is created with two zones for use during a Cyber Monday sales event. One zone is used to display general search results while the other is used to promote specific products for the sale.
The zones used in this article use different query profiles. To follow the instructions as described, you will need to begin with an app that has at least two query profiles.
To begin, access Templates in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor) by navigating to **Relevance** > **Rules** > **Templates**.
The course for **Predictive Merchandiser: Template Manager** focuses on how to use the templates included in Predictive Merchandiser and create your own custom templates.
## Create a new template
1. From the Templates screen, click the **New Template** button:
2. Assign the template a unique name in the **Template Name** field. In this example, the value `Cyber Monday` is assigned.
3. Choose a **Template Type**. This example uses the `landing` template type.
This value is read-only and for informational purposes only.
4. Click the **Save** button to save the new template:
For full configuration, see:
* [Templates UI Configuration Options](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/templates-config)
* [Templating API Configuration Reference](/api-reference/templates-controller/get-template-information)
Images may display other releases if the information between releases does not differ. For example, the Fusion Templating API image reflects version 5.3.5 in both the Fusion 5.2 and 5.3 releases, but reflects 5.4.0 in the Fusion 5.4 release.
## Add a zone to a template
Zones enable Fusion users to create rules designed for specific query profiles. Combined with templates, this allows a search application to display results from multiple query profiles in the same view.
In this example, two new zones are created for the **Cyber Monday** template.
1. Add an existing zone to the template by clicking the **Existing** button. In this example, the **Main Results List** zone is added by selecting the zone and clicking the **Apply** button.
2. Create a new zone by clicking the **New** button:
3. Assign the zone a unique name in the **Display Name** field. For this example, the value `Promoted` is assigned.
4. Choose the **Query Profile** for this zone. This example uses a query profile named **promotional-results**. This query profile is configured with rules that pin products to the top of the results.
5. Choose the **Zone Type** for this zone. This example uses **main-results-list**.
This value is read-only and for informational purposes only.
6. Set the configuration options for the product image, title, and description per your preference.
7. Click the **Save** button to save the new zone:
For full zone configuration details, see [Zone Configuration Options](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/templates-config).
## Edit an existing zone
You can edit a zone that was already added to a template:
1. Select the **Start Task** button.
2. Select the pencil icon to begin editing the chosen page.
3. Each zone has a trash can icon to delete the zone. Select the wrench icon to edit the zone.
4. In the form that appears, make any necessary edits.
5. Click the **Save** button to save your changes.
## Set triggering options
Triggers describe conditions that must be met for a template to display. Trigger options include a set date range and specific search terms. See [Triggering Configuration Options](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/templates-config) for additional details.
### Date range
1. Click the **0 Active** button under **Triggering**:
If there are active triggers, this button changes to reflect that number. For example, this button will read **3 Active** when 3 triggers are active.
2. Click the date range selector and select a start and end date. For this example, choose a date range that includes today’s date.
3. Click the **Save** button to save your changes:
### Search terms
1. Click the **0 Active** button under **Triggering**.
2. Click the **Trigger** button:
3. In the **Search Terms** field, enter `cyber monday`.
4. Set the **Match Mode** field to `phrase`.
5. Click the **Add** button.
6. Repeat the process to add a second trigger with **Search Terms** set to `sales event` and **Match Mode** to `phrase`.
7. Click the **Save** button to save your changes:
## Test the template
1. Navigate to the **Merchandiser** screen.
2. Enter one of the [search terms](#search-terms) you configured in the **Set triggering options** instructions above. This example used the search terms `cyber monday` and `sales event`.
If today’s date does not fall within the template’s [date range trigger](#date-range), the template will *not* load.
If everything was configured correctly, you should see the **Cyber Monday** template is loaded. You can now create different rules for both zones, including pinning promoted products in the **Promoted** zone.
### Fusion
* Users can now manage API keys from their browser. See [API Keys](/docs/5/fusion/operations/security/access-control/api-keys) for more information.
* Added a new query pipeline stage, [Graph Security Trimming Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/security-trimming-graph-query-stage).\
The Graph Security Trimming stage restricts query results according to the user ID as an alternative to [Security Trimming Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/security-trimming-query-stage). Whereas the Security Trimming stage has one Solr filter query per data source, Graph Security Trimming uses a single filter query for all data sources.
### Fusion Connectors
* Added a new connector, the [SharePoint Optimized V2 Connector](/docs/fusion-connectors/connectors/sharepoint-v2-optimized). The SharePoint Optimized V2 connector retrieves content and metadata from an on-premises SharePoint repository. This connector replaces the SharePoint V2 Connector in Fusion 5.6.0.
* Added a new connector, the [LDAP-ACLs Connector](/docs/fusion-connectors/connectors/ad-acl-ldap). This connector replaces the LDAP V2 Connector.
## Bug Fixes
### Fusion
* The Jobs panel no longer refreshes constantly while a job is running.
* The `fieldList` property in Index stages now displays the designated title and description.
* Fixed an issue that caused [Spark jobs](/docs/5/fusion/operations/jobs-and-scheduling/spark-jobs) to fail when Fusion 5 was deployed using Gatekeeper service on Kubernetes.
* Configured [Realms](/docs/5/fusion/operations/security/access-control/security-realms) now maintain sign in information.
* In App Import, the **Import** button is no longer inactive if the wrong file type is selected.
* Fixed an issue that caused the [Job History API](/docs/5/fusion/operations/jobs-and-scheduling/job-history) to display inaccurate job status information.
* Fixed an issue that caused stage information to be deleted after moving them in the pipeline and saving.
* The `f5-monitoring-kube-state-metrics` service tooltip now displays.
* Fixed an issue that caused jobs to not run after adding a new schedule.
* Fixed an issue that caused an internal server error when creating a new app using numbers as an app name.
* Attempting to access the Rules page with a developer role no longer causes a forbidden error.
* Using the pipe symbol as a Field Delimiter in the Apply Rules query pipeline stage no longer causes a forbidden error.
* Fixed an issue that caused the timezone value to not save.
### Predictive Merchandiser
* Fixed an issue that caused published templates to not display when clicking Start Task on the Merchandiser page.
## Removals
* Removed **Selection of Solr Cluster** on the **Create New Collection** screen in the Admin UI.
* Removed support for Apache Pulsar and replaced it with Apache Kafka. As a result:
* Logging via Pulsar has been removed.
* The Log Viewer has been removed from the Admin UI.
* Remote connector support via Pulsar is no longer possible. This functionality will be replaced in a future release.
* The ability to access Pulsar from the Subscriptions API has been removed. You are no longer able to create a custom subscription topic.
* The Send to Message Bus index stage has been removed.
* [LDAP-ACLs V2 Connector](/docs/fusion-connectors/connectors/ad-acl-ldap) replaces the [LDAP V2 Connector](/docs/fusion-connectors/connectors/ad-acl-v2).
* [SharePoint Optimized V2 Connector](/docs/fusion-connectors/connectors/sharepoint-v2-optimized) replaces the [SharePoint V2 Connector](/docs/fusion-connectors/connectors/sharepoint-v2).
## Known issues
* New Kerberos security realms cannot be configured successfully in this version of Fusion.
* A bug was discovered in the Spark 3.2.x component that prevents signal aggregation for signals stored in the cloud in Parquet format. For more information about the issue, see the [Spark bug report](https://issues.apache.org/jira/browse/SPARK-38318).
This issue is fixed in Fusion 5.7.1.
* The link to the [Banana dashboards](/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/overview) in the Fusion UI does not work. You can access Banana dashboards using a direct link at `https://FUSION_HOST:FUSION_PORT/admin/banana/index.html#/dashboard/`.
This issue is fixed in Fusion 5.7.0.
# Fusion 5.6.1
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/more/5.6.1-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.6.1-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.6.1-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/5hi7ax
**Release date:** September 20, 2022
**Important**
Fusion 5.6.1 is available only to Lucidworks Search customers.
**Component versions:**
| Component | Version |
| - | - |
| **Solr** | 8.11.1 |
| **ZooKeeper** | 3.7.1 |
| **Spark** | 3.2.1 |
| **Kubernetes** | GKE, AKS, EKS 1.23 Rancher (RKE) and OpenShift 4 compatible with Kubernetes 1.23 OpenStack and customized Kubernetes installs not supported. See [Kubernetes support](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support) for end of support dates. |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
**More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).**
## New Features
### Fusion Connectors
* Fusion 5.6.1 adds a new connector, the [Web V2 Connector](/docs/fusion-connectors/connectors/web-v2). This brings our Web connector from the classic framework to our V2 framework.\
The Web V2 connector retrieves data from a Web site using HTTP and starting from a specified URL.
* As part of our transition from Pulsar to Kafka, 5.6.1 returns the ability to **Remotely Launch a Plugin**.
Follow this configuration to use a remote connector in Fusion.
A known issue is after idling for five minutes without any new connector start requests, the remote connector stops responding to server requests and must restart. This causes connector start requests to hang. A workaround is to schedule a connector to run every three minutes. It is recommended to use a connector instance that doesn’t fetch any documents to keep the remote connector functional without frequent restarts.
1. Execute the following command.
```
java -jar connector-plugin-client-standalone.jar [YAML_CONFIG_FILE]
```
2. Check that the `YAML_CONFIG_FILE` has the following format.
```yaml theme={"dark"}
kafka-bridge:
target: mynamespace-connectors-backend.lucidworkstest.com:443 # mandatory
plain-text: false # optional
proxy-server: # optional - needed when a forward proxy server is used to provide outbound access to the standalone connector
host: host
port: 111
user: user
password: password
trust: # optional - needed when the client’s system doesn’t trust the server’s certificate
cert-collection-filepath: path1
proxy: # mandatory fusion-proxy
user: admin
password: password123
url: https://fusiontest.com/ # needed only when the connector plugin requires blob store access
plugin: # mandatory
path: ./fs.zip
type:
suffix: remote
```
3. Use this table for descriptions of the YAML properties.
| Key | Value description |
| - | - |
| `kafka-bridge.target` | The Kafka-gRPC bridge host:port pair. |
| `kafka-bridge.plain-text` | Enable when TLS ingress is not used. |
| `kafka-bridge.proxy-server.host` | The Forward Proxy Server Host. |
| `kafka-bridge.proxy-server.port` | The Forward Proxy Server Port. |
| `kafka-bridge.proxy-server.user` | The Forward Proxy Server User. |
| `kafka-bridge.proxy-server.password` | The Forward Proxy Server Password. |
| `kafka-bridge.trust.cert-collection-filepath` | The Private Key file path. |
| `proxy.url` | Fusion proxy url. |
| `proxy.user` | Fusion proxy user. |
| `proxy.password` | Fusion proxy password. |
| `plugin.path` | The path of the plugin zip file. |
| `plugin.type` | Optional suffix added to the plugin `id`. For example, if the plugin `id` is `lucidworks.test` and the suffix is `remote`, the plugin will be launched with `lucidworks.test.remote`. |
Example `YAML_CONFIG_FILE`:
```yaml theme={"dark"}
kafka-bridge:
target: mynamespace-connectors.example.com:443
plain-text: false
proxy:
user: admin
password: "password123"
plugin:
path: ./testplugin.zip
```
## Bug Fixes
### Fusion
* The [blocked rule form](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/block-documents-query-stage) now validates the "Parameter Name" when closing the modal. This fixes an error where blocked parameters in the Query Workbench could not be deleted via the UI.
* User with read-only access no longer see the **Quickstart** splash page, as they are unable to use it to create applications or other Fusion objects.
The Quickstart is a wizard that lets you explore some of Fusion’s core capabilities:
* Creating or selecting an app
* Selecting a collection (where data is stored)
* Indexing data
* Searching
The course for **Using The Quickstart Wizard** focuses on using the wizard to practice using the key functions of Fusion.
## Step through the Quickstart wizard
If it is not already open, open the Quickstart wizard. In the Fusion launcher, click **New here? Get started...**.
1. Click **Continue**.
On the **Select an App** screen, you can select or create an [app](/docs/4/fusion-server/concepts/overview) for your quickstart data.
If you click **Create new app**, Fusion prompts you to enter an app name (and optionally a description), then click **Create App** to return to the Quickstart. Your new app is selected by default.
2. After selecting an app, click **Continue**.
On the **Select a Collection** screen, you can select or create a [collection](/docs/4/fusion-server/concepts/overview) for your quickstart data. A collection with the same name as your app is created automatically:
3. Click **Continue**.\
On the **Index Data** screen, you can either select one of the built-in sample datasets or click **Use my data** to upload your own:
4. Click **Continue**.\
On the **Query Data** screen, you can see all search results and enter your own search queries to test the indexed dataset. You can also select the display fields or view the parsed documents:
5. Click **Continue**.
From here, you can do the following:
* Open the Index Workbench to change the index pipeline.
* [Fusion 4.x Index Workbench](/docs/4/fusion-server/concepts/indexing/datasources/index-workbench)
* [Fusion 5.x Index Workbench](/docs/5/fusion/intro/ui-tour/index-workbench)
* Open the Query Workbench to change the query pipeline.
* [Fusion 4.x Query Workbench](/docs/4/fusion-server/concepts/querying/query-workbench)
* [Fusion 5.x Query Workbench](/docs/5/fusion/getting-data-out/query-basics/query-workbench)
The workbenches are essentials tools in the Fusion workflow.
* Open [App Studio](/docs/4/app-studio/overview) to create a user interface for searching this collection.
App Studio is only available in Fusion 4.1 and 4.2.
### Fusion Connectors
* The [AWS S3 V2 Connector](/docs/fusion-connectors/connectors/amazon-aws-s3-pro) connector is now able to crawl non-public buckets when the object key is not specified in the configuration.
### Predictive Merchandiser
* Rules page now displays correctly for users with only the [developer role](/docs/5/fusion/operations/security/access-control/roles).
## Removals
Fusion SQL was removed as of Fusion 5.6.1. It is available for Fusion releases 5.5 and earlier.
## Known issues
* New Kerberos security realms cannot be configured successfully in this version of Fusion.
* A bug was discovered in the Spark 3.2.x component that prevents signal aggregation for signals stored in the cloud in Parquet format. For more information about the issue, see the [Spark bug report](https://issues.apache.org/jira/browse/SPARK-38318).
This issue is fixed in Fusion 5.7.1.
* The link to the [Banana dashboards](/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/overview) in the Fusion UI does not work. You can access Banana dashboards using a direct link at `https://FUSION_HOST:FUSION_PORT/admin/banana/index.html#/dashboard/`.
This issue is fixed in Fusion 5.7.0.
# Fusion 5.7.0
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/more/5.7.0-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.7.0-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.7.0-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/3c6x8l
**Release date:** December 13, 2022
**Component versions:**
| Component | Version |
| - | - |
| **Solr** | fusion-solr 8.11.1 *(based on Solr 8.11.1)* |
| **ZooKeeper** | 3.7.1 |
| **Spark** | 3.2.2 |
| **Kubernetes** | GKE, AKS, EKS 1.24 Rancher (RKE) and OpenShift 4 compatible with Kubernetes 1.24 OpenStack and customized Kubernetes installs not supported. See [Kubernetes support](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support) for end of support dates. |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
**More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).**
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
## Improvements
### Fusion
* Fusion now supports Kubernetes 1.24. This applies to GKE, AKS, and EKS. It also applies to Rancher (RKE) and OpenShift 4 versions that are compatible with Kubernetes 1.24.
* Fusion now uses a custom Solr image called fusion-solr. This version of Solr is based on Solr 8.11.1 and adds functionality to support Fusion features.
* Upgraded Fusion to use Tika 2.4.1.
* Disabled browser autofill in Fusion data source configuration forms to prevent accidentally filling fields incorrectly.
### Fusion Connectors
* The recrawl process is updated for the LDAP ACLs V2 connector. For more information, see [LDAP ACLs V2 connector incremental crawls](/docs/fusion-connectors/connectors/ad-acl-ldap).
* As part of the transition from Pulsar to Kafka, the ability to **Remotely Launch a Plugin** is returned.
Follow this configuration to use a remote connector in Fusion.
A known issue is after idling for five minutes without any new connector start requests, the remote connector stops responding to server requests and must restart. This causes connector start requests to hang. A workaround is to schedule a connector to run every three minutes. It is recommended to use a connector instance that doesn’t fetch any documents to keep the remote connector functional without frequent restarts.
1. Execute the following command.
```
java -jar connector-plugin-client-standalone.jar [YAML_CONFIG_FILE]
```
2. Check that the `YAML_CONFIG_FILE` has the following format.
```yaml theme={"dark"}
kafka-bridge:
target: mynamespace-connectors-backend.lucidworkstest.com:443 # mandatory
plain-text: false # optional
proxy-server: # optional - needed when a forward proxy server is used to provide outbound access to the standalone connector
host: host
port: 111
user: user
password: password
trust: # optional - needed when the client’s system doesn’t trust the server’s certificate
cert-collection-filepath: path1
proxy: # mandatory fusion-proxy
user: admin
password: password123
url: https://fusiontest.com/ # needed only when the connector plugin requires blob store access
plugin: # mandatory
path: ./fs.zip
type:
suffix: remote
```
3. Use this table for descriptions of the YAML properties.
| Key | Value description |
| - | - |
| `kafka-bridge.target` | The Kafka-gRPC bridge host:port pair. |
| `kafka-bridge.plain-text` | Enable when TLS ingress is not used. |
| `kafka-bridge.proxy-server.host` | The Forward Proxy Server Host. |
| `kafka-bridge.proxy-server.port` | The Forward Proxy Server Port. |
| `kafka-bridge.proxy-server.user` | The Forward Proxy Server User. |
| `kafka-bridge.proxy-server.password` | The Forward Proxy Server Password. |
| `kafka-bridge.trust.cert-collection-filepath` | The Private Key file path. |
| `proxy.url` | Fusion proxy url. |
| `proxy.user` | Fusion proxy user. |
| `proxy.password` | Fusion proxy password. |
| `plugin.path` | The path of the plugin zip file. |
| `plugin.type` | Optional suffix added to the plugin `id`. For example, if the plugin `id` is `lucidworks.test` and the suffix is `remote`, the plugin will be launched with `lucidworks.test.remote`. |
Example `YAML_CONFIG_FILE`:
```yaml theme={"dark"}
kafka-bridge:
target: mynamespace-connectors.example.com:443
plain-text: false
proxy:
user: admin
password: "password123"
plugin:
path: ./testplugin.zip
```
* A new connector, the [Web V2 Connector](/docs/fusion-connectors/connectors/web-v2), is added. This brings our Web connector from the classic framework to our V2 framework.
* The [Kaltura V2 Connector](/docs/fusion-connectors/connectors/kaltura) is now available.
### Predictive Merchandiser
* Improved UI elements in the Predictive Merchandiser.
## Bug Fixes
### Fusion
* Fixed an issue where pointing a Fusion collection at a Solr collection with a different name breaks queries to `solrServer` in Javascript stages.
* Fixed an issue when adding a new Graph security trimming stage; now you can enter a comma-separated list of datasources in **Exclude Data Source ID(s)** and **Inclusive Data Source ID(s)** fields with spaces. For example, `ds1, ds2, ds3`.
* Fixed an issue where documentation links pointed to Fusion 5.4 version, instead of the currently used version of Fusion.
### Fusion Connectors
* The [LDAP ACLs V2 connector](/docs/fusion-connectors/connectors/ad-acl-ldap) is updated to reduce crawl bugs, prevent a false error from reporting when the crawl is stopped, and ensure permissions settings are kept in sync.
* Fixed an issue with network policies that prevented the SharePoint Optimized V2 connector plugin from indexing some data.
* Fixed an issue with the ServiceNow connector where empty fields would cause an error and break the crawl.
### Predictive Merchandiser
* Made the text box size larger to make it easier to enter multiple Boost Queries in the Rules Manager.
## Known issues
* New Kerberos security realms cannot be configured successfully in this version of Fusion.
* A bug was discovered in the Spark 3.2.x component that prevents signal aggregation for signals stored in the cloud in Parquet format. For more information about the issue, see the [Spark bug report](https://issues.apache.org/jira/browse/SPARK-38318).
This issue is fixed in Fusion 5.7.1.
## Removals
* Removed deprecated NLP Annotator stage from Query Workbench and OpenNLP NER stage from Index Workbench.
# Fusion 5.7.1
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/more/5.7.1-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.7.1-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.7.1-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/dvb7jr
**Release date:** January 30, 2023
**Component versions:**
| Component | Version |
| - | - |
| **Solr** | fusion-solr 8.11.1 *(based on Solr 8.11.1)* |
| **ZooKeeper** | 3.7.1 |
| **Spark** | 3.2.2 |
| **Kubernetes** | GKE, AKS, EKS 1.24 Rancher (RKE) and OpenShift 4 compatible with Kubernetes 1.24 OpenStack and customized Kubernetes installs not supported. See [Kubernetes support](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support) for end of support dates. |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
## Improvements
* Remote connector jobs now have the ability to recover from temporary network interruptions.
## Bug Fixes
* Fixed a bug that caused errors with the Argo workflow due to an issue in the Helm chart.
* Fixed a bug that can prevent the Fusion indexing microservices from running with Transport Layer Security (TLS) enabled. Other microservices, which rely on the indexing microservice, were subsequently prevented from running.
* Fixed a bug with [Spark jobs](/docs/5/fusion/operations/jobs-and-scheduling/spark-jobs) that prevented the deletion of the temporary `job-impl.jar` file. This bug resulted in excessive pod disk space usage. Now, the directory containing the file is deleted when the job finishes.
* Fixed a bug with the Smart Answers Supervised Training job that prevented a trained model from deploying.
* Fixed a bug in the Spark 3.2.x component that prevented signal aggregation for signals stored in the cloud in Parquet format. For more information about the issue, see the [Spark bug report](https://issues.apache.org/jira/browse/SPARK-38318).
## Known issues
* New Kerberos security realms cannot be configured successfully in this version of Fusion.
# Fusion 5.8.0
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/more/5.8.0-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/5.8.0-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/5.8.0-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/2k5m8y
**Release date:** March 22, 2023
**Component versions:**
| Component | Version |
| - | - |
| **Solr** | fusion-solr 5.8.0 *(based on Solr 9.1.1)* |
| **ZooKeeper** | 3.7.1 |
| **Spark** | 3.2.2 |
| **Kubernetes** | GKE, AKS, EKS 1.24 Rancher (RKE) and OpenShift 4 compatible with Kubernetes 1.24 OpenStack and customized Kubernetes installs not supported. See [Kubernetes support](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support) for end of support dates. |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
**More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).**
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
Rosette Entity Extractor (REX) or Rosette Base Linguistics (RBL) for the **Use Advanced Linguistics with Babel Street** are not compatible with Solr 9 included in this version of Fusion. If you rely on the Babel Street language module, do not upgrade until this compatibility issue is resolved.
## New Features
* Lucidworks Search only: Added a new feature called Dynamic Pricing which improves scalability for custom pricing. This feature lets B2B organizations with large product and pricing inventories sort, facet, boost, and filter on custom prices and entitlements.
The course for **Dynamic Pricing** focuses on how Dynamic Pricing maximizes custom pricing strategies.
* Lucidworks Search only: Fusion now supports Reverse Search, which lets you set up monitoring queries that automatically include new documents. Instead of running a query multiple times to see if new documents have been added, this feature matches incoming documents to existing relevant queries, improving content awareness and productivity.
* Tika Asynchronous Parsing improves document crawl speeds and prevents memory and stability issues during connector processes. You can **Use Tika Asynchronous Parsing** to separate document crawling from document parsing, which is useful for large sets of complex documents. For more information, see [Asynchronous Tika Parsing](/docs/5/fusion/getting-data-in/indexing/asynchronous-tika-parsing).
This document describes how to set up your application to use Tika asynchronous parsing.
Unlike synchronous Tika parsing, which uses a parser stage, asynchronous Tika parsing is configured in the datasource and index pipeline. For more information, see [Asynchronous Tika Parsing](/docs/5/fusion/getting-data-in/indexing/asynchronous-tika-parsing).
**Field names change with asynchronous Tika parsing.**
In contrast to synchronous parsing, asynchronous Tika parsing prepends `parser_` to fields added to a document. System fields, which start with `\_lw_`, are not prepended with `parser_`. If you are migrating to asynchronous Tika parsing, and your search application configuration relies on specific field names, update your search application to use the new fields.
## Configure the connectors datasource
1. Navigate to your datasource.
2. Enable the **Advanced** view.
3. Enable the **Async Parsing** option.
**Fusion 5.9.11 and later uses your parser configuration when using asynchronous parsing.**
The asynchronous parsing service performs Tika parsing using Apache Tika Server. In Fusion 5.8 through 5.9.10, other parsers, such as HTML and JSON, are not supported by the asynchronous parsing service. By enabling asynchronous parsing, the parser configuration linked to your datasource is ignored. In Fusion 5.9.11 and later, other parsers, such as HTML and JSON, are supported by the asynchronous parsing service. By enabling asynchronous parsing, the parser configuration linked to your datasource is used.
4. Save the datasource configuration.
## Configure the parser stage
You must do this step in Fusion 5.9.11 and later.
1. Navigate to **Parsers**.
2. Select the parser, or create a new parser.
3. From the **Add a parser stage** menu, select **Apache Tika Container Parser**.
4. (Optional) Enter a label for this stage. This label changes the names from Apache Tika Container Parser to the value you enter in this field.
5. If the Apache Tika Container Parser stage is not already the first stage, drag and drop the stage to the top of the stage list so it is the first stage that runs.
## Configure the index pipeline
1. Go to the **Index Pipeline** screen.
2. Add the **Solr Partial Update Indexer** stage.
3. Turn off the **Reject Update if Solr Document is not Present** option and turn on the **Process All Pipeline Doc Fields** option:
4. Include an extra update field in the stage configuration using any update type and field name. In this example, an incremental field `docs_counter_i` with an increment value of `1` is added:
5. Enable the **Allow reserved fields** option:
6. Click **Save**.
7. Turn off or remove the **Solr Indexer stage**, and move the **Solr Partial Update Indexer stage** to be the last stage in the pipeline.
Asynchronous Tika parsing setup is now complete. Run the datasource indexing job and monitor the results.
The Apache Tika and Forked Tika stages are now deprecated. Follow the migration steps to begin using asynchronous parsing.
## Improvements
* Improved recoverability for [on-prem connectors](/docs/fusion-connectors/developers/remote-v2-connectors) in high network traffic environments.
* Fusion’s custom Solr image has been updated to fusion-solr 5.8.0. This upgrade includes the [benefits and new features in Solr 9](https://solr.apache.org/guide/solr/latest/upgrade-notes/major-changes-in-solr-9.html), while also including custom plugins to support Dynamic Pricing, Reverse Search, and autoscaling.
* Developed new authentication methods for the [MongoDB connector](/docs/fusion-connectors/connectors/v1/mongodb).
## Bug Fixes
### Fusion
* Fixed a bug where the indexing service failed to load some classes from some JDBC drivers.
* Updated the Helm charts used when deploying Prometheus, Grafana, Loki, and Promtail for monitoring.
* Fixed an error with permissions required for the [Upload Model Parameters To Cloud](/docs/5/fusion/reference/config-ref/jobs/upload-model-parameters) job.
* [Graph Security Trimming](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/security-trimming-graph-query-stage) stage now works when collections have multiple shards and replicas.
* Fixed a bug where having the same document updated twice in the same job could cause the job to hang.
* Fixed an issue where the [Solr API](/api-reference/solr/get-solr-service-status) was unable to pass through raw requests using the proxy.
* Updated the query pipeline and indexing container base images to use Java 11 so they are more secure.
* Removed UI link to view logs dashboard as its target screen is no longer available.
* Fixed a UI bug where zone display fields could not be manually removed.
* Fusion panel text editors can now scroll as expected in Firefox.
### Predictive Merchandiser
* Fixed a bug in [Predictive Merchandiser](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/overview) where templates having a higher precedence using a specified trigger phrase and facet were not appearing when that phrase was searched with that facet selected.
## Deprecations
* [Field Parser Index Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/field-parser-index-stage) is no longer used by Fusion connectors. It is officially deprecated in this release and will be removed entirely in a later release.
* Streaming documents to the `/index` and `/reindex` endpoints of the [Index Pipelines API](/api-reference/index-pipelines-api/get-the-service-status)is deprecated and will eventually stop working altogether in the continuing switch to asynchronous parsing.
* [Tika Server Parser](/docs/5/fusion/reference/config-ref/parser-stages/apache-tika-server) is deprecated and will be replaced by Tika Asynchronous Parser.
* [Apache Tika Parser](/docs/5/fusion/reference/config-ref/parser-stages/apache-tika-parser) stage is deprecated and will be removed in a later release.
* The [Forked Apache Tika Parser](/docs/5/fusion/reference/config-ref/parser-stages/forked-apache-tika-parser) stage is deprecated and will be removed completely in a later release.
## Known issues
* New Kerberos security realms cannot be configured successfully in this version of Fusion.
* When using the JavaScript query stage to query Solr, you have to provide parameters including `rows`. Previously `rows` accepted an integer, but now it must be entered as a string as in `("rows", "1")`.
```js wrap theme={"dark"}
map.put("q", 'okta_uid_s:"{identifier}" OR username_s:"{identifier}" OR email_s:"{identifier}"'
.replace(/{identifier}/ig, userIdentifier))
map.put("rows", "1")
```
# Fusion 5.8.1
Source: https://doc.lucidworks.com/docs/5/fusion/release-notes/more/5.8.1-release-notes
[localhost link]: http://localhost:3000/docs/5/fusion/release-notes/more/5.8.1-release-notes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/release-notes/more/5.8.1-release-notes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/608v1q
**Release date:** June 8, 2023
**Component versions:**
| Component | Version |
| - | - |
| **Solr** | fusion-solr 5.8.1 *(based on Solr 9.1.1)* |
| **ZooKeeper** | 3.7.1 |
| **Spark** | 3.2.2 |
| **Kubernetes** | GKE, AKS, EKS 1.24 Rancher (RKE) and OpenShift 4 compatible with Kubernetes 1.24 OpenStack and customized Kubernetes installs not supported. See [Kubernetes support](/docs/policies/lifecycle-policies/lw-version-support-lifecycle#kubernetes-support) for end of support dates. |
| **Ingress Controllers** | Nginx, Ambassador (Envoy), GKE Ingress Controller Istio not supported. |
More information about support dates can be found at [Lucidworks Fusion Product Lifecycle](/docs/policies/lifecycle-policies/lw-version-support-lifecycle).
**Looking to upgrade?**
See [Fusion 5 Upgrades](/docs/5/fusion/operations/fusion-5-upgrades) for detailed instructions.
Rosette Entity Extractor (REX) or Rosette Base Linguistics (RBL) for the **Use Advanced Linguistics with Babel Street** or are not compatible with Solr 9 included in this version of Fusion. If you rely on the Babel Street language module, do not upgrade until this compatibility issue is resolved.
## Bug Fixes
Fusion 5.8.1 fixes a bug in the Fusion 5.8.0 Helm chart that prevented horizontal pod autoscaling from working.
If you are not using horizontal pod autoscaling, you do not need to upgrade to Fusion 5.8.1. This release does not make any other changes to your deployment.
To use horizontal pod autoscaling in Fusion 5.8.1, follow these steps:
1. Add the metrics server to your Fusion 5.8.1 deployment.
2. Ensure the following changes are made your custom values YAML file for horizontal pod autoscaling:
1. Add service limits to the `resources` object of a service. For example:
```yaml theme={"dark"}
query-pipeline:
resources:
limits:
cpu: "2"
memory: "2Gi"
requests:
cpu: "500m"
memory: "1Gi"
```
Resource limits vary depending on your deployment. Use values specific to your needs.
Do not use horizontal scaling with Argo.
Horizontal scaling is not available with the Argo workflow controller. Do not apply horizontal autoscaling, or any other type of horizontal scaling, to the Argo service.
Applying horizontal autoscaling to Argo can cause unexpected behavior, such as pods being unnecessarily terminated, jobs failing to launch, or models failing to deploy.
2. Use the new keys and values to support autoscaling. For example, the key `targetAverageUtilization` is now `target`, which requires the keys `type` and `averageUtilization`:
```yaml theme={"dark"}
autoscaling:
enabled: true
minReplicas: 1
maxReplicas: 5
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 10
```
To verify that horizontal pod autoscaling is running, use the `k get HorizontalPodAutoscaler` command. The output should resemble the following:
```bash theme={"dark"}
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
f5-query-pipeline Deployment/f5-query-pipeline 1%/10% 1 5 1 84m
```
## Known issues
* New Kerberos security realms cannot be configured successfully in this version of Fusion.
# Fusion Advanced Linguistics Package
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/advanced-linguistics
The Fusion Advanced Linguistics Package provides Lucidworks language packs for multilingual text analysis. To improve search recall, Base Linguistics handles the unique linguistic phenomena of more than 30 Asian and European languages. Entity Extractor identifies named entities such as people, locations, and organizations, allowing you to quickly refine your search, remove noise, and increase search relevance.
## Using Named Entities
Entity Extractor (REX) extracts named entities in multiple languages, including English, Chinese (traditional and simplified), and German. In English, it extracts multiple entity types and subtypes. This includes the following entity types (along with their associated subtypes):
* `PERSON`
* `LOCATION`
* `ORGANIZATION`
* `PRODUCT`
* `TITLE`
* `NATIONALITY`
* `RELIGION`
In this tutorial, you will extract named entities from English news articles.
For more information, see [Entity Extractor](https://docs.babelstreet.com/r/Entity-Extractor-Overview-and-Technical-Specifications/Key-features) and [Entity Types](https://docs.babelstreet.com/r/Entity-Extractor-Overview-and-Technical-Specifications/Entity-types).
### Create an application
To begin, create a new application called "entities".
### Configurate the application
#### Edit the Solr configuration
You will begin by adding the required library elements to the `solrconfig.xml` file. You will also add a new update processor to perform the entity extraction.
1. Navigate to **System > Solr Config** to edit the `solrconfig.xml` file.
2. Fusion 5.8 and earlier: In the `` directive section, add the lines below. Fusion 5.9 and later already contain these lines.
```xml theme={"dark"}
```
For Fusion 4.x users, the `dir` paths are the local REX installation path.
3. In the `` section, add the following lines after the existing processor chains:
```xml theme={"dark"}
/opt/basistech/rex-jetext_eng
```
Note the reference to a field called `text_eng`. You will create this field through the Fusion UI in the next step.
4. Save your changes to `solrconfig.xml`.
#### Define fields
The data file you will use, [eng\_docs.csv](/assets/attachments/basistech/eng_docs.csv), contains two fields:
* `title`: an article headline.
* `article_text`: the text content of the article.
You will index these two static fields and also define a set of dynamic fields to hold the extracted entities. To create new fields, navigate to **Collections > Fields** and click **Add a Field**. Create the following fields:
| Field name | Field type | Other options |
| - | - | - |
| `title` | `string` | Use default options. |
| `text_eng` | `text_en` | Use default options. |
| `text_eng_REX_*` | `string` | Create this field as a dynamic field by clicking the **Dynamic** checkbox. Click the **Multivalued** checkbox. Leave other options as defaults. |
Save each field after creating it.
### Index data
#### Create an index pipeline
1. Navigate to **Indexing > Index Pipelines**.
2. Click **Add** and create a new pipeline called `test-entities`.
3. Select the **Field Mapping** stage.
4. In the **Field Translations** section, add a new row with source `article_text` and target `text_eng`. Set the **Operation** to `move`.
The CSV file you will upload contains a field title, but since this matches the title field that you created earlier, there is no need to map it. Leave the **Unmapped Fields** section with its default option, `keep`.
5. Select the **Solr Indexer** stage.
6. In the **Additional Update Request Parameters** section, add a new row with parameter name `update.chain` and value `rex`.
7. Save the new pipeline.
#### Create a datasource
In this step, you will upload and index our documents from the data file.
1. Navigate to **Indexing > Datasources**.
2. Click **Add** and select **File Upload V2** from the dropdown menu.
3. Enter `eng_docs` for the **Datasource ID**. Alternatively, use a name you prefer.
4. Select `test-entities` for the **Pipeline ID**.
5. In the **File Upload** field, choose the sample file `eng_docs.csv` and click **Upload File**.
The File ID field will be automatically populated. Leave all other values as their defaults.
6. Save the new datasource. The form will refresh, adding a set of buttons at the top.
7. Click **Run** and then **Start**. When the job is finished, you will see "Success" in the popup form.
### Query data
1. Navigate to **Querying > Query Workbench**. The default query is `*:*`, which should bring up three documents.
2. For the document with title "SpaceX Successfully Launches its First Crewed Spaceflight", select **Show fields**. You will see a number of entities listed under the `text_eng_REX_*` fieldnames.
3. Search on these multivalued fields. For example, set your query to `text_eng_REX_LOCATION:"New York"` to return the article that contains a mention of New York.
### Customize (advanced)
When setting up the Solr configuration, you specified the rootDirectory and fields options in your processor chain. REX provides a number of [other configuration options](https://docs.babelstreet.com/r/Base-Linguistics-RBL/Introduction) you can set to control how entities are extracted. For example, if you are finding false positives, you can set parameters instructing REX to return only entities above a [confidence threshold](https://www.babelstreet.com/blog/fine-tune-your-match-threshold-and-parameters). The confidence threshold is a value between `0` and `1` and applies to entities extracted by the statistical model. Lucidworks recommends starting with a low value, around `0.2`. In your `solrconfig.xml` file, add the options `calculateConfidence` and `confidenceThreshold` to your processor chain definition:
```xml theme={"dark"}
/opt/basistech/rex-jetext_engTrue0.2
```
Save the changes, re-index your data, and perform the same query on `*:*`. Note that for the SpaceX article, "Falcon", is now correctly omitted from the list of LOCATIONs.
If there is a particular entity you want to make sure is extracted or rejected, or if you wish to create a custom entity type, REX also supports [gazetteers and regular expressions](https://docs.babelstreet.com/r/Model-Training-Suite-System-Administrator-Guide/Install-the-training-environment).
#### Gazetteers
A gazetteer is a UTF-8 text file in which the first line is the entity type. It is followed by the names of entities you wish to extract, separated by newlines, in the language of your documents. Comments can be prefixed by the `#` symbol. Create a file `spacecraft_gaz.txt` with the following lines:
```
SPACECRAFT
ISS
International Space Station
Vostok
Soyuz
Dragon
Crew Dragon
Cargo Dragon
Voyager
Apollo
```
#### Regular expressions
REX uses the Tcl regular expression engine. Create a file, `zulu_time_regex.xml`, file with the following lines:
```xml theme={"dark"}
(?i)\m(?:[01]?\d|2[0-4])(?:[0-5]\d) (?:UTC|GMT)
```
This regular expression will extract as entity type `ZULU_TIME` all spans that consist of a 4-digit military time unit followed by the time zone designator UTC or GMT.
#### Example
To instruct REX to use the gazetteer and RegEx file, edit your `solrconfig.xml` file.
##### addGazetteers
The language code for the gazetteer, such as `eng` for English.
The file path to the gazetteer file.
Whether to accept or reject the entities in the gazetteer.
Whether the matching should be case-sensitive.
For example:
```xml theme={"dark"}
eng,/path/to/spacecraft_gaz.txt,True,True
```
##### addRegularExpressions
The file path to the regular expressions XML file.
Whether to accept or reject the entities matched by the regular expressions.
For example:
```xml theme={"dark"}
/path/to/zulu_time_regex.xml,True
```
The result in the `solrconfig.xml` file:
```xml theme={"dark"}
/opt/basistech/rex-jetext_engTrue0.2eng,/path/to/spacecraft_gaz.txt,True,True/path/to/zulu_time_regex.xml,True
```
Now, when you re-index your data and search `*:*`, the SpaceX document will have new entities listed in the `text_eng_REX_SPACECRAFT` and `text_eng_REX_ZULU_TIME` dynamic fields.
Additional Fusion deployment configurations are needed to use the REX gazetteer and RegEx options.
## Use multilingual search
The Lucidworks language packs provide a set of linguistic tools to prepare your data for analysis. Language-specific models provide base forms (lemmas) of words, parts-of-speech tagging, compound components, normalized tokens, stems, and roots.
In this tutorial, you will index and query headlines in English, Chinese, and German to demonstrate the linguistics capabilities of RBL: lemmatization, tokenization, and decompounding.
### Create an application
To begin, create a new application called "multilingual".
### Configure the application
#### Edit the Solr configuration
You will begin by adding the required library elements to the `solrconfig.xml` file.
1. Navigate to **System > Solr Config** to edit the `solrconfig.xml` file.
2. In the `` directive section, add the following lines:
```xml theme={"dark"}
```
For Fusion 4.x users, the `dir` path is the local REX installation path.
3. Save your changes to `solrconfig.xml`.
#### Edit the schema
Add a `fieldType` element for each language to be processed by the application. The `fieldType` element includes two analyzers: one for indexing documents and one for querying documents. Each analyzer contains a tokenizer and a token filter. The `language` attribute is set to the [language code](https://docs.babelstreet.com/r/Match-Overview-and-Technical-Specifications/Languages), equal to the ISO 639-3 code in most cases. The `rootDirectory` points to the RBL directory.
1. Navigate to **System > Solr Config** to edit the `managed-schema.xml` file.
2. In the **fieldType** section, add the following new field types: `basis_english`, `basis_chinese`, and `basis_german`.
```xml theme={"dark"}
```
You can incorporate any additional Solr filters you need, such as the Solr lowercase filter. However, filters should be added into the chain *after* the Base Linguistics token filter. If you modify the token stream too significantly before RBL, you degrade its ability to analyze the text.
3. Save your changes to `managed-schema.xml`.
#### Define fields
The data file you will use, [multilingual\_headlines.csv](/assets/attachments/basistech/multilingual_headlines.csv), contains fields for headlines in three languages: `eng_headline`, `zho_headline`, and `deu_headline`. The analysis chain requires a field definition with a `type` attribute that maps to the `fieldType` you defined in the schema.
To create new fields, navigate to **Collections > Fields** and click **Add a Field**. Create the following fields:
| Field name | Field type | Other options |
| - | - | - |
| `text_eng` | `basis_english` | Use default options. |
| `text_zho` | `basis_chinese` | Use default options. |
| `text_deu` | `basis_german` | Use default options. |
Save each field after creating it.
### Index data
#### Create an index pipeline
1. Navigate to **Indexing > Index Pipelines**.
2. Click **Add** and create a new pipeline called `test-multilingual`.
3. Select the **Field Mapping** stage.
4. In the **Field Translations** section, add three new rows:
| Field name | Field name | Operation |
| - | - | - |
| `eng_headline` | `text_eng` | `move` |
| `zho_headline` | `text_zho` | `move` |
| `deu_headline` | `text_deu` | `move` |
5. Save the new pipeline.
#### Create a datasource
In this step, you will upload and index our documents from the data file.
1. Navigate to **Indexing > Datasources**.
2. Click **Add** and select **File Upload** from the dropdown menu.
3. Enter `multilingual_headlines` for the **Datasource ID**. Alternatively, use a name you prefer.
4. Select `test-multilingual` for the **Pipeline ID**.
5. In the **File Upload** field, choose the sample file `multilingual_headlines.csv` and click **Upload File**.
The File ID field will be automatically populated. Leave all other values as their defaults.
6. Save the new datasource. The form will refresh, adding a set of buttons at the top.
7. Click **Run** and then **Start**. When the job is finished, you will see "Success" in the popup form.
### Query data
1. Navigate to **Querying > Query Workbench**. The default query is `*:*`, which should bring up ten documents.
2. Follow the examples in the subsections below to see how Fusion's Advanced Linguistics capabilities can improve your search results.
#### Lemmatization
A "lemma" is the canonical form of a word, or the version of a word that you find in the dictionary. For example, the lemma of "mice" is "mouse". The words "speaks", "speaking", "spoke", and "spoken" all share the same lemma: "speak".
With RBL, you can perform searches by lemma, thus increasing your search results. This example demonstrates this practice with the words "knife" and "knives" below.
1. For ease of viewing results, select the **Display Fields** dropdown and enter `text_eng` in the **Description** field.
2. Enter the query `text_eng:knife` in the search box.
Two documents are returned. One of the headlines, "Iowa City Man Arrested After Alleged Altercation with a Knife", is an exact match on the query term `knife`. With a standard Solr text field type, this would be the only result returned. However, the special type `basis_english` you configured allows the search engine to recognize "knives" as a form of "knife". Therefore, the article "The Best Ways to Sharpen Kitchen Knives at Home" is also returned.
RBL can significantly reduce your dependence on creating, maintaining, and using large synonym lists.
#### Tokenization
Tokenization is the process of separating a piece of text into smaller units called "tokens". Tokens can be words, characters, or subwords, depending on how they are defined and analyzed. The RBL tokenizer first determines sentence boundaries, then segments each sentence into individual tokens. The most useful tokens are often words, though they may also be numbers or other characters.
In some languages like Chinese and Japanese, word tokens are not separated by whitespace, and words can consist of one, two, or more characters. For example, the tokens in 我喜歡貓 (I like cats) are 我 (I), 喜歡 (like), and 貓 (cats). RBL uses statistical models to identify token boundaries, allowing for more accurate search results.
1. For ease of viewing results, select the **Display Fields** dropdown and enter `text_zho` in the **Description** field.
2. Enter the query `text_zho:美國` (United States) in the search box.
The document "美國男子染疫無呼吸道症狀但難以說話行走" ("U.S. Man has no Respiratory Symptoms but has Difficulty Talking and Walking") is returned, as 美國 (United States) is a token in the headline.
With a standard Solr text field type, this headline would be naively tokenized with one character per token. Therefore, a search for `美` (beautiful) would trigger a false positive match, even though it is not a word in this context. However, with the advanced analytics you have configured here, the query `text_zho:美` will correctly return zero results.
#### Compounds
RBL can [decompose](https://docs.babelstreet.com/r/Base-Linguistics-Elasticsearch-Plugin/Analyzers/Compounds) Chinese, Danish, Dutch, German, Hungarian, Japanese, Korean, Norwegian, and Swedish compounds, returning the lemmas of each of the components. The lemmas may differ from their surface form in the compound, such that the concatenation of the components is not the same as the original compound (or its lemma). Components are often connected by elements that are present only in the compound form. RBL allows Solr to index and query on these components, increasing recall of search results.
1. For ease of viewing results, select the **Display Fields** dropdown and enter `text_deu` in the **Description** field.
2. Enter the query `text_deu:Land` in the search box.
The result "Mitarbeiter sitzen in ihren Heimatländern fest" ("Employees are Stuck in Their Home Countries") returned.
This headline contains the compound "Heimatländern" (home countries). A search on `Land` (country) with a standard Solr text field type would not trigger a match. However, because RBL performs decompounding with lemmatization, searching on `Heimat` or `Land` will return a result.
### Customize (advanced)
When setting up the Solr configuration, you specified the language and rootDirectory options in your field type definition. This is sufficient for most use cases. However, RBL does provide [more options](https://docs.babelstreet.com/r/Base-Linguistics-Elasticsearch-Plugin/Options/General-Options) to control the behavior of the tokenizer and analyzer. For example, the default tokenization does not consider URLs. As a result, `https://lucidworks.com` is tokenized as `https`, `lucidworks`, and `com`.
If you wish to recognize URLs, you can add the option `urls="true"` to the tokenizer in your field type definition:
```xml wrap theme={"dark"}
```
This will instruct RBL to consider URLs as a single token, for example: `https://lucidworks.com`.
To see a list of all options, consult the full [RBL documentation](https://docs.babelstreet.com/r/Base-Linguistics-Elasticsearch-Plugin/Introduction).
# Blob Storage
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/blob-storage
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-in/blob-storage
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/blob-storage
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/153
Fusion accepts large binary objects (blobs) for upload, and stores them in Solr. Blob uploads are used to install models, lookup lists, JDBC drivers, and more.
## Blob Types
A `resourceType` query parameter can be used to specify the blob type. For example, specify `file` when uploading a text or CSV file, like this:
```bash wrap theme={"dark"}
curl -H 'Content-Type: text/csv' -u USERNAME:PASSWORD -X PUT 'https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/blobs/quickstart/EXAMPLE.csv?resourceType=file' --data-binary @/path/EXAMPLE.csv
```
The complete list of valid values for `resourceType` is below:
| Type | Description |
| - | - |
| `banana` | A Banana [dashboard](/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/overview) |
| `catalog` | An analytics catalog |
| `driver:jdbc` | A JDBC Driver |
| `file:js-index` | A JavaScript file for use with a [Managed Javascript index stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/managed-javascript-index-stage) |
| `file:js-query` | A JavaScript file for use with a [Managed Javascript query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/managed-javascript-query-stage) |
| `file` | Any uploaded file, such as from the Quickstart or the [Index Workbench](/docs/5/fusion/intro/ui-tour/index-workbench) |
| `unspecified` | A blob of unknown type If no `resourceType` is specified on upload, "other" is assigned by default. |
| `plugin:index-stage` | An [index stage plugin](/docs/5/fusion/dev-portal/index-stage-sdk) |
| `plugin:query-stage` | A [query stage plugin](/docs/5/fusion/dev-portal/query-stage-sdk/overview) |
The JDBC V2 connector is supported, and fetches documents from a relational database via SQL queries. Under the hood, this connector implements the Solr [DataImportHandler (DIH)](https://wiki.apache.org/solr/DataImportHandler) plugin.
Fusion stores JDBC drivers in the blob store. You can upload a driver using the Fusion UI or the Blob Store API.
## How to upload a JDBC driver using the Fusion UI
1. In the Fusion UI, navigate to **System** > **Blobs**.
2. Click **Add**.
3. Select **JDBC Driver**.\
The "New 'JDBC Driver' Upload" panel appears.
4. Click **Choose File** and select the .jar file from your file system.
5. Click **Upload**.\
The new driver’s blob manifest appears.
From this screen you can also delete or replace the driver.
## How to install a JDBC driver using the API
1. Upload the JAR file to Fusion’s blob store using the [`/blobs/{id}` endpoint](/api-reference/blobs/upload-a-blob).
Specify an arbitrary blob ID, and a `resourceType` value of `plugin:connector`, as in this example:
```bash theme={"dark"}
curl -u USERNAME:PASSWORD -H "content-type:application/java-archive" -H "content-length:707261" -X PUT --data-binary @postgresql-42.0.0.jar http://localhost:8764/api/blobs/mydriver?resourceType=driver:jdbc
```
Success response:
```json theme={"dark"}
{
"name" : "mydriver",
"contentType" : "application/java-archive",
"size" : 707261,
"modifiedTime" : "2017-06-09T19:00:48.919Z",
"version" : 0,
"md5" : "c67163ca764bfe632f28229c142131b5",
"metadata" : {
"subtype" : "driver:jdbc",
"drivers" : "org.postgresql.Driver",
"resourceType" : "driver:jdbc"
}
}
```
Fusion automatically publishes the event to the cluster, and the listeners perform the driver installation process on each node.
If the blob ID is identical to an existing one, the old driver will be uninstalled and the new driver will installed in its place. To get the list of existing blob IDs, run: `curl -u USERNAME:PASSWORD https://FUSION_HOST:FUSION_PORT/api/blobs`
2. To verify the uploaded driver, run:
```bash theme={"dark"}
curl -u USERNAME:PASSWORD https://FUSION_HOST:FUSION_PORT/api/blobs/BLOB_ID/manifest
```
Where the `BLOB_ID` is the name specified during upload, such as "mydriver" above. A success response looks like this:
```json theme={"dark"}
{
"name" : "mydriver",
"contentType" : "application/java-archive",
"size" : 707261,
"modifiedTime" : "2017-06-09T19:05:17.897Z",
"version" : 1569755095787110400,
"md5" : "c67163ca764bfe632f28229c142131b5",
"metadata" : {
"subtype" : "driver:jdbc",
"drivers" : "org.postgresql.Driver",
"resourceType" : "driver:jdbc"
}
}
```
## The Blob manager
In addition to the [Blob Store API](/api-reference/blobs/get-blob-store-service-status), the Fusion UI provides an interface to the blob store in the [Fusion workspace](/docs/5/fusion/intro/ui-tour/overview) at **System** > **Blobs**.
* Click **Add** to upload a new blob.
* Select an uploaded blob to view, replace, or delete it.
* Click **Add** to upload a new blob.
* Select an uploaded blob to view, replace, or delete it.
### Blob editor
The blob editor gives the ability to edit blobs in the Fusion UI in a variety of formats:
* CSV
* Java
* JavaScript
* Python
* SCALA
* Typescript
* Plain text
Navigate to **System > Blobs**, select a blob, and click the **Edit blob** button to open the editor:
* If the **Edit blob** button is not available, the blob type cannot be edited with the blob editor.
* For Fusion 5.4.x and later, JavaScript can be edited directly in the Fusion UI.
The editor has several elements worth noting:
| Elements | Description |
| - | - |
| | Copies the JSON to the clipboard. |
| | Pastes the clipboard into the editor. Only valid JSON is accepted. |
| | Expands the editor to a fullscreen view. |
| | Condenses the editor to a compact view. |
| | Saves the JSON in the editor and updates the blob. |
| | Resets changes made since the last time changes were applied. |
For instructions, see **Use the Blob Editor**.
The blob editor gives the ability to edit blobs in the Fusion UI in a variety of formats:
* CSV
* Java
* JavaScript
* Python
* SCALA
* Typescript
* Plain text
This article focuses on editing an existing blob. In this case, the blob is a [stopwords file](/docs/5/fusion/getting-data-out/query-enhancement/stopwords-files) that needs to be updated.
1. Navigate to **System > Blobs** and select a valid blob.
2. Click the **Edit blob** button. This button is only available for valid blob types.
The editor appears:
3. (optional) Click **Expand editor** to view the editor in fullscreen mode.
4. (optional) Click the **Copy** button to copy the blob details. This can be saved somewhere, if desired.
5. Copy the contents of the new stop words file, and click the **Paste** button to replace the contents of the blob. Alternatively, delete the existing contents and manually paste the new contents.
6. Click the **Save** button to save your changes.
# Asynchronous Tika Parsing
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/asynchronous-tika-parsing
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-in/indexing/asynchronous-tika-parsing
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/asynchronous-tika-parsing
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/lo9hdj
In synchronous Tika parsing, parsing and indexing are performed concurrently. This can result in slow indexing for a large number of documents, as the parser and indexer must share resources.
Asynchronous Tika parsing, on the other hand, performs parsing in the background. This allows Fusion to continue indexing documents while the parser is processing others, resulting in improved indexing performance for large numbers of documents.
**FAQ**
How does the configuration differ from synchronous Tika parsing?
* In Fusion 5.9.10 and earlier, asynchronous parsing does not use a Fusion parser stage. Instead, the configuration is made in the datasource and index pipeline.
* In Fusion 5.9.11 and later, asynchronous parsing uses the [Apache Tika Container parser stage](/docs/5/fusion/reference/config-ref/parser-stages/apache-tika-container) in addition to configuration in the datasource and index pipeline. The parser ID set in the connector datasource configuration is used, allowing you to use any parser stage configuration except deprecated Tika stages.
By default, the asynchronous parsing service is deployed with a single instance, or pod. Depending on the number of documents and their size, you may consider scaling up or down the service. It is also possible to configure:
* **Cron expression:** The cron expression that controls when the task is be executed.
* **Number of datasources:** The number of datasources to select per task.
* **Number of documents:** The number of documents to select per datasource per task.
* **Task execution timeout:** The maximum amount of time that a task is allowed to run before it is terminated.
**Important**
* In Fusion 5.9.10 and earlier, *your parser configuration is ignored* when using asynchronous parsing.\
The asynchronous parsing service performs Tika parsing using Apache Tika Server. Other parsers, such as HTML and JSON, are not supported by the asynchronous parsing service.\
Although your datasource is linked to a parser configuration, this link is ignored when asynchronous parsing is used.
* In Fusion 5.9.11 and later, *your parser configuration is used* when using asynchronous parsing.\
Asynchronous parsing service performs Tika parsing using [Apache Tika Container](/docs/5/fusion/reference/config-ref/parser-stages/apache-tika-container). Other parsers, such as HTML and JSON, are now supported by the asynchronous parsing service.\
Your datasource is linked to a parser configuration. This link is used when asynchronous parsing is used.
## Requirements
Asynchronous parsing works for V2 connectors only, which use the Java SDK framework. It is enabled in the datasource configuration by toggling the **Advanced** view and turning on the **Async Parsing** option. Learn more about [V1 and V2 connectors](/docs/fusion-connectors/concepts/v1-v2-connectors).
Additionally, the index pipeline must include a [Solr Partial Update Indexer](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/solr-partial-update-indexer-stage) stage. This stage replaces the Solr indexer stage, which should be removed or turned off. This is required because the connector plugin and the asynchronous parsing services generate one document each, one from the fetching process and another from the parsing process, respectively. Both documents need to be merged into a single document. The Solr Partial Update Indexer merges both documents, while the Solr indexer stage overrides documents.
For more information on asynchronous parsing setup, see **Use Tika Asynchronous Parsing**.
This document describes how to set up your application to use Tika asynchronous parsing.
Unlike synchronous Tika parsing, which uses a parser stage, asynchronous Tika parsing is configured in the datasource and index pipeline. For more information, see [Asynchronous Tika Parsing](/docs/5/fusion/getting-data-in/indexing/asynchronous-tika-parsing).
**Field names change with asynchronous Tika parsing.**
In contrast to synchronous parsing, asynchronous Tika parsing prepends `parser_` to fields added to a document. System fields, which start with `\_lw_`, are not prepended with `parser_`. If you are migrating to asynchronous Tika parsing, and your search application configuration relies on specific field names, update your search application to use the new fields.
## Configure the connectors datasource
1. Navigate to your datasource.
2. Enable the **Advanced** view.
3. Enable the **Async Parsing** option.
**Fusion 5.9.11 and later uses your parser configuration when using asynchronous parsing.**
The asynchronous parsing service performs Tika parsing using Apache Tika Server. In Fusion 5.8 through 5.9.10, other parsers, such as HTML and JSON, are not supported by the asynchronous parsing service. By enabling asynchronous parsing, the parser configuration linked to your datasource is ignored. In Fusion 5.9.11 and later, other parsers, such as HTML and JSON, are supported by the asynchronous parsing service. By enabling asynchronous parsing, the parser configuration linked to your datasource is used.
4. Save the datasource configuration.
## Configure the parser stage
You must do this step in Fusion 5.9.11 and later.
1. Navigate to **Parsers**.
2. Select the parser, or create a new parser.
3. From the **Add a parser stage** menu, select **Apache Tika Container Parser**.
4. (Optional) Enter a label for this stage. This label changes the names from Apache Tika Container Parser to the value you enter in this field.
5. If the Apache Tika Container Parser stage is not already the first stage, drag and drop the stage to the top of the stage list so it is the first stage that runs.
## Configure the index pipeline
1. Go to the **Index Pipeline** screen.
2. Add the **Solr Partial Update Indexer** stage.
3. Turn off the **Reject Update if Solr Document is not Present** option and turn on the **Process All Pipeline Doc Fields** option:
4. Include an extra update field in the stage configuration using any update type and field name. In this example, an incremental field `docs_counter_i` with an increment value of `1` is added:
5. Enable the **Allow reserved fields** option:
6. Click **Save**.
7. Turn off or remove the **Solr Indexer stage**, and move the **Solr Partial Update Indexer stage** to be the last stage in the pipeline.
Asynchronous Tika parsing setup is now complete. Run the datasource indexing job and monitor the results.
## Learn more
The course for **Asynchronous Parsing Service** focuses on how to use asynchronous parsing to index your data more efficiently.
# Fields Editor UI
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/collections/fields-editor-ui
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-in/indexing/collections/fields-editor-ui
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/collections/fields-editor-ui
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/nmi4j0
The Fusion UI includes a space under Collections to edit Fields. Fields help build the schema file and are modified from Fusion with the Fields Editor UI. These fields define how Solr should interpret data in a given field and how to query the fields. You can find descriptions for these fields in the Field Type Definitions section of the [Solr Reference Guide](https://solr.apache.org/guide/8_9/field-type-definitions-and-properties.html) associated with your Fusion release.
To use the Field Editor, navigate to **Collections** > **Fields** and click **Add a Field+**.
Field options displayed in the UI include:
| Option | Input | Notes |
| - | - | - |
| **Dynamic** | checkbox | Fields that are indexed even if they weren’t included in the schema. |
| **Field Name** | alphanumeric | |
| **Field Type** | menu drop-down | |
| **Indexed** | checkbox | |
| **Stored** | checkbox | |
| **Multi-valued** | checkbox | |
| * *Required** | checkbox | |
| **Default Value** | text | |
| **Copy Fields** | uses the plus sign to add rows | **static** can copy to `raw_content` or `text`. **dynamic** can copy to any `raw_content`, `text`, or any other dynamic field. |
Advanced toggles checkboxes for **Doc Values**, **Omit Norms**, **Omit Positions**, **Omit term freq and positions**, **Term Vectors**, **Term Positions**, and **Term Offsets**.
# Collections
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/collections/overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-in/indexing/collections/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/collections/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/8831
Your data is organized into collections. When you create an app, Fusion automatically creates a collection with the same name. You can create additional collections in any app.
A primary collection contains the data that your users will search. Every primary collection is associated with a set of auxiliary collections that contain related data, such as signals, aggregations, and more.
Under the hood, a Fusion collection is a distributed index in Solr, defined by a named configuration stored in ZooKeeper, with these properties:
* **Number of shards.** Documents are distributed across this number of partitions.
* **Document routing strategy.** How documents are assigned to shards.
* **Replication factor.** How many copies of each document in the collection.
* **Replica placement strategy.** Where to place replicas in the cluster.
If your data is already stored in a Solr instance or cluster, you can manage this collection
in Fusion by creating a Fusion collection that imports the existing Solr collection.
Collection names are case-insensitive, but Fusion preserves case when displaying collection names.
## Auxiliary Collections
Every primary collection is associated with a set of auxiliary collections that contain related data, such as signals, aggregations, and more.
Some auxiliary collections are created for every primary collection. Others are created only for the app’s default collection, one per app.
Auxiliary collections are described below:
| | | |
| - | - | - |
| `APP_NAME_job_reports` | Output from Fusion [experiments](/docs/5/fusion/getting-data-out/data-analytics/experiments/overview), [Ranking Metrics jobs](/docs/5/fusion/reference/config-ref/jobs/ranking-metrics), and [Head/Tail Analysis jobs](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis). | 1 per app |
| `APP_NAME_query_rewrite` | A collection of documents to use for [rewriting queries](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting), optimized for high-volume traffic. These documents originate from the `COLLECTION_NAME_query_rewrite_staging` collection. Certain Fusion query pipeline stages read from this collection:
● [Modify Response with Rules](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/rules-augment-response-query-stage) | 1 per app |
| `APP_NAME_query_rewrite_staging` | A collection of documents created by the Rules Editor or by certain [Fusion jobs](/docs/5/fusion/reference/config-ref/jobs/overview), not optimized for production traffic. Documents move from this collection to the `COLLECTION_NAME_query_rewrite` collection as follows:
● Job output documents with high confidence contain a `review=auto` field and are moved to the `COLLECTION_NAME_query_rewrite` collection automatically.
● Job output documents with low confidence contain a `review=pending` field. When these are approved by a Fusion user, Fusion copies them to the `COLLECTION_NAME_query_rewrite` collection. | 1 per app |
| `COLLECTION_NAME_signals` | A search query logs and signals collection. | 1 per collection |
| `COLLECTION_NAME_signals_aggr` | A collection for aggregated signals. | 1 per collection |
| `APP_NAME_user_prefs` | A collection of data to support App Studio’s social features, such as user-generated tags, bookmarks, comments, ratings, and so on. | 1 per app |
Don’t create primary collections with names that end in the suffixes above; these are reserved for Fusion auxiliary collections, which are created and managed by Fusion directly.
Fusion maintains a set of Solr collections that store Fusion’s own
log files and other internal information.
These are called [System Collections](#system-collections), described below.
Don’t create primary collections named "logs" or beginning with "system\_".
These names are reserved for Fusion system collections.
Fusion uses ZooKeeper to register information about all collections,
and the Fusion components and services related to a collection.
The Fusion components associated with a collection include:
* Datasources
* Pipelines
* Profiles
* Signals and aggregations
* Analytics dashboards
## System Collections
Fusion automatically creates some collections that are used for internal purposes and shared across all apps:
* **system\_autocomplete** stores the content that the Fusion UI displays when you use the search bar.
* **system\_blobs** stores [blobs](/docs/5/fusion/getting-data-in/blob-storage) in Solr. This is used to store model files for the NLP components and other binary files used by Fusion components.
* **system\_history** keeps a record of configuration changes, start and stop times for services and experiments, and more.
* **system\_jobs\_history** keeps a record of Fusion [jobs](/docs/5/fusion/reference/config-ref/jobs/overview), including start/stop times and status.
## Collection Configuration Properties
Collections have properties that you can configure only when you are creating a collection using the
[Collections API](/api-reference/collections/get-collections-service-status).
| Property | Description | Default behavior |
| - | - | - |
| signals\* | The `signals` property determines whether to create auxiliary collections with suffixes `_signals` and `_signals_aggr`. | When you create a collection in the Fusion UI, `signals` defaults to **true**. When you create a collection using the Fusion API, this property defaults to **false**. |
| searchLogs | The `searchLogs` property determines whether to create an auxiliary search query logs collection with suffix `_logs`. | When you create a collection in the Fusion UI, this property defaults to **true**. When you create a collection using the Fusion API, this property defaults to **false**. |
| commitWithin | The `commitWithin` property guarantees that the data is committed and available for searching within the time specified in the value. | The default of `10000` milliseconds saves the data and makes that data available for searching within 10 seconds. The default for signal collections is `1000` milliseconds. |
| autoCommit | The `autoCommit` property (Solr hard commit) is inherited from the collection's `solrconfig.xml`. | By default, this setting is typically set to `15` seconds with `openSearcher=false`. It saves the data, but does not force the search results to refresh immediately. With this setting, search performance is not slowed, but the new data may not show in search results until the next refresh. This property can be used instead of the `commitWithin` property and is set using the [Solr configuration](/api-reference/solr-configuration/replace-a-solr-configuration-file). |
| autoSoftCommit | The `autoSoftCommit` property (Solr soft commit) does not save the data, but makes the data visible to searches almost immediately. If the system crashes, that new data is lost because it has not been saved. This property is set using the [Solr configuration](/api-reference/solr-configuration/replace-a-solr-configuration-file). | The default setting is turned off, and search visibility is managed using the `commitWithin` setting. |
\*Signals are events with timestamps that can be used to improve search results.
For more information about signals in Fusion, see [Signals](/docs/5/fusion/getting-data-out/query-enhancement/signals/overview) in the Fusion documentation.
In schemaless mode, if a document contains a field not currently in the Solr schema, Solr processes the field value to determine what the field type should be defined as, and then adds a new field to the schema with the field name and field type.
This behavior can be convenient during preliminary application development, but it’s rarely appropriate in a production environment.
## Using profiles to associate collections with pipelines
Index pipelines and query pipelines aren’t connected to a specific collection by default. Index profiles and query profiles are configurations that create consistent endpoints for indexing and querying, each with a specific pipeline and collection.
* [Index Profiles](/docs/5/fusion/getting-data-in/indexing/index-pipelines/index-profiles) work with index pipelines for getting content into the system.
* [Query Profiles](/docs/5/fusion/getting-data-out/query-basics/query-pipelines/query-profiles) work with query pipelines for user queries.
## Field Editor UI
The Fields Editor UI allows you to create and configure the schema file directly from Fusion. For instructions, see [Fields Editor UI](/docs/5/fusion/getting-data-in/indexing/collections/fields-editor-ui).
## Learn more
The course for **Fusion Applications and Collections** focuses on how Fusion transforms your siloed data into personalized insights unique to each user.
A Fusion collection can be configured to map to multiple Solr collections, known as partitions in this context, where each partition contains data from a specific time range. An example is time-based partitioning for logs:
* In the UI, you can only enable time-based partitioning for *new* collections.
* In the API, you can only enable time-based partitioning for *existing* collections.
## Enablement using the Fusion UI
1. Open the Collections Manager:
2. Click **New**.\
In the UI, you can only enable time-based partitioning for *new* collections. To enable it for an existing collection, [use the API](#enablement-using-the-api).
3. Click **Advanced**.
4. Scroll down to "Time Series Partitioning".
5. Click **Enable**.\\
When you enable this option, Fusion displays the time series partitioning configuration options.
6. Save the collection.
## Enablement using the API
Use the Collection Features API to enable time-based partitioning for an existing collection.
**Enable time-based partitioning using the default configuration:**
```bash wrap theme={"dark"}
curl -X PUT -H 'Content-type: application/json' -d '{"enabled": true}' http://localhost:8764/api/collections/COLLECTION_NAME/features/partitionByTime
```
No response is returned.
Submit an empty request to the same endpoint to verify that time-based partitioning is enabled:
```bash wrap theme={"dark"}
curl -X GET http://localhost:8764/api/collections/COLLECTION_NAME/features/partitionByTime
```
Response:
```json theme={"dark"}
{
"name" : "partitionByTime",
"collectionId" : "COLLECTION_NAME",
"params" : { },
"enabled" : true
}
```
# Connectors and Datasources
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/connectors
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-in/indexing/connectors
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/connectors
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/164
Connectors are the out-of-the-box components for pulling your data into Fusion. Lucidworks provides a wide variety of connectors, each specialized for a particular data type. When you add a [datasource](/docs/fusion-connectors/concepts/datasources) to a [collection](/docs/5/fusion/getting-data-in/indexing/collections/overview), you specify the connector to use for ingesting data.
Connectors are distributed separately from Fusion. For complete information, see [Fusion Connectors](/docs/fusion-connectors/overview).
# Data Models
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/data-models/overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-in/indexing/data-models/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/data-models/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/8829
Data models simplify the process of getting started with Fusion by providing pre-configured objects to reduce the effort spent on basic starting tasks. This helps keep documents consistent between datasources and intuitive to the object’s type.
## Components
Data models consist of the following:
* **[Object type](#object-type)** - Describes the object type, or what a document in the index is. For example, an object may be a file, email, person, product, etc.
* **[Index pipeline](#index-pipeline)** - Specifies the index pipeline that is indexing documents relevant to the data model.
* **[Query pipeline](#query-pipeline)** - Specifies the query pipeline that is invoked when a query involves the data model.
* **[Fields configuration](#fields-configuration)** - Creates a schema map that describes how the object schema should be translated into the Solr index. For example, the field name `first_name` can map to Solr fields `first_name_s` and `first_name_t`.
### Object type
The object type defines what a document is and what associated data is expected.
**Example**
When indexing employees for a company directory, certain fields are expected by a user. This includes the employees’s name, title, and contact information. An object type, `person`, is created for this purpose. The data model object is created to include this information in a clear, intuitive way:
| Data Model Field | Solr Fields |
| - | - |
| `first_name` | `firstName_t` `firstName_s` |
| `last_name` | `lastName_t` `lastName_s` |
| `email` | `email_s` |
| `job_title` | `jobTitle_t` `lastName_s` |
### Index pipeline
When you add a connector in Fusion, a default index pipeline is created that is used to map the documents' fields to the data model fields. The pipeline contains two pipeline stages, the **[Data Model Mapping](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/datamodel-mapping-index-stage)** index stage and the **[Call Data Model Pipeline](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/call-datamodel-pipeline-index-stage)** index stage. Once configured, this stage indexes the documents as various object types. Fields irrelevant to that object type are optionally discarded.
### Query pipeline
The query pipeline is applied by default whenever a user is searching documents of the object type. This enables query pipeline configurations specialized for the object type. For example, a query pipeline for the object type `person` could boost or filter results for current employees over previous employees.
The query pipeline is optional for data models.
### Fields configuration
Data model fields allow you to map document fields to Solr fields in a way that respects the data model object type. The fields configuration allows this map to be intuitive to new users, offering a guided experience for configuring these options.
## UI fields configuration
| Field | Required | Description | Example |
| - | - | - | - |
| **Name of the Data Model** | ✅ | `string` A user-friendly name for the data model. This typically describes the object type. | `person` |
| **Index Pipeline** | ✅ | `string` The name of the index pipeline used for the data model. | `companyDirectory-index-pipeline` |
| **Query Pipeline** | ✅ | `string` The name of the query pipeline used for the data model. | `companyDirectory-query-pipeline` |
| **Field Name** | ✅ | `string` | `first_name` |
| * *Required** | ✘ | `boolean` | `true` |
| ***Solr Fields* ↓** | | | |
| **Solr Field Name** | ✅ | `string` | `firstName_t` |
| **Is Query Field** | ✘ | `boolean` | `true` |
| **Enable Phrase Match** | ✘ | `boolean` | `false` |
| **Boost Value** | ✘ | `integer` The amount of boost to give to the query. | `1` |
| **Phrase Boost** | ✘ | `integer` The amount of boost to give to the query, if it matches as an exact phrase. | `2` |
## More information
* **Configure a Data Model**
* [Data Models API reference](/api-reference/data-models/get-data-models-service-status)
* [Data Model Mapping Index Stage reference](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/datamodel-mapping-index-stage)
* [Call Data Model Pipeline Index Stage reference](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/call-datamodel-pipeline-index-stage)
Data models simplify the process of getting started with Fusion by providing pre-configured objects to reduce the effort spent on basic starting tasks. This helps keep documents consistent between datasources and intuitive to the object’s type.
See the [Data Models](/docs/5/fusion/getting-data-in/indexing/data-models/overview) topic for more information.
## Configure the datasource
This section references the Slack V2 Connector.
1. Navigate to **Indexing > Datasources**.
2. Click the **Add** button, and choose the Slack V2 connector from the list.
Some connectors include built-in data models as a standard component. Others require you to manually create data models.
3. Complete the datasource configuration:
* Ideally, use a pipeline specifically created for this data model. For now, you can create the pipeline by navigating to **Indexing > Index Pipelines** and clicking the **Add** button. Configure it using the information in [Configure the index pipeline](#configure-the-index-pipeline).
* Under **Profiles Settings**, select *Index profiles*:
* Under **Channels and Messages Settings**, select *Index channels* and *Index from public channels*:
4. Click **Save**.
5. Click **Run** to run the indexing job.
## Configure the data model
Some connectors include built-in data models with pre-configured object types. However, you can add new data models or customize existing ones to fit your particular needs.
1. Navigate to **Indexing > Data Models**.
2. Click the **Add** button.
3. Create a new data model. For example, `person`.
4. Assign the index pipeline for the data model. In this example, create a new index pipeline named `companyDirectory-data-pipeline`.
This pipeline is ideally for operation with data models only. For the sake of this example, we will only be using the default stages. Applying additional logic/stages specific to the object would occur here.
5. Assign the query pipeline you will use to query the indexed documents. In this example, create a new query pipeline named `companyDirectory-query-pipeline`.
6. Click the **New** button under **Fields Configuration**.
7. Complete the required configurations, as detailed in [Data Models API Reference](/api-reference/data-models/get-data-models-service-status). Create fields for the following:
| Field Name | Solr Fields |
| - | - |
| `first_name` | `firstName_t`, `firstName_s` |
| `last_name` | `lastName_t`, `lastName_s` |
| `email` | `email_s` |
| `job_title` | `jobTitle_t`, `jobTitle_s` |
8. If needed, edit the JSON for the data model before saving. After saving the data model, this JSON viewer becomes read-only.
9. Click the **Save** button to save the data model.
## Configure the index pipeline
To begin, navigate to **Indexing > Index Workbench**. Alternatively, the `companyDirectory-index-pipeline` pipeline can be configured in **Indexing > Index Pipelines**, but you are not able to preview results.
The raw data fields in your index coming from Slack may differ from the example data fields used in this article.
### Data Model Mapping stage
1. Click the **Add a Stage** button, and choose **Data Model Mapping** from the list.
2. Use the **Data Model Type** dropdown to select the `person` data model.
3. *(optional)* Check the **Match Trigger** checkbox and assign the following values:
| Field | Value |
| - | - |
| **Field to match** | `type_s` |
| **Value to match** | `user` |
The **Value to match** field supports RegEx. You can assign multiple values. Alternatively, you can create additional **Data Model Mapping** stages.
4. Assign field mappings for the Slack datasource’s raw datasources:
| Source Field | Data Model Field |
| - | - |
| `first_name` | `first_name` |
| `last_name` | `last_name` |
| `email` | `email` |
5. *(optional)* Check the **Keep unmapped fields** checkbox to preserve fields that are unmatched to the data model.
6. Click **Apply**.
### Call Data Model Pipeline stage
This stage must be placed *after* the Data Model Mapping stage.
1. Click the **Add a Stage** button, and choose **Call Data Model Pipeline** from the list.
2. Assign the value `_lw_data_model_type_s` to the **Data Model Type Field** field. This field is created when a document meets the criteria specified in the **Data Model Mapping**.
3. Click **Apply**.
## Next steps
With the index pipeline configured, you are now ready to complete the indexing job by clicking the **Start Job** button.
Once complete, the documents are ready for viewing in the Query Workbench:
1. Navigate to **Querying > Query Workbench**.
2. Click the **Load** button.
3. Choose the query pipeline you specified when creating the data model, `companyDirectory-query-pipeline`.
## More information
* [Data Models](/docs/5/fusion/getting-data-in/indexing/data-models/overview)
* [Data Models API reference](/api-reference/data-models/get-data-models-service-status)
* [Data Model Mapping Index Stage reference](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/datamodel-mapping-index-stage)
* [Call Data Model Pipeline Index Stage reference](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/call-datamodel-pipeline-index-stage)
# Custom JavaScript Stages for Index Pipelines
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/index-pipelines/custom-javascript-index-stages
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-in/indexing/index-pipelines/custom-javascript-index-stages
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/index-pipelines/custom-javascript-index-stages
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/v5n255
The JavaScript Index stage allows you to write a custom processing logic using JavaScript to manipulate Pipeline Documents and the index pipeline context, which will be compiled by the JDK into Java bytecode that is executed by the Fusion pipeline. The first time that the pipeline is run, Fusion compiles the JavaScript program into Java bytecode using the JDK’s JavaScript engine.
Users who can create or modify code obtain access to the broader Fusion environment. This access can be used to create intentional or unintentional damage to Fusion.
For a JavaScript Index stage, the JavaScript code must return either:
* A single document or array of documents
*or*
* The null value or an empty array.\
In the latter case, no further processing is possible, which means that the document will not be indexed or updated. For example, Solr commits have a `null` value and are dropped. For information about how to skip stages when Solr commits are sent, see the [Skip JavaScript stages during Solr commits](#skip-javascript-stages-during-solr-commits).
## JavaScript Index Stage Global Variables
[JavaScript](http://en.wikipedia.org/wiki/JavaScript) is a lightweight scripting language.
In a JavaScript stage, Fusion uses the Nashorn engine, which implements [ECMAScript](http://en.wikipedia.org/wiki/ECMAScript) version 5.1. Although Nashorn does include some ECMAScript 6 (ES6) features such as `let`, `const`, or template strings, Fusion does not enable ES6 by default, so ES6 support is not guaranteed.
What a JavaScript program can do depends on the container in which it runs.
For a JavaScript Index stage, the container is a Fusion index pipeline.
The following global pipeline variables are available:
| Name | Type | Description |
| - | - | - |
| `doc` | [ PipelineDocument ](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/apollo/common/pipeline/PipelineDocument.html) | The contents of each document submitted to the pipeline. |
| `ctx` | [ Context ](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/apollo/pipeline/Context.html) | A map that stores miscellaneous data created by each stage of the pipeline. **Important** Use the `ctx` variable instead of the deprecated `_context` global variable. The `ctx` variable is used to: • Pass data from one stage to another • Store data that needs to be passed from one custom stage to a later custom stage
The data can differ between stages: • If the previous stage changes the data • Based on the configuration of each stage If the data is modified in one stage, it may cause a later stage to function irregularly. |
| `collection` | String | The name of the Fusion collection being indexed or queried. |
| `solrServer` | [ BufferingSolrServer ](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/apollo/component/BufferingSolrServer.html) | The Solr server instance that manages the pipeline’s default Fusion collection. All indexing and query requests are done by calls to methods on this object. See [solrClient](https://lucene.apache.org/solr/5_2_1/solr-solrj/org/apache/solr/client/solrj/SolrClient.html) for details. |
| `solrServerFactory` | [ SolrClusterComponent ](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/cloud/api/solr/SolrClusterComponent.html) | The SolrCluster server used for lookups by collection name which returns a Solr server instance for that collection. For example: `var productsSolr = solrServerFactory.getSolrServer("products");` |
### Syntax Variants
JavaScript stages can be written using ***function syntax***. With function syntax, global variables are passed as function parameters.
Support for legacy syntax was removed in Fusion 5.8.
#### Function Syntax
Function syntax is used for moderately complex tasks.
```js wrap theme={"dark"}
function (doc) {
// do some work ...
return doc;
}
```
**Important**
*Function syntax* is used for the examples in this document.
#### Advanced Syntax
Advanced syntax is used for complex tasks and when multiple functions are needed.
```js wrap theme={"dark"}
/* globals Java, logger*/
(function () {
"use strict";
return function main(doc , ctx, collection, solrServer, solrServerFactory) {
// do some work ...
return doc;
};
})();
```
### JavaScript Use
The JavaScript in a JavaScript Index stage must return either a single document or an array of documents.
This can be accomplished by either:
* a series of statements where the final statement evaluates to a document or array of documents
* a function that returns a document or an array of documents
All pipeline variables referenced in the body of the JavaScript function are passed in as arguments to the function. For example, in order to access the PipelineDocument in global variable 'doc',
the JavaScript function is written as follows:
```js wrap theme={"dark"}
function (doc) {
// do some work ...
return doc;
}
```
The allowed set of function declarations are:
```js wrap theme={"dark"}
function doWork(doc) { ... return doc; }
function doWork(doc, ctx) { ... return doc; }
function doWork(doc, ctx, collection) { ... return doc; }
function doWork(doc, ctx, collection, solrServer) { ... return doc; }
function doWork(doc, ctx, collection, solrServer, SolrClusterComponent) { ... return doc; }
```
The order of these arguments is according to the (estimated) frequency of use.
The assumption is that most processing only requires access to the document object itself,
and the next-most frequent type of processing requires only the document and read-only access of some context parameters.
If you need to reference the solrServerFactory global variable, you must use the 5-arg function declaration.
In order to use other functions in your JavaScript program, you can define and use them, as long as the final statement
in the program returns a document or documents.
### Global variable `logger`
The logs are output to the indexing service logs for custom index stages. Access the Log Viewer and filter on this service to view the information.
## The JavaScript Engine Used by Fusion
The
default
JavaScript engine used by Fusion is the Nashorn engine from Oracle.
See [The Nashorn Java API](https://docs.oracle.com/javase/8/docs/technotes/guides/scripting/nashorn/api.html) for details.
In Fusion 5.9.6 and up, you also have the option to select OpenJDK Nashorn.
While Nashorn is the default option, it is in the process of being deprecated and will eventually be removed, so it is recommended to use OpenJDK Nashorn when possible.
You can select the JavaScript engine in the pipeline views or in the workbenches.
Your JavaScript pipeline stages are interpreted by the selected engine.
### Creating and accessing Java types
The following information is taken from
Oracle’s JavaScript programming guide section 3,
[Using Java From Scripts](https://docs.oracle.com/javase/8/docs/technotes/guides/scripting/prog_guide/javascript.html#A1147207).
To create script objects that access and reference Java types from Javascript use the `Java.type()` function:
```js wrap theme={"dark"}
var ArrayList = Java.type("java.util.ArrayList");
var a = new ArrayList;
```
## Examples
### Set the condition field
The JavaScript Index stage lets you define a condition to trigger the script body.
The condition field is evaluated as either true or false. Do not precede with `if` or include `;` at the end of the line.
* Works:
* `doc.hasField("title_s") === true`
* `doc.hasField("title_s") === false`
* `doc.hasField("title_s")`
* Does not work:
* `if doc.hasField("title_s") === false;`
#### Skip JavaScript stages during Solr commits
When JavaScript stages contain data manipulation such as values in the `conditions` field, errors may be generated and Solr commits are dropped because the value is `null`. To skip those stages when a Solr commit is sent through the pipeline, add the following condition to those stages: `doc.getCommands().size()==0`.
### Add a field to a document
```js wrap theme={"dark"}
function (doc) {
doc.addField('some-new-field', 'some-value');
return doc;
}
```
### Join two fields
The following example conjoins separate latitude and longitude fields into a single geo-coordinate field,
whose field name follows Solr schema conventions and ends in "\_p".
It also removes the original latitude and longitude fields from the document.
```js wrap theme={"dark"}
function (doc) {
var value = "";
if (doc.hasField("myGeo_Lat") && doc.hasField("myGeo_Long")) {
value = doc.getFirstFieldValue("myGeo_Lat") + "," + doc.getFirstFieldValue("myGeo_Long");
doc.addField("myGeo_p", value);
doc.removeFields("myGeo_Lat");
doc.removeFields("myGeo_Long");
logger.debug("conjoined Lat, Long: " + value);
}
return doc;
}
```
### Return an array of documents
```js wrap theme={"dark"}
function (doc) {
var subjects = doc.getFieldValues("subjects");
var id = doc.getId();
var newDocs = [];
for (i = 0; i < subjects.size(); i++) {
var pd = new com.lucidworks.apollo.common.pipeline.PipelineDocument(id+'-'+i );
pd.addField('subject', subjects.get(i));
newDocs.push( pd );
}
return newDocs;
}
```
### Parse a JSON-escaped string into a JSON object
While it is simpler to use a JSON Parsing index stage, the following code example
shows you how to parse a JSON-escaped string representation into a JSON object.
This code parses a JSON object into an array of attributes,
and then find the attribute "tags" which has as its value a list of strings.
Each item in the list is added to a multi-valued document field named "tag\_ss".
```js wrap expandable theme={"dark"}
function (doc, ctx) {
/**
* Returned body is a JSON string. Example:
{
"telescope": "Hubble",
"location": "low earth orbit",
"primary_mirror_size": "2.4m"
"tags": ["launched 1990","repaired 1993","repaired 1997","repaired 1997","repaired 1999","repaired 2002","repaired 2009"]
}
*/
for (var key in myData) {
var obj = myData[key];
if (obj == "tags") {
logger.debug("extracting tags from tag field {}", JSON.stringify(obj))
for (var i = 0; i < obj.length; i++) {
var tag = obj[i][0];
doc.addField("tag_ss", tag);
}
}
}
return doc;
}
```
### Do a lookup on another Fusion collection
```js wrap theme={"dark"}
function doWork(doc, ctx, collection, solrServer, solrServerFactory) {
var sku = doc.getFirstFieldValue("sku");
if (!doc.hasField("mentions")) {
var mentions = ""
var productsSolr = solrServerFactory.getSolrServer("products");
if( productsSolr != null ){
var q = "sku:"+sku;
var query = new org.apache.solr.client.solrj.SolrQuery();
query.setRows(100);
query.setQuery(q);
var res = productsSolr.query(query);
mentions = res.getResults().size();
doc.addField("mentions",mentions);
}
}
return doc;
}
```
### Reject a document
If the function returns `null` or an empty array, it will not be indexed or updated into Fusion.
```js wrap theme={"dark"}
function doWork(doc) {
if (!doc.hasField("required_field")) {
return null;
}
return doc;
}
```
### Drop a document by ID
```js wrap theme={"dark"}
function(doc) {
var id = doc.getId();
if (id !== null) {
var pattern = "https://www.mydomain.com/links/contact/?";
// 0 means the pattern was found so drop the doc
return (id.indexOf(pattern) == 0) ? null : doc;
}
return doc;
}
```
### Format Date to Solr Date
```js wrap theme={"dark"}
// For example:
// From: 26/Mar/2015:14:38:48 -0700
// To: 2015-03-26T14:38:48Z (Solr format)
function(doc) {
if (doc.getId() !== null) {
var inboundPattern = "dd/MMM/yyyy':'HH:mm:ss Z"; // modify this to match the format of the inbound date
var solrDatePattern = "yyyy-MM-dd'T'HH:mm:ss'Z'"; // leave this alone
var dateFieldName = "apachelogtime"; // change this to your date field name
var solrFormatter = new java.text.SimpleDateFormat(solrDatePattern);
var apacheParser = new java.text.SimpleDateFormat(inboundPattern);
var dateString = doc.getFirstFieldValue(dateFieldName);
logger.info("**** dateString: " + dateString);
var inboundDate = apacheParser.parse(dateString);
logger.info("**** inboundDate: " + inboundDate.toString());
var solrDate = solrFormatter.format(inboundDate);
logger.info("**** solrDate: " + solrDate.toString());
doc.setField(dateFieldName, solrDate.toString());
}
return doc;
}
```
### Replace whitespace and newlines
```js wrap theme={"dark"}
function(doc) {
if (doc.getId() !== null) {
var fields = ["col1", "col2", "col3"];
for (i = 0; i < fields.length; i++ ) {
var field = fields[i];
var value = doc.getFirstFieldValue(field);
logger.info("BEFORE: Field " + field + ": *" + value + "*");
if (value != null) {
value = value.replace(/^\s+/, ""); // remove leading whitespace
logger.info("AFTER: Field " + field + ": *" + value + "*");
value = value.replace(/\s+$/, ""); // remove trailing whitespace
logger.info("AFTER: Field " + field + ": *" + value + "*");
value = value.replace(/\s+/g, " "); // multiple whitespace to one space
logger.info("AFTER: Field " + field + ": *" + value + "*");
doc.setField(field, value);
}
}
}
return doc;
}
```
### Split the values in a field
```js wrap expandable theme={"dark"}
// Split On a delimiter. In this case, a newline
function(doc){
if (doc.getId() !== null) {
var fromField = "company2_ss";
var toField = "company2_ss";
var delimiter = "\n";
var oldList = doc.getFieldValues(fromField);
var values = [];
// parse the entries one at a time
doc.removeFields(toField); // clear out the target field
for (i = 0; i < oldList.size(); i++) {
values[i] = oldList.get(i);
// get the list of strings split by the delimiter
newList = values[i].split(delimiter);
for(j = 0; j < newList.length; j++ ){
doc.addField(toField, newList[j]);
}
}
}
return doc;
}
```
### Prevent global variables in JavaScript
#### Variable declared using the `var` keyword
If a variable is declared using the `var` keyword, the JavaScript interpreter processes the value sequentially.
In this example, the values for `var i = 0` are logged in order as `0`, `1`, `2`, `3`, `4`, etc.
```js wrap theme={"dark"}
var queries = ["cat","the cat", "the cat in the hat","the cat in the hat is back"]
for(var i = 0; i < queries.length; i++){
logger.info("query {} is '{}'",i,queries[i])
}
```
#### Variable not declared using the `var` keyword
If a variable is not declared using the `var` keyword, the JavaScript interpreter moves the declaration of variable and functions to the top of the declared (global) scope. Because Fusion pipeline stages execute in a multi-threaded environment, these global (shared) variables make the stages not thread-safe.
For more detailed information, see [Hoisting](https://developer.mozilla.org/en-US/docs/Glossary/Hoisting#:~:text=JavaScript%20Hoisting%20refers%20to%20the,defined%20in%20the%20ECMAScript%20specification).
```js wrap theme={"dark"}
var queries = ["cat","the cat", "the cat in the hat","the cat in the hat is back"]
for(i = 0; i < queries.length; i++){
logger.info("query {} is '{}'",i,queries[i])
}
```
Issues and errors that may occur include:
For multi-threaded environments, the value of `i` may not proceed sequentially from 0 to 4 as the loop is processed. Instead, values may be logged based on the execution state of the other pipeline requests. For example, `0`, `1`, `3`, `1`, `2`, etc., which logs the values as `"cat", "the cat", "the cat in the hat is back", "the cat", "the cat in the hat"`.
However, if only one thread is incrementing the `i` variable, the values proceed sequentially (`0`, `1`, `2`, `3`, `4`, etc.)
If the `queries` array varies in length from document to document, the loop may generate an `ArrayIndexOutOfBounds` exception for a Java array or an `undefined` error for a JavaScript array.
Threads may not log all four queries.
#### Setting the `"use strict"` directive
Setting the `"use strict"` directive tells the JavaScript engine to require non-global declarations of all functions and variables.
The following example demonstrates how to create a copy of a `PipelineDocument` and return both the original and the copy to the pipeline for processing.
```js wrap theme={"dark"}
/* globals Java, logger*/ // This is an optional line that informs the syntax checker about known global objects. This avoids excessive flagging of what it perceives as errors.
(function () { //Defines a top-level scope
"use strict"; // This directs the interpreter to require `var` for all variable declarations to help ensure thread safety. Any global scope variables generate compile errors.
/***
* NOTE: All code within the outer scope, (everything from other than the `main` function returned by the `function` declared on line two) is only run when the JavaScript engine JIT-compiles the script. Place comments, static declarations, functions, and initialization code here to improve readability and performance.
*/
var PipelineDocument = Java.type('com.lucidworks.apollo.common.pipeline.PipelineDocument');
var ArrayList = Java.type("java.util.ArrayList")
/**
* Take a PipelineDocument like the one passed to this stage and clone it. Then give it a new id.
*/
function clonePipelineDoc(pipelineDoc, id) {
var clone = new PipelineDocument(pipelineDoc);
if (id) {
clone.setId(id);
}
logIfDebug("Cloned document with id '{}' and gave the clone id '{}'",pipelineDoc.getId(),id)
return clone;
};
var isDebug = false // This setting is optional and determines if log messages are turned off or on. It works in conjunction with the `logIfDebug` function.
function logIfDebug(m){if(isDebug && m)logger.info(m, Array.prototype.slice.call(arguments).slice(1));}
return function main(doc , ctx, collection, solrServer, solrServerFactory) {
//This returns the function to be called by Fusion when the stage executes, and guarantees Fusion will always call this function. When a stage does not use advanced syntax, but declares multiple top-level functions, Fusion may not be able to determine which function to call. Duplicate doc and give it an ID that is the reverse of the original.
var doc2 = clonePipelineDoc(doc, doc.getId().split("").reverse().join(""))
var list = new ArrayList()
list.add(doc)
list.add(doc2)
//Return both docs to the downstream stages.
return listOfDocs;
};
})();
```
## Learn more
The course for **JavaScript in Fusion** focuses on how to leverage JavaScript in Fusion to build powerful and responsive scripts at index and query time.
## See also
* [JavaScript Index Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/javascript-index-stage)
* [Fusion Pipeline Javadocs](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/)
# Index Pipeline Stages
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/index-pipelines/index-pipeline-stages
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-in/indexing/index-pipelines/index-pipeline-stages
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/index-pipelines/index-pipeline-stages
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/8799
An Index Pipeline takes content and transforms it into a document suitable for indexing by Solr via a series of modular operations called stages.
The objects sent from stage to stage are PipelineDocument objects.
Fusion provides many specialized index stages as well as a [JavaScript Index stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/javascript-index-stage) that allows for custom processing via a JavaScript program. The general outline of the Extract/Transform/Load processing performed by an index pipeline is:
* Raw content is parsed into one or more PipelineDocument objects.
* Any number of intermediate stages operate on the document fields directly, or, in the case of specialized NLP tools, add annotations to a document.
* Finally, the PipelineDocument is sent to Solr for indexing.
A pipeline stage definition associates a unique ID with a set of properties.
Pipeline definitions are stored in ZooKeeper for reuse across pipelines and search applications.
The Fusion UI provides stage-specific panels used to define and configure each pipeline stage.
Alternatively, JSON can be used to specify the sequence of pipeline stages and registered via the Fusion REST API.
Some stages require additional resources, e.g., text files that contain lists of names, synonyms, places, or binary files which NLP language models.
These resources can be uploaded via the Fusion UI or the REST API.
The [Index Pipeline Stages Reference](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/overview) section contains articles that detail the configuration for each of the Fusion index pipeline stages. The configuration information includes field names and values, as well as examples and other key reference information.
## Pipeline stage JSON editor
The pipeline stage JSON editor gives the ability to create and copy pipeline stages by pasting JSON objects in the Fusion UI. Only JSON is supported, and JSON validation is included to prevent the user from saving an invalid object.
Navigate to your pipelines, select a pipeline stage, and click the **JSON View** button to open the editor:
Existing stages are considered **READ-ONLY**. They will have only a copy button which can be used to create stages. New stages will have the option to copy or paste valid JSON. Changes are applied to the stage with the **Apply** button.
Users will still need to **Save** the stage for the changes made in the editor to be saved.
The editor has several elements worth noting:
| Elements | Description |
| - | - |
| | Copies the JSON to the clipboard. |
| | Pastes the clipboard into the editor. Only valid JSON is accepted. |
| | Expands the editor to a fullscreen view. |
| | Condenses the editor to a compact view. |
| | Applies the JSON in the editor and updates the stage. The stage must be saved to preserve the changes. |
| | Resets changes made since the last time changes were applied. |
For instructions, see **Use the Pipeline Stage JSON Editor**.
This article will focus on copying the JSON configuration for an existing pipeline stage and creating a new stage with the JSON configuration.
The instructions in this article focus on query pipeline stages. However, the steps are the same for index pipeline stages, which are accessed at **Indexing > Index Pipelines > ``**.
1. Navigate to **Querying > Query Pipelines** and select a pipeline.
2. Select an existing pipeline stage to view in the editor.
3. Click the **JSON View** button. The editor will appear:
Existing stages are considered **READ-ONLY**. They will only have a copy button.
4. (optional) Click **Expand editor** to view the editor in fullscreen mode.
5. Click the **Copy** button. The JSON configuration for that stage is copied to your clipboard.
6. Click **Add a new pipeline stage** to create a new stage. Create the same stage type.
7. Click the **JSON View** button.
8. Use the **Paste** button to paste the JSON configuration from your clipboard into the editor. Alternatively, delete the existing JSON configuration and manually paste the new configuration.\
At this time, you can make changes to the JSON configuration inside the editor.
9. Click the **Apply** button to apply your changes to the stage. Although the changes are immediately shown in the UI, the stage is not saved until you click the **Save** button.
10. Save the stage.
## JavaScript selector
Now your pipeline definitions can include your choice of JavaScript engine, either Nashorn or OpenJDK Nashorn.
While Nashorn is the default option, it is in the process of being deprecated and will eventually be removed, so it is recommended to use OpenJDK Nashorn when possible.
You can select the JavaScript engine in the pipeline views or in the workbenches.
Your JavaScript pipeline stages are interpreted by the selected engine.
The quick learning for **Top 3 Index Pipeline Stages** focuses on useful indexing stages.
# Index Profiles
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/index-pipelines/index-profiles
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-in/indexing/index-pipelines/index-profiles
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/index-pipelines/index-profiles
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/160
Index profiles let your applications import Data with the REST API (the index profile endpoint) and change the underlying index pipeline as needed.
It is often possible to get documents into Fusion by configuring a datasource with [the appropriate connector](/docs/5/fusion/getting-data-in/indexing/connectors).
But if there are obstacles to using connectors, it can be simpler to index documents with a REST API call to an index profile or pipeline.
## Push documents to Fusion using index profiles
[Index profiles](/docs/5/fusion/getting-data-in/indexing/index-pipelines/index-profiles) allow you to send documents to a consistent endpoint (the profile alias) and change the backend index pipeline as needed. The profile is also a simple way to use one pipeline for multiple collections without any one collection "owning" the pipeline.
You can send documents directly to an index using the [Index Profiles REST API](/docs/5/fusion/getting-data-in/indexing/index-pipelines/index-profiles). The request path is:
```bash theme={"dark"}
/api/apps/APP_NAME/index/INDEX_PROFILE
```
These requests are sent as a POST request. The request header specifies the format of the contents of the request body. Create an index profile in the Fusion UI.
To send a streaming list of JSON documents, you can send the JSON file that holds these objects to the API listed above with `application/json` as the content type. If your JSON file is a list or array of many items, the endpoint operates in a streaming way and indexes the docs as necessary.
### Send data to an index profile that is part of an app
Accessing an index profile through an app lets a Fusion admin secure and manage all objects on a per-app basis. Security is then determined by whether a user can access an app. This is the recommended way to manage permissions in Fusion.
The syntax for sending documents to an index profile that is part of an app is as follows:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/json' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE --data-binary @my-json-data.json
```
Spaces in an app name become underscores. Spaces in an index profile name become hyphens.
To prevent the terminal from displaying all the data and metadata it indexes--useful if you are indexing a large file--you can optionally append `?echo=false` to the URL.
Be sure to set the content type header properly for the content being sent. Some frequently used content types are:
* Text: `application/json`, `application/xml`
* PDF documents: `application/pdf`
* MS Office:
* DOCX: `application/vnd.openxmlformats-officedocument.wordprocessingml.document`
* XLSX: `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet`
* PPTX: `application/vnd.vnd.openxmlformats-officedocument.presentationml.presentation`
* More types: [http://filext.com/faq/office\_mime\_types.php](http://filext.com/faq/office_mime_types.php)
### Example: Send JSON data to an index profile under an app
In `$FUSION_HOME/apps/solr-dist/example/exampledocs` you can find a few sample documents. This example uses one of these, `books.json`.
To push JSON data to an index profile under an app:
1. Create an index profile. In the Fusion UI, click **Indexing > Index Profiles** and follow the prompts.
2. From the directory containing `books.json`, enter the following, substituting your values for username, password, and index profile name:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/json' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE?echo=false --data-binary @books.json
```
3. Test that your data has made it into Fusion:
1. Log into the Fusion UI.
2. Navigate to the app where you sent your data.
3. Navigate to the Query Workbench.
4. Search for `*:*`.
5. Select relevant Display Fields, for example `author` and `name`.
### Example: Send JSON data without defining an app
In most cases it is best to delegate permissions on a per-app basis. But if your use case requires it, you can push data to Fusion without defining an app.
To send JSON data without app security, issue the following curl command:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/json' https://FUSION_HOST:FUSION_PORT/api/index/INDEX_PROFILE --data-binary @my-json-data.json
```
### Example: Send XML data to an index profile with an app
To send XML data to an app, use the following:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/xml' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE --data-binary @my-xml-file.xml
```
In Fusion 5, documents can be created on the fly using the [PipelineDocument](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/apollo/common/pipeline/PipelineDocument.html) JSON notation.
## Remove documents
### Example 1
The following example removes content:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/vnd.lucidworks-document' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE --data-binary @del-json-data.json
```
### Example 2
A more specific example removes data from `books.json`. To delete "The Lightning Thief" and "The Sea of Monsters" from the index, use their id values in the JSON file.
The `del-json-data.json` file to delete the two books:
```json wrap theme={"dark"}
[{ "id": "978-0641723445","commands": [{"name": "delete","params": {}}]},{ "id": "978-1423103349","commands": [{"name": "delete","params": {}}, {"name": "commit","params": {}}]}]
```
You can use `?echo=false` to turn off the response to the terminal.
### Example 3
Another example to delete items using the Push API is:
```bash wrap theme={"dark"}
curl -u admin:XXX -X POST 'http://FUSION_HOST:FUSION_PORT/api/apps/APP/index/INDEX' -H 'Content-Type: application/vnd.lucidworks-document' -d '[
{
"id": "1663838589-44",
"commands":
[
{
"name": "delete",
"params":
{}
},
{
"name": "commit",
"params":
{}
}
]
}, ...
]'
```
## Send documents to an index pipeline
Although sending documents to an index profile is recommended, if your use case requires it, you can send documents directly to an index pipeline.
For more information about index pipeline REST API reference documentation, see [Fusion 5.x Index Pipelines API](/api-reference/index-pipelines-api/list-all-index-pipelines).
### Specify a parser
When you push data to a pipeline, you can specify the name of the parser by adding a parserId querystring parameter to the URL.
For example: `https://FUSION_HOST:FUSION_PORT/api/index-pipelines/INDEX_PIPELINE/collections/COLLECTION_NAME/index?parserId=PARSER`.
If you do not specify a parser, and you are indexing outside of an app (`https://FUSION_HOST:FUSION_PORT/api/index-pipelines/...`), then the `_system` parser is used.
If you do not specify a parser, and you are indexing in an app context (`https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index-pipelines/...`), then the parser with the same name as the app is used.
## Indexing CSV Files
In the usual case, to index a CSV or TSV file, the file is split into records, one per row, and each row is indexed as a separate document.
An index profile is also a simple way to use one pipeline for multiple collections, without any one collection "owning" the pipeline. Associating a profile with an index pipeline, or an index pipeline and a collection, is simply a mapping.
Finally, you can also specify configuration parameters for an index pipeline in the index profile.
You can use the API or UI to manage index profiles.
The course for **Index and Query Profiles** focuses on how index and query profiles operate, and how to configure and manage them.
## Index Profiles in the API
* [Index Profiles CRUD API](/api-reference/index-profiles-crud-api/list-all-index-profiles) (`/index-profiles`)\
Create, read, update, and delete index profiles.
## Index Profiles in the UI
To configure index profiles, click **Indexing** > **Index Profiles**.
## Learn more
1. In the Fusion workspace, navigate to **Indexing** > **Index Profiles**.
2. Click **New**.
The **Add Index Profile** panel appears.
3. Enter an **Index Profile ID**.
4. Select an index pipeline to associate with this profile.
5. Select a parser to associate with this profile.
6. Select a collection to associate with this profile.
7. Optionally, click **New params...** to enter Solr request parameters to add to the request URL when indexing documents using this profile.
8. Click **Save**.
The index profile window displays the request URL for sending documents to this profile.
The course for **Index and Query Profiles** focuses on how index and query profiles operate, and how to configure and manage them.
1. In the Fusion workspace, navigate to **Indexing** > **Index Profiles**.
2. In the profiles list, select the index profile to delete.
3. In the index profile configuration panel, click **Delete profile**.
# Index Pipelines
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/index-pipelines/overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-in/indexing/index-pipelines/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/index-pipelines/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/162
Index pipelines transform incoming data into `PipelineDocument` objects for indexing by Fusion-managed Solr service.
An index pipeline consists of a series of configurable
[index pipeline stages](/docs/5/fusion/getting-data-in/indexing/index-pipelines/index-pipeline-stages),
each performing a different transformation on the data before passing the result to the next stage in the pipeline.
The final stage is the
[Solr Indexer stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/solr-indexer-stage),
which transforms the PipelineDocument into a Solr document and submits it to Solr for indexing in a specific
[Collection](/docs/5/fusion/getting-data-in/indexing/collections/overview).
Each configured datasource has an associated index pipeline and uses a
[connector](/docs/fusion-connectors/connectors/overview)
to fetch data to parse and then input into the index pipeline.
Alternatively, documents can be submitted directly to an index pipeline or profile with the REST API; see
**Import Data with the REST API**.
It is often possible to get documents into Fusion by configuring a datasource with [the appropriate connector](/docs/5/fusion/getting-data-in/indexing/connectors).
But if there are obstacles to using connectors, it can be simpler to index documents with a REST API call to an index profile or pipeline.
## Push documents to Fusion using index profiles
[Index profiles](/docs/5/fusion/getting-data-in/indexing/index-pipelines/index-profiles) allow you to send documents to a consistent endpoint (the profile alias) and change the backend index pipeline as needed. The profile is also a simple way to use one pipeline for multiple collections without any one collection "owning" the pipeline.
You can send documents directly to an index using the [Index Profiles REST API](/docs/5/fusion/getting-data-in/indexing/index-pipelines/index-profiles). The request path is:
```bash theme={"dark"}
/api/apps/APP_NAME/index/INDEX_PROFILE
```
These requests are sent as a POST request. The request header specifies the format of the contents of the request body. Create an index profile in the Fusion UI.
To send a streaming list of JSON documents, you can send the JSON file that holds these objects to the API listed above with `application/json` as the content type. If your JSON file is a list or array of many items, the endpoint operates in a streaming way and indexes the docs as necessary.
### Send data to an index profile that is part of an app
Accessing an index profile through an app lets a Fusion admin secure and manage all objects on a per-app basis. Security is then determined by whether a user can access an app. This is the recommended way to manage permissions in Fusion.
The syntax for sending documents to an index profile that is part of an app is as follows:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/json' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE --data-binary @my-json-data.json
```
Spaces in an app name become underscores. Spaces in an index profile name become hyphens.
To prevent the terminal from displaying all the data and metadata it indexes--useful if you are indexing a large file--you can optionally append `?echo=false` to the URL.
Be sure to set the content type header properly for the content being sent. Some frequently used content types are:
* Text: `application/json`, `application/xml`
* PDF documents: `application/pdf`
* MS Office:
* DOCX: `application/vnd.openxmlformats-officedocument.wordprocessingml.document`
* XLSX: `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet`
* PPTX: `application/vnd.vnd.openxmlformats-officedocument.presentationml.presentation`
* More types: [http://filext.com/faq/office\_mime\_types.php](http://filext.com/faq/office_mime_types.php)
### Example: Send JSON data to an index profile under an app
In `$FUSION_HOME/apps/solr-dist/example/exampledocs` you can find a few sample documents. This example uses one of these, `books.json`.
To push JSON data to an index profile under an app:
1. Create an index profile. In the Fusion UI, click **Indexing > Index Profiles** and follow the prompts.
2. From the directory containing `books.json`, enter the following, substituting your values for username, password, and index profile name:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/json' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE?echo=false --data-binary @books.json
```
3. Test that your data has made it into Fusion:
1. Log into the Fusion UI.
2. Navigate to the app where you sent your data.
3. Navigate to the Query Workbench.
4. Search for `*:*`.
5. Select relevant Display Fields, for example `author` and `name`.
### Example: Send JSON data without defining an app
In most cases it is best to delegate permissions on a per-app basis. But if your use case requires it, you can push data to Fusion without defining an app.
To send JSON data without app security, issue the following curl command:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/json' https://FUSION_HOST:FUSION_PORT/api/index/INDEX_PROFILE --data-binary @my-json-data.json
```
### Example: Send XML data to an index profile with an app
To send XML data to an app, use the following:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/xml' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE --data-binary @my-xml-file.xml
```
In Fusion 5, documents can be created on the fly using the [PipelineDocument](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/apollo/common/pipeline/PipelineDocument.html) JSON notation.
## Remove documents
### Example 1
The following example removes content:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/vnd.lucidworks-document' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE --data-binary @del-json-data.json
```
### Example 2
A more specific example removes data from `books.json`. To delete "The Lightning Thief" and "The Sea of Monsters" from the index, use their id values in the JSON file.
The `del-json-data.json` file to delete the two books:
```json wrap theme={"dark"}
[{ "id": "978-0641723445","commands": [{"name": "delete","params": {}}]},{ "id": "978-1423103349","commands": [{"name": "delete","params": {}}, {"name": "commit","params": {}}]}]
```
You can use `?echo=false` to turn off the response to the terminal.
### Example 3
Another example to delete items using the Push API is:
```bash wrap theme={"dark"}
curl -u admin:XXX -X POST 'http://FUSION_HOST:FUSION_PORT/api/apps/APP/index/INDEX' -H 'Content-Type: application/vnd.lucidworks-document' -d '[
{
"id": "1663838589-44",
"commands":
[
{
"name": "delete",
"params":
{}
},
{
"name": "commit",
"params":
{}
}
]
}, ...
]'
```
## Send documents to an index pipeline
Although sending documents to an index profile is recommended, if your use case requires it, you can send documents directly to an index pipeline.
For more information about index pipeline REST API reference documentation, see [Fusion 5.x Index Pipelines API](/api-reference/index-pipelines-api/list-all-index-pipelines).
### Specify a parser
When you push data to a pipeline, you can specify the name of the parser by adding a parserId querystring parameter to the URL.
For example: `https://FUSION_HOST:FUSION_PORT/api/index-pipelines/INDEX_PIPELINE/collections/COLLECTION_NAME/index?parserId=PARSER`.
If you do not specify a parser, and you are indexing outside of an app (`https://FUSION_HOST:FUSION_PORT/api/index-pipelines/...`), then the `_system` parser is used.
If you do not specify a parser, and you are indexing in an app context (`https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index-pipelines/...`), then the parser with the same name as the app is used.
## Indexing CSV Files
In the usual case, to index a CSV or TSV file, the file is split into records, one per row, and each row is indexed as a separate document.
A pipeline can be reused across multiple collections. Fusion provides a set of built-in pipelines. You can use the Index Workbench or the
[REST API](/api-reference/index-pipelines-api/list-all-index-pipelines)
to develop custom index pipelines to suit any datasource or application.
Fusion creates a default index pipeline when you create an app. The index pipeline has the same name as the app. This pipeline consists of a [Field Mapping index stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/field-mapper-index-stage).
## Debugging
If your pipeline is not producing the desired result, these debugging tips can help you identify the source of the issue and resolve it.
### View parameters
When debugging a pipeline, it helps to see the parameters that are being passed to or from each stage.
There are several ways to exposed those parameters:
Follow these steps to add a Logging stage to your pipeline:
1. In Fusion UI, navigate to **Indexing > Index Pipelines** (for index pipelines) or **Querying > Query Pipelines** (for query pipelines).
2. Click **Add a new pipeline stage**.
3. Select **Logging** from the Troubleshooting section.
4. In the **Label** field, enter a descriptive name (for example, "Debug After Vectorize").
5. Set the **detailed** property to `true` to print the full Request or PipelineDocument object.
6. Place the Logging stage after the stage you want to debug.
7. Click **Save**.
8. Run your pipeline and retrieve logs for your pipeline type:
* Query pipelines: `kubectl logs -f -l app=query-pipeline -n NAMESPACE`
* Index pipelines: `kubectl logs -f -l app=fusion-indexing -n NAMESPACE`
If Grafana with Loki is installed, see [Grafana and Loki](/docs/5/fusion/operations/monitoring-and-reporting/grafana-loki) for query guidance.
Follow these steps to inspect context variables:
1. Add a **JavaScript** stage to your pipeline.
2. Use the `ctx` variable to inspect context data:
```javascript theme={"dark"}
function (request, response, ctx) {
// View all context variables
logger.info("Context: " + JSON.stringify(ctx));
// Check specific variable (for example, vector from LWAI stage)
if(ctx.get('vector') !== null) {
logger.info("Vector: " + ctx.get('vector'));
}
}
```
3. Retrieve logs from the `fusion-indexing` pod: `kubectl logs -f -l app=fusion-indexing -n NAMESPACE`
Follow these steps to debug using Query Workbench:
1. Navigate to **Querying > Query Workbench**.
2. Enter a test query and click **Search**.
3. Click the **Debug** tab. The debug view displays the following information:
* Request parameters
* Pipeline stage execution details
* Response data including `responseHeader` and `debug.explain`
4. Switch to **View As: JSON** to see the full response structure.
### Enable **Fail on Error**
The **Fail on Error** setting determines whether silent failures can occur in your pipeline.
By enabling **Fail on Error** during development, testing, and troubleshooting, you ensure that configuration issues, authentication problems, or model errors are immediately visible rather than producing incomplete or incorrect data that can be difficult to troubleshoot later.
The **Fail on Error** setting controls whether pipeline execution stops when an LWAI stage encounters an error. Follow these steps to configure this setting:
1. In Fusion UI, navigate to your pipeline (Index or Query).
2. Click the LWAI stage you want to configure (for example, **LWAI Vectorize Field** or **LWAI Vectorize Query**).
3. Locate the **Fail on Error** checkbox at the bottom of the stage configuration.
4. Select the checkbox to enable any of the following behaviors:
* Stop pipeline processing and throw an exception on errors
* Get immediate feedback when LWAI models fail or are misconfigured
* Guarantee data quality by preventing indexing of documents without vectors
In production environments, keep this feature disabled to ensure that service is not interrupted when errors occur.
6. Click **Save**.
During development and testing, enable **Fail on Error** to identify configuration issues early. In production, consider disabling **Fail on Error** if you want resilient pipelines that can handle occasional model failures. For critical vector fields, enable **Fail on Error** to ensure that all documents have required embeddings.
Follow these steps to verify your Fail on Error configuration:
1. Trigger an intentional error (for example, use an invalid model name or account).
2. Verify that the pipeline fails (when Fail on Error is enabled) or continues (when Fail on Error is disabled).
3. Retrieve logs from the `fusion-indexing` pod: `kubectl logs -f -l app=fusion-indexing -n NAMESPACE`
# The Indexing Workbench
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/index-workbench
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-in/indexing/overview-workbench
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/overview-workbench
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/3223
The Index Workbench is a powerful tool that combines key aspects of the data indexing configuration process into one user-friendly panel. It guides the user through the workflow for configuring datasources, parsers and index pipelines, and then running a datasource job to index your data.
## Datasources
A datasource is a complete data ingest configuration that consists of the following components:
* **Connector.** Connectors are out-of-the-box components for pulling your data into Fusion. Lucidworks provides a wide variety of connectors, each specialized for a particular data type. For a list of connectors, see [Fusion Connectors](/docs/fusion-connectors/overview).
* **Parser.** Parsers provide fine-grained configuration for inbound data.
* For Fusion 5.x, see [Parsers](/docs/5/fusion/reference/config-ref/parser-stages/overview) for more information.
* For Fusion 4.x, see [Parsers](/docs/4/fusion-server/reference/parser-stages/overview) for more information.
* **Index pipeline.** Index pipelines transform incoming data for indexing by Fusion.
* For Fusion 5.x, see [Index Pipelines](/docs/5/fusion/getting-data-in/indexing/index-pipelines/overview) for more information.
* For Fusion 4.x, see [Index Pipelines](/docs/4/fusion-server/concepts/indexing/datasources/index-pipelines) for more information.
To set up a new datasource in the Index Workbench, click **New**. The **Add A New Datasource** panel appears, offering three simplified methods of configuring how Fusion fetches your data:
* By URL
* By local file upload
* By selecting an installed connector to configure\
See [Connectors Configuration Reference](/docs/fusion-connectors/connectors/overview) for a complete list of available connectors. Some are installed by default, and others require separate download and installation.
## Parsers
[Parsers](/docs/5/fusion/reference/config-ref/parser-stages/overview) are a configurable component of the indexing workflow, for flexibility and specificity when parsing inbound data.
A parser consists of an ordered list of parser stages that is completely customizable. The same parser stage can be added to a given configuration multiple times if the different specified settings within those stages best suits the parsing of the data. There is no limit to the number of parser stages that can be included in a parser, and the order in which they run is also completely flexible.
In a parser, after all of the doctype-specific parser stages have run, the Tika and Fallback stages are useful catch-all stages that can attempt to parse anything that has not yet been matched. Tika is used for parsing many types of unstructured documents like PDFs, DOCX, and many more. If all of the other stages in the parser fail to completely parse the data, the Fallback stage can copy the raw bytes directly to Solr.
## Index pipelines
An [Index Pipeline](/docs/5/fusion/getting-data-in/indexing/index-pipelines/overview) transforms incoming data into a document suitable for indexing by Solr via a series of modularized operations called stages. Fusion provides a variety of specialized index stages to index data effectively. Stages can be selected, configured, and enabled or disabled in the Index Pipeline section of the Index Workbench.
Once you finish configuring a datasource using the Index Workbench, you can move on to setting up queries using the [Query Workbench](/docs/5/fusion/intro/ui-tour/query-workbench), which provides a similar workflow for configuring and previewing search results.
### Adding a pipeline stage
Click **Add a Stage** to add index pipeline stages that can perform data filtering, transformations, and more.
The [Solr Indexer stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/solr-indexer-stage)
is the only pipeline stage that is required to complete the indexing process, and therefore exists in every query pipeline. It is always the last stage in a series.
### Re-ordering pipeline stages
The order of the pipeline stages matters, because the output from one stage becomes the input to the next stage. For example, the
[Solr Indexer stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/solr-indexer-stage)
must always come last in the sequence, so that data is indexed only after it has been processed by all other stages. Putting this stage first in the sequence means that subsequent stages have no effect on the indexed data.
Click and drag any stage in the pipeline to move it up or down in the sequence of stages. The preview panel automatically updates the results to reflect the output of the new sequence.
### Enabling and disabling pipeline stages
By default, every stage in an index pipeline is enabled. While working with a pipeline, it can be helpful to disable stages without removing them completely. This allows you to preserve a stage’s configuration while observing how the search results change in its absence. You can re-enable the stage at any time. When you save a query pipeline, the enabled/disabled state of each stage is also saved.
Click the circle next to any stage in order to enable or disable it.
# Indexing Data Flow
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/indexing-data-flow
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-in/indexing/overviewing-data-flow
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/overviewing-data-flow
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/3222
With Fusion, your data is indexed in a set of collections in Fusion’s Solr core.
A *primary collection* holds your searchable content, such as your product catalog, knowledge base, blog articles, product reviews, and so on.
A set of *secondary collections* are associated with your primary collection to hold related data that Fusion can use to enhance the relevancy of your search results.
This topic shows you how different types of data flow through Fusion to be indexed in your primary and secondary collections.
## 1. Index your content
No matter where your content is located, Fusion can index it. The Fusion collection where your searchable content is indexed is called the primary collection. Secondary collections are automatically created to hold related data, such as signals, machine learning job output, and so on.
There are a few ways to index your searchable content to the primary collection:
* [Connectors](/docs/fusion-connectors/overview): Lucidworks has a wide variety of connectors for many types of data sources. [Find your connector](/docs/fusion-connectors/overview). Once the connector fetches your data, [parsers](/docs/5/fusion/reference/config-ref/parser-stages/overview) read it before passing it to the [index pipeline](/docs/5/fusion/getting-data-in/indexing/index-pipelines/overview).
* The gRpc remote framework is configured on the client side and only works with V2 type connectors.
* V1 classic connectors support ingesting data using IP white lists, VPN tunnels, and public channels.
* The [Parallel Bulk Loader (PBL)](/docs/5/fusion/getting-data-in/parallel-bulk-loader): The PBL can send your data to an index pipeline *or* directly to the primary collection, depending on whether the data requires transformation before indexing. It does not support parsers and is not recommended for production environments.
* Import Data with the REST API: Send your content to an index profile using the Fusion REST API. Index profiles are saved configurations of parsers and index pipelines.
It is often possible to get documents into Fusion by configuring a datasource with [the appropriate connector](/docs/5/fusion/getting-data-in/indexing/connectors).
But if there are obstacles to using connectors, it can be simpler to index documents with a REST API call to an index profile or pipeline.
## Push documents to Fusion using index profiles
[Index profiles](/docs/5/fusion/getting-data-in/indexing/index-pipelines/index-profiles) allow you to send documents to a consistent endpoint (the profile alias) and change the backend index pipeline as needed. The profile is also a simple way to use one pipeline for multiple collections without any one collection "owning" the pipeline.
You can send documents directly to an index using the [Index Profiles REST API](/docs/5/fusion/getting-data-in/indexing/index-pipelines/index-profiles). The request path is:
```bash theme={"dark"}
/api/apps/APP_NAME/index/INDEX_PROFILE
```
These requests are sent as a POST request. The request header specifies the format of the contents of the request body. Create an index profile in the Fusion UI.
To send a streaming list of JSON documents, you can send the JSON file that holds these objects to the API listed above with `application/json` as the content type. If your JSON file is a list or array of many items, the endpoint operates in a streaming way and indexes the docs as necessary.
### Send data to an index profile that is part of an app
Accessing an index profile through an app lets a Fusion admin secure and manage all objects on a per-app basis. Security is then determined by whether a user can access an app. This is the recommended way to manage permissions in Fusion.
The syntax for sending documents to an index profile that is part of an app is as follows:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/json' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE --data-binary @my-json-data.json
```
Spaces in an app name become underscores. Spaces in an index profile name become hyphens.
To prevent the terminal from displaying all the data and metadata it indexes--useful if you are indexing a large file--you can optionally append `?echo=false` to the URL.
Be sure to set the content type header properly for the content being sent. Some frequently used content types are:
* Text: `application/json`, `application/xml`
* PDF documents: `application/pdf`
* MS Office:
* DOCX: `application/vnd.openxmlformats-officedocument.wordprocessingml.document`
* XLSX: `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet`
* PPTX: `application/vnd.vnd.openxmlformats-officedocument.presentationml.presentation`
* More types: [http://filext.com/faq/office\_mime\_types.php](http://filext.com/faq/office_mime_types.php)
### Example: Send JSON data to an index profile under an app
In `$FUSION_HOME/apps/solr-dist/example/exampledocs` you can find a few sample documents. This example uses one of these, `books.json`.
To push JSON data to an index profile under an app:
1. Create an index profile. In the Fusion UI, click **Indexing > Index Profiles** and follow the prompts.
2. From the directory containing `books.json`, enter the following, substituting your values for username, password, and index profile name:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/json' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE?echo=false --data-binary @books.json
```
3. Test that your data has made it into Fusion:
1. Log into the Fusion UI.
2. Navigate to the app where you sent your data.
3. Navigate to the Query Workbench.
4. Search for `*:*`.
5. Select relevant Display Fields, for example `author` and `name`.
### Example: Send JSON data without defining an app
In most cases it is best to delegate permissions on a per-app basis. But if your use case requires it, you can push data to Fusion without defining an app.
To send JSON data without app security, issue the following curl command:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/json' https://FUSION_HOST:FUSION_PORT/api/index/INDEX_PROFILE --data-binary @my-json-data.json
```
### Example: Send XML data to an index profile with an app
To send XML data to an app, use the following:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/xml' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE --data-binary @my-xml-file.xml
```
In Fusion 5, documents can be created on the fly using the [PipelineDocument](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/apollo/common/pipeline/PipelineDocument.html) JSON notation.
## Remove documents
### Example 1
The following example removes content:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/vnd.lucidworks-document' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE --data-binary @del-json-data.json
```
### Example 2
A more specific example removes data from `books.json`. To delete "The Lightning Thief" and "The Sea of Monsters" from the index, use their id values in the JSON file.
The `del-json-data.json` file to delete the two books:
```json wrap theme={"dark"}
[{ "id": "978-0641723445","commands": [{"name": "delete","params": {}}]},{ "id": "978-1423103349","commands": [{"name": "delete","params": {}}, {"name": "commit","params": {}}]}]
```
You can use `?echo=false` to turn off the response to the terminal.
### Example 3
Another example to delete items using the Push API is:
```bash wrap theme={"dark"}
curl -u admin:XXX -X POST 'http://FUSION_HOST:FUSION_PORT/api/apps/APP/index/INDEX' -H 'Content-Type: application/vnd.lucidworks-document' -d '[
{
"id": "1663838589-44",
"commands":
[
{
"name": "delete",
"params":
{}
},
{
"name": "commit",
"params":
{}
}
]
}, ...
]'
```
## Send documents to an index pipeline
Although sending documents to an index profile is recommended, if your use case requires it, you can send documents directly to an index pipeline.
For more information about index pipeline REST API reference documentation, see [Fusion 5.x Index Pipelines API](/api-reference/index-pipelines-api/list-all-index-pipelines).
### Specify a parser
When you push data to a pipeline, you can specify the name of the parser by adding a parserId querystring parameter to the URL.
For example: `https://FUSION_HOST:FUSION_PORT/api/index-pipelines/INDEX_PIPELINE/collections/COLLECTION_NAME/index?parserId=PARSER`.
If you do not specify a parser, and you are indexing outside of an app (`https://FUSION_HOST:FUSION_PORT/api/index-pipelines/...`), then the `_system` parser is used.
If you do not specify a parser, and you are indexing in an app context (`https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index-pipelines/...`), then the parser with the same name as the app is used.
## Indexing CSV Files
In the usual case, to index a CSV or TSV file, the file is split into records, one per row, and each row is indexed as a separate document.
When using the REST API for synchronous parsing, you can send content directly to the Fusion indexing endpoint for Fusion to process. You can designate an index profile that consists of a parser, collection, and a pipeline. If you do not want to use an index profile, you can specify the parser, collection, and pipeline in the API call.
When using asynchronous parsing, you can send content directly to the Fusion indexing endpoint for Fusion to process. You can designate an index profile that consists of a parser, collection, and a pipeline. If you do not want to use an index profile, you can specify the parser, collection, and pipeline in the API call.
Asynchronous parsing only uses V2 connectors.
The [index pipeline](/docs/5/fusion/getting-data-in/indexing/index-pipelines/overview) consists of one or more configurable [index pipeline stages](/docs/5/fusion/getting-data-in/indexing/index-pipelines/index-pipeline-stages), each performing a different type of transformation on the incoming data. Each connector has a default index pipeline, but you can modify these or create new ones.
The last stage in any index pipeline should be the [Solr Indexer](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/solr-indexer-stage) stage, which submits the documents to Solr for indexing.
## 2. Index your signals
Signals are event records that provide historical data about user behavior, such as clicks, likes, purchases, and so on. You don’t need to index signals about query responses; Fusion indexes response signals automatically by default. If you are using [App Studio](/docs/5/fusion/dev-portal/appkit/overview) or [App Insights](/docs/5/fusion/getting-data-out/data-analytics/app-insights/overview), then you need to index request signals. Learn more about [signal types and required fields](/docs/5/fusion/getting-data-out/query-enhancement/signals/types-and-structures).
To index your signals, you send them to Fusion using the [Signals API](/api-reference/signal-indexing-api/index-an-event-signal), which points to the hidden index pipeline designed especially for signals.
**Indexing your signals**
Raw signals are indexed in a secondary collection called `COLLECTION_NAME_signals`. For example, if your primary collection is called `Products`, then the raw signals collection is `Products_signals`.
## 3. Fusion jobs that index signals-related data
When you **Enable or Disable Signals**, Fusion creates jobs and secondary collections for analyzing and aggregating your raw signals. Some of this data enables [query rewriting](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) and [automatic boosting](/docs/5/fusion/getting-data-out/query-enhancement/boosting/boost-with-signals), while other data becomes useful when you enable [recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/overview).
You can enable and disable signals using the Fusion UI or the REST API.
When you disable signals, the [aggregation jobs](/docs/4/fusion-ai/concepts/signals-and-aggregations/aggregations/sql-aggregations) are deleted, but the `_signals` and `_signals_aggr` collections are not, your legacy signal data remains intact.
## Using the UI
When you create a collection using the Fusion UI, signals are enabled and a signals collection created by default. You can also enable and disable signals for existing collections using the Collections Manager.
**Enable signals for a collection**
1. In the Fusion workspace, navigate to **Collections** > **Collections Manager**.
2. Hover over the primary collection for which you want to enable signals.
3. Click **Configure** to open the drop-down menu.
4. Click **Enable Signals**.\
The **Enable Signals** window appears, with a list of collections and jobs that are created when you enable signals.
5. Click **Enable Signals**.
**Disable signals for a collection**
1. In the Fusion workspace, navigate to **Collections** > **Collections Manager**.
2. Hover over the primary collection for which you want to disable signals.
3. Click **Configure** to open the drop-down menu.
4. Click **Disable Signals**.\
The **Disable Signals** window appears, with a list of jobs that are created when you enable signals.
5. Click **Disable Signals**.\
Your `_signals` and `_signals_aggr` collections remain intact so that you can access your legacy signals data.
## Using the Collection Features API
Using the API, the [`/collections/{collection}/features/{feature}`](/api-reference/collections/get-collection-features) endpoint enables or disables signals for any collection:
**Check whether signals are enabled for a collection**
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD http://localhost:{api-port}/api/collections/COLLECTION_NAME/features/signals
```
**Enable signals for a collection**
```json wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X PUT -H "Content-type: application/json" -d '{"enabled" : true}' http://localhost:{api-port}/api/collections/COLLECTION_NAME/features/signals
```
**Disable signals for a collection**
```json wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X PUT -H "Content-type: application/json" -d '{"enabled" : false}' http://localhost:{api-port}/api/collections/COLLECTION_NAME/features/signals
```
**Fusion jobs that index signals-related data**
* The [`_session_rollup` job](/docs/5/fusion/reference/config-ref/jobs/aggregations/built-in-sql-aggregation-jobs#collection-name-session-rollup) creates session signals from your raw signals and adds them to the `_signals` collection.
* [Automatic query rewrites](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) are created by [several jobs](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) and indexed in the `_query_rewrite_staging` collection for review before they are published to the `_query_rewrite` collection.
* The [`_head_tail` job](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis) sends [rewrites for underperforming queries](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) to the `_query_rewrite_staging` collection. It also sends analytics tables to the `_job_reports` collection.
* The `_signals_aggr` collection stores unique combinations of query, document ID, and filters found in your raw signals, each with a timestamp, weight, and count. These are generated by the [`_click_signals_aggregation` job](/docs/5/fusion/reference/config-ref/jobs/aggregations/overview) and the [`_user_query_history_agg` job](/docs/5/fusion/reference/config-ref/jobs/aggregations/built-in-sql-aggregation-jobs).
* The [`user_item_prefs_agg` job](/docs/5/fusion/reference/config-ref/jobs/aggregations/built-in-sql-aggregation-jobs) identifies unique pairs of document IDs and user IDs, assigns a timestamp, weight, and count to each pair, and indexes it in the `_recs_aggr` collection.
## 4. Fusion jobs that index recommendations
When you [enable recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/getting-started#enable-recommendations), another set of jobs and secondary collections is created.
**Fusion jobs that index recommendations**
The default recommendation jobs read data from different collections depending on the type of recommendations being generated:
* [Content-based recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-item) are produced by the [`_content_recs` job](/docs/5/fusion/reference/config-ref/jobs/content-based-recommender) by analyzing your primary collection.
* [Queries-for-query recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/queries-for-query) are produced by the [`_query_recs` job](/docs/5/fusion/reference/config-ref/jobs/query-to-query-session-based-similarity) by analyzing your raw signals.
* BPR-based [items-for-item](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-item#collaborative-items-for-item-recommendations) and [items-for-user](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-user) recommendations are produced by the [`_bpr_item_recs` job](/docs/5/fusion/reference/config-ref/jobs/bpr-recommender) by analyzing aggregated signals.
For more recommendations, try **Identify Trending Items or Queries**.
The Trending Recommender job analyzes signals to measure customer engagement over time. Use this job to identify spikes in popularity for specific items or queries, then display those items to your users or analyze the trends for business purposes. You can configure any time window, such as daily, weekly, or monthly.
For complete details about the job’s configuration options, see [Trending Recommender Jobs](/docs/5/fusion/reference/config-ref/jobs/trending-recommender).
**How to identify trending items or queries**
1. Navigate to **Collections** > **Jobs** > **Add +** > **Trending Recommender**.
2. Configure the job:
1. Enter an ID for this job.
2. In the **Reference Time Days** field, enter the number of days to use as a *baseline* for identifying trends, starting from today.\
For example, enter 21 days to analyze three weeks of signals data to use as a baseline.
3. In the **Target Time Days** field, enter the number of days to use as a *target* for identifying trends, starting from today.\
For example, enter 7 days to get documents or products whose popularity has spiked in the past week.
**Reference Time Days** and **Target Time Days** do not overlap. For example, with the values suggested above, a total of 28 days of signals are analyzed, and the first 21 days are compared to the last 7 days.
4. If you want to identify trending queries instead of trending items, change the value of the **Document ID Field** from `doc_id_s` to `query_s`.
This field must be present in your signals. See the required signals fields in the [Trending Recommender Jobs](/docs/5/fusion/reference/config-ref/jobs/trending-recommender) reference topic.
5. In the **Training Collection** field, enter the Solr collection or cloud path where signals are stored (the `COLLECTION_NAME_signals` collection by default).
6. In the **Output Collection** field, enter the Solr collection or cloud path where trend analysis data will be stored.
7. If you are using a format other than `solr`, enter it in the **Data Format** field.
8. In the **Solr Fields to Read** field, enter one or more field names containing text training data.
You can enter multiple field names with weights, as in `field1:weight1,field2:weight2...`.
9. In the **Event Count Field Name** field, enter the name of the event count field in your training data, usually `count_i`.
3. Click **Save**.
4. Click **Run** > **Start** to run the job.
The job outputs documents similar to this example:
```json theme={"dark"}
{
"doc_id":250,
"ref_hits":1,
"ref_rank":1,
"trgt_hits":1,
"trgt_rank":1,
"vol_diff":0.5,
"average_weekly_vol":0.5,
"hit_vol_ratio":2.0,
"combine_score":1.0,
"vol_diff_ratio":1.0,
"ref_wt_vol_diff_ratio":1.0,
"vol_diff_wt_vol_diff_ratio":0.5,
"log_diff_wt_ratio":1.3068528194400546,
"trend-type":"prds_weekly",
"id":"284f930d-d750-49a2-90ac-be4692bddda9",
"_version_":1682995654191742976
}
```
5. Configure a query pipeline to retrieve trending items from the job’s output collection for display or further analysis. For information about pipelines created when recommendations are enabled, see [enable recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/getting-started).
Search on the `log_diff_wt_ratio` field to find the top trending items in the output collection.
## What’s next?
See [Query Data Flow](/docs/5/fusion/intro/query-dataflow) to learn about querying your content and recommendations.
# Indexing Signals
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/indexing-signals/overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-in/indexing/indexing-signals/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/indexing-signals/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/3225
Signals are indexed just like other data, but instead of using a connector, you use the [Signals API](/api-reference/signal-indexing-api/index-an-event-signal).
This page describes Fusion's native signals system.
For Platform Analytics signals integration, see [Platform Signals Integration](/docs/5/fusion/getting-data-in/platform-signals/overview).
## Signals data flow
This diagram shows the flow of signals data from the search app through Fusion. The numbered steps are explained below.
1. The search app sends a query to a Fusion query pipeline.\
The query request should include a user ID and session query parameter to identify the user.
2. Optionally, the Fusion query pipeline queries the `COLLECTION_NAME_signals_aggr` collection to get boosts for the main query based on aggregated click data.
3. The search app also sends a request signal to the Fusion [`/signals` endpoint](/api-reference/signal-indexing-api/index-an-event-signal).\
The primary intent of a request signal is to capture the raw user query and contextual information about the user’s current activity in the app, such as the user agent and the page where they generated the query. The request signal does not contain any information about the results sent to Solr; it is created before a query is processed.
4. Once Solr returns the response to Fusion, the SearchLogger component indexes the complete request/response data into the `COLLECTION_NAME_signals` collection as a response signal using the `_signals_ingest` pipeline. Therefore, the response signal captures all results from Fusion as it related to the original query.
This is a departure from pre-4.0 versions of Fusion where query impressions were logged in a separate `COLLECTION_NAME_logs` collection. Query activity is no longer indexed into the `_logs` collection. All response signals use the `fusion_query_id` (see below) as the unique document ID in Solr.
5. When the user clicks a link in the search results, the search app sends a click event to the Fusion signals endpoint (which invokes the `_signals_ingest` pipeline behind the scenes).\
The click signal *must* include a field named `fusion_query_id` in the `params` object of the raw click signal. The `fusion_query_id` field is returned in the query response (from step 1) in a response header named `x-fusion-query-id`. This allows Fusion to associate a click signal with the response signal generated in step 4. The `fusion_query_id` is also used by Fusion to associate click signals with [experiments](/docs/5/fusion/getting-data-out/data-analytics/experiments/overview). For experiments to work, each click signal must contain the corresponding `fusion_query_id` that produced the document/item that was clicked.
6. The `_signals_ingest` pipeline enriches signals before indexing into the `COLLECTION_NAME_signals` collection.\
This enrichment includes field mapping, geolocation resolution, and updating the `has_clicks` flag to "true" on request signals when the first click signal is encountered for a given request using the [Update Related Document index stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/update-related-document-index-stage).
7. Fusion queries the `COLLECTION_NAME_signals` collection through a Fusion query pipeline to generate query analytics reports from raw signals.
8. Behind the scenes, the [SQL aggregation](/docs/5/fusion/reference/config-ref/jobs/aggregations/sql-aggregation) framework aggregates click signals to compute a weight for each query + `doc_id` + filters group.\
The resulting metrics are saved to the `COLLECTION_NAME_signals_aggr` collection to generate boosts on queries to the main collection (step 2 above).
9. Recommendations also use aggregated documents in the `COLLECTION_NAME_signals_aggr` collection to build a [collaborative filtering-based recommender model](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods).
## Default index pipeline for signals
When indexing signals of any type for any Fusion app, Fusion always uses a default index pipeline named `_signals_ingest` unless you explicitly specify a different index pipeline.
Because this pipeline is not associated with any Fusion app, it does not automatically appear in the list of index pipelines. You can find it in the [Object Explorer](/docs/5/fusion/intro/ui-tour/object-explorer) by clicking the **In No Apps** filter.
### Default stages
The `_signals_ingest` index pipeline has several stages:
1. [Format Signals stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/signal-formatter-index-stage)
2. [Field Mapping stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/field-mapper-index-stage)
3. [GeoIP Lookup stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/geoip-lookup-index-stage)
4. [Solr Indexer stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/solr-indexer-stage)
5. Update `has_clicks` flag stage\
The Update `has_clicks` flag stage is an instance of the [Update Related Document stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/update-related-document-index-stage) that updates the `has_clicks` flag to "true" on an existing request signal after the first click signal is processed for the request.
The Update `has_clicks` flag stage works as follows:
1. When a click signal is encountered (`type==click`)
2. Look at the incoming click signal for a field named `request_id_s`, which gets set by the Format Signals stage using a distributed cache of recently processed request signals.\
If the `request_id_s` field is set, then send a real-time `GET` query to Solr to find a request signal with ID equal to the value of the `request_id_s` field on the click signal. To avoid re-updating request signals, the RTG query also filters on `has_clicks==false`, which avoids duplicate atomic updates on the same document in Solr. Real-time `GET` is used to avoid timing issues between a request signal being sent to Solr and when it gets committed. This prevents missing updates when clicks occur soon after the initial request signal is sent by the search app.
3. If the click signal does not have the `request_id_s` field set, then do a normal Solr lookup for the request signal using: `+query_id:"${query_id}" +type:request +has_clicks:false`. A click signal may not have a `request_id_s` if there is a cache miss in the distributed cache used by the Format Signals stage.
4. If the stage performs a normal query, there may be multiple request signals that have the same `query_id`. This is because the `query_id` is based on `session` + `query` + `filter`, so if a user sends the same `query` + `filter` during the same session, there will be multiple request signals with the same `query_id` value. Thus, the stage sorts to get the latest request signal to update.
5. If a related document is found (in this case a request signal), then the stage updates the `has_clicks` field to true and performs an atomic update in Solr.
This stage performs its work in a background thread, so it does not impact the indexing performance of the click signal.
# Indexing Your Data
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-in/indexing/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/indexing/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/155
Indexing stores your data in a format that is optimized for searching. How you index your data is critical for ensuring that your data is stored in a format that is ideal for your search application.
* [Index pipelines](/docs/5/fusion/getting-data-in/indexing/index-pipelines/overview) determine the details of the conversion.
* [Index pipeline stages](/docs/5/fusion/getting-data-in/indexing/index-pipelines/index-pipeline-stages) are the building blocks of index pipelines.
* The [Index Workbench](/docs/5/fusion/intro/ui-tour/index-workbench) is Fusion’s index pipeline development tool.
### Collections
Collections are a way of grouping data sets so that related data sets can be managed together. Every data set that you ingest belongs to a collection. Any app can contain one or more collections. See [Collection Management](/docs/5/fusion/getting-data-in/indexing/collections/overview).
### Datasources
Datasources are configurations that determine how your data is handled during ingest by Fusion’s connectors, parsers, and index pipelines. When you run a fully-configured datasource, the result is an indexed data set that is optimized for search, depending on the shape of your data and how you want to search it. See **Configure a New Datasource**.
1. In your Fusion app, navigate to **Indexing** > **Datasources**.
2. Click **Add**.
3. Select your connector.
The connector configuration panel appears. The specific configuration options vary depending on the connector.
If you do not see your connector in the list, you may need to install it.
1. Configure your connector’s options:
1. Enter a useful name for your datasource in the **Datasource ID** field.
2. You can leave the **Pipeline ID** and **Parser** fields as-is to start with the defaults, or select different ones if you have them.
3. Configure the connector-specific options.
Select your specific release and connector detail from the [configuration reference topic](/docs/fusion-connectors/connectors/overview).
2. Click **Save**.
3. Test your datasource configuration:
1. Navigate to the [Index Workbench](/docs/5/fusion/intro/ui-tour/index-workbench) at **Indexing** > **Index Workbench**.
2. Click **Load...**.
3. Select the datasource ID you specified when you created the datasource.\
Now you can see your datasource configuration and a simulation of the results you can expect when you run this datasource job to index your data.
4. Adjust the configurations of your datasource, parsers, and index pipeline until the simulated results are satisfactory.
5. Click **Save**.
4. Index your data:
1. In the Index Workbench, click **Start Job**.
2. When the job status is **Finished**, navigate to the [Query Workbench](/docs/5/fusion/getting-data-out/query-basics/query-workbench) at **Querying** > **Query Workbench** to view the indexed documents and configure your query pipeline.
### Connectors
Connectors are Fusion components that ingest and parse specific kinds of data. There is a Fusion [connector](/docs/fusion-connectors/concepts/datasources) for just about any data type.
## Learn more
The quick learning for **Indexing Menu Tour** focuses on the Indexing Menu features and functionality along with a brief description of each screen available in the menu.
The quick learning for **Anatomy of Indexing** focuses on the Fusion objects that work together to index your data.
The course for **Indexing Data** focuses on how to ingest and store your data in a format that’s optimized for search.
# Getting Data In
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-in/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/3218
There are many methods for getting data into Fusion, depending on the type of data:
* Your searchable content, [signals](/docs/5/fusion/getting-data-out/query-enhancement/signals/overview), and logs are all ingested by [indexing them](/docs/5/fusion/getting-data-in/indexing/overview) in Solr.
* [Connectors](/docs/fusion-connectors/overview), [machine learning models](/docs/5/fusion/intro/machine-learning/machine-learning-models), JDBC drivers, analytics catalogs, and [Banana dashboards](/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/overview) can be uploaded to the blob store.
* The [Parallel Bulk Loader (PBL)](/docs/5/fusion/getting-data-in/parallel-bulk-loader) can perform distributed reads on any data source that implements the SparkSQL (2.2.1 or later) Data Sources API.
* In some cases, you might find that it’s best to use other ingestion methods, such as
**Import Data with Hive**, **Import Data with Pig**, or pushing data to a **Import Data with the REST API** endpoint.
The JDBC V2 connector is supported, and fetches documents from a relational database via SQL queries. Under the hood, this connector implements the Solr [DataImportHandler (DIH)](https://wiki.apache.org/solr/DataImportHandler) plugin.
Fusion stores JDBC drivers in the blob store. You can upload a driver using the Fusion UI or the Blob Store API.
## How to upload a JDBC driver using the Fusion UI
1. In the Fusion UI, navigate to **System** > **Blobs**.
2. Click **Add**.
3. Select **JDBC Driver**.
The "New 'JDBC Driver' Upload" panel appears.
4. Click **Choose File** and select the .jar file from your file system.
5. Click **Upload**.
The new driver’s blob manifest appears.
From this screen you can also delete or replace the driver.
## How to install a JDBC driver using the API
1. Upload the JAR file to Fusion’s blob store using the [`/blobs/{id}` endpoint](/api-reference/blobs/upload-a-blob).
Specify an arbitrary blob ID, and a `resourceType` value of `plugin:connector`, as in this example:
```bash theme={"dark"}
curl -u USERNAME:PASSWORD -H "content-type:application/java-archive" -H "content-length:707261" -X PUT --data-binary @postgresql-42.0.0.jar http://localhost:8764/api/blobs/mydriver?resourceType=driver:jdbc
```
Success response:
```json theme={"dark"}
{
"name" : "mydriver",
"contentType" : "application/java-archive",
"size" : 707261,
"modifiedTime" : "2017-06-09T19:00:48.919Z",
"version" : 0,
"md5" : "c67163ca764bfe632f28229c142131b5",
"metadata" : {
"subtype" : "driver:jdbc",
"drivers" : "org.postgresql.Driver",
"resourceType" : "driver:jdbc"
}
}
```
Fusion automatically publishes the event to the cluster, and the listeners perform the driver installation process on each node.
If the blob ID is identical to an existing one, the old driver will be uninstalled and the new driver will installed in its place. To get the list of existing blob IDs, run: `curl -u USERNAME:PASSWORD https://FUSION_HOST:FUSION_PORT/api/blobs`
2. To verify the uploaded driver, run:
```bash theme={"dark"}
curl -u USERNAME:PASSWORD https://FUSION_HOST:FUSION_PORT/api/blobs/BLOB_ID/manifest
```
Where the `BLOB_ID` is the name specified during upload, such as "mydriver" above. A success response looks like this:
```json theme={"dark"}
{
"name" : "mydriver",
"contentType" : "application/java-archive",
"size" : 707261,
"modifiedTime" : "2017-06-09T19:05:17.897Z",
"version" : 1569755095787110400,
"md5" : "c67163ca764bfe632f28229c142131b5",
"metadata" : {
"subtype" : "driver:jdbc",
"drivers" : "org.postgresql.Driver",
"resourceType" : "driver:jdbc"
}
}
```
Fusion ships with a Serializer/Deserializer (SerDe) for Hive, included in the distribution as `lucidworks-hive-serde-v2.2.6.jar` in `$FUSION_HOME/apps/connectors/resources/lucid.hadoop/jobs`.
For Fusion 4.1.x and 4.2.x, the preferred method of importing data with Hive is to use the Parallel Bulk Loader. The import procedure does not apply to Fusion 5.x.
## Features
* Index Hive table data to Solr.
* Read Solr index data to a Hive table.
* Kerberos support for securing communication between Hive and Solr.
* As of v2.2.4 of the SerDe, integration with Fusion is supported.
\*Fusion's index pipelines can be used to index data to Fusion.
\*Fusion's query pipelines can be used to query Fusion's Solr instance for data to insert into a Hive table.
## Add the SerDe Jar to Hive Classpath
In order for the Hive SerDe to work with Solr, the SerDe jar must be added to Hive's classpath using the `hive.aux.jars.path` capability. There are several options for this, described below.
It's considered a best practice to use a single directory for all auxiliary jars you may want to add to Hive so you only need to define a single path. However, you must then copy any jars you want to use to that path.
The following options all assume you have created such a directory at `/usr/hive/auxlib`; if you use another path, update the path in the examples accordingly.
If you use Hive with Ambari (as with the Hortonworks HDP distribution), go to **menu:Hive\[Configs > Advanced]**, and scroll down to **menu:Advanced hive-env\[hive-env template]**. Find the section where the `HIVE_AUX_JARS_PATH` is defined, and add the path to each line which starts with `export`. What you want will end up looking like:
```java wrap theme={"dark"}
# Folder containing extra libraries required for hive compilation/execution can be controlled by:
if [ "${HIVE_AUX_JARS_PATH}" != "" ]; then
if [ -f "${HIVE_AUX_JARS_PATH}" ]; then
export HIVE_AUX_JARS_PATH=${HIVE_AUX_JARS_PATH},/usr/hive/auxlib
elif [ -d "/usr/hdp/current/hive-webhcat/share/hcatalog" ]; then
export HIVE_AUX_JARS_PATH=/usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar,/usr/hive/auxlib
fi
elif [ -d "/usr/hdp/current/hive-webhcat/share/hcatalog" ]; then
export HIVE_AUX_JARS_PATH=/usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar,/usr/hive/auxlib
fi
```
If you are not using Ambari or a similar cluster management tool, you can add the jar location to `hive/conf/hive-site.xml`:
```xml theme={"dark"}
hive.aux.jars.path/usr/hive/auxlib
```
Another option is to launch Hive with the path defined with the `auxpath` variable:
hive --auxpath /usr/hive/auxlib
There are also other approaches that could be used. Keep in mind, though, that the jar *must* be loaded into the classpath, adding it with the `ADD JAR` function is not sufficient.
### Indexing Data to Fusion
If you use Lucidworks Fusion, you can index data from Hive to Solr via Fusion's index pipelines. These pipelines allow you several options for further transforming your data.
If you are using Fusion v3.0.x, you already have the Hive SerDe in Fusion's `./apps/connectors/resources/lucid.hadoop/jobs` directory. The SerDe jar that supports Fusion is v2.2.4 or higher. This was released with Fusion 3.0.
If you are using Fusion 3.1.x and higher, you will need to download the Hive SerDe from [http://lucidworks.com/connectors/](http://lucidworks.com/connectors/). Choose the proper Hadoop distribution and the resulting .zip file will include the Hive SerDe.
A 2.2.4 or higher jar built from this repository will also work with Fusion 2.4.x releases.
This is an example Hive command to create an external table to index documents in Fusion and to query the table later.
```bash wrap theme={"dark"}
hive> CREATE EXTERNAL TABLE fusion (id string, field1_s string, field2_i int)
STORED BY 'com.lucidworks.hadoop.hive.FusionStorageHandler'
LOCATION '/tmp/fusion'
TBLPROPERTIES('fusion.endpoints' = 'http://localhost:8764/api/apollo/index-pipelines//collections//index',
'fusion.fail.on.error' = 'false',
'fusion.buffer.timeoutms' = '1000',
'fusion.batchSize' = '500',
'fusion.realm' = 'KERBEROS',
'fusion.user' = 'fusion-indexer@FUSIONSERVER.COM',
'java.security.auth.login.config' = '/path/to/JAAS/file',
'fusion.jaas.appname' = 'FusionClient',
'fusion.query.endpoints' = 'http://localhost:8764/api/apollo/query-pipelines/pipeline-id/collections/collection-id',
'fusion.query' = '*:*');
```
In this example, we have created an external table named "fusion", and defined a custom storage handler (`STORED BY 'com.lucidworks.hadoop.hive.FusionStorageHandler'`) that a class included with the Hive SerDe jar designed for use with Fusion.
Note that all of the same caveats about field types discussed in the section `<>` apply to Fusion as well. In Fusion, however, you have the option of using an index pipeline to perform specific field mapping instead of using dynamic fields.
The LOCATION indicates the location in HDFS where the table data will be stored. In this example, we have chosen to use `/tmp/fusion`.
In the section TBLPROPERTIES, we define several properties for Fusion so the data can be indexed to the right Fusion installation and collection:
* `fusion.endpoints`:
The full URL to the index pipeline in Fusion. The URL should include the pipeline name and the collection data will be indexed to.
* `fusion.fail.on.error`:
If `true`, when an error is encountered, such as if a row could not be parsed, indexing will stop. This is `false` by default.
* `fusion.buffer.timeoutms`:
The amount of time, in milliseconds, to buffer documents before sending them to Fusion. The default is 1000. Documents will be sent to Fusion when either this value or `fusion.batchSize` is met.
* `fusion.batchSize`:
The number of documents to batch before sending the batch to Fusion. The default is 500. Documents will be sent to Fusion when either this value or `fusion.buffer.timeoutms` is met.
* `fusion.realm`:
This is used with `fusion.user` and `fusion.password` to authenticate to Fusion for indexing data. Two options are supported, `KERBEROS` or `NATIVE`.
Kerberos authentication is supported with the additional definition of a JAAS file. The properties `java.security.auth.login.config` and `fusion.jaas.appname` are used to define the location of the JAAS file and the section of the file to use.
Native authentication uses a Fusion-defined username and password. This user must exist in Fusion, and have the proper permissions to index documents.
* `fusion.user`:
The Fusion username or Kerberos principal to use for authentication to Fusion. If a Fusion username is used (`'fusion.realm' = 'NATIVE'`), the `fusion.password` must also be supplied.
* `fusion.password`:
This property is not shown in the example above. The password for the `fusion.user` when the `fusion.realm` is `NATIVE`.
* `java.security.auth.login.config`:
This property defines the path to a JAAS file that contains a service principal and keytab location for a user who is authorized to read from and write to Fusion and Hive.
The JAAS configuration file *must* be copied to the same path on every node where a Node Manager is running (i.e., every node where map/reduce tasks are executed). Here is a sample section of a JAAS file:
```bash theme={"dark"}
Client { <1>
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/data/fusion-indexer.keytab" <2>
storeKey=true
useTicketCache=true
debug=true
principal="fusion-indexer@FUSIONSERVER.COM"; <3>
};
```
* `<1>` The name of this section of the JAAS file. This name will be used with the `fusion.jaas.appname` parameter.
* `<2>` The location of the keytab file.
* `<3>` The service principal name. This should be a different principal than the one used for Fusion, but must have access to both Fusion and Hive. This name is used with the `fusion.user` parameter described above.
* `fusion.jaas.appname`:
Used only when indexing to or reading from Fusion when it is secured with Kerberos.
This property provides the name of the section in the JAAS file that includes the correct service principal and keytab path.
* `fusion.query.endpoints`:
The full URL to a query pipeline in Fusion. The URL should include the pipeline name and the collection data will be read from. You should also specify the request handler to be used.
If you do not intend to query your Fusion data from Hive, you can skip this parameter.
* `fusion.query`:
The query to run in Fusion to select records to be read into Hive. This is `\*:*` by default, which selects all records in the index.
If you do not intend to query your Fusion data from Hive, you can skip this parameter.
## Query and Insert Data to Hive
Once the table is configured, any syntactically correct Hive query will be able to query the index.
For example, to select three fields named "id", "field1\_s", and "field2\_i" from the "solr" table, you would use a query such as:
```bash theme={"dark"}
hive> SELECT id, field1_s, field2_i FROM solr;
```
Replace the table name as appropriate to use this example with your data.
To join data from tables, you can make a request such as:
```bash theme={"dark"}
hive> SELECT id, field1_s, field2_i FROM solr left
JOIN sometable right
WHERE left.id = right.id;
```
And finally, to insert data to a table, simply use the Solr table as the target for the Hive INSERT statement, such as:
```bash theme={"dark"}
hive> INSERT INTO solr
SELECT id, field1_s, field2_i FROM sometable;
```
## Example Indexing Hive to Solr
Solr includes a small number of sample documents for use when getting started. One of these is a CSV file containing book metadata. This file is found in your Solr installation, at `$SOLR_HOME/example/exampledocs/books.csv`.
Using the sample `books.csv` file, we can see a detailed example of creating a table, loading data to it, and indexing that data to Solr.
```bash wrap theme={"dark"}
CREATE TABLE books (id STRING, cat STRING, title STRING, price FLOAT, in_stock BOOLEAN, author STRING, series STRING, seq INT, genre STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ','; <1>
LOAD DATA LOCAL INPATH '/solr/example/exampledocs/books.csv' OVERWRITE INTO TABLE books; <2>
CREATE EXTERNAL TABLE solr (id STRING, cat_s STRING, title_s STRING, price_f FLOAT, in_stock_b BOOLEAN, author_s STRING, series_s STRING, seq_i INT, genre_s STRING) <3>
STORED BY 'com.lucidworks.hadoop.hive.LWStorageHandler' <4>
LOCATION '/tmp/solr' <5>
TBLPROPERTIES('solr.zkhost' = 'zknode1:2181,zknode2:2181,zknode3:2181/solr',
'solr.collection' = 'gettingstarted',
'solr.query' = '*:*', <6>
'lww.jaas.file' = '/data/jaas-client.conf'); <7>
INSERT OVERWRITE TABLE solr SELECT b.* FROM books b;
```
* `<1>` Define the table `books`, and provide the field names and field types that will make up the table.
* `<2>` Load the data from the `books.csv` file.
* `<3>` Create an external table named `solr`, and provide the field names and field types that will make up the table. These will be the same field names as in your local Hive table, so we can index all of the same data to Solr.
* `<4>` Define the custom storage handler provided by the `lucidworks-hive-serde-v2.2.6.jar`.
* `<5>` Define storage location in HDFS.
* `<6>` The query to run in Solr to read records from Solr for use in Hive.
* `<7>` Define the location of Solr (or ZooKeeper if using SolrCloud), the collection in Solr to index the data to, and the query to use when reading the table. This example also refers to a JAAS configuration file that will be used to authenticate to the Kerberized Solr cluster.
You can use Pig to import data into Fusion, using the `lucidworks-pig-functions-v2.2.6.jar` file found in `$FUSION_HOME/apps/connectors/resources/lucid.hadoop/jobs`.
## Available Functions
The Pig functions included in the `{packageUser}-pig-functions-{connectorVersion}.jar` are three UserDefined Functions (UDF) and two Store functions. These functions are:
* `com/lucidworks/hadoop/pig/SolrStoreFunc.class`
* `com/lucidworks/hadoop/pig/FusionIndexPipelinesStoreFunc.class`
* `com/lucidworks/hadoop/pig/EpochToCalendar.class`
* `com/lucidworks/hadoop/pig/Extract.class`
* `com/lucidworks/hadoop/pig/Histogram.class`
## Using the Functions
### Register the Functions
There are two approaches to using functions in Pig: `REGISTER` them in the script, or load them with your Pig command line request.
If using `REGISTER`, the Pig function jars must be put in HDFS in order to be used by your Pig script. It can be located anywhere in HDFS; you can either supply the path in your script or use a variable and define the variable with `-p` property definition.
The example below uses the second approach, loading the jars with the `-Dpig.additional.jars` system property when launching the script. With this approach, the jars can be located anywhere on the machine where the script will be run.
### Indexing Data to Fusion
When indexing data to Fusion, there are several parameters to pass with your script in order to output data to Fusion for indexing.
These parameters can be made into variables in the script, with the proper values passed on the command line when the script is initiated. The example script below shows how to do this for Solr. The theory is the same for Fusion, only the parameter names would change as appropriate:
* `fusion.endpoints`:
The full URL to the index pipeline in Fusion. The URL should include the pipeline name and the collection data will be indexed to.
* `fusion.fail.on.error`:
If `true`, when an error is encountered, such as if a row could not be parsed, indexing will stop. This is `false` by default.
* `fusion.buffer.timeoutms`:
The amount of time, in milliseconds, to buffer documents before sending them to Fusion. The default is 1000. Documents will be sent to Fusion when either this value or `fusion.batchSize` is met.
* `fusion.batchSize`:
The number of documents to batch before sending the batch to Fusion. The default is 500. Documents will be sent to Fusion when either this value or `fusion.buffer.timeoutms` is met.
* `fusion.realm`:
This is used with `fusion.user` and `fusion.password` to authenticate to Fusion for indexing data. Two options are supported, `KERBEROS` or `NATIVE`.
Kerberos authentication is supported with the additional definition of a JAAS file. The properties `java.security.auth.login.config` and `fusion.jaas.appname` are used to define the location of the JAAS file and the section of the file to use. These are described in more detail below.
Native authentication uses a Fusion-defined username and password. This user must exist in Fusion, and have the proper permissions to index documents.
* `fusion.user`:
The Fusion username or Kerberos principal to use for authentication to Fusion.
If a Fusion username is used (`'fusion.realm' = 'NATIVE'`), the `fusion.password` must also be supplied.
* `fusion.pass`:
This property is not shown in the example above. The password for the `fusion.user` when the `fusion.realm` is `NATIVE`.
#### Indexing to a Kerberized Fusion Installation
When Fusion is secured with Kerberos, Pig scripts must include the full path to a JAAS file that includes the service principal and the path to a keytab file that will be used to index the output of the script to Fusion.
Additionally, a Kerberos ticket must be obtained on the server for the principal using `kinit`.
* `java.security.auth.login.config`:
This property defines the path to a JAAS file that contains a service principal and keytab location for a user who is authorized to write to Fusion.
The JAAS configuration file *must* be copied to the same path on every node where a Node Manager is running (i.e., every node where map/reduce tasks are executed). Here is a sample section of a JAAS file:
```bash theme={"dark"}
Client { <1>
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/data/fusion-indexer.keytab" <2>
storeKey=true
useTicketCache=true
debug=true
principal="fusion-indexer@FUSIONSERVER.COM"; <3>
};
```
* `<1>` The name of this section of the JAAS file. This name will be used with the `fusion.jaas.appname` parameter.
* `<2>` The location of the keytab file.
* `<3>` The service principal name. This should be a different principal than the one used for Fusion, but must have access to both Fusion and Pig. This name is used with the `fusion.user` parameter described above.
* `fusion.jaas.appname`:
Used only when indexing to or reading from Fusion when it is secured with Kerberos.
This property provides the name of the section in the JAAS file that includes the correct service principal and keytab path.
### Sample CSV Script
The following Pig script will take a simple CSV file and index it to Solr.
```bash theme={"dark"}
set solr.zkhost '$zkHost';
set solr.collection '$collection'; <1>
A = load '$csv' using PigStorage(',') as (id_s:chararray,city_s:chararray,country_s:chararray,code_s:chararray,code2_s:chararray,latitude_s:chararray,longitude_s:chararray,flag_s:chararray); <2>
--dump A;
B = FOREACH A GENERATE $0 as id, 'city_s', $1, 'country_s', $2, 'code_s', $3, 'code2_s', $4, 'latitude_s', $5, 'longitude_s', $6, 'flag_s', $7; <3>
ok = store B into 'SOLR' using com.lucidworks.hadoop.pig.SolrStoreFunc(); <4>
```
This relatively simple script is doing several things that help to understand how the Solr Pig functions work.
* `<1>` This and the line above define parameters that are needed by `SolrStoreFunc` to know where Solr is. `SolrStoreFunc` needs the properties `solr.zkhost` and `solr.collection`, and these lines are mapping the `zkhost` and `collection` parameters we will pass when invoking Pig to the required properties.
* `<2>` Load the CSV file, the path and name we will pass with the `csv` parameter. We also define the field names for each column in CSV file, and their types.
* `<3>` For each item in the CSV file, generate a document id from the first field (`$0`) and then define each field name and value in `name, value` pairs.
* `<4>` Load the documents into Solr, using the `SolrStoreFunc`. While we don't need to define the location of Solr here, the function will use the `zkhost` and `collection` properties that we will pass when we invoke our Pig script.
When using `SolrStoreFunc`, the document ID *must* be the first field.
When we want to run this script, we invoke Pig and define several parameters we have referenced in the script with the `-p` option, such as in this command:
```bash wrap theme={"dark"}
./bin/pig -Dpig.additional.jars=/path/to/{packageUser}-pig-functions-{connectorVersion}.jar -p csv=/path/to/my/csv/airports.dat -p zkHost=zknode1:2181,zknode2:2181,zknode3:2181/solr -p collection=myCollection ~/myScripts/index-csv.pig
```
The parameters to pass are:
* `csv`:
The path and name of the CSV file we want to process.
* `zkhost`:
The ZooKeeper connection string for a SolrCloud cluster, in the form of `zkhost1:port,zkhost2:port,zkhost3:port/chroot`. In the script, we mapped this to the `solr.zkhost` property, which is required by the `SolrStoreFunc` to know where to send the output documents.
* `collection`:
The Solr collection to index into. In the script, we mapped this to the `solr.collection` property, which is required by the `SolrStoreFunc` to know the Solr collection the documents should be indexed to.
The `zkhost` parameter above is only used if you are indexing to a SolrCloud cluster, which uses ZooKeeper to route indexing and query requests.
If, however, you are not using SolrCloud, you can use the `solrUrl` parameter, which takes the location of a standalone Solr instance, in the form of `http://host:port/solr`.
In the script, you would change the line that maps `solr.zkhost` to the `zkhost` property to map `solr.server.url` to the `solrUrl` property. For example:
`set solr.server.url '$solrUrl';`
It is often possible to get documents into Fusion by configuring a datasource with [the appropriate connector](/docs/5/fusion/getting-data-in/indexing/connectors).
But if there are obstacles to using connectors, it can be simpler to index documents with a REST API call to an index profile or pipeline.
## Push documents to Fusion using index profiles
[Index profiles](/docs/5/fusion/getting-data-in/indexing/index-pipelines/index-profiles) allow you to send documents to a consistent endpoint (the profile alias) and change the backend index pipeline as needed. The profile is also a simple way to use one pipeline for multiple collections without any one collection "owning" the pipeline.
You can send documents directly to an index using the [Index Profiles REST API](/docs/5/fusion/getting-data-in/indexing/index-pipelines/index-profiles). The request path is:
```bash theme={"dark"}
/api/apps/APP_NAME/index/INDEX_PROFILE
```
These requests are sent as a POST request. The request header specifies the format of the contents of the request body. Create an index profile in the Fusion UI.
To send a streaming list of JSON documents, you can send the JSON file that holds these objects to the API listed above with `application/json` as the content type. If your JSON file is a list or array of many items, the endpoint operates in a streaming way and indexes the docs as necessary.
### Send data to an index profile that is part of an app
Accessing an index profile through an app lets a Fusion admin secure and manage all objects on a per-app basis. Security is then determined by whether a user can access an app. This is the recommended way to manage permissions in Fusion.
The syntax for sending documents to an index profile that is part of an app is as follows:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/json' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE --data-binary @my-json-data.json
```
Spaces in an app name become underscores. Spaces in an index profile name become hyphens.
To prevent the terminal from displaying all the data and metadata it indexes--useful if you are indexing a large file--you can optionally append `?echo=false` to the URL.
Be sure to set the content type header properly for the content being sent. Some frequently used content types are:
* Text: `application/json`, `application/xml`
* PDF documents: `application/pdf`
* MS Office:
* DOCX: `application/vnd.openxmlformats-officedocument.wordprocessingml.document`
* XLSX: `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet`
* PPTX: `application/vnd.vnd.openxmlformats-officedocument.presentationml.presentation`
* More types: [http://filext.com/faq/office\_mime\_types.php](http://filext.com/faq/office_mime_types.php)
### Example: Send JSON data to an index profile under an app
In `$FUSION_HOME/apps/solr-dist/example/exampledocs` you can find a few sample documents. This example uses one of these, `books.json`.
To push JSON data to an index profile under an app:
1. Create an index profile. In the Fusion UI, click **Indexing > Index Profiles** and follow the prompts.
2. From the directory containing `books.json`, enter the following, substituting your values for username, password, and index profile name:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/json' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE?echo=false --data-binary @books.json
```
3. Test that your data has made it into Fusion:
1. Log into the Fusion UI.
2. Navigate to the app where you sent your data.
3. Navigate to the Query Workbench.
4. Search for `*:*`.
5. Select relevant Display Fields, for example `author` and `name`.
### Example: Send JSON data without defining an app
In most cases it is best to delegate permissions on a per-app basis. But if your use case requires it, you can push data to Fusion without defining an app.
To send JSON data without app security, issue the following curl command:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/json' https://FUSION_HOST:FUSION_PORT/api/index/INDEX_PROFILE --data-binary @my-json-data.json
```
### Example: Send XML data to an index profile with an app
To send XML data to an app, use the following:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/xml' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE --data-binary @my-xml-file.xml
```
In Fusion 5, documents can be created on the fly using the [PipelineDocument](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/apollo/common/pipeline/PipelineDocument.html) JSON notation.
## Remove documents
### Example 1
The following example removes content:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/vnd.lucidworks-document' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE --data-binary @del-json-data.json
```
### Example 2
A more specific example removes data from `books.json`. To delete "The Lightning Thief" and "The Sea of Monsters" from the index, use their id values in the JSON file.
The `del-json-data.json` file to delete the two books:
```json wrap theme={"dark"}
[{ "id": "978-0641723445","commands": [{"name": "delete","params": {}}]},{ "id": "978-1423103349","commands": [{"name": "delete","params": {}}, {"name": "commit","params": {}}]}]
```
You can use `?echo=false` to turn off the response to the terminal.
### Example 3
Another example to delete items using the Push API is:
```bash wrap theme={"dark"}
curl -u admin:XXX -X POST 'http://FUSION_HOST:FUSION_PORT/api/apps/APP/index/INDEX' -H 'Content-Type: application/vnd.lucidworks-document' -d '[
{
"id": "1663838589-44",
"commands":
[
{
"name": "delete",
"params":
{}
},
{
"name": "commit",
"params":
{}
}
]
}, ...
]'
```
## Send documents to an index pipeline
Although sending documents to an index profile is recommended, if your use case requires it, you can send documents directly to an index pipeline.
For more information about index pipeline REST API reference documentation, see [Fusion 5.x Index Pipelines API](/api-reference/index-pipelines-api/list-all-index-pipelines).
### Specify a parser
When you push data to a pipeline, you can specify the name of the parser by adding a parserId querystring parameter to the URL.
For example: `https://FUSION_HOST:FUSION_PORT/api/index-pipelines/INDEX_PIPELINE/collections/COLLECTION_NAME/index?parserId=PARSER`.
If you do not specify a parser, and you are indexing outside of an app (`https://FUSION_HOST:FUSION_PORT/api/index-pipelines/...`), then the `_system` parser is used.
If you do not specify a parser, and you are indexing in an app context (`https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index-pipelines/...`), then the parser with the same name as the app is used.
## Indexing CSV Files
In the usual case, to index a CSV or TSV file, the file is split into records, one per row, and each row is indexed as a separate document.
The course for **Ingesting Data** focuses on understanding connectors and how to configure datasources, parsers, and index pipelines.
# Parallel Bulk Loader (PBL)
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/parallel-bulk-loader
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-in/parallel-bulk-loader
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/parallel-bulk-loader
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/452
Fusion Parallel Bulk Loader (PBL) jobs enable bulk ingestion of structured and semi-structured data from big data systems, NoSQL databases, and common file formats like Parquet and Avro.
The Parallel Bulk Loader leverages the popularity of Spark as a prominent distributed computing platform for big data. A number of companies invest heavily in building production-ready Spark SQL data source implementations for big data and NoSQL systems, much as Lucidworks has done for Solr. The Parallel Bulk Loader uses connectors provided by the experts who develop these complex systems.
## Available data sources
The Parallel Bulk Loader can load documents from any system that implements the Data Sources API for Spark SQL 2.2.1 or later.
These are data sources that the Parallel Bulk Loader can use. For data sources that use Spark SQL connectors, the source of the connector is indicated.
* Solr databases\
Connector (Lucidworks): [spark-solr](https://github.com/lucidworks/spark-solr)
* Files in these common formats: JSON, CSV, XML, Apache Avro, and Apache Parquet
* JDBC-compliant databases
* Apache HBase databases\
Connector (Hortonworks): [Apache Spark - Apache HBase Connector](https://github.com/hortonworks-spark/shc)
[Spark-on-HBase](https://hortonworks.com/blog/spark-hbase-dataframe-based-hbase-connector)
* Datasets accessible through Apache Hive
* Apache Cassandra NoSQL databases\
Connector (DataStax): [Spark-Cassandra connector](https://github.com/datastax/spark-cassandra-connector)
* Elastic databases\
Connector: [Elasticsearch-Hadoop connector](https://github.com/elastic/elasticsearch-hadoop)\
Use the package: `org.elasticsearch:elasticsearch-spark-20_2.11:6.2.2`
* MongoDB databases
* Riak databases
* Couchbase NoSQL databases\
Connector: [Couchbase-Spark connector](https://spark-packages.org/package/couchbase/couchbase-spark-connector)\
Use the package: `com.couchbase.client:spark-connector_2.11:2.2.0`
* Redis in-memory data structure stores
* Google data sources, including Google Analytics, Sheets, and BigQuery\
Connectors:
[Analytics](https://mvnrepository.com/artifact/com.crealytics/spark-google-analytics)
[Sheets](https://github.com/potix2/spark-google-spreadsheets)
[Big Query](https://github.com/spotify/spark-bigquery)
* Microsoft Azure DataLake, Cosmos DB, and SQL Database
## Key features
Key features of the Parallel Bulk Loader are:
* **Load distribution.** To distribute load and maximize performance, the Parallel Bulk Loader parallelizes operations and distributes them across the Fusion Spark cluster.
* **No parsing.** No parsing is needed. The Spark Data Sources API returns a DataFrame (RDD + schema) that has an easy-to-use tabular format.
* **Dynamic resolution of dependencies.** There is nothing to download or install. Users just provide the Maven coordinates of dependencies during configuration, and Spark distributes the necessary JAR files to worker nodes.
* **Leverage integration libraries.** Similar to JDBC, the Parallel Bulk Loader leverages integration libraries built by the experts of the underlying systems, for example, Databricks, DataStax, Hortonworks, Elastic, Lucidworks, Microsoft, and so forth.
* **Direct write operations.** The Parallel Bulk Loader writes directly to Solr (for maximum performance) or to Fusion index pipelines (for maximum flexibility).
* **Solr atomic updates.** The Parallel Bulk Loader uses atomic updates to update existing documents in Solr.
* **Incremental queries.** To obtain the latest changes to data sources, macros built into the Parallel Bulk Loader use timestamps to filter queries.
* **Seamless integration with Spark-NLP.** Do natural language processing, including part-of-speech tagging, stemming or lemmatization, sentiment analysis, named-entity recognition, and other NLP tasks.
* **SQL joins.** Use SQL to join data from multiple Solr collections.
* **Load Fusion ML models.** To classify incoming documents, load Fusion Machine Learning models stored in the Fusion blob store.
* **SQL transformations.** Leverage the full power of the Spark Scala’s DataFrame APIs and SQL to filter and transform data.
* **UDF and UDAF functions.** Select from hundreds of user-defined functions and user-defined aggregate functions.
## Differences between the Parallel Bulk Loader and Fusion classic connectors
The primary difference between the Bulk Loader and Fusion classic connectors is that the Bulk Loader uses Spark SQL and Spark/Solr integration to perform distributed reads from data sources.
Here are some examples of how the Parallel Bulk Loader performs distributed reads:
* **HBase table.** To support high-volume data ingestion into Solr, the Parallel Bulk loader can distribute queries sent to HBase tables across multiple region servers.
* **Parquet files.** The Parallel Bulk loader processes a directory of Parquet files in HDFS in parallel using the built-in support for computing splits in HDFS files.
* **Spark/Solr integration.** With Spark/Solr integration, the Parallel Bulk Loader uses a [Spark-Solr data source](https://github.com/lucidworks/spark-solr) to send queries to all replicas of a collection, so it can read from Solr in parallel across the Spark cluster.
This diagram depicts how the Spark-Solr data source partitions queries across all shards/replicas of a Solr collection to better utilize cluster resources and to improve read performance. Most Spark-SQL data sources do something similar for their respective databases, which is one of the main benefits of using the Parallel Bulk Loader job.
In contrast, most classic connectors have only a single-fetcher mode. To scale the fetching with classic connectors, you must distribute the connector itself, which differs from relying on the built-in parallelization of Spark.
Also, most classic connectors rely on Fusion parsers and index pipelines to prepare data for indexing, whereas no parsing is needed for the Parallel Bulk Loader, which can achieve maximum indexing performance by writing directly to Solr.
## Learn more
You can use the Parallel Bulk Loader to copy Solr collections from one collection to another. This is helpful for copying collections from a production environment to a testing or development environment and using real data in the development and testing process.
## Parallel Bulk Loader Job Configuration
In the Fusion UI:
1. Navigate to **Collections > Jobs**.
2. Click Add and select Parallel Bulk Loader from the menu.
3. Enter a name for your job, and enter `solr` as the format.
4. Set the parameter name and value for the collection you want to import.
* Parameter name: `collection`
* Parameter value: the name of the collection to import from
5. If the source collection is in a different Fusion app, add an additional parameter name and value pair.
* Parameter name: `zkHost`
* Parameter value: the location of zkHost
6. Enter the output collection, or where you want to import the collection to.
* Output collection: the name of the output collection
* Send to index pipeline: `_system`
7. Save your job.
When you’re ready, run the job or schedule the job to run at an interval of your choice.
## Parallel Bulk Loader JSON Configuration
Use this sample JSON payload to load the Parallel Bulk Loader to the \[park Jobs API:
```json theme={"dark"}
{
"type" : "parallel-bulk-loader",
"id" : "copy_sigs_with_timestamp",
"format" : "solr",
"readOptions" : [ {
"key" : "collection",
"value" : "SOURCE_COLLECTION"
}, {
"key" : "query",
"value" : "timestamp_tdt:{$lastTimestamp*}* TO *]"
} ],
"outputCollection" : "OUTPUT_COLLECTION",
"timestampFieldName" : "timestamp_tdt",
"defineFieldsUsingInputSchema" : true
}
```
Replace `SOURCE_COLLECTION` with the name of the collection being exported. Replace `OUTPUT_COLLECTION` with the name of the destination collection.
Use the following Spark Jobs API call to run the job. Replace `data.json` with your JSON payload file.
```bash theme={"dark"}
curl -X POST \
-u USERNAME:PASSWORD \
'https://FUSION_HOST:6764/api/apps/Documentation/spark/configurations' \
-H 'accept: */*' \
-H 'Content-Type: application/json' \
-d 'data.json'
```
# Enable Platform Signals
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/platform-signals/configuration
Configure Fusion to consume aggregated signals from Lucidworks Platform for query boosting and result reranking.
## Prerequisites
Before configuring Platform signals in Fusion, make sure you have the following:
* A Lucidworks Platform application configured with a [Signals Store](/docs/lw-platform/lw-analytics/signals/signals-store)
* [Signals Beacon](/docs/lw-platform/lw-analytics/signals/signals-beacon) or [Platform Signals API](/docs/lw-platform/lw-analytics/signals/signals-api) configured
* A Platform integration created for self-hosted Fusion
* Platform Client ID and Secret
* Access to modify Fusion Helm values and ConfigMaps
For Platform setup, see [Platform signals documentation](/docs/lw-platform/lw-analytics/signals/overview).
## Configuration steps
Add Platform credentials to Fusion via Helm values.
These credentials allow Fusion to authenticate with Lucidworks Platform and retrieve aggregated signals.
Add this to your Helm values:
```yaml theme={"dark"}
lwai-gateway:
lwai:
credentials: |
fusion.lwai.account[0].name:
fusion.lwai.account[0].scopes: signals.push,machinelearning.predict,machinelearning.model
fusion.lwai.account[0].clientId:
fusion.lwai.account[0].clientSecret:
fusion.lwai.account[0].baseUrl: https://api.lucidworks.com/applications/
fusion.lwai.default.authEndpoint: https://identity.lucidworks.com/oauth2/ausao8uveaPmyhv0v357/v1/token
```
Replace the placeholders with your actual values:
* `` is the name of your Platform integration.
* `` is the Client ID from the Platform integration.
* `` is the Client Secret from the Platform integration.
* `` is the Platform application ID.
The `authEndpoint` URL is the shared Lucidworks Platform production authentication endpoint.
Use this exact URL as shown; it is not environment-specific.
Apply the Helm values update and restart the affected pods.
Configure which Platform signals Fusion should retrieve and which collections to populate.
Open the `-cloud-signals-config` ConfigMap for editing:
```bash theme={"dark"}
kubectl edit configmap -cloud-signals-config
```
Add your configuration to the `signalsOperator` section:
```yaml theme={"dark"}
signalsOperator: |
:
_signals_aggr:
formula1:
CLICK: 2.0
CART: 3.0
PURCHASE: 4.0
```
The configuration includes the following elements:
* `` is the name of your Platform application.
* `_signals_aggr` is the name of the Fusion collection for storing aggregated signals. Replace `` with your Fusion app ID (for example, `my_app_signals_aggr`).
* `formula1` is the aggregation formula name and must match Platform configuration.
* `CLICK`, `CART`, `PURCHASE` are the signal types and their respective weights.
The formula configuration defines how different signal types are weighted.
The weights should match the aggregation formula configured in Lucidworks Platform.
Save the ConfigMap and restart the signal processing component:
```bash theme={"dark"}
kubectl delete pod -l app.kubernetes.io/component=signals-operator
```
If a Fusion app doesn't exist, create one using these steps:
1. Log in to Fusion Admin UI.
2. Navigate to **Apps**.
3. Click **Create App**.
4. Enter the app name and configure as needed.
Fusion automatically creates the `COLLECTION_NAME_signals_aggr` collection when you configure signal retrieval.
You don't need to create the collection manually.
Add the Boost with Signals stage to your query pipeline using these steps:
1. Navigate to your app's default query pipeline.
2. Add a **Boost with Signals** stage.
3. Configure the stage parameters as shown in the table below:
| Parameter | Value | Description |
| - | - | - |
| **Aggregation Type** | `formula1` | This must match the formula name in the signal retrieval configuration. |
| **Rollup Field** | `doc_id` | This is the field containing the document ID in aggregated signals. |
| **Rollup Weight Strategy** | `weight_d` | This is the field containing the calculated weight. |
See [Boost with Signals Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommendation-boosting-query-stage) for detailed configuration options.
After configuration, verify the integration is working:
1. **Check signal retrieval logs**:
```bash theme={"dark"}
kubectl logs -l app.kubernetes.io/component=signals-operator --tail=100
```
Look for successful run messages:
```
INFO c.l.c.s.c.fetcher.SignalsOperator - Starting control loop
INFO c.l.c.s.c.fetcher.SignalsOperator - Job status: SUCCEEDED
INFO c.l.c.s.c.fetcher.IndexingRunner - Converting to SolrDoc
```
2. **Verify collection population**:
* Navigate to the `COLLECTION_NAME_signals_aggr` collection in Fusion.
* Open Query Workbench.
* Run a query: `q=*:*`.
* Verify documents exist with expected fields.
3. **Test query boosting**:
* Execute a query through your configured pipeline.
* Verify that results with signal data receive appropriate boosts.
If the integration is not working, see [Troubleshooting Platform Signals](/docs/5/fusion/getting-data-in/platform-signals/troubleshooting).
## Next steps
* [Troubleshoot Platform signals](/docs/5/fusion/getting-data-in/platform-signals/troubleshooting)
* [Configure Boost with Signals stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommendation-boosting-query-stage)
* [Platform integration guide](/docs/lw-platform/lw-platform/settings/fusion-integrations/overview)
# Platform Signals
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/platform-signals/overview
Fusion integrates with Lucidworks Platform to consume pre-aggregated signals for query boosting and result reranking.
This integration lets Fusion leverage user behavior data captured by the [Signals Beacon](/docs/lw-platform/lw-analytics/signals/signals-beacon) or [Platform Signals API](/docs/lw-platform/lw-analytics/signals/signals-api) and aggregated in a Lucidworks Platform [Signals Store](/docs/lw-platform/lw-analytics/signals/signals-store).
Platform signals provide an alternative to Fusion's native signals system.
Instead of capturing and aggregating signals within Fusion, the Platform signals integration lets Fusion consume signals that have already been collected and aggregated by Platform Analytics.
## Architecture
The Platform signals integration involves three main components:
1. **Signal collection** via [Signals Beacon](/docs/lw-platform/lw-analytics/signals/signals-beacon) (browser-based) or [Platform Signals API](/docs/lw-platform/lw-analytics/signals/signals-api) sends raw signals to the Platform Signals Store.
2. **Fusion** reads aggregated signals from the Platform Signals Store and indexes them for use in query boosting.
3. **Query Pipelines** use the aggregated signals for query boosting via the [Boost with Signals stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommendation-boosting-query-stage).
## Signal flow
User interactions are captured via Signals Beacon or Platform Signals API and sent to the Platform Signals Store for storage and aggregation.
Fusion periodically retrieves these aggregated signals and indexes them in the `signals_aggr` collection.
Query pipelines then use these signals through the Boost with Signals stage to rerank search results based on user behavior patterns.
```mermaid theme={"dark"}
graph LR
accTitle: Platform signals data flow
accDescr: A flowchart showing platform signals flowing from a beacon or signals API through the Platform Signals Store to Lucidworks Search for query pipeline boosting
A["Beacon or Platform Signals API"] --> B["Platform Signals Store"]
B --> C[Fusion]
C --> D["signals_aggr collection"]
D --> E["Query Pipeline Boost Stage"]
```
## When to use Platform signals
Use the Platform Signals integration if any of the following apply:
* **You already use Analytics Studio.** This integration uses your existing Platform signals in Fusion without additional infrastructure.
* **You want adaptive behavioral relevance.** Platform Signals feeds real customer behavior into Fusion's boost stage, enabling relevance that adapts continuously to your users.
* **You manage multiple search applications.** Platform Signals aggregates behavior from multiple sites and applications using the Signals Beacon (web) or Signals API (any application type), giving you a unified view across your search ecosystem.
* **You want advanced signal aggregation.** Platform's aggregation formulas produce refined signals that reflect meaningful behavior patterns rather than raw event counts.
## Comparison with Fusion native signals
| Feature | Fusion native signals | Platform signals |
| - | - | - |
| Signal collection | Fusion Signals API | Beacon (browser-based) or Platform Signals API (any application type) |
| Signal storage | Fusion collections | Platform Signals Store |
| Signal aggregation | Fusion SQL aggregations | Platform aggregation engine |
## Collections
When you enable Platform signals integration, Fusion creates and populates the `COLLECTION_NAME_signals_aggr` collection that stores aggregated signals retrieved from the Platform Signals Store.
Unlike Fusion-native signals, Platform signals integration does *not* create the following collections because they are not needed:
* `COLLECTION_NAME_signals` is the raw signals collection. Raw signals remain in the Platform Signals Store.
* `COLLECTION_NAME_signals_session_aggr` is for session-level aggregations. Platform handles these aggregations.
## Next steps
* [Configure Platform signals integration](/docs/5/fusion/getting-data-in/platform-signals/configuration)
* [Configure Boost with Signals stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommendation-boosting-query-stage)
* [Troubleshoot Platform signals](/docs/5/fusion/getting-data-in/platform-signals/troubleshooting)
## Related topics
* [Indexing Signals](/docs/5/fusion/getting-data-in/indexing/indexing-signals/overview) - Fusion's native signals system
* [Platform Analytics Signals](/docs/lw-platform/lw-analytics/signals/overview) - Platform signals documentation
# Troubleshooting Platform Signals
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-in/platform-signals/troubleshooting
Common issues and solutions for Platform signals integration in Fusion.
## Signal retrieval not working
**Symptom**: Fusion's signal processing component is not running or crashes repeatedly.
**Diagnosis**:
```bash theme={"dark"}
kubectl get pods -l app.kubernetes.io/component=signals-operator
kubectl logs -l app.kubernetes.io/component=signals-operator --tail=50
```
**Common Causes**:
1. **Missing or incorrect LWAI credentials**
* LWAI gateway credentials are not configured in Helm values.
* Client ID and Secret don't match the Platform integration.
* The `baseUrl` doesn't point to the correct Platform application.
2. **ConfigMap syntax errors**
* The `cloud-signals-config` ConfigMap syntax is not valid YAML.
* The `signalsOperator` section has incorrect indentation.
**Solution**:
Review and correct the credentials, then restart the pod:
```bash theme={"dark"}
kubectl delete pod -l app.kubernetes.io/component=signals-operator
```
## Authentication failures
**Symptom**: Signal retrieval logs show authentication errors.
**Example Log**:
```
ERROR c.l.c.s.c.f.s.SignalsServiceClientImpl - Response code:400, message:{"error":"validation","message":"Requested account profile is not known."}
```
**Diagnosis**:
Check the signal retrieval logs for authentication errors:
```bash theme={"dark"}
kubectl logs -l app.kubernetes.io/component=signals-operator | grep -i "error\|failed"
```
**Common Causes**:
1. **Integration name mismatch**
* The integration name in Helm values doesn't match the Platform integration name.
2. **Expired or invalid credentials**
* The Client ID or Secret are incorrect.
* The integration was deleted or modified in Platform.
3. **Incorrect auth endpoint**
* The `authEndpoint` URL is wrong or inaccessible.
**Solution**:
1. Verify integration name matches exactly (case-sensitive).
2. Regenerate Client ID and Secret in Platform.
3. Update Helm values with correct credentials.
4. Redeploy or restart affected pods.
## No signals in collection
**Symptom**: The `COLLECTION_NAME_signals_aggr` collection is empty or not populated.
**Diagnosis**:
1. Check whether the collection exists in Fusion Admin UI:
* Navigate to Collections.
* Search for `COLLECTION_NAME_signals_aggr`.
2. Check signal retrieval logs:
```bash theme={"dark"}
kubectl logs -l app.kubernetes.io/component=signals-operator --tail=100
```
**Common Causes**:
1. **No signals in Platform Signals Store**
* Beacon is not configured or not tracking events.
* The Platform aggregation job hasn't run yet.
2. **Signal retrieval not processing**
* The job status shows `FAILED`.
* No export job is found in logs.
3. **Collection name mismatch**
* The collection name in the ConfigMap doesn't match the Fusion app name.
**Solution**:
1. Verify signal collection is working:
* Check the Platform dashboard for signal activity.
* Verify your [Beacon](/docs/lw-platform/lw-analytics/signals/signals-beacon) or [Platform Signals API](/docs/lw-platform/lw-analytics/signals/signals-api) configuration.
2. Review signal retrieval logs for processing status:
```
INFO c.l.c.s.c.fetcher.SignalsOperator - Found jobId in progress:
INFO c.l.c.s.c.fetcher.SignalsOperator - Job status: SUCCEEDED
```
3. Verify collection name format:
* The collection name should be `COLLECTION_NAME_signals_aggr`.
* The name must exactly match the ConfigMap configuration.
## Incorrect signal weights
**Symptom**: Documents are boosted, but weights don't match expected values.
**Diagnosis**:
Query the signals collection to inspect weights, usually in the `weight_d` field.
**Common Causes**:
1. **Formula configuration mismatch**
* Formula weights in the signal retrieval configuration don't match the Platform formula.
* The wrong formula name is specified in the Boost with Signals stage.
2. **Field mapping issues**
* The Rollup Weight Field in the query stage doesn't match the signal document field.
**Solution**:
1. Verify formula configuration matches Platform:
* Check Platform formula definition.
* Update signal retrieval configuration to match.
2. Verify Boost with Signals stage configuration:
* **Aggregation Type** must match formula name (e.g., `formula1`).
* **Rollup Weight Strategy** must be `weight_d`.
## Query boosting not working
**Symptom**: Queries execute successfully but results are not boosted based on signals.
**Diagnosis**:
1. Check whether the Boost with Signals stage is enabled in the query pipeline.
2. Run a query with debug enabled: `debug=true`.
3. Review boost parameters in the debug output.
**Common Causes**:
1. **Stage not in pipeline**
* The Boost with Signals stage is missing or disabled.
2. **Wrong aggregation type**
* The Aggregation Type in the stage doesn't match the signals formula.
3. **Field name mismatch**
* The Rollup Field value doesn't match the signal document field names.
4. **Empty signals collection**
* No matching signals exist for the query.
**Solution**:
1. Add or enable the Boost with Signals stage in the query pipeline.
2. Verify the stage configuration:
* Aggregation Type: `formula1`
* Rollup Field: `doc_id`
* Rollup Weight Strategy: `weight_d`
3. Test with a known query that has signals:
* Query the signals collection to find a query with aggregated data.
* Execute that query through the pipeline.
* Verify boosting is applied in the debug output.
## Slow query performance
**Symptom**: Queries with signal boosting are significantly slower than without.
**Diagnosis**:
Enable query timing and examine the Boost with Signals stage performance:
```
debug=true&debugQuery=true
```
**Common Causes**:
1. **Large number of signals retrieved**
* The `numSignals` parameter is set too high.
2. **Inefficient signals collection query**
* Solr Query Parameters are missing or incorrect.
**Solution**:
1. Tune Boost with Signals stage parameters:
* Reduce `numSignals` to a reasonable value (50-100).
* Reduce `numRecommendations` if appropriate.
2. Optimize signals collection:
* Make sure appropriate fields are indexed.
* Consider adding filters to limit signal retrieval scope.
## Log analysis
### Successful run
Successful signal retrieval shows:
```
INFO c.l.c.s.c.fetcher.SignalsOperator - Starting control loop
INFO c.l.c.s.c.fetcher.SignalsOperator - Found jobId in progress:
INFO c.l.c.s.c.f.s.SignalsServiceClientImpl - Checking export job status
INFO c.l.c.s.c.fetcher.SignalsOperator - Job status: SUCCEEDED
INFO c.l.c.s.c.fetcher.IndexingRunner - Found input file: gs://...
INFO c.l.c.s.c.fetcher.IndexingRunner - Object node: {"query_s":"...","doc_id":"...","type":"...","weight_d":...}
INFO c.l.c.s.c.fetcher.IndexingRunner - Converting to SolrDoc
```
### Failed run
A failed run shows:
```
ERROR c.l.c.s.c.f.s.SignalsServiceClientImpl - Response code:400, message:{...}
INFO c.l.c.s.c.fetcher.SignalsOperator - Job status: FAILED
```
## Getting help
If you continue to experience issues after following this guide, take these steps:
1. Collect diagnostic information:
* Signal retrieval logs (last 200 lines).
* Helm values configuration (redact secrets).
* Signal retrieval configuration.
* Sample signals collection documents.
2. Verify Platform integration:
* Check Platform application status.
* Verify integration credentials.
* Review Platform signals aggregation jobs.
3. Contact Lucidworks Support with the collected diagnostic information.
## Related topics
* [Configure Platform Signals](/docs/5/fusion/getting-data-in/platform-signals/configuration)
* [Boost with Signals Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommendation-boosting-query-stage)
* [Platform Signals](/docs/lw-platform/lw-analytics/signals/overview)
# Curating Your Search Experience
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/curating-your-search-experience/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/cx3sp3
Predictive Merchandiser and Experience Optimizer are both AI-powered tools that provide insights, customer recommendations, and search optimization in a single comprehensive product.
Determining the best product for your needs depends on the type of materials you would like to tailor your search experience for.
## Experience Optimizer vs Predictive Merchandiser
Experience Optimizer is typically for Knowledge Management while Predictive Merchandiser is typically for eCommerce. Knowledge Management uses files and documents and wants to see additional text and links, with less emphasis on images. Online stores tend to put a greater emphasis on images than text.
In both situations, items chosen as most important are put into zones and shown for all users with natural search results below. You can pin important products, files, or folders as the first links.
| Product | Experience Optimizer | Predictive Merchandiser |
| - | - | - |
| Layout | Optimizes list view | Thumbnail based grid view |
| Images | Small thumbnail; if no image available then show generic images | Larger images |
| Description | Page title with heading and excerpt | Less text, typically just a title |
## Choosing the right product for your needs
When you enter the **Rules** section from Fusion for the first time, you will initiate a wizard that will allow you to choose between two solution types, **Product Discovery** or **Knowledge Management**. From there, you will see your prospective workspace with predefined defaults.
This selection is permanent. Please select with caution.
### Exploring solution types
The Product Discovery route activates Predictive Merchandiser. This route is designed for online retailers and similar clients who want to create an enhanced shopping experience for their customers. [Predictive Merchandiser’s](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/overview) grid-view interface is helpful for displaying products in the best light, with larger product imagery and a product title.
The Knowledge Management route activates Experience Optimizer. This route is intended for search managers and similar clients who want to facilitate access to documents, files, or important news. [Experience Optimizer’s](/docs/5/fusion/getting-data-out/curating-your-search-experience/experience-optimizer/overview) list-view conveys more information in search results with additional lines of text and smaller imagery. It also contains assigned imagery for abstract items, or items with no image.
# Hierarchical Facets
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/hierarchical-faceting
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/hierarchical-faceting
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/hierarchical-faceting
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/vfdq6w
Facets of multiple levels under a parent facet are known as hierarchical facets. Hierarchical facets allow faster navigation through facet submenus and breadcrumbs when navigating through the available options. Facets with children can be indicated by an arrow to indicate they have children that become visible when the arrow is selected. Selecting a child facet after expanding the facet menu allows the user to bypass the options between the top page and their choice.
Hierarchical facets are split according to the delimiter set in the [Apply Rules](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/query-rules-query-stage) query pipeline stage.
For more information, see **Use Hierarchical Facets**.
## Fusion
To configure hierarchical fields settings:
1. Access the application and click **Relevance > Rules > Templates**.
2. Click the three vertical dots to the right of the application name to display the Hierarchical fields settings section.
3. Complete one of the following:
* To add a new field, click **+ Add**. Enter values in the **Field** and **Delimiter** fields and click **Apply**.
Any symbol can be used as a delimiter.
* To edit an existing field, click **Field** and select the value to edit. Enter the appropriate value in the **Delimiter** field and click **Apply**.
* To delete a field setting, select the value in **Field**, click **X**, and then click **Apply**.
* To cancel any changes before you apply them, click **Cancel**.
# Merchandising Zero Results
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/merchandising-zero-results
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/merchandising-zero-results
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/merchandising-zero-results
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/597
As a merchandiser, search queries that generate zero results can drive customers away from the site. Predictive Merchandiser allows you to ensure this does not happen. Instead of display zero results, you can choose to display certain products when a customer reaches a zero results page.
Learn how to use the **Merchandising Zero Results** feature.
1. Enter a search query the generates 0 results:
2. Click the **Add** button:
3. Select products or [product groups](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/product-grouping) that you want to associate with the 0 results query:
4. Click the **Insert** button. The selected products are now displayed when the search query is used, resolving the 0 results query:
# Predictive Merchandiser
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/591
Predictive Merchandiser is an AI-powered merchandising tool for Product Discovery solutions. It provides insights and recommendations for optimizing search results and product placement.
It provides an easy-to-use interface, allowing you to:
* Connect to a Fusion data source and simulate the shopping experience of your customers.
* Pin, boost, bury, and block specific products in your catalog.
* Create business rules to display custom banners, or redirect customers to a different page on your site, for example.
* Create search rewriting strategies to manage spelling mistakes, synonym matching, phrase matching, and poorly performing queries.
* Review query rewriting strategies that were generated automatically using machine learning models in Fusion.
* Deploy new business rules and search rewrites to the Fusion data source where they can be triggered by live queries.
* View analytics reports that show, for example, the variation in site visits over time, the most clicked on products, and the most searched for terms.
## Licensing
Predictive Merchandiser is integrated with Fusion and requires a valid license.
To verify your Predictive Merchandiser license, click **Account Settings** and select **License Details**.
If you have a valid Predictive Merchandiser license, an asterisk `*` displays after **Fusion AI Pm**.
To upload a valid license, click **Upload**.
## Access Predictive Merchandiser
To access Predictive Merchandiser:
1. Sign in to Fusion and click **Relevance > Rules**.
2. Click the **Merchandiser** icon.
## Learn more
The course for **Predictive Merchandiser: Getting Started** focuses on surfacing products that match an individual shopper’s taste and preferences using signal-driven recommendations.
You can block a product so that it does not appear in the search results when your customers run a particular query. This is helpful if a product has been removed from your catalog, or it is not yet available, for example.
When you block a product, a new rule is created automatically.
1. Click the **Start Task** button.
2. Hover your cursor over a product. A menu overlay will appear.
3. Hover your cursor over the **Actions** button. A dropdown menu will appear.
4. Select the **Block Product**.
You can also block multiple products at once by clicking the checkbox in the top-left corner of the product tiles and selecting the **Block** button above the top row of products.
You can boost a product so that it occupies a higher position in the search results when your customers run a particular query. This can be helpful if you have new products in your catalog, or large stocks of some products, for example.
Boosting a product increases the product’s position in the results, but takes other factors into consideration while doing so. As a result, it does not guarantee the product will occupy a specific position in the list. Pinning a product, however, places the product at a specified position regardless of other factors.
When you boost a product, a new rule is created automatically.
1. Click the **Start Task** button.
2. Hover your cursor over a product. A menu overlay will appear.
3. Hover your cursor over the **Actions** button. A dropdown menu will appear.
4. Click **Boost**.
You can also boost multiple products at once by clicking the checkbox in the top-left corner of the product tiles and selecting the **Boost** button above the top row of products.
You can bury a product so that it occupies a lower position in the search results when your customers run a particular query. This can be helpful if some of the products in your catalog are temporarily out of stock, for example.
Burying a product typically moves it to the end of the list. However, its exact position will depend on multiple factors, including its relevance score and user signals in relation to other products.
When you bury a product, a new rule is created automatically.
1. Click the **Start Task** button.
2. Hover your cursor over a product. A menu overlay will appear.
3. Hover your cursor over the **Actions** button. A dropdown menu will appear.
4. Select the **Bury** button:
You can also bury multiple products at once by clicking the checkbox in the top-left corner of the product tiles and selecting the **Bury** button above the top row of products.
You can pin a product so that it occupies a fixed position in the search results when your customers run a particular query. This can be helpful if you have bestsellers or popular items that you want customers to see first, for example.
Boosting a product increases the product’s position in the results, but takes other factors into consideration while doing so. As a result, it does not guarantee the product will occupy a specific position in the list. Pinning a product, however, places the product at a specified position regardless of other factors.
When you pin a product, either a new rule is created automatically or an existing pin rule is modified.
## Use Actions
1. Click the **Start Task** button.
2. Hover your cursor over a product. A menu overlay will appear.
3. Hover your cursor over the **Actions** button. A dropdown menu will appear.
4. Select the **Pin Product** button:
## Drag and Pin a Product
1. Click the **Start Task** button.
2. Hover your cursor over a product. A menu overlay will appear.
3. Click-and-hold the drag bar.
4. Move the product to the desired location.
5. Release the click.
## Insert and Pin a Product
You can search for and insert an alternative product from your catalog into a list of search results, and pin it at a fixed position. This is helpful if:
* You want to include a product that is not included in the results list generated by the query.
* The results list is long and you need to search for a particular product to locate and pin it.
* You want to pin multiple products simultaneously.
When you pin a product, either a new rule is created automatically or an existing pin rule is modified.
1. Hover your cursor between two products. A **+** button will appear:
2. Click the **+** button.
3. Locate and select one or more products you want to pin.
4. Click the **Insert** button:
## Choose Pin Position
1. Click the **Start Task** button.
2. Hover your cursor over a product. A menu overlay will appear.
3. Select the value of the **Position** parameter.
4. Enter the desired replacement value, or adjust the value using the arrows.
## Start a task
1. Click the **Start Task** button:
This enables Interactive Mode. In Interactive Mode, you can interact directly with some of the widgets on the Predictive Merchandiser board, creating and managing business rules as you simulate your customers' searches. You can:
* Pin, block, boost and bury products.
* Set additional facets.
* Create search rewrites from poorly performing search terms.
* See which rules and search rewrites are triggered by a search, then edit or delete them as needed.
* Sort search results by ascending or descending order, or by relevance.
* Check product information.
## Undo changes in a task
1. Click the **Undo All** button: . A confirmation dialog box will appear.
2. Select **Undo** to confirm.
## Exit a task
Exiting a task will
1. Click the **Exit** button:
## Publish a task
You can publish any changes you make to rules back to the Fusion data source where they can impact live queries.
1. Click the **Publish** button: . A confirmation dialog box will appear.
2. Select **Publish** to confirm.
Predictive Merchandiser counts unpublished changes in the top navigation bar:
1. Click the **Start Task** button.
2. Hover your cursor over a product. A menu overlay will appear.
3. Hover your cursor over the **Information** button . The Product Details panel will appear.
4. Use the **Details** and **Signals** buttons to toggle between panes.\\
# Page Metadata
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/page-metadata
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/page-metadata
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/page-metadata
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/d9o8jb
Metadata describe the content of a web page. The Page Metadata feature in Predictive Merchandiser lets you configure metadata using **Use Predictive Merchandiser Templates**.
This article teaches you how to use the [Templates](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/templates) feature in Predictive Merchandiser, including how to:
* [Create a template](#create-a-new-template)
* [Add zones](#add-a-zone-to-a-template) to the template
* Conditionally [trigger the template](#set-triggering-options)
* [Test](#test-the-template) the template in the Predictive Merchandiser screen
In the instructions below, a new template is created with two zones for use during a Cyber Monday sales event. One zone is used to display general search results while the other is used to promote specific products for the sale.
The zones used in this article use different query profiles. To follow the instructions as described, you will need to begin with an app that has at least two query profiles.
To begin, access Templates in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor) by navigating to **Relevance** > **Rules** > **Templates**.
The course for **Predictive Merchandiser: Template Manager** focuses on how to use the templates included in Predictive Merchandiser and create your own custom templates.
## Create a new template
1. From the Templates screen, click the **New Template** button:
2. Assign the template a unique name in the **Template Name** field. In this example, the value `Cyber Monday` is assigned.
3. Choose a **Template Type**. This example uses the `landing` template type.
This value is read-only and for informational purposes only.
4. Click the **Save** button to save the new template:
For full configuration, see:
* [Templates UI Configuration Options](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/templates-config)
* [Templating API Configuration Reference](/api-reference/templates-controller/get-template-information)
Images may display other releases if the information between releases does not differ. For example, the Fusion Templating API image reflects version 5.3.5 in both the Fusion 5.2 and 5.3 releases, but reflects 5.4.0 in the Fusion 5.4 release.
## Add a zone to a template
Zones enable Fusion users to create rules designed for specific query profiles. Combined with templates, this allows a search application to display results from multiple query profiles in the same view.
In this example, two new zones are created for the **Cyber Monday** template.
1. Add an existing zone to the template by clicking the **Existing** button. In this example, the **Main Results List** zone is added by selecting the zone and clicking the **Apply** button.
2. Create a new zone by clicking the **New** button:
3. Assign the zone a unique name in the **Display Name** field. For this example, the value `Promoted` is assigned.
4. Choose the **Query Profile** for this zone. This example uses a query profile named **promotional-results**. This query profile is configured with rules that pin products to the top of the results.
5. Choose the **Zone Type** for this zone. This example uses **main-results-list**.
This value is read-only and for informational purposes only.
6. Set the configuration options for the product image, title, and description per your preference.
7. Click the **Save** button to save the new zone:
For full zone configuration details, see [Zone Configuration Options](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/templates-config).
## Edit an existing zone
You can edit a zone that was already added to a template:
1. Select the **Start Task** button.
2. Select the pencil icon to begin editing the chosen page.
3. Each zone has a trash can icon to delete the zone. Select the wrench icon to edit the zone.
4. In the form that appears, make any necessary edits.
5. Click the **Save** button to save your changes.
## Set triggering options
Triggers describe conditions that must be met for a template to display. Trigger options include a set date range and specific search terms. See [Triggering Configuration Options](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/templates-config) for additional details.
### Date range
1. Click the **0 Active** button under **Triggering**:
If there are active triggers, this button changes to reflect that number. For example, this button will read **3 Active** when 3 triggers are active.
2. Click the date range selector and select a start and end date. For this example, choose a date range that includes today’s date.
3. Click the **Save** button to save your changes:
### Search terms
1. Click the **0 Active** button under **Triggering**.
2. Click the **Trigger** button:
3. In the **Search Terms** field, enter `cyber monday`.
4. Set the **Match Mode** field to `phrase`.
5. Click the **Add** button.
6. Repeat the process to add a second trigger with **Search Terms** set to `sales event` and **Match Mode** to `phrase`.
7. Click the **Save** button to save your changes:
## Test the template
1. Navigate to the **Merchandiser** screen.
2. Enter one of the [search terms](#search-terms) you configured in the **Set triggering options** instructions above. This example used the search terms `cyber monday` and `sales event`.
If today’s date does not fall within the template’s [date range trigger](#date-range), the template will *not* load.
If everything was configured correctly, you should see the **Cyber Monday** template is loaded. You can now create different rules for both zones, including pinning promoted products in the **Promoted** zone.
When you trigger Predictive Merchandiser templates, the page metadata is added to the response payload, which looks like this:
```json wrap theme={"dark"}
{
"triggered": {
"app": "test",
"id": "5d9cd572-5928-4a16-a9de-5bab582538a2",
"name": "example-template",
...
"name": "Metadata of example-template",
...
"inTemplates": ["5d9cd572-5928-4a16-a9de-5bab582538a2"],
"pageMetadata": [{
"metadata": {
"pageTitle": "Example page title",
"pageDescription": "Example description",
"pageKeywords": ["example"],
"imageUrl": null,
"pageUrl": null
}
}],
"neverPublished": false,
"staging": true,
"published": false
}
}
```
The Page Metadata feature also allows you to filter your facets by existing tags, clone the values of your facets, or select filters/facets with predefined metadata. If you have a large number of pages, you can also search your domain by **Path URL** using the **URL Search**.
For more information, see **Use Page Metadata**.
Use the Page Metadata feature to edit the following fields in a Predictive Merchandiser template:
* `page-title`
* `page-description`
* `page-keywords`
* `image-url`
* `page-url`
## Access and Edit Page Metadata
You must be within an active template to use the **More Menu**.
1. Choose your app and navigate to **Rules** > **Merchandiser**.
2. Click
3. Click .
This opens a dialog with predefined fields, based on the current information in place for your page.
4. Make your changes, then click **Save**.
### Publish a template
After editing your page metadata, your changes will not be applied until you publish the template. The **Publish** function is available from the **Rules toolbar**:
1. Click .
In the **Template Changed** section, you will see short description of the page that you edited the metadata for.
2. Hover to the right of the page description to see the button.
3. Review the changes and click the button at the end of the dialogue to publish the template.
Your metadata changes have now been applied to your site.
## Clone Facet Values
1. In the Merchandiser tab, click the
2. Click
3. Select one or more of your existing facets and click
## Find a page using URL Search
1. In the Merchandiser tab, click
2. Click
3. In the **Domain URL** field, select the domain URL for the website you wish to search in the dropdown menu. For example, `https://www.lucidworks.com`.
4. Type the path URL you wish to search for in the **Path URL** field. For example, `/products/fusion`.
5. Click **Search**.
# Supported Browsers
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/pm-browser-support
Predictive Merchandiser
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/pm-browser-support
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/pm-browser-support
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/600
Currently, Predictive Merchandiser is officially supported in Google Chrome *only*. Although Predictive Merchandiser is available in unsupported browsers, features and functionality may be limited.
| Browser | Supported |
| - | - |
| Google Chrome | ✅ |
| Firefox | ✘ |
| Safari | ✘ |
| Opera | ✘ |
| Edge | ✘ |
# Product Grouping
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/product-grouping
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/product-grouping
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/product-grouping
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/596
Predictive Merchandiser makes it easy to group products together so they can be presented to the user. This can be useful when you want to show a product in all of its color variants or the various versions of a product (basic or advanced, for example).
Product groups can be boosted or blocked. If you want to promote a product group that is on sale, for example, you can boost the group to the top of the results.
You can still apply rules individually to products within a group. This allows you to boost a product based on a specific user’s search behavior. If a user typically favors the color red, you can boost the red variant of the product to the top of the group.
## Best practices
These parameters group products, and are defined in the Additional Query Parameters Stage:
| Parameter Name | Parameter Value | Update Policy |
| - | - | - |
| group | true | default |
| group.format | grouped | default |
| group.ngroups | true | default |
| group.field | style\_id\_s | default
NOTE: You must configure `groupField=` if you use [App Studio](/docs/5/fusion/dev-portal/appkit/overview) to create your search application. |
For more information about adding a query pipeline and additional query parameters, see:
* **Add a Query Pipeline Stage**
* [Additional Query Parameters Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/set-query-params-stage)
In the Query Workbench, click **Add a Stage** to add query pipeline stages that can perform query setup, results relevancy, troubleshooting, and more.
The Solr Query stage is the only pipeline stage that is required for querying processes to complete, and therefore exists in every query pipeline. It is always the last stage in a series.
**See these pages for more information:**
Fusion 5.x:
* [Query Workbench](/docs/5/fusion/intro/ui-tour/query-workbench)
* [query pipeline stages](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/overview)
* [Solr Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/solr-query-stage)
Fusion 4.x:
* [Query Workbench](/docs/4/fusion-server/concepts/querying/query-workbench)
* [query pipeline stages](/docs/4/fusion-server/reference/pipeline-stages/query/overview)
* [Solr Query stage](/docs/4/fusion-server/reference/pipeline-stages/query/solr-query-stage)
### Collapse/Expand parser grouping
The Collapse/Expand parser allows grouping with the following parameters:
| Parameter Name | Parameter Value |
| - | - |
| expand | true |
| enableElevation | true |
| group | false |
See [Collapse and Expand Results](https://solr.apache.org/guide/collapse-and-expand-results.html) for more information.
# Rule Conflict Resolution
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/rule-conflict-resolution
Predictive Merchandiser
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/rule-conflict-resolution
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/rule-conflict-resolution
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/598
Sometimes, rules created in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor) and Predictive Merchandiser conflict with each other. For example, if you have three rules to boost, bury, and block the same product, Predictive Merchandiser highlights these conflicting rules so you can address them.
To highlight conflicting rules, the **Rules Fired** section displays a red warning icon next to the dominant rule and a red flag on the conflicting rules.
Click the **Edit** button to view the details of the conflicting rules and resolve the conflict.
The details of the conflicting rules affecting specific products are found in the details panel. Click the **Start Task** button , hover over a product, and click the **Details** button .
The course for **Predictive Merchandiser: Rules Manager** focuses on how to create and utilize business rules to create more relevant search results for your users.
# Search Rewrites
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/search-rewrites
Predictive Merchandiser
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/search-rewrites
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/search-rewrites
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/595
Search rewriting is a way of modifying search queries so that they more accurately reflect the intentions of your customers. You can create a misspelling search rewrite, for example, to correct a spelling mistake that commonly occurs when searching for a particular product. The misspelled query will then produce the same results as a correctly spelled search.
There are four rewrite types:
* **Head/Tail** - improves poorly performing searches.
* **Misspelling detection** - corrects common spelling mistakes.
* **Phrase detection** - identifies products with matching phrases.
* **Synonym detection** - includes alternative words with the same meaning.
You can create query rewrite rules in Predictive Merchandiser. This is similar to using the Rules Editor to create query rewrite rules, except there are additional options in Predictive Merchandiser.
1. In Fusion, navigate to **Relevance** > **Rules** > **Merchandiser**.
2. Enter a search term or phrase in the search bar.
3. Click **Start Task**.
4. Hover your cursor next to the query. A **+** button will appear:
5. Click the **+** button. A list of query rewrites options will appear: [Head/Tail](#head-tail), [Misspelling](#misspelling), [Phrase](#phrase), [Synonym](#synonym), and [Remove Words](#remove-words).
## Head/Tail
You can create a Head/Tail rewrite to improve search results using methods other than correcting for misspellings or synonym expansion. When a poorly defined search term is identified, the original term is replaced by an improved search term.
For example, a search for `iphone case yellow` could be improved by using the search term `iphone case yellow +color:"yellow"` (in this case making use of the color field in the data).
Most Head/Tail rewrites are typically created automatically via machine learning. However, if desired, custom rewrites can be manually created using the following steps.
1. From the list of query rewrite options, select **Head/Tail**. A form will appear:
This form contains the following fields:
| | | |
| - | - | - |
| Parameter | Description | Example Value |
| Tail Query | The tail query itself. | `iphone case yellow` |
| Improved Query | The query that will replace the tail query phrase. | `iphone case yellow +color:"yellow"` |
| Tags | Optional metadata tags that can be used to identify and organize rewrites. | `blackfridaysale` |
2. Enter one or more improved search terms in the **Improved Query** field.
3. Click the **Save** button.
The search is re-run using the improved search term, and a Search box information panel drops down to show that the new query rewrite has fired:
## Misspelling
You can create a misspelling query rewrite to detect and correct common spelling mistakes. When a customer enters a search term containing a known misspelling, the incorrect spelling is replaced with the spelling correction.
For example, if your customers frequently misspell or mistype the word `sweater` as `sweatre`, you can set up a query rewrite to automatically correct it.
1. From the list of query rewrite options, select **Misspelling**. A form will appear:
This form contains the following fields:
| | | |
| - | - | - |
| Parameter | Description | Example Value |
| Misspelled Term | The phrase itself. | `sweatre` |
| Corrected Term | The term that will replace the misspelled term. | `sweater` |
| Action | Action to perform. | |
| Confidence | Confidence score from the phrase job. A confidence level of `1` represents 100% confidence. For rules created automatically via machine learning, the confidence level will reflect the output from the machine learning model. | `1` |
| Tags | Optional metadata tags that can be used to identify and organize rewrites. | `blackfridaysale` |
2. Enter one or more spelling corrections in the **Corrected Term** field.
3. Click the **Save** button.
The search is re-run using the corrected spelling, and a Search box information panel drops down to show that the new query rewrite has fired:
## Phrase
You can use query rewriting to identify phrases used in search terms so that products with matching phrases are boosted in the search results. This is helpful when users do not use quotation marks to identify phrases in their search terms.
For example, without phrase detection a search for the words `remote control` would show results for both `remote` and `control`. With phrase detection, this search would correctly boost results for `"remote control"`.
1. From the list of query rewrite options, select **Phrase**. A form will appear:
This form contains the following fields:
| | | |
| - | - | - |
| Parameter | Description | Example Value |
| Surface Form | The phrase itself. | `remote control` |
| Word Count | Indicates how many words are included in the phrase. | `2` |
| Confidence | Confidence score from the phrase job. A confidence level of `1` represents 100% confidence. For rules created automatically via machine learning, the confidence level will reflect the output from the machine learning model. | `1` |
| Tags | Optional metadata tags that can be used to identify and organize rewrites. | `blackfridaysale` |
| Phrase Count | Denotes how many times this phrase was found in the source. This value is automatically set via machine learning. It does not need to be set manually. | `5` |
| Boost Factor | The factor to use to boost this phrase in matching queries. | `2.0` |
| Slop Factor | Phrase slop, or the distance between the terms of the query while still considering it a phrase match. | `10` |
2. Enter the number of words in the phrase in the **Word Count** field.
3. Click the **Save** button.
The search is re-run, with your search words identified as a phrase, and a Search box information panel drops down to show that the new query rewrite has fired:
## Synonym
You can specify synonyms for a specified search term so that alternative words with the same meaning are automatically used in the search query. When a customer enters a search term with a synonym match, the alternative words are used instead of, or in addition to, the original search term.
For example, the search term `sweater` could have the synonyms `pullover` and `jumper`.
1. From the list of query rewrite options, select **Synonym**. A form will appear:
This form contains the following fields:
| | | |
| - | - | - |
| Parameter | Description | Example Value |
| Surface Form | The term that has synonyms. | `sweater` |
| Direction | With a **oneway** search, the original search term is replaced by the synonym. In the example above, `sweater` would be replaced by the alternative words `pullover` and `jumper`. With a **symmetric** search, the search query is expanded to include the original term and the synonyms, resulting in a greater number of potential hits. In the example above, this time the query would include `sweater`, `pullover`, and `jumper`. | `symmetric` |
| Synonym Mappings | Synonyms for the surface form. | `pullover`, `jumper` |
| Confidence | Confidence score from the phrase job. A confidence level of `1` represents 100% confidence. For rules created automatically via machine learning, the confidence level will reflect the output from the machine learning model. | `1` |
| Tags | Optional metadata tags that can be used to identify and organize rewrites. | `blackfridaysale` |
| Count | How many times this term occurred in the signal data when it was discovered. This value is optional when a rewrite is being defined manually. | `5` |
2. Choose whether the direction is **oneway** or **symmetric**.
3. Enter one or more alternative words in the **Synonym Mappings** field.
4. Click the **Save** button.
The search is re-run using the synonyms, and a Search box information panel drops down to show that the new query rewrite has fired:
## Remove Words
The Remove Words feature is available in Fusion 5.4 and later.
You can create a Remove Words query rewrite to remove words from a query. This query rewrite is helpful when a word in the search query does not add value to the search results.
For example, you can rewrite a search query for `case study examples` to remove `examples` and then display results for `case study`.
1. From the list of query rewrite options, select Remove Words. A form appears:
This form contains the following fields:
| | | |
| - | - | - |
| Parameter | Description | Example Value |
| Phrase to remove | The words to remove from the trigger phrase. | `examples` |
| Trigger phrases | The query that prompts the removal of the phrase. The trigger phrase is not necessarily a complete query. If the query contains the trigger phrase, then Fusion removes the phrase in the Phrase to Remove field. | `case study examples` |
2. Enter a phrase to remove and a trigger phrase. Note that the phrase to remove is auto-populated with the query.
3. Click **Save**.
The search is re-run, and a Search box information panel drops down to show that the new query rewrite has fired:
## Learn more
The course for **Predictive Merchandiser: Rewrites Manager** focuses on using search rewrites to boost conversions and create better customer experiences by improving your relevancy.
In Predictive Merchandiser you can create search rewrites manually. You can also edit, test, review and publish the search rewrites that are generated automatically in Fusion from signals data.
The course for **Predictive Merchandiser: Rewrites Manager** focuses on using search rewrites to boost conversions and create better customer experiences by improving your relevancy.
# Templates
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/templates
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/templates
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/templates
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/3289
The Templates screen in the Rules Editor allows you to design, test, and implement a wide variety of search experiences. Results are conditionally delivered from multiple different query pipelines, enabling you to use Fusion’s search, browse, and AI-driven functionality throughout your site.
Access Templates from the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor) by navigating to **Relevance > Rules > Templates**. See **Use Predictive Merchandiser Templates** for instructions on using Templates.
This article teaches you how to use the [Templates](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/templates) feature in Predictive Merchandiser, including how to:
* [Create a template](#create-a-new-template)
* [Add zones](#add-a-zone-to-a-template) to the template
* Conditionally [trigger the template](#set-triggering-options)
* [Test](#test-the-template) the template in the Predictive Merchandiser screen
In the instructions below, a new template is created with two zones for use during a Cyber Monday sales event. One zone is used to display general search results while the other is used to promote specific products for the sale.
The zones used in this article use different query profiles. To follow the instructions as described, you will need to begin with an app that has at least two query profiles.
To begin, access Templates in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor) by navigating to **Relevance** > **Rules** > **Templates**.
The course for **Predictive Merchandiser: Template Manager** focuses on how to use the templates included in Predictive Merchandiser and create your own custom templates.
## Create a new template
1. From the Templates screen, click the **New Template** button:
2. Assign the template a unique name in the **Template Name** field. In this example, the value `Cyber Monday` is assigned.
3. Choose a **Template Type**. This example uses the `landing` template type.
This value is read-only and for informational purposes only.
4. Click the **Save** button to save the new template:
For full configuration, see:
* [Templates UI Configuration Options](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/templates-config)
* [Templating API Configuration Reference](/api-reference/templates-controller/get-template-information)
Images may display other releases if the information between releases does not differ. For example, the Fusion Templating API image reflects version 5.3.5 in both the Fusion 5.2 and 5.3 releases, but reflects 5.4.0 in the Fusion 5.4 release.
## Add a zone to a template
Zones enable Fusion users to create rules designed for specific query profiles. Combined with templates, this allows a search application to display results from multiple query profiles in the same view.
In this example, two new zones are created for the **Cyber Monday** template.
1. Add an existing zone to the template by clicking the **Existing** button. In this example, the **Main Results List** zone is added by selecting the zone and clicking the **Apply** button.
2. Create a new zone by clicking the **New** button:
3. Assign the zone a unique name in the **Display Name** field. For this example, the value `Promoted` is assigned.
4. Choose the **Query Profile** for this zone. This example uses a query profile named **promotional-results**. This query profile is configured with rules that pin products to the top of the results.
5. Choose the **Zone Type** for this zone. This example uses **main-results-list**.
This value is read-only and for informational purposes only.
6. Set the configuration options for the product image, title, and description per your preference.
7. Click the **Save** button to save the new zone:
For full zone configuration details, see [Zone Configuration Options](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/templates-config).
## Edit an existing zone
You can edit a zone that was already added to a template:
1. Select the **Start Task** button.
2. Select the pencil icon to begin editing the chosen page.
3. Each zone has a trash can icon to delete the zone. Select the wrench icon to edit the zone.
4. In the form that appears, make any necessary edits.
5. Click the **Save** button to save your changes.
## Set triggering options
Triggers describe conditions that must be met for a template to display. Trigger options include a set date range and specific search terms. See [Triggering Configuration Options](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/templates-config) for additional details.
### Date range
1. Click the **0 Active** button under **Triggering**:
If there are active triggers, this button changes to reflect that number. For example, this button will read **3 Active** when 3 triggers are active.
2. Click the date range selector and select a start and end date. For this example, choose a date range that includes today’s date.
3. Click the **Save** button to save your changes:
### Search terms
1. Click the **0 Active** button under **Triggering**.
2. Click the **Trigger** button:
3. In the **Search Terms** field, enter `cyber monday`.
4. Set the **Match Mode** field to `phrase`.
5. Click the **Add** button.
6. Repeat the process to add a second trigger with **Search Terms** set to `sales event` and **Match Mode** to `phrase`.
7. Click the **Save** button to save your changes:
## Test the template
1. Navigate to the **Merchandiser** screen.
2. Enter one of the [search terms](#search-terms) you configured in the **Set triggering options** instructions above. This example used the search terms `cyber monday` and `sales event`.
If today’s date does not fall within the template’s [date range trigger](#date-range), the template will *not* load.
If everything was configured correctly, you should see the **Cyber Monday** template is loaded. You can now create different rules for both zones, including pinning promoted products in the **Promoted** zone.
Templates are only available with a valid license for [Predictive Merchandiser](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/overview). If available, you will see the templates icon in the left navigation panel:
For configuration information, see [Templates UI Configurations Reference](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/templates-config) and the [Templating API Configuration Reference](/api-reference/templates-controller/get-template-information).
The course for **Predictive Merchandiser: Template Manager** focuses on how to use the templates included in Predictive Merchandiser and create your own custom templates.
## Templates
Templates are comprised of two primary elements, the [template](#templates) itself and its associated [zones](#zones). A variety of conditions, or *triggers*, determine which template is shown to the customer. These triggers can include whether the template is active, the template’s precedence over other templates, the customer’s query, and whether the template is triggered within a specific date range (if set).
### Default templates
A new app starts with built-in templates: Detail Page, Search Results, and Typeahead. The Detail Page and Search Results Page templates are immediately available in the **Templates** screen.
**Detail Page:** The Detail Page defines what a customer sees when they view the details of a product. It can be configured in the **Templates** screen or by hovering over a product in the **Predictive Merchandiser** screen and clicking the **Detail Page** button that appears.
**Search Results page:** The Search Results Page defines what a customer sees when they reach a search results page. This template can also be configured in the **Template** screen. Alternatively, you can click **Merchandiser > Start Task > Edit Template**.
**Typeahead Template:** The Typeahead Template allows you to configure what is displayed when query terms are typed into the search bar. To add this template, click **Merchandiser > Start Task > enter a search query term**. When the system displays "Typeahead not created. Do you want to create a new typeahead template?", click **New** to create the typeahead template.
### Custom templates
In some cases, you may want to create custom template types to add more options for curating pages in your search application. You can create a custom template type in the **Template** screen by following the steps in **Create a Custom Predictive Merchandiser Template Type**.
This article teaches you how to create custom [Templates](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/templates) for Predictive Merchandiser.
To begin, access Templates in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor) by navigating to **Relevance** > **Rules** > **Templates**.
## Create a new template type
1. From the Template Manager screen, click **New Template**.
2. Add a user-friendly **Display Name** for your template.
3. In the **type** field, create a new template type by entering the name and clicking **Add**.
4. Click **Save**. The custom template type and template display in the list of templates.
5. Configure the template as desired.
6. Click **Publish**. Review the changes, and click **Confirm and Publish** to publish your custom template type and template. The template is now available in the **Merchandiser** screen. The template activates based on your configurations.
### Hierarchical template triggers
Hierarchical template triggers allow you to fire a search configuration template or rules at a parent or child level in a hierarchy, and even at a faceted level.
**Ways to use hierarchical templates:**
* For a sale on "Winter Coats," you can fire a template at the "Outerwear" level or at any of its children.
* For an overstock on a particular clothing size, you could fire a template directly at the “Women’s Outerwear, Size Small” filter.
* When rolling out a new company policy, you could fire a zone on all pages from the homepage down so everyone sees the most important links and documents for the new policy.
#### Hierarchical fields settings
To configure hierarchical fields settings:
1. Access the application and click **Relevance > Rules > Templates**.
2. Click the three vertical dots to the right of the application name to display the Hierarchical fields settings section.
3. Complete one of the following:
Any symbol can be used as a delimiter.
* To edit an existing field, click **Field** and select the value to edit. Enter the appropriate value in the **Delimiter** field and click **Apply**.
* To delete a field setting, select the value in **Field**, click **X**, and then click **Apply**.
* To cancel any changes before you apply them, click **Cancel**.
#### Hierarchical trigger settings
To configure hierarchical trigger settings:
1. Access the application and click **Relevance > Rules > Templates**.
2. Click to open a template.
3. In the Triggering section, click the trigger to display the settings. You can also click **+ Trigger** to add a new trigger.
4. Enter values in the **Search terms** and **Match mode** fields.
5. In the Filters section of the Edit Trigger screen (or New Trigger screen), click **+ Add** to add a new filter.
6. Enter values in the **Field name** and **Field value** fields.
7. Click the **Apply to Descendant** option to apply the trigger at child levels of the hierarchy.
The **Apply to Descendant** function only applies to fields that are configured as hierarchical.
8. Click **Exact** to enable exact filter matches.
9. Save the configuration. To close the window without saving changes click **X** or **Cancel**.
The URL Content tab fields and values do not apply to hierarchical trigger functions.
## Zones
Zones enable Fusion users to create rules designed for specific query profiles. Combined with templates, this allows a search application to display results from multiple query profiles in the same view.
For example, if a customer searches for `climbing`, one zone can fetch results for climbing apparel, and another zone can fetch blog posts about climbing. The same zone can be used in multiple templates, and changes to a zone will be made on all templates that zone is used in.
In the example above, a unique template is fired for the query `cyber monday`. This template has two zones:
* **Promoted** - Display advertised products.
* **Main Results List** - Display general search results.
# Lucidworks MCP
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/mcp
Lucidworks MCP (Model Context Protocol) enables agentic search using Claude to search your Fusion data.
You configure the connection once, and Claude can immediately search your indexed content.
**Self-hosted Fusion users**: See [Monitor MCP Server usage](/docs/5/fusion/operations/monitoring-and-reporting/mcp-monitoring) to set up Grafana dashboards for tracking MCP request volume, latency, and error rates.
## Benefits
Lucidworks MCP server provides these advantages:
* **Self-service integration:** Configure Claude to access Fusion data without custom API development.
* **Enterprise security:** Leverages Fusion's existing security controls, including field-level security, role-based access, and audit logging.
* **Multi-pipeline access:** Access multiple query pipelines through a single MCP endpoint, enabling Claude to search across different data sources and configurations based on context.
* **Multi-tenant architecture:** Support multiple use cases with different data access patterns using query profiles and collection isolation.
* **Query pipeline integration:** Claude benefits from your custom query pipelines, ML models, and relevance tuning.
* **Extensible tool framework:** Add custom tools programmatically that automatically become available to Claude.
## Supported AI clients
Lucidworks MCP works with any MCP-compatible AI client, including these popular clients:
* **Claude Code** - Command-line AI agent with built-in MCP support (recommended for easiest setup).
* **Claude Desktop** - Desktop application for conversational AI.
## MCP server setup
### Before you begin
Before configuring MCP access, ensure you have the following:
* Fusion 5.17 or later deployed with the MCP Server service running
* A Fusion app configured with indexed content
* A query profile created for MCP access
* Claude Code (recommended) or Claude Desktop
The MCP server runs on your Fusion infrastructure at `https://FUSION_HOST/mcp`.
### Set up authentication
Fusion MCP Server supports two authentication methods:
Generate an [API key](/docs/5/fusion/operations/security/access-control/api-keys) in Fusion with the appropriate permissions for your transport mode.
**HTTP mode (Claude Code, custom integrations):**
```
POST,GET:/mcp
```
**STDIO mode (Claude Desktop):**
```
GET:/query-profiles/search-experiences
GET:/collections/*/schema/fields
GET:/collections/*/schema/dynamicfields
GET:/apps/*/query-profiles/**
GET:/apps/*/query/**
```
You can replace the wildcards with specific values from your search experience configuration:
```
GET:/query-profiles/search-experiences
GET:/collections//schema/fields
GET:/collections//schema/dynamicfields
GET:/apps//query-profiles/
GET:/apps//query/
```
To grant access to multiple search experiences, duplicate the permissions for each collection, app, and query profile combination.
Fusion 5.18 and later supports OAuth 2.1 authentication using external identity providers that support the OAuth 2.1 protocol.
This enables enterprise single sign-on workflows and provides automatic token lifecycle management.
**Advantages over API keys:**
* User-specific access control tied to authenticated users
* Automatic token expiration and refresh (access tokens expire after 30 minutes)
* Immediate revocation when users are deleted
* Standard protocol support for all OAuth 2.1-compliant AI clients
**Prerequisites:**
Your external identity provider must support the following:
* OAuth 2.0 / 2.1 with Authorization Code flow and PKCE (RFC 7636)
* Authorization Server metadata (RFC 8414)
* OAuth redirect URIs for callback handling:
* **Azure AD, Keycloak**: Support wildcard redirect URIs like `http://localhost/callback`
* **Okta**: Requires specific callback ports (for example, `http://localhost:3118/callback`)
**Configuration steps:**
Create a JWT realm in Fusion with the following configuration:
* **Issuer**: Must match the external IDP's issuer from the Authorization Server metadata
* **JWK Set URL**: Must point to the external IDP's public keys endpoint
See [Configure Fusion for JWT](/docs/5/fusion/reference/config-ref/configure-fusion-for-jwt-5x) for detailed setup instructions.
Create an OAuth client (Single Page Application type) in your external identity provider with the appropriate redirect URI:
* **Azure AD, Keycloak**: Use wildcard redirect URI `http://localhost/callback`
* **Okta**: Use specific callback port like `http://localhost:3118/callback` (or configure multiple ports)
Note the client ID for use in the next step.
Claude Code starts a local server on a random port for OAuth callbacks.
IDPs that support wildcard redirect URIs work automatically.
For Okta, users can specify a callback port using `--callback-port` when adding the MCP server.
Set the following environment variables in the API Gateway service:
* `OAUTH_CLIENT_ID`: The OAuth client identifier (pre-created in the external IDP) that will be returned to all MCP clients during registration
* `OAUTH_AUTHORIZATION_SERVER_URL`: Base URL of the external IDP's authorization server used to fetch OAuth metadata via `/.well-known/oauth-authorization-server`
**Registration modes:**
Fusion supports two OAuth registration modes:
* **Dynamic registration**: Use when your IDP does not require authorization for its registration endpoint. MCP clients can register directly using the `registration_endpoint` discovered from `/.well-known/oauth-authorization-server`.
* **Static registration**: Use when your IDP requires authorization for its registration endpoint (such as Okta). Fusion exposes a `/oauth2/register` endpoint that returns the pre-configured `OAUTH_CLIENT_ID` to all MCP clients.
### Configure the MCP server
In Fusion 5.18 and later, you configure MCP search experiences through the Query Profile panel in the Fusion UI.
This is the only supported configuration method for 5.18+.
Fusion 5.17 uses a configuration file (configmap) instead.
For any query profile, you can configure an MCP search experience in the Query Profiles panel of the Fusion UI.
Repeat these steps for each query profile that you want to make available as an MCP search experience.
In Fusion, navigate to **Querying** > **Query Profiles** and select the query profile you want to make available as an MCP search experience.
Scroll to the **MCP Search Experience** section and select the **Enable for MCP Processing** checkbox:
Fill in the required fields:
* **Name**: Display name for this search experience (required)
* **Description**: Description that helps Claude understand when to use this search experience (required)
* **Content Field**: Optional field name to use as the document description source
The form validates that required fields are populated before allowing you to save.
**How document descriptions are determined:**
When **Content Field** is specified:
* If the field contains a value in the Query Profile response, that value is used.
* If the field is empty, the MCP server falls back to checking default fields like `body_t`.
When **Content Field** is empty:
* The MCP server checks default fields like `body_t` to generate document descriptions.
Click **Save** to apply your changes.
The search experience is immediately available to MCP clients.
Repeat these steps for each query profile you want to expose as an MCP search experience.
Open your Fusion `values.yaml` configuration file.
Add the MCP server configuration to the `query-pipeline` service configuration with at least one search experience, as in this example:
```yaml wrap theme={"dark"}
query-pipeline:
searchExperiences:
- name: "Product Catalog"
description: "Search outdoor gear, camping equipment, and sporting goods inventory"
application-name: "products"
query-profile: "catalog-search"
- name: "Employee Resources"
description: "HR policies, benefits information, and internal documentation"
application-name: "hr-docs"
query-profile: "internal-search"
- name: "Customer Support"
description: "Product guides, troubleshooting articles, and warranty information"
application-name: "support"
query-profile: "support-search"
```
Replace the example values above with your own Fusion configuration.
You can configure multiple search experiences that point to different applications and query profiles.
Claude automatically selects the appropriate search experience based on the query context and the search experience descriptions.
The MCP server validates the configuration at deployment and requires at least one valid search experience with all required fields populated.
If no valid search experiences are configured, the service fails to start with an error message.
**Search experience configuration parameters:**
List of available search experiences.
Each search experience defines how the MCP server routes queries to Fusion applications.
You must configure at least one search experience.
Display name of the search experience.
Claude uses this name to route queries.
Description of what this search experience contains.
This helps Claude determine which search experience to use for a given query.
A well-crafted description helps provide a seamless experience for users.
Make your descriptions as detailed as possible to enable Claude to quickly identify the right search experience for any prompt.
Fusion application name to query.
Fusion query profile to use for queries.
**Optional: Tool description configuration**
You can also configure custom descriptions for the MCP tools.
Description for the `search_documents` tool.
This parameter is optional and has a safe default that includes critical dependency information.
Only override this if you understand [how the tools work together](#how-the-tools-work-together).
The default description instructs Claude to first call `list_search_experiences` to obtain valid search experience names.
Incorrect descriptions can break the automatic workflow.
Description for the `list_search_experiences` tool.
This parameter is optional and has a safe default that includes usage examples.
Only override this if you understand [how the tools work together](#how-the-tools-work-together).
Modifying this incorrectly can prevent Claude from discovering available search experiences properly.
### Connect an AI agent
Connect your AI agent using the authentication method you configured in the previous step: API key or OAuth 2.1.
Claude Code connects with a single command.
```bash wrap theme={"dark"}
claude mcp add --transport http fusion https://FUSION_HOST/mcp
```
Replace `FUSION_HOST` with your Fusion host.
Claude Code opens a browser window to complete the OAuth flow with your identity provider.
**Okta**: Okta requires a specific callback port instead of a wildcard redirect URI.
Add `--callback-port` to match the port configured in your OAuth client:
```bash wrap theme={"dark"}
claude mcp add --transport http --callback-port 3118 fusion https://FUSION_HOST/mcp
```
If you receive errors when running this command, check your Claude Code version.
The `--transport` and `--callback-port` flags require a recent version of Claude Code.
```bash wrap theme={"dark"}
claude mcp list
```
You should see `fusion` listed, with the HTTP transport.
```bash wrap theme={"dark"}
claude mcp add --transport http fusion https://FUSION_HOST/mcp --header "x-api-key: FUSION_API_KEY"
```
Replace the placeholder values with your Fusion host, port, and API key.
If you receive errors when running this command, check your Claude Code version.
The `--transport` and `--scope` flags require a recent version of Claude Code.
By default, MCP servers are scoped to the current project.
To make the server available globally across all projects, add `--scope user`.
To explicitly scope it to the current project, add `--scope project`.
```bash wrap theme={"dark"}
claude mcp list
```
You should see `fusion` listed, with the HTTP transport.
Lucidworks provides an MCP Bundle (`.mcpb`) file; this is the recommended method.
Alternatively, you can configure Claude Desktop manually by editing the JSON configuration file.
1. Make sure Claude Desktop is already installed.
2. Download the MCPB file that matches your Fusion version from the [fusion-cloud-native `mcp` directory](https://github.com/lucidworks/fusion-cloud-native/tree/master/mcp):
* Fusion 5.17.x: [`fusion-5-17-0.mcpb`](https://github.com/lucidworks/fusion-cloud-native/raw/refs/heads/master/mcp/fusion-5-17-0.mcpb)
* Fusion 5.18.x: [`fusion-5-18-0.mcpb`](https://github.com/lucidworks/fusion-cloud-native/raw/refs/heads/master/mcp/fusion-5-18-0.mcpb)
3. Double-click the downloaded file. The file opens in Claude Desktop, where you can review the file's details.
4. In Claude Desktop, click **Install**:
5. In the next dialog, click **Install** to confirm the installation.
6. Enter your MCP server URL (as in `https://FUSION_HOST/mcp`) and your API key:
7. Click **Save**.
8. Click the **Disabled** switch to toggle it to **Enabled**:
9. Close the Lucidworks Fusion modal in Claude Desktop.
10. Verify the connection.
Go to **Customize** > **Connectors** in the Claude Desktop interface to confirm the MCP server is connected.
Ask Claude to search Fusion to verify it uses the Lucidworks MCP tools automatically.
1. Locate the configuration file for your operating system:
* **macOS:** `~/Library/Application Support/Claude/claude_desktop_config.json`
* **Windows:** `%APPDATA%\Claude\claude_desktop_config.json`
* **Linux:** `~/.config/Claude/claude_desktop_config.json`
2. Edit the file to add the Fusion MCP server:
```json theme={"dark"}
{
"mcpServers": {
"fusion": {
"command": "docker",
"args": [
"run", "--rm", "-i", "--name", "mcp-server",
"-e", "FUSION_URL=https://FUSION_HOST/mcp",
"-e", "FUSION_API_KEY=API_KEY",
"lucidworks/mcp-server-stdio:5.18.0"
]
}
}
}
```
Replace the placeholder values with your specific credentials.
Use the image tag that matches your Fusion version, such as `5.17.0` or `5.18.0`.
3. Restart Claude Desktop.
4. Verify the connection.
Look for the hammer icon (🔨) in the Claude interface to confirm the MCP server is connected.
Ask Claude to search Fusion, and it should automatically use the `fusion` tool.
### Test the connection
After configuring your AI agent, test the connection with these example prompts.
These prompts verify that the MCP server is working correctly and that Claude can discover and use the configured search experiences.
Test whether Claude can successfully retrieve the configured search experiences from Fusion.
Using the Lucidworks MCP's `list_search_experiences` tool, show me all available search experiences in this Fusion instance.
Test whether Claude can identify the correct search experience by name and execute a search query.
Using the Lucidworks MCP's `search_documents` tool, search the Product Catalog search experience for wireless headphones and return the results.
Test that Claude can automatically route queries to the correct search experience based on context, demonstrating access to multiple query pipelines through a single MCP endpoint.
Using the Lucidworks MCP, first search the Product Catalog for wireless headphones, then search Employee Resources for the return policy. Show results from both search experiences.
Test whether Claude can match natural language queries to search experience descriptions and return relevant results.
Using the Lucidworks MCP's `search_documents` tool, identify the appropriate search experience and search for information about benefits and time off policies, then return the results.
## Search with MCP
Lucidworks MCP server provides four tools that work together in a discovery-to-execution pattern.
Claude automatically determines which tool to call based on the user's query.
### List search experiences
The `list_search_experiences` tool discovers available search experiences configured in Fusion.
You can use this tool in the following situations:
* You need to see what search experiences are available.
* You need to determine which search experience to use for a specific query.
* You need to help users who don't know what they can search for.
Claude automatically calls this tool when users ask questions like "What can I search here?" or "What do I find in Fusion?".
The tool returns a list of configured search experiences with their names and descriptions.
This tool takes no parameters.
### Search documents
The `search_documents` tool searches for documents and returns relevant results from Fusion collections.
Based on user input, it automatically selects a search experience and can optionally filter and sort results.
For example, a user can input "Find all reports from the last six months and sort them with the newest ones first" to trigger a query that uses `filters` to specify the document type and date range and `sort` to order the results by date, newest to oldest.
### Get a document
The `get_document` tool fetches a specific document by its ID.
This is useful when `search_documents` has already been called and the user inquires about a specific document returned in those results.
The tool returns complete details about the document, such as its title, description, URL, similar documents (if [MoreLikeThis](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/solr-more-like-this-query-stage) is implemented), and highlighted fields (if [highlighting](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/highlight-query-stage) is implemented).
### Get the field schema
The `get_schema` tool retrieves available fields within a specific search experience and returns schema information with sortable and searchable capability flags.
You can use this tool in the following situations:
* You need to discover what fields exist within a particular search experience.
* You need to determine which fields can be used for sorting.
* You need to determine which fields can be used for filtering.
* You need to display field information to users with human-readable names.
Claude automatically calls this tool before using the `sort` or `filters` parameters when searching documents, or when users ask questions like "What fields can I sort by?" or "What can I filter by?".
The tool returns field names with their display values and capability flags.
By default, the tool returns only populated fields.
Set `includeEmptyFields=true` to retrieve the complete schema.
### How the tools work together
Claude uses these tools in the following order:
1. When a user asks a question without specifying a search experience, Claude calls `list_search_experiences` to discover available options.
2. Claude analyzes the search experiences and matches them to the user's query intent.
3. Claude calls `search_documents` with the appropriate search experience.
4. Claude returns results to the user.
For example, when a user asks "Find hiking backpacks," Claude automatically completes the following steps:
1. Calls `list_search_experiences` and receives the following data:
```json wrap theme={"dark"}
{
"searchExperiences": [
{"name": "Product Catalog", "description": "Outdoor gear and sporting equipment"},
{"name": "Support Articles", "description": "Product guides, warranties, and care instructions"}
]
}
```
2. Identifies the "Product Catalog" search experience as the one that best matches the prompt.
3. Calls `search_documents` with `searchExperience: "Product Catalog"` and the user's query.
4. Returns the product results.
This pattern allows users who know nothing about Fusion's configuration to simply ask questions and get results without needing to understand the underlying structure.
## Example use cases
The following examples demonstrate common MCP integration patterns.
An employee uses Claude to search internal technical documentation:
```text User query wrap theme={"dark"}
Find all technical documentation about JWT authentication from the past six months
```
```json MCP call (automatic) wrap theme={"dark"}
{
"method": "tools/call",
"params": {
"name": "search_documents",
"arguments": {
"searchExperience": "technical_docs",
"query": "JWT authentication",
"filters": [
{"field": "doc_type", "operator": "equals", "value": "technical"},
{"field": "date", "operator": "relative_date_range", "timeUnit": "MONTHS", "timeValue": 6}
],
"rows": 10
}
}
}
```
```text Claude output wrap theme={"dark"}
I found eight technical documents about JWT authentication from the past six months:
1. **JWT Authentication Best Practices 2026** (Jan 15, 2026)
This guide covers modern JWT authentication patterns including...
[View Document](https://docs.company.com/security/jwt-guide)
2. **Implementing JWT in Node.js** (Feb 2, 2026)
Step-by-step tutorial for implementing JWT tokens...
[View Document](...)
```
A Japanese-speaking employee searches in their native language:
**User query (Japanese):**
```text wrap theme={"dark"}
認証に関する最新のドキュメントを教えてください
```
Fusion processes the Japanese query, applies language-specific tokenization, and returns results with Japanese snippets.
Claude displays results in Japanese, maintaining the natural language interaction.
If your query pipeline is configured with [highlighting](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/highlight-query-stage) or [MoreLikeThis](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/solr-more-like-this-query-stage), the MCP server can include those features in its responses:
```json Example response wrap theme={"dark"}
{
"jsonrpc": "2.0",
"id": 10,
"result": {
"content": [],
"isError": false,
"structuredContent": {
"summary": "Found 3 documents for query: search",
"documents": [
{
"id": "doc-123",
"title": "Introduction to Search Analytics",
"url": "https://docs.example.com/search-analytics",
"moreLikeThis": [
{
"id": "doc-456",
"title": "Query Performance Optimization"
},
{
"id": "doc-789",
"title": "Relevance Tuning Best Practices"
},
{
"id": "doc-321",
"title": "Understanding Search Metrics"
}
],
"highlighting": {
"title_t": ["Introduction to Search Analytics"],
"description_t": ["Learn how to analyze and improve your search performance"]
}
}
],
"searchExperiences": [],
"querySpellcheckSuggestions": []
}
}
}
```
A user asks Claude to search across multiple related content types in a single query:
**User query:**
```text wrap theme={"dark"}
Find all customer feedback about our mobile app performance
```
**Search experience configuration:**
The workspace has three search experiences configured:
* "Customer Support Tickets" - Support cases and resolutions
* "Product Reviews" - App store reviews and ratings
* "User Surveys" - Customer satisfaction surveys and NPS data
**MCP behavior:**
Claude recognizes that customer feedback spans multiple sources and automatically searches all three experiences in parallel, then synthesizes the results.
**Claude output:**
```text wrap theme={"dark"}
I found customer feedback about mobile app performance across three sources:
**Support Tickets:**
- 23 recent tickets mention slow loading times on iOS devices
- Most common issue: image gallery performance (12 tickets)
[View tickets](https://...)
**App Store Reviews:**
- Average 3.2 stars for performance (down from 4.1 last quarter)
- Top complaint: "App freezes when scrolling through photos"
- 47 mentions of battery drain during video playback
[View reviews](https://...)
**User Surveys:**
- 67% of respondents rate mobile app speed as "needs improvement"
- Performance is the #2 requested enhancement after offline mode
- Users on Android 12+ report better experience than iOS users
[View survey results](https://...)
**Summary:** Performance concerns are consistent across all feedback channels, with image and video handling being the primary pain points, particularly on iOS devices.
```
This demonstrates how a single query can benefit from searching multiple related content sources, with Claude automatically determining which search experiences are relevant and synthesizing results from all of them.
## Example multi-tenant configurations
Lucidworks MCP server supports multi-tenant architectures where different users or use cases access different data subsets.
You can configure separate MCP endpoints with different query profiles:
```bash Employee configuration (full access) wrap theme={"dark"}
claude mcp add --transport http fusion-employee \
https://FUSION_HOST/mcp \
--header "x-api-key: EMPLOYEE_KEY"
```
```bash Contractor configuration (restricted access) wrap theme={"dark"}
claude mcp add --transport http fusion-contractor \
https://FUSION_HOST/mcp \
--header "x-api-key: CONTRACTOR_KEY"
```
* The employee API key accesses a server with search experiences configured for full access to all collections.
* The contractor API key accesses a server with search experiences limited to public documentation and shared resources.
You can support both public (unauthenticated) and authenticated access patterns:
```json Public kiosk configuration wrap theme={"dark"}
{
"mcpServers": {
"fusion-public": {
"type": "http",
"url": "https://FUSION_HOST/mcp",
"headers": {
"x-api-key": "PUBLIC_READONLY_KEY"
}
}
}
}
```
```json Authenticated employee configuration wrap theme={"dark"}
{
"mcpServers": {
"fusion-internal": {
"type": "http",
"url": "https://FUSION_HOST/mcp",
"headers": {
"x-api-key": "EMPLOYEE_KEY"
}
}
}
}
```
This approach enables use cases like showroom kiosks (public read-only access with search experiences configured for public documents) and internal employee tools (authenticated with search experiences configured for full access and security trimming) using the same Fusion instance.
## Troubleshooting
If you cannot connect to the MCP server, verify the following:
1. Verify the endpoint is accessible:
```bash expandable wrap theme={"dark"}
curl -X POST https://FUSION_HOST/mcp \
-H "Content-Type: application/json" \
-H "Accept: text/event-stream, application/json" \
-H "x-api-key: API_KEY" \
-i \
-d '{
"jsonrpc": "2.0",
"id": 1,
"method": "initialize",
"params": {
"protocolVersion": "2025-11-25",
"capabilities": {},
"clientInfo": {
"name": "test-client",
"version": "1.0.0"
}
}
}'
```
The protocol version in this example (`2025-11-25`) may differ from your installed version. See the [MCP schema specification](https://modelcontextprotocol.io/specification/2025-11-25/schema) for the current version.
If the endpoint is accessible, it returns output similar to the example below.
Any other result means that the endpoint is not accessible.
```bash expandable wrap theme={"dark"}
{
"jsonrpc": "2.0",
"id": 1,
"result": {
"protocolVersion": "2024-11-05",
"capabilities": {
"logging": {},
"prompts": {
"listChanged": false
},
"resources": {
"subscribe": false,
"listChanged": true
},
"tools": {
"listChanged": true
}
},
"serverInfo": {
"name": "Lucidworks MCP Server",
"version": "1.0.0"
}
}
```
}
````
2. Confirm that the Fusion version is 5.17 or later:
```bash wrap
curl https://FUSION_HOST:FUSION_PORT/api/admin/version
````
3. Check the URL path is exactly `/mcp` (no trailing slash, not `/api/mcp`).
4. Test with verbose logging:
```bash wrap theme={"dark"}
claude mcp test fusion --verbose
```
1. Regenerate the API key in Fusion Admin (the key may be expired or revoked).
2. Verify the header format is exactly `x-api-key: API_KEY` (case-sensitive).
3. Test the key directly against Fusion's API:
```bash wrap theme={"dark"}
curl -H "x-api-key: abc123" https://FUSION_HOST:FUSION_PORT/api/apps/YOUR_APP/query/YOUR_PROFILE
```
1. Verify the query profile exists in the specified Fusion app.
2. Check the profile name is an exact match (case-sensitive).
3. Ensure the API key's role has permission to execute queries on this profile.
The `x-api-key` header is required for authentication:
```bash wrap theme={"dark"}
claude mcp add --transport http fusion https://FUSION_HOST/mcp --header "x-api-key: API_KEY"
```
Run this command to verify your configuration:
```bash wrap theme={"dark"}
claude mcp list
```
The response should show your MCP URL:
```bash wrap theme={"dark"}
fusion - https://FUSION_HOST/mcp (http)
```
If searches complete successfully but return no results, try these steps.
1. Verify the Fusion app contains indexed content:
```bash wrap theme={"dark"}
curl "https://FUSION_HOST:FUSION_PORT/api/apps/YOUR_APP/query/YOUR_PROFILE?q=*:*"
```
2. Check that the query profile has access to the expected collections.
3. Test the same query directly in Fusion's Query Workbench to confirm results exist.
4. Review security trimming rules that might filter out documents for the API key's role.
If MCP searches are slow, try these steps.
1. Check the query profile's pipeline stages for expensive operations.
2. Review the query complexity (complex filter queries can impact performance).
3. Monitor Fusion's query logs for slow queries.
4. Consider optimizing ML model execution or caching strategies.
5. Verify network latency between Claude and the Fusion instance.
If the Lucidworks MCP server fails to start with an error message about search experiences, follow these steps.
**Symptom:**
```text wrap theme={"dark"}
ERROR - No valid search experiences configured. Server cannot start without at least one valid search experience.
Exception in thread "main" java.lang.IllegalStateException: No valid search experiences configured...
```
**Solution:**
1. Add at least one search experience with all required fields to `values.yaml`:
```yaml wrap theme={"dark"}
query-pipeline:
searchExperiences:
- name: "Company Knowledge" # Required
description: "Internal documentation..." # Required
application-name: "APP_NAME" # Required
query-profile: "default" # Required
```
2. Restart the server.
3. Verify startup logs show: `Found 1 valid search experience(s): [Company Knowledge]`
If `list_search_experiences` returns fewer search experiences than you configured, follow these steps.
**Solution:**
1. Check server startup logs for: `INFO - Found N valid search experience(s): [names]`
2. Compare logged names with your configuration.
3. Verify all required fields are populated (`name`, `description`, `application-name`, `query-profile`).
4. Fix configuration and restart.
If your search query returns an error message about the search experience not being configured, verify the search experience name.
**Symptom:**
```json wrap theme={"dark"}
{
"summary": "Search experience is not configured",
"documents": [],
"searchExperiences": [...]
}
```
**Solution:**
1. Call `list_search_experiences` to get exact names.
2. Use the exact name (case-sensitive).
Common mistakes:
* ❌ "engineering docs" (wrong case)
* ✅ "Engineering Docs" (exact match)
If you receive a generic error message about being unable to process the request, verify your Fusion configuration and connectivity.
**Symptom:**
```json wrap theme={"dark"}
{
"content": [{"type": "text", "text": "Error: Unable to process request"}],
"isError": true
}
```
**Solution:**
Follow these troubleshooting steps:
1. Verify Fusion is running and accessible.
2. Check that `application-name` exists in Fusion (case-sensitive).
3. Verify `query-profile` exists and is enabled in Fusion.
4. Check authentication and network connectivity.
5. Review MCP server logs for detailed error messages.
If your search completes successfully but returns no documents, verify your data and query configuration.
**Symptom:**
```json wrap theme={"dark"}
{
"summary": "Found 0 documents",
"documents": [],
"searchExperiences": []
}
```
**Solution:**
Follow these troubleshooting steps:
1. Verify documents are indexed in Fusion (check Collections in Fusion UI).
2. Try omitting the query parameter to return all documents (defaults to `*:*`):
```json wrap theme={"dark"}
{
"searchExperience": "Company Knowledge"
}
```
3. Try a broader query.
4. Test the same query directly in Fusion's Query Workbench.
5. Verify `application-name` points to the correct Fusion application.
6. Check query profile settings for permission filters.
# Getting Data Out
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/3228
Getting data out of Fusion can mean querying your content or your signals, or accessing other data for analytics.
## Querying Fusion
Querying Fusion means sending a request to Fusion’s [Query API endpoints](/api-reference/query-profiles-api/get-the-service-status), specifying the query profile and any Solr query parameters. The request is processed by the query pipeline associated with the specified profile, and the response includes any enhancements configured in the query pipeline. You can also enter queries in the [Query Workbench](/docs/5/fusion/intro/ui-tour/query-workbench) to test your query pipeline.
Start by learning [the basics of querying](/docs/5/fusion/getting-data-out/query-basics/overview), then about the out-of-the-box [query enhancement features](/docs/5/fusion/getting-data-out/query-enhancement/overview). For even more sophisticated personalization of search results, see [Advanced Query Enhancement](/docs/5/fusion/getting-data-out/advanced-query-enhancement/overview).
## Data analytics
Fusion provides search and analytic functions that allow you to:
* Explore and analyze the data retrieved
* Run standard reports and create custom reports
* Confirm the search results are what you intended to retrieve
* Verify the accuracy of the results
* Enable and run enhanced query options that can power virtual assistants and chatbots
* Run options to assist with predictive analysis
See [Data Analytics](/docs/5/fusion/getting-data-out/data-analytics/overview) for more information.
# Domain Specific Language (DSL)
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-basics/domain-specific-language
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-basics/domain-specific-language
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-basics/domain-specific-language
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/8833
Fusion Domain Specific Language (DSL) provides expected search results as a JSON response in a way that reduces search query complexity for the user.
Previously, users needed to understand complex syntax to express certain search queries in Fusion (for example, the best way to express a facet filter). DSL gives Fusion control over how to execute a query by transforming a structured query input into a Solr request, where we can add intelligence around the index and the user’s intent.
DSL is deprecated in Fusion 5.9.4 and will be removed in a future release. Be aware that using Search DSL can cause performance degradation during queries. You can use the [DSL to Legacy Parameters](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/dsl-to-legacy-query-stage) query pipeline stage to convert your DSL requests to the legacy (Solr) format.
## Use existing DSL query pipelines with Commerce Studio
DSL does not work with [Commerce Studio](/docs/lw-platform/lw-cs/overview). If you have existing query pipelines with DSL that you want to use with Commerce Studio, instead of recreating the entire pipeline, you can keep the pipeline and remove DSL by doing the following.
1. For DSL-enabled pipelines where supported search mode = `all` or `dsl`, select `legacy` option instead.
2. Update or disable pipeline stages used solely for DSL.
1. For query stage parameters, remove any fields with DSL values mapped.
2. In JavaScript, comment out or remove instances of `with(DSL)` code blocks.
## Why Search DSL
Fusion Search Domain Specific Language (DSL) reduces search query complexity for the user. Fusion Search DSL supports expressive search queries and responses via a structured, modern JSON format.
Search DSL is an alternative to the current (now referred to as) “Legacy” format for performing search queries. Previously, users needed to understand complex syntax to express certain search queries in Fusion (for example, the best way to express a facet filter).
Compared to the legacy Solr parameter format, Search DSL is structured to more closely align with the central concepts of Fusion and provide a more usable alternative for expressing complex Fusion queries.
Search DSL gives Fusion control over how to execute a query by transforming a structured query input into a Solr request, at which time Fusion can add intelligence around the index and the user’s intent.
### Limitations
Despite the advantages, there are still some important limitations to be aware of when deciding whether to use the Fusion Search DSL.
These limitations only apply to DSL queries. Legacy queries can still be issued separately from DSL queries without being subject to these limitations.
1. **Head / Tail Rewrite**. Support is currently limited when using DSL queries:
1. In order to work on DSL queries, the `Improved Query` must be entered in as a JSON string representing the desired `main` query that should be issued (this replaces the `queryDefinition.main` field in the rewritten DSL query)
2. The query rewrites produced by the built-in head/tail rewrite job will NOT work on DSL queries, as the job only outputs legacy-style rewrites
2. **Query Stage Support**. All query stages are fully supported
3. **Rules Support**. All kinds of rules are fully supported with the exceptions noted below
1. **Set Params**. Not supported
2. **Custom Rule**. Not supported
## Request Format
This section describes each top-level field of a Search DSL request.
### Overview
The Search DSL format is supported in the following endpoints:
| | |
| - | - |
| **query pipelines** | `query-pipelines/QUERY_PIPELINE/collections/COLLECTION_ID` |
| **query profiles** | `query/QUERY_PROFILE` |
| **templating render** | `templating/renderDSL/APP_NAME` |
In all cases, it should be a POST with a `Content-Type: application/json` header.
The below table briefly summarizes the function of each of the top-level fields.
| Field | Description |
| - | - |
| `queryDefinition` | Defines the logic of what to query for. The `userQuery` holds the raw user query (if there is one) which is used by various Fusion components such as Rules to determine which rewrites / rules trigger. The `main` and `filter` fields accept a variety of different possible query types (terms and singleTerm are shown here) and together define the matching criteria. Additional fields are also possible here for boosting and grouping - `boostsByValues`, `boostsByQuery`, `groupedQuery`, `groupedFilters`. |
| `results` | Defines how the results should be displayed and organized. A variety of fields are available here, such as `cursorMark`, `start`, and `size` for pagination, sort for sorting, fields for the list of fields to display for each document, highlight for highlighting features, and group for grouping. |
| `facets` | The faceting configuration for the query. Defines the fields to perform faceting on as well as the desired behavior of the returned facet values. Supports range facets in the `ranges` field and field facets in the `fields` field. |
| `context` | Accepts parameters used by some query stages as well as DSL hints. Should not be necessary for typical use cases. For example, `tags` can be provided here to adjust which rules should trigger based on their tags, and `lw.rules.debug` can be specified with a value of “1” to include rule triggering debug info in the response. |
| `params` | Allows arbitrary query parameters to be added to the underlying Solr query. Should not be necessary for most use cases - the other DSL fields should be used when possible, but this field can be used when those fields do not suffice. If the Security Trimming Stage is in use, this field can be used to supply the various parameters for that stage (username, user identity key, collection, shards, etc…). |
The following sections provide an example of each field in use.
More detailed information on each field can be found in the [Fusion API Javadocs](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/overview-summary).
### queryDefinition
```json wrap theme={"dark"}
{
"queryDefinition": {
"userQuery": "cyberpunk novels",
"main": {
"type": "terms",
"field": "body_t",
"values": ["cyberpunk", "novels"],
"method": "booleanQuery"
},
"filters": [
{
"type": "singleTerm",
"field": "category_t",
"value": ["books"]
}
]
}
}
```
The `queryDefinition` defines the logic of what to query for. The above example queries for “cyberpunk novels” against the `body_t` field, additionally filtering to only show matches that have “books” in the `category_t` field.
### results
```json wrap theme={"dark"}
{
"results": {
"start": 20,
"size": 20,
"group": {
"field": "author",
"size": 5
}
}
}
```
The `results` defines how the results should be displayed and organized. The above example is set to show the next 20 results starting from the 21st matched document (as with Solr, `start` is 0-based). Additionally, results will be grouped by author, showing 5 results per group. In the DSL response, the head document of each group will show up in the results list with the other documents in the group in the head doc’s `groupedDocs` field.
### facets
```json wrap theme={"dark"}
{
"facets": {
"fields": [
{
"field": "category",
"limit": 5
},
{
"field": "brand",
"offset": "10"
}
],
"ranges": [
{
"field": "published_dt",
"gap": "+1YEAR",
"start": "2006-01-01T00:00:00Z",
"end": "2020-01-01T00:00:00Z"
}
]
}
}
```
The `facets` field controls faceting for the query. In the above example, faceting is enabled on the `category` field (limiting to show only 5 values), brand field (showing all values starting from the 11th value), and `published_dt` field (showing 1 year increments between 2006 and 2020).
### context
```json wrap theme={"dark"}
{
"context": {
"tags": "sometag",
"lw.rules.debug": "1"
}
}
```
The `context` field accepts parameters used by various query stages as well as DSL hints. In the above example, the tags parameter has been specified for the Apply Rules stage which will cause only rules with the “sometag” tag to be triggered. The `lw.rules.debug` parameter has also been specified to return extra rule triggering information in the response.
### params
```json wrap theme={"dark"}
{
"params": {
"uid": "je985"
}
}
```
The `params` field allows arbitrary query parameters to be added to the underlying Solr query and is also used to supply Security Trimming Stage parameters. In the above query, the user’s id has been supplied for the Security Trimming Stage.
## Response Format
This section describes each top-level field of a Search DSL response.
### Overview
The below table briefly summarizes the function of each of the top-level fields. The following sections provide an example of each field in a real response (some parts omitted for brevity).
| Field | Description |
| - | - |
| `results` | Holds the results list with pagination info, scoring, and hit count. When grouping is performed via the `results.group` setting in the DSL request, the groups show up here with the head doc at the top level and the other docs in the group in the `groupedDocs` field. |
| `facets` | Holds the returned facets and facet values. Each facet has an associated `label` which can serve as the user-friendly display name and can be defined using the Set Facets rule. As this is a map, the order of facets should not be relied upon. Facet ordering can be determined instead using the Set Facets rule and the `responseValues.facet_labels` field. The `groupFacets` flag controls the level on which facets are calculated: before grouping (`groupFacets=false`) or after grouping (`groupFacets=true`). Default value is `true`. |
| `rules` | Holds data returned by any rules that triggered. There are array fields for data from particular rule types: `redirects` for Redirect, `banners` for Banner, and `jsonBlobs` for JSON Blob (organized into separate arrays based on blob type). The `responseValues` field holds values set by the Response Value rules or any rules that provide a response values setting, as well as the `facet_labels` from the Set Facets rule. When multiple rules of a given kind trigger, the one with the higher precedence appears first in the corresponding list (breaking ties on creation date, newer rule comes first). |
| `spellcheck` | Holds all the spelling suggestions. Suggestions are provided for any misspelled words via the `wordSuggestions` field, and for the entire query via the `querySuggestions` field. |
| `meta` | Assorted metadata about the query, such as timing and debug information. The `debug.solrParams` field provides a list of the query parameters that were sent in the underlying Solr query. The `timing.mainQuery` provides the QTime of the underlying Solr query, while the `timing.total` field measures the entire DSL query (both in milliseconds). The timings of individual pipeline stages are broken down in the `timing.pipeline` field. |
### results
```json wrap theme={"dark"}
{
"results": {
"list": {
"hits": 1024,
"maxScore": 1.0,
"pagination": {
"start": 10
},
"docs": [
{
"collection": "books",
"type": "generic",
"id": "281",
"score": 1.0,
"fields": {
"author": "Iain Banks"
"title": "Player of Games"
},
"groupedDocs": {
"hits": 228,
"docs": [
{
"collection": "books",
"type": "generic",
"id": "283",
"score": 1.0,
"fields": {
"author": "Iain Banks"
"title": "Consider Phlebas"
}
}
]
}
},
...additional docs...
]
}
}
}
```
The `results` field holds the results list with pagination info, scoring, and hit count. In the above, grouping has been performed on the `author` field.
### facets
```json wrap theme={"dark"}
{
"facets": {
"field": {
"category": {
"label": "Category",
"counts": [
{
"name": "Sci-Fi",
"count": 7
},
{
"name": "Fantasy",
"count": 3
}
...additional values...
]
},
... additional field facets...
}
},
"range": {
"published_dt": {
"label": "Publication Date",
"gap": "+1YEAR",
"start": "2006-01-01T00:00:00Z",
"end": "2020-01-01T00:00:00Z",
"counts": [
...range values...
]
}
}
}
```
The `facets` field holds the returned facets and facet values. In the above, we see facet results for the `category` field and a range facet on the `published_dt` field, each of which has a corresponding `label` configured via a Set Facets rule for a user-friendly display name for the facet.
### rules
```json wrap theme={"dark"}
{
"rules": {
"responseValues": {
"facet_labels": [
"category:Category,published_dt:Publication Date"
]
},
"jsonBlobs": {
"default": [
{
"someField": "someValue"
},
{
"someField": "someValue2"
}
]
}
}
}
```
The `rules` field holds data returned by any rules that triggered. In the above, 3 rules have triggered. A Set Facets rule created a facet\_labels entry defining the desired ordering and labels for the facets (the labels are also present in the `facets` section of the response). Two JSON Blob rules of type `default` have triggered, the first blob in the list having a higher precedence value than the second (and thus appearing first).
### spellcheck
```json wrap theme={"dark"}
{
"spellcheck" : {
"correctlySpelled" : false,
"wordSuggestions" : {
"whiskeys" : {
"startOffset" : 6,
"endOffset" : 14,
"origFreq" : 0,
"wordFreqList" : [ {
"word" : "whiskey",
"freq" : 4764
}, {
"word" : "whiskies",
"freq" : 15963
}, {
"word" : "whisky",
"freq" : 2206
} ]
},
"scoch" : {
"startOffset" : 0,
"endOffset" : 5,
"origFreq" : 0,
"wordFreqList" : [ {
"word" : "scotch",
"freq" : 1556
}, {
"word" : "scott",
"freq" : 3340
}, {
"word" : "stock",
"freq" : 78
}, {
"word" : "shock",
"freq" : 9
} ]
}
},
"querySuggestions" : [ {
"query" : "scotch whiskey",
"hits" : 6320,
"corrections" : {
"whiskeys" : "whiskey",
"scoch" : "scotch"
}
} ]
}
}
```
The `spellcheck` field holds all the spelling suggestions. Above, the query has misspelled “scotch whiskey” as “scoch whiskeys” and a number of suggestions are provided for each word as well as a collation with the fully corrected query.
### meta
```json wrap theme={"dark"}
{
"meta": {
"timing": {
"total": 105,
"mainQuery": 14,
"pipeline": [
... pipeline stage timing info..
]
},
"debug": {
"solrParams": {
"fl": "*,score",
"facet": "true"
...additional solr params...
}
}
}
}
```
The `meta` field holds assorted metadata about the query, such as timing and debug information. Above, in the `timing.mainQuery` field we see that the underlying Solr query took 14 ms (taken from Solr’s QTime) while the `total` field shows that the entire DSL query took 105 ms. The timings of individual pipeline stages are broken down in the `pipeline` field. The `debug` field here shows the various parameters that were sent in the underlying Solr query.
# Faceting
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-basics/faceting
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-basics/faceting
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-basics/faceting
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/175
Faceting is the name given to a set of computed counts over a search result returned together with the documents which match the search query.
Facets are most often used to create additional navigational controls on the search results page or panel. Users can expand and restrict their search criteria in a natural way, without having to construct complicated queries.
For example, popular e-commerce facets include product category, price range, availability, and user ratings.
Fusion leverages [Solr’s Faceting](https://cwiki.apache.org/confluence/display/solr/Faceting) search components.
## Field faceting
In Solr the most straightforward kind of faceting is field faceting, in which Solr’s FacetComponent computes the top values for a field and returns the list of those values along with a count of the subset of documents in the search results which match that term. Field faceting works best over fields which contain a single label or set of labels from a finite, controlled lexicon such as product category. Fusion’s [Facet Query Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/facet-query-stage) can be used to configure field faceting as part of the search query pipeline.
## Range faceting
Range facets are used for fields which contain date or number values. Values can be grouped into ranges by specifying additional query parameters.
To configure range faceting, use the [Additional Query Parameters Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/set-query-params-stage) to specify [Solr range faceting parameters](https://cwiki.apache.org/confluence/display/solr/Faceting#Faceting-RangeFaceting).
The quick learning for **Configuring Range Facets** focuses on how to group values into ranges for faceting.
## JSON faceting
In eCommerce, one product often contains multiple SKUs. In this case, it’s ideal to group search results on the product ID, such that only one result for all SKUs of a given product will be returned. This is achieved by using the `unique` parameter in [JSON facets](https://solr.apache.org/guide/solr/latest/query-guide/json-facet-api.html).
You can use JSON facets by creating Query Params in a **Create a Query Profile** or with the [Additional Query Parameters Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/set-query-params-stage) in a Query Pipeline.
Configure facets with a JSON script in your query pipeline.
Facets that come from Rules must be converted in code to JSON facets. they cannot be passed through as name/value params.
You can use the [Additional Query Parameters stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/set-query-params-stage) when a JSON facet is hardcoded in the pipeline as a static `json.facet` string.
1. In the Fusion workspace, navigate to **Querying** > **Query Profiles**.
2. Click **New**. The **Add Query Profile** panel appears:
3. Enter a search mode to use with this profile.
* `dsl` ([Domain Specific Language](/docs/5/fusion/getting-data-out/query-basics/domain-specific-language)) uses expressive search queries and responses via a structured, modern JSON format.
* `legacy` primarily uses Solr parameters. See the [Solr Query Language cheat sheet](/docs/5/fusion/getting-data-out/query-basics/query-language-cheat-sheet).
* `all` uses both DSL and legacy search modes. This is the default value.
The default value of `all` works well for most use cases.
The **Search Mode** field is available for Fusion 5.4.x and later.
4. Enter a **Query Profile ID**.
5. Select a query pipeline to associate with this profile.
6. Enter a [Solr search handler](https://wiki.apache.org/solr/SearchHandler) to use with this profile.
The default of `select` works well for most use cases.
7. Select a collection to associate with this profile.
8. Optionally, click **New params...** to enter Solr request parameters to add to the request URL when submitting queries using this profile.
9. Optionally, select **Enable experimentation** to configure this profile for use with [experiments](/docs/4/fusion-ai/concepts/experiments/overview).
1. Select the percentage of experiment traffic to direct to this profile.
2. Select the experiment to associate with this profile, or select **Add Experiment** to configure a new one.
10. Click **Save**.
The query profile window displays the request URL for sending queries to this profile.
The course for **Index and Query Profiles** focuses on how index and query profiles operate, and how to configure and manage them.
### JSON faceting example
For example, if a product is available in SKUs of multiple colors, the following should apply:
1. All available colors (in any SKU in the result set) are represented in the facets
2. The `count` for those facets reflects matching product counts, not SKU counts.
In this example, eight SKU records are used across two products.
| SKU | Product | Color |
| - | - | - |
| Sku11 | Prod1 | Red |
| Sku12 | Prod1 | Blue |
| Sku13 | Prod1 | Blue |
| Sku14 | Prod1 | Blue |
| Sku21 | Prod2 | Red |
| Sku22 | Prod2 | Blue |
| Sku23 | Prod2 | Green |
| Sku24 | Prod2 | Green |
With the preceding data, the color facet should show:
```bash theme={"dark"}
Color:
Red (2)
Blue (2)
Green (1)
```
## Facet stats
The [Stats component](https://solr.apache.org/guide/solr/latest/query-guide/stats-component.html) is useful for computing statistics against fields within your document set. For example, when performing a query, you can invoke the Stats component to return information about the `mean` price of a product or determine how many documents are missing a particular field.
### Facet stats example
```sh wrap theme={"dark"}
https://FUSION_HOST:FUSION_PORT/api/apps/exampleapp/query/examplepipeline/select?q=Cooking&stats=true&stats.field=price
```
```json wrap theme={"dark"}
...
"stats":{
"stats_fields":{
"price":{
"min":12.34,
"max":57.65,
"mean":34.56,
...
```
## Faceting concepts
Key Facet Concepts:
* **Term**: A specific value from a field.
* **Limit**: The maximum number of terms to be returned.
* **Offset**: The number of top facet values to skip in the response (just like paging through search results and choosing an offset of 51 to start on page 2 when showing 50 results per page).
* **Sort**: The order in which to list facet values: `count` ordering is by documents per term, descending, and `index` ordering is sorted on term values themselves.
* **Missing**: The number of documents in the results set which have no value for the facet field.
* **Choice of facet method (advanced)**: Specify Solr algorithm used to calculate facet counts. (See [Facet Method Configuration](https://cwiki.apache.org/confluence/display/solr/Faceting#Faceting-Thefacet.methodParameter) for details).\
One of:
* `enum`. Small number of distinct categories.
* `fc` ("field cache"). Many different values in the field, each document has low number of values, multi-valued field.
* `fcs` ("single value string fields"). Good for rapidly changing indexes.
## Learn more
The course for **Faceting** focuses on how to use facets to ensure faster, smarter search results.
## See also
* [https://lucidworks.com/blog/pivot-facets-inside-and-out/](https://lucidworks.com/blog/pivot-facets-inside-and-out/)
* [https://lucidworks.com/blog/apache-solr-statistics/](https://lucidworks.com/blog/apache-solr-statistics/)
* [https://lucidworks.com/blog/pivoting-to-the-query-using-pivot-facets-to-build-a-multi-field-suggester/](https://lucidworks.com/blog/pivoting-to-the-query-using-pivot-facets-to-build-a-multi-field-suggester/)
# Query Basics
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-basics/overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-basics/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-basics/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/168
## Query Pipelines
Query pipelines process search requests before sending them to Fusion’s Solr core. A query pipeline can perform transformations on a search request in order to customize the search results that Solr returns. See [Query Pipeline Configuration](/docs/5/fusion/getting-data-out/query-basics/query-pipelines/overview).
## Search Applications
Search applications are the front-end interfaces that you build on top of Fusion. Your application makes calls to Fusion’s REST API in order to retrieve search results or perform other actions. Certain features, like autocomplete and synonyms, require some configuration on the Fusion back end. See [Application Development](/docs/5/fusion/dev-portal/front-end-dev/overview).
## Query Language
Whenever you are getting data out of Fusion, you may find it handy to consult the [Query Language Cheat Sheet](/docs/5/fusion/getting-data-out/query-basics/query-language-cheat-sheet).
If you are using Fusion, you can generate and retrieve additional data for analysis or to enhance the end-user experience. For example, Fusion can produce sophisticated recommendations to guide end users to the best available results, including results that do not exactly match the original user-submitted query.
It can also perform machine learning functions that automatically improve search results based on the past activities of users. For more information, see:
* [Natural Language Processing](/docs/4/fusion-ai/concepts/nlp)
* [Machine Learning](/docs/5/fusion/intro/machine-learning/overview)
* [Machine Learning Models in Fusion](/docs/5/fusion/intro/machine-learning/machine-learning-models)
* [Machine Learning Jobs](/docs/5/fusion/intro/machine-learning/ml-jobs)
## Learn more
The quick learning for **Anatomy of Querying** focuses on the basics of query related objects and workflows in Fusion.
The course for **Queries** focuses on basic query language syntax and the concepts of query pipelines and query profiles.
# Query Data Flow
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-basics/query-dataflow
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-basics/query-dataflow
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-basics/query-dataflow
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/3234
[Indexing Data Flow](/docs/5/fusion/intro/indexing-dataflow) shows you how your content, signals, and recommendations are indexed in Fusion. This topic explains how to query your indexed data.
## 1. Query your content
When you query your content using the default query pipeline *and* you have [signals enabled](/docs/5/fusion/intro/indexing-dataflow), you also get [query rewrites](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) and [automatic boosting](/docs/5/fusion/getting-data-out/query-enhancement/boosting/boost-with-signals) to enhance the relevancy of your search results. In addition to matches from your primary collection, you’ll get results produced by Fusion’s machine learning jobs, based on [signals](/docs/5/fusion/getting-data-out/query-enhancement/signals/overview) from your users.
[Query profiles](/docs/5/fusion/getting-data-out/query-basics/query-pipelines/query-profiles) give you a stable endpoint that can be associated with any [query pipeline](/docs/5/fusion/getting-data-out/query-basics/query-pipelines/overview). Just specify the query profile when sending queries to the [Query Profiles API](/api-reference/query-profiles-api/get-the-service-status) endpoint. You can always reconfigure the query profile to point to a different pipeline, or to an [experiment](/docs/5/fusion/getting-data-out/data-analytics/experiments/overview).
* Several stages perform [query rewriting](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) and [response rewriting](/docs/5/fusion/getting-data-out/query-enhancement/response-rewriting) by reading from the `_query_rewrite` collection:
* The [Text Tagger stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/text-tagger-query-stage) finds [misspellings](/docs/5/fusion/getting-data-out/query-enhancement/misspelling-detection), [phrases](/docs/5/fusion/getting-data-out/query-enhancement/phrase-detection), [underperforming query strings](/docs/5/fusion/getting-data-out/query-enhancement/underperforming-queries), and [synonyms](/docs/5/fusion/reference/config-ref/jobs/synonym-detection) in the incoming query.
* The [Apply Rules stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/query-rules-query-stage) applies your [business rules](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/overview) to the incoming query.
* The [Modify Response with Rules stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/rules-augment-response-query-stage)
* The [Boost with Signals stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommendation-boosting-query-stage) reads aggregated signals to boost search results that have been popular with users.
* The [Solr Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/solr-query-stage) fetches the final set of relevant search results from the primary collection.
## 2. Query for recommendations
To get even more relevant results based on AI-powered analysis of your signals data, your query pipeline can perform any number of secondary queries to the collections where the output from machine learning jobs is indexed. Your secondary queries can provide additional search results based on Fusion’s analysis of your signals.
* Use the [Recommend Items for Item stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommend-items-for-item-query-stage) to query for [collaborative items-for-item recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-item#collaborative-items-for-item-recommendations).
* The [Recommend Items for User stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommend-items-for-user-query-stage) queries for [items-for-user recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-user).
* The [Recommend Similar Queries stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommend-similar-queries-query-stage) provides [queries-for-query recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/queries-for-query).
# Solr Query Language Cheat Sheet
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-basics/query-language-cheat-sheet
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-basics/query-language-cheat-sheet
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-basics/query-language-cheat-sheet
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/199
This cheat sheet is a quick reference to the Solr query language. Use this syntax when querying Fusion via the [Query Profiles API](/api-reference/query-profiles-api/get-the-service-status).
Additional information about Solr query syntax can be found in the [Solr Query](https://solr.apache.org/guide/8_7/the-standard-query-parser.html#specifying-terms-for-the-standard-query-parser) documentation.
There are two ways to query a Fusion collection using the parameters below:
* Enter query parameters in the [Query Workbench](/docs/5/fusion/intro/ui-tour/query-workbench) or the Quickstart.
* Append query parameters to the [`/query/{id}` endpoint](/api-reference/query-profiles-api/get-the-service-status).
## Wildcards
Wildcards and regular expressions are supported.
* `?`: Single-character wildcard
* `*`: Multi-character wildcard
## Escape characters
Some characters in query fields must be escaped using a backslash `\`. Adding `\` before the following includes them in the query as literal values:
* `?`
* `*`
* `\`
* `/`
* `"`
Queries against facets are case sensitive.
## Common query parameters
| | |
| - | - |
| `q` | Defines a query using [standard query syntax](https://solr.apache.org/guide/8_8/the-standard-query-parser.html#standard-query-parser-parameters). This parameter is mandatory. |
| `fq` | [Filter query](https://solr.apache.org/guide/8_8/common-query-parameters.html#fq-filter-query-parameter). A query string that limits the query results without influencing their scores. |
| `sort` | [Sort field/direction](https://solr.apache.org/guide/8_8/common-query-parameters.html#sort-parameter). The field on which to sort, followed by a space and direction (`desc` or `asc`). You can specify multiple sort fields like this: `sort=title asc,year desc`. |
| `rows` | [Max results per page](https://solr.apache.org/guide/8_8/common-query-parameters.html#rows-parameter). This sets the "page size" for paginated search results. |
| `start` | [Pagination offset](https://solr.apache.org/guide/8_8/common-query-parameters.html#start-parameter). The number of results to skip, for pagination purposes. |
| `fl` | [Field List](https://solr.apache.org/guide/8_8/common-query-parameters.html#fl-field-list-parameter). The list of fields to return in the query results. |
| `df` | [Default field](https://solr.apache.org/guide/8_8/the-standard-query-parser.html#standard-query-parser-parameters). Used to configure the `q` and `fq` parameters. If not specified, the default field is `text`. |
| `wt` | [Response writer](https://solr.apache.org/guide/8_8/common-query-parameters.html#wt-parameter). Select the response format by specifying one of [Solr’s response writers](https://solr.apache.org/guide/8_8/response-writers.html). |
## Query examples
1. Search only the `title` field in the "docs" collection for the term "solr", and format the results as JSON:
```sh wrap theme={"dark"}
http://FUSION_HOST:FUSION_PORT/api/query-pipelines/default/collections/docs/select?q=solr&fl=title&wt=json
```
2. Use the following query with wildcards and character escaping in the Query Workbench to search for both `BLU RAY MOVIES` and `CAMERA` classes in the `VIDEO/CD` and `PHOTO/CAMERA` departments:
* `class_s:*RA* AND department_s:*O\/C*`
3. Use a movie collection to select the movie ID and rating where the rating is equal to 4.
```sh wrap theme={"dark"}
http://FUSION_HOST:FUSION_PORT/api/query-pipelines/default/collections/docs/select?q=*:*&fl=movie_id,rating&fq=rating:4
```
4. Search for movies with "black" in the title, starting at the 20th result for pagination, then return the results as XML:
```sh wrap theme={"dark"}
http://FUSION_HOST:FUSION_PORT/api/query-pipelines/default/collections/docs/select?q=black&fl=title&start=20&rows=20&wt=xml
```
## Learn more
The course for **Advanced Query Tools** focuses on advanced query language techniques, such as using query parameters, function queries, nested queries, and different query parsers.
## See also
* The [Apache Solr Query Guide](https://solr.apache.org/guide/solr/latest/query-guide/common-query-parameters.html) provides complete information about all aspects of Solr queries.
# Custom JavaScript Stages for Query Pipelines
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-basics/query-pipelines/custom-javascript-query-stages
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-basics/query-pipelines/custom-javascript-query-stages
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-basics/query-pipelines/custom-javascript-query-stages
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/170
The JavaScript Query stage allows you to write custom processing logic using JavaScript to manipulate search requests and responses.
The first time that the pipeline is run, Fusion compiles the JavaScript program into Java bytecode using the JDK’s JavaScript engine.
The JavaScript Query stage allows you to run JavaScript functions over search requests and responses by manipulating variables called "request" and "response" which are [Request objects](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/apollo/pipeline/query/Request.html) and [Response objects](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/apollo/pipeline/query/Response.html), respectively.
Users who can create or modify code obtain access to the broader Fusion environment. This access can be used to create intentional or unintentional damage to Fusion.
## The JavaScript Engine Used by Fusion
The default JavaScript engine used by Fusion is the Nashorn engine from Oracle.
See [The Nashorn Java API](https://docs.oracle.com/javase/8/docs/technotes/guides/scripting/nashorn/api.html) for details.
In Fusion 5.9.6 and up, you also have the option to select OpenJDK Nashorn.
While Nashorn is the default option, it is in the process of being deprecated and will eventually be removed, so it is recommended to use OpenJDK Nashorn when possible.
You can select the JavaScript engine in the pipeline views or in the workbenches.
Your JavaScript pipeline stages are interpreted by the selected engine.
## JavaScript Query Stage Global Variables
[JavaScript](http://en.wikipedia.org/wiki/JavaScript) is a lightweight scripting language.
In a JavaScript stage, Fusion uses the Nashorn engine, which implements [ECMAScript](http://en.wikipedia.org/wiki/ECMAScript) version 5.1. Although Nashorn does include some ECMAScript 6 (ES6) features such as `let`, `const`, or template strings, Fusion does not enable ES6 by default, so ES6 support is not guaranteed.
What a JavaScript program can do depends on the container in which it runs.
For a JavaScript Query stage, the container is a Fusion query pipeline.
[ Request for a Regular Query ](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/apollo/pipeline/query/Request.html) The `request` variable contains Solr query information and is referred to as a **regular** request. A regular query request does *not* include Search DSL (domain specific language) parameters.
[ DSL Request ](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/search/dsl/request/DslRequest.html) The `request` variable is also used when the query contains parameters for a Search DSL (domain specific language) request. See [Domain Specific Language](/docs/5/fusion/getting-data-out/query-basics/domain-specific-language) for more information.
[ Response for a Regular Query ](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/apollo/pipeline/query/Response.html) The `response` variable contains Solr response information, and is used to return information from a **regular** request. Because a regular request does *not* include Search DSL (domain specific language) parameters, the response will not return Search DSL results.
[ DSL Response ](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/search/dsl/response/DslResponse.html) The `response` variable is also used to return information from a Search DSL (domain specific language) request. See [Domain Specific Language](/docs/5/fusion/getting-data-out/query-basics/domain-specific-language) for more information.
[ Context ](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/apollo/pipeline/Context.html)
A map that stores miscellaneous data created by each stage of the pipeline.
**Important** Use the `ctx` variable instead of the deprecated `_context` global variable.
The `ctx` variable is used to:
* Pass data from one stage to another
* Store data that needs to be passed from one custom stage to a later custom stage
The data can differ between stages:
* If the previous stage changes the data
* Based on the configuration of each stage
If the data is modified in one stage, it may cause a later stage to function irregularly.
String
The name of the Fusion collection being indexed or queried.
[ BufferingSolrServer ](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/apollo/component/BufferingSolrServer.html) The Solr server instance that manages the pipeline’s default Fusion collection. All indexing and query requests are done by calls to methods on this object. See [solrClient](https://lucene.apache.org/solr/5_2_1/solr-solrj/org/apache/solr/client/solrj/SolrClient.html) for details.
[ SolrClusterComponent ](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/cloud/api/solr/SolrClusterComponent.html)\
The SolrCluster server used for lookups by collection name which returns a Solr server instance for that collection, e.g. `var productsSolr = solrServerFactory.getSolrServer("products");`.
[ QueryRequestAndResponse ](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/apollo/pipeline/query/QueryRequestAndResponse.html) Used to create query pipeline requests and responses.
[ JSONResponse ](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/apollo/solr/response/JSONResponse.html) Returns list of info as a string.
See [Custom JavaScript Query Stage Examples](/docs/5/fusion/reference/example-custom-javascript-stages) for more information.
### Syntax Variants
JavaScript stages can be written using ***function syntax***. With function syntax, global variables are passed as function parameters.
Support for legacy syntax was removed in Fusion 5.8.
#### Function Syntax
```js wrap theme={"dark"}
function(request,response) {
request.addParam("foo", "bar");
}
```
**Important**
*Function syntax* is used for the examples in this document.
### Global variable `logger`
The logs are output to the query service logs for custom query stages. Access the Log Viewer and filter on this service to view the information.
## See also
* [Fusion Pipeline Javadocs](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/)
* [JavaScript Query Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/javascript-query-stage)
* [Custom JavaScript Query Stage Examples](/docs/5/fusion/reference/example-custom-javascript-stages)
* [Custom JavaScript Stages Global Variables](/docs/5/fusion/reference/config-ref/pipeline-stages/custom-javascript-stages-global-variables)
* [Managed JavaScript Query Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/managed-javascript-query-stage)
The course for **JavaScript in Fusion** focuses on how to leverage JavaScript in Fusion to build powerful and responsive scripts at index and query time.
# Query Pipelines
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-basics/query-pipelines/overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-basics/query-pipelines/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-basics/query-pipelines/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/173
A Query Pipeline transforms a set of inputs into a Solr query request and it can execute requests and manipulate the Solr response as well, via a set of modularized operations called [Query Stages](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/overview). The objects sent from stage to stage are Request objects and Response objects.
Fusion stores pipeline names and definitions, allowing a pipeline to be reused across applications. Pipeline definitions can be modified, so that as an application evolves, the pipelines used by that application can evolve accordingly. During application development, the Fusion UI can be used to develop and debug a Query Pipeline.
The available stage types allow setting specific parameters for the query, such as the number of results to return or the query parser to use. You can also define facets and recommendations to be returned with the results. If Access Control Lists (ACLs) are in use, you can apply a security-trimming stage to apply user access restrictions to the results.
For details about the available REST APIs, see [Query Pipelines API](/api-reference/query-pipelines-api/get-all-query-pipelines) and [Query Stages API](/api-reference/query-stage-api/get-the-service-status).
## Search mode
When creating a query pipeline, you can select a search mode for the pipeline to use.
* `dsl` ([Domain Specific Language](/docs/5/fusion/getting-data-out/query-basics/domain-specific-language)) uses expressive search queries and responses via a structured, modern JSON format.
* `legacy` primarily uses Solr parameters. See the [Solr Query Language cheat sheet](/docs/5/fusion/getting-data-out/query-basics/query-language-cheat-sheet).
* `all` uses both DSL and legacy search modes. This is the default value.
The default value of `all` works well for most use cases.
## Default query pipelines
Fusion creates a default query pipeline when you create an app. The query pipeline has the same name as the app.
The default query pipeline has the following pre-configured stages:
* [Text Tagger](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/text-tagger-query-stage)
This stage uses the [SolrTextTagger](https://solr.apache.org/guide/solr/latest/query-guide/tagger-handler.html) handler to identify known entities in the query by searching the [`COLLECTION_NAME_query_rewrite` collection](/docs/5/fusion/getting-data-in/indexing/collections/overview). See **Manage Collections in the Fusion UI** for more information.
Collections can be created or removed using the Fusion UI or the REST API.
For information about using the REST API to manage collections, see [Collections API](/api-reference/collections/get-collections-service-status) in the REST API Reference:
## Creating a Collection
When you create an app, by default Fusion Server creates a collection and associated objects.
To create a new collection in the Fusion UI:
1. From within an app, click **Collections > Collections Manager**.
2. At the upper right of the panel, click **New**.
3. Enter a **Collection name**. This name cannot be changed later.
4. To create the collection in the default Solr cluster and with other default settings, click **Save Collection**.
## Creating a Collection with Advanced Options
To access advanced options for creating a collection in the Fusion UI:
1. From within an app, click **Collections > Collections Manager**.
2. At the upper right of the panel, click **New**.
3. Enter a **Collection name**. This name cannot be changed later.
4. Click **Advanced**.
5. Configure advanced options. The options are described below.
6. Click **Save Collection**.
### Solr Cluster
By default, a new collection is associated with the Solr instance that is associated with the `default` Solr cluster.
If Fusion has multiple Solr clusters, choose from the list which cluster you want to associate your collection with.
The cluster must exist first.
### Solr Cluster Layout
The next section lets you define a **Replication Factor** and **Number of Shards**.
Define these options only if you are creating a new collection in the Solr cluster.
If you are linking Fusion to an existing Solr collection, you can skip these settings.
### Solr Collection Import
Import a Solr collection to associate the new Fusion collection with an existing Solr collection.
Enter a **Solr Collection Name** to associate the collection with an existing Solr collection.
Then, enter a **Solr Config Set** to tell ZooKeeper to use the configurations from an existing collection in Solr when creating this collection.
### Time Series Partitioning
Available in 4.x only.
You can map a Fusion collection to multiple Solr collections, known here as partitions, where each partition contains data from a specific time range.
To configure time-based partitioning, under **Time Series Partitioning** click **Enable**.
See [Time-Based Partitioning](/docs/4/fusion-server/concepts/indexing/collections/time-based-partitioning) for more information.
## Configuring Collections
The Collections menu lets you configure your existing collection, including datasources, fields, jobs, stopwords, and synonyms.
In the Fusion UI, from any app, the Collections icon displays on the left side of the screen.
Some tasks related to managing a collection are available in other menus:
* Configure a profile in **Indexing > Indexing Profiles** or **Querying > Query Profiles**.
* View reports about your collection’s activity in **Analytics > Dashboards**.
### Collections Manager
The Collections Manager page displays details about the collection, such as how many datasources are configured, how many documents are in the index, and how much disk space the index consumes.
This page also lets you [create a new collection](#creating-a-collection), disable search logs or signals, enable recommendations, issue a commit command to Solr, or clear a collection.
#### Disable search logs
When you first create a collection, the search logs are created by default. The search logs populate the panels in **Analytics > Dashboards**.
1. Hover over your collection name until the gear icon appears at the end of the line.
2. Click the gear icon.
3. Click **Disable Search Logs**.
4. On the confirmation screen, click **Disable Search Logs**.
Note that if you disable search logs, you cannot see any data for this collection in **Analytics > Dashboards**.
See Dashboards:
* **Fusion 5.x.** [Dashboards](/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/overview)
* **Fusion 4.x.** [Dashboards](/docs/4/fusion-server/concepts/dashboards/overview)
#### Disable signals
When you first create a collection, the signals and aggregated signals collections are created by default.
1. Hover over your collection name until the gear icon appears at the end of the line.
2. Click the gear icon.
3. Click **Disable Signals**.
4. On the confirmation screen, click **Disable Signals**.
#### Hard commit a collection
1. Hover over your collection name until the gear icon appears at the end of the line.
2. Click the gear icon.
3. Click **Hard Commit Collection**.
4. On the confirmation screen, click **Hard Commit Collection**.
Read internal details about how Solr processes commits on [our blog](https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/).
### Datasources
To access the Datasources page, click **Indexing > Datasources**. By default, there are no datasources configured right after installation.
To add a new datasource, click **New** at the upper right of the panel.
See the [Connectors and Datasources Reference](/docs/fusion-connectors/connectors/overview) for details on how to configure a datasource. Options vary depending on the repository you would like to index.
After you configure a datasource, it appears in a list on this screen. Click the name of a datasource to edit its properties. Click **Start** to start the datasource. Click **Stop** to stop the datasource before it completes.
To the right, view information on the last completed job, including the date and time started and stopped, and the number of documents found as new, skipped, or failed.
When you stop a datasource, Fusion attempts to safely close connector threads, finishing processing documents through the pipeline and indexing documents to Solr. Some connectors take longer to complete these processes than others, so might stay in a "stopping" state for several minutes.
To stop a datasource immediately, choose **Abort** instead of **Stop**.
There is also a [REST API for datasources](/api-reference/datasource-configuration-v2-api/list-all-v2-datasources).
### Stopwords
The Stopwords page lets you edit a stopwords list for your collection.
To add or delete stop words:
1. Click the name of the text file you wish to edit.
2. Add a new word on a new line.
3. When you are done with your changes, click **Save**.
To import a stop words list:
1. Click **System > Import Fusion Objects**.
2. Choose the file to upload.
3. Click **Import >>**.
Read more about stopwords:
* **Fusion 5.x.** [stopwords](/docs/5/fusion/getting-data-out/query-enhancement/stopwords-files)
* **Fusion 4.x.** [stopwords](/docs/4/fusion-server/concepts/querying/apps/stopwords-files)
### Synonyms
Fusion has the same synonym functionality that Solr supports. This includes a list of words that are synonyms (where the synonym list expands on the terms entered by the user), as well as a full mapping of words, where a word is substituted for what the user has entered (that is, the term the user has entered is replaced by a term in the synonym list).
See more about synonyms:
* **Fusion 5.x.** [synonyms](/docs/5/fusion/getting-data-out/query-enhancement/synonyms/overview)
* **Fusion 4.x.** [synonyms](/docs/4/fusion-server/concepts/querying/apps/synonyms-files)
You can edit the synonyms list for your collection.
To access the Synonyms page in the Fusion UI, in any app, click **Collections > Synonyms**.
Filter the list of synonym definitions by typing in the **Filter...** box.
To import a synonyms list:
1. From the Synonyms page, click **Import and Save**. A dialog box opens.
2. Choose the file to import.
To edit a synonyms list:
* Enter new synonym definitions one per line.
* To enter a string of terms that expand on the terms the user entered, enter the terms separated by commas, like `Television, TV`.
* To enter a term that should be mapped to another term, enter the terms separated by an equal sign then a right angle bracket, `=>`, like `i-pod=>ipod`.
* Remove a line by clicking the **x** at the end of the line.
* Once you are finished with edits, click **Save**.
To export the synonyms list, click **Export**. This downloads the list to your computer using your browser download capability.
### Profiles
Profiles allow you to create an alias for an index or query pipeline.
This allows you to send documents or queries to a consistent endpoint and change the underlying pipeline or collection as needed.
Read about profiles in Index Profiles and Query Profiles:
* **Fusion 5.x.**
* [Index Profiles](/docs/4/fusion-server/concepts/indexing/datasources/index-profiles)
* [Query Profiles](/docs/4/fusion-server/concepts/querying/pipelines/query-profiles)
* **Fusion 4.x.**
* [Index Profiles](/docs/4/fusion-server/concepts/indexing/datasources/index-profiles)
* [Query Profiles](/docs/4/fusion-server/concepts/querying/pipelines/query-profiles)
To access the Solr Config page, from any app, click **System > Solr Config**.
## Learn more
The quick learning for **Collections Menu Tour** focuses on the Collections Menu features and functionality along with a brief description of each screen available in the menu.
For Fusion 5.x.x organizations that do *not* have a Predictive Merchandiser license, the Solr Text Tagger handler also searches the `COLLECTION_NAME_query_rewrite_staging` collection in the case of the Fusion query rewriting [Simulator](/docs/4/fusion-ai/concepts/query-rewriting/simulator).
* [Boost with Signals](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommendation-boosting-query-stage)\
The Boost with Signals query pipeline stage uses [aggregated signals](/docs/5/fusion/getting-data-out/query-enhancement/signals/aggregations) to selectively boost items in the set of search results.
* [Query Fields](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/search-fields-query-stage)\
The Query Fields query pipeline stage defines common Solr query parameters for the [edismax query parser](https://cwiki.apache.org/confluence/display/solr/The+Extended+DisMax+Query+Parser). If using a less-than sign (\<) with DisMax, it must be escaped using a backslash.\
An alternative to this stage is the
[Additional Query Parameters stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/set-query-params-stage).
* [Field Facet](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/facet-query-stage)\
The Field Facet query pipeline stage is used to add a
[Solr Field Facet query](https://cwiki.apache.org/confluence/display/solr/Faceting#Faceting-Field-ValueFacetingParameters)
to the search query pipeline.
* [Apply Rules](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/query-rules-query-stage)\
This stage looks up rules that have been deployed to the [`COLLECTION_NAME_query_rewrite` collection](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) and matches them against the query. Matching rules that perform [query rewriting](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) are applied at this stage, while matching rules that perform [response rewriting](/docs/5/fusion/getting-data-out/query-enhancement/response-rewriting) are applied by the [Modify Response with Rules](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/rules-augment-response-query-stage) stage later in the pipeline.
To trigger a rule that contains a tag, specify the tagname in the request URL of the user search app. See [Easily define triggers in tags](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/general-config) for more information.
* A [Solr Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/solr-query-stage)\
The Solr Query stage transforms the Fusion query pipeline
[Request](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/apollo/pipeline/query/Request.html)
object into a Solr query and sends it to Solr.
* [Modify Response with Rules](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/rules-augment-response-query-stage)\
Most rules operate on the request, but some [rule types](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting), such as banner rules or redirect rules, do their work when the response comes back. The Modify Response with Rules stage applies those rules to the response. For example, a banner rule can add a banner URL to the response before returning it to the client.
## Custom query pipelines
Using the [Query Workbench](/docs/5/fusion/intro/ui-tour/query-workbench) or the [REST API](/api-reference/query-pipelines-api/get-all-query-pipelines), you can develop custom pipelines to suit any search application. Start with any of Fusion’s built-in query pipelines, then add, remove, and re-order the [pipeline stages](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/overview) as needed to produce the appropriate query results.
## Asynchronous query pipeline processing
Query pipeline processing performance can be improved by enabling asynchronous processing for certain stages that make requests to secondary collections, external databases, and so on. The following stages support asynchronous processing:
* [Active Directory Security Trimming](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/active-directory-security-trimming-query-stage)
* [Apply Rules](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/query-rules-query-stage)
* [Boost with Signals](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommendation-boosting-query-stage)
* [JDBC Lookup](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/jdbc-query-stage)
* [LWAI Prediction](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/lucidworks-ai-prediction-query-stage)
* [LWAI Vectorize Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/lwai-vectorize-query)
* [Security Trimming](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/security-trimming-query-stage)
* [Solr Subquery](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/subquery-query-stage)
This feature uses the fork-and-join model, where any of the stages above can create a fork on the pipeline. The parallel processes are joined again using the [Merge Async Results stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/merge-async-results-query-stage) at a later point in the pipeline.
### Monitoring for asynchronous query pipeline stages
The monitoring feature provides a framework that integrates with tools such as Grafana for seamless observability, and supports complex AI-driven search and generative AI (Gen-AI) workloads.
Monitoring provides critical, real-time visibility into the performance and reliability of asynchronous query pipeline stages. The information includes tracking execution times, failures, and performance bottlenecks. In addition, monitoring information enables system optimization and the ability to more quickly and easily troubleshoot issues.
Use these execution metrics and failure analysis to reduce downtime and accelerate issue resolution, as well as optimize search and AI-driven applications for efficiency and scalability. This ensures better search relevance, improved operational stability, and readiness for next-generation AI search experiences.
## Debugging
If your pipeline is not producing the desired result, these debugging tips can help you identify the source of the issue and resolve it.
### View parameters
When debugging a pipeline, it helps to see the parameters that are being passed to or from each stage.
There are several ways to exposed those parameters:
Follow these steps to add a Logging stage to your pipeline:
1. In Fusion UI, navigate to **Indexing > Index Pipelines** (for index pipelines) or **Querying > Query Pipelines** (for query pipelines).
2. Click **Add a new pipeline stage**.
3. Select **Logging** from the Troubleshooting section.
4. In the **Label** field, enter a descriptive name (for example, "Debug After Vectorize").
5. Set the **detailed** property to `true` to print the full Request or PipelineDocument object.
6. Place the Logging stage after the stage you want to debug.
7. Click **Save**.
8. Run your pipeline and retrieve logs for your pipeline type:
* Query pipelines: `kubectl logs -f -l app=query-pipeline -n NAMESPACE`
* Index pipelines: `kubectl logs -f -l app=fusion-indexing -n NAMESPACE`
If Grafana with Loki is installed, see [Grafana and Loki](/docs/5/fusion/operations/monitoring-and-reporting/grafana-loki) for query guidance.
Follow these steps to inspect context variables:
1. Add a **JavaScript** stage to your pipeline.
2. Use the `ctx` variable to inspect context data:
```javascript theme={"dark"}
function (request, response, ctx) {
// View all context variables
logger.info("Context: " + JSON.stringify(ctx));
// Check specific variable (for example, vector from LWAI stage)
if(ctx.get('vector') !== null) {
logger.info("Vector: " + ctx.get('vector'));
}
}
```
3. Retrieve logs from the `query-pipeline` pod: `kubectl logs -f -l app=query-pipeline -n NAMESPACE`
Follow these steps to debug using Query Workbench:
1. Navigate to **Querying > Query Workbench**.
2. Enter a test query and click **Search**.
3. Click the **Debug** tab. The debug view displays the following information:
* Request parameters
* Pipeline stage execution details
* Response data including `responseHeader` and `debug.explain`
4. Switch to **View As: JSON** to see the full response structure.
### Enable **Fail on Error**
The **Fail on Error** setting determines whether silent failures can occur in your pipeline.
By enabling **Fail on Error** during development, testing, and troubleshooting, you ensure that configuration issues, authentication problems, or model errors are immediately visible rather than producing incomplete or incorrect data that can be difficult to troubleshoot later.
The **Fail on Error** setting controls whether pipeline execution stops when an LWAI stage encounters an error. Follow these steps to configure this setting:
1. In Fusion UI, navigate to your pipeline (Index or Query).
2. Click the LWAI stage you want to configure (for example, **LWAI Vectorize Field** or **LWAI Vectorize Query**).
3. Locate the **Fail on Error** checkbox at the bottom of the stage configuration.
4. Select the checkbox to enable any of the following behaviors:
* Stop pipeline processing and throw an exception on errors
* Get immediate feedback when LWAI models fail or are misconfigured
* Guarantee data quality by preventing indexing of documents without vectors
In production environments, keep this feature disabled to ensure that service is not interrupted when errors occur.
6. Click **Save**.
During development and testing, enable **Fail on Error** to identify configuration issues early. In production, consider disabling **Fail on Error** if you want resilient pipelines that can handle occasional model failures. For critical vector fields, enable **Fail on Error** to ensure that all documents have required embeddings.
Follow these steps to verify your Fail on Error configuration:
1. Trigger an intentional error (for example, use an invalid model name or account).
2. Verify that the pipeline fails (when Fail on Error is enabled) or continues (when Fail on Error is disabled).
3. Retrieve logs from the `query-pipeline` pod: `kubectl logs -f -l app=query-pipeline -n NAMESPACE`
# Query Pipeline Stages
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-basics/query-pipelines/query-pipeline-stages
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-basics/query-pipelines/query-pipeline-stages
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-basics/query-pipelines/query-pipeline-stages
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/169
A query pipeline is made up of a series of query stages that process incoming search queries.
A pipeline stage definition associates a unique ID with a set of properties.
These definitions are registered with the Fusion API service and stored in ZooKeeper for re-use across pipelines and search applications.
Fusion includes a number of specialized query stages as well as a JavaScript stage that allows advanced processing via a JavaScript program.
The [Query Pipeline Stages Reference](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/overview) section contains articles that detail the configuration for each of the Fusion query pipeline stages. The configuration information includes field names and values, as well as examples and other key reference information.
## Configuring query pipeline stages
* In the Fusion UI, the [Query Workbench](/docs/5/fusion/intro/ui-tour/query-workbench) provides an environment for configuring the stages in a query pipeline.
* The [Query Stages API](/api-reference/query-stage-api/get-the-service-status) is used to create, list, update, or delete query stages using JSON. See also the [Query Pipelines API](/api-reference/query-pipelines-api/get-all-query-pipelines).
### Pipeline stage JSON editor
The pipeline stage JSON editor gives the ability to create and copy pipeline stages by pasting JSON objects in the Fusion UI. Only JSON is supported, and JSON validation is included to prevent the user from saving an invalid object.
Navigate to your pipelines, select a pipeline stage, and click the **JSON View** button to open the editor:
Existing stages are considered **READ-ONLY**. They will have only a copy button which can be used to create stages. New stages will have the option to copy or paste valid JSON. Changes are applied to the stage with the **Apply** button.
Users will still need to **Save** the stage for the changes made in the editor to be saved.
The editor has several elements worth noting:
| Elements | Description |
| - | - |
| | Copies the JSON to the clipboard. |
| | Pastes the clipboard into the editor. Only valid JSON is accepted. |
| | Expands the editor to a fullscreen view. |
| | Condenses the editor to a compact view. |
| | Applies the JSON in the editor and updates the stage. The stage must be saved to preserve the changes. |
| | Resets changes made since the last time changes were applied. |
For instructions, see **Use the Pipeline Stage JSON Editor**.
This article will focus on copying the JSON configuration for an existing pipeline stage and creating a new stage with the JSON configuration.
The instructions in this article focus on query pipeline stages. However, the steps are the same for index pipeline stages, which are accessed at \*\*Indexing > Index Pipelines > ``.
1. Navigate to **Querying > Query Pipelines** and select a pipeline.
2. Select an existing pipeline stage to view in the editor.
3. Click the **JSON View** button. The editor will appear:
Existing stages are considered **READ-ONLY**. They will only have a copy button.
4. (optional) Click **Expand editor** to view the editor in fullscreen mode.
5. Click the **Copy** button. The JSON configuration for that stage is copied to your clipboard.
6. Click **Add a new pipeline stage** to create a new stage. Create the same stage type.
7. Click the **JSON View** button.
8. Use the **Paste** button to paste the JSON configuration from your clipboard into the editor. Alternatively, delete the existing JSON configuration and manually paste the new configuration.\
At this time, you can make changes to the JSON configuration inside the editor.
9. Click the **Apply** button to apply your changes to the stage. Although the changes are immediately shown in the UI, the stage is not saved until you click the **Save** button.
10. Save the stage.
## JavaScript selector
Now your pipeline definitions can include your choice of JavaScript engine, either Nashorn or OpenJDK Nashorn.
While Nashorn is the default option, it is in the process of being deprecated and will eventually be removed, so it is recommended to use OpenJDK Nashorn when possible.
You can select the JavaScript engine in the pipeline views or in the workbenches.
Your JavaScript pipeline stages are interpreted by the selected engine.
## Asynchronous query pipeline processing
Query pipeline processing performance can be improved by enabling asynchronous processing for certain stages that make requests to secondary collections, external databases, and so on. The following stages support asynchronous processing:
* [Active Directory Security Trimming](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/active-directory-security-trimming-query-stage)
* [Apply Rules](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/query-rules-query-stage)
* [Boost with Signals](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommendation-boosting-query-stage)
* [JDBC Lookup](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/jdbc-query-stage)
* [LWAI Prediction](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/lucidworks-ai-prediction-query-stage)
* [LWAI Vectorize Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/lwai-vectorize-query)
* [Security Trimming](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/security-trimming-query-stage)
* [Solr Subquery](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/subquery-query-stage)
This feature uses the fork-and-join model, where any of the stages above can create a fork on the pipeline. The parallel processes are joined again using the [Merge Async Results stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/merge-async-results-query-stage) at a later point in the pipeline.
### Monitoring for asynchronous query pipeline stages
The monitoring feature provides a framework that integrates with tools such as Grafana for seamless observability, and supports complex AI-driven search and generative AI (Gen-AI) workloads.
Monitoring provides critical, real-time visibility into the performance and reliability of asynchronous query pipeline stages. The information includes tracking execution times, failures, and performance bottlenecks. In addition, monitoring information enables system optimization and the ability to more quickly and easily troubleshoot issues.
Use these execution metrics and failure analysis to reduce downtime and accelerate issue resolution, as well as optimize search and AI-driven applications for efficiency and scalability. This ensures better search relevance, improved operational stability, and readiness for next-generation AI search experiences.
## Conditional query processing
Query Pipeline stages are used to modify Request objects
and Response objects. Each stage can include a conditional JavaScript expression (the `condition` property in its configuration) that can access these objects.
For example, this condition first checks that the property `fusion-user-name` is specified in the Request object, then checks for a particular value:
```js wrap theme={"dark"}
request.hasParam("fusion-user-name") && request.getFirstParam("fusion-user-name").equals("SuperUser");
```
## Learn more
The quick learning for **Top 3 Query Pipeline Stages** focuses on useful querying stages.
# Query Profiles
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-basics/query-pipelines/query-profiles
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-basics/query-pipelines/query-profiles
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-basics/query-pipelines/query-profiles
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/174
Query profiles let you consistently point your search application at a static endpoint, but give you the flexibility to change the actual query pipeline being used (and optionally, the collection and/or configuration parameters for the query pipeline).
For example, an e-commerce site might want to create a query pipeline to support a month-long promotion. After the query pipeline is configured, it can be easily enabled by changing the query profile in use by the front-end application to use the new pipeline.
To run a query through a query profile, use the REST API as described below. For CRUD operations, you can use either the REST API or the Fusion UI.
## Rules Simulator query profile
**Important**
Rules Simulator is only available for Fusion 5.x.x organizations that do *not* have a valid Predictive Merchandiser or Experience Optimizer license.
The [Rules Simulator](/docs/4/fusion-ai/concepts/query-rewriting/simulator) allows product owners to experiment with rules and other query rewrites in the `COLLECTION_NAME_query_rewrite_staging` collection before deploying them to the `COLLECTION_NAME_query_rewrite` collection.
Each app has a `COLLECTION_NAME_rules_simulator` query profile, configured to use the `COLLECTION_NAME_query_rewrite_staging` collection for query rewrites instead of the `COLLECTION_NAME_query_rewrite` collection. This profile is created automatically whenever a new app is created.
See **Configure the Rules Simulator Query Profile** for more information about configuration.
Each app has a `_rules_simulator` query profile, configured to use the `_query_rewrite_staging` collection for [query rewrites](/docs/4/fusion-ai/concepts/query-rewriting/overview) instead of the `_query_rewrite` collection. This profile is created automatically whenever a new app is created.
By default, this query profile points to your default query pipeline and collection. You can configure it to point to any pipeline or collection, for example when testing a new pipeline before it has been deployed.
**How to change the query pipeline, collection, and query parameters used by the `_rules_simulator` query profile**
1. Open the Fusion UI.
2. Navigate to **Querying** > **Query Profiles**.
3. Select the `_rules_simulator` query profile for your app.
For example, if your app is called "Demo" then the name of the query profile is `Demo_rules_simulator`.
4. Modify the configuration as desired.
5. Click **Save**.
The course for **Index and Query Profiles** focuses on how index and query profiles operate, and how to configure and manage them.
## Query profiles in the REST API
* [Query Profiles CRUD API](/api-reference/query-profiles-crud-api/get-the-service-status) (`/query-profiles`)\
Create, read, update, and delete query profiles.
* [Query Profiles API](/api-reference/query-profiles-api/get-the-service-status) (`/query`)\
Run a query through a query profile by specifying the profile ID and appending the request with a [Solr query string](/docs/5/fusion/getting-data-out/query-basics/query-language-cheat-sheet), as in `/api/query/?`.
## Query profiles in the UI
To configure query profiles, click **Querying** > **Query Profiles**.
# Restrictive Mode
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-basics/query-pipelines/restrictive-mode
Restrictive mode safeguards the pipeline against unintended or unsafe changes. You can enable restrictive mode for any query pipeline.
This feature is available starting in Fusion 5.9.15.
The default mode is permissive mode.
When a query pipeline is in restrictive mode:
* **Only certain query parameters are allowed.**\
Restrictive mode blocks parameters that could be used destructively.
* **Strict parameter validation and sanitization is applied.**
* The `q` and `fq` parameters are rejected if they contain LocalParams syntax (`{!...}`), which prevents parser-injection attacks. General Solr query syntax (field:value, booleans, wildcards, ranges, phrase queries) is still permitted.
* The `rows` and `start` parameters must be less than 100 (valid range: 0–99).
* The `sort` parameter is allowlisted but values are not validated against specific sortable fields.
## Enabling restrictive mode
You can enable restrictive mode for any individual pipeline, in the Query Pipelines panel.
1. Navigate to **Query** > **Query Pipelines**.
2. Select the pipeline you want to secure.
3. In the **Parameter Validation** field, select **Restrictive**.
## Allowed parameters
The parameters listed below are allowed in restrictive mode.
All other parameters are prohibited.
* `bf`
* `boost`
* `bq`
* `callback`
* `collection`
* `context`
* `cursorMark`
* `debug`
* `defType`
* `echoParams`
* `explainOther`
* `facet.field`
* `facet.limit`
* `facet.mincount`
* `facet.pivot`
* `facet.query`
* `facet.range`
* `facet.sort`
* `facet`
* `fl`
* `fq`
* `hl.fl`
* `hl.fragsize`
* `hl.simple.post`
* `hl.simple.pre`
* `hl.snippets`
* `hl`
* `indent`
* `json.facet`
* `json.nl`
* `json.wrf`
* `mm`
* `omitHeader`
* `pf`
* `ps`
* `q`
* `qf`
* `qs`
* `queryProfileID`
* `requestHandler`
* `rows`
* `segmentTerminateEarly`
* `sort`
* `start`
* `tie`
* `timeAllowed`
* `wt`
# The Query Workbench
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-basics/query-workbench
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-basics/query-workbench
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-basics/query-workbench
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/10918
The Query Workbench is where you edit query pipeline stages, results parameters, and more, then preview the impacts of the changes in real time before saving them. Relevancy tools such as boosting and blocking can be accessed and altered through the Query Workbench. The Query Workbench streamlines the process of querying indexed results and fine-tuning Query Pipelines to surface the results that best satisfy the end user’s needs.
See the [Query Language Cheat Sheet](/docs/5/fusion/getting-data-out/query-basics/query-language-cheat-sheet) for help constructing queries.
With this sandbox paradigm, you can experiment with query pipelines without consequence because Fusion uses a copy of a pipeline to simulate the new results, and it is not permanently modified in the collection until you click **Save**.
## Query pipelines and pipeline stages
[Query pipelines](/docs/5/fusion/getting-data-out/query-basics/query-pipelines/overview) work by processing search requests and returning an ordered list of matching documents. Each pipeline consists of a series of query stages that can be added, ordered, and configured using the Query Workbench.
Select any stage in the pipeline to open its configuration panel:
## Boosting and blocking
As you search your data and inspect the results, you can manipulate the rankings of individual documents. Boosting a document raises its ranking, while blocking a document removes it from search results.
Boosting and blocking affects the results for specific queries. For example, when you search for "citizen" and boost the document for "Citizen Kane", your end users will see that title boosted to the top when they also search for "citizen", but not necessarily when they search for "kane".
The Query Workbench provides convenient buttons for instantly blocking or boosting the documents that you see in the preview panel. Clicking **Boost** automatically adds the [Boost Documents stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/boost-documents-query-stage) to your query pipeline, with a boost rule that matches your query and the document you clicked. Likewise, clicking **Block** adds and configures the [Block Documents stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/block-documents-query-stage).
## Compare mode
Compare mode allows you to compare search results using two different query pipelines, or compare search results from the original query pipeline and the copy being modified in the Query Workbench. On the right is the working pipeline, which you can edit. On the left is one of your saved pipelines.
## Learn more
1. How to boost a document in the Query Workbench
2. In the preview panel, hover over the document you want to boost.
3. Click **Boost**.
The preview panel automatically updates the rankings of the search results, and tags the differences:
Notice that the Boost Documents stage is now in our pipeline. You can click this stage to view the boost rules you have added. This is also where you can remove boost rules:
4. When you are satisfied with the results, save the query pipeline.
**How to block a document in the Query Workbench**
1. In the preview panel, hover over the document you want to block.
2. Click **Block**
The preview panel automatically updates the rankings of the search results, and tags the differences.
Notice that the Block Documents stage is now in our pipeline. You can click this stage to view the block rules you have added. This is also where you can remove block rules:
3. When you are satisfied with the results, save the query pipeline.
The Query Workbench allows you to edit the search parameters. Click **Parameters** to display the current search parameters and the **Edit parameters** button:
Then, click **Edit parameters** to open the "Parameters and Values" window where you can add, modify, or remove search parameters:
1. In the Query Workbench, open the query pipeline you want to modify.
2. Click **Add a Stage** and select a stage that supports asynchronous processing:
* Active Directory Security Trimming
* Security Trimming
* Boost with Signals
* Apply Rules
* JDBC Lookup
* LWAI Vectorize Query
* Solr Subquery
* LWAI Prediction Query
3. In the stage configuration panel, select **Asynchronous Execution Config**.
4. Select **Enable Async Execution**.\\
Fusion automatically assigns an **Async ID** value to this stage. Change this to a more memorable string that describes the asynchronous stages you are merging, such as `signals` or `access_control`.
5. Copy the **Async ID** value.
6. Click **Apply**.\
Verify that the stage is in the correct position in your pipeline.
7. Click **Add a Stage** and select the Merge Async Results stage.\\
8. In the stage configuration panel, click the Add button next to **Async IDs**.
9. Paste the ID from the asynchronous query stage into the new ID field.
When using multiple asynchronous stages, the order in which IDs are entered here is the order in which they are joined back into the pipeline.
10. Click **Apply**.\
The Merge Async Results stage is now the first stage in the pipeline.
11. Drag the Merge Async Results stage down so that it appears immediately before the Solr Query stage:
12. Click **Save**.
By default, every stage in a query pipeline is enabled. While working with a query pipeline, it can be helpful to disable stages without removing them completely. This allows you to preserve a stage’s configuration while observing how the search results change in its absence. You can re-enable the stage at any time. When you save a query pipeline, the enabled/disabled state of each stage is also saved.
Click the circle next to any stage in order to enable or disable it.
Federated search lets you query across multiple collections in Fusion. This is useful if you keep separate data collections for security or compliance reasons or maintain different collections based on data type.
1. In the Fusion workspace, navigate to **Querying > Query Workbench**.
2. Click **Solr Query** to open the Solr Query panel.
3. Enable **Allow Federated Search** then click **Apply**.\\
4. In the workbench area, click **Parameters** then click **Edit Parameters**.
5. Click Add to add a parameter. For the **Name**, enter `collection` and for the **Parameter Value**, enter a comma-separated list of collections you want to query. For example, `movies-collection,books-collection,music-collection`.\\
6. Click **Close**.
7. Check that your pipeline is querying documents from all specified collections.\\
Ultimately, your search application will format the search results that end users see. The Query Workbench provides some formatting options for the preview panel.
At the bottom of the screen, click **Format Results** to configure how results are displayed in the QWB:
These options only affect how the Query Workbench displays results; they have no effect on how your search application displays them.
1. Use the Index Workbench to set up datasource(s).
2. Run a crawl on the data.
3. Open the Query Workbench.
4. Select a query pipeline to use or create a new, blank pipeline (the Default pipeline is created automatically and can be used as-is or with modification).
5. Modify the pipeline stages and set the order in which the queries will run.
6. Examine the live results in the Search panel.
7. Modify any pertinent stage settings to fine tune your results.
8. Click **Save** to commit the new settings to the pipeline
After a custom Query Pipeline configuration is saved, it becomes available throughout Fusion and can be re-used and modified.
The order of the pipeline stages matters, because the output from one stage becomes the input to the next stage. For example, the Solr Query stage must always come last in the sequence, so that data is indexed only after it has been processed by all other stages. Putting this stage first in the sequence means that subsequent stages have no effect on the indexed data.
Drag any stage in the pipeline to move it up or down in the sequence of stages. The preview panel automatically updates the search results to reflect the output of the new sequence.
While the Query Fields stage configures which fields match against incoming queries, the search application itself determines which fields to display in the search results.
For example, the search results below use the `title_s` field as the document name and the `year_i` field as the document description:
Adding `year_i` as the description removed it from the list of field facets and now we see the year of release for each movie title in the search results:
1. In the Query Workbench, click **Compare**.\
Another preview panel opens. In this view, you can compare results from one query pipeline side by side with another query pipeline.
2. In the left panel, select a pipeline to compare to your working pipeline.\
Now you can see how the search results differ between the two pipelines:
When you click a document in one panel, the other panel automatically scrolls to the same document.
To exit compare mode, click the Close icon.
# Stored Parameters Collection
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-basics/stored-parameters-collection
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-basics/stored-parameters-collection
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-basics/stored-parameters-collection
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/172
A Stored Parameters collection contains lists of parameters, where each list is has a unique identifier called a "key".
These lists of parameters can be plugged into Fusion query pipelines. The Stored Parameters facility provides a
mapping layer between the keys and specific actions carried during search and is somewhat similar to database triggers
or stored procedures. The use for this collection is to store Solr facet parameters for a particular document or product category, where
the facet parameters are stored as a JSON object.
An example of what the input JSON is shown in a section below.
A Stored Parameters collection is an auxiliary Fusion collection which is associated with a primary collection via naming conventions:
for a primary collection, the name of the associated Stored Parameters collection consists of the name of the primary collection plus the suffix "\_stored\_parameters".
In the Fusion UI, Stored Parameter collections are not displayed in the list of collections in the Collections Admin panel,
nor are there controls on the UI for creating or populating these collections; all collection administration is done via calls
to the REST API.
## Stored Parameters for Facets
* An existing facets configuration can be converted into the required format via a once-off custom script (e.g. written by Lucidworks solution engineers or by a customer’s technical team).
* Note that if a caller wants particular behavior e.g. specifying that all results should be from a given category, then it is up to them to specify that in their original request (e.g. using a filter query "fq" setting).
* A new Stored Parameter Query Stage can then be used in a query pipeline to add the required parameters for a given key (e.g. a category ID).
## Stored Parameters Example
The following example shows how faceting can be carried out over a hierarchy of categories and subcategories where there
are many known subcategory-specific attributes to query on.
The example data is taken from the Best Buy product catalog data.
In this list of stored parameter objects, the set of fields used are:
* "id" : record ID (or "key" in the sense of a "Key-Value" store) of the category to facet on.
* "label\_s" : human-readable label for this record, i.e., the category name
* "in\_ids\_ss" : parent categories, if an intermediate category in a category hierarchy. Intended use is in traversing a directed graph of categories.
* "out\_ids\_ss" : child categories, if an intermediate category in a category hierarchy. Intended use is in traversing a directed graph of categories.
* "parameters\_ss" : list of arbitrary Solr parameters that can be used to construct a query that is relevant for the entity described by this record. In this example, it is the list of facet parameters to use for this specific category.
```json wrap theme={"dark"}
[
{"id": "abcat0600000",
"label_s": "Movies & Music",
"in_ids_ss": ["cat00000"],
"out_ids_ss": ["cat02001", "cat02015"] },
{ "id": "cat02001",
"label_s": "Music",
"in_ids_ss": ["abcat0600000"],
"out_ids_ss": ["cat02673", "cat02012", "cat02004"] },
{ "id": "cat02010",
"label_s": "Rock",
"in_ids_ss": ["cat02001"],
"out_ids_ss": ["cat02673", "cat02687", "cat02661", "cat02662", "cat02675", "cat02678", "cat02680"],
"parameters_ss": ["facet.field=attr_format_", "facet.field=attr_albumLabel_", "facet.field=attr_genre_"] },
{ "id": "cat02673",
"label_s": "Hard Rock",
"in_ids_ss": ["cat02010"],
"parameters_ss": ["facet.field=attr_format_", "facet.field=attr_albumLabel_", "facet.field=attr_genre_"] },
{ "id": "abcat0208001",
"label_s": "Batteries" },
{ "id": "abcat0208002",
"label_s": "Alkaline Batteries",
"in_ids_ss": ["abcat0208001"],
"parameters_ss": ["facet.field=attr_hardGoodType_"] }
]
```
## Direct Access and Navigation
A caller can query the `COLLECTION_NAME_stored_parameters` system collection directly.
For example, a Solr request to get the human-readable label stored in field "label\_s" for a specific category ID is:
```bash theme={"dark"}
`http://127.0.0.1:8983/solr/COLLECTION_NAME_stored_parameters/query?q=id:&fl=label_s`
```
Direct access allows for navigation up and down a hierarchy of categories by listing incoming and outgoing links that specify a category’s location.
For example, the Solr requests to find the parent(s) and children of a given category, respectively, are:
```bash theme={"dark"}
`http://127.0.0.1:8983/solr/COLLECTION_NAME_stored_parameters/query?q=id:&fl=in_ids_ss`
`http://127.0.0.1:8983/solr/COLLECTION_NAME_stored_parameters/query?q=id:&fl=out_ids_ss`
```
Note that this kind of "graph navigation" can become inefficient if it requires submitting a large number of queries to navigate up and down the hierarchy.
# Boost With Signals
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/boosting/boost-with-signals
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/boosting/boost-with-signals
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/boosting/boost-with-signals
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/483
The Boost With Signals query pipeline stage performs automatic boosts based on the contents of the aggregated signals collection. For this type of boosting, signals must be enabled but recommendations need not be.
Using the main query and the stage configuration parameters, this stage performs a secondary query to the `COLLECTION_NAME_signals_aggr` collection and returns updated boost weights for the items in the main query’s search results. Items that have received more user interaction also receive higher boost weights.
Some of the important configuration parameters are discussed below. For complete details about all configuration parameters for this stage, see [Boost with Signals](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommendation-boosting-query-stage).
## Configuration overview
The fields below are especially useful to understand when configuring this stage.
Sets the `rows` query param in the main query as the maximum number of query results which will be boosted by this pipeline stage.
Sets the `rows` query param in the query that searches the `COLLECTION_NAME_signals_aggr` collection, so only the specified number of aggregated signals are retrieved and used for boosting. When signals boosting is applied to a query, aggregated signals records are queried from the appropriate `_signals_aggr` collection to find out the popularity or boost weight for documents which have signals. `numSignals` limits the number of records to be queried from a `COLLECTION_NAME_signals_aggr` collection and used to calculate this boost.
A filter to retrieve aggregated signals in the `COLLECTION_NAME_signals_aggr` collection per each aggregated signal’s `aggr_type_s` field value.
The document field in the main collection on which to perform boosting. Typically it should use the default field, which is `id`.
This field corresponds to the **Rollup Field**/`rollupField` field. Together, these two fields act like a `FIELD:VALUE` pair in the query modification for boosting.
This adds a query parameter to the original query, either “query-param” or “query-parser”. The result is `(“query-param” or “query-parser”) + Boost Param (“boost” or “bq”)`, as in the examples below:
* “query-param” + "boost", result boost query param
```js theme={"dark"}
boost="map(query({!field f='id' v=‘6239046,13026192}), 0, 0, 1, 27.1705)"
```
* “query-parser” + "boost", result boost query param
```js theme={"dark"}
bp_xxx_bbqx="map(query({!field f='id' v=‘6239046,13026192}), 0, 0, 1, 27.1705)"
```
When `Boost Param` uses `bq`, similar logic applies. When Boost Param/`boostingParam` uses “boost”, it works with both “query-param” and “query-parser”.
Indicates which aggregated signal document field the boost parameter will use for the final boosting. It works in combination with the **Solr Field to Boost On**/`boostId` field.
This should be set to the field in the aggregated signal collection that stores the doc list that is aggregated as one record. By default it’s set to `doc_id_s`.
Indicates the final boost weight used to calculate the new score for docs retrieved by the main query.
Similar to **Rollup Field**/`rollupField` above, this should be set to the field in the aggregated signal collection that stores the final weight that was calculated. By default it’s `weight_d`.
Calculates the final weight using the weight and score retrieved from the `COLLECTION_NAME_signals_aggr` collection.
The default value is:
```text theme={"dark"}
math:log(weight_d + 1) + 10 * math:log(score+1)
```
## Solr query parameters
These parameters are used in the **Solr Query parameters**/`queryParams` field for retrieving signal aggregation docs from the `COLLECTION_NAME_signals_aggr` collection. These Solr query params will affect which aggregated signals are used for producing the boosting parameter on the main query.
| | |
| - | - |
| `qf=query_t` | Defines which field to query. In the default case, the query searches on the `query_t` field of aggregated signal docs. |
| `pf=query_t^50` | Boosts docs within the set of retrieved docs using phrase matching. |
| `pf2=query_t^20` | `pf2` is similar to `pf`; the difference is that `pf2` works on bigram phrases. |
| `pf3=query_t^10` | `pf3` is similar to `pf`; the difference is that `pf3` works on trigram phrases. |
## FAQs
**If there is `fq` in the main query, how is it matched with the correct aggregated signal?**
In this case, you need to use the `lw.rec.fq` query parameter in the main query. `lw.rec.fq` can be parsed by the Boost with Signals stage, and therefore the filters specified in it can be added to the Solr query that is retrieving the aggregated signals.
For example, if we have filter query param `fq=format:CD&fq=name:Latin`, this needs to be translated into `lw.rec.fq=filters_s:"format:cd $ name:latin"`. Values must be lowercase. The final main query should be:
```sh wrap theme={"dark"}
http://FUSION_HOST:FUSION_PORT/api/apps/demo_app/query-pipelines/demo_app/collections/demo_app/select?echoParams=all&wt=json&json.nl=arrarr&sort&start=0&q=apple&debug=true&rows=10&lw.rec.fq=filters_s:"format:cd $ name:latin"
```
Now the Boost with Signals stage will only retrieve aggregated signals that have the same filter query.
If there are multiple `fq` values (for example, `format:cd` and `name:latin`), they are ordered alphabetically as strings and joined with " $" (a$ with a space on each side). In the example, `"format:cd $ name:latin"`.
**What if my aggregated signals are in a different collection?**
You can point the Boost with Signals stage to a different signal collection by adding a `collection` parameter in the `Solr Query Parameters` section.
# Boosting
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/boosting/overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/boosting/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/boosting/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/3242
[Signals](/docs/5/fusion/getting-data-out/query-enhancement/signals/overview) contain data about how users interact with search results. Once your Fusion app has accumulated a sufficient number of [aggregated signals](/docs/5/fusion/reference/config-ref/jobs/aggregations/overview), they become useful for automatically producing recommendations and boosts for better relevance and higher conversion rates.
The same data used to produce recommendations can also be used for automatic boosting. So although recommendations and boosts are different (as explained below), these topics use "recommender data" to refer to the data that can be used for boosting as well as for recommendations.
## Recommendations vs boosts
Recommendations and boosts are two ways of presenting AI-powered estimations about which items are most likely to interest a user:
| Recommendations | Boosts |
| - | - |
|
Personalized search results from a special, automatic query, *whether or not the user has performed a query*. Recommendations can be based on a variety of criteria, as in the examples below: • "Based on your purchase history, you might be interested in these items." • "People who viewed this item also viewed these items." • "People who searched for 'ipad' also searched for these items."
|
Modifications to the *scores* of the items in a query’s search results. The set of search results does not change, but their ranking does. For example: • Boost the most popular items • Boost items this user has clicked before • Boost seasonal items
Boosts can also be configured manually, using the [ Query Workbench ](/docs/5/fusion/intro/ui-tour/query-workbench) or the [ Query Rewriting UI ](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/overview) . You can also perform boosts within a set of recommendations.
|
These topics explain how to configure and use recommendations and boosts:
* [Getting Started](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/getting-started) shows you how to quickly enable the basic feature set and see some results.
* [Recommendation Methods](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods) explains the available approaches to recommendations and boosting, including methods that do not rely on signals.
## See also
* [Boost with Signals](/docs/5/fusion/getting-data-out/query-enhancement/boosting/boost-with-signals)
# Actions
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/business-rules/actions
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/business-rules/actions
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/business-rules/actions
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/8836
Business rule actions define what actions are taken when a rule fires.
Some rule actions are only available to specific version of Fusion.
## Configuration options
### Banner action
The Banner action returns an image outside of the search results that can be displayed in the frontend.
For example, an e-commerce website may display a banner to promote a sale or new product release.
| Required | Parameter | Description | Example |
| - | - | - | - |
| ✅ | **Banner URL** | The URL of the desired banner image. | `https://example-site.com/assets/promotion.png` |
| | **Banner zone** | An arbitrary value used by the frontend, generally to instruct the search application where to place the banner image. | `TOP` |
### Block list
The Block List action blocks the associated values from appearing in the search results.
For example, a workplace may block documents that the user does not have permission to view.
| Required | Parameter | Description | Example |
| - | - | - | - |
| ✅ | **Field name** | The name of the field that the rule action applies to. The documents in the datasource determine what fields are available. | `title` |
| ✅ | **Field values** | The values of the field that the rule action applies to. The field determines what field values are available. | `Voyager Jacket` |
### Boost attributes
The Boost Attributes action boosts documents with specific attributes in a search. Boosts can be additive or multiplicative.
For example, an e-commerce website may boost the laptop manufacturer’s name if the search query contains a manufacturer.
| Required | Parameter | Description | Example |
| - | - | - | - |
| ✅ | **Boost param** | ● `bq`. An additive boost that is added to the main query’s score. ● `boost`. A multiplicative boost based on document scores. This boost increases or decreases the score by a relative amount. | `boost` |
| ✅ | **Boost query** | The query that the boost will apply to. | `bq`: `category_s:"Small Business"^5` `boost`: `category_s:"Small Business"` |
### Boost list
The Boost List action boosts documents with a large multiplicative boost so the documents appear at the top of the results. Larger boosts can override this action.
For example, an e-commerce website may boost new or seasonal items.
| Required | Parameter | Description | Example |
| - | - | - | - |
| ✅ | **Field name** | The name of the field that the rule action applies to. The documents in the datasource determine what fields are available. | `title` |
| ✅ | **Field values** | The values of the field that the rule action applies to. The field determines what field values are available. | `Voyager Jacket` |
### Bury list
The Bury List action buries documents with the selected values. This action minimizes certain documents without blocking them.
For example, a workplace may bury out-of-date documents.
| Required | Parameter | Description | Example |
| - | - | - | - |
| ✅ | **Field name** | The name of the field that the rule action applies to. The documents in the datasource determine what fields are available. | `title` |
| ✅ | **Field values** | The values of the field that the rule action applies to. The field determines what field values are available. | `Voyager Jacket` |
### Filter list
The Filter List action applies [filter queries](https://solr.apache.org/guide/common-query-parameters.html) (`fq`) to the query and changes the results so a pre-selected set of content displays.
For example, an e-commerce website may display only movies rated G or PG when a user searches for children’s movies.
| Required | Parameter | Description | Example |
| - | - | - | - |
| ✅ | **Field name** | The name of the field that the rule action applies to. The documents in the datasource determine what fields are available. | `title` |
| ✅ | **Field values** | The values of the field that the rule action applies to. The field determines what field values are available. | `Voyager Jacket` |
### Ingroup actions
Ingroup actions perform a specific action within a [product group](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/product-grouping). You can boost, bury, or pin specific items within a group.
Ingroup actions require a grouping field to perform the action within.
#### Ingroup boost list
The Ingroup Boost List action boosts an item within a product group. This action does not boost the product group.
| Required | Parameter | Description | Example |
| - | - | - | - |
| ✅ | **Field name** | The name of the field that the rule action applies to. The documents in the datasource determine what fields are available. | `title` |
| ✅ | **Field values** | The values of the field that the rule action applies to. The field determines what field values are available. | `Voyager Jacket` |
| ✅ | **Grouping field** | The name of the group field that the rule action applies to. This group field *must* have a `group-field` parameter or the rule does not apply. | `color_group_name` |
#### Ingroup bury list
The Ingroup Bury List action buries an item within a product group. This action does not bury the product group.
| Required | Parameter | Description | Example |
| - | - | - | - |
| ✅ | **Field name** | The name of the field that the rule action applies to. The documents in the datasource determine what fields are available. | `title` |
| ✅ | **Field values** | The values of the field that the rule action applies to. The field determines what field values are available. | `Voyager Jacket` |
| ✅ | **Grouping field** | The name of the group field that the rule action applies to. This group field *must* have a `group-field` parameter or the rule does not apply. | `color_group_name` |
#### Ingroup pinned
The Ingroup Pinned action sticks a document to a specific location within a product group. This ensures that users view a result in a specified place. This action does not pin the product group.
| Required | Parameter | Description | Example |
| - | - | - | - |
| ✅ | **Field name** | The name of the field that the rule action applies to. The documents in the datasource determine what fields are available. | `title` |
| ✅ | **Field values** | The values of the field that the rule action applies to. The field determines what field values are available. | `Voyager Jacket` |
| ✅ | **Grouping field** | The name of the group field that the rule action applies to. This group field *must* have a `group-field` parameter or the rule does not apply. | `color_group_name` |
| ✅ | **Pin within a specific group value** | The values of the grouping field that the rule action applies to. The field determines what field values are available. Only a single group value is supported. | `red` |
| ✅ | **Pinned documents** | The individual documents that the rule action applies to. | `Summit Pants (red)` |
### Pinned
The Pinned action sticks a document to a specific location within the search results. This ensures that users view a result in a specified place.
For example, a workplace may pin documents answering frequently asked questions.
| Required | Parameter | Description | Example |
| - | - | - | - |
| ✅ | **Field name** | The name of the field that the rule action applies to. The documents in the datasource determine what fields are available. | `id` |
| ✅ | **Pinned documents** | Individual documents to pin. Specify the following: ● Value. The values of the field that the rule action applies to. The field determines what field values are available. ● Position. Position within the results where the document is pinned, with `1` being the 1st result. Each position is available to only a single document. | Value: `1000026` Position: `4` |
### Redirect
The Redirect action displays a message at the top of the search results that the frontend can utilize to redirect a user to a specific URL. This action guides users to the desired URL when the search query isn’t included in the collection.
For example, a workplace may create a redirect action that displays a link to the internal help desk when an employee searches for "computer broken".
| Required | Parameter | Description | Example |
| - | - | - | - |
| ✅ | **Redirect** | The URL that the user will redirect to. | `https://www.activate-conf.com/` |
### Response value
The Response Value action sends an arbitrary value to the frontend or pipeline, which the frontend of pipeline can use to perform another action. The Response Value action can be combined with other actions within the same rule.
For example, an e-commerce website may use the response value action to display an advertisement.
| Required | Parameter | Description | Example |
| - | - | - | - |
| ✅ | **Field name** | The name of the field that the response value is found under. | `ad-code` |
| ✅ | **Field value** | The response value that is included in the results response. | `AD_CODE_99` |
### Set facets
The Set Facets action customizes the facets returned in the query response. These facets are different from the facets set in the Query Workbench.
For example, an e-commerce website may display different facets on the homepage, the primary search page, and category pages.
| Required | Parameter | Description | Example |
| - | - | - | - |
| ✅ | **Position** | Position of the facet field in the list of facets, with `1` being the top of the list. Each position is available to only a single facet. | `3` |
| ✅ | **Facet field** | The name of the facet field. | `Color` |
| ✅ | **Facet type** | The type of facet: `field` - A field found in the datasource. `range` - An accepted range for the facet field to exist within. Specify the following: • Range start—the lower bounds of the facet range. For example, `0`. • Range end—the upper bounds of the facet range. For example, `50`. • Range gap—the increment amount dividing the range. For a facet range of `1-50` and a range gap of `20`, 3 facets are available: `0-20`, `20-40`, `40-60`. | `field` |
**Advanced options for `field` facet types**
| Parameter | Description | Example |
| - | - | - |
| **Select type** | Determines whether the user can select a single facet or multiple facets. | `multi` |
| **Facet sort** | Sort order of the individual facets within the facet field: • `count`—Facets are sorted by the number of documents in descending order. • `index`—Facets are sorted in alphanumeric order. | `count` |
| **Min count** | Sets a minimum requirement for the number of documents in the facet to display. | `10` |
| **Boost values** | Boost individual facets in the order added, overriding the default **Facet sort**. For example, boost `tshirt` and `shorts` over `hoodie` during summer. | `tshirt` |
| **Bury values** | Bury individual facets with the facet field in the order added. | `hoodie` |
| **Suppress values** | Prevent individual facets from appearing within the facet field. | `coat` |
**Advanced options for `range` facet types**
| Parameter | Description | Example |
| - | - | - |
| **Select type** | Determines whether the user can select a single facet or multiple facets. | `multi` |
| **Facet sort** | Sort order of the individual facets within the facet field: • `count`—Facets are sorted by the number of documents in descending order. • `index`—Facets are sorted in alphanumeric order. | `count` |
| **Min count** | Sets a minimum requirement for the number of documents in the facet to display. | `10` |
### Set Params
The Set Params action sets a [parameter name and value](https://solr.apache.org/guide/common-query-parameters.html) to a search query.
For example, a workplace may use the name and value to display only specific filetypes in the results.
| Required | Parameter | Description | Example |
| - | - | - | - |
| ✅ | **Parameter name** | The name of the [query parameter](https://wiki.apache.org/solr/QueryParametersIndex). | `fl` |
| ✅ | **Parameter value** | The value of the query parameter. | `discountAmount` |
| ✅ | **Update policy** | The manner in which the parameter affects the results response: ● `append`. Add the parameter to the default parameters in the results response. ● `default`. Add the parameter to the default parameters in the results response if it is not already included in the request or the default specification in the Solr config file solrconfig.xml. ● `remove`. Remove the parameter from the results response. ● `replace`. Replace the default parameter in the results response. | `append` |
### JSON Blob
The JSON Blob action returns an [arbitrary JSON blob](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/overview) to your frontend when a business rule fires. The JSON blob displays outside of the search results.
| Required | Parameter | Description | Example |
| - | - | - | - |
| ✅ | **Blob Type** | An arbitrary value used by the frontend, generally to indicate what kind of data the JSON blob contains. | `default` |
| ✅ | **JSON Blob** | An arbitrary JSON value used by the frontend. When the rule is triggered, the JSON is sent with the results response. This behaves similarly to the **Banner** rule, given that it returns the JSON object *outside* of the search results. | `{"parameter1":"value", "parameter2":["value_a", "value_b"]}` |
## Additional configuration options
* **Set response values.** Combine **Response Value** rules with other rules for purposes as [described above](#response-value).
* **Use Query Elevation Component (QEC).** Apply the rule action to configure the top results regardless of the document’s score. However, enabling QEC does not boost the document’s score. The QEC only supports elevation through the `id` field. If QEC is enabled but is not configured correctly in Solr, the rule does not fire.\
To use the Query Elevation Component, you must first **Use the Query Elevation Component**. Note that query elevation *does not boost scores*.
Fusion currently supports the use of the Query Elevation Component (QEC) with boost lists and filter lists. You must configure Solr in order to enable the QEC option.
Although it will not be accessed for the elevation process, Fusion requires an independent `elevate.xml` file for the use of built-in rule actions with QEC due to a known [issue in Solr](https://issues.apache.org/jira/browse/SOLR-14970).
1. Create an XML file named `elevate.xml` in the same directory as the `solrconfig.xml` file.
2. Use the following as the contents of the `elevate.xml` file:
```xml theme={"dark"}
```
3. In the `solrconfig.xml` file, add the following in the elevator search Component (`searchComponent name="elevator"`). Before adding, check to see if the `searchComponent` already exists within the file.
```xml theme={"dark"}
elevate.xml
```
4. Add `elevator` as a 'last component' part of the select handler in the `solrconfig.xml` file. If a `last-components` array is not already defined, create it at the end of the select handler’s configuration. If one is already defined, add `elevator`.
```xml theme={"dark"}
elevator
```
5. When you create a new business rule, check the **USE QUERY ELEVATION COMPONENT** checkbox to use elevation.
The Query Elevation Component only elevates documents within the [query rewriting rules engine](/docs/4/fusion-ai/concepts/query-rewriting/overview) by the document `id` field. Ensure `id` is entered in the "FIELD NAME" option.
The chart below shows additional configuration support for each business rule action:
| Action | Response value | Query Elevation Component |
| - | - | - |
| **Banner** | ✘ | ✘ |
| **Block List** | ✅ | ✘ |
| **Boost Attributes** | ✅ | ✘ |
| **Boost List** | ✅ | ✅ |
| **Bury List** | ✅ | ✘ |
| **Filter List** | ✅ | ✅ |
| **Ingroup Boost List** | ✅ | ✘ |
| **Ingroup Bury List** | ✅ | ✘ |
| **Ingroup Pinned** | ✅ | ✘ |
| **Pinned** | ✅ | ✅ |
| **Redirect** | ✘ | ✘ |
| **Response Value** | ✅ | ✘ |
| **Set Facets** | ✅ | ✘ |
| **Set Params** | ✅ | ✘ |
# Conditions
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/business-rules/conditions
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/business-rules/conditions
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/business-rules/conditions
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/8837
Conditions, or triggers, define the environment necessary for the rule action to fire. Any combination of conditions (date, field value, query, and query profile) can be defined for a rule.
Query profile triggers can be tested in the [Query Workbench](/docs/5/fusion/intro/ui-tour/query-workbench) or with the [Query Pipelines API](/api-reference/query-pipelines-api/get-all-query-pipelines) by setting `queryProfileID` parameter on the request.
For more information, see [Business Rules Conditions Concepts](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/conditions).
## Configuration options
### Condition types
#### Dates
| Required | Parameter | Description | Example |
| - | - | - | - |
| ✅ | **Dates** | The date and time range for the rule. | `01/01/2021 12:00 to 01/31/2021 12:00` |
#### Field value
| Required | Parameter | Description | Example |
| - | - | - | - |
| ✅ | **Filters** | The filters that determine whether the rule fires: ● Field name. The name of the field that the rule action applies to. The documents in the datasource determine what fields are available. ● Field value. The values of the field that the rule action applies to. The field determines what field values are available. | Field name: `color_group_name` Field value: `red` |
#### Query
| Required | **Parameter** | Description | Example |
| - | - | - | - |
| ✅ | **Match query using** | Method used for query matching. Choose from the following: ● `keywords`. The query must match the keywords of the **Search terms** value exactly. For example, if the **Search terms** value is `red pants`, only that value triggers the rule. The rule is not triggered if the value is `red`, `pants`, or `small red pants`. ● `phrase`. The search query must contain one of the **Search terms** values as a subphrase, and in the same order. For example, if the **Search terms** value is `yellow shirt`, the rule is triggered by `yellow shirt` and `small yellow shirt`. The rule is not triggered if the value is `yellow small shirt`. ● `text`. At least one of the **Search terms** values must match the query, but unlike `phrase`, the query only needs to match part of the value. For example, if the **Search terms** value is `green hat`, the rule is triggered by `green hat`, `green`, `hat`, and `large green hat`. | `keywords` |
| ✅ | **Search terms** | Search terms that will trigger the rule. Terms can be words or phrases. | `sale`, `shoes`, `red` |
#### Query profile
| Required | Parameter | Description | Example |
| - | - | - | - |
| ✅ | **Query profile ID** | The ID of the query profile that allows the rule to fire. Accepts multiple values. | `myFusionApp` |
# Custom Rule Actions
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/business-rules/custom-rules-actions
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/business-rules/custom-rules-actions
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/business-rules/custom-rules-actions
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/8834
Custom rule actions allow you to define a unique action to take when certain conditions are met. They involve three primary components: a query pipeline, a custom rule type, and a rule that is triggered by conditions defined by the user.
To learn use custom rule actions with the API, see **Create Custom Rule Actions with the API**.
Custom rule actions allow you to define a unique action to take when certain conditions are met. They involve three primary components: a query pipeline, a custom rule type, and a rule that is triggered by conditions defined by the user.
This article describes the general workflow for creating a custom rule action. The example assumes the following components are used:
| Component | Name |
| - | - |
| Query pipeline | custom-rule-pipeline |
| Custom rule type | custom-rule-type |
| Rule | custom-rule-action |
1. Create a query pipeline, `custom-rule-pipeline`, that is used to complete the rule action.
2. Create a custom rule type, `custom-rule-type`, that uses your query pipeline. Here are some important parameters to include in your `POST` request.
Rule properties, included as part of the `schema
A unique ID for the custom rule type. The `id` value must be unique across all apps in Fusion (global namespace).\
**Example:** `custom-rule-type`
The ID of the pipeline that is invoked during rule processing.\
**Example:** `custom-rule-pipeline`
A user-friendly name for the custom rule type.\
**Example:** `My Custom Rule`
The ID of the rule’s parameter. This field is used in the API but is not visible in the Rules Editor.\
**Example:** `1st-rule-parameter`
The type of user-input.\
**Example:** `string`
A user-friendly name for the parameter.\
**Example:** `Manufacturer`
A user-friendly description of what the field is used for. The `description` value is shown in the field’s tooltip in the Rules Editor.\
**Example:** The manufacturer of the product to which this rule applies.
For example:
```json theme={"dark"}
curl -u USERNAME:PASSWORD -XPOST -H "Content-type:application/json" \
https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/apps/myFusionApp/custom-rule-types -d '{
"id":"custom-rule-type",
"pipeline_id":"custom-rule-pipeline",
"display_type":"My Custom Rule",
"schema": {
"type" : "object",
"properties" : {
"custom_param1" : {
"type" : "string",
"title" : "Manufacturer",
"description" : "The manufacturer of the product this rule action applies to."
},
"custom_param2" : {
"type": "string",
"title": "Type",
"description": "The type of product this rule action applies to."
}
}
}
}'
```
3. Verify the custom rule type was created by visiting `https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/apps/myFusionApp/custom-rule-types`. Custom rule types are also available in the Rules Editor:
4. Create a rule, `custom-rule-action`, that uses your custom rule, `custom-rule-type`. If you are creating the rule using the Query Rewrite API, the following image depicts which parameters are shared between the custom rule type and the rule:
Note that the custom rule type `id` value is used for the rule’s `type` *and* `custom_type` fields. Remember to assign values to the custom rule properties, `custom_param1` and `custom_param2`.
```json theme={"dark"}
curl -u USERNAME:PASSWORD -XPOST -H "Content-type:application/json" \
https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/apps/myFusionApp/query-rewrite/instances -d '{
"id":"123456",
"type":"custom-rule-type",
"custom_type":"custom-rule-type",
"name":"My Rule",
"description":"A rule that uses my custom rule, custom-rule-type.",
"search_terms":["upcoming sale"],
"custom_param1":"ABC",
"custom_param2":"XYZ",
"pipeline_id":"custom-rule-pipeline",
"display_type":"My Custom Rule"
}'
```
5. Verify your rule was created by checking the rule ID, `123456`, with the Query Rewrite API: `https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/apps/myFusionApp/query-rewrite/instances/123456`
6. Test your custom rule using your normal rule development process.
## Configuration options
| Required | Parameter | Description | Example |
| - | - | - | - |
| ✅ | `id` | A unique ID for the custom rule type. The `id` value must be unique across all apps in Fusion (global namespace). | `custom-rule-type` |
| ✅ | `pipeline_id` | The ID of the pipeline that is invoked during rule processing. | `custom-rule-pipeline` |
| ✅ | `display_type` | A user‑friendly name for the custom rule type. | `My Custom Rule` |
**Rule properties**, included as part of the `schema`, define the values required from the user to create a rule that uses the custom rule type.
| Required | Parameter | Description | Example |
| - | - | - | - |
| ✅ | `custom_param1` | The ID of the rule’s parameter. This field is used in the API but is not visible in the Rules Editor. | `1st-rule-parameter` |
| ✅ | `type` | The type of user‑input. | `string` |
| ✅ | `title` | A user‑friendly name for the parameter. | `Manufacturer` |
| ✅ | `description` | A user‑friendly description of what the field is used for. The `description` value is shown in the field’s tooltip in the Rules Editor. | `The manufacturer of the product this rule action applies to.` |
# General Configurations
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/business-rules/general-config
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/business-rules/general-config
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/business-rules/general-config
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/8838
General configurations are used to characterize a rule, including its name, description, group, and more. They are also used to set a rule’s precedence, or priority over other rules.
## Configuration options
| Required | Parameter | Description | Example |
| - | - | - | - |
| ✅ | Name | A user-friendly name for the rule. | `headphones-sale-01` |
| | Description | A description of the rule for the person who manages business rules. This is an arbitrary value which has no effect on the rule’s purpose or execution. | `"Rule for the upcoming sale on headphones. Boosts limited stock items."` |
| | Rule Group | See [Rule Group](#rule-group) for more information. | |
| | Tags | See [Tags](#tags) for more information. | `browse-rule`, `search-rule` |
| | Precedence | Determines when rules should take effect in relation to other triggered rules. A higher number means higher precedence. For example, a rule with precedence of `10` will take effect before a rule with precedence of `3`. | `10` |
| ✅ | Enabled | Determines whether the rule is enabled. If set to `false`, the rule will not fire under any conditions. | `true` |
### Rule group
Rule groups are used for logically grouping rules for interpretation and debugging purposes. The rule group is specified in the debug response. It does not affect which rules fire.
### Tags
Tags are assigned in the UI. By using tags, you can:
* **Quickly sort rules.** Tags appear in the facet options of the Business Rules editor for fast sorting. Click a tag facet to see all rules associated with the tags.
* **Easily define triggers.** Triggers are associated with tags to determine the conditions under which the rules fire. This reduces the number of triggers needed and speeds up the development process.\
To target which rules fire, you can specify one or more tagnames in the request URL of the user search app. The rule containing the matching tagname is triggered. For example:
* One pin rule is defined with "pin the document titled **Brussel Sprouts** when someone searches for garden", and it is tagged with `vegetable`.
* A second pin rule is defined with "pin the document titled **Tomato** when someone searches for garden", but it is *not* tagged with `vegetable`. Instead, this rule is tagged with `fruit`.
* A third pin rule is defined with "pin the document titled **Vegetables** when someone searches for garden", and it is tagged with `vegetable`.\
If the request URL is:
`https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/query/QUERY_PROFILE?q=garden&tags=vegetable`, then only the first and third rules are fired.\
To fire all three rules, the request URL is:
`https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/query/QUERY_PROFILE?q=garden&tags=vegetable,fruit`.
* **Separate rules by development stage.** You can add the tags `development`, `testing`, and `production` to distinguish between rules in various stages of development.
* **Define different search experiences.** In some cases, you may want to define a different experience for users who browse and users who search. Keep track of these experiences by tagging the rules accordingly.
Tags are also included in the response results:
```json wrap theme={"dark"}
{
"response" : {
"numFound" : 213,
"start" : 0,
"maxScore" : 0.0,
"docs" : [ ... ]
},
"facet_counts" : {
"facet_fields" : {
"tags" : {
"development" : 1,
"testing" : 3,
"production" : 2,
}
},
"facet_ranges" : { }
}
}
```
# Business Rules
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/business-rules/overview
Overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/business-rules/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/business-rules/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/8835
Business rules are manually-created formulas for rewriting queries. This is the most versatile strategy for creating custom query rewrites. It supports a variety of conditions and actions to address a wide range of use cases. When you need a very specific query rewrite, this is the best strategy.
Business rules are applied in the [Apply Rules stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/query-rules-query-stage) of the query pipeline.
To learn how to use business rules in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor), see **Use Business Rules in the Rules Editor**.
This article describes how to use the [Business Rules](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/overview) screen in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor).
The course for **Query Rewriting with Business Rules and Search Rewrites** focuses on how using business rules and search rewrites gives you the opportunity to boost conversions and create better customer experiences by improving your relevancy.
## Facet rules
### Facet menu
To facet the rules list by **Action**, **Status**, **Published**, or **Tags**, click one of the facet items within the facet categories.
To facet the rules list by **Created Date** or **Expiration Date**:
1. Click the date start value. A calendar will appear.
2. Click a date to select the start of your facet range.
3. Click the bottom date and select the end of your facet range.
### Search
Enter a term in the search bar to quickly facet your business rules.
## Rules
### Create a rule
1. Click the button in the top-right corner of the card. Alternatively, click the button in the bottom-left corner of the card. The **New Rule** dialog box will appear.
2. Complete the required fields, at a minimum.
| | | |
| - | - | - |
| General | Condition | Action |
| In the **General** column, only the **Name** field is required. Other fields are optional:
**Description** is an arbitrary string you can use to describe this rule.
**Rule Group** can be a user-defined group that you use to organize your rules.
**Tags** are another way to organize your rules. Tags appear as facets in the Business Rules interface, so you can filter the set of visible rules by tag.
**Priority** can be used to determine which rule should apply first if multiple rules are activated for the same request. Priority is an integer value, ranging from 1 to infinity, with the value 1 given lowest priority. If multiple rules have the same value, those rules are applied in a random order.
**Enabled** means that this rule is applied but not necessarily published. Disabling a rule helps ensure that it is not accidentally published.
| Conditions are triggers that activate the rule when they match the current date and time, query, or field values. See [Rule conditions](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) for more information. | A rule can take different types of actions when the specified conditions are met. See [Action types](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) for more information. You can also create [custom actions](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting). |
Hover over the icon for more information.
3. Once complete, click the button.
### Copy a rule
1. Hover over a rule. A row of buttons will appear on the right.
2. Click the button. The **Copy Rule** dialog box will appear.
3. At a minimum, change the title of the rule.
4. Once complete, click the button.
### Edit a rule
1. Hover over a rule. A row of buttons will appear on the right.
2. Click the button. The **Edit Rule** dialog box appears.
3. Make your desired changes to the rule.
4. Once complete, click the button.
### Approve/deny rules
1. Hover over a rule. A checkbox will appear on the left.
2. Click the checkbox to select the rule.
3. Repeat this process to select multiple rules. *(optional)*
4. Click the Approve button to approve the selected rules, or click the Deny button to deny the rules.
### Enable/disable a rule
1. Hover over a rule. A checkbox will appear on the left.
2. Click the checkbox to select the rule.
3. Repeat this process to select multiple rules. *(optional)*
4. Click the button to enable the selected rules, or click the button to disable the rules.
### Publish a rule
1. Hover over a rule. A checkbox will appear on the left.
2. Click the checkbox to select the rule.
3. Repeat this process to select multiple rules. *(optional)*
4. Click the button to publish the selected rules.
### Tag a rule
Tags are a powerful tool for organizing and publishing rules according to a keyword, such as `black-friday-sale`.
1. Hover over a rule. A checkbox will appear on the left.
2. Click the checkbox to select the rule.
3. Repeat this process to select multiple rules. *(optional)*
4. Click the button. A dialog box will appear.
5. Enter a tag. To enter multiple tags, press **Tab** after writing each tag.
6. Click the button to add the tag to the rule.
See [General Configuration Tags](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/general-config) for information about how to use tagnames in the request URL to fire specific rules.
### Delete rules
#### Delete a single rule
1. Hover over a rule. A row of buttons will appear on the right.
2. Click the button. A confirmation box will appear.
3. Click the button to confirm.
#### Delete multiple rules
1. Hover over a rule. A checkbox will appear on the left.
2. Click the checkbox to select the rule.
3. Repeat this process to select multiple rules. *(optional)*
4. Click the button to delete the selected rules. A confirmation box will appear.
5. Click the button to confirm.
### Sort rules
1. Hover over column titles. A icon will appear for some titles to indicate the rules list can be sorted by that column.
2. Click on the title column to sort the rules.
The column that is currently sorting the rules list is indicated by an or icon.
### Export rules
To export the rules list, simply click the button. This generates a `.csv` file containing your rules.
## Columns
### Customize columns
To customize the columns:
1. Hover over the row of column titles. A button will appear on the right.
2. Click the button. The **Customize Columns to Display** dialog box will appear.
3. Click the checkboxes to select or deselect the columns you want to display.
4. Click the button to save your changes.
## Configuration options
### General
General configurations are used to characterize a rule, including its name, description, group, and more. They are also used to set a rule’s precedence, or priority over other rules.
For complete details, see [General Configurations](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/general-config).
### Condition
Conditions, or triggers, define the environment necessary for the rule action to fire. Any combination of conditions (date, field value, query, and query profile) can be defined for a rule.
For complete details, see [Conditions](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/conditions).
### Action
Business rule actions define what actions are taken when a rule fires.
For complete details, see [Actions](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/actions).
## Custom Rule Actions
Custom rule actions allow you to define a unique action to take when certain conditions are met. They involve three primary components: a query pipeline, a custom rule type, and a rule that is triggered by conditions defined by the user.
For complete details, see [Custom Rule Actions](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/custom-rules-actions). To learn use custom rule actions with the API, see **Create Custom Rule Actions with the API**.
Custom rule actions allow you to define a unique action to take when certain conditions are met. They involve three primary components: a query pipeline, a custom rule type, and a rule that is triggered by conditions defined by the user.
This article describes the general workflow for creating a custom rule action. The example assumes the following components are used:
| Component | Name |
| - | - |
| Query pipeline | custom-rule-pipeline |
| Custom rule type | custom-rule-type |
| Rule | custom-rule-action |
1. Create a query pipeline, `custom-rule-pipeline`, that is used to complete the rule action.
2. Create a custom rule type, `custom-rule-type`, that uses your query pipeline. Here are some important parameters to include in your `POST` request.
Rule properties, included as part of the `schema
A unique ID for the custom rule type. The `id` value must be unique across all apps in Fusion (global namespace).\
**Example:** `custom-rule-type`
The ID of the pipeline that is invoked during rule processing.\
**Example:** `custom-rule-pipeline`
A user-friendly name for the custom rule type.\
**Example:** `My Custom Rule`
The ID of the rule’s parameter. This field is used in the API but is not visible in the Rules Editor.\
**Example:** `1st-rule-parameter`
The type of user-input.\
**Example:** `string`
A user-friendly name for the parameter.\
**Example:** `Manufacturer`
A user-friendly description of what the field is used for. The `description` value is shown in the field’s tooltip in the Rules Editor.\
**Example:** The manufacturer of the product to which this rule applies.
For example:
```json theme={"dark"}
curl -u USERNAME:PASSWORD -XPOST -H "Content-type:application/json" \
https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/apps/myFusionApp/custom-rule-types -d '{
"id":"custom-rule-type",
"pipeline_id":"custom-rule-pipeline",
"display_type":"My Custom Rule",
"schema": {
"type" : "object",
"properties" : {
"custom_param1" : {
"type" : "string",
"title" : "Manufacturer",
"description" : "The manufacturer of the product this rule action applies to."
},
"custom_param2" : {
"type": "string",
"title": "Type",
"description": "The type of product this rule action applies to."
}
}
}
}'
```
3. Verify the custom rule type was created by visiting `https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/apps/myFusionApp/custom-rule-types`. Custom rule types are also available in the Rules Editor:
4. Create a rule, `custom-rule-action`, that uses your custom rule, `custom-rule-type`. If you are creating the rule using the Query Rewrite API, the following image depicts which parameters are shared between the custom rule type and the rule:
Note that the custom rule type `id` value is used for the rule’s `type` *and* `custom_type` fields. Remember to assign values to the custom rule properties, `custom_param1` and `custom_param2`.
```json theme={"dark"}
curl -u USERNAME:PASSWORD -XPOST -H "Content-type:application/json" \
https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/apps/myFusionApp/query-rewrite/instances -d '{
"id":"123456",
"type":"custom-rule-type",
"custom_type":"custom-rule-type",
"name":"My Rule",
"description":"A rule that uses my custom rule, custom-rule-type.",
"search_terms":["upcoming sale"],
"custom_param1":"ABC",
"custom_param2":"XYZ",
"pipeline_id":"custom-rule-pipeline",
"display_type":"My Custom Rule"
}'
```
5. Verify your rule was created by checking the rule ID, `123456`, with the Query Rewrite API: `https://EXAMPLE_COMPANY.b.lucidworks.cloud/api/apps/myFusionApp/query-rewrite/instances/123456`
6. Test your custom rule using your normal rule development process.
## Query Elevation Component
Fusion currently supports the use of the Query Elevation Component (QEC) with boost lists and filter lists. You must configure Solr in order to enable the QEC option.
For instructions about using the QEC, see **Use the Query Elevation Component**.
Fusion currently supports the use of the Query Elevation Component (QEC) with boost lists and filter lists. You must configure Solr in order to enable the QEC option.
Although it will not be accessed for the elevation process, Fusion requires an independent `elevate.xml` file for the use of built-in rule actions with QEC due to a known [issue in Solr](https://issues.apache.org/jira/browse/SOLR-14970).
1. Create an XML file named `elevate.xml` in the same directory as the `solrconfig.xml` file.
2. Use the following as the contents of the `elevate.xml` file:
```xml theme={"dark"}
```
3. In the `solrconfig.xml` file, add the following in the elevator search Component (`searchComponent name="elevator"`). Before adding, check to see if the `searchComponent` already exists within the file.
```xml theme={"dark"}
elevate.xml
```
4. Add `elevator` as a 'last component' part of the select handler in the `solrconfig.xml` file. If a `last-components` array is not already defined, create it at the end of the select handler’s configuration. If one is already defined, add `elevator`.
```xml theme={"dark"}
elevator
```
5. When you [create a new business rule](#create-a-rule), check the **USE QUERY ELEVATION COMPONENT** checkbox to use elevation.
The Query Elevation Component only elevates documents within the [query rewriting rules engine](/docs/4/fusion-ai/concepts/query-rewriting/overview) by the document `id` field. Ensure `id` is entered in the "FIELD NAME" option.
## Additional details
### Query pipeline stages for rules
These stages are part of the [default query pipeline](/docs/5/fusion/getting-data-out/query-basics/query-pipelines/overview):
* [Apply Rules](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/query-rules-query-stage)\
This stage looks up rules that have been deployed to the [`COLLECTION_NAME_query_rewrite` collection](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) and matches them against the query. Matching rules that perform [query rewriting](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) are applied at this stage, while matching rules that perform [response rewriting](/docs/5/fusion/getting-data-out/query-enhancement/response-rewriting) are applied by the [Modify Response with Rules](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/rules-augment-response-query-stage) stage later in the pipeline.
To trigger a rule that contains a tag, specify the tagname in the request URL of the user search app. See [Easily define triggers in tags](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/general-config) for more information.
* [Modify Response with Rules](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/rules-augment-response-query-stage)\
Most rules operate on the request, but some [rule types](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting), such as banner rules or redirect rules, do their work when the response comes back. The Modify Response with Rules stage applies those rules to the response. For example, a banner rule can add a banner URL to the response before returning it to the client.
Generally, if you are using rules, you need both of these stages enabled in your query pipeline. The Apply Rules stage must come *before* the Solr Query stage, while the Modify Response with Rules stage must come *after* the Solr Query stage. Disabling or removing the Apply Rules stage will disable rules entirely. Disabling or removing the Modify Response with Rules stage will disable only the rules that perform [response rewriting](/docs/5/fusion/getting-data-out/query-enhancement/response-rewriting), if any.
### Rules on response signals
Response signals capture a list of rule IDs that match the query in a multi-valued field named `rule_ss`.
### Rules and experiments
The easiest way to integrate rules and experiments is to create one query pipeline with rules enabled and one without. Keep all other settings constant between the two pipelines. The pipeline without rules enabled must be configured manually.
To create an experimental rule, use tags on the rule and set the tag in one of the variant pipelines.
## FAQ
What are arbitrary values?
Arbitrary values are values that are not actively used by Fusion. Instead, they are used by the user or the search application. For example, the **Description** is considered an arbitrary value, as it serves only to inform the user what the business rule is used for.
## Learn more
The course for **Query Rewriting with Business Rules and Search Rewrites** focuses on how using business rules and search rewrites gives you the opportunity to boost conversions and create better customer experiences by improving your relevancy.
# Collapsing Results
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/collapse
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/collapse
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/collapse
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/s25sw9
You can configure the [Query Fields stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/search-fields-query-stage) to collapse search results based on a field, such as `product_id`, to group all variations of each item into a single search result.
This is useful for ecommerce sites where you want to show a single product with multiple SKUs or variations.
You can also control how Fusion selects the variation that represents the collapsed group; the default is the one most relevant to the user’s query.
For example, a user who searches for "red shoes" sees all of the red variations of shoes first, with the option to drill down and see all the variations.
For additional collapse options, see the [Query Fields stage configuration reference](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/search-fields-query-stage).
# Fuzzy Search
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/fuzzy-search
Fuzzy search matches terms that are similar to, but not an exact match for, a query term. Unlike semantic search, which matches terms by meaning using vector embeddings, fuzzy search matches by spelling, using character-level edit distance to catch typos and minor variants.
While Fusion has no single fuzzy search feature, you can combine Solr's native fuzzy matching with Fusion query pipeline stages, choosing an approach based on your search architecture and how much precision you're willing to trade for recall.
## Choose an approach
Depending on your use case and existing pipeline stages, there are several possible approaches to fuzzy search:
The simplest option. Surfaces "Did you mean?" suggestions when a query returns few results. No custom code required.
Rewrites query terms with Solr's fuzzy tilde operator before they reach Solr. Use this to make fuzzy matching automatic for every query.
Applies the `{!fuzzy}` query parser to a specific field, such as a product title, without affecting other fields.
If you're already running NHS, its semantic vector matching handles most misspelling-like cases without any fuzzy configuration.
| Situation | Recommended approach |
| - | - |
| General fuzzy tolerance, minimal setup | Spell Check stage |
| Catch typos automatically at query time | JavaScript Query stage with `~1` |
| Fuzzy matching on one field only | `{!fuzzy}` via Additional Query Parameters stage |
| Neural Hybrid Search is enabled | NHS handles this natively; avoid injecting fuzzy syntax into `q` |
| High-traffic queries with known typos | Manual [misspelling corrections](/docs/5/fusion/getting-data-out/query-enhancement/misspelling-detection) |
## Spell check stage
Add a [Spell Check query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/spell-check-query-stage) after your Solr Query stage (or Neural Hybrid Query stage) in the query pipeline. It triggers Solr's spell checker when result counts fall below a threshold you configure, and returns spelling suggestions without any custom code.
## JavaScript query stage
Solr supports fuzzy matching natively with the tilde operator: `roam~1` matches terms within an edit distance of 1, such as `foam` or `roams`. Edit distance ranges from 0–2, but start with `1`. An edit distance of `2` expands matching aggressively and can degrade both precision and query performance.
Add a [JavaScript Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/javascript-query-stage) before your Solr Query stage to rewrite the `q` parameter:
```javascript theme={"dark"}
function(request, response) {
var q = request.getFirstParam('q');
if (!q || q.indexOf('~') !== -1 || q.indexOf('"') !== -1) return;
var tokens = q.split(/\s+/);
var fuzzy = tokens.map(function(t) {
if (t.length >= 4 && t.indexOf(':') === -1 && t[0] !== '+' && t[0] !== '-') {
return t + '~1';
}
return t;
}).join(' ');
request.putSingleParam('q', fuzzy);
}
```
Never apply this to an NHS pipeline. NHS requires `q` to be a raw user query string, not a Solr query parser string. See [Neural Hybrid Search](#neural-hybrid-search).
Skip terms shorter than 4 characters, terms that already contain an operator (`:`, `+`, `-`), quoted phrases, and terms that are already fuzzy. Don't apply tilde fuzzy to wildcard queries (`*`); the two syntaxes conflict.
For general background on writing and testing pipeline scripts, see [Custom JavaScript Stages for Query Pipelines](/docs/5/fusion/getting-data-out/query-basics/query-pipelines/custom-javascript-query-stages).
## Field-specific fuzzy matching
To apply fuzzy matching to one field only, such as `title_t`, use the `{!fuzzy}` query parser with the [Additional Query Parameters stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/set-query-params-stage):
```
q={!fuzzy f=title_t maxEdits=1 prefixLength=2 maxExpansions=50}searchterm
```
Maximum edit distance, from 0–2. Start with 1.
Number of leading characters that must match exactly. Setting this to 2 or higher significantly reduces the number of terms Solr must evaluate and improves performance.
Caps how many term variants Solr evaluates per fuzzy term. Lower this value if you see latency spikes.
When `true`, uses Damerau-Levenshtein distance (counts transpositions, such as swapped adjacent letters, as a single edit). When `false`, uses classic Levenshtein distance.
To rank exact matches first while still surfacing fuzzy matches, keep `q` exact and add the fuzzy expression as a boost query instead: `bq=title_t:searchterm~1`.
## Neural Hybrid Search
Don't inject fuzzy tilde syntax or `{!fuzzy}` local params into the `q` parameter in a Neural Hybrid Search pipeline. NHS requires `q` to be a raw user query string; injecting Solr query parser syntax breaks vectorization. See [Neural Hybrid Search](/docs/5/fusion/hybrid-search/overview).
In an NHS environment:
* Rely on NHS's semantic vector matching, which already handles many misspelling-like cases through embedding proximity.
* Apply fuzzy matching at the Solr request handler level in `solrconfig.xml`, outside the NHS query path, if lexical fuzzy matching is still needed.
* Build a separate lexical-and-fuzzy fallback pipeline that runs only on zero-result queries.
Fusion's [Misspelling Detection](/docs/5/fusion/getting-data-out/query-enhancement/misspelling-detection) feature is deprecated as of Fusion 5.9.15. Lucidworks recommends Neural Hybrid Search as the replacement for spelling tolerance.
# Misspelling Detection
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/misspelling-detection
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/misspelling-detection
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/misspelling-detection
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/488
The Misspelling Detection feature maps misspellings to their corrected spellings. When Fusion receives a query containing a known misspelling, it rewrites the query using the corrected spelling in order to return relevant results instead of an empty or irrelevant results set.
This feature is deprecated in Fusion 5.9.15 and will be removed in a future release.
Lucidworks recommends migrating to [Neural Hybrid Search](/docs/5/fusion/hybrid-search/overview), which achieves superior relevance compared to legacy machine learning methods.
Spelling corrections are applied in the [Text Tagger stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/text-tagger-query-stage) of the query pipeline.
The [Token and Phrase Spell Correction job](/docs/5/fusion/reference/config-ref/jobs/token-and-phrase-spell-correction) automatically creates spelling corrections based on your AI-generated data. When you navigate to **Relevance** > **Rules** > **Rewrite** and select the **Misspelling** tab, the application displays the **Misspelling Detection** screen.
You can:
* **Use Misspelling Detection** the output from the job
* **Use Misspelling Detection**
* **Use Misspelling Detection** from the `_query_rewrite_staging` collection
The course for **Query Analytics** focuses on how Fusion provides query analytics to detect and improve underperforming queries.
## Reviewing auto-generated spelling corrections
Spelling corrections that are automatically generated by the [Token and Phrase Spell Correction job](/docs/5/fusion/reference/config-ref/jobs/token-and-phrase-spell-correction) are assigned one of the following status values:
* **Auto**
There are three values for confidence level:
| Value | Confidence | Label |
| - | - | - |
| 0 | low confidence | Pending |
| 0.5 | median confidence | Auto |
| 1 | high confidence | Auto |
No action is required on these results, but you can edit them if you wish.
* **Pending**
The confidence level is ambiguous, and the result must be reviewed by a user before it can be deployed. It will only be moved from the `_query_rewrite_staging` collection to the `_query_rewrite` collection when its status has changed to "Approved" *and* it has been published.
1. Navigate to **Relevance** > **Rules** > **Rewrite**.
2. Select **Misspelling** tab. The application displays the **Misspelling Detection** screen.
Notice the **Status** facet on the left. Click **Pending** to view only the items that need review.
3. Click the icon next to the spelling correction.
4. In the **Status** column, select either "Approved" or "Denied".
Optionally, you can also edit the spelling correction itself.
## Adding new spelling corrections
You can manually add spelling corrections in addition to any generated by the Fusion 5.x.x [Token and Phrase Spell Correction job](/docs/5/fusion/reference/config-ref/jobs/token-and-phrase-spell-correction) release.
1. Navigate to **Relevance** > **Rules** > **Rewrite**.
2. Select **Misspelling** tab. The application displays the **Misspelling Detection** screen.
3. At the bottom of the rules list, click the icon.
A new spelling correction appears at the top of the list:
4. Enter the misspelled word or phrase.
5. Enter one or more spelling corrections.
It is not necessary to set a confidence value.
6. Select the spelling correction’s status, depending on whether you want to deploy it the next time you publish your changes ("Approved") or save it for further review ("Pending").
7. Click the check mark to save the new spelling correction:
## Publishing your changes
**How to publish updated spelling corrections**
1. In the **Misspelling Detection** screen, click the **PUBLISH** button.
Fusion prompts you to confirm that you want to publish your changes.
2. Click **PUBLISH**.
You can un-publish a query rewrite by changing its status to "denied", then clicking **PUBLISH**.
## Tuning the misspelling detection job
The default configuration for the [Token and Phrase Spell Correction job](/docs/5/fusion/reference/config-ref/jobs/token-and-phrase-spell-correction), respectively, is designed for high accuracy and works well with most signal datasets, depending on the volume and quality of the signals. If you are seeing too few results, or too many inaccurate results, then you can try tuning the job to achieve better results.
To modify job configurations, you must be a Fusion user with one of the following roles or permissions that include access to job configurations:
* [admin](/docs/5/fusion/operations/security/access-control/roles)
* [developer](/docs/5/fusion/operations/security/access-control/roles)
* [permissions](/docs/5/fusion/operations/security/access-control/permissions)
## Query rewrite jobs post-processing cleanup
To perform more extensive cleanup of query rewrites, complete the procedures in Query Rewrite Jobs Post-processing Cleanup.
**LucidAcademy**
Lucidworks offers free training to help you get started.
The **Course** for **Query Analytics** focuses on how Fusion provides query analytics to detect and improve underperforming queries:
Visit the [LucidAcademy](https://academy.lucidworks.com) to see the full training catalog.
## Reviewing auto-generated spelling corrections
Spelling corrections that are automatically generated by the [Token and Phrase Spell Correction job](/docs/5/fusion/reference/config-ref/jobs/token-and-phrase-spell-correction) are assigned one of the following status values:
* **Auto**
There are three values for confidence level:
| Value | Confidence | Label |
| - | - | - |
| 0 | low confidence | Pending |
| 0.5 | median confidence | Auto |
| 1 | high confidence | Auto |
No action is required on these results, but you can edit them if you wish.
* **Pending**
The confidence level is ambiguous, and the result must be reviewed by a user before it can be deployed. It will only be moved from the `_query_rewrite_staging` collection to the `_query_rewrite` collection when its status has changed to "Approved" *and* it has been published.
1. Navigate to **Relevance** > **Rules** > **Rewrite**.
2. Select **Misspelling** tab. The application displays the **Misspelling Detection** screen.
Notice the **Status** facet on the left. Click **Pending** to view only the items that need review.
3. Click the icon next to the spelling correction.
4. In the **Status** column, select either "Approved" or "Denied".
Optionally, you can also edit the spelling correction itself.
## Adding new spelling corrections
You can manually add spelling corrections in addition to any generated by the Fusion 5.x.x [Token and Phrase Spell Correction job](/docs/5/fusion/reference/config-ref/jobs/token-and-phrase-spell-correction) release.
1. Navigate to **Relevance** > **Rules** > **Rewrite**.
2. Select **Misspelling** tab. The application displays the **Misspelling Detection** screen.
3. At the bottom of the rules list, click the icon.
A new spelling correction appears at the top of the list:
4. Enter the misspelled word or phrase.
5. Enter one or more spelling corrections.
It is not necessary to set a confidence value.
6. Select the spelling correction’s status, depending on whether you want to deploy it the next time you publish your changes ("Approved") or save it for further review ("Pending").
7. Click the check mark to save the new spelling correction:
## Publishing your changes
**How to publish updated spelling corrections**
1. In the **Misspelling Detection** screen, click the **PUBLISH** button.
Fusion prompts you to confirm that you want to publish your changes.
2. Click **PUBLISH**.
You can un-publish a query rewrite by changing its status to "denied", then clicking **PUBLISH**.
## Tuning the misspelling detection job
The default configuration for the [Token and Phrase Spell Correction job](/docs/5/fusion/reference/config-ref/jobs/token-and-phrase-spell-correction), respectively, is designed for high accuracy and works well with most signal datasets, depending on the volume and quality of the signals. If you are seeing too few results, or too many inaccurate results, then you can try tuning the job to achieve better results.
To modify job configurations, you must be a Fusion user with one of the following roles or permissions that include access to job configurations:
* [admin](/docs/5/fusion/operations/security/access-control/roles)
* [developer](/docs/5/fusion/operations/security/access-control/roles)
* [permissions](/docs/5/fusion/operations/security/access-control/permissions)
## Query rewrite jobs post-processing cleanup
To perform more extensive cleanup of query rewrites, complete the procedures in Query Rewrite Jobs Post-processing Cleanup.
**LucidAcademy**
Lucidworks offers free training to help you get started.
The **Course** for **Query Analytics** focuses on how Fusion provides query analytics to detect and improve underperforming queries:
Visit the [LucidAcademy](https://academy.lucidworks.com) to see the full training catalog.
## Reviewing auto-generated spelling corrections
Spelling corrections that are automatically generated by the [Token and Phrase Spell Correction job](/docs/5/fusion/reference/config-ref/jobs/token-and-phrase-spell-correction) are assigned one of the following status values:
* **Auto**
There are three values for confidence level:
| Value | Confidence | Label |
| - | - | - |
| 0 | low confidence | Pending |
| 0.5 | median confidence | Auto |
| 1 | high confidence | Auto |
No action is required on these results, but you can edit them if you wish.
* **Pending**
The confidence level is ambiguous, and the result must be reviewed by a user before it can be deployed. It will only be moved from the `_query_rewrite_staging` collection to the `_query_rewrite` collection when its status has changed to "Approved" *and* it has been published.
1. Navigate to **Relevance** > **Rules** > **Rewrite**.
2. Select **Misspelling** tab. The application displays the **Misspelling Detection** screen.
Notice the **Status** facet on the left. Click **Pending** to view only the items that need review.
3. Click the icon next to the spelling correction.
4. In the **Status** column, select either "Approved" or "Denied".
Optionally, you can also edit the spelling correction itself.
## Adding new spelling corrections
You can manually add spelling corrections in addition to any generated by the Fusion 5.x.x [Token and Phrase Spell Correction job](/docs/5/fusion/reference/config-ref/jobs/token-and-phrase-spell-correction) release.
1. Navigate to **Relevance** > **Rules** > **Rewrite**.
2. Select **Misspelling** tab. The application displays the **Misspelling Detection** screen.
3. At the bottom of the rules list, click the icon.
A new spelling correction appears at the top of the list:
4. Enter the misspelled word or phrase.
5. Enter one or more spelling corrections.
It is not necessary to set a confidence value.
6. Select the spelling correction’s status, depending on whether you want to deploy it the next time you publish your changes ("Approved") or save it for further review ("Pending").
7. Click the check mark to save the new spelling correction:
## Publishing your changes
**How to publish updated spelling corrections**
1. In the **Misspelling Detection** screen, click the **PUBLISH** button.
Fusion prompts you to confirm that you want to publish your changes.
2. Click **PUBLISH**.
You can un-publish a query rewrite by changing its status to "denied", then clicking **PUBLISH**.
## Tuning the misspelling detection job
The default configuration for the [Token and Phrase Spell Correction job](/docs/5/fusion/reference/config-ref/jobs/token-and-phrase-spell-correction), respectively, is designed for high accuracy and works well with most signal datasets, depending on the volume and quality of the signals. If you are seeing too few results, or too many inaccurate results, then you can try tuning the job to achieve better results.
To modify job configurations, you must be a Fusion user with one of the following roles or permissions that include access to job configurations:
* [admin](/docs/5/fusion/operations/security/access-control/roles)
* [developer](/docs/5/fusion/operations/security/access-control/roles)
* [permissions](/docs/5/fusion/operations/security/access-control/permissions)
## Query rewrite jobs post-processing cleanup
To perform more extensive cleanup of query rewrites, complete the procedures in Query Rewrite Jobs Post-processing Cleanup.
When you manually add new spelling corrections, subsequent job runs use those documents as input for machine learning to improve the job’s output. Unlike job-generated documents, manually-added query rewriting documents are never overwritten by new job output.
Misspelled terms are completely replaced by their corrected terms. If you want to expand the query to include all alternative terms, set the synonyms to bi-directional. See [Synonym Detection](/docs/5/fusion/reference/config-ref/jobs/synonym-detection) for more information.
To learn how to use misspelling detection in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor), see **Use Query Rewrites in the Rules Editor**.
This article describes how to use the Rewrites screen in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor).
## Query rewrite rule types
* **Head/Tail.** Replace underperforming queries with queries that produce higher click-through rates. Query improvements always require review before they are published. Schedule the Head/Tail Analysis job to get AI-generated query improvements based on your signals collection. \
\
Manually created Head/Tail rewrites must provide an "Improved Query" in a format appropriate to the type of request the rewrite will be applied to. For Legacy requests, this means a valid Solr query fit for use as the `q` parameter. For DSL requests this means a JSON string representing the main DSL query. \
\
For more information, see:
* [Underperforming Query Rewriting](/docs/5/fusion/getting-data-out/query-enhancement/underperforming-queries)
* [Head/Tail Analysis Jobs](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis)
* **Misspelling.** Fusion replaces misspelled query terms with their spelling corrections for better search results. Schedule the Token and Phrase Spell Correction job to get AI-generated spelling corrections based on your signals collection. For more information, see [Misspelling Detection](/docs/5/fusion/getting-data-out/query-enhancement/misspelling-detection).
* **Phrase.** Known phrases are boosted when they appear in new queries. Schedule the Phrase Extraction job to get AI-generated phrases based on your signals collection.
For more information, see [Phrase Detection](/docs/5/fusion/getting-data-out/query-enhancement/phrase-detection).
* **Synonym.** Synonyms are used to expand queries so that they include all known synonyms of the query terms. To retrieve AI-generated synonyms based on your signals collection, schedule the [Synonym Detection job](/docs/5/fusion/reference/config-ref/jobs/synonym-detection).
* **Remove Words** - Remove particular phrases from queries. Unlike other rewrites, these should be entered manually and are not generated by a job. For more information, see [Remove Words](/docs/5/fusion/getting-data-out/query-enhancement/remove-words).
## Facets
### Facet menu
To facet the rules list by **Status** or **Published**, click one of the facet items within the facet categories.
To facet the rules by **Confidence**, click and drag either slider to set the range of confidence you would like to see:
Faceting rules by confidence is not available for Head/Tail rewrites.
### Search
Enter a term in the search bar to quickly facet your query rewrite rules.
## Rules
Begin by selecting one of the query rewrite rule types you will create.
### Create a rule
1. Click **+Add**.
2. In the new table row, complete the required fields, and additional fields as needed.
3. Click the checkmark to save, or close to cancel the changes.
### Edit a rule
1. Hold the pointer over the rule to edit.
2. Click the pencil icon.
3. Enter desired changes and click the checkmark to save, or close to cancel the changes.
### Approve/deny rules
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To modify multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Approve** or **Deny** as needed.
### Publish a rule
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To publish multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Publish Selected** to publish the selected rules.
#### Publish an individual rule
1. Hold the pointer over a rule.
2. Click the button.
3. To confirm, click **Publish**.
### Delete rules
#### Delete a single rule
1. Hold the pointer over a rule.
2. Click the trash can.
3. To confirm, click **Delete**.
#### Delete multiple rules
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To delete multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Delete**.
5. To confirm, click **Delete** again.
### Sort rules
1. Hold the pointer over column titles. A icon displays for some titles to indicate the rules list can be sorted by that column.
2. Click the title column to sort the rules.
The column that is currently sorting the rules list is indicated by an or icon.
### Export rules
To export the rules list, click **Export**. This generates a `.csv` file that contains your rules.
## Columns
### Customize columns
To customize columns:
1. Hold the pointer over the row of column titles.
2. Click the button.
3. In the Customize Columns to Display screen, select the checkboxes for the columns you want to display. Clear the checkboxes for columns you do not want to display.
4. Click **Save**. To exit without saving your changes, click **Cancel**.
The course for **Query Analytics** focuses on how Fusion provides query analytics to detect and improve underperforming queries.
# Query Enhancement
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/3236
Query enhancement refers to features that improve search results by analyzing query intent, correcting malformed queries, boosting popular items, recommending items from related queries, and so on.
These topics show you how to use the basic query enhancement features:
* [Signals](/docs/5/fusion/getting-data-out/query-enhancement/signals/overview) are the foundation of many query enhancement strategies. Enabling and collecting signals allows you to enhance query results based on the past behavior of your users.
* Each Fusion app comes with a basic list of [stopwords](/docs/5/fusion/getting-data-out/query-enhancement/stopwords-files) that you can modify to suit your use case.
* [Boosting](/docs/5/fusion/getting-data-out/query-enhancement/boosting/overview) can be done manually or using signals to boost the most desirable items to the top of search results.
* [Synonyms](/docs/5/fusion/getting-data-out/query-enhancement/synonyms/overview) are alternative query terms used to expand the original query.
* [Business rules](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/overview) are a versatile strategy for manually rewriting queries to address a wide range of use cases.
* [Collapsing results](/docs/5/fusion/getting-data-out/query-enhancement/collapse) is a feature that groups all variations of each item into a single search result. This is useful for e-commerce sites where you want to show a single product with multiple SKUs or variations.
* [Misspelling detection](/docs/5/fusion/getting-data-out/query-enhancement/misspelling-detection) automatically rewrites misspelled queries to return relevant results.
* [Phrase detection](/docs/5/fusion/getting-data-out/query-enhancement/phrase-detection) automatically finds phrases so that results with matching phrases can be boosted.
* [Response rewriting](/docs/5/fusion/getting-data-out/query-enhancement/response-rewriting) modifies Solr’s response, rather than the user’s query.
* The [underperforming queries](/docs/5/fusion/getting-data-out/query-enhancement/underperforming-queries) feature identifies queries that can be rewritten to obtain more relevant results.
* [Recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/overview) can be signals-based or content-based, and items are recommended based on the current item, user, or query.
* [Predictive Merchandiser](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/overview) is a visual tool for manipulating search results to present the best items to e-commerce shoppers.
* [Experience Optimizer](/docs/5/fusion/getting-data-out/curating-your-search-experience/experience-optimizer/overview), similar to Predictive Merchandiser, is a visual tool for curating search results in a knowledge management environment.
Many of these are examples of [query rewriting](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting).
See also [Advanced Query Enhancement](/docs/5/fusion/getting-data-out/advanced-query-enhancement/overview) for even more sophisticated features designed for implementation by data scientists.
The course for **Using Advanced Query Tools in Fusion** focuses on how to apply parameters, LocalParams, function queries, and query re-ranking using the Fusion UI.
The learning path for **Query Fine-Tuning** focuses on best practices in tuning query relevancy to get the best results out of Fusion.
# Phrase Detection
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/phrase-detection
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/phrase-detection
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/phrase-detection
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/487
Phrase detection identifies phrases in your signals so that results with matching phrases can be boosted. This helps compensate for queries where phrases are not distinguished with quotation marks. For example, the query `ipad case` is rewritten as `“ipad case”~10^2`, meaning if `ipad` and `case` appear **within 10** terms (whitespace-delimited tokens) of each other, then boost the result by a factor of **2**.
This feature is deprecated in Fusion 5.9.15 and will be removed in a future release.
Lucidworks recommends migrating to [Neural Hybrid Search](/docs/5/fusion/hybrid-search/overview), which achieves superior relevance compared to legacy machine learning methods.
Phrases are applied in the [Text Tagger stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/text-tagger-query-stage) of the query pipeline.
For more information about how to use phrase detection in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor), see:
* **Use Query Rewrites in the Rules Editor**
* **Use Phrase Detection**
This article describes how to use the Rewrites screen in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor).
## Query rewrite rule types
* **Head/Tail.** Replace underperforming queries with queries that produce higher click-through rates. Query improvements always require review before they are published. Schedule the Head/Tail Analysis job to get AI-generated query improvements based on your signals collection. \
\
Manually created Head/Tail rewrites must provide an "Improved Query" in a format appropriate to the type of request the rewrite will be applied to. For Legacy requests, this means a valid Solr query fit for use as the `q` parameter. For DSL requests this means a JSON string representing the main DSL query. \
\
For more information, see:
* [Underperforming Query Rewriting](/docs/5/fusion/getting-data-out/query-enhancement/underperforming-queries)
* [Head/Tail Analysis Jobs](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis)
* **Misspelling.** Fusion replaces misspelled query terms with their spelling corrections for better search results. Schedule the Token and Phrase Spell Correction job to get AI-generated spelling corrections based on your signals collection. For more information, see [Misspelling Detection](/docs/5/fusion/getting-data-out/query-enhancement/misspelling-detection).
* **Phrase.** Known phrases are boosted when they appear in new queries. Schedule the Phrase Extraction job to get AI-generated phrases based on your signals collection.
For more information, see [Phrase Detection](/docs/5/fusion/getting-data-out/query-enhancement/phrase-detection).
* **Synonym.** Synonyms are used to expand queries so that they include all known synonyms of the query terms. To retrieve AI-generated synonyms based on your signals collection, schedule the [Synonym Detection job](/docs/5/fusion/reference/config-ref/jobs/synonym-detection).
* **Remove Words** - Remove particular phrases from queries. Unlike other rewrites, these should be entered manually and are not generated by a job. For more information, see [Remove Words](/docs/5/fusion/getting-data-out/query-enhancement/remove-words).
## Facets
### Facet menu
To facet the rules list by **Status** or **Published**, click one of the facet items within the facet categories.
To facet the rules by **Confidence**, click and drag either slider to set the range of confidence you would like to see:
Faceting rules by confidence is not available for Head/Tail rewrites.
### Search
Enter a term in the search bar to quickly facet your query rewrite rules.
## Rules
Begin by selecting one of the query rewrite rule types you will create.
### Create a rule
1. Click **+Add**.
2. In the new table row, complete the required fields, and additional fields as needed.
3. Click the checkmark to save, or close to cancel the changes.
### Edit a rule
1. Hold the pointer over the rule to edit.
2. Click the pencil icon.
3. Enter desired changes and click the checkmark to save, or close to cancel the changes.
### Approve/deny rules
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To modify multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Approve** or **Deny** as needed.
### Publish a rule
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To publish multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Publish Selected** to publish the selected rules.
#### Publish an individual rule
1. Hold the pointer over a rule.
2. Click the button.
3. To confirm, click **Publish**.
### Delete rules
#### Delete a single rule
1. Hold the pointer over a rule.
2. Click the trash can.
3. To confirm, click **Delete**.
#### Delete multiple rules
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To delete multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Delete**.
5. To confirm, click **Delete** again.
### Sort rules
1. Hold the pointer over column titles. A icon displays for some titles to indicate the rules list can be sorted by that column.
2. Click the title column to sort the rules.
The column that is currently sorting the rules list is indicated by an or icon.
### Export rules
To export the rules list, click **Export**. This generates a `.csv` file that contains your rules.
## Columns
### Customize columns
To customize columns:
1. Hold the pointer over the row of column titles.
2. Click the button.
3. In the Customize Columns to Display screen, select the checkboxes for the columns you want to display. Clear the checkboxes for columns you do not want to display.
4. Click **Save**. To exit without saving your changes, click **Cancel**.
The [Phrase Extraction job](/docs/5/fusion/reference/config-ref/jobs/phrase-extraction) automatically creates phrases based on your AI-generated data. When you navigate to **Relevance** > **Rules** > **Rewrite** and select the **Phrase** tab, the application displays the **Phrase Detection** screen.
When you manually add new phrases, subsequent job runs use those documents as input for machine learning to improve the job’s output. Unlike job-generated documents, manually-added query rewriting documents are never overwritten by new job output.
The course for **Query Analytics** focuses on how Fusion provides query analytics to detect and improve underperforming queries.
## Reviewing auto-generated phrases
Phrases that are automatically generated by the [Phrase Extraction job](/docs/5/fusion/reference/config-ref/jobs/phrase-extraction) are assigned one of the following status values:
* **Auto**
These results have a confidence level as a threshold to automatically deploy them to the `_query_rewrite` collection. This threshold can be specified in the configuration parameter **Minimum Likelihood Score** (default value 0.1).
No action is required on these results, but you can edit them if you wish.
* **Pending**
The confidence level is ambiguous, and the result must be reviewed by a user before it can be deployed. It will only be moved from the `_query_rewrite_staging` collection to the `_query_rewrite` collection when its status has changed to "Approved" *and* it has been published.
### How to review a pending phrase result
1. Navigate to **Relevance** > **Rules** > **Rewrite**.
2. Select **Phrase** tab. The application displays the **Phrase Detection** screen.
Notice the **Status** facet on the left. Click **Pending** to view only the items that need review.
3. Click the icon next to the phrase.
4. In the **Status** column, select either "Approved" or "Denied".
Optionally, you can also edit the phrase itself.
Although the Confidence field is also editable, changing its value makes no difference.
5. Click the **Close** icon next to the updated phrase:
Approving a phrase does not automatically deploy it to the `_query_rewrite` collection. When you have finished your review, you must click **Publish** to deploy your changes.
## Adding new phrases
You can manually add phrases in addition to any generated by the [Phrase Extraction job](/docs/5/fusion/reference/config-ref/jobs/phrase-extraction).
### How to add a phrase
1. Navigate to **Relevance** > **Rules** > **Rewrite**.
2. Select **Phrase** tab. The application displays the **Phrase Detection** screen.
3. At the bottom of the rules list, click the icon. A new phrase appears at the top of the list.
4. Enter the phrase.
It is not necessary to set a confidence value.
5. Select the phrase’s status, depending on whether you want to deploy it the next time you publish your changes ("Approved") or save it for further review ("Pending").
6. Click the check mark to save the new phrase.
## Publishing your changes
### How to publish updated phrases
1. In the **Phrase Detection** screen, click the **PUBLISH** button.
Fusion prompts you to confirm that you want to publish your changes.
2. Click **PUBLISH**.
You can un-publish a query rewrite by changing its status to "denied", then clicking **PUBLISH**.
The course for **Query Analytics** focuses on how Fusion provides query analytics to detect and improve underperforming queries.
# Query Rewriting
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/489
Query rewriting is a strategy for improving relevancy using AI-generated data. Many of Fusion’s features can be used to rewrite incoming queries prior to submitting them to Fusion’s Solr core. These rewrites produce more relevant search results with higher conversion rates.
For example, when spelling corrections are used for query rewriting, a misspelled query can return the same search results as a correctly-spelled query, instead of returning irrelevant results or no results. Spelling corrections are one of several available [query rewriting strategies](#query-rewriting-strategies). Apply all available strategies for best results.
**See also the [Query Rewriting API](/api-reference/query-rewrite-api/get-the-query-rewrite-status).**
Fusion can also rewrite Solr’s responses before returning them to the search application; see [Response Rewriting](/docs/5/fusion/getting-data-out/query-enhancement/response-rewriting).
## Query rewriting strategies
Fusion provides a variety of query rewriting strategies to improve relevancy:
* [Business rules](#business-rules)
* [Underperforming query rewriting](#underperforming-query-rewriting)
* [Misspelling detection](#misspelling-detection)
* [Phrase detection](#phrase-detection)
* [Synonym detection](#synonym-detection)
* [Remove Words](#remove-words)
With the exception of business rules, which are always manually created, these strategies correspond to certain [Spark jobs](#spark-jobs-for-query-rewriting). Lucidworks recommends configuring and scheduling all of these jobs for best results. You can also train the jobs by manually adding documents to their output. Manually-added documents are used for machine learning and are never overwritten by new job output.
In Fusion 5.9.10 and higher, [Neural Hybrid Search](/docs/5/fusion/hybrid-search/overview) provides superior relevance over Fusion's query rewriting Spark jobs.
Query rewriting strategies are applied in the following order:
1. **Business rules** - If a query triggers a business rule, then the business rule overrides any query rewriting strategies that conflict with it.
2. **Query rewrites**
1. **Underperforming query rewriting** - If a query triggers an underperforming query rewrite, then this strategy overrides all subsequent query rewriting strategies.
2. **Remove words** - To help increase the number of results returned, Remove Words query rewrites removes words from the users' search.
3. **Synonym detection**
4. **Misspelling detection *and* phrase detection** - The query rewriting results from both of these strategies are applied together. To use only the strategy with the longest surface form, you can configure the Text Tagger query stage with **Overlapping Tag Policy** set to "LONGEST\_DOMINANT\_RIGHT".
### Business rules
Business rules are manually-created formulas for rewriting queries. This is the most versatile strategy for creating custom query rewrites. It supports a variety of conditions and actions to address a wide range of use cases. When you need a very specific query rewrite, this is the best strategy.
Business rules are applied in the [Apply Rules stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/query-rules-query-stage) of the query pipeline.
See [Business Rules](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/overview) to learn how to create, edit, and publish business rules.
### Underperforming query rewriting
Head/tail analysis is:
* Also known as the Underperforming Query Rewriting feature
* Uses signals data to identify underperforming queries
* Suggests improved queries that could produce better conversion rates
When an incoming query contains a matching underperforming query, the original query is replaced by an improved query.
These improvements can be:
* Suggested by the [Head/Tail Analysis Spark job](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis) operating on your signals data
This job is deprecated in Fusion 5.9.15 and will be removed in a future release.
Lucidworks recommends using [Neural Hybrid Search](/docs/5/fusion/hybrid-search/overview), which achieves superior relevance compared to legacy machine learning methods.
* Created manually using the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor) or underlying API
Query improvements are applied in the [Text Tagger stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/text-tagger-query-stage) of the query pipeline.
See [Head/Tail Analysis (Underperforming Query Rewriting)](/docs/5/fusion/getting-data-out/query-enhancement/underperforming-queries) to learn how to review, edit, create, and publish query improvements.
### Misspelling detection
The Misspelling Detection feature maps misspellings to their corrected spellings. When Fusion receives a query containing a known misspelling, it rewrites the query using the corrected spelling in order to return relevant results instead of an empty or irrelevant results set.
The Misspelling Detection job is deprecated in Fusion 5.9.15 and will be removed in a future release.
Lucidworks recommends using [Neural Hybrid Search](/docs/5/fusion/hybrid-search/overview), which achieves superior relevance compared to legacy machine learning methods.
Spelling corrections are applied in the [Text Tagger stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/text-tagger-query-stage) of the query pipeline.
Misspelled terms are completely replaced by their corrected terms. If you want to expand the query to include all alternative terms, set the synonyms to bi-directional. See [Synonym Detection](/docs/5/fusion/reference/config-ref/jobs/synonym-detection) for more information.
See:
* [Misspelling Detection](/docs/5/fusion/getting-data-out/query-enhancement/misspelling-detection) for general information
* **Use Misspelling Detection** for information about how to review, edit, create, and publish spelling corrections
The course for **Query Analytics** focuses on how Fusion provides query analytics to detect and improve underperforming queries.
## Reviewing auto-generated spelling corrections
Spelling corrections that are automatically generated by the [Token and Phrase Spell Correction job](/docs/5/fusion/reference/config-ref/jobs/token-and-phrase-spell-correction) are assigned one of the following status values:
* **Auto**
There are three values for confidence level:
| Value | Confidence | Label |
| - | - | - |
| 0 | low confidence | Pending |
| 0.5 | median confidence | Auto |
| 1 | high confidence | Auto |
No action is required on these results, but you can edit them if you wish.
* **Pending**
The confidence level is ambiguous, and the result must be reviewed by a user before it can be deployed. It will only be moved from the `_query_rewrite_staging` collection to the `_query_rewrite` collection when its status has changed to "Approved" *and* it has been published.
1. Navigate to **Relevance** > **Rules** > **Rewrite**.
2. Select **Misspelling** tab. The application displays the **Misspelling Detection** screen.
Notice the **Status** facet on the left. Click **Pending** to view only the items that need review.
3. Click the icon next to the spelling correction.
4. In the **Status** column, select either "Approved" or "Denied".
Optionally, you can also edit the spelling correction itself.
## Adding new spelling corrections
You can manually add spelling corrections in addition to any generated by the Fusion 5.x.x [Token and Phrase Spell Correction job](/docs/5/fusion/reference/config-ref/jobs/token-and-phrase-spell-correction) release.
1. Navigate to **Relevance** > **Rules** > **Rewrite**.
2. Select **Misspelling** tab. The application displays the **Misspelling Detection** screen.
3. At the bottom of the rules list, click the icon.
A new spelling correction appears at the top of the list:
4. Enter the misspelled word or phrase.
5. Enter one or more spelling corrections.
It is not necessary to set a confidence value.
6. Select the spelling correction’s status, depending on whether you want to deploy it the next time you publish your changes ("Approved") or save it for further review ("Pending").
7. Click the check mark to save the new spelling correction:
## Publishing your changes
**How to publish updated spelling corrections**
1. In the **Misspelling Detection** screen, click the **PUBLISH** button.
Fusion prompts you to confirm that you want to publish your changes.
2. Click **PUBLISH**.
You can un-publish a query rewrite by changing its status to "denied", then clicking **PUBLISH**.
## Tuning the misspelling detection job
The default configuration for the [Token and Phrase Spell Correction job](/docs/5/fusion/reference/config-ref/jobs/token-and-phrase-spell-correction), respectively, is designed for high accuracy and works well with most signal datasets, depending on the volume and quality of the signals. If you are seeing too few results, or too many inaccurate results, then you can try tuning the job to achieve better results.
To modify job configurations, you must be a Fusion user with one of the following roles or permissions that include access to job configurations:
* [admin](/docs/5/fusion/operations/security/access-control/roles)
* [developer](/docs/5/fusion/operations/security/access-control/roles)
* [permissions](/docs/5/fusion/operations/security/access-control/permissions)
## Query rewrite jobs post-processing cleanup
To perform more extensive cleanup of query rewrites, complete the procedures in Query Rewrite Jobs Post-processing Cleanup.
### Phrase detection
Phrase detection identifies phrases in your signals so that results with matching phrases can be boosted. This helps compensate for queries where phrases are not distinguished with quotation marks. For example, the query `ipad case` is rewritten as `“ipad case”~10^2`, meaning if `ipad` and `case` appear **within 10** terms (whitespace-delimited tokens) of each other, then boost the result by a factor of **2**.
The Phrase Detection job is deprecated in Fusion 5.9.15 and will be removed in a future release.
Lucidworks recommends using [Neural Hybrid Search](/docs/5/fusion/hybrid-search/overview), which achieves superior relevance compared to legacy machine learning methods.
Phrases are applied in the [Text Tagger stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/text-tagger-query-stage) of the query pipeline.
See:
* [Phrase Detection](/docs/5/fusion/getting-data-out/query-enhancement/phrase-detection) for general information
* **Use Phrase Detection** for information about how to review, edit, create, and publish spelling corrections
The [Phrase Extraction job](/docs/5/fusion/reference/config-ref/jobs/phrase-extraction) automatically creates phrases based on your AI-generated data. When you navigate to **Relevance** > **Rules** > **Rewrite** and select the **Phrase** tab, the application displays the **Phrase Detection** screen.
When you manually add new phrases, subsequent job runs use those documents as input for machine learning to improve the job’s output. Unlike job-generated documents, manually-added query rewriting documents are never overwritten by new job output.
The course for **Query Analytics** focuses on how Fusion provides query analytics to detect and improve underperforming queries.
## Reviewing auto-generated phrases
Phrases that are automatically generated by the [Phrase Extraction job](/docs/5/fusion/reference/config-ref/jobs/phrase-extraction) are assigned one of the following status values:
* **Auto**
These results have a confidence level as a threshold to automatically deploy them to the `_query_rewrite` collection. This threshold can be specified in the configuration parameter **Minimum Likelihood Score** (default value 0.1).
No action is required on these results, but you can edit them if you wish.
* **Pending**
The confidence level is ambiguous, and the result must be reviewed by a user before it can be deployed. It will only be moved from the `_query_rewrite_staging` collection to the `_query_rewrite` collection when its status has changed to "Approved" *and* it has been published.
### How to review a pending phrase result
1. Navigate to **Relevance** > **Rules** > **Rewrite**.
2. Select **Phrase** tab. The application displays the **Phrase Detection** screen.
Notice the **Status** facet on the left. Click **Pending** to view only the items that need review.
3. Click the icon next to the phrase.
4. In the **Status** column, select either "Approved" or "Denied".
Optionally, you can also edit the phrase itself.
Although the Confidence field is also editable, changing its value makes no difference.
5. Click the **Close** icon next to the updated phrase:
Approving a phrase does not automatically deploy it to the `_query_rewrite` collection. When you have finished your review, you must click **Publish** to deploy your changes.
## Adding new phrases
You can manually add phrases in addition to any generated by the [Phrase Extraction job](/docs/5/fusion/reference/config-ref/jobs/phrase-extraction).
### How to add a phrase
1. Navigate to **Relevance** > **Rules** > **Rewrite**.
2. Select **Phrase** tab. The application displays the **Phrase Detection** screen.
3. At the bottom of the rules list, click the icon. A new phrase appears at the top of the list.
4. Enter the phrase.
It is not necessary to set a confidence value.
5. Select the phrase’s status, depending on whether you want to deploy it the next time you publish your changes ("Approved") or save it for further review ("Pending").
6. Click the check mark to save the new phrase.
## Publishing your changes
### How to publish updated phrases
1. In the **Phrase Detection** screen, click the **PUBLISH** button.
Fusion prompts you to confirm that you want to publish your changes.
2. Click **PUBLISH**.
You can un-publish a query rewrite by changing its status to "denied", then clicking **PUBLISH**.
### Synonym detection
The Synonym Detection feature generates pairs of synonyms and pairs of similar queries. Two words are considered potential synonyms when they are used in a similar context in similar queries. A query that contains a matching term is expanded to include all of its synonyms, with the original term boosted by a factor of two.
Synonyms are applied in the [Text Tagger stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/text-tagger-query-stage) of the query pipeline.
The Synonym Detection job is deprecated in Fusion 5.9.15 and will be removed in a future release.
Lucidworks recommends using [Neural Hybrid Search](/docs/5/fusion/hybrid-search/overview), which achieves superior relevance compared to legacy machine learning methods.
See:
* [Synonym Detection](/docs/5/fusion/reference/config-ref/jobs/synonym-detection) for general information
* **Use Synonym Detection** for information about how to review, edit, create, and publish synonym corrections
Based on the release, synonyms are automatically created based on your AI-generated data. When you navigate to **Relevance** > **Rules** > **Rewrite** and select the **Synonym** tab, the application displays the **Synonym Detection** screen. For more information, see:
When you manually add new synonym pairs, subsequent job runs use those documents as input for machine learning to improve the job’s output. Unlike job-generated documents, manually-added query rewriting documents are never overwritten by new job output.
The course for **Query Analytics** focuses on how Fusion provides query analytics to detect and improve underperforming queries.
## Reviewing auto-generated synonym pairs
Synonyms that are automatically generated by the [Fusion 5.2.x and later Synonym Detection job](/docs/5/fusion/reference/config-ref/jobs/synonym-detection) are assigned the following status value:
* **Pending**
The confidence level is ambiguous, and the result must be reviewed by a user before it can be deployed. It will only be moved from the `_query_rewrite_staging` collection to the `_query_rewrite` collection when its status has changed to "Approved" *and* it has been published.
By default, all results from a synonym job are set to "Pending", since there are usually a limited number of synonyms, and synonym expansion can have high impact on relevancy.
### How to review a pending synonym pair result
1. Navigate to **Relevance** > **Rules** > **Rewrite**.
2. Select **Synonym** tab. The application displays the **Synonym Detection** screen.
Notice the **Status** facet on the left. Click **Pending** to view only the items that need review.
3. Click the icon next to the synonym pair.
4. In the **Status** column, select either "Approved" or "Denied".
Where alternative synonyms were detected, you can click **Suggestions** to view and select them as replacements for the displayed synonym pair.
Although the Confidence field is also editable, changing its value makes no difference.
5. Click the **Close** icon next to the updated synonym pair:
Approving a synonym pair does not automatically deploy it to the `_query_rewrite` collection. When you have finished your review, you must click **Publish** to deploy your changes.
## Adding new synonym pairs
You can manually add synonym pairs in addition to any generated by your Fusion release:
* [Fusion 5.2.x and later Synonym Detection job](/docs/5/fusion/reference/config-ref/jobs/synonym-detection)
### How to add a synonym pair
1. Navigate to **Relevance** > **Rules** > **Rewrite**.
2. Select **Synonym** tab. The application displays the **Synonym Detection** screen.
3. At the bottom of the rules list, click the icon. A new synonym pair appears at the top of the list.
4. Enter the query term.
It is not necessary to set a confidence value.
5. Select the synonym pair’s status, depending on whether you want to deploy it the next time you publish your changes ("Approved") or save it for further review ("Pending").
6. Click the check mark to save the new synonym pair.
## Publishing your changes
### How to publish updated synonym pairs
1. In the **Synonym Detection** screen, click the **PUBLISH** button.
Fusion prompts you to confirm that you want to publish your changes.
2. Click **PUBLISH**.
You can un-publish a query rewrite by changing its status to "denied", then clicking **PUBLISH**.
### Remove words
Use a Remove Words query rewrite to remove particular phrases from queries. Unlike other rewrites, Remove Words rules are entered manually and aren’t generated by a job.
You can create a Remove Words query rewrite to remove words from a query. This query rewrite is helpful when a word in the search query does not add value to the search results.
For example, you can rewrite a search query for `case study examples` to remove `examples` and then display results for `case study`.
See [Remove Words](/docs/5/fusion/getting-data-out/query-enhancement/remove-words) to learn how to remove words from your users' searches.
## Rules Editor
The Rules Editor allows you to view, edit, create, approve, enable, and publish rules powered by Fusion. Access the Rules Editor from the Fusion UI by navigating to **Relevance** > **Rules**:
Read more about the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor).
## Query rewrite collections
For detailed information about query rewriting, see:
* **Manage Collections in the Fusion UI**
* [Collections](/docs/5/fusion/getting-data-in/indexing/collections/overview)
Collections can be created or removed using the Fusion UI or the REST API.
For information about using the REST API to manage collections, see [Collections API](/api-reference/collections/get-collections-service-status) in the REST API Reference:
## Creating a Collection
When you create an app, by default Fusion Server creates a collection and associated objects.
To create a new collection in the Fusion UI:
1. From within an app, click **Collections > Collections Manager**.
2. At the upper right of the panel, click **New**.
3. Enter a **Collection name**. This name cannot be changed later.
4. To create the collection in the default Solr cluster and with other default settings, click **Save Collection**.
## Creating a Collection with Advanced Options
To access advanced options for creating a collection in the Fusion UI:
1. From within an app, click **Collections > Collections Manager**.
2. At the upper right of the panel, click **New**.
3. Enter a **Collection name**. This name cannot be changed later.
4. Click **Advanced**.
5. Configure advanced options. The options are described below.
6. Click **Save Collection**.
### Solr Cluster
By default, a new collection is associated with the Solr instance that is associated with the `default` Solr cluster.
If Fusion has multiple Solr clusters, choose from the list which cluster you want to associate your collection with.
The cluster must exist first.
### Solr Cluster Layout
The next section lets you define a **Replication Factor** and **Number of Shards**.
Define these options only if you are creating a new collection in the Solr cluster.
If you are linking Fusion to an existing Solr collection, you can skip these settings.
### Solr Collection Import
Import a Solr collection to associate the new Fusion collection with an existing Solr collection.
Enter a **Solr Collection Name** to associate the collection with an existing Solr collection.
Then, enter a **Solr Config Set** to tell ZooKeeper to use the configurations from an existing collection in Solr when creating this collection.
### Time Series Partitioning
Available in 4.x only.
You can map a Fusion collection to multiple Solr collections, known here as partitions, where each partition contains data from a specific time range.
To configure time-based partitioning, under **Time Series Partitioning** click **Enable**.
See [Time-Based Partitioning](/docs/4/fusion-server/concepts/indexing/collections/time-based-partitioning) for more information.
## Configuring Collections
The Collections menu lets you configure your existing collection, including datasources, fields, jobs, stopwords, and synonyms.
In the Fusion UI, from any app, the Collections icon displays on the left side of the screen.
Some tasks related to managing a collection are available in other menus:
* Configure a profile in **Indexing > Indexing Profiles** or **Querying > Query Profiles**.
* View reports about your collection’s activity in **Analytics > Dashboards**.
### Collections Manager
The Collections Manager page displays details about the collection, such as how many datasources are configured, how many documents are in the index, and how much disk space the index consumes.
This page also lets you [create a new collection](#creating-a-collection), disable search logs or signals, enable recommendations, issue a commit command to Solr, or clear a collection.
#### Disable search logs
When you first create a collection, the search logs are created by default. The search logs populate the panels in **Analytics > Dashboards**.
1. Hover over your collection name until the gear icon appears at the end of the line.
2. Click the gear icon.
3. Click **Disable Search Logs**.
4. On the confirmation screen, click **Disable Search Logs**.
Note that if you disable search logs, you cannot see any data for this collection in **Analytics > Dashboards**.
See Dashboards:
* **Fusion 5.x.** [Dashboards](/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/overview)
* **Fusion 4.x.** [Dashboards](/docs/4/fusion-server/concepts/dashboards/overview)
#### Disable signals
When you first create a collection, the signals and aggregated signals collections are created by default.
1. Hover over your collection name until the gear icon appears at the end of the line.
2. Click the gear icon.
3. Click **Disable Signals**.
4. On the confirmation screen, click **Disable Signals**.
#### Hard commit a collection
1. Hover over your collection name until the gear icon appears at the end of the line.
2. Click the gear icon.
3. Click **Hard Commit Collection**.
4. On the confirmation screen, click **Hard Commit Collection**.
Read internal details about how Solr processes commits on [our blog](https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/).
### Datasources
To access the Datasources page, click **Indexing > Datasources**. By default, there are no datasources configured right after installation.
To add a new datasource, click **New** at the upper right of the panel.
See the [Connectors and Datasources Reference](/docs/fusion-connectors/connectors/overview) for details on how to configure a datasource. Options vary depending on the repository you would like to index.
After you configure a datasource, it appears in a list on this screen. Click the name of a datasource to edit its properties. Click **Start** to start the datasource. Click **Stop** to stop the datasource before it completes.
To the right, view information on the last completed job, including the date and time started and stopped, and the number of documents found as new, skipped, or failed.
When you stop a datasource, Fusion attempts to safely close connector threads, finishing processing documents through the pipeline and indexing documents to Solr. Some connectors take longer to complete these processes than others, so might stay in a "stopping" state for several minutes.
To stop a datasource immediately, choose **Abort** instead of **Stop**.
There is also a [REST API for datasources](/api-reference/datasource-configuration-v2-api/list-all-v2-datasources).
### Stopwords
The Stopwords page lets you edit a stopwords list for your collection.
To add or delete stop words:
1. Click the name of the text file you wish to edit.
2. Add a new word on a new line.
3. When you are done with your changes, click **Save**.
To import a stop words list:
1. Click **System > Import Fusion Objects**.
2. Choose the file to upload.
3. Click **Import >>**.
Read more about stopwords:
* **Fusion 5.x.** [stopwords](/docs/5/fusion/getting-data-out/query-enhancement/stopwords-files)
* **Fusion 4.x.** [stopwords](/docs/4/fusion-server/concepts/querying/apps/stopwords-files)
### Synonyms
Fusion has the same synonym functionality that Solr supports. This includes a list of words that are synonyms (where the synonym list expands on the terms entered by the user), as well as a full mapping of words, where a word is substituted for what the user has entered (that is, the term the user has entered is replaced by a term in the synonym list).
See more about synonyms:
* **Fusion 5.x.** [synonyms](/docs/5/fusion/getting-data-out/query-enhancement/synonyms/overview)
* **Fusion 4.x.** [synonyms](/docs/4/fusion-server/concepts/querying/apps/synonyms-files)
You can edit the synonyms list for your collection.
To access the Synonyms page in the Fusion UI, in any app, click **Collections > Synonyms**.
Filter the list of synonym definitions by typing in the **Filter...** box.
To import a synonyms list:
1. From the Synonyms page, click **Import and Save**. A dialog box opens.
2. Choose the file to import.
To edit a synonyms list:
* Enter new synonym definitions one per line.
* To enter a string of terms that expand on the terms the user entered, enter the terms separated by commas, like `Television, TV`.
* To enter a term that should be mapped to another term, enter the terms separated by an equal sign then a right angle bracket, `=>`, like `i-pod=>ipod`.
* Remove a line by clicking the **x** at the end of the line.
* Once you are finished with edits, click **Save**.
To export the synonyms list, click **Export**. This downloads the list to your computer using your browser download capability.
### Profiles
Profiles allow you to create an alias for an index or query pipeline.
This allows you to send documents or queries to a consistent endpoint and change the underlying pipeline or collection as needed.
Read about profiles in Index Profiles and Query Profiles:
* **Fusion 5.x.**
* [Index Profiles](/docs/4/fusion-server/concepts/indexing/datasources/index-profiles)
* [Query Profiles](/docs/4/fusion-server/concepts/querying/pipelines/query-profiles)
* **Fusion 4.x.**
* [Index Profiles](/docs/4/fusion-server/concepts/indexing/datasources/index-profiles)
* [Query Profiles](/docs/4/fusion-server/concepts/querying/pipelines/query-profiles)
To access the Solr Config page, from any app, click **System > Solr Config**.
## Learn more
The quick learning for **Collections Menu Tour** focuses on the Collections Menu features and functionality along with a brief description of each screen available in the menu.
For each app, two [auxiliary collections](/docs/5/fusion/getting-data-in/indexing/collections/overview) are dedicated to documents used for query rewriting:
* `COLLECTION_NAME_query_rewrite_staging`\
[Certain Spark jobs](#spark-jobs-for-query-rewriting) send their output to this collection. [Rules](#business-rules) are also written to this collection initially\
Some of the content in this collection requires manual review before it can be migrated to the `COLLECTION_NAME_query_rewrite`, where query pipelines can read it. See below for details.
* `COLLECTION_NAME_query_rewrite`
This collection is optimized for high-volume traffic. Query pipelines can read from this collection to find rules, synonyms, spelling corrections, and more with which to rewrite queries and responses.
Each app contains exactly one of each of these collections, associated with the app’s default collection. They are not created again for additional collections created within the same app.
Documents move from `COLLECTION_NAME_query_rewrite_staging` to the `COLLECTION_NAME_query_rewrite` collection only when they are approved (either automatically on the basis of their confidence scores or manually by a human reviewer) *and* a Fusion user clicks **Publish**. The `review` field value indicates whether a document will be published when the user clicks **Publish**:
| | |
| - | - |
| `review=auto` | A job-generated document has a sufficiently high confidence score and is automatically approved for publication. |
| `review=pending` | A job-generated document has an ambiguous confidence score and must be reviewed by a Fusion user. |
| `review=approved` | A Fusion user has reviewed the document and approved it for publication. |
| `review=denied` | A job-generated document has a low confidence score, or a Fusion user has reviewed and denied it for publication. |
In the query rewriting UI, the value of the `review` field appears in the **Status** column.
You can review and approve or deny documents using the query rewriting UI. You can also change a document’s status to "pending" to save it for later review.
## Rules Simulator query profile
**Important**
Rules Simulator is only available for Fusion 5.x.x organizations that do *not* have a valid Predictive Merchandiser or Experience Optimizer license.
The [Rules Simulator](/docs/4/fusion-ai/concepts/query-rewriting/simulator) allows product owners to experiment with rules and other query rewrites in the `COLLECTION_NAME_query_rewrite_staging` collection before deploying them to the `COLLECTION_NAME_query_rewrite` collection.
Each app has a `COLLECTION_NAME_rules_simulator` query profile, configured to use the `COLLECTION_NAME_query_rewrite_staging` collection for query rewrites instead of the `COLLECTION_NAME_query_rewrite` collection. This profile is created automatically whenever a new app is created.
See **Configure the Rules Simulator Query Profile** for more information about configuration.
Each app has a `_rules_simulator` query profile, configured to use the `_query_rewrite_staging` collection for [query rewrites](/docs/4/fusion-ai/concepts/query-rewriting/overview) instead of the `_query_rewrite` collection. This profile is created automatically whenever a new app is created.
By default, this query profile points to your default query pipeline and collection. You can configure it to point to any pipeline or collection, for example when testing a new pipeline before it has been deployed.
**How to change the query pipeline, collection, and query parameters used by the `_rules_simulator` query profile**
1. Open the Fusion UI.
2. Navigate to **Querying** > **Query Profiles**.
3. Select the `_rules_simulator` query profile for your app.
For example, if your app is called "Demo" then the name of the query profile is `Demo_rules_simulator`.
4. Modify the configuration as desired.
5. Click **Save**.
## Query pipeline stages for query rewriting
These query rewriting stages are part of any default query pipeline:
* [Apply Rules query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/query-rules-query-stage)\
This stage looks up rules that have been deployed to the [`COLLECTION_NAME_query_rewrite` collection](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) and matches them against the query. Matching rules that perform [query rewriting](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) are applied at this stage, while matching rules that perform [response rewriting](/docs/5/fusion/getting-data-out/query-enhancement/response-rewriting) are applied by the [Modify Response with Rules](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/rules-augment-response-query-stage) stage later in the pipeline.
To trigger a rule that contains a tag, specify the tagname in the request URL of the user search app. See [Easily define triggers in tags](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/general-config) for more information.
* [Text Tagger query pipeline stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/text-tagger-query-stage)
This stage uses the [SolrTextTagger](https://solr.apache.org/guide/solr/latest/query-guide/tagger-handler.html) handler to identify known entities in the query by searching the [`COLLECTION_NAME_query_rewrite` collection](/docs/5/fusion/getting-data-in/indexing/collections/overview).
For Fusion 5.x.x organizations that do *not* have a Predictive Merchandiser license, the Solr Text Tagger handler also searches the `COLLECTION_NAME_query_rewrite_staging` collection in the case of the Fusion query rewriting [Simulator](/docs/4/fusion-ai/concepts/query-rewriting/simulator)).
The purpose of the search is to perform query rewriting using matches from the following items:
* [Spelling corrections](/docs/5/fusion/getting-data-out/query-enhancement/misspelling-detection)
* [Phrase boosts](/docs/5/fusion/getting-data-out/query-enhancement/phrase-detection)
* [Head/Tail Analysis (Underperforming query improvements)](/docs/5/fusion/getting-data-out/query-enhancement/underperforming-queries)
* [Synonym Detection](/docs/5/fusion/reference/config-ref/jobs/synonym-detection)
## Spark jobs for query rewriting
This section describes how Spark jobs support query rewriting. These jobs read from the signals collection and write their output to the `COLLECTION_NAME_query_rewrite_staging` collection. High-confidence results are automatically migrated from there to the `COLLECTION_NAME_query_rewrite` collection, while ambiguous results remain in the staging collection until they are reviewed and approved. You can review job results in the Query Rewriting UI.
* [Daily query rewriting jobs](#daily-query-rewriting-jobs) are created and scheduled automatically when you create a new app.
* [Additional query rewriting jobs](#additional-query-rewriting-jobs) can be created manually.
For best relevancy, enable all of these jobs.
### Daily query rewriting jobs
When a new app is created, the jobs below are also created and scheduled to run daily, beginning 15 minutes after app creation, in the following order:
1. [Token and Phrase Spell Correction job](/docs/5/fusion/reference/config-ref/jobs/token-and-phrase-spell-correction)\
Detect misspellings in queries or documents using the numbers of occurrences of words and phrases.
2. [Phrase Extraction job](/docs/5/fusion/reference/config-ref/jobs/phrase-extraction)\
Identify multi-word phrases in signals.
3. [Synonym Detection](/docs/5/fusion/reference/config-ref/jobs/synonym-detection)\
Use this job to generate pairs of synonyms and pairs of similar queries. Two words are considered potential synonyms when they are used in a similar context in similar queries.
#### Process flow
The first and second jobs can provide input to improve the Synonym job’s output:
* Token and Phrase Spell Correction job results can be used to avoid finding mainly misspellings, or mixing synonyms with misspellings.
* Phrase Extraction job results can be used to find pairs of synonyms with multiple tokens, such as "lithium ion"/"ion battery".
The Phrase Extraction and Synonym Detection jobs are triggered by the success of the previous job: the phrase detection job runs only if the spell correction job succeeds, and the synonym job runs only if the phrase detection job succeeds.
### Additional query rewriting jobs
These jobs also produce results that are used for query rewriting, but must be created manually:
* [Head/Tail Analysis job](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis)\
Perform head/tail analysis of queries from collections of raw or aggregated signals, to identify underperforming queries and the reasons. This information is valuable for improving overall conversions, Solr configurations, auto-suggest, product catalogs, and SEO/SEM strategies, in order to improve conversion rates.
* [Ground Truth job](/docs/5/fusion/reference/config-ref/jobs/ground-truth)\
Ground truth or gold standard datasets are used in the ground truth jobs and query relevance metrics to define a specific set of documents.
Ground truth jobs estimate ground truth queries using click signals and query signals, with document relevance per query determined using a click/skip formula.
Use this job along with the [Ranking Metrics job](/docs/5/fusion/reference/config-ref/jobs/ranking-metrics) to calculate relevance metrics, such as Normalized Discounted Cumulative Gain (nDCG).
To create a ground truth job, sign in to Fusion and click **Collections > Jobs**. Then click **Add+** and in the Experiment Evaluation Jobs section, select **Ground Truth**. You can enter basic and advanced parameters to configure the job. If the field has a default value, it is populated when you click to add the job.
## Basic parameters
To enter advanced parameters in the UI, click **Advanced**. Those parameters are described in [the advanced parameters section](#advanced-parameters).
* **Spark job ID.** The unique ID for the Spark job that references this job in the API. This is the `id` field in the configuration file. Required field.
* **Input/Output Parameters.** This section includes the **Signals collection** field, which is the Solr collection that contains click signals and its associated search log identifier. This is the `signalsCollection` field in the configuration file. Required field.
## Advanced parameters
If you click the **Advanced** toggle, the following optional fields are displayed in the UI.
* **Spark Settings.** This section lets you enter `parameter name:parameter value` options to use in this job. This is the `sparkConfig` field in the configuration file.
* **Additional Options.** This section includes the following options:
* **Search logs pipeline.** The pipeline ID associated with search log entries. This is the `searchLogsPipeline` field in the configuration file.
* **Join key (query signals).** The common key that joins the query signals in the signals collection. This is the `joinKeySignals` field in the configuration file.
* **Join key (click signals).** The common key that joins the click signals in the signals collection. This is the `joinKeySignals` field in the configuration file.
* **Search logs and options.** This section lets you enter `property name:property value` options to when loading the search logs collection. This is the `searchLogsAddOpts` field in the configuration file.
* **Additional signals options.** This section lets you enter `property name:property value` options when loading the signals collection. This is the `signalsAddOpts` field in the configuration file.
* **Filter queries.** The `array[string]` filter query to apply when selecting top queries from the query signals in the signals collection. This is the `filterQueries` field in the configuration file.
* **Top queries limit.** The total number of queries to select for ground truth calculations when this job is run. This is the `topQueriesLimit` field in the configuration file.
For more information, see [Ground truth query rewrite API configurations](/api-reference/experiments-api/get-ground-truth-results).
### "rules" role for query rewriting users
The "rules" [role](/docs/5/fusion/operations/security/access-control/roles) provides permissions to access query rewriting features for all Fusion apps. A Fusion admin can create a user account with this role to give a business user access to the Query Rewriting UI.
## Query rewrite jobs post-processing cleanup
To perform more extensive cleanup of query rewrites, complete the procedures in **Query Rewrite Jobs Post-processing Cleanup**.
The Synonym Detection job uses the output of the Misspelling Detection job and Phrase Extraction job. Therefore, post processing *must* occur in the order specified in this topic for the [Synonym detection job cleanup](#synonym-detection-job-cleanup), [Phrase extraction job cleanup](#phrase-extraction-job-cleanup), and [Misspelling detection job cleanup](#misspelling-detection-job-cleanup) procedures. The [Head-Tail Analysis job cleanup](#head-tail-analysis-job-cleanup) can occur in any order.
## Synonym detection job cleanup
Use this job to remove low confidence synonyms.
### Prerequisites
Complete this:
* **AFTER** the Misspelling Detection and Phrase Extraction jobs have successfully completed.
* **BEFORE** removing low confidence synonym suggestions generated in the *post processing* phrase extraction cleanup and misspelling detection cleanup procedures detailed later in this topic.
### Remove low confidence synonym suggestions
Use either a [Synonym cleanup method 1 - API call](#synonym-cleanup-method-1-api-call) or the [Synonym cleanup method 2 - Fusion Admin UI](#synonym-cleanup-method-2-fusion-admin-ui) to remove low confidence synonym suggestions.
#### Synonym cleanup method 1 - API call
1. Open the `delete_lowConf_synonyms.json` file.
```json theme={"dark"}
{
"type" : "rest-call",
"id" : "DC_Large_QR_DELETE_LOW_CONFIDENCE_SYNONYMS",
"callParams" : {
"uri" : "solr://DC_Large_query_rewrite_staging/update",
"method" : "post",
"queryParams" : {
"wt" : "json"
},
"headers" : { },
"entity" : "type:synonym AND confidence:[0 TO 0.0005]"
},
"type" : "rest-call",
"type" : "rest-call"
}
```
**REQUEST ENTITY** specifies the threshold for low confidence synonyms. Edit the upper range from 0.0005 to increase or decrease the threshold based on your data.
2. Enter `` in the **uri** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
3. Change the `id` field if applicable.
4. Specify the **upper confidence level** in the **entity** field.
The **entity** field specifies the threshold for low confidence synonyms. Edit the upper range to increase or decrease the threshold based on your data.
#### Synonym cleanup method 2 - Fusion Admin UI
1. Log in to Fusion and select **Collections > Jobs**.
2. Select **Add+ > Custom and Other Jobs > REST Call**.
3. Enter **delete-low-confidence-synonyms** in the **ID** field.
4. Enter `` in the **ENDPOINT URI** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
5. Enter **POST** in the **CALL METHOD** field.
6. In the **QUERY PARAMETERS** section, select **+** to add a property.
7. Enter **wt** in the **Property Name** field.
8. Enter **json** in the **Property Value** field.
9. In the **REQUEST PROTOCOL HEADERS** section, select **+** to add a property.
10. Enter the following as a **REQUEST ENTITY (AS STRING)**
`type:synonym AND confidence: [0 TO 0.0005]`
**REQUEST ENTITY** specifies the threshold for low confidence synonyms. Edit the upper range from 0.0005 to increase or decrease the threshold based on your data.
### Delete all synonym suggestions
To delete *all* of the synonym suggestions, enter the following in the **REQUEST ENTITY** section:
`type:synonym`
This entry may be helpful when tuning the synonym detection job and testing different configuration parameters.
## Phrase extraction job cleanup
Use this job to remove low confidence phrase suggestions.
### Prerequisites
Complete this:
* **AFTER** you complete [Synonym detection job cleanup](#synonym-detection-job-cleanup)
### Remove low confidence phrase suggestions
Use either a [Phrase cleanup method 1 - API call](#phrase-cleanup-method-1-api-call) or the [Phrase cleanup method 2 - Fusion Admin UI](#phrase-cleanup-method-2-fusion-admin-ui) to remove low confidence phrase suggestions.
#### Phrase cleanup method 1 - API call
1. Open the `delete_lowConf_phrases.json` file.
```json theme={"dark"}
{
"type" : "rest-call",
"id" : "DC_Large_QR_DELETE_LOW_CONFIDENCE_PHRASES",
"callParams" : {
"uri" : "solr://DC_Large_query_rewrite_staging/update",
"method" : "post",
"queryParams" : {
"wt" : "json"
},
"headers" : { },
"entity" : " type:phrase AND confidence:[0 TO ]"
},
"type" : "rest-call",
"type" : "rest-call"
}
```
2. Enter `` in the **uri** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
3. Change the **id** field if applicable.
4. Specify the **upper confidence level** in the **entity** field.
The **entity** field specifies the threshold for low confidence phrases. Edit the upper range to increase or decrease the threshold based on your data.
#### Phrase cleanup method 2 - Fusion Admin UI
1. Log in to Fusion and select **Collections > Jobs**.
2. Select **Add+ > Custom and Other Jobs > REST Call**.
3. Enter **remove-low-confidence-phrases** in the **ID** field.
4. Enter `` in the **ENDPOINT URI** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
5. Enter **POST** in the **CALL METHOD** field.
6. In the **QUERY PARAMETERS** section, select **+** to add a property.
7. Enter **wt** in the **Property Name** field.
8. Enter **json** in the **Property Value** field.
9. In the **REQUEST PROTOCOL HEADERS** section, select **+** to add a property.
10. Enter the following as a **REQUEST ENTITY (AS STRING)**
`type:phrase AND confidence: [0 TO ]`
**REQUEST ENTITY** specifies the threshold for low confidence phrases. Edit the upper range to increase or decrease the threshold based on your data.
### Delete all phrase suggestions
To delete *all* of the phrase suggestions, enter the following in the **REQUEST ENTITY** section:
`type:phrase`
This entry may be helpful when tuning the phrase extraction job and testing different configuration parameters.
## Misspelling detection job cleanup
Use this job to remove low confidence spellings (also referred to as misspellings).
### Prerequisites
Complete this:
* **AFTER** you complete [Synonym detection job cleanup](#synonym-detection-job-cleanup) and [Phrase extraction job cleanup](#phrase-extraction-job-cleanup)
### Remove misspelling suggestions
Use either a [Misspelling cleanup method 1 - API call](#misspelling-cleanup-method-1-api-call) or the [Misspelling cleanup method 2 - Fusion Admin UI](#misspelling-cleanup-method-2-fusion-admin-ui) to remove misspelling suggestions.
#### Misspelling cleanup method 1 - API call
1. Open the `delete_lowConf_misspellings.json` file.
```json theme={"dark"}
{
"type" : "rest-call",
"id" : "DC_Large_QR_DELETE_LOW_CONFIDENCE_MISSPELLINGS",
"callParams" : {
"uri" : "solr://DC_Large_query_rewrite_staging",
"method" : "post",
"queryParams" : {
"wt" : "json"
},
"headers" : { },
"entity" : "type:spell AND confidence:[0 TO 0.5]"
},
"type" : "rest-call",
"type" : "rest-call"
}
```
2. Enter `` in the **uri** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
3. Change the **id** field if applicable.
4. Specify the **upper confidence level** in the **entity** field.
The **entity** field specifies the threshold for low confidence spellings. Edit the upper range to increase or decrease the threshold based on your data.
#### Misspelling cleanup method 2 - Fusion Admin UI
1. Log in to Fusion and select **Collections > Jobs**.
2. Select **Add+ > Custom and Other Jobs > REST Call**.
3. Enter **remove-low-confidence-spellings** in the **ID** field.
4. Enter `` in the **ENDPOINT URI** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
5. Enter **POST** in the **CALL METHOD** field.
6. In the **QUERY PARAMETERS** section, select **+** to add a property.
7. Enter **wt** in the **Property Name** field.
8. Enter **json** in the **Property Value** field.
9. In the **REQUEST PROTOCOL HEADERS** section, select **+** to add a property.
10. Enter the following as a **REQUEST ENTITY (AS STRING)**
`type:spell AND confidence: [0 TO 0.5]`
**REQUEST ENTITY** specifies the threshold for low confidence spellings. Edit the upper range from 0.5 to increase or decrease the threshold based on your data.
### Delete all misspelling suggestions
To delete *all* of the misspelling suggestions, enter the following in the **REQUEST ENTITY** section:
`type:spell`
This entry may be helpful when tuning the misspelling detection job and testing different configuration parameters.
## Head-tail analysis job cleanup
The head-tail analysis job puts tail queries into one of multiple reason categories. For example, a tail query that includes a number might be assigned to the 'numbers' reason category. If the output in a particular category is not useful, you can remove it from the results. The examples in this section remove the numbers category.
### Prerequisites
The head-tail analysis job cleanup does *not* have to occur in a specific order.
### Remove head-tail analysis query suggestions
Use either a [Head-tail analysis cleanup method 1 - API call](#head-tail-analysis-cleanup-method-1-api-call) or the [Head-tail analysis cleanup method 2 - Fusion Admin UI](#head-tail-analysis-cleanup-method-2-fusion-admin-ui) to remove query category suggestions.
#### Head-tail analysis cleanup method 1 - API call
1. Open the `delete_lowConf_headTail.json` file.
```json theme={"dark"}
{
"type" : "rest-call",
"id" : "DC_Large_QR_HEAD_TAIL_CLEANUP",
"callParams" : {
"uri" : "solr://DC_Large_query_rewrite_staging/update",
"method" : "post",
"queryParams" : {
"wt" : "json"
},
"headers" : { },
"entity" : "reason_code_s:(\"number\" \"number spelling\" \"number rare-term\" \"question number other-specific\" \"number others\" \"number other-specific\" \"number other-extra\" \"product number other-specific\" \"product number other-extra\" \"product number spelling\" \"product number others\" \"product number rare-term\" \"product question number\" \"product number re-wording\" \"question number other-extra\" \"number re-wording\")"
},
"type" : "rest-call",
"type" : "rest-call"
}
```
2. Enter `` in the **uri** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
3. Change the **id** field if applicable.
#### Head-tail analysis cleanup method 2 - Fusion Admin UI
1. Log in to Fusion and select **Collections > Jobs**.
2. Select **Add+ > Custom and Other Jobs > REST Call**.
3. Enter **remove-low-confidence-head-tail** in the **ID** field.
4. Enter `` in the **ENDPOINT URI** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
5. Enter **POST** in the **CALL METHOD** field.
6. In the **QUERY PARAMETERS** section, select **+** to add a property.
7. Enter **wt** in the **Property Name** field.
8. Enter **json** in the **Property Value** field.
9. In the **REQUEST PROTOCOL HEADERS** section, select **+** to add a property.
10. Enter the following as a **REQUEST ENTITY (AS STRING)**
```
reason_code_s:("number" "number spelling" "number rare-term" "question number other-specific" "number others" "number other-specific" "number other-extra" "product number other-specific" "product number other-extra" "product number spelling" "product number others" "product number rare-term" "product question number" "product number re-wording" "question number other-extra" "number re-wording")
```
### Delete all head-tail suggestions
To delete *all* of the head-tail suggestions, enter the following in the **REQUEST ENTITY** section:
`type:tail`
This entry may be helpful when tuning the head-tail job and testing different configuration parameters.
## Learn more
1. In the Fusion UI, navigate to **Relevance** > **Query Rewriting**.
2. Under **Business Rules**, click **View**.
3. Hover over the rule you want to edit.
4. Click the icon next to the rule.
## Publishing your changes
**How to publish updated business rules**
1. In the **Business Rules** screen, click the **PUBLISH** button.
Fusion prompts you to confirm that you want to publish your changes.
2. Click **PUBLISH**.
You can un-publish a query rewrite by changing its status to "denied", then clicking **PUBLISH**.For **My Custom Rules** to show in the UI, you must first create a `custom_rule_type` via the Query Rewrite API.
1. Navigate to **Relevance** > **Query Rewriting**.
2. Under **Business Rules**, click **View**.
3. Click the icon.
The **New Rule** window appears.
4. Enter the general parameters and conditions for this custom rule.
5. Under **Action**, select **My Custom Rule**.
6. Select the query pipeline to use when this rule is triggered.
7. Enter a name for this custom rule type.
8. Optionally, you can click **Add** under **Return Parameter Policy Override** to override the value of a parameter returned from the specified pipeline:
1. Enter the name of the parameter.
2. Enter the desired value.
3. Select one of the following policies:
* **Replace** the returned value with the specified value.
* **Append** the specified value to the returned value.
* **Remove** the specified parameter from the set of returned parameters.
* **Default**
4. Enter a value for **From Custom Rule**.
By default, all query rewriting strategies are enabled, and all of the data in the `_query_rewrite_staging` collection is applied in the Simulator. To see how relevancy is affected by individual strategies, you can selectively enable or disable strategies in the query rewriting dashboard.
**How to enable or disable query rewriting strategies**
1. From the Simulator, click **Query Rewriting Dashboard**.\
The query rewriting dashboard appears:
2. Click **Enabled** to disable an active strategy, or click **Disabled** to enable an inactive strategy.
3. Click **Simulator** to return to the Simulator and see how your changes affect search results.
# Getting Started with Recommendations and Boosting
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/recommendations/getting-started
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/recommendations/getting-started
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/recommendations/getting-started
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/480
[Signals](/docs/5/fusion/getting-data-out/query-enhancement/signals/overview) provide the data that Fusion uses to generate [collaborative recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods). The simplest way to get started is to enable signals and recommendations in one of your primary collections.
Once you do this, Fusion automatically creates a set of [default objects](#default-objects-for-recommendations) and begins creating and updating collections of recommendations on a regular schedule.
[Content-based recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods) can be used without enabling signals or recommendations, but they require manual configuration.
The course for **Recommendations** focuses on how Fusion uses signals to automatically produce custom recommendations to offer users the items most relevant to their experience.
## Enable signals
Signals are enabled by default for new collections when you have a Fusion license installed. You can enable or disable signals for any collection at **Collections** > **Collections Manager**.
Enabling signals automatically creates a set of [aggregation jobs](/docs/5/fusion/reference/config-ref/jobs/aggregations/sql-aggregation) which create the input data for recommendations. See [Signals and Aggregations](/docs/5/fusion/getting-data-out/query-enhancement/signals/overview) for complete details.
## Enable recommendations
Recommendations are not enabled by default. You can enable recommendations using one of the following methods:
* **Collections Manager.** Sign in to Fusion and select your application. Then click **Collections** > **Collections Manager**. Click the gear icon of your collection and select **Enable Recommendations**.
The system displays the Enable Recommendations dialog box. Click **Enable Recommendations** to automatically enable the [items-for-user](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods) and [items-for-item](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods) recommendation methods. To use additional [recommendation methods](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods), you must configure them separately.
* **Relevance.** Sign in to Fusion and select your application. Then click **Relevance > Recommendations**. The system displays the Enable Recommendations dialog box. Click **Enable Recommendations** to automatically enable the [items-for-user](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods) and [items-for-item](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods) recommendation methods. To use additional [recommendation methods](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods), you must configure them separately.
* **REST API.** Use this command to enable recommendations:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X PUT http://FUSION_HOST:FUSION_PORT/api/collections/COLLECTION_NAME/features/recommendations -H 'content-type: application/json' -d '{"enabled":true}'
```
## Default objects for recommendations
When recommendations are enabled, Fusion automatically creates a default set of collections, jobs, schedules, and query pipelines that provide basic functionality for recommendations.
You can tune the default jobs and pipelines as needed to refine the results, or create new ones, then configure your search application to request recommendations from the query pipelines.
**See also the default objects created when you [enable signals](/docs/5/fusion/getting-data-out/query-enhancement/signals/overview). These must already exist when you enable recommendations.**
### Collections
* `COLLECTION_NAME_items_for_item_recommendations`
Collection to hold generated item-item similarities (by default 10 per item). No `user_id_s` data is present. A Recommend Items for Item query pipeline stage can use the similarities to return item recommendations. For example, a query in which `doc_id_s = docA` would return an ordered list of other `doc_id_s` values for documents that are similar to document `docA`, along with the similarities. For example: `[("docB", 0.83), ("docC", 0.55), ("docD", 0.43), …, ("docK", 0.22)]`.
* `COLLECTION_NAME_items_for_user_recommendations`
Collection to hold recommended items for a user. By default the job creates 10 recommendations per user.
* `COLLECTION_NAME_recs_aggr`
Collection to hold the output of recommender aggregation jobs.
* `COLLECTION_NAME_bpr_user_recs` and `COLLECTION_NAME_bpr_item_recs`
Collections for Bayesian Personalized Ranking (BPR) recs, user-item, and item-item (respectively).
* `COLLECTION_NAME_content_recs`
Collection for content-based recommender recs.
* `COLLECTION_NAME_recommender_models`
Collection that stores ALS models.
* `COLLECTION_NAME_queries_query_recs`
Collection for output of the Query-Query Similarity Job.
### Jobs and schedules
Enabling recommendations creates the new jobs below, each using a different method to produce items-for-item and items-for-user recommendations. These jobs consume the output of the [signals aggregation jobs](/docs/5/fusion/reference/config-ref/jobs/aggregations/sql-aggregation).
| | [BPR Recommender](/docs/5/fusion/reference/config-ref/jobs/bpr-recommender) | [Content-Based Recommender](/docs/5/fusion/reference/config-ref/jobs/content-based-recommender) | [Query-to-Query Session-Based Similarity](/docs/5/fusion/reference/config-ref/jobs/query-to-query-session-based-similarity) | Default job name |
| - | - | - | - | - |
| `COLLECTION_NAME_item_recommendations` | `COLLECTION_NAME_bpr_item_recs` | `COLLECTION_NAME_content_recs` | `COLLECTION_NAME_query_recs` | Default input collection |
| `COLLECTION_NAME_signals_aggr` | `COLLECTION_NAME_signals_aggr` | `COLLECTION_NAME` | `COLLECTION_NAME_signals` | Default output collections |
| `COLLECTION_NAME_items_for_user_recommendations` `COLLECTION_NAME_items_for_item_recommendations` | `COLLECTION_NAME_bpr_user_recs` `COLLECTION_NAME_bpr_item_recs` | `COLLECTION_NAME_content_recs` | `COLLECTION_NAME_queries_query_recs` | Recommendation type |
None of the jobs are scheduled by default; you must manually run or schedule them.
NOTE:
* The `COLLECTION_NAME_user_item_prefs_agg` job provides input data for the ALS and BPR jobs and must run before them. See [SQL Aggregations](/docs/5/fusion/reference/config-ref/jobs/aggregations/built-in-sql-aggregation-jobs#collection-name-user-item-preferences-aggregation) for details.
* The `COLLECTION_NAME_user_query_history_agg` job generates aggregations for user-item preferences from signal information.
Fusion does *not* automatically create a [Query-to-Query Session-Based Similarity job](/docs/5/fusion/reference/config-ref/jobs/query-to-query-session-based-similarity), which is needed for certain [recommender types](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods#content-based-vs-collaborative-recommendations).
### Query pipelines
* `COLLECTION_NAME_items_for_user_recommendations`
Query pipeline to generate recommendations of items for a user.
* `COLLECTION_NAME_items_for_item_recommendations`
Query pipeline to generate recommendations of items similar to an item.
* `COLLECTION_NAME_queries_query_recs`
Query pipeline to generate [queries-for-query recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/queries-for-query) using a model created by the [Query-to-Query Session-Based Similarity](/docs/5/fusion/reference/config-ref/jobs/query-to-query-session-based-similarity) job.
# Items-For-Item Recommendations
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-item
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-item
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-item
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/476
Items-for-item recommendations present items that are similar to a specified item. For example, when the user is viewing a BMX bicycle, Fusion can recommend other BMX bicycles. Similarity can be based on different criteria, such as click patterns, people who bought this also bought that, percentage match of document tags, and so on.
## Collaborative items-for-item recommendations
If you have enabled signals and recommendations for a collection, then the default `COLLECTION_NAME_bpr_item_recs`, a [BPR Recommender](/docs/5/fusion/reference/config-ref/jobs/bpr-recommender) job is already created and configured to produce items-for-item recommendations (as well as [items-for-user recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-user)).
This is the recommended job type, and produces better results with a short runtime. Download the APPName\_item\_item\_rec\_pipelines\_bpr.json file and **Migrate Fusion Objects** to create the query pipeline that consumes this job’s output. See **Fetch Items-for-Item Recommendations (Collaborative/BPR Method)** for details.
You can use the Fusion UI and the [Objects API](/api-reference/objects/get-objects-service-status) to migrate collections and related objects, including your searchable data, configuration data, pipelines, aggregations, and other objects on which your collections depend. You can also migrate entire apps.
To upgrade from one Fusion version to a 5.x version, see Fusion 5 Upgrades to migrate objects.
You might need to migrate objects in the following circumstances:
* When migrating data from development environments into testing and production environments
* To back up data, so you can restore it after an incident of data loss
* When the migrator script was not able to migrate an object automatically
When you export and import objects to migrate them, make sure you that any data gets to where it is going before deleting the sources.
## Migration approaches
Several approaches are available for migrating Fusion objects. This table summarizes the approaches.
| | Export an app | Import an app | Export an object | Import an object | Add an object to an app |
| - | - | - | - | - | - |
| **Fusion UI** | App configuration | Launcher (entire app) App configuration (combine apps) | - | - | Object Explorer |
| **Objects API** (endpoints) | GET from `export` endpoint | POST to `import` endpoint | GET from `export` endpoint | POST to `import` endpoint | - |
For more information about using the Objects API to export and import objects, see [Objects API](/api-reference/objects/get-objects-service-status).
The remainder of this topic describes approaches in the Fusion UI.
Use the parts of the Fusion UI indicated in the table to export and import apps and specific objects. Exporting creates a zip file. To import, you select a data file and possibly a variable file.
The approach with Object Explorer differs. With Object Explorer, you can add objects from other apps (or that are not linked to any apps) to the currently open app.
## Export an app with the Fusion UI
**How to export an app with the Fusion UI**
1. Navigate to the launcher.
2. Hover over the app you want to export and click the Configure icon:
3. In the app config window, click **Export app to zip**:
See import for information to import the downloaded zip file into other instances of Fusion 5.x.
## Import an app with the Fusion UI
**How to import an app with the Fusion UI**
1. Navigate to the launcher.
2. Click **Import app**.
3. Under **Data File**, click **Choose File** and select the zip file containing the app you want to import.
4. If your app has [usernames and passwords](/api-reference/objects/list-imported-variables) in a separate file, select it under **Variables File**.
If the Variables File is needed, it *must* be a separate file that is not in a .zip file. It is a .json map of variables to values. The following is an example:
```json theme={"dark"}
{
"secret.dataSources.Inventory_BR_S3_DS.password":"SOMETHING",
"secret.dataSources.LLM_A_BR_S3_DS.password":"FmJSaDE9Tj5REDACTED",
"secret.dataSources.LLM_BR_Inventory_S3_DS.password":"FmJSaDE9Tj5GzIVvethAC4Huh",
"secret.dataSources.LLM_BR_Load_S3_DS.password":"FmJSaDE9Tj5GzIVvethAC4"
}
```
5. You can sometimes edit parameter values to use the new values in the imported app. If this is the case, Fusion displays a dialog box that lets you edit the parameter values.
Make desired changes, and then click **Import**.
## Copy an app
To copy an app from one deployment to a different one, [export the app](#export-an-app-with-the-fusion-ui) on the source deployment, and then [import the app](#import-an-app-with-the-fusion-ui) on the target deployment.
## Import objects into an app
You can import objects into the currently open app.
**How to import objects into an open app**
1. In the Fusion launcher, click the app into which you want to import objects.
The Fusion workspace appears.
2. Click **System > Import Fusion Objects**.
The Import Fusion Objects window opens.
3. Select the data file from your local filesystem.
If you are importing [usernames and passwords](/api-reference/objects/import-objects) in a separate file, select it under **Variables File**.
4. Click **Import**.
If there are conflicts, Fusion prompts you to specify an import policy:
* Click **Overwrite** to overwrite the objects on the target system with the ones in the import file.
* Click **Merge** to skip all conflicting objects and import only the non-conflicting objects.
* Click **Start Over** to abort the import.
Fusion confirms that the import was successful:
5. Click **Close** to close the Import Fusion Objects window.
## Add an object to an app
You can add objects present in other apps (or in no apps) to the open app. Some objects are linked to other apps. You can also add those directly to an app.
* **Add an object to an app** – While in the Fusion workspace for the app to which you want to add an object, open Object Explorer and click **In Any App**. Search for or browse to the object you want to add. Hover over the object, click the App icon, and then click **Add to this app**.
* **Add an object to an app directly** – In cases when an object is linked to an app, but is not linked directly to the app (it is linked via some dependency), you can add the object to an app directly.
While in the Fusion workspace for the app to which you want to add an object directly, open Object Explorer and click **In Any App**. Search for or browse to the object you want to add. Hover over the object, click the App icon, and then click **Add to this app directly**.
In Fusion, you can download and import a query pipeline that works out of the box with the default [BPR Recommender](/docs/5/fusion/reference/config-ref/jobs/bpr-recommender) job.
There are two separate pipelines attached below which work in different ways to query and return recommendations.
**Query for recommendations only**
1. Download the `APPName_item_item_rec_pipelines_bpr.json` file.
2. Rename the file to replace `APPName` with the name of your Fusion app, such as `ProductCatalog_item_item_rec_pipelines_bpr.json`.
3. Open the JSON file, replace all instances of `APPName` with the name of your Fusion app, such as `ProductCatalog`, and save it.
4. Import the JSON file into your Fusion instance using the Query Pipelines REST API:
```
curl -u USERNAME:PASSWORD /api/query-pipelines -XPOST -H 'content-type:application/json' -d@
```
5. In the Fusion UI, navigate to **Query** > **Query Pipelines** to verify that the new pipeline is available.
This pipeline should be used to query the collection where the recommendations are stored. It makes a query against the `itemId` field and only returns the recommended `itemId` values. To get the actual items, you need to make a second query to the respective catalog collection with the returned `itemId` values.
No additional configuration is needed to use this pipeline with the default BPR job configuration.
**Boost recommended products from catalog**
1. Download the `APPName_items_for_item_bpr_boost.json` file
2. Rename the file to replace `APPName` with the name of your Fusion app, such as `ProductCatalog_item_item_rec_pipelines_bpr.json`.
3. Open the JSON file, replace all instances of `APPName` with the name of your Fusion app, such as `ProductCatalog`.
4. Fill in the `collection` name field in the first `Recommend Items for Item` stage and save the file.
5\. Import the JSON file into your Fusion instance using the Query Pipelines REST API:
```
curl -u USERNAME:PASSWORD /api/query-pipelines -XPOST -H 'content-type:application/json' -d@
```
5. In the Fusion UI, navigate to **Query** > **Query Pipelines** to verify that the new pipeline is available.
This pipeline queries the recommendations collection and then makes a subsequent query to the actual catalog collection boosting the recommended items and returning the actual items from the catalog. This pipeline will therefore also return recommendations even if none were generated/available.This pipeline expects a request parameter called `id=` to be appended to the request in order to work. An example query URL to this pipeline would look like `https://example-recs-fusion.com/api/query-pipelines/APPName_items_for_item_bpr_boost/collections/catalog/select?q=**:**&id=SomeItemId`If the pipeline does not appear in the **Query Pipelines** panel, you may need to attach it to your app like this: Go to **System** > **Object Explorer**, click the **In No Apps** filter, hover over the pipeline, click the icon, and select **Add to this app**.
No additional configuration is needed to use this pipeline with the default BPR job configuration.
**See also**
* [Items-For-Item Recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-item)
* [Getting Started with Recommendations and Boosting](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/getting-started)
```mermaid theme={"dark"}
flowchart TD
accTitle: BPR collaborative recommendations dataflow
accDescr: A flowchart showing the BPR Collaborative Recommendations Job processing signals and populating bpr_item_recs and bpr_user_recs collections used by their respective query pipelines
UI["UI"]
subgraph item_profile["Query Profile"]
Pipe1["bpr_item_recs Query Pipeline"]
end
subgraph user_profile["Query Profile"]
Pipe2["bpr_user_recs Query Pipeline"]
end
Col1[("bpr_item_recs collection")]
Col2[("bpr_user_recs collection")]
Job["BPR Collaborative Recommendations Job (Experimental)"]
Sig[("signals collection")]
UI <--> item_profile
Pipe1 --> Col1
UI <--> user_profile
Pipe2 --> Col2
Sig --> Job
Job --> Col1
Job --> Col2
```
If you want to use different parameters for items-for-item recommendations and [items-for-user recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-user), simply create separate jobs for each, where one job configuration includes an output collection for items-for-item recommendations only and the other includes an output collection for items-for-user recommendations only.
## Content-based items-for-item recommendations
The [Content-Based Recommender](/docs/5/fusion/reference/config-ref/jobs/content-based-recommender) job computes item similarities based on their content, such as product descriptions. This is a useful option when you do not have enough signals for accurate results using the collaborative method described above. Download the APPName\_item\_item\_rec\_pipelines\_content.json file and **Migrate Fusion Objects** to create the query pipeline that consumes this job’s output.
You can use the Fusion UI and the [Objects API](/api-reference/objects/get-objects-service-status) to migrate collections and related objects, including your searchable data, configuration data, pipelines, aggregations, and other objects on which your collections depend. You can also migrate entire apps.
To upgrade from one Fusion version to a 5.x version, see Fusion 5 Upgrades to migrate objects.
You might need to migrate objects in the following circumstances:
* When migrating data from development environments into testing and production environments
* To back up data, so you can restore it after an incident of data loss
* When the migrator script was not able to migrate an object automatically
When you export and import objects to migrate them, make sure you that any data gets to where it is going before deleting the sources.
## Migration approaches
Several approaches are available for migrating Fusion objects. This table summarizes the approaches.
| | Export an app | Import an app | Export an object | Import an object | Add an object to an app |
| - | - | - | - | - | - |
| **Fusion UI** | App configuration | Launcher (entire app) App configuration (combine apps) | - | - | Object Explorer |
| **Objects API** (endpoints) | GET from `export` endpoint | POST to `import` endpoint | GET from `export` endpoint | POST to `import` endpoint | - |
For more information about using the Objects API to export and import objects, see [Objects API](/api-reference/objects/get-objects-service-status).
The remainder of this topic describes approaches in the Fusion UI.
Use the parts of the Fusion UI indicated in the table to export and import apps and specific objects. Exporting creates a zip file. To import, you select a data file and possibly a variable file.
The approach with Object Explorer differs. With Object Explorer, you can add objects from other apps (or that are not linked to any apps) to the currently open app.
## Export an app with the Fusion UI
**How to export an app with the Fusion UI**
1. Navigate to the launcher.
2. Hover over the app you want to export and click the Configure icon:
3. In the app config window, click **Export app to zip**:
See import for information to import the downloaded zip file into other instances of Fusion 5.x.
## Import an app with the Fusion UI
**How to import an app with the Fusion UI**
1. Navigate to the launcher.
2. Click **Import app**.
3. Under **Data File**, click **Choose File** and select the zip file containing the app you want to import.
4. If your app has [usernames and passwords](/api-reference/objects/list-imported-variables) in a separate file, select it under **Variables File**.
If the Variables File is needed, it *must* be a separate file that is not in a .zip file. It is a .json map of variables to values. The following is an example:
```json theme={"dark"}
{
"secret.dataSources.Inventory_BR_S3_DS.password":"SOMETHING",
"secret.dataSources.LLM_A_BR_S3_DS.password":"FmJSaDE9Tj5REDACTED",
"secret.dataSources.LLM_BR_Inventory_S3_DS.password":"FmJSaDE9Tj5GzIVvethAC4Huh",
"secret.dataSources.LLM_BR_Load_S3_DS.password":"FmJSaDE9Tj5GzIVvethAC4"
}
```
5. You can sometimes edit parameter values to use the new values in the imported app. If this is the case, Fusion displays a dialog box that lets you edit the parameter values.
Make desired changes, and then click **Import**.
## Copy an app
To copy an app from one deployment to a different one, [export the app](#export-an-app-with-the-fusion-ui) on the source deployment, and then [import the app](#import-an-app-with-the-fusion-ui) on the target deployment.
## Import objects into an app
You can import objects into the currently open app.
**How to import objects into an open app**
1. In the Fusion launcher, click the app into which you want to import objects.
The Fusion workspace appears.
2. Click **System > Import Fusion Objects**.
The Import Fusion Objects window opens.
3. Select the data file from your local filesystem.
If you are importing [usernames and passwords](/api-reference/objects/import-objects) in a separate file, select it under **Variables File**.
4. Click **Import**.
If there are conflicts, Fusion prompts you to specify an import policy:
* Click **Overwrite** to overwrite the objects on the target system with the ones in the import file.
* Click **Merge** to skip all conflicting objects and import only the non-conflicting objects.
* Click **Start Over** to abort the import.
Fusion confirms that the import was successful:
5. Click **Close** to close the Import Fusion Objects window.
## Add an object to an app
You can add objects present in other apps (or in no apps) to the open app. Some objects are linked to other apps. You can also add those directly to an app.
* **Add an object to an app** – While in the Fusion workspace for the app to which you want to add an object, open Object Explorer and click **In Any App**. Search for or browse to the object you want to add. Hover over the object, click the App icon, and then click **Add to this app**.
* **Add an object to an app directly** – In cases when an object is linked to an app, but is not linked directly to the app (it is linked via some dependency), you can add the object to an app directly.
While in the Fusion workspace for the app to which you want to add an object directly, open Object Explorer and click **In Any App**. Search for or browse to the object you want to add. Hover over the object, click the App icon, and then click **Add to this app directly**.
```mermaid theme={"dark"}
flowchart LR
accTitle: Content-based recommender job dataflow
accDescr: A flowchart showing the content_recs query pipeline connecting to the content_recs collection, fed by the Content-Based Recommender Job processing the documents collection
UI["UI"]
subgraph profile["Query Profile"]
Pipe["content_recs Query Pipeline"]
end
Col[("content_recs collection")]
Job["Content-Based Recommender Job (Experimental)"]
Docs[("documents collection")]
UI <--> profile
Pipe <--> Col
Docs --> Job <--> Col
```
Once you have run the recommender job, see **Fetch Items-for-Item Recommendations (Content-Based Method)**.
In Fusion, you can download and import a query pipeline that works out of the box to fetch the items-for-item recommendations generated by the default [Content-Based Recommender](/docs/5/fusion/reference/config-ref/jobs/content-based-recommender) job.
There are two separate pipelines attached below which work in different ways to query and return recommendations.
**Query for recommendations only**
1. Download the `APPName_item_item_rec_pipelines_content.json` file.
2. Rename the file to replace `APPName` with the name of your Fusion app, such as `ProductCatalog_item_item_rec_pipelines_bpr.json`.
3. Open the JSON file, replace all instances of `APPName` with the name of your Fusion app, such as `ProductCatalog`, and save it.
4. Import the JSON file into your Fusion instance using the Query Pipelines REST API:
```
curl -u USERNAME:PASSWORD /api/query-pipelines -XPOST -H 'content-type:application/json' -d@
```
5. In the Fusion UI, navigate to **Query** > **Query Pipelines** to verify that the new pipeline is available.
This pipeline should be used to query the collection where the recommendations are stored. It makes a query against the `itemId` field and returns the recommended `itemId` values. To get the actual items, you need to make a second query to the respective catalog collection with the returned `itemId` values.
No additional configuration is needed to use this pipeline with the default BPR job configuration.
**Boost recommended products from catalog**
1. Download the `APPName_content_boost.json` file
2. Rename the file to replace `APPName` with the name of your Fusion app, such as `ProductCatalog_item_item_rec_pipelines_bpr.json`.
3. Open the JSON file, replace all instances of `APPName` with the name of your Fusion app, such as `ProductCatalog`.
4. Fill in the `collection` name field in the first `Recommend Items for Item` stage and save the file.
5. Import the JSON file into your Fusion instance using the Query Pipelines REST API:
```
curl -u USERNAME:PASSWORD /api/query-pipelines -XPOST -H 'content-type:application/json' -d@
```
6. In the Fusion UI, navigate to **Query** > **Query Pipelines** to verify that the new pipeline is available.
This pipeline queries the recommendations collection and then makes a subsequent query to the actual catalog collection boosting the recommended items and returning the actual items from the catalog. This pipeline will therefore also return recommendations even if none were generated/available.This pipeline expects a request parameter called `id=itemId` to be appended to the request in order to work. An example query URL to this pipeline would look like `https://example-recs-fusion.com/api/query-pipelines/APPName_items_for_user_bpr_boost/collections/catalog/select?q=**:**&id=SomeItemId`If the pipeline does not appear in the **Query Pipelines** panel, you may need to attach it to your app like this: Go to **System** > **Object Explorer**, click the **In No Apps** filter, hover over the pipeline, click the icon, and select **Add to this app**.
**See also**
* [Items-For-Item Recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-item)
* [Content-Based Recommender job](/docs/5/fusion/reference/config-ref/jobs/content-based-recommender)
* [Getting Started with Recommendations and Boosting](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/getting-started)
# Items-For-Query Recommendations
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-query
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-query
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-query
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/481
The Items for Query recommender is the primary algorithm that supports recommendations on the search results page.
Items-for-query recommendations are based on how other users interacted with search results from the same query. This is one type of [collaborative recommendation](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods#content-based-vs-collaborative-recommendations).
For example, if users have searched for "titanic" in the past and many of them clicked on the search result for the DVD of the movie *Titanic* (as opposed to books or memorabilia), then this method boosts the DVD item on subsequent searches for "titanic".
Below is an example of items-for-query recommendations based on a search for "madonna":
To use this type of boosting, signals must be enabled but recommendations need not be.
## Job configuration
The items-for-query recommendations are implemented as a [Query-to-Query Session-Based Similarity](/docs/5/fusion/reference/config-ref/jobs/query-to-query-session-based-similarity) job.
## Query pipeline configuration
The [Boost with Signals](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommendation-boosting-query-stage) query stage is part of the default query pipeline, so usually you do not need to add it. It takes its input from the aggregated signals collection (`COLLECTION_NAME_signals_aggr`) at query time.
# Items-For-User Recommendations
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-user
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-user
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-user
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/477
Items-for-user recommendations use the [Recommend Items for User](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommend-items-for-user-query-stage) query stage to present items that are similar to ones in which the user has previously shown interest, based on the user’s search history, browsing history, purchase history, and so on. This is one type of [collaborative recommendation](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods#content-based-vs-collaborative-recommendations).
If you have enabled signals and recommendations for a collection, then the default `COLLECTION_NAME_bpr_item_recs`, a [BPR Recommender](/docs/5/fusion/reference/config-ref/jobs/bpr-recommender) job is already created and configured to produce items-for-user recommendations.
This is the recommended job type, and produces better results with a short runtime. Download the APPName\_item\_item\_rec\_pipelines\_bpr.json file and **Migrate Fusion Objects** to create the query pipeline that consumes this job’s output. See **Fetch Items-for-User Recommendations (Collaborative/BPR Method)** for details.
You can use the Fusion UI and the [Objects API](/api-reference/objects/get-objects-service-status) to migrate collections and related objects, including your searchable data, configuration data, pipelines, aggregations, and other objects on which your collections depend. You can also migrate entire apps.
To upgrade from one Fusion version to a 5.x version, see Fusion 5 Upgrades to migrate objects.
You might need to migrate objects in the following circumstances:
* When migrating data from development environments into testing and production environments
* To back up data, so you can restore it after an incident of data loss
* When the migrator script was not able to migrate an object automatically
When you export and import objects to migrate them, make sure you that any data gets to where it is going before deleting the sources.
## Migration approaches
Several approaches are available for migrating Fusion objects. This table summarizes the approaches.
| | Export an app | Import an app | Export an object | Import an object | Add an object to an app |
| - | - | - | - | - | - |
| **Fusion UI** | App configuration | Launcher (entire app) App configuration (combine apps) | - | - | Object Explorer |
| **Objects API** (endpoints) | GET from `export` endpoint | POST to `import` endpoint | GET from `export` endpoint | POST to `import` endpoint | - |
For more information about using the Objects API to export and import objects, see [Objects API](/api-reference/objects/get-objects-service-status).
The remainder of this topic describes approaches in the Fusion UI.
Use the parts of the Fusion UI indicated in the table to export and import apps and specific objects. Exporting creates a zip file. To import, you select a data file and possibly a variable file.
The approach with Object Explorer differs. With Object Explorer, you can add objects from other apps (or that are not linked to any apps) to the currently open app.
## Export an app with the Fusion UI
**How to export an app with the Fusion UI**
1. Navigate to the launcher.
2. Hover over the app you want to export and click the Configure icon:
3. In the app config window, click **Export app to zip**:
See import for information to import the downloaded zip file into other instances of Fusion 5.x.
## Import an app with the Fusion UI
**How to import an app with the Fusion UI**
1. Navigate to the launcher.
2. Click **Import app**.
3. Under **Data File**, click **Choose File** and select the zip file containing the app you want to import.
4. If your app has [usernames and passwords](/api-reference/objects/list-imported-variables) in a separate file, select it under **Variables File**.
If the Variables File is needed, it *must* be a separate file that is not in a .zip file. It is a .json map of variables to values. The following is an example:
```json theme={"dark"}
{
"secret.dataSources.Inventory_BR_S3_DS.password":"SOMETHING",
"secret.dataSources.LLM_A_BR_S3_DS.password":"FmJSaDE9Tj5REDACTED",
"secret.dataSources.LLM_BR_Inventory_S3_DS.password":"FmJSaDE9Tj5GzIVvethAC4Huh",
"secret.dataSources.LLM_BR_Load_S3_DS.password":"FmJSaDE9Tj5GzIVvethAC4"
}
```
5. You can sometimes edit parameter values to use the new values in the imported app. If this is the case, Fusion displays a dialog box that lets you edit the parameter values.
Make desired changes, and then click **Import**.
## Copy an app
To copy an app from one deployment to a different one, [export the app](#export-an-app-with-the-fusion-ui) on the source deployment, and then [import the app](#import-an-app-with-the-fusion-ui) on the target deployment.
## Import objects into an app
You can import objects into the currently open app.
**How to import objects into an open app**
1. In the Fusion launcher, click the app into which you want to import objects.
The Fusion workspace appears.
2. Click **System > Import Fusion Objects**.
The Import Fusion Objects window opens.
3. Select the data file from your local filesystem.
If you are importing [usernames and passwords](/api-reference/objects/import-objects) in a separate file, select it under **Variables File**.
4. Click **Import**.
If there are conflicts, Fusion prompts you to specify an import policy:
* Click **Overwrite** to overwrite the objects on the target system with the ones in the import file.
* Click **Merge** to skip all conflicting objects and import only the non-conflicting objects.
* Click **Start Over** to abort the import.
Fusion confirms that the import was successful:
5. Click **Close** to close the Import Fusion Objects window.
## Add an object to an app
You can add objects present in other apps (or in no apps) to the open app. Some objects are linked to other apps. You can also add those directly to an app.
* **Add an object to an app** – While in the Fusion workspace for the app to which you want to add an object, open Object Explorer and click **In Any App**. Search for or browse to the object you want to add. Hover over the object, click the App icon, and then click **Add to this app**.
* **Add an object to an app directly** – In cases when an object is linked to an app, but is not linked directly to the app (it is linked via some dependency), you can add the object to an app directly.
While in the Fusion workspace for the app to which you want to add an object directly, open Object Explorer and click **In Any App**. Search for or browse to the object you want to add. Hover over the object, click the App icon, and then click **Add to this app directly**.
In Fusion you can download and import a query pipeline that works out of the box to fetch items-for-user recommendations generated by the default [BPR Recommender](/docs/5/fusion/reference/config-ref/jobs/bpr-recommender) job.
There are two separate pipelines attached below which work in different ways to query and return recommendations.
**Query for recommendations only**
1. Download the `APPName_item_user_rec_pipelines_bpr.json` file.
2. Rename the file to replace `APPName` with the name of your Fusion app, such as `ProductCatalog_item_item_rec_pipelines_bpr.json`.
3. Open the JSON file, replace all instances of `APPName` with the name of your Fusion app, such as `ProductCatalog`, and save it.
4. Import the JSON file into your Fusion instance using the Query Pipelines REST API:
```
curl -u USERNAME:PASSWORD /api/query-pipelines -XPOST -H 'content-type:application/json' -d@
```
5. In the Fusion UI, navigate to **Query** > **Query Pipelines** to verify that the new pipeline is available.
This pipeline should be used to query the collection where the recommendations are stored. It makes a query against the `userId` field and only returns the recommended `itemId` values. To get the actual items, you need to make a second query to the respective catalog collection with the returned `itemId` values.
No additional configuration is needed to use this pipeline with the default BPR job configuration.
**Boost recommended products from catalog**
1. Download the `APPName_items_for_user_bpr_boost.json` file
2. Rename the file to replace `APPName` with the name of your Fusion app, such as `ProductCatalog_items_for_user_bpr_boost.json`.
3. Open the JSON file, replace all instances of `APPName` with the name of your Fusion app, such as `ProductCatalog`.
4. Fill in the `collection` name field in the first `Recommend Items for User` stage and save the file.
5. Import the JSON file into your Fusion instance using the Query Pipelines REST API:
```
curl -u USERNAME:PASSWORD /api/query-pipelines -XPOST -H 'content-type:application/json' -d@
```
6. In the Fusion UI, navigate to **Query** > **Query Pipelines** to verify that the new pipeline is available.
This pipeline queries the recommendations collection and then makes a subsequent query to the actual catalog collection boosting the recommended items and returning the actual items from the catalog. This pipeline will therefore also return recommendations even if none were generated/available.This pipeline expects a request parameter called `id=` to be appended to the request in order to work. An example query URL to this pipeline would look like `https://example-recs-fusion.com/api/query-pipelines/APPName_items_for_user_bpr_boost/collections/catalog/select?q=**:**&id=SomeUserId`If the pipeline does not appear in the **Query Pipelines** panel, you may need to attach it to your app like this: Go to **System** > **Object Explorer**, click the **In No Apps** filter, hover over the pipeline, click the icon, and select **Add to this app**.
**See also**
* [Items-For-User Recommendations](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommend-items-for-user-query-stage)
* [BPR Recommender job](/docs/5/fusion/reference/config-ref/jobs/bpr-recommender)
* [Getting Started with Recommendations and Boosting](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/getting-started)
# Recommendations
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/recommendations/overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/recommendations/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/recommendations/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/3241
[Signals](/docs/5/fusion/getting-data-out/query-enhancement/signals/overview) contain data about how users interact with search results. Once your Fusion app has accumulated a sufficient number of [aggregated signals](/docs/5/fusion/reference/config-ref/jobs/aggregations/overview), they become useful for automatically producing recommendations and boosts for better relevance and higher conversion rates.
The same data used to produce recommendations can also be used for automatic boosting. So although recommendations and boosts are different (as explained below), these topics use "recommender data" to refer to the data that can be used for boosting as well as for recommendations.
## Recommendations vs boosts
Recommendations and boosts are two ways of presenting AI-powered estimations about which items are most likely to interest a user:
| Recommendations | Boosts |
| - | - |
|
Personalized search results from a special, automatic query, *whether or not the user has performed a query*. Recommendations can be based on a variety of criteria, as in the examples below: • "Based on your purchase history, you might be interested in these items." • "People who viewed this item also viewed these items." • "People who searched for 'ipad' also searched for these items."
|
Modifications to the *scores* of the items in a query’s search results. The set of search results does not change, but their ranking does. For example: • Boost the most popular items • Boost items this user has clicked before • Boost seasonal items Boosts can also be configured manually, using the [Query Workbench](/docs/lucidworks-search/03-ui-tour/query-workbench) or the [Query Rewriting UI](/docs/lucidworks-search/07-improve-your-queries/business-rules/overview). You can also perform boosts within a set of recommendations.
|
These topics explain how to configure and use recommendations and boosts:
* [Getting Started](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/getting-started) shows you how to quickly enable the basic feature set and see some results.
* [Recommendation Methods](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods) explains the available approaches to recommendations and boosting, including methods that do not rely on signals.
## Recommendations data flow
The diagram below shows the flow of data between the [default objects](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/getting-started#default-objects-for-recommendations) created when you [enable recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/getting-started#enable-recommendations). You can create additional object for different [recommendation methods](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods).
1. *Signals data* is aggregated into a format that can be consumed by recommender jobs.\
Your search application should be sending [well-formed signals data](/docs/5/fusion/getting-data-out/query-enhancement/signals/overview) to Fusion whenever there is user activity on your site.\
By default, the `COLLECTION_NAME_click_signals_aggregation` job runs every 15 minutes while the `_user_item_preferences_aggregation` and `_user_query_history_aggregation` jobs run once per day.\
You can use signals data for boosting, without enabling recommendations, using the [Boost with Signals](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommendation-boosting-query-stage) query pipeline stage. The [default query pipeline](/docs/5/fusion/getting-data-out/query-basics/query-pipelines/overview) includes this stage.
2. *Recommender data* is produced when recommender jobs analyze aggregated signals.\
The default `COLLECTION_NAME_item_recommendations` job must be manually [scheduled](/docs/5/fusion/operations/jobs-and-scheduling/overview) or run on demand. Make sure that the `COLLECTION_NAME_user_item_preferences_aggregation` job runs first.
3. Query pipelines retrieve recommender data to produce recommendations and boosts.\
When recommendations are enabled, Fusion creates two query pipelines for recommendations:
**`COLLECTION_NAME_items_for_user_recommendations`** `COLLECTION_NAME_items_for_item_recommendations`
## Recommendations storage requirements
Collaborative recommendations are derived from data generated by Fusion [jobs](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/getting-started#jobs-and-schedules) and stored in special collections. You can estimate the required storage for this data if you know the following:
* the number of unique users in the signals collection (`num_users`)\
To find the number of unique values of the `user_id` field, navigate to **Analytics** > **App Insights** > **Analytics** > **Users**. The **Number of Distinct Data Values** panel displays the number of unique users:
If no value is displayed here or the value is not accurate, see [Important fields for signals](/docs/5/fusion/getting-data-out/query-enhancement/signals/overview) to verify that your signals data is well-formed.
* the number of unique items in the main collection (`num_items`)\
Navigate to **Collections** > **Collections Manager** to see the total number of documents in your main collection.
* the number of recommendations to be computed per user (`num_recs_per_user`)\
This is a configuration parameter in the recommender job, with a default value of 10.
* the number of item similarities to be computed per item (`num_item_sims`)\
This is a configuration parameter in the [recommender job](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/getting-started#jobs-and-schedules), with a default value of 10.
Every time the recommender job runs, this many new recommender documents are created:
```js theme={"dark"}
num_users * num_recs_per_user + num_items * num_item_sims
```
Each of these documents is about 300 bytes. This is the minimum size; they can be larger if you configure the recommender job to pull item metadata fields from the main collection for inclusion in recommender documents. These fields are useful when displaying recommendations in your front-end search application.
In your recommender jobs, verify that **Delete Old Recommendations** is selected so that only the latest recommender data is stored.
## Learn more
The course for **Recommendations** focuses on how Fusion uses signals to automatically produce custom recommendations to offer users the items most relevant to their experience.
When you enable recommendations on a per-collection basis:
A set of default objects is created, including jobs you can schedule and pipelines you can configure to fetch the recommendation data.
Signals are enabled.
## How to enable recommendations for a collection
1. From the collection picker, select the collection for which you want to enable recommendations, if it is not selected already.
2. Navigate to **Relevance > Recommendations**.\
The Enable Recommendations window appears, displaying the list of objects that will be created when you enable recommendations.\
If recommendations are already enabled for this collection, then the Object Explorer appears, displaying the recommendations-related objects in this app.
3. Click Enable Recommendations.
It may take a few moments for Fusion to create the default objects.
# Queries-for-Query Recommendations
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/recommendations/queries-for-query
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/recommendations/queries-for-query
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/recommendations/queries-for-query
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/479
Queries-for-query recommendations are queries performed by other users who also performed the current query. For example, when a user searches for "madonna", your app may also suggest searches for "evita", "a league of their own", "lady gaga", and so on.
The "Suggested Searches" shown below are one example of queries-for-query recommendations:
**To enable queries-for-query recommendations, use the [Query-to-Query Session-Based Similarity](/docs/5/fusion/reference/config-ref/jobs/query-to-query-session-based-similarity) job, which is session-based unlike the [Synonym Detection job](/docs/5/fusion/reference/config-ref/jobs/synonym-detection).**
The Query-to-Query Similarity recommender job was deprecated in Fusion 5.2.
The job generates two sets of recommendations based on the two approaches described below, then merges and de-duplicates them to present unique query-recommender pairs.
| Similar queries based on documents clicked | Similar queries based on co-occurrence in sessions |
| - | - |
| Queries are considered for recommendation if two queries have similar sets of document IDs clicked according to the signals data. This is directly implemented from the similar queries portion of the [Synonym Detection job](/docs/5/fusion/reference/config-ref/jobs/synonym-detection). This approach can work on both raw and aggregated signals. | Queries are considered for recommendation if two queries have co-occurred in the same session based on the assumption that users search for similar items in a single search session (this may or may not hold true depending on the data). This approach, based on session/user IDs, needs raw signals to work. |
```mermaid theme={"dark"}
flowchart LR
accTitle: Query-to-query session-based similarity job dataflow
accDescr: A flowchart showing the Query-Query Session Based Similarity Job processing signals and populating the queries_query_recs collection used by the query pipeline
UI["UI"]
subgraph profile["Query Profile"]
Pipe["queries_query_recs Query Pipeline"]
end
Col[("queries_query_recs collection")]
Job["Query-Query Session Based Similarity Job (Experimental)"]
Sig[("signals collection")]
UI <--> profile
Pipe <--> Col
Sig --> Job <--> Col
```
A default Query-to-Query Session-Based Similarity job (`COLLECTION_NAME_query_recs`) and a dedicated [collection](#the-similar-queries-collection) and [pipeline](#the-query-pipeline) are created when you [enable recommendations for a collection](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/getting-started).
At a minimum, you must configure these:
* an ID for this job
* the input collection containing the signals data, usually `COLLECTION_NAME_signals`
* the data format, usually `solr`
* the query field name, usually `query_s`
* the document ID field name, usually `doc_id_s`
* optionally, the user session ID or user ID field name
If this field is not specified, then the job generates click-based recommendations only, without session-based recommendations.
## Data tips
* Running the job on other types of data than signals is not recommended and may yield unexpected results.
* To get about 90% query coverage with the query pipeline, we recommend a raw signals dataset of about \~170k unique queries. More signals will generally improve coverage.
* On a raw signal dataset of about 3 million records, the job finishes execution in about 7-8 minutes on two executor pods with one CPU and about 3G of memory each. Your performance may vary depending on your configuration.
## Boosting recommendations
Generally if a query and recommendation has some token overlap, then they’re closely related and we want to highlight these. Therefore, query-recommendation pair similarity scores can be boosted based on token overlap. This overlap is calculated in terms of the number or fractions of tokens that overlap.
For example, consider the pair (“a red polo shirt”, “red polo”). If the minimum match parameter is set to 1, then there should be 1 token in common. For this example there is 1 token in common and therefore it is boosted. If it is set to 0.5, then at least half of the tokens from the shorter string (in terms of space separated tokens) should match. Here, the shorter string is “red polo” which is 2 tokens long. Therefore, to satisfy the boosting requirement, at least 1 token should match.
## Tuning tips
These tuning tips pertain to the advanced Model Tuning Parameters:
* **Special characters to be filtered out.** Special characters can cause problems with matching queries and are therefore removed in the job.
Only the characters are removed, not the queries, so a query like `ps3$` becomes `ps3`.
* **Query similarity threshold.** This is for use by the similar queries portion of the job and is the same as that used in the Synonym and Similar Queries Detection job.
* **Boost on token overlap.** This enables or disables boosting of query recommendation pairs where all or some tokens match. How much match is required to boost can be configured using the next parameter.\
For example, if this is enabled, then a query-recommendation pair like `(playstation 3, playstation console)` is boosted with a similarity score of 1, provided the minimum match is set to 1 token or 0.5.
* **Minimum match for token overlap.** Similar to the `mm` param in Solr, this defines the number/fraction of tokens that should overlap if boosting is enabled. Queries and recommendations are split by “ “ (space) and each part is considered a token. If using a less-than sign (\<), it must be escaped using a backslash.\
The value can be an integer, such as 1, in which case that many tokens should match. So in the previous example, pair is boosted because the term “playstation” is common to both and the `mm` value is set to 1.\
The value can also be a fraction, in which case that fraction of the shorter member of the query and recommendation pair should match. For example, if the value is set to 0.5 and query is 4 tokens long and recommendation is 6 tokens long then there should be at least 2 common tokens between query and recommendation.\
Here the stopwords specified in the [list of stopwords](/docs/5/fusion/getting-data-out/query-enhancement/stopwords-files) are ignored while calculating the overlap.
* **Query clicks threshold.** The minimum number of clicked documents needed for comparing queries.
* **Minimum query-recommendation pair occurrence count.** Minimum limit for the number of times a query-recommendation pair needs to be generated to make it to the final similar query recommendation list. Default is set to 2. Higher value will improve quality but reduce coverage.
## The similar queries collection
The following fields are stored in the `COLLECTION_NAME_queries_query_recs` collection:
* `query_t`
* `recommendation_t`
* `similarity_d`, the similarity score
* `source_s`, the approach that generated this pair, one of the following: `SessionBased` or `ClickedDocumentBased`
* `query_count_l`, the number of times the query occurred in signals
* `recommendation_count_l`, the number of times recommendations occurred in signals
* `pair_count_l`, the number of instances of the pair generated in the final recommendations using either of the approaches
* `type_s`, always set to `similar_queries`
## The query pipeline
When you enable recommendations, a default query pipeline, `COLLECTION_NAME_queries_query_recs`. is created.
# Methods for Recommendations and Boosting
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/475
This topic provides an overview of different methods for generating recommendations and boosts. "Content-based" and "collaborative" recommendations are two broad categories. More specific recommendation methods are also explained.
## Content-based vs collaborative recommendations
Recommendation methods can be divided into two broad categories: collaborative recommendations and content-based recommendations.
Collaborative recommendations always require signals. Some also require a job that pre-computes a matrix of values that can be queried. When these values are used for boosting, then certain query pipeline stages are needed as well.
**Collaborative recommendations** These methods require aggregated signals to analyze how users have interacted with documents.
| Method | Required job | Boosting stage |
| - | - | - |
| [Items-for-item](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-item) | [BPR Recommender](/docs/5/fusion/reference/config-ref/jobs/bpr-recommender) | Download the `APPName_item_item_rec_pipelines_bpr.json` |
| [Items-for-user](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-user) | [BPR Recommender](/docs/5/fusion/reference/config-ref/jobs/bpr-recommender) | [Recommend Items for User](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommend-items-for-user-query-stage) |
| [Items-for-query](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-query) | [Query-to-Query Session-Based Similarity jobs](/docs/5/fusion/reference/config-ref/jobs/query-to-query-session-based-similarity) | |
| [Queries-for-query](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/queries-for-query) | [Query-to-Query Session-Based Similarity jobs](/docs/5/fusion/reference/config-ref/jobs/query-to-query-session-based-similarity) | |
| [Boost with signals](/docs/5/fusion/getting-data-out/query-enhancement/boosting/boost-with-signals) | | [Boost with Signals](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommendation-boosting-query-stage) |
| [Trending items or queries](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/trending-items) | [Trending Recommender](/docs/5/fusion/reference/config-ref/jobs/trending-recommender) | |
| [Users-for-item recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/users-for-item) | [BPR Recommender](/docs/5/fusion/reference/config-ref/jobs/bpr-recommender) | |
**Content-based recommendations** These methods analyze the content of documents or queries.
| Method | Required job | Boosting stage |
| - | - | - |
| [Items-for-item](#items-for-item-recommendations) | [Content-Based Recommender](/docs/5/fusion/reference/config-ref/jobs/content-based-recommender) | Download the `APPName_item_item_rec_pipelines_content.json` |
| [Boost documents](#boost-documents) | | [Boost Documents](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/boost-documents-query-stage) |
| [Parameterized boosting](#parameterized-boosting) | | [Parameterized Boosting](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/advanced-boosting-query-stage) |
| [More Like This](#more-like-this) | | [Solr MoreLikeThis](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/solr-more-like-this-query-stage) |
| [Business rules](#business-rules) | | [Apply Rules](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/query-rules-query-stage) |
## Fusion recommendation and boosting methods
This section provides an overview of each method, with links to more detailed topics.
| Collaborative recommendations | Content-based recommendations |
| - | - |
| ● [Items-for-item](#items-for-item-recommendations) ● [Items-for-user](#items-for-user-recommendations) ● [Items-for-query](#items-for-query-recommendations) ● [Queries-for-query](#queries-for-query-recommendations) ● [Boost with signals](#boost-with-signals) ● [Trending items or queries](#trending-items-or-queries) ● [Users-for-item](#users-for-item-recommendations) | ● [Items-for-item](#items-for-item-recommendations) ● [Boost Documents](#boost-documents) ● [Parameterized Boosting](#parameterized-boosting) ● [More Like This](#more-like-this) ● [Business rules](#business-rules) |
When you [enable recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/getting-started#enable-recommendations), this automatically enables the items-for-user and items-for-item recommendation methods. To use additional recommendation methods, you must configure them separately.
### Items-for-item recommendations
Items-for-item recommendations present items that are similar to a specified item. For example, when the user is viewing a BMX bicycle, Fusion can recommend other BMX bicycles. Similarity can be based on different criteria, such as click patterns, people who bought this also bought that, percentage match of document tags, and so on.
See [Items-for-Item Recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-item) for details.
### Items-for-user recommendations
Items-for-user recommendations use the [Recommend Items for User](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommend-items-for-user-query-stage) query stage to present items that are similar to ones in which the user has previously shown interest, based on the user’s search history, browsing history, purchase history, and so on. This is one type of [collaborative recommendation](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods#content-based-vs-collaborative-recommendations).
See [Items-for-User Recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-user) for details.
### Items-for-query recommendations
The Items for Query recommender is the primary algorithm that supports recommendations on the search results page.
Items-for-query recommendations are based on how other users interacted with search results from the same query. This is one type of [collaborative recommendation](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods#content-based-vs-collaborative-recommendations).
For example, if users have searched for "titanic" in the past and many of them clicked on the search result for the DVD of the movie *Titanic* (as opposed to books or memorabilia), then this method boosts the DVD item on subsequent searches for "titanic".
See [Items-for-Query Recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-query) for details.
### Queries-for-query recommendations
Queries-for-query recommendations are queries performed by other users who also performed the current query. For example, when a user searches for "madonna", your app may also suggest searches for "evita", "a league of their own", "lady gaga", and so on.
See [Queries for Query](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/queries-for-query) for details.
### Boost with signals
The Boost With Signals query pipeline stage performs automatic boosts based on the contents of the aggregated signals collection. For this type of boosting, signals must be enabled but recommendations need not be.
See [Boost With Signals](/docs/5/fusion/getting-data-out/query-enhancement/boosting/boost-with-signals) for details.
### Trending items or queries
The [Trending Recommender job](/docs/5/fusion/reference/config-ref/jobs/trending-recommender) analyzes signals to measure customer engagement over time. Use this job to identify spikes in popularity for specific items or queries, then display those items to your users or analyze the trends for business purposes. You can configure any time window, such as daily, weekly, or monthly.
See [Trending items or queries](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/trending-items) for details.
### Users-for-item recommendations
Users-for-item recommendations retrieve the set of users who have interacted with an item, weighted by the number of interactions. This can be useful for community-driven use cases or for marketing campaigns. For example, on your organization’s intranet, clicking a search result could display the item plus a list of colleagues who have interacted with the same item.
See [Users-for-Item Recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/users-for-item) for details.
### Boost documents
The Boost Documents query pipeline stage adds boosting parameters to matched documents based on user-defined rules. Boosts are defined with a term value to boost and the boost factor to add. The boosting parameters are added to the `bq` Solr query parameter.
See [Boost Documents](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/boost-documents-query-stage) for details.
### Parameterized boosting
The Parameterized Boosting query pipeline stage reads the `boostValues` (in `List` format) from the context variable (added by a [Rollup Aggregation stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/rollup-aggregator-query-stage) or a [JavaScript stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/javascript-query-stage)), and adds boosts to the main query using `bq` or `boost` based on the stage configuration. The weights for the boost values can also be scaled.
See [Parameterized Boosting](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/advanced-boosting-query-stage) for details.
### More Like This
This stage uses the content of the current document to query for similar documents, using [Solr’s MoreLikeThis component](https://solr.apache.org/guide/solr/latest/query-guide/morelikethis.html).
This stage provides content-based recommendations. For collaborative recommendations, use the [Recommend Items for Item stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommend-items-for-item-query-stage).
See [Solr MoreLikeThis](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/solr-more-like-this-query-stage) for details.
### Business rules
Business rules are manually-created formulas for rewriting queries. This is the most versatile strategy for creating custom query rewrites. It supports a variety of conditions and actions to address a wide range of use cases. When you need a very specific query rewrite, this is the best strategy.
Business rules are applied in the [Apply Rules stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/query-rules-query-stage) of the query pipeline.
See [Business Rules](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/overview) to learn how to create, edit, and publish business rules. See [Query Rewriting](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) for additional query rewriting methods.
## BPR algorithm
The Bayesian Personalized Ranking (BPR) algorithm is used to configure items-for-item, items-for-user, or user-for-item recommendations. The BPR job generates effective results in a short run time. See the [BPR Recommender](/docs/5/fusion/reference/config-ref/jobs/bpr-recommender) for more information.
# Trending Items and Queries
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/recommendations/trending-items
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/recommendations/trending-items
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/recommendations/trending-items
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/crfvw5
The [Trending Recommender job](/docs/5/fusion/reference/config-ref/jobs/trending-recommender) analyzes signals to measure customer engagement over time. Use this job to identify spikes in popularity for specific items or queries, then display those items to your users or analyze the trends for business purposes. You can configure any time window, such as daily, weekly, or monthly.
For detailed steps to configure this job, see **Identify Trending Items or Queries**.
The Trending Recommender job analyzes signals to measure customer engagement over time. Use this job to identify spikes in popularity for specific items or queries, then display those items to your users or analyze the trends for business purposes. You can configure any time window, such as daily, weekly, or monthly.
For complete details about the job’s configuration options, see [Trending Recommender Jobs](/docs/5/fusion/reference/config-ref/jobs/trending-recommender).
**How to identify trending items or queries**
1. Navigate to **Collections** > **Jobs** > **Add +** > **Trending Recommender**.
2. Configure the job:
1. Enter an ID for this job.
2. In the **Reference Time Days** field, enter the number of days to use as a *baseline* for identifying trends, starting from today.\
For example, enter 21 days to analyze three weeks of signals data to use as a baseline.
3. In the **Target Time Days** field, enter the number of days to use as a *target* for identifying trends, starting from today.\
For example, enter 7 days to get documents or products whose popularity has spiked in the past week.
**Reference Time Days** and **Target Time Days** do not overlap. For example, with the values suggested above, a total of 28 days of signals are analyzed, and the first 21 days are compared to the last 7 days.
4. If you want to identify trending queries instead of trending items, change the value of the **Document ID Field** from `doc_id_s` to `query_s`.
This field must be present in your signals. See the required signals fields in the [Trending Recommender Jobs](/docs/5/fusion/reference/config-ref/jobs/trending-recommender) reference topic.
5. In the **Training Collection** field, enter the Solr collection or cloud path where signals are stored (the `COLLECTION_NAME_signals` collection by default).
6. In the **Output Collection** field, enter the Solr collection or cloud path where trend analysis data will be stored.
7. If you are using a format other than `solr`, enter it in the **Data Format** field.
8. In the **Solr Fields to Read** field, enter one or more field names containing text training data.
You can enter multiple field names with weights, as in `field1:weight1,field2:weight2...`.
9. In the **Event Count Field Name** field, enter the name of the event count field in your training data, usually `count_i`.
3. Click **Save**.
4. Click **Run** > **Start** to run the job.
The job outputs documents similar to this example:
```json theme={"dark"}
{
"doc_id":250,
"ref_hits":1,
"ref_rank":1,
"trgt_hits":1,
"trgt_rank":1,
"vol_diff":0.5,
"average_weekly_vol":0.5,
"hit_vol_ratio":2.0,
"combine_score":1.0,
"vol_diff_ratio":1.0,
"ref_wt_vol_diff_ratio":1.0,
"vol_diff_wt_vol_diff_ratio":0.5,
"log_diff_wt_ratio":1.3068528194400546,
"trend-type":"prds_weekly",
"id":"284f930d-d750-49a2-90ac-be4692bddda9",
"_version_":1682995654191742976
}
```
5. Configure a query pipeline to retrieve trending items from the job’s output collection for display or further analysis. For information about pipelines created when recommendations are enabled, see [enable recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/getting-started).
Search on the `log_diff_wt_ratio` field to find the top trending items in the output collection.
# Users-for-Item Recommendations
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/recommendations/users-for-item
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/recommendations/users-for-item
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/recommendations/users-for-item
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/478
Users-for-item recommendations retrieve the set of users who have interacted with an item, weighted by the number of interactions. This can be useful for community-driven use cases or for marketing campaigns. For example, on your organization’s intranet, clicking a search result could display the item plus a list of colleagues who have interacted with the same item.
This is one type of [collaborative recommendation](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods#content-based-vs-collaborative-recommendations).
## Learn more
To enable this recommendation method, you only need to create a new collection and modify the configuration of the default recommender job, as explained below.
1. How to enable users-for-item recommendations
2. Navigate to **Collections** > **Collections Manager**.
3. Click **New**
4. Enter a collection name, such as `_users_for_item_recommendations`.
5. Click **Save Collection**.
You can safely disable signals for this new collection.
6. In your main collection, verify that signals and recommendations are enabled.
7. Navigate to **Collections** > **Jobs**.
8. Open the `COLLECTION_NAME_item_recommendations` job.\
Alternatively, you can create a BPR Recommender job.
9. Click **Advanced**.
10. In the **Users-for-items Recommendation Collection** field, enter the collection name `COLLECTION_NAME_users_for_item_recommendations`.
11. Click **Save**.
# Remove Words
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/remove-words
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/remove-words
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/remove-words
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/z6tv3f
Use a Remove Words query rewrite to remove particular phrases from queries. Unlike other rewrites, Remove Words rules are entered manually and aren’t generated by a job.
You can create a Remove Words query rewrite to remove words from a query. This query rewrite is helpful when a word in the search query does not add value to the search results.
For example, you can rewrite a search query for `case study examples` to remove `examples` and then display results for `case study`.
1. From the list of query rewrite options, select Remove Words. A form appears:
2. This form contains the following fields:
| Parameter | Description | Example Value |
| - | - | - |
| Phrase to remove | The words to remove from the trigger phrase. | `examples` |
| Trigger phrases | The query that prompts the removal of the phrase. The trigger phrase is not necessarily a complete query. If the query contains the trigger phrase, then Fusion removes the phrase in the Phrase to Remove field. | `case study examples` |
1. Enter a phrase to remove and a trigger phrase. Note that the phrase to remove is auto-populated with the query.
2. Click **Save**.
The search is re-run, and a Search box information panel drops down to show that the new query rewrite has fired:
To learn how to use Remove Words and other query rewrites in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor), see **Use Query Rewrites in the Rules Editor**.
This article describes how to use the Rewrites screen in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor).
## Query rewrite rule types
* **Head/Tail.** Replace underperforming queries with queries that produce higher click-through rates. Query improvements always require review before they are published. Schedule the Head/Tail Analysis job to get AI-generated query improvements based on your signals collection. \
\
Manually created Head/Tail rewrites must provide an "Improved Query" in a format appropriate to the type of request the rewrite will be applied to. For Legacy requests, this means a valid Solr query fit for use as the `q` parameter. For DSL requests this means a JSON string representing the main DSL query. \
\
For more information, see:
* [Underperforming Query Rewriting](/docs/5/fusion/getting-data-out/query-enhancement/underperforming-queries)
* [Head/Tail Analysis Jobs](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis)
* **Misspelling.** Fusion replaces misspelled query terms with their spelling corrections for better search results. Schedule the Token and Phrase Spell Correction job to get AI-generated spelling corrections based on your signals collection. For more information, see [Misspelling Detection](/docs/5/fusion/getting-data-out/query-enhancement/misspelling-detection).
* **Phrase.** Known phrases are boosted when they appear in new queries. Schedule the Phrase Extraction job to get AI-generated phrases based on your signals collection.
For more information, see [Phrase Detection](/docs/5/fusion/getting-data-out/query-enhancement/phrase-detection).
* **Synonym.** Synonyms are used to expand queries so that they include all known synonyms of the query terms. To retrieve AI-generated synonyms based on your signals collection, schedule the [Synonym Detection job](/docs/5/fusion/reference/config-ref/jobs/synonym-detection).
* **Remove Words** - Remove particular phrases from queries. Unlike other rewrites, these should be entered manually and are not generated by a job. For more information, see [Remove Words](/docs/5/fusion/getting-data-out/query-enhancement/remove-words).
## Facets
### Facet menu
To facet the rules list by **Status** or **Published**, click one of the facet items within the facet categories.
To facet the rules by **Confidence**, click and drag either slider to set the range of confidence you would like to see:
Faceting rules by confidence is not available for Head/Tail rewrites.
### Search
Enter a term in the search bar to quickly facet your query rewrite rules.
## Rules
Begin by selecting one of the query rewrite rule types you will create.
### Create a rule
1. Click **+Add**.
2. In the new table row, complete the required fields, and additional fields as needed.
3. Click the checkmark to save, or close to cancel the changes.
### Edit a rule
1. Hold the pointer over the rule to edit.
2. Click the pencil icon.
3. Enter desired changes and click the checkmark to save, or close to cancel the changes.
### Approve/deny rules
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To modify multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Approve** or **Deny** as needed.
### Publish a rule
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To publish multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Publish Selected** to publish the selected rules.
#### Publish an individual rule
1. Hold the pointer over a rule.
2. Click the button.
3. To confirm, click **Publish**.
### Delete rules
#### Delete a single rule
1. Hold the pointer over a rule.
2. Click the trash can.
3. To confirm, click **Delete**.
#### Delete multiple rules
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To delete multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Delete**.
5. To confirm, click **Delete** again.
### Sort rules
1. Hold the pointer over column titles. A icon displays for some titles to indicate the rules list can be sorted by that column.
2. Click the title column to sort the rules.
The column that is currently sorting the rules list is indicated by an or icon.
### Export rules
To export the rules list, click **Export**. This generates a `.csv` file that contains your rules.
## Columns
### Customize columns
To customize columns:
1. Hold the pointer over the row of column titles.
2. Click the button.
3. In the Customize Columns to Display screen, select the checkboxes for the columns you want to display. Clear the checkboxes for columns you do not want to display.
4. Click **Save**. To exit without saving your changes, click **Cancel**.
# Response Rewriting
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/response-rewriting
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/response-rewriting
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/response-rewriting
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/449
Response rewriting is a strategy for improving relevancy using AI-generated data by modifying Solr’s response before passing it to the search application. There are two approaches to configuring response rewriting:
## Rules
Any [rule type](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) can add content to the response by modifying the `responseValues` field. These two rule types can replace the whole response:
* Banner\
Display a user-defined banner message when the rule fires.
* Redirect\
Send users to a different URL instead of the search results.
See the main [Rules](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/overview) topic for details.
## Query pipeline stages
Query pipeline stages that perform response rewriting must appear *after* the [Solr Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/solr-query-stage). These stages fall into two categories:
* **Stages that act on the whole set of results**
* [Response Shuffle stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/response-shuffle-query-stage)\
"De-bias" results by shuffling the top N results randomly.
* [Response Pairwise Swap stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/response-pairwise-swap-query-stage)\
"De-bias" results by swapping the search results at any two positions, such as positions 1 and 2, positions 3 and 4, and so on.
* **Stages that act on individual documents**
* [Response Document Exclusion stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/response-document-exclusion-query-stage)\
Drop all documents that match all of the specified rules.
* [Modify Response with Rules stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/rules-augment-response-query-stage)
Apply [rules](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/overview) to the response.
# Rules Editor
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/rules-editor
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/rules-editor
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/rules-editor
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/491
The Rules Editor allows you to view, edit, create, approve, enable, and publish rules powered by Fusion. Access the Rules Editor from the Fusion UI by navigating to **Relevance** > **Rules**:
A rule’s various states determine whether the rule is in effect for everyone using your search application:
| State | Options | Description |
| - | - | - |
| **Status** | *Enabled* or *Disabled* | If a rule’s status is *Enabled*, the rule is in effect. If a rule’s status is *Disabled*, the rule is not in effect. This is only true if the rule is published, as described below. |
| **Review** | *Approved* or *Denied* | If a rule’s review state is *Approved*, and you publish the changes, the latest changes to that rule are in effect. If the state is *Denied*, and you publish the changes, the Rules Manager deletes the rule. **Caution**
Unlike deleting the rule with the UI, if you delete the rule by publishing it while in a *Denied* review state, you cannot undo the action. |
| **Published** | *Yes* or *No* | If a rule’s published state is *Yes*, the latest changes to the rule are in effect. If the state is *No*, the latest changes are not in effect. This also applies to changes to the rule’s status, such as disabling to rule. |
For more information:
* **Use the Analytics Screen in the Rules Editor**
* **Use Business Rules in the Rules Editor**
* **Use Query Rewrites in the Rules Editor**
This article describes how to use the [Analytics screen](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor) in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor).
Job configuration affects the data that displays on the dashboard. For example, the [`COLLECTION_NAME_session_rollup` job](/docs/5/fusion/reference/config-ref/jobs/aggregations/built-in-sql-aggregation-jobs) aggregates related user activity into a session signal that contains activity count, duration, and keywords (based on user search terms).
### UI elements
#### Data date range
* **Quick Range Selectors.** Click one of the quick range selectors to facet your data.
* **Calendar Selector.** Use the calendar selector to specify a start and end date and time facet for your data.
#### Visitor data
* **Data Cards**
* **Site Visits.** The total number of visits.
* **Total Searches.** The total number of searches performed.
* **Unique Visitors.** The number of unique visitors to your site.
* **Average Session.** The average length of time spent on your site.
* **Data Graph.** Detailed information for the selected data card.
## View data by date range
The Rules Editor analytics allows you to view data by a date range. To do this, you can either:
* Click one of the date range presets:
* Click the date range details to display and select your range on the calendar that appears:
## View additional card details
Click one of the analytics cards to view additional details for that card’s data.
## Configure Analytics cards
1. Hover over one of the cards.
2. Click the **Edit** button that appears:
3. Complete the form that appears.
4. Click the **Save** button.
If the data is empty, check to make sure the date range includes the data you want to display.
This article describes how to use the [Business Rules](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/overview) screen in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor).
The course for **Query Rewriting with Business Rules and Search Rewrites** focuses on how using business rules and search rewrites gives you the opportunity to boost conversions and create better customer experiences by improving your relevancy.
## Facet rules
### Facet menu
To facet the rules list by **Action**, **Status**, **Published**, or **Tags**, click one of the facet items within the facet categories.
To facet the rules list by **Created Date** or **Expiration Date**:
1. Click the date start value. A calendar will appear.
2. Click a date to select the start of your facet range.
3. Click the bottom date and select the end of your facet range.
### Search
Enter a term in the search bar to quickly facet your business rules.
## Rules
### Create a rule
1. Click the button in the top-right corner of the card. Alternatively, click the button in the bottom-left corner of the card. The **New Rule** dialog box will appear.
2. Complete the required fields, at a minimum.
| | | |
| - | - | - |
| General | Condition | Action |
| In the **General** column, only the **Name** field is required. Other fields are optional:
**Description** is an arbitrary string you can use to describe this rule.
**Rule Group** can be a user-defined group that you use to organize your rules.
**Tags** are another way to organize your rules. Tags appear as facets in the Business Rules interface, so you can filter the set of visible rules by tag.
**Priority** can be used to determine which rule should apply first if multiple rules are activated for the same request. Priority is an integer value, ranging from 1 to infinity, with the value 1 given lowest priority. If multiple rules have the same value, those rules are applied in a random order.
**Enabled** means that this rule is applied but not necessarily published. Disabling a rule helps ensure that it is not accidentally published.
| Conditions are triggers that activate the rule when they match the current date and time, query, or field values. See [Rule conditions](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) for more information. | A rule can take different types of actions when the specified conditions are met. See [Action types](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) for more information. You can also create [custom actions](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting). |
Hover over the icon for more information.
3. Once complete, click the button.
### Copy a rule
1. Hover over a rule. A row of buttons will appear on the right.
2. Click the button. The **Copy Rule** dialog box will appear.
3. At a minimum, change the title of the rule.
4. Once complete, click the button.
### Edit a rule
1. Hover over a rule. A row of buttons will appear on the right.
2. Click the button. The **Edit Rule** dialog box appears.
3. Make your desired changes to the rule.
4. Once complete, click the button.
### Approve/deny rules
1. Hover over a rule. A checkbox will appear on the left.
2. Click the checkbox to select the rule.
3. Repeat this process to select multiple rules. *(optional)*
4. Click the Approve button to approve the selected rules, or click the Deny button to deny the rules.
### Enable/disable a rule
1. Hover over a rule. A checkbox will appear on the left.
2. Click the checkbox to select the rule.
3. Repeat this process to select multiple rules. *(optional)*
4. Click the button to enable the selected rules, or click the button to disable the rules.
### Publish a rule
1. Hover over a rule. A checkbox will appear on the left.
2. Click the checkbox to select the rule.
3. Repeat this process to select multiple rules. *(optional)*
4. Click the button to publish the selected rules.
### Tag a rule
Tags are a powerful tool for organizing and publishing rules according to a keyword, such as `black-friday-sale`.
1. Hover over a rule. A checkbox will appear on the left.
2. Click the checkbox to select the rule.
3. Repeat this process to select multiple rules. *(optional)*
4. Click the button. A dialog box will appear.
5. Enter a tag. To enter multiple tags, press **Tab** after writing each tag.
6. Click the button to add the tag to the rule.
See [General Configuration Tags](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/general-config) for information about how to use tagnames in the request URL to fire specific rules.
### Delete rules
#### Delete a single rule
1. Hover over a rule. A row of buttons will appear on the right.
2. Click the button. A confirmation box will appear.
3. Click the button to confirm.
#### Delete multiple rules
1. Hover over a rule. A checkbox will appear on the left.
2. Click the checkbox to select the rule.
3. Repeat this process to select multiple rules. *(optional)*
4. Click the button to delete the selected rules. A confirmation box will appear.
5. Click the button to confirm.
### Sort rules
1. Hover over column titles. A icon will appear for some titles to indicate the rules list can be sorted by that column.
2. Click on the title column to sort the rules.
The column that is currently sorting the rules list is indicated by an or icon.
### Export rules
To export the rules list, simply click the button. This generates a `.csv` file containing your rules.
## Columns
### Customize columns
To customize the columns:
1. Hover over the row of column titles. A button will appear on the right.
2. Click the button. The **Customize Columns to Display** dialog box will appear.
3. Click the checkboxes to select or deselect the columns you want to display.
4. Click the button to save your changes.
This article describes how to use the Rewrites screen in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor).
## Query rewrite rule types
* **Head/Tail.** Replace underperforming queries with queries that produce higher click-through rates. Query improvements always require review before they are published. Schedule the Head/Tail Analysis job to get AI-generated query improvements based on your signals collection. \
\
Manually created Head/Tail rewrites must provide an "Improved Query" in a format appropriate to the type of request the rewrite will be applied to. For Legacy requests, this means a valid Solr query fit for use as the `q` parameter. For DSL requests this means a JSON string representing the main DSL query. \
\
For more information, see:
* [Underperforming Query Rewriting](/docs/5/fusion/getting-data-out/query-enhancement/underperforming-queries)
* [Head/Tail Analysis Jobs](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis)
* **Misspelling.** Fusion replaces misspelled query terms with their spelling corrections for better search results. Schedule the Token and Phrase Spell Correction job to get AI-generated spelling corrections based on your signals collection. For more information, see [Misspelling Detection](/docs/5/fusion/getting-data-out/query-enhancement/misspelling-detection).
* **Phrase.** Known phrases are boosted when they appear in new queries. Schedule the Phrase Extraction job to get AI-generated phrases based on your signals collection.
For more information, see [Phrase Detection](/docs/5/fusion/getting-data-out/query-enhancement/phrase-detection).
* **Synonym.** Synonyms are used to expand queries so that they include all known synonyms of the query terms. To retrieve AI-generated synonyms based on your signals collection, schedule the [Synonym Detection job](/docs/5/fusion/reference/config-ref/jobs/synonym-detection).
* **Remove Words** - Remove particular phrases from queries. Unlike other rewrites, these should be entered manually and are not generated by a job. For more information, see [Remove Words](/docs/5/fusion/getting-data-out/query-enhancement/remove-words).
## Facets
### Facet menu
To facet the rules list by **Status** or **Published**, click one of the facet items within the facet categories.
To facet the rules by **Confidence**, click and drag either slider to set the range of confidence you would like to see:
Faceting rules by confidence is not available for Head/Tail rewrites.
### Search
Enter a term in the search bar to quickly facet your query rewrite rules.
## Rules
Begin by selecting one of the query rewrite rule types you will create.
### Create a rule
1. Click **+Add**.
2. In the new table row, complete the required fields, and additional fields as needed.
3. Click the checkmark to save, or close to cancel the changes.
### Edit a rule
1. Hold the pointer over the rule to edit.
2. Click the pencil icon.
3. Enter desired changes and click the checkmark to save, or close to cancel the changes.
### Approve/deny rules
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To modify multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Approve** or **Deny** as needed.
### Publish a rule
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To publish multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Publish Selected** to publish the selected rules.
#### Publish an individual rule
1. Hold the pointer over a rule.
2. Click the button.
3. To confirm, click **Publish**.
### Delete rules
#### Delete a single rule
1. Hold the pointer over a rule.
2. Click the trash can.
3. To confirm, click **Delete**.
#### Delete multiple rules
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To delete multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Delete**.
5. To confirm, click **Delete** again.
### Sort rules
1. Hold the pointer over column titles. A icon displays for some titles to indicate the rules list can be sorted by that column.
2. Click the title column to sort the rules.
The column that is currently sorting the rules list is indicated by an or icon.
### Export rules
To export the rules list, click **Export**. This generates a `.csv` file that contains your rules.
## Columns
### Customize columns
To customize columns:
1. Hold the pointer over the row of column titles.
2. Click the button.
3. In the Customize Columns to Display screen, select the checkboxes for the columns you want to display. Clear the checkboxes for columns you do not want to display.
4. Click **Save**. To exit without saving your changes, click **Cancel**.
The course for **Experience Optimizer: Rules Manager** focuses on how to create and utilize business rules to create more relevant search results for your users.
## Signals data
* **Top Null Searches.** The top search terms that yielded 0 results. A null search is considered a poor user experience. Use **Use Query Rewrites in the Rules Editor** to reduce the likelihood of a visitor receiving 0 results.
* **Top Search Terms.** The most commonly searched terms.
* **Top Clicked Products.** The most commonly clicked products or documents. This may include searches, recommendations, static links, or outside referrals.
* **Top Clicked Searches.** The most common searches that resulted in a click.
* **Top Purchased Products.** The most commonly purchased products. This may include searches, recommendations, static links, or referrals.
* **Top Purchased Searches.** The most commonly purchased products that were identified from a search.
Use query rewrites in the Rules Editor to adapt your search results and guide visitors to top clicked and top purchased products.
This article describes how to use the Rewrites screen in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor).
## Query rewrite rule types
* **Head/Tail.** Replace underperforming queries with queries that produce higher click-through rates. Query improvements always require review before they are published. Schedule the Head/Tail Analysis job to get AI-generated query improvements based on your signals collection. \
\
Manually created Head/Tail rewrites must provide an "Improved Query" in a format appropriate to the type of request the rewrite will be applied to. For Legacy requests, this means a valid Solr query fit for use as the `q` parameter. For DSL requests this means a JSON string representing the main DSL query. \
\
For more information, see:
* [Underperforming Query Rewriting](/docs/5/fusion/getting-data-out/query-enhancement/underperforming-queries)
* [Head/Tail Analysis Jobs](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis)
* **Misspelling.** Fusion replaces misspelled query terms with their spelling corrections for better search results. Schedule the Token and Phrase Spell Correction job to get AI-generated spelling corrections based on your signals collection. For more information, see [Misspelling Detection](/docs/5/fusion/getting-data-out/query-enhancement/misspelling-detection).
* **Phrase.** Known phrases are boosted when they appear in new queries. Schedule the Phrase Extraction job to get AI-generated phrases based on your signals collection.
For more information, see [Phrase Detection](/docs/5/fusion/getting-data-out/query-enhancement/phrase-detection).
* **Synonym.** Synonyms are used to expand queries so that they include all known synonyms of the query terms. To retrieve AI-generated synonyms based on your signals collection, schedule the [Synonym Detection job](/docs/5/fusion/reference/config-ref/jobs/synonym-detection).
* **Remove Words** - Remove particular phrases from queries. Unlike other rewrites, these should be entered manually and are not generated by a job. For more information, see [Remove Words](/docs/5/fusion/getting-data-out/query-enhancement/remove-words).
## Facets
### Facet menu
To facet the rules list by **Status** or **Published**, click one of the facet items within the facet categories.
To facet the rules by **Confidence**, click and drag either slider to set the range of confidence you would like to see:
Faceting rules by confidence is not available for Head/Tail rewrites.
### Search
Enter a term in the search bar to quickly facet your query rewrite rules.
## Rules
Begin by selecting one of the query rewrite rule types you will create.
### Create a rule
1. Click **+Add**.
2. In the new table row, complete the required fields, and additional fields as needed.
3. Click the checkmark to save, or close to cancel the changes.
### Edit a rule
1. Hold the pointer over the rule to edit.
2. Click the pencil icon.
3. Enter desired changes and click the checkmark to save, or close to cancel the changes.
### Approve/deny rules
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To modify multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Approve** or **Deny** as needed.
### Publish a rule
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To publish multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Publish Selected** to publish the selected rules.
#### Publish an individual rule
1. Hold the pointer over a rule.
2. Click the button.
3. To confirm, click **Publish**.
### Delete rules
#### Delete a single rule
1. Hold the pointer over a rule.
2. Click the trash can.
3. To confirm, click **Delete**.
#### Delete multiple rules
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To delete multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Delete**.
5. To confirm, click **Delete** again.
### Sort rules
1. Hold the pointer over column titles. A icon displays for some titles to indicate the rules list can be sorted by that column.
2. Click the title column to sort the rules.
The column that is currently sorting the rules list is indicated by an or icon.
### Export rules
To export the rules list, click **Export**. This generates a `.csv` file that contains your rules.
## Columns
### Customize columns
To customize columns:
1. Hold the pointer over the row of column titles.
2. Click the button.
3. In the Customize Columns to Display screen, select the checkboxes for the columns you want to display. Clear the checkboxes for columns you do not want to display.
4. Click **Save**. To exit without saving your changes, click **Cancel**.
## Predictive Merchandiser and Experience Optimizer
Predictive Merchandiser and Experience Optimizer are both AI-powered tools that provide insights, customer recommendations, and search optimization in a single comprehensive product.
Predictive Merchandiser and Experience Optimizer are separate offerings from Fusion. See the [Predictive Merchandiser documentation](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/overview) for more information.
## Rules
The Rules screen is used to create [Business Rules](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/overview) within the Rules Editor.
Business rules are manually-created formulas for rewriting queries. This is the most versatile strategy for creating custom query rewrites. It supports a variety of conditions and actions to address a wide range of use cases. When you need a very specific query rewrite, this is the best strategy.
Business rules are applied in the [Apply Rules stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/query-rules-query-stage) of the query pipeline.
Learn how to use the **Use Business Rules in the Rules Editor**.
This article describes how to use the [Business Rules](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/overview) screen in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor).
The course for **Query Rewriting with Business Rules and Search Rewrites** focuses on how using business rules and search rewrites gives you the opportunity to boost conversions and create better customer experiences by improving your relevancy.
## Facet rules
### Facet menu
To facet the rules list by **Action**, **Status**, **Published**, or **Tags**, click one of the facet items within the facet categories.
To facet the rules list by **Created Date** or **Expiration Date**:
1. Click the date start value. A calendar will appear.
2. Click a date to select the start of your facet range.
3. Click the bottom date and select the end of your facet range.
### Search
Enter a term in the search bar to quickly facet your business rules.
## Rules
### Create a rule
1. Click the button in the top-right corner of the card. Alternatively, click the button in the bottom-left corner of the card. The **New Rule** dialog box will appear.
2. Complete the required fields, at a minimum.
| | | |
| - | - | - |
| General | Condition | Action |
| In the **General** column, only the **Name** field is required. Other fields are optional:
**Description** is an arbitrary string you can use to describe this rule.
**Rule Group** can be a user-defined group that you use to organize your rules.
**Tags** are another way to organize your rules. Tags appear as facets in the Business Rules interface, so you can filter the set of visible rules by tag.
**Priority** can be used to determine which rule should apply first if multiple rules are activated for the same request. Priority is an integer value, ranging from 1 to infinity, with the value 1 given lowest priority. If multiple rules have the same value, those rules are applied in a random order.
**Enabled** means that this rule is applied but not necessarily published. Disabling a rule helps ensure that it is not accidentally published.
| Conditions are triggers that activate the rule when they match the current date and time, query, or field values. See [Rule conditions](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) for more information. | A rule can take different types of actions when the specified conditions are met. See [Action types](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) for more information. You can also create [custom actions](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting). |
Hover over the icon for more information.
3. Once complete, click the button.
### Copy a rule
1. Hover over a rule. A row of buttons will appear on the right.
2. Click the button. The **Copy Rule** dialog box will appear.
3. At a minimum, change the title of the rule.
4. Once complete, click the button.
### Edit a rule
1. Hover over a rule. A row of buttons will appear on the right.
2. Click the button. The **Edit Rule** dialog box appears.
3. Make your desired changes to the rule.
4. Once complete, click the button.
### Approve/deny rules
1. Hover over a rule. A checkbox will appear on the left.
2. Click the checkbox to select the rule.
3. Repeat this process to select multiple rules. *(optional)*
4. Click the Approve button to approve the selected rules, or click the Deny button to deny the rules.
### Enable/disable a rule
1. Hover over a rule. A checkbox will appear on the left.
2. Click the checkbox to select the rule.
3. Repeat this process to select multiple rules. *(optional)*
4. Click the button to enable the selected rules, or click the button to disable the rules.
### Publish a rule
1. Hover over a rule. A checkbox will appear on the left.
2. Click the checkbox to select the rule.
3. Repeat this process to select multiple rules. *(optional)*
4. Click the button to publish the selected rules.
### Tag a rule
Tags are a powerful tool for organizing and publishing rules according to a keyword, such as `black-friday-sale`.
1. Hover over a rule. A checkbox will appear on the left.
2. Click the checkbox to select the rule.
3. Repeat this process to select multiple rules. *(optional)*
4. Click the button. A dialog box will appear.
5. Enter a tag. To enter multiple tags, press **Tab** after writing each tag.
6. Click the button to add the tag to the rule.
See [General Configuration Tags](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/general-config) for information about how to use tagnames in the request URL to fire specific rules.
### Delete rules
#### Delete a single rule
1. Hover over a rule. A row of buttons will appear on the right.
2. Click the button. A confirmation box will appear.
3. Click the button to confirm.
#### Delete multiple rules
1. Hover over a rule. A checkbox will appear on the left.
2. Click the checkbox to select the rule.
3. Repeat this process to select multiple rules. *(optional)*
4. Click the button to delete the selected rules. A confirmation box will appear.
5. Click the button to confirm.
### Sort rules
1. Hover over column titles. A icon will appear for some titles to indicate the rules list can be sorted by that column.
2. Click on the title column to sort the rules.
The column that is currently sorting the rules list is indicated by an or icon.
### Export rules
To export the rules list, simply click the button. This generates a `.csv` file containing your rules.
## Columns
### Customize columns
To customize the columns:
1. Hover over the row of column titles. A button will appear on the right.
2. Click the button. The **Customize Columns to Display** dialog box will appear.
3. Click the checkboxes to select or deselect the columns you want to display.
4. Click the button to save your changes.
### UI elements
#### Facets
* **Action.** The [built-in rule action](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/overview) for the rule.
* **Status.** Whether a rule is enabled or disabled.
* **Published.** Whether or not a rule has been published.
* **Created Date.** The date the rule was created.
* **Expiration Date.** The date the rule will expire.
#### Rules list
The rules list can be faceted, searched, or sorted according to the various values the rules hold.
## Rewrite
Learn how to use the **Use Query Rewrites in the Rules Editor**.
This article describes how to use the Rewrites screen in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor).
## Query rewrite rule types
* **Head/Tail.** Replace underperforming queries with queries that produce higher click-through rates. Query improvements always require review before they are published. Schedule the Head/Tail Analysis job to get AI-generated query improvements based on your signals collection. \
\
Manually created Head/Tail rewrites must provide an "Improved Query" in a format appropriate to the type of request the rewrite will be applied to. For Legacy requests, this means a valid Solr query fit for use as the `q` parameter. For DSL requests this means a JSON string representing the main DSL query. \
\
For more information, see:
* [Underperforming Query Rewriting](/docs/5/fusion/getting-data-out/query-enhancement/underperforming-queries)
* [Head/Tail Analysis Jobs](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis)
* **Misspelling.** Fusion replaces misspelled query terms with their spelling corrections for better search results. Schedule the Token and Phrase Spell Correction job to get AI-generated spelling corrections based on your signals collection. For more information, see [Misspelling Detection](/docs/5/fusion/getting-data-out/query-enhancement/misspelling-detection).
* **Phrase.** Known phrases are boosted when they appear in new queries. Schedule the Phrase Extraction job to get AI-generated phrases based on your signals collection.
For more information, see [Phrase Detection](/docs/5/fusion/getting-data-out/query-enhancement/phrase-detection).
* **Synonym.** Synonyms are used to expand queries so that they include all known synonyms of the query terms. To retrieve AI-generated synonyms based on your signals collection, schedule the [Synonym Detection job](/docs/5/fusion/reference/config-ref/jobs/synonym-detection).
* **Remove Words** - Remove particular phrases from queries. Unlike other rewrites, these should be entered manually and are not generated by a job. For more information, see [Remove Words](/docs/5/fusion/getting-data-out/query-enhancement/remove-words).
## Facets
### Facet menu
To facet the rules list by **Status** or **Published**, click one of the facet items within the facet categories.
To facet the rules by **Confidence**, click and drag either slider to set the range of confidence you would like to see:
Faceting rules by confidence is not available for Head/Tail rewrites.
### Search
Enter a term in the search bar to quickly facet your query rewrite rules.
## Rules
Begin by selecting one of the query rewrite rule types you will create.
### Create a rule
1. Click **+Add**.
2. In the new table row, complete the required fields, and additional fields as needed.
3. Click the checkmark to save, or close to cancel the changes.
### Edit a rule
1. Hold the pointer over the rule to edit.
2. Click the pencil icon.
3. Enter desired changes and click the checkmark to save, or close to cancel the changes.
### Approve/deny rules
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To modify multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Approve** or **Deny** as needed.
### Publish a rule
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To publish multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Publish Selected** to publish the selected rules.
#### Publish an individual rule
1. Hold the pointer over a rule.
2. Click the button.
3. To confirm, click **Publish**.
### Delete rules
#### Delete a single rule
1. Hold the pointer over a rule.
2. Click the trash can.
3. To confirm, click **Delete**.
#### Delete multiple rules
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To delete multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Delete**.
5. To confirm, click **Delete** again.
### Sort rules
1. Hold the pointer over column titles. A icon displays for some titles to indicate the rules list can be sorted by that column.
2. Click the title column to sort the rules.
The column that is currently sorting the rules list is indicated by an or icon.
### Export rules
To export the rules list, click **Export**. This generates a `.csv` file that contains your rules.
## Columns
### Customize columns
To customize columns:
1. Hold the pointer over the row of column titles.
2. Click the button.
3. In the Customize Columns to Display screen, select the checkboxes for the columns you want to display. Clear the checkboxes for columns you do not want to display.
4. Click **Save**. To exit without saving your changes, click **Cancel**.
### UI elements
#### Facets
* **Status.** Whether a rule is enabled or disabled.
* **Published.** Whether or not a rule has been published.
* **Confidence.** The confidence score for the rule. A confidence score is automatically set by Fusion. It can be manually set for user-created rules. The confidence score is used, in part, to decided whether a rule is automatically published or not. See [Query Rewriting concepts](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) for more information.
**Confidence** is not used for Head/Tail query rewrites.
#### Rules list
You can facet, search, or sort the rules list according to the values the rules hold.
##### Rule type selectors
There are buttons at the top of the rules list that allow you to choose what query rewrite rule types you would like to work with:
* **Head/Tail.** Replace underperforming queries with queries that produce higher click-through rates. Query improvements always require review before they are published. Schedule the Head/Tail Analysis job to get AI-generated query improvements based on your signals collection. See [Underperforming Query Rewriting](/docs/5/fusion/getting-data-out/query-enhancement/underperforming-queries).
* **Misspelling.** Fusion replaces misspelled query terms with their spelling corrections for better search results. Schedule the Token and Phrase Spell Correction job to get AI-generated spelling corrections based on your signals collection. See [Misspelling Detection](/docs/5/fusion/getting-data-out/query-enhancement/misspelling-detection).
* **Phrase.** Known phrases are boosted when they appear in new queries. Schedule the Phrase Extraction job to get AI-generated phrases based on your signals collection. See [Phrase Detection](/docs/5/fusion/getting-data-out/query-enhancement/phrase-detection).
* **Synonym.** Synonyms expand queries so that they include all known synonyms of the query terms. Schedule the Synonym Detection job to get AI-generated synonyms based on your signals collection. See [Synonym Detection](/docs/5/fusion/reference/config-ref/jobs/synonym-detection).
* **Remove Words.** Remove particular phrases from queries. Unlike other rewrites, these should be entered manually and are not generated by a job. See [Remove Words](/docs/5/fusion/getting-data-out/query-enhancement/remove-words).
## Templates
The Templates screen in the Rules Editor allows you to design, test, and implement a wide variety of search experiences. Results are conditionally delivered from multiple different query pipelines, enabling you to use Fusion’s search, browse, and AI-driven functionality throughout your site.
Templates are only available with a valid license for [Predictive Merchandiser](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/overview). If available, you will see the templates icon in the left navigation panel:
## Simulator
The Simulator provides an interactive preview of how your staged rules affect relevancy, using your search data and a simple search interface. When you enter a query, the Simulator shows you the triggered rules, in the order in which they were triggered, along with any triggered facets.
You can edit any of the triggered rules, then re-run the query to see new results.
The Simulator sends requests to the `COLLECTION_NAME_rules_simulator` query profile, which you can configure to point to any pipeline and collection in your app.
The Simulator is only visible in the Rules Editor if you *do not* have a valid license for [Predictive Merchandiser](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/overview). If available, you will see the simulator icon in the left navigation panel.
### Search
Use the search bar in the top nav panel to simulate, or preview, your staged rules.
### Rule indicators
After entering a search query, the active rules affecting the results are displayed above the results. For example, a query for the term `keyboard` with a query rewrite rule would display an indicator for the rewritten query `wireless keyboard` in the left panel of the Simulator:
You can edit the query rewrite rule by hovering over this indicator and clicking the edit button. A dialog box appears:
Similarly, when a business rule is triggered by a search condition, it appears in the right panel of the Simulator:
## Analytics
The Analytics screen is available as part of the [Predictive Merchandiser](#predictive-merchandiser-and-experience-optimizer), a separate offering from Fusion. See the [Predictive Merchandiser documentation](/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/overview) for more information.
Learn how to use the **Use the Analytics Screen in the Rules Editor**.
This article describes how to use the [Analytics screen](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor) in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor).
Job configuration affects the data that displays on the dashboard. For example, the [`COLLECTION_NAME_session_rollup` job](/docs/5/fusion/reference/config-ref/jobs/aggregations/built-in-sql-aggregation-jobs) aggregates related user activity into a session signal that contains activity count, duration, and keywords (based on user search terms).
### UI elements
#### Data date range
* **Quick Range Selectors.** Click one of the quick range selectors to facet your data.
* **Calendar Selector.** Use the calendar selector to specify a start and end date and time facet for your data.
#### Visitor data
* **Data Cards**
* **Site Visits.** The total number of visits.
* **Total Searches.** The total number of searches performed.
* **Unique Visitors.** The number of unique visitors to your site.
* **Average Session.** The average length of time spent on your site.
* **Data Graph.** Detailed information for the selected data card.
## View data by date range
The Rules Editor analytics allows you to view data by a date range. To do this, you can either:
* Click one of the date range presets:
* Click the date range details to display and select your range on the calendar that appears:
## View additional card details
Click one of the analytics cards to view additional details for that card’s data.
## Configure Analytics cards
1. Hover over one of the cards.
2. Click the **Edit** button that appears:
3. Complete the form that appears.
4. Click the **Save** button.
If the data is empty, check to make sure the date range includes the data you want to display.
### UI elements
#### Data date range
* **Quick Range Selectors.** Click one of the quick range selectors to facet your data.
* **Calendar Selector.** Use the calendar selector to specify a start and end date and time facet for your data.
#### Visitor data
* **Data Cards**
* **Site Visits.** The total number of visits.
* **Total Searches.** The total number of searches performed.
* **Unique Visitors.** The number of unique visitors to your site.
* **Average Session.** The average length of time spent on your site.
* **Data Graph.** Detailed information for the selected data card.
## Learn more
This article teaches you how to keep business rules consistent between clusters. Use a `PUT` API call to the Query Rewrite) API’s `/query-rewrite/instances` endpoint for bulk operations. The JSON payload sent with the `PUT` call must adhere to the following format:
```json theme={"dark"}
{
"create": [
This article describes how to use the Analytics screen in the Rules Editor.
The Analytics screen was renamed from the Dashboard screen in Fusion 5.2.0. All functionality remained the same.
### UI elements
#### Data date range
* **Quick Range Selectors.** Click one of the quick range selectors to facet your data.
* **Calendar Selector.** Use the calendar selector to specify a start and end date and time facet for your data.
#### Visitor data
* **Data Cards**
* **Site Visits.** The total number of visits.
* **Total Searches.** The total number of searches performed.
* **Unique Visitors.** The number of unique visitors to your site.
* **Average Session.** The average length of time spent on your site.
* **Data Graph.** Detailed information for the selected data card.
## View data by date range
The Rules Editor analytics allows you to view data by a date range. To do this, you can either:
* Click one of the date range presets:
* Click the date range details to display and select your range on the calendar that appears:
## View additional card details
Click one of the analytics cards to view additional details for that card’s data.
# Aggregations
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/signals/aggregations
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/signals/aggregations
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/signals/aggregations
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/aclc1h
Aggregations compile [Signals](/docs/5/fusion/getting-data-out/query-enhancement/signals/overview) into a set of summaries that you can use to enrich the search experience through [recommendations and boosting](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods).
Fusion uses SQL to produce these signal summaries. SQL is a familiar query language that is well suited to data aggregation. Fusion’s SQL Aggregation Engine is powerful and flexible.
You can perform a SQL aggregation on a signals collection for a datasource (or on some other collection) through the Fusion UI or using a Fusion API. For more information, see **Create and Run a SQL Aggregation Job**.
You can perform a SQL aggregation on a signals collection for a datasource (or on some other collection), through the Fusion UI or using the Fusion API.
## Preliminaries
Before you can create and run a SQL aggregation job, you must create an app, create a collection (or use the default collection for the app), and add a datasource. Before you run a SQL aggregation job, you need signal data. Otherwise, there is nothing to aggregate.
## Use the Fusion UI
You can use the Fusion UI to perform a SQL aggregation.
1. In the Fusion application, navigate to **Collections > Jobs**.
2. Click **Add** and select **SQL Aggregation** from the dropdown list.
3. Specify an arbitrary **Spark Job ID**.
4. For the **Source Collection**, select the collection that contains the data to aggregate.
This is *not* the base collection. For example, to aggregate the signals in `experiment_signals`, you would select `experiment_signals`, not `experiment`.
5. Click the **SQL** field to enter aggregation statements. Click **Close** to close the dialog box.
Select the **Advanced** option in the top right corner to enter detailed parameters. See [SQL Aggregation Jobs](/docs/5/fusion/reference/config-ref/jobs/aggregations/sql-aggregation) for more information.
6. Enter applicable **Signal Types**.
7. Enter the **Data Format**.
8. Save the job.
**Run a SQL aggregation job**
1. With the app open, navigate to **Collections > Jobs**.
2. In the list of jobs, click the job you want to run, and then click **Run**.
If you do not want to run the job manually, you can also schedule a job.
## Use the Fusion API
You can use the Fusion API to perform a SQL aggregation. For example:
```json theme={"dark"}
curl ":{api-port}/api/spark/configurations/experiment_signals_aggregation"
{
"type" : "spark",
"id" : "experiment_click_signals_aggregation",
"definition" : {
"id" : "experiment_click_signals_aggregation",
"sql" : "SELECT SUM(count_i) AS aggr_count_i, query AS query_s, doc_id AS doc_id_s, time_decay(count_i, timestamp_tdt) AS weight_d FROM default_signals WHERE type_s='click' GROUP BY query, doc_id",
"selectQuery" : "*:*",
"outputPipeline" : "_system",
"rollupAggregator" : "SQL",
"sourceRemove" : false,
"sourceCatchup" : true,
"outputRollup" : true,
"parameters" : [ {
"key" : "optimizeOutput",
"value" : "4"
} ]
},
"inputCollection" : "experiment_click_signals",
"rows" : 10000
}
```
```json theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H "Content-Type: application/json" http://localhost:{api-port}/api/jobs/spark:experiment_click_signals_aggregation/actions -d '{"action": "start"}'
```
## Rollup SQL
Most aggregation jobs run with the catch-up flag set to `true`, which means that Fusion only computes aggregations for new signals that have arrived *since* the last time the job was run, and *up to and including* `ref_time`, which is usually the run time of the current job. Fusion must "roll up" the newly aggregated rows into any existing aggregated rows in the `COLLECTION_NAME_aggr` collection.
If you are using SQL to do aggregation but have not supplied a custom rollup SQL, Fusion generates a basic rollup SQL script automatically by consulting the schema of the aggregated documents. If your rollup logic is complex, you can provide a custom rollup SQL script.
Fusion’s basic rollup is a SUM of numeric types (long, integer, double, and float) and can support `time_decay` for the weight field. If you are not using `time_decay` in your weight calculations, then the weight is calculated using `weight_d`. If you do include `time_decay` with your weight calculations, then the weight is calculated as a combination of timestamp, `halfLife`, `ref_time`, and `weight_d`.
The basic rollup SQL can be grouped by `DOC_ID`, `QUERY`, `USER_ID`, and `FILTERS`. The last GROUP BY field in the main SQL is used so it will ultimately group any newly aggregated rows with existing rows.
This is an example of a rollup query:
```sql wrap theme={"dark"}
SELECT query_s, doc_id_s, time_decay(1, timestamp_tdt, "30 days", ref_time, weight_d)
AS weight_d , SUM(aggr_count_i) AS aggr_count_i
FROM `commerce_signals_aggr` GROUP BY query_s, doc_id_s
```
## Time-range filtering
When Fusion rolls up new data into an aggregation, time-range filtering lets you ensure that Fusion does not aggregate the same data over and over again.
Fusion applies a time-range filter when loading rows from Solr, before executing the aggregation SQL statement. In other words, the SQL executes over rows that are already filtered by the appropriate time range for the aggregation job.
Notice that the examples [Perform the Default SQL Aggregation](/docs/5/fusion/reference/config-ref/jobs/aggregations/sql-aggregation-examples) and [Use Different Weights Based on Signal Types](/docs/5/fusion/reference/config-ref/jobs/aggregations/sql-aggregation-examples#use-different-weights-based-on-signal-types) do not include a time-range filter. Fusion computes the time-range filter automatically as follows:
* If the catch-up flag is set to `true`, Fusion uses the last time the job was run and `ref_time` (which you typically set to the current time). This is equivalent to the WHERE clause `WHERE time > last_run_time AND time <= ref_time`.
* If the catch-up flag is not set to `true`, Fusion uses a filter with `ref_time` (and no start time). This is equivalent to the WHERE clause `WHERE time <= ref_time`.
The built-in time logic should suffice for most use cases. You can set the time range filter to **TO** and specify a WHERE clause filter to achieve more complex time based filtering.
Time range in Fusion is equivalent to [date range in Solr](https://solr.apache.org/guide/working-with-dates.html).
**Example values for time range:**
* `[* TO NOW]` - all past events
* `[2000-11-01 TO 2020-12-01]` – specify by exact date
* `[* TO 2020-12-01]` – from the first data point to the end of the day specified
* `[2000 TO 2020]` - from the start of a year to the end of another year
## SQL functions
A Spark SQL aggregation query can use any of the functions provided by [Spark SQL](https://spark.apache.org/docs/latest/api/sql/index.html). Use these functions to perform complex aggregations or to enrich aggregation results.
## Weight aggregated values using a time-decay function
Fusion automatically uses a default `time_decay` function to compute and apply appropriate weights to aggregation groups during aggregation. Larger weights are assigned to more recent events. This reduces the impact of less-recent signals. Intuitively, older signals (and the user behavior they represent) should count less than newer signals.
If the default `time_decay` function does not meet your needs, you can modify it. The `time_decay` function is implemented as a [User Defined Aggregate Function](https://spark.apache.org/docs/latest/sql-ref-functions-udf-aggregate.html) (UDAF).
#### Full function signature
This is the UDAF signature of the default `time_decay` function:
```js wrap theme={"dark"}
time_decay(count: Long,
timestamp: Timestamp,
halfLife: String (calendar interval),
ref_time: Timestamp,
weight_d: Double)
```
**Abbreviated function signature and default values**
Your function call can also use this abbreviated UDAF signature, that omits `halfLife`, `ref_time`, and `weight_d`:
```js wrap theme={"dark"}
time_decay(count: Long,
timestamp: Timestamp)
```
In this case, Fusion fills in these values for the omitted parameters: `halfLife` = `30 days`, `ref_time` = `NOW`, and `weight_d` = `0.1`.
Users indexing custom signals with `weight_d` specified should ensure that the "default" value matches the `weight_d` parameter used by `time_decay`.
#### Matching legacy aggregation
To match the results of legacy aggregation, either use the abbreviated function signature or supply these values for the mentioned parameters: `halfLife` = `30 days`, `ref_time` = `NOW`, and `weight_d` = `0.1`.
#### Parameters
Parameters for `time_decay` are:
| Parameter | Description |
| - | - |
| `count` | Number of occurrences of the event. Typically, the increment is 1, though there is no reason it could not be some other number. In most cases, you pass `count_i`, which is the event count field used by Fusion signals, as shown in the [SQL aggregation examples](/docs/5/fusion/reference/config-ref/jobs/aggregations/sql-aggregation-examples). |
| `timestamp` | The date-and-time for the event. This time is the beginning of the interval used to calculate the time-based decay factor. |
| `halfLife` | Half life for the exponential decay that Fusion calculates. It is some interval of time, for example, `30 days` or `10 minutes`. The `interval` prefix is optional. Fusion treats `30 days` as equivalent to `interval 30 days`. |
| `ref_time` | Reference time used to compute the age of an event for the time-decay computation. It is usually the time when the aggregation job runs (`NOW`). The reference time is not present in the data; Fusion determines the reference time at runtime. Fusion automatically attaches a `ref_time` column to every row before executing the SQL. |
| `weight_d` | Initial weight for an event if not specified in the signal, prior to the decay calculation. This value is typically not present in the signal data. If some signals *do* contain `weight_d` values, this parameter should be set to match the "neutral" value there. You can use SQL to compute `weight_d`; see [Use Different Weights Based on Signal Types](/docs/5/fusion/reference/config-ref/jobs/aggregations/sql-aggregation-examples#use-different-weights-based-on-signal-types) for an example. |
#### Sample calculation of the age of a signal
This is an example of how Fusion calculates the age of a signal:
Imagine a SQL aggregation job that runs at `Tuesday, July 11, 2017 1:00:00 AM (1499734800)`.
For a signal with the timestamp `Tuesday, July 11, 2017 12:00:00 AM (1499731200)`, the age of the signal in relation to the reference time is 1 hour.
## Learn more
Fusion’s basic aggregation jobs aggregate using the document ID. You can also aggregate at a more coarse-grained level using other fields available for documents (item metadata), such as manufacturer or brand for products. Aggregating with item metadata is useful for building personalization boosts into your search application.
The following PUT request creates additional aggregation jobs that join signals with the primary `products` collection to compute an aggregated weight for a `manufacturer` field:
```bash wrap theme={"dark"}
curl -X PUT -H "Content-type:application/json" -d '{"enabled":true, "metadata_column":"manufacturer"}' "http://localhost:{api-port}/api/collections/products/features/signals"
```
After performing the PUT request shown above, you will have two additional aggregation jobs in Fusion.
### COLLECTION\_NAME\_user\_METADATA\_preferences\_aggregation
This job computes an aggregated weight for each user/item metadata combination, e.g. user/manufacturer, found in the signals collection. Fusion computes the weight for each group using an exponential time-decay on signal count (30 day half-life) and a weighted sum based on the signal type. Use the `signalTypeWeights` parameter to set the correct signal types and weights for your dataset. Use the `primaryCollectionMetadataField` parameter to set the name of a field from the primary collection to join into the results, e.g. `manufacturer`. You can use the results of this job to boost queries based on user preferences regarding item-specific metadata such as manufacturer (e.g. Ford vs. BMW) or brand (e.g. Ralph Lauren vs. Abercrombie & Fitch).
### COLLECTION\_NAME\_query\_METADATA\_preferences\_aggregation
This job computes an aggregated weight for each query/item metadata combination, e.g. query/manufacturer, found in the signals collection. Fusion computes the weight for each group using an exponential time-decay on signal count (30 day half-life) and a weighted sum based on the signal type. Use the `signalTypeWeights` parameter to set the correct signal types and weights for your dataset. Use the `primaryCollectionMetadataField` parameter to set the name of a field from the primary collection to join into the results.
These additional item item/metadata aggregation jobs also serve as examples of how to join between the signals and primary collections to perform aggregations on fields other than the document ID. You can re-execute the same PUT request shown above using a different metadata field name in the `metadata_column` parameter.
# Signals Data Flow
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/signals/data-flow
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/signals/data-flow
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/signals/data-flow
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/467
This diagram shows the flow of signals data from the search app through Fusion. The numbered steps are explained below.
1. The search app sends a query to a Fusion query pipeline.\
The query request should include a user ID and session query parameter to identify the user.
2. Optionally, the Fusion query pipeline queries the `COLLECTION_NAME_signals_aggr` collection to get boosts for the main query based on aggregated click data.
3. The search app also sends a request signal to the Fusion [`/signals` endpoint](/api-reference/signal-indexing-api/index-an-event-signal).\
The primary intent of a request signal is to capture the raw user query and contextual information about the user’s current activity in the app, such as the user agent and the page where they generated the query. The request signal does not contain any information about the results sent to Solr; it is created before a query is processed.
4. Once Solr returns the response to Fusion, the SearchLogger component indexes the complete request/response data into the `COLLECTION_NAME_signals` collection as a response signal using the `_signals_ingest` pipeline. Therefore, the response signal captures all results from Fusion as it related to the original query.
This is a departure from pre-4.0 versions of Fusion where query impressions were logged in a separate `COLLECTION_NAME_logs` collection. Query activity is no longer indexed into the `_logs` collection. All response signals use the `fusion_query_id` (see below) as the unique document ID in Solr.
5. When the user clicks a link in the search results, the search app sends a click event to the Fusion signals endpoint (which invokes the `_signals_ingest` pipeline behind the scenes).
The click signal *must* include a field named `fusion_query_id` in the `params` object of the raw click signal. The `fusion_query_id` field is returned in the query response (from step 1) in a response header named `x-fusion-query-id`. This allows Fusion to associate a click signal with the response signal generated in step 4. The `fusion_query_id` is also used by Fusion to associate click signals with [experiments](/docs/5/fusion/getting-data-out/data-analytics/experiments/overview). For experiments to work, each click signal must contain the corresponding `fusion_query_id` that produced the document/item that was clicked.
6. The `_signals_ingest` pipeline enriches signals before indexing into the `COLLECTION_NAME_signals` collection.
This enrichment includes field mapping, geolocation resolution, and updating the `has_clicks` flag to "true" on request signals when the first click signal is encountered for a given request using the [Update Related Document index stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/update-related-document-index-stage).
7. Fusion queries the `COLLECTION_NAME_signals` collection through a Fusion query pipeline to generate query analytics reports from raw signals.
8. Behind the scenes, the [SQL aggregation](/docs/5/fusion/reference/config-ref/jobs/aggregations/sql-aggregation) framework aggregates click signals to compute a weight for each query + `doc_id` + filters group.\
The resulting metrics are saved to the `COLLECTION_NAME_signals_aggr` collection to generate boosts on queries to the main collection (step 2 above).
9. Recommendations also use aggregated documents in the `COLLECTION_NAME_signals_aggr` collection to build a [collaborative filtering-based recommender model](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods).
# Deleting Old Signals
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/signals/deleting-old-signals
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/signals/deleting-old-signals
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/signals/deleting-old-signals
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/468
Signals are not automatically deleted by default, and over time they occupy an increasing amount of storage space.
To avoid running out of storage space as a result of your growing collection of signals, you must decide on a signals retention policy, then configure and schedule a [REST Call job](/docs/5/fusion/reference/config-ref/jobs/rest-call) that regularly deletes old signals.
The duration for which signals should be kept depends on your use case and your organization’s policies. For example, in some cases signals could be deleted after 30 days, while in other cases they may remain useful for a year, or even forever.
## Learn more
1. Navigate to **Collections** > **Jobs**.
2. Click **Add** and select **REST Call**.\
The REST Call job configuration panel appears.
3. Enter an arbitrary ID for this job, such as "Delete-old-signals".
4. Enter the following endpoint URI, substituting the name of your signals collection for `signalsCollectionName`:
```bash theme={"dark"}
solr://signalsCollectionName/update
```
5. In the **Call Method** field, select "post".
6. Under **Query Parameters**, enter the property name "wt" with the property value "json".
7. In the **Request entity (as string)** field, enter the following:
```html wrap theme={"dark"}
timestamp_tdt:[* TO NOW-2WEEKS]
```
See [Working with Dates](https://solr.apache.org/guide/8_6/working-with-dates.html) for details about date formatting.
Your job configuration should look similar to this:
You can configure a schedule for this job at **System** > **Scheduler**.
# Signals
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/signals/overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/signals/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/signals/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/469
A *signal* is a recorded event related to one or more documents in a collection. Signals can record any kind of event that is useful to your organization. Click signals are the most common type of signals as this is the most common action a user takes with an item. In addition, other signal types can be defined, such as "addToCart", "purchase", and so on.
Using a sufficiently large collection of signals, Fusion can automatically generate [recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods) such as these:
* Based on the user’s search query, which items are most likely to interest them?
* Based on the user’s similarity to other users, which additional items are likely to interest them?
Signals are indexed in a secondary collection which is linked to the primary collection by the naming convention `_signals`. So, if your main collection is named `products`, the associated signals collection is named `products_signals`. The signals collection is created automatically when signals are enabled for the primary collection. Signals are enabled by default whenever a new collection is created.
Signals are indexed just like ordinary documents. The signals collection can be searched like any other collection, for example by using the [Query Workbench](/docs/5/fusion/intro/ui-tour/query-workbench) with the signals collection selected.
[App Insights](/docs/5/fusion/getting-data-out/data-analytics/app-insights/overview) provides visualizations and reports with which to analyze your signals. App Insights mainly uses raw signals, but also uses some [*aggregated*](/docs/5/fusion/reference/config-ref/jobs/aggregations/overview) signals. Currently only the signal types Request, Response and Click are supported within the App Insights dashboards.
See [signals types and structure](/docs/5/fusion/getting-data-out/query-enhancement/signals/types-and-structures) for more information.
## The cold start problem
The "cold start" problem means it is hard to personalize the search experience when insufficient signals have been aggregated. For example, it is hard to offer recommendations to users who have never visited before, or for queries that have never been issued before, or for items that have been recently introduced into the system.
Fusion provides solutions for this problem using its query pipelines. A query pipeline that includes stages for blocking, boosting, or recommending based on signals can also include stages that provide fallbacks. In the case where there is not enough data to provide specialized blocking, boosting, or recommendations, the pipeline can return a simpler set of search results using Solr’s normal relevancy calculation.
A common solution to the cold start problem is to sort or boost on a certain field to provide pseudo-recommendations when more specific recommendations are not available. For example, you can sort on the `sales_rank` field to recommend the most popular products, or boost on the `date_added` field to recommend the newest items.
## Indexing or importing Signals
Normally, signals are indexed as streaming data during the natural activity of users. See [Indexing Signals](/docs/5/fusion/getting-data-in/indexing/indexing-signals/overview) for details.
For more information, see [Parallel Bulk Loader](/docs/5/fusion/reference/config-ref/jobs/parallel-bulk-loader).
## Learn more
The course for **Fusion Signals** focuses on how Fusion utilizes signal information to boost search relevance.
1. In the Fusion UI, select your signals collection.
2. Open the Query Workbench by navigating to **Query** > **Query Workbench**.
3. Click **Add a field facet**.
4. Select the `type` field.\
The list of signal types appears in the facet panel:
# Required Signals for App Insights
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/signals/signals-reqs-for-insights
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/signals/signals-reqs-for-insights
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/signals/signals-reqs-for-insights
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/drf2uz
Although there are many fields that are relevant to [signals](/docs/5/fusion/getting-data-out/query-enhancement/signals/overview), this topic describes which fields are **required** for each [type of signal](/docs/5/fusion/getting-data-out/query-enhancement/signals/types-and-structures). Without these fields, reports and dashboards may not function as expected.
| Report | Signal Types Used | Required Field | Type | Description | Example |
| - | - | - | - | - | - |
| Search queries | ✘ Annotation ✅ Click ✘ Login ✅ Request ✅ Response | `query` | string | Terms of the query. | `physics` |
| Facets used | ✘ Annotation ✘ Click ✘ Login ✅ Request ✘ Response | `filter_field` | array of string | List of filters associated with the query, which in turn is associated with signal. | `["categories_s"]` |
| Facet filters applied | ✘ Annotation ✘ Click ✘ Login ✅ Request ✘ Response | `filter` | array of string | List of **applied** filters associated with the query, which in turn is associated with signal. | `["categories_s/math.OA"]` |
| Application servers | ✘ Annotation ✅ Click ✘ Login ✅ Request ✘ Response | `host` | string | Host name of the server that is hosting the app, which in turn is generating the signal. | `lucidworks` |
| Response times | ✘ Annotation ✘ Click ✘ Login ✘ Request ✅ Response | `time` | number | Response time (in milliseconds). | `17` |
| URLs clicked | ✘ Annotation ✅ Click ✘ Login ✘ Request ✘ Response | `url` | string | URL of the page that was selected from the search results. This typically results from a click signal. | `http://arxiv.org/pdf/astro-ph/0611688v1` |
| Search pages | ✘ Annotation ✘ Click ✘ Login ✅ Request ✘ Response | `page_title` | string | Title of the page that the signal originated from. | `Search Preview` |
| Search platforms | ✘ Annotation ✅ Click ✘ Login ✅ Request ✅ Response | `platform` | string | Name of the search platform. | `fusion` |
| Visitor countries | ✘ Annotation ✅ Click ✘ Login ✅ Request ✘ Response | `geo_country_name` | string | Originating country of the user. | `United States` |
| Visitor cities | ✘ Annotation ✅ Click ✘ Login ✅ Request ✘ Response | `geo_city_name` | string | Originating city of the user. | `San Francisco` |
| Browsers | ✘ Annotation ✅ Click ✘ Login ✅ Request ✘ Response | `browser_name` | string | Name of the browser that generated the signal. | `Firefox` |
| Operating System | ✘ Annotation ✅ Click ✘ Login ✅ Request ✘ Response | `os_name` | string | Name of the operating system used by the user. | `Ubuntu` |
| Device types | ✘ Annotation ✅ Click ✘ Login ✅ Request ✘ Response | `os_device` | string | Type of the device used by the user. | `Computer` |
| Websites people are coming from | ✘ Annotation ✅ Click ✘ Login ✘ Request ✘ Response | `referrer_domain` | string | URL of the page that the signal originated from. | `http://www.google.com` |
| Users | ✘ Annotation ✅ Click ✘ Login ✅ Request ✘ Response | `user_id` | string | Unique ID for the user that generated the signal. | `admin` |
| User domains | N/A | `domain` | string | Domain of the user that generated the signal. | `lucidworks.com` |
| Types of event | ✅ Annotation ✅ Click ✅ Login ✅ Request ✅ Response | `type` | string | Type of the signal. | `click` |
| Head Tail analysis | ✘ Annotation | | | This report is populated by running a head tail job in Fusion. With a collection called `foo`, for example, run the `foo_head_tail` job. | N/A |
| ✅ Click | type | string | N/A | | |
| ✅ Click | query | string | N/A | | |
| ✘ Login | | | N/A | | |
| ✘ Request | | | N/A | | |
| ✅ Response | count\_i | number | N/A | | |
The fields mentioned above are defined as follows:
| Field Name | Example Value | Description |
| - | - | - |
| `count_i` | 1 | Number of times an interaction event occurred with this item |
| `doc_id` | NMDDV | Product ID or Item ID |
| `query` | xwearabletech | A query string from the user |
| `session_id` | 91aa66d11af44b6c90ccef44d055cf9a | Id for session in which user generated the signal |
| `type` | quick\_view\_click | Type of session the user used to interact with the platform |
| `user_id` | 11506893 | ID of user during the session |
| `timestamp_tdt` | 2018-11-20T17:58:57.650Z | Time when signal was generated |
Some [signal types](/docs/5/fusion/getting-data-out/query-enhancement/signals/overview), including [custom signal types](/docs/5/fusion/getting-data-out/query-enhancement/signals/types-and-structures#custom-signal-types), may include additional fields.
# Required Signals for Jobs
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/signals/signals-reqs-for-jobs
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/signals/signals-reqs-for-jobs
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/signals/signals-reqs-for-jobs
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/t08vnl
A number of Fusion jobs ingest signals. Make sure your signals include the fields required by the jobs you plan to use. The table below shows which jobs require specific fields in your signals.
| | query | count\_i | type | timestamp\_tdt | user\_id | doc\_id | session\_id | fusion\_query\_id |
| - | - | - | - | - | - | - | - | - |
| Created by default | | | | | | | | |
| [COLLECTION\_NAME\_click\_signals\_aggregation](/docs/5/fusion/reference/config-ref/jobs/aggregations/overview) | ✅ | ✅ | ✅ | ✅ | | ✅ | | \[**1**] |
| [COLLECTION\_NAME\_phrase\_extraction](/docs/5/fusion/reference/config-ref/jobs/phrase-extraction) | ✅ | ✅ | ✅ | | | | | |
| [COLLECTION\_NAME\_spell\_correction](/docs/5/fusion/reference/config-ref/jobs/token-and-phrase-spell-correction) | ✅ | ✅ | ✅ | | | | | |
| [COLLECTION\_NAME\_synonym\_detection](/docs/5/fusion/reference/config-ref/jobs/synonym-detection) | ✅ | ✅ | ✅ | | | ✅ | | |
| [COLLECTION\_NAME\_head\_tail](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis) | ✅ | ✅ | ✅ | | | | | |
| [COLLECTION\_NAME\_user\_item\_prefs\_agg](/docs/5/fusion/reference/config-ref/jobs/aggregations/built-in-sql-aggregation-jobs) | | ✅ | ✅ | ✅ | ✅ | ✅ | | |
| [COLLECTION\_NAME\_user\_query\_history\_agg](/docs/5/fusion/reference/config-ref/jobs/aggregations/built-in-sql-aggregation-jobs) | ✅ | ✅ | ✅ | ✅ | ✅ | | | |
| Created when recommendations are enabled | | | | | | | | |
| [COLLECTION\_NAME\_bpr\_item\_recs](/docs/5/fusion/reference/config-ref/jobs/bpr-recommender) | | ✅ | ✅ | ✅ | ✅ | ✅ | | |
| Created when experiments are enabled | | | | | | | | |
| [Experiments](/docs/5/fusion/getting-data-out/data-analytics/experiments/overview) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| Created manually | | | | | | | | |
| [Query-to-Query Collaborative Similarity](/docs/5/fusion/reference/config-ref/jobs/query-to-query-collaborative-similarity) | ✅ | ✅ | ✅ | ✅ | ✅ | | | |
| [Query-to-Query Session-Based Similarity](/docs/5/fusion/reference/config-ref/jobs/query-to-query-session-based-similarity) | ✅ | ✅ | \[**2**] | \[**3**] | \[**4**] | ✅ | ✅ | |
| [Trending Recommender](/docs/5/fusion/reference/config-ref/jobs/trending-recommender) | \[**5**] | ✅ | ✅ | ✅ | | ✅ | | |
* **1** Required if you are using response signals.
* **2** Required if you want to weight types differently.
* **3** Required if you want to use time decay.
* **4** Required if no `session_id` field is available. Either `user_id` or `session_id` is needed on response signals if doing any click path analysis from signals.
* **5** Required when identifying trending queries instead of trending items.
The fields mentioned above are defined as follows:
| Field Name | Example Value | Description |
| - | - | - |
| `count_i` | 1 | Number of times an interaction event occurred with this item |
| `doc_id` | NMDDV | Product ID or Item ID |
| `query` | xwearabletech | A query string from the user |
| `session_id` | 91aa66d11af44b6c90ccef44d055cf9a | Id for session in which user generated the signal |
| `type` | quick\_view\_click | Type of session the user used to interact with the platform |
| `user_id` | 11506893 | ID of user during the session |
| `timestamp_tdt` | 2018-11-20T17:58:57.650Z | Time when signal was generated |
Some [signal types](/docs/5/fusion/getting-data-out/query-enhancement/signals/overview), including [custom signal types](/docs/5/fusion/getting-data-out/query-enhancement/signals/types-and-structures#custom-signal-types), may include additional fields.
# Signals Types and Structures
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/signals/types-and-structures
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/signals/types-and-structures
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/signals/types-and-structures
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/466
Signals can be broadly categorized as [implicit or explicit](#implicit-signals-vs-explicit-signals). When signals are enabled, Fusion produces several [built-in signal types](#built-in-signal-types) by default, all of which are implicit signals. You can also create [custom signal types](#custom-signal-types), including explicit signals. Be sure to verify that your signals include all of the [important fields](#required-signal-fields) for best results. It is also useful to [rank your signal types](#signal-type-ranking) in terms of how strongly each type indicates a user’s interest in an item.
## Implicit signals vs explicit signals
Signals can reveal a user’s level of interest in an item in two main ways:
* Implicit\
The user shows interest by engaging with the item/document through clicks, searches, and so on. Since this type of interaction requires no additional effort on the user’s part, these types of signals tend to be plentiful. They can be used to infer a measurable value of interest in order to build an accurate recommender system.
* Explicit\
An explicit signal is created when a user intentionally assigns a clear, measurable value to an item, such as by giving it a rating. This value can be used to rank items, for example. Since this requires the user to invest extra time to provide the information, the number of ratings tends to be small compared to the total number of users interacting with the item.
You can create [recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods) based on implicit signals out of the box. For recommenders based on explicit signals, contact your Lucidworks Professional Services representative.
## Built-in signal types
There are five built-in signal types:
| | |
| - | - |
| [annotation](#annotation-signals) | Annotation signals are generated when a user bookmarks, likes, or comments on a document. Annotation signals are likewise generated when the user removes a bookmark, like, or comment. Annotation signals are generated by [App Studio](/docs/5/fusion/dev-portal/appkit/overview). If you are not using App Studio, this type of signal is not relevant to your search application. |
| [login](#login-signals) | Login signals record information about specific users when they log in to an application. This includes a time stamp and various session details. |
| [request](#request-signals) | A request signal is generated by a front-end search app and captures the raw user query and other contextual information about a user and their journey through the search app. |
| [response](#response-signals) | Response signals are automatically generated by a query pipeline when the signals feature is enabled for a collection. |
| [click](#click-signals) | Click signals are generated when a user clicks on a page element that is being monitored by the search app. Click signals are sent from the search app to Fusion. |
**Important**
Because response signals and their fields are automatically generated, this topic does not cover what response signal fields are required.
### Annotation signals
Annotation signals are generated when a user bookmarks, likes, or comments on a document. Annotation signals are likewise generated when the user removes a bookmark, like, or comment.
Annotation signals are generated by [App Studio](/docs/5/fusion/dev-portal/appkit/overview). If you are not using App Studio, this type of signal is not relevant to your search application.
### Login signals
Login signals record information about specific users when they log in to an application. This includes a time stamp and various session details.
### Request signals
A request signal is generated by a front-end search app and captures the raw user query and other contextual information about a user and their journey through the search app. The request signal contains no information about the documents the user searched for.
A request signal should have the following fields:
```json wrap theme={"dark"}
[
{
"id":"288fe4f7-6680-403e-8d18-27647cdd9989",
"timestamp":1518717749409,
"type":"request",
"params":{
"user_id":"admin",
"session":"ef4e00cd-91bb-45b4-be80-e81f9f9c5b27",
"query":"USER QUERY HERE",
"app_id":"SEARCH APP ID",
"ip_address":"0:0:0:0:0:0:0:1",
"host":"Lucids-MacBook-Pro-5.local",
"filter":[
"field1/value",
...
],
"filter_field":[
"field1"
]
}
}
]
```
Additional optional fields are used by [App Insights](/docs/5/fusion/getting-data-out/data-analytics/app-insights/overview). In the raw signal, optional fields should be inside the `params` object. Optional fields are as follows:
```json wrap theme={"dark"}
{
"page_title":"Fusion Search",
"path":"/search",
"browser_type":"Browser",
"browser_version":"64.0.3282.140",
"browser_name":"Chrome",
"referrer":"http://localhost:8080/",
"ctx_prev_uri":"/",
"ctx_prev_query":"",
"ctx_prev_path":"/",
"os_manufacturer":"Apple Inc.",
"os_name":"Mac OS X",
"os_id":"778",
"os_device":"Computer",
"os_group":"Mac OS X"
}
```
### Response signals
Response signals are automatically generated when a query pipeline receives a search request and that request is processed. This occurs only when the signals feature is enabled for a collection.
Front-end search applications should not send response signals to Fusion directly, as those would conflict with the auto-generated signals.
Response signals capture much of the same information as a request signal. Response signals also capture information about the actual results returned to the front-end search application, such as the search query, the query state (including sorts, pagination, and applied filters) and all the results from Fusion that are related to the query. Response signals can also identify search requests that returned no results to the user or requests that resulted in no clicks from the user.
Fusion creates a unique `fusion-query-id` to each response signal. This is used to correlate the downstream click signals with the original search request. After a user receives the results from their search, they can interact with the documents. In order to know which results (if any) were shown to the user, we use the response signals correlated with the unique `fusion-query-id`.
A response signal has the following explicit fields, plus any additional query parameters sent by the search application for a query:
| Field Name | Description | Example |
| - | - | - |
| `id` | The x-fusion-query-id generated by the query-pipeline used for associating click signals with queries in experiments and aggregation jobs. | `TwWCn3Dz` |
| `type` | Signal type | `response` |
| `response_type` | Used by Insights to determine if this query had results or was empty | `results \| empty` |
| `session` | User session ID; the search app should pass the session ID in the query params for a query | `UUID` |
| `query` | The actual query string sent to Solr from Fusion | `ipad` |
| `query_orig_s` | The incoming query from the search app before it is enriched by the query pipeline | `ipad` |
| `query_id` | A hash generated from the session, query, and filters fields; used as a rollup key in Insights to group activity by a specific | `SHA1 hash` |
| `filters_s` | Filter queries sent to Solr; the Fusion SearchLogger component combines multiple fq parameters into a single value delimited by `$` | `{!tag=format}format:(vhs) $ {!tag=type}type:(movie)` |
| `filter` | Reformatted filter queries | `field1/value` |
| `user_id` | User ID; the search app should pass the user\_id in the query params | `admin` |
| `doc_ids_s` | A comma-delimited list of document IDs returned for the page of results; this field is used by Fusion Spark jobs, such as the ground truth job, to perform click/skip analysis | `123,456,789` |
| `pipeline_id` | Fusion query pipeline that processed this query | `_system` |
| `collection` | Fusion collection | `my_collection` |
| `qtime` | Query time from Solr, in milliseconds | `10` |
| `rows` | Number of rows requested for this query | `10` |
| `hits` | Total number of documents matching the query | `10000` |
| `totaltime` | Total processing time of this query in milliseconds, includes Solr qtime and Fusion query processing time | `15` |
| `timestamp_tdt` | Timestamp when the query request was received by Fusion | `2018-02-15T18:17:42.560Z` |
| `res_offset` | Offset of results; this field is used by experiment metrics to calculate MRR | `0` |
| `res_pos` | Position of the clicked result within the list of results | `3` |
| `params.*` | Any other query param sent from the search app to Fusion that was not already mapped to a declared field | `params.defType_ss=edismax` |
Fusion’s experiment framework relies heavily on response signals and the linking between response and clicks signals using the `fusion_query_id`.
### Click signals
Click signals are explicit events that capture any type of user interaction that the business is interested in keeping track of. The basic click signal records the action of a user clicking an item in a context, whether that context is within search results, category browse, type ahead suggestions, or other locations. Each unique action receives a name such as click2pdp, add2cart or purchase.
When a user clicks a search result, your search app should send a click signal to Fusion. All click signals should include a `fusion_query_id` field pulled from the query response header `x-fusion-query-id`.
In addition, click signals should include the following fields:
```json wrap theme={"dark"}
[
{
"id":"SOME UUID HERE",
"timestamp":1518725351750,
"type":"click",
"params":{
"fusion_query_id":"ABkaEA11",
"user_id":"admin",
"session":"b3a15101-9e30-4e28-8a23-d1f663c2ee06",
"query":"tiger woods",
"ctype":"result",
"res_offset":0,
"filter":[
"type/Game"
],
"ip_address":"0:0:0:0:0:0:0:1",
"host":"Lucids-MacBook-Pro-5.local",
"doc_id":"9502308",
"app_id":"SEARCH APP ID",
"res_pos":1,
"filter_field":[
"type"
]
}
}
]
```
Additional optional fields are used by [App Insights](/docs/5/fusion/getting-data-out/data-analytics/app-insights/overview). In the raw signal, optional fields should be inside the `params` object. Optional fields are as follows:
```json wrap theme={"dark"}
{
"browser_type":"Browser",
"browser_version":"64.0.3282.140",
"browser_name":"Chrome",
"referrer":"http://localhost:8080/",
"ctx_prev_uri":"/",
"ctx_prev_query":"",
"ctx_prev_path":"/",
"os_manufacturer":"Apple Inc.",
"os_name":"Mac OS X",
"os_id":"778",
"os_device":"Computer",
"os_group":"Mac OS X",
"url":"http://localhost:8080/#/product/9502308",
"label":"Tiger Woods PGA Tour 09 All-Play - Nintendo Wii",
}
```
## Custom signal types
The signal `type` parameter can also take arbitrary values for custom signal types. For example, these custom events are important for e-commerce sites:
* Add-to-favorites
* Add-to-cart
* Remove-from-cart
* Purchase
* Hover/quick-view
To collect custom signals, configure your front-end search application to send signals to Fusion using a custom value for the `type` field. Custom signals should also include the fields described [below](#required-signal-fields) in order to get the best results from [aggregation](/docs/5/fusion/reference/config-ref/jobs/aggregations/overview) and [recommendation](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods) jobs.
To use custom signals in [recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods), you must add them to the value of the `signalTypeWeights` parameter in the configuration for the [`COLLECTION_NAME_user_item_preferences_aggregation`](/docs/5/fusion/reference/config-ref/jobs/aggregations/sql-aggregation) job and the [`COLLECTION_NAME_user_query_history_aggregation`](/docs/5/fusion/reference/config-ref/jobs/aggregations/sql-aggregation) job.
Custom signals can be analyzed in [App Insights](/docs/5/fusion/getting-data-out/data-analytics/app-insights/overview) just like pre-defined signal types.
## Required signal fields
Depending on how you use signals, certain fields are required. These are signals collection field names and not the JSON field names in the in-bound signals document. An example is when sending the **user id**, write it as `params.user_id`.
The fields mentioned in this section are defined as follows:
| Field Name | Example Value | Description |
| - | - | - |
| `count_i` | 1 | Number of times an interaction event occurred with this item |
| `doc_id` | NMDDV | Product ID or Item ID |
| `query` | xwearabletech | A query string from the user |
| `session_id` | 91aa66d11af44b6c90ccef44d055cf9a | Id for session in which user generated the signal |
| `type` | quick\_view\_click | Type of session the user used to interact with the platform |
| `user_id` | 11506893 | ID of user during the session |
| `timestamp_tdt` | 2018-11-20T17:58:57.650Z | Time when signal was generated |
Some [signal types](/docs/5/fusion/getting-data-out/query-enhancement/signals/overview), including [custom signal types](#custom-signal-types), may include additional fields.
### Required fields by use case
[Aggregations](/docs/5/fusion/reference/config-ref/jobs/aggregations/overview), [recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods), and [App Insights](/docs/5/fusion/getting-data-out/data-analytics/app-insights/overview) work best when certain fields are present in your signals. See these topics for details:
* [Required Signals for Jobs](/docs/5/fusion/getting-data-out/query-enhancement/signals/signals-reqs-for-jobs)
* [Required Signals for App Insights](/docs/5/fusion/getting-data-out/data-analytics/app-insights/report-requirements)
### Required fields by signal type
The following table describes which fields are required for annotation, click, login, and request signals.
Requests (or queries) can also require additional, available user data for the search.
| Field | Type | Description | Example | Required |
| - | - | - | - | - |
| `id` | string | Unique ID for the signal. | `b0ee5307-6223-4150-ac5a-d0d8113aa480` | ✅ Annotation
✅ Click
✅ Login
✅ Request |
| `doc_id` | string | Product ID or Item ID of the clicked result. | `NMDDV` | ✘ Annotation
✅ Click
✘ Login
✘ Request |
| `date` | timestamp | Timestamp of when the signal was generated. This timestamp follows Unix epoch time formatting. | `1518717749409` | ✅ Annotation
✅ Click
✅ Login
✅ Request |
| `user_id` | string | Unique ID for the user that generated the signal. | `admin` | ✅ Annotation
✅ Click
✅ Login
✅ Request |
| `session` | string | Unique ID for the user’s browser session. | `ef4e00cd-91bb-45b4-be80-e81f9f9c5b27` | ✅ Annotation
✅ Click
✅ Login
✅ Request |
| `host` | string | Host name of the server which is hosting the app that is generating the signal. | `x.y.z` | ✅ Annotation
✅ Click
✅ Login
✅ Request |
| `ip_address` | string | IP address of the user that generated the signal. | `80.6.99.35` | ✅ Annotation
✅ Click
✅ Login
✅ Request |
| `app_id` | string | Name of the application that is generating the signal. | `myApp` | ✅ Annotation
✅ Click
✅ Login
✅ Request |
| `annotation_type` | string | Type of the annotation signal, either "comment", "bookmark", or "like".
Required fields for "comment": ●`comment`. The comment itself. ●`target`. The target that the comment refers to.
Required fields for "bookmark": ●`title`. The bookmark title. ●`url`. The bookmark url.
✘ Request |
| `query` | string | Terms of the query. | `ipad` | ✘ Annotation
✅ Click
✘ Login
✅ Request |
| `fusion_query_id` | string | Unique ID for the `query` that is automatically generated from the Fusion response signal. | `ABkaEA11` | ✘ Annotation
✅ Click
✘ Login
✘ Request |
| `filter` | array of string | List of filters associated with the query, which in turn is associated with signal. | `[“type/tablet”,”category/electronics”]` | ✘ Annotation
✅ Click
✘ Login
✅ Request |
| `ctype` | string | Type of click. | `result` | ✘ Annotation
✅ Click
✘ Login
✘ Request |
| `res_pos` | number | Position of the clicked result within the list of results. | `3` | ✘ Annotation
✅ Click
✘ Login
✘ Request |
| `res_offset` | number | Result page. | `2` | ✘ Annotation
✅ Click
✘ Login
✘ Request |
| `url` | string | URL of the page that the signal originated from. | `http://localhost:8080/products/search` | ✘ Annotation
✘ Click
✘ Login
✅ Request |
| `path` | string | URL path of the page that the signal originated from. | `/search` | ✘ Annotation
✘ Click
✘ Login
✅ Request |
| `page_title` | string | Title of the page that the signal originated from. | `Search Page` | ✘ Annotation
✘ Click
✘ Login
✅ Request |
## Field name suffixes
Fusion can add suffixes when fields are indexed. This table lists common suffix values.
| Single Value Suffix | Multivalued Suffix | Type |
| - | - | - |
| `*_b` | `*_bs` | boolean |
| `*_d` | `*_ds` | double |
| `*_dt` | `*_dts` | date |
| `*_f` | `*_fs` | float |
| `*_i` | `*_ii` | int |
| `*_l` | `*_ls` | long |
| `*_s` | `*_ss` | string |
| `*_t` | `*_ts` | text |
## Signal field count analysis
Lucidworks recommends performing signal field count analysis to determine whether any of the fields above are missing from some of your signals.
The table below shows how to query for specific fields using the [Query Workbench](/docs/5/fusion/intro/ui-tour/query-workbench) in order to compare the number of results for each field with the total number of documents in the signals collection. In the examples in the third column, some fields appear in all 33,477,919 signals documents, while others appear in fewer documents.
| Field name | Query | Example number of documents |
| - | - | - |
| `ALL` | `*:*` | 33,477,919 |
| `count_i` | `count_i:[* TO *]` | 11,101,165 |
| `doc_id` | `doc_id:[* TO *]` | 23,216,297 |
| `id` | `id:[* TO *]` | 33,477,919 |
| `query` | `query:[* TO *]` | 19,724,598 |
| `session_id` | `session_id:[* TO *]` | 11,101,165 |
| `type` | `type:[* TO *]` | 33,477,919 |
| `user_id` | `user_id:[* TO *]` | 26,117,399 |
| `timestamp_tdt` | `timestamp_tdt:[* TO *]` | 26,117,399 |
You can also get the number of signals documents that contain all of the required fields by using the following query:
```bash theme={"dark"}
count_i:[* TO *] doc_id:[* TO *] id:[* TO *] query:[* TO *] type:[* TO *] user_id:[* TO *] timestamp_tdt:[* TO *] session_id:[* TO *]
```
## The `query_id` field
For each incoming signal, Fusion calculates a value for the `query_id` field.
[App Insights](/docs/5/fusion/getting-data-out/data-analytics/app-insights/overview) uses to create group-by-query reports like the one shown below:
The `query_id` field should not be confused with the `fusion_query_id`, which is a unique ID for each query processed by a Fusion query pipeline, or with `query_s` which is the query string.
To calculate the value, Fusion creates a hash based on `session`, `query`, and `filter` fields, then saves it into the `query_id` field.
The `filter` field can either be passed in by the search app,
or computed by the SignalFormatterStage (the first stage in the [`_signals_ingest` pipeline](/docs/5/fusion/getting-data-in/indexing/indexing-signals/overview)) using the raw filter queries. For instance, on a response signal that is generated by a query pipeline, the following `fq` query params get translated into the multi-valued `filter` field:
* Raw query parameters:
```
fq={!tag=format}format:(VHS)&fq={!tag=type}type:(Movie)
```
* `filters_s` field (created by the SearchLogger component):
```json theme={"dark"}
{!tag=format}format:(vhs) $ {!tag=type}type:(movie)
```
* `filter` field:
```
"filter":["format/VHS", "type/Movie"]
```
[App Insights](/docs/5/fusion/getting-data-out/data-analytics/app-insights/overview) uses the `filter` field to generate various reports.
## Signal type ranking
When you have defined some custom fields, it is useful to rank them according to how strongly they indicate a user’s interest in an item. While it’s not necessary to exclude certain signal types from the main signals collection, some can be excluded from signal aggregations in order to focus on the most important fields when generating [recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods).
# Stopwords Files
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/stopwords-files
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/stopwords-files
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/stopwords-files
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/181
Fusion collections are Solr collections that are managed by Fusion.
Solr itself manages a set of [resources](https://cwiki.apache.org/confluence/display/solr/Managed+Resources) for a collection.
Stopword lists are one such resource.
Stopwords are words that are filtered out of a user-entered search request. Examples of common English stopwords are *a*, *an*, and *the*.
The Fusion UI provides a stopwords manager tool that is reached from the **Stopwords** page in the **Collections** menu.
This tool lets you view the contents of all configured stopwords files in an editable browser.
# Synonyms
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/synonyms/overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/synonyms/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/synonyms/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/3240
Synonyms are interchangeable terms that can be used to expand a query to retrieve more or better search results. For example, "TV" and "television" can be considered synonyms. When a user enters one of these terms, Fusion can also look for results containing the other term.
The Synonym Detection job described in this topic is deprecated in Fusion 5.9.15 and will be removed in a future release.
Lucidworks recommends migrating to [Neural Hybrid Search](/docs/5/fusion/hybrid-search/overview), which achieves superior relevance compared to legacy machine learning methods.
You can still manually add synonyms to Solr's `synonyms.txt` file.
More specifically, synonyms are:
* Words that mean the same thing, within the context where they are used.
* Used in searches. And synonym expansion allows Fusion to return results that match the meaning of the query terms, but not the words themselves.
* Important for mapping query terms such as:
* Acronyms to their names
* Jargon to public terms
* Misspellings to correct spellings
* Old to new personal or corporate names
* Bridging the gap between the user vocabulary and terms in the original text
Fusion uses the Solr `synonyms.txt` and Solr collections, which are managed by Fusion.
Solr itself manages a set of [resources](https://cwiki.apache.org/confluence/display/solr/Managed+Resources) to apply synonym expansion, with configuration through the Fusion API and the Fusion UI.
However, Fusion synonyms are not interchangeable with Solr synonyms files.
If you have a Fusion license, see also the [Synonym Detection Jobs](/docs/5/fusion/reference/config-ref/jobs/synonym-detection) which automatically detects synonyms to use in [query rewriting](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting).
See **Use Synonym Detection** for more information.
Based on the release, synonyms are automatically created based on your AI-generated data. When you navigate to **Relevance** > **Rules** > **Rewrite** and select the **Synonym** tab, the application displays the **Synonym Detection** screen. For more information, see:
When you manually add new synonym pairs, subsequent job runs use those documents as input for machine learning to improve the job’s output. Unlike job-generated documents, manually-added query rewriting documents are never overwritten by new job output.
The course for **Query Analytics** focuses on how Fusion provides query analytics to detect and improve underperforming queries.
## Reviewing auto-generated synonym pairs
Synonyms that are automatically generated by the [Fusion 5.2.x and later Synonym Detection job](/docs/5/fusion/reference/config-ref/jobs/synonym-detection) are assigned the following status value:
* **Pending**
The confidence level is ambiguous, and the result must be reviewed by a user before it can be deployed. It will only be moved from the `_query_rewrite_staging` collection to the `_query_rewrite` collection when its status has changed to "Approved" *and* it has been published.
By default, all results from a synonym job are set to "Pending", since there are usually a limited number of synonyms, and synonym expansion can have high impact on relevancy.
### How to review a pending synonym pair result
1. Navigate to **Relevance** > **Rules** > **Rewrite**.
2. Select **Synonym** tab. The application displays the **Synonym Detection** screen.
Notice the **Status** facet on the left. Click **Pending** to view only the items that need review.
3. Click the icon next to the synonym pair.
4. In the **Status** column, select either "Approved" or "Denied".
Where alternative synonyms were detected, you can click **Suggestions** to view and select them as replacements for the displayed synonym pair.
Although the Confidence field is also editable, changing its value makes no difference.
5. Click the **Close** icon next to the updated synonym pair:
Approving a synonym pair does not automatically deploy it to the `_query_rewrite` collection. When you have finished your review, you must click **Publish** to deploy your changes.
## Adding new synonym pairs
You can manually add synonym pairs in addition to any generated by your Fusion release:
* [Fusion 5.2.x and later Synonym Detection job](/docs/5/fusion/reference/config-ref/jobs/synonym-detection)
### How to add a synonym pair
1. Navigate to **Relevance** > **Rules** > **Rewrite**.
2. Select **Synonym** tab. The application displays the **Synonym Detection** screen.
3. At the bottom of the rules list, click the icon. A new synonym pair appears at the top of the list.
4. Enter the query term.
It is not necessary to set a confidence value.
5. Select the synonym pair’s status, depending on whether you want to deploy it the next time you publish your changes ("Approved") or save it for further review ("Pending").
6. Click the check mark to save the new synonym pair.
## Publishing your changes
### How to publish updated synonym pairs
1. In the **Synonym Detection** screen, click the **PUBLISH** button.
Fusion prompts you to confirm that you want to publish your changes.
2. Click **PUBLISH**.
You can un-publish a query rewrite by changing its status to "denied", then clicking **PUBLISH**.
## Synonym types
There are three kinds of search synonyms, depending on the requirements of the search for each specific term.
### Replacement synonyms
Replacements are used to change the query, to replace it with a more standard term or terms. For example:
```bash theme={"dark"}
lucid => lucidworks
```
In this case, "lucid" by itself not an approved term, so there should be no instances where the company name is a partial word.
### One-way expansion synonyms
One-way expansions expand original terms with more standard terms while retaining the original term. However, they do not do the opposite, meaning they do not expand standard terms to the original non-standard terms:
```bash theme={"dark"}
monitor => monitor, display
```
In this case, "display" is the standard term, but "monitor" is the original term.
### Multi-way expansion synonyms
Where each term is considered equally standard, multi-way synonyms expand the query so any items with any of the terms is retrieved:
```csv theme={"dark"}
login,logon,signin,signon
```
This example shows terms that are used interchangeably by authors. For this search engine, there is no need to distinguish among them, and considerable value in increasing recall to find all items discussing this topic. However, other content stores may use them differently.
Note that "logging" and "signing" have specific meanings in many contexts, so they might not be candidates for synonyms.
**Example of synonym expansion:**
Results before synonyms:
Results after synonyms:
Viewing the query using the `debug=true` parameter shows how it is expanded:
```json theme={"dark"}
"querystring": "logon",
"parsedquery": "(+DisjunctionMaxQuery((Synonym(_text_:login _text_:logon _text_:signin _text_:signon))))/no_coord",
"parsedquery_toString": "+(Synonym(_text_:login _text_:logon _text_:signin _text_:signon))",
```
## Multi-word synonyms
**To enable multi-word synonyms in Fusion, create an Additional Parameter stage for disabling the split on whitespace tokenization process (which applies to synonyms only):**
```bash theme={"dark"}
sow=false
```
Using eDisMax, this allows the new Solr SynonymGraphFilter to create the graph representations of token streams containing overlapping synonyms of varying word counts, and expand the queries with additional terms. If using a less-than sign (\<) with eDisMax, it must be escaped using a backslash.
**Examples:**
```bash theme={"dark"}
appstudio => app studio
signup =>signup,sign up
login,log in,logon,log on,signin,sign in,signon,sign on
```
Multi-word synonyms work just like single-word synonyms, expanding the parsed query with additional query terms. For Solr details, see: [Multi-Word Synonyms: Solr Adds Query-Time Support](https://lucidworks.com/post/multi-word-synonyms-solr-adds-query-time-support//).
# Template Expressions
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/template-expressions/overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/template-expressions/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/template-expressions/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/10911
Template expressions are a way to map field values in a document. They are used to configure some Fusion index pipeline stages, query pipeline stages, and messaging services. When the value of a configuration parameter is a template expression, that expression is dynamically evaluated at runtime by the [StringTemplate](https://github.com/antlr/stringtemplate4/blob/master/doc/cheatsheet.md) library.
Fusion template expressions are delimited by angle bracket characters `<` and `>`.
The expression consists of the name of a variable in the scope of that component.
Since these variables are Java objects, if object **foo** of type Foo has a field named **bar**, the expression **``** will evaluate to the string representation of the contents of field **bar**.
For example, given a PipelineDocument where a field named **title** has a value of **Star Wars**, the template expression `` evaluates to **Star Wars**.
## Index pipeline stage template expression variables
Index stages have the following variables available:
* `ctx`. [Context](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/apollo/pipeline/Context.html): used to pass variables between stages
* `doc`. [PipelineDocument](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/apollo/common/pipeline/PipelineDocument.html): Fusion internal representation of a Solr Document. Please note that if you change this class in ways that affect the serialized form (adding fields, for example) you need to change the deserialization code in `GenericJSONDocumentMessageReader`.
The format required is ``.
## Query pipeline stage template expression variables
Query stages have the following variables available:
* `ctx`. [Context](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/apollo/pipeline/Context.html): used to pass variables between stages
* `reqResp`. [QueryRequestAndResponse](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/apollo/pipeline/query/QueryRequestAndResponse.html): deserializes pre-request parameters and makes them available
The format required is ``.
## Example
Check that a query has an `fq` filter on a given field for a given value:
In the UI, set property name to `ids` and Property Value to `req.hasParam("fq")`
## Messaging services template expression variables
Messages have a set of variables which correspond to the parts of a system message:
* id
* to
* from
* subject
* body
* type
* schedule
A system message is the result of evaluating the following expression:
```xml wrap theme={"dark"}
```
# Head/Tail Analysis
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/underperforming-queries
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/query-enhancement/underperforming-queries
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/query-enhancement/underperforming-queries
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/486
Head/tail analysis is:
* Also known as the Underperforming Query Rewriting feature
* Uses signals data to identify underperforming queries
* Suggests improved queries that could produce better conversion rates
This feature is deprecated in Fusion 5.9.15 and will be removed in a future release.
Lucidworks recommends migrating to [Neural Hybrid Search](/docs/5/fusion/hybrid-search/overview), which achieves superior relevance compared to legacy machine learning methods.
When an incoming query contains a matching underperforming query, the original query is replaced by an improved query.
These improvements can be:
* Suggested by the [Head/Tail Analysis Spark job](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis) operating on your signals data
* Created manually using the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor) or underlying API
Query improvements are applied in the [Text Tagger stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/text-tagger-query-stage) of the query pipeline.
The [Head/Tail Analysis job](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis) automatically creates query improvements based on your AI-generated data. When you navigate to **Relevance** > **Rules** > **Rewrite** and select the **Head/Tail** tab, the application displays the **Underperforming Query Rewriting** screen.
Job-generated query improvements are always assigned an initial status of "Pending", never "Auto". Query improvements must be explicitly approved and published in order to be copied to the `COLLECTION_NAME_query_rewrite` collection.
The course for **Query Analytics** focuses on how Fusion provides query analytics to detect and improve underperforming queries.
## Reviewing auto-generated query improvements
Query improvements that are automatically generated by the [Head/Tail Analysis job](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis) are assigned the following status value:
* **Auto**
All results will be automatically deployed to the `_query_rewrite` collection.\
No action is required on these results, but you can edit them if you wish.
## Adding new query improvements
You can manually add query improvements in addition to any generated by the [Head/Tail Analysis job](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis).
### How to add a query improvement
1. Navigate to **Relevance** > **Rules** > **Rewrite**.
2. Select the **Head/Tail** tab. The application displays the **Underperforming Query Rewriting** screen.
3. At the bottom of the rules list, click the icon.\
A new query improvement appears at the top of the list:
4. Enter the underperforming query.
5. Enter one or more query improvements.
It is not necessary to set a confidence value.
6. Select the query improvement’s status, depending on whether you want to deploy it the next time you publish your changes ("Approved") or save it for further review ("Pending").
7. Click the check mark to save the new query improvement.
## Publishing your changes
### How to publish updated query improvements
1. In the **Underperforming Query Rewriting** screen, click the **PUBLISH** button.\
Fusion prompts you to confirm that you want to publish your changes.
2. Click **PUBLISH**.
You can un-publish a query rewrite by changing its status to "denied", then clicking **PUBLISH**.
For more information about how to use underperforming query rewrites in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor), see **Use Query Rewrites in the Rules Editor**.
This article describes how to use the Rewrites screen in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor).
## Query rewrite rule types
* **Head/Tail.** Replace underperforming queries with queries that produce higher click-through rates. Query improvements always require review before they are published. Schedule the Head/Tail Analysis job to get AI-generated query improvements based on your signals collection. \
\
Manually created Head/Tail rewrites must provide an "Improved Query" in a format appropriate to the type of request the rewrite will be applied to. For Legacy requests, this means a valid Solr query fit for use as the `q` parameter. For DSL requests this means a JSON string representing the main DSL query. \
\
For more information, see:
* [Underperforming Query Rewriting](/docs/5/fusion/getting-data-out/query-enhancement/underperforming-queries)
* [Head/Tail Analysis Jobs](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis)
* **Misspelling.** Fusion replaces misspelled query terms with their spelling corrections for better search results. Schedule the Token and Phrase Spell Correction job to get AI-generated spelling corrections based on your signals collection. For more information, see [Misspelling Detection](/docs/5/fusion/getting-data-out/query-enhancement/misspelling-detection).
* **Phrase.** Known phrases are boosted when they appear in new queries. Schedule the Phrase Extraction job to get AI-generated phrases based on your signals collection.
For more information, see [Phrase Detection](/docs/5/fusion/getting-data-out/query-enhancement/phrase-detection).
* **Synonym.** Synonyms are used to expand queries so that they include all known synonyms of the query terms. To retrieve AI-generated synonyms based on your signals collection, schedule the [Synonym Detection job](/docs/5/fusion/reference/config-ref/jobs/synonym-detection).
* **Remove Words** - Remove particular phrases from queries. Unlike other rewrites, these should be entered manually and are not generated by a job. For more information, see [Remove Words](/docs/5/fusion/getting-data-out/query-enhancement/remove-words).
## Facets
### Facet menu
To facet the rules list by **Status** or **Published**, click one of the facet items within the facet categories.
To facet the rules by **Confidence**, click and drag either slider to set the range of confidence you would like to see:
Faceting rules by confidence is not available for Head/Tail rewrites.
### Search
Enter a term in the search bar to quickly facet your query rewrite rules.
## Rules
Begin by selecting one of the query rewrite rule types you will create.
### Create a rule
1. Click **+Add**.
2. In the new table row, complete the required fields, and additional fields as needed.
3. Click the checkmark to save, or close to cancel the changes.
### Edit a rule
1. Hold the pointer over the rule to edit.
2. Click the pencil icon.
3. Enter desired changes and click the checkmark to save, or close to cancel the changes.
### Approve/deny rules
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To modify multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Approve** or **Deny** as needed.
### Publish a rule
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To publish multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Publish Selected** to publish the selected rules.
#### Publish an individual rule
1. Hold the pointer over a rule.
2. Click the button.
3. To confirm, click **Publish**.
### Delete rules
#### Delete a single rule
1. Hold the pointer over a rule.
2. Click the trash can.
3. To confirm, click **Delete**.
#### Delete multiple rules
1. Hold the pointer over a rule.
2. Select the checkbox that displays for that rule.
3. To delete multiple rules, hold the pointer over each rule and select its corresponding checkbox.
4. Click **Delete**.
5. To confirm, click **Delete** again.
### Sort rules
1. Hold the pointer over column titles. A icon displays for some titles to indicate the rules list can be sorted by that column.
2. Click the title column to sort the rules.
The column that is currently sorting the rules list is indicated by an or icon.
### Export rules
To export the rules list, click **Export**. This generates a `.csv` file that contains your rules.
## Columns
### Customize columns
To customize columns:
1. Hold the pointer over the row of column titles.
2. Click the button.
3. In the Customize Columns to Display screen, select the checkboxes for the columns you want to display. Clear the checkboxes for columns you do not want to display.
4. Click **Save**. To exit without saving your changes, click **Cancel**.
## Query rewrite jobs post-processing cleanup
To perform more extensive cleanup of query rewrites, complete the procedures in **Query Rewrite Jobs Post-processing Cleanup**.
The Synonym Detection job uses the output of the Misspelling Detection job and Phrase Extraction job. Therefore, post processing *must* occur in the order specified in this topic for the [Synonym detection job cleanup](#synonym-detection-job-cleanup), [Phrase extraction job cleanup](#phrase-extraction-job-cleanup), and [Misspelling detection job cleanup](#misspelling-detection-job-cleanup) procedures. The [Head-Tail Analysis job cleanup](#head-tail-analysis-job-cleanup) can occur in any order.
## Synonym detection job cleanup
Use this job to remove low confidence synonyms.
### Prerequisites
Complete this:
* **AFTER** the Misspelling Detection and Phrase Extraction jobs have successfully completed.
* **BEFORE** removing low confidence synonym suggestions generated in the *post processing* phrase extraction cleanup and misspelling detection cleanup procedures detailed later in this topic.
### Remove low confidence synonym suggestions
Use either a [Synonym cleanup method 1 - API call](#synonym-cleanup-method-1-api-call) or the [Synonym cleanup method 2 - Fusion Admin UI](#synonym-cleanup-method-2-fusion-admin-ui) to remove low confidence synonym suggestions.
#### Synonym cleanup method 1 - API call
1. Open the `delete_lowConf_synonyms.json` file.
```json theme={"dark"}
{
"type" : "rest-call",
"id" : "DC_Large_QR_DELETE_LOW_CONFIDENCE_SYNONYMS",
"callParams" : {
"uri" : "solr://DC_Large_query_rewrite_staging/update",
"method" : "post",
"queryParams" : {
"wt" : "json"
},
"headers" : { },
"entity" : "type:synonym AND confidence:[0 TO 0.0005]"
},
"type" : "rest-call",
"type" : "rest-call"
}
```
**REQUEST ENTITY** specifies the threshold for low confidence synonyms. Edit the upper range from 0.0005 to increase or decrease the threshold based on your data.
2. Enter `` in the **uri** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
3. Change the `id` field if applicable.
4. Specify the **upper confidence level** in the **entity** field.
The **entity** field specifies the threshold for low confidence synonyms. Edit the upper range to increase or decrease the threshold based on your data.
#### Synonym cleanup method 2 - Fusion Admin UI
1. Log in to Fusion and select **Collections > Jobs**.
2. Select **Add+ > Custom and Other Jobs > REST Call**.
3. Enter **delete-low-confidence-synonyms** in the **ID** field.
4. Enter `` in the **ENDPOINT URI** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
5. Enter **POST** in the **CALL METHOD** field.
6. In the **QUERY PARAMETERS** section, select **+** to add a property.
7. Enter **wt** in the **Property Name** field.
8. Enter **json** in the **Property Value** field.
9. In the **REQUEST PROTOCOL HEADERS** section, select **+** to add a property.
10. Enter the following as a **REQUEST ENTITY (AS STRING)**
`type:synonym AND confidence: [0 TO 0.0005]`
**REQUEST ENTITY** specifies the threshold for low confidence synonyms. Edit the upper range from 0.0005 to increase or decrease the threshold based on your data.
### Delete all synonym suggestions
To delete *all* of the synonym suggestions, enter the following in the **REQUEST ENTITY** section:
`type:synonym`
This entry may be helpful when tuning the synonym detection job and testing different configuration parameters.
## Phrase extraction job cleanup
Use this job to remove low confidence phrase suggestions.
### Prerequisites
Complete this:
* **AFTER** you complete [Synonym detection job cleanup](#synonym-detection-job-cleanup)
### Remove low confidence phrase suggestions
Use either a [Phrase cleanup method 1 - API call](#phrase-cleanup-method-1-api-call) or the [Phrase cleanup method 2 - Fusion Admin UI](#phrase-cleanup-method-2-fusion-admin-ui) to remove low confidence phrase suggestions.
#### Phrase cleanup method 1 - API call
1. Open the `delete_lowConf_phrases.json` file.
```json theme={"dark"}
{
"type" : "rest-call",
"id" : "DC_Large_QR_DELETE_LOW_CONFIDENCE_PHRASES",
"callParams" : {
"uri" : "solr://DC_Large_query_rewrite_staging/update",
"method" : "post",
"queryParams" : {
"wt" : "json"
},
"headers" : { },
"entity" : " type:phrase AND confidence:[0 TO ]"
},
"type" : "rest-call",
"type" : "rest-call"
}
```
2. Enter `` in the **uri** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
3. Change the **id** field if applicable.
4. Specify the **upper confidence level** in the **entity** field.
The **entity** field specifies the threshold for low confidence phrases. Edit the upper range to increase or decrease the threshold based on your data.
#### Phrase cleanup method 2 - Fusion Admin UI
1. Log in to Fusion and select **Collections > Jobs**.
2. Select **Add+ > Custom and Other Jobs > REST Call**.
3. Enter **remove-low-confidence-phrases** in the **ID** field.
4. Enter `` in the **ENDPOINT URI** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
5. Enter **POST** in the **CALL METHOD** field.
6. In the **QUERY PARAMETERS** section, select **+** to add a property.
7. Enter **wt** in the **Property Name** field.
8. Enter **json** in the **Property Value** field.
9. In the **REQUEST PROTOCOL HEADERS** section, select **+** to add a property.
10. Enter the following as a **REQUEST ENTITY (AS STRING)**
`type:phrase AND confidence: [0 TO ]`
**REQUEST ENTITY** specifies the threshold for low confidence phrases. Edit the upper range to increase or decrease the threshold based on your data.
### Delete all phrase suggestions
To delete *all* of the phrase suggestions, enter the following in the **REQUEST ENTITY** section:
`type:phrase`
This entry may be helpful when tuning the phrase extraction job and testing different configuration parameters.
## Misspelling detection job cleanup
Use this job to remove low confidence spellings (also referred to as misspellings).
### Prerequisites
Complete this:
* **AFTER** you complete [Synonym detection job cleanup](#synonym-detection-job-cleanup) and [Phrase extraction job cleanup](#phrase-extraction-job-cleanup)
### Remove misspelling suggestions
Use either a [Misspelling cleanup method 1 - API call](#misspelling-cleanup-method-1-api-call) or the [Misspelling cleanup method 2 - Fusion Admin UI](#misspelling-cleanup-method-2-fusion-admin-ui) to remove misspelling suggestions.
#### Misspelling cleanup method 1 - API call
1. Open the `delete_lowConf_misspellings.json` file.
```json theme={"dark"}
{
"type" : "rest-call",
"id" : "DC_Large_QR_DELETE_LOW_CONFIDENCE_MISSPELLINGS",
"callParams" : {
"uri" : "solr://DC_Large_query_rewrite_staging",
"method" : "post",
"queryParams" : {
"wt" : "json"
},
"headers" : { },
"entity" : "type:spell AND confidence:[0 TO 0.5]"
},
"type" : "rest-call",
"type" : "rest-call"
}
```
2. Enter `` in the **uri** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
3. Change the **id** field if applicable.
4. Specify the **upper confidence level** in the **entity** field.
The **entity** field specifies the threshold for low confidence spellings. Edit the upper range to increase or decrease the threshold based on your data.
#### Misspelling cleanup method 2 - Fusion Admin UI
1. Log in to Fusion and select **Collections > Jobs**.
2. Select **Add+ > Custom and Other Jobs > REST Call**.
3. Enter **remove-low-confidence-spellings** in the **ID** field.
4. Enter `` in the **ENDPOINT URI** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
5. Enter **POST** in the **CALL METHOD** field.
6. In the **QUERY PARAMETERS** section, select **+** to add a property.
7. Enter **wt** in the **Property Name** field.
8. Enter **json** in the **Property Value** field.
9. In the **REQUEST PROTOCOL HEADERS** section, select **+** to add a property.
10. Enter the following as a **REQUEST ENTITY (AS STRING)**
`type:spell AND confidence: [0 TO 0.5]`
**REQUEST ENTITY** specifies the threshold for low confidence spellings. Edit the upper range from 0.5 to increase or decrease the threshold based on your data.
### Delete all misspelling suggestions
To delete *all* of the misspelling suggestions, enter the following in the **REQUEST ENTITY** section:
`type:spell`
This entry may be helpful when tuning the misspelling detection job and testing different configuration parameters.
## Head-tail analysis job cleanup
The head-tail analysis job puts tail queries into one of multiple reason categories. For example, a tail query that includes a number might be assigned to the 'numbers' reason category. If the output in a particular category is not useful, you can remove it from the results. The examples in this section remove the numbers category.
### Prerequisites
The head-tail analysis job cleanup does *not* have to occur in a specific order.
### Remove head-tail analysis query suggestions
Use either a [Head-tail analysis cleanup method 1 - API call](#head-tail-analysis-cleanup-method-1-api-call) or the [Head-tail analysis cleanup method 2 - Fusion Admin UI](#head-tail-analysis-cleanup-method-2-fusion-admin-ui) to remove query category suggestions.
#### Head-tail analysis cleanup method 1 - API call
1. Open the `delete_lowConf_headTail.json` file.
```json theme={"dark"}
{
"type" : "rest-call",
"id" : "DC_Large_QR_HEAD_TAIL_CLEANUP",
"callParams" : {
"uri" : "solr://DC_Large_query_rewrite_staging/update",
"method" : "post",
"queryParams" : {
"wt" : "json"
},
"headers" : { },
"entity" : "reason_code_s:(\"number\" \"number spelling\" \"number rare-term\" \"question number other-specific\" \"number others\" \"number other-specific\" \"number other-extra\" \"product number other-specific\" \"product number other-extra\" \"product number spelling\" \"product number others\" \"product number rare-term\" \"product question number\" \"product number re-wording\" \"question number other-extra\" \"number re-wording\")"
},
"type" : "rest-call",
"type" : "rest-call"
}
```
2. Enter `` in the **uri** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
3. Change the **id** field if applicable.
#### Head-tail analysis cleanup method 2 - Fusion Admin UI
1. Log in to Fusion and select **Collections > Jobs**.
2. Select **Add+ > Custom and Other Jobs > REST Call**.
3. Enter **remove-low-confidence-head-tail** in the **ID** field.
4. Enter `` in the **ENDPOINT URI** field. An example URI value for an app called `DC_Large` would be `DC_Large_query_rewrite_staging/update`.
5. Enter **POST** in the **CALL METHOD** field.
6. In the **QUERY PARAMETERS** section, select **+** to add a property.
7. Enter **wt** in the **Property Name** field.
8. Enter **json** in the **Property Value** field.
9. In the **REQUEST PROTOCOL HEADERS** section, select **+** to add a property.
10. Enter the following as a **REQUEST ENTITY (AS STRING)**
```
reason_code_s:("number" "number spelling" "number rare-term" "question number other-specific" "number others" "number other-specific" "number other-extra" "product number other-specific" "product number other-extra" "product number spelling" "product number others" "product number rare-term" "product question number" "product number re-wording" "question number other-extra" "number re-wording")
```
### Delete all head-tail suggestions
To delete *all* of the head-tail suggestions, enter the following in the **REQUEST ENTITY** section:
`type:tail`
This entry may be helpful when tuning the head-tail job and testing different configuration parameters.
# Milvus
Source: https://doc.lucidworks.com/docs/5/fusion/intro/fusion-stack/milvus
[localhost link]: http://localhost:3000/docs/5/fusion/intro/fusion-stack/milvus
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/intro/fusion-stack/milvus
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/8821
**Deprecated**
As of Fusion 5.9.5, Milvus is deprecated and will be removed in a future release. Use [Seldon](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/vectorize-query-via-seldon-query-stage) or [Lucidworks AI](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/lwai-vectorize-query) vector query stages instead. For more information, see the following topics:
* [Deprecations and Removals](/docs/5/fusion/deprecations-and-removals)
* [Seldon](/docs/5/fusion/intro/fusion-stack/seldon)
* [Lucidworks AI](/docs/lw-platform/lw-ai/overview)
[Milvus](https://milvus.io/) is an open source vector similarity search engine, integrated into Fusion to streamline its deep learning capabilities and reduce the workload on Solr. See [Milvus v0.10.2 documentation](https://www.milvus.io/docs/v0.10.2/overview.md).
## Milvus collections
Milvus stores the vectors produced by machine learning models in one or more Milvus collections.
For each collection, the following parameters need to be set:
* `Collection Name`. A name for the Milvus collection you are creating. Milvus requires that collection names only include alphanumeric characters and the underscore `_`.
* `Dimension`. The dimension size of the vectors to store in this Milvus collection. The Dimension should match the size of the vectors returned by the encoding model. For example, the `Smart Answers Pre-trained Coldstart` models output vectors of 512 dimension size. Dimensionality of encoders trained by `Smart Answers Supervised Training` job depends on the provided parameters and printed in the training job logs. See **Train a Smart Answers Supervised Model** for more details.
* `Index file size`. Vectors are added to the raw data files and then indexed. Files larger than this will trigger index building for raw data files.
* `Metric`. The type of metric used to determine how close the search vector is to the vectors in the collection. It is used to calculate the vector similarity scores. In most cases we suggest to use Cosine Similarity which can be obtained by using the `Inner Product` metric over normalized vectors. It produces values between -1 and 1, where a higher value means higher similarity.
The [Supervised solution for Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/faq-solution) begins with training a model using your existing data and the [Smart Answers Supervised Training job](/docs/5/fusion/reference/config-ref/jobs/smart-answers-supervised-training), as explained in this topic. The job includes an auto-tune feature that you can use instead of manually tuning the configuration.
See also [Advanced Model Training Configuration for Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/smart-answers-advanced-model-config).
## Training job requirements
**Storage**
150GB plus 2.5 times the total input data size.
**Processor and memory**
The memory requirements depend on whether you choose GPU or CPU processing:
| GPU | CPU |
| - | - |
|
one core
11GB RAM
|
32 cores
32GB RAM
|
If your training data contains more than 1 million entries, use GPU.
## Prepare the input data
1. Format your input data as question/answer pairs, that is, a query and its corresponding response in each row.
You can do this in any format that Fusion supports.
If there are multiple possible answers for a unique question, then repeat the questions and put the pair into different rows to make sure each row has one question and one answer, as in the example JSON below:
```json theme={"dark"}
[{"question":"How to transfer personal auto lease to business auto lease?","answer":"I would approach the lender that you are getting the lease from..."}
{"question":"How to transfer personal auto lease to business auto lease?","answer":"See what the contract says about transfers or subleases..."}]
```
2. Index the input data in Fusion.
If you wish to have the training data in Fusion, index it into a separate collection for training data such as `model_training_input`. Otherwise you can use it directly from the cloud storage.
## Configure the training job
1. In Fusion, navigate to **Collections** > **Jobs**.
2. Select **Add** > **Smart Answers Supervised Training**:
3. In the **Training Collection** field, specify the input data collection that you created when you [prepared the input data](#prepare-the-input-data).
In Fusion 5.3 and later, you can also configure this job to read from or write to cloud storage.
4. Enter the names of the **Question Field** and the **Answer Field** in the training collection.
5. Enter a **Model Deployment Name**.
The new machine learning model will be saved in the blob store with this name. You will reference it later when you configure your pipelines.
6. *Fusion 5.3 and later:* Configure the **Model base**.
There are several pre-trained word and [BPE](https://nlp.h-its.org/bpemb) embeddings for different languages, as well as a few pre-trained BERT models.
If you want to train custom embeddings, select `word_custom` or `bpe_custom`. This trains Word2vec on the provided data and specified fields. It might be useful in cases when your content includes unusual or domain-specific vocabulary.
If you have content in addition to the query/response pairs that can be used to train the model, then specify it in the **Texts Data Path**.
When you use the pre-trained embeddings, the log shows the percentage of processed vocabulary words. If this value is high, then try using custom embeddings.
The job trains a few (configurable) RNN layers on top of word embeddings or fine-tunes a BERT model on the provided training data. The result model uses an attention mechanism to average word embeddings to obtain the final single dense vector for the content.
Dimension size of vectors for Transformer-based models is 768. For RNN-based models it is 2 times the number units of the last layer. To find the dimension size: download the model, expand the zip, open the log and search for `Encoder output dim size:` line. You might need this information when creating collections in Milvus.
7. *Optional:* Check **Perform auto hyperparameter tuning** to use auto-tune.
Although training module tries to select the most optimal default parameters based on the training data statistics, auto-tune can extend it by automatically finding even better training configuration through hyper-parameter search.
Although this is a resource-intensive operation, it can be useful to identify the best possible RNN-based configuration. Transformer-based models like BERT are not used during auto hyperparameter tuning as they usually perform better yet they are much more expensive on both training and inference time.
8. Click **Save**.
If using solr as the training data source ensure that the source collection contains the `random_*` dynamic field defined in its `managed-schema.xml`. This field is required for sampling the data. If it is not present, add the following entry to the `managed-schema.xml` alongside other dynamic fields `` and \ alongside other field types.
9. Click **Run** > **Start**.
After training is finished the model is deployed into the cluster and can be used in index and query pipelines.
## Next steps
1. See A Smart Answers Supervised Job’s Status and Output
2. Configure The Smart Answers Pipelines
3. Evaluate a Smart Answers Query Pipeline
The Metric options for Milvus collections are: Euclidean, Inner Product, Hamming, Jaccard, Tanimoto, Substructure, and Superstructure. The following metrics all require that the Dimension is a multiple of 8: Hamming, Jaccard, Tanimoto, Substructure, and Superstructure. Otherwise the Create Milvus Collection job will fail. See [Milvus metrics documentation](https://www.milvus.io/docs/v0.10.2/metric.md#floating) for more details.
Milvus can be configured to pre-load one or more collections via the Milvus parameter `preload_collection`. Fusion does not currently ship with a method to manage this parameter, but the Milvus API can be used directly (after first setting up port forwarding via `kubectl port-forward milvus-writeable-- 19121:19121`).
For an example of how to create and utilize a Milvus collection, see **Configure the Smart Answers Pipelines (5.3 and later)**.
Before beginning this procedure, train a machine learning model using either the FAQ method or the cold start method.
Regardless of how you set up your model, the deployment procedure is the same:
1. [Create the `Milvus` collection](#create-the-milvus-collection).
2. [Configure the `smart-answers` index pipeline](#configure-the-index-pipeline).
3. [Configure the `smart-answers` query pipeline](#configure-the-query-pipeline).
See also [Best Practices](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/best-practices), [Advanced Model Training Configuration for Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/smart-answers-advanced-model-config), and Smart Answers Detailed Pipeline Setup.
## Create the Milvus collection
For complete details about job configuration options, see the [Create Collections in Milvus job](/docs/5/fusion/reference/config-ref/jobs/create-collections-in-milvus).
1. Navigate to **Collections** > **Jobs** > **Add +** and select **Create Collections in Milvus**.
2. Configure the job:
1. Enter an ID for this job.
2. Under **Collections**, click **Add**.
3. Enter a collection name.
4. In the **Dimension** field, enter the dimension size of vectors to store in this Milvus collection. The Dimension should match the size of the vectors returned by the encoding model. For example, the `Smart Answers Pre-trained Coldstart` models outputs vectors of 512 dimension size. Dimensionality of encoders trained by `Smart Answers Supervised Training` job depends on the provided parameters and printed in the training job logs.
3. Click **Save**.
The `Create Collections in Milvus` job can be used to create multiple collections at once. In this image, the first collection is used in the [indexing](#configure-the-index-pipeline) and [query](#configure-the-query-pipeline) steps. The other two collections are used in the [example](#pipeline-setup-example).
4. Click **Run** > **Start** to run the job.
## Configure the index pipeline
1. Open the Index Workbench.
2. Load or create your datasource using the default smart-answers index pipeline.
3. Configure the
[Encode into Milvus stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/encode-into-milvus-index-stage):
1. change the value of **Model ID** to match the model deployment name you chose when you configured the model training job.
2. Change `Field to Encode` to the document field name to be processed and encoded into dense vectors.
3. Ensure the `Encoder Output Vector` matches the output vector from the chosen model.
4. Ensure the `Milvus Collection Name` matches the collection name created via the `Create Milvus Collection` job.
To test out your settings, turn on `Fail on Error` in the `Encode into Milvus` stage and Apply the changes. This will cause an error message to display if any settings need to be changed.
4. Save the datasource.
5. Index your data.
## Configure the query pipeline
1. Open the Query Workbench.
2. Load the default smart-answers query pipeline.
3. Configure the [Milvus Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-query-stage):
1. Change the **Model ID** value to match the model deployment name you chose when you configured the model training job.
2. Ensure the `Encoder Output Vector` matches the output vector from the chosen model.
3. Ensure the `Milvus Collection Name` matches the collection name created via the `Create Milvus Collection` job.
4. `Milvus Results Context Key` can be changed as needed. It will be used in the Milvus Ensemble Query Stage to calculate the query score.
4. In the [Milvus Ensemble Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-ensemble-query-stage), update the `Ensemble math expression` as needed based on your model and the name used in the prior stage for the storing the Milvus results.
In versions 5.4 and later, you can also set the `Threshold` so that the Milvus Ensemble Query Stage will only return items with a score **greater than** or **equal to** the configured value.
5. Save the query pipeline.
## Pipeline Setup Example
### Index and retrieve the question and answer together
To show question and answer together in one document (that is, treat the question as the title and the answer as the description), you can index them together in the same document. You can still use the default `smart-answers` index and query pipelines with a few additional changes.
Prior to configuring the Smart Answers pipelines, use the `Create Milvus Collection` job to create two collections, `question_collection` and `answer_collection`, to store the encoded "questions" and the encoded "answers", respectively.
#### Index Pipeline
As shown in the pictures below, you will need two Encode into Milvus stages, named Encode Question and Encode Answer respectively.
**Encode Question (Encode Into Milvus) stage**
**Encode Answer (Encode Into Milvus) stage**
In the Encode Question stage, specify `Field to Encode` to be `title_t` and change the `Milvus Collection Name` to match the new Milvus collection, `question_collection`.
In the Encode Answer stage, specify `Field to Encode` to be `description_t` and change the `Milvus Collection Name` to match the new Milvus collection, `answer_collection`.
#### Query Pipeline
Since we have two dense vectors generated during indexing, at query time we need to compute both query to question distance and query to answer distance. This can be set up as the pictures shown below with two Milvus Query Stages, one for each of the two Milvus collections.
To store those two distances separately, the `Milvus Results Context Key` needs to be different in each of these two stages.
In the Query Questions stage, we set the `Milvus Results Context Key` to `milvus_questions` and the Milvus collection name to `question_collection`.
Query Questions (Milvus Query) stage:
In the Query Answers stage, we set the `Milvus Results Context Key` to `milvus_answers` and the Milvus collection name to `answer_collection`.
Query Answers (Milvus Query) stage:
Now we can ensemble them together with the Milvus Ensemble Query Stage with the `Ensemble math expression` combining the results from the two query stages. If we want the question scores and answer scores weighted equally, we would use: `0.5 * milvus_questions + 0.5 * milvus_answers`.
This is recommended especially when you have limited FAQ dataset and want to utilize both question and answer information.
**Milvus Ensemble Query stage**
## Evaluate the query pipeline
The [Evaluate QnA Pipeline job](/docs/5/fusion/reference/config-ref/jobs/smart-answers-evaluate-pipeline) evaluates the rankings of results from any Smart Answers pipeline and finds the best set of weights in the ensemble score.
## Detailed pipeline setup
Typically, you can use the [default pipelines](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/overview) included with Fusion AI. These pipelines now utilize Milvus to store encoded vectors and to calculate vector similarity. This topic provides information you can use to customize the Smart Answers pipelines.
| | | |
| - | - | - |
| **"smart-answers" index pipeline** | | [Encode into Milvus stage](#the-encode-into-milvus-index-stage) |
| **"smart-answers" query pipeline** | |
|
### Create the Milvus collection
Prior to indexing data, the [Create Collections in Milvus job](/docs/5/fusion/reference/config-ref/jobs/create-collections-in-milvus) can be used to create the Milvus collection(s) used by the Smart Answers pipelines (see [Milvus overview](/docs/5/fusion/intro/fusion-stack/milvus)).
* `Job ID`. A unique identifier for the job.
* `Collection Name`. A name for the Milvus collection you are creating. This name is used in both the Smart Answer Index and the Smart Answer Query pipelines.
* `Dimension`. The dimension size of the vectors to store in this Milvus collection. The Dimension should match the size of the vectors returned by the encryption model. For example, if the model was created with either the `Smart Answers Coldstart Training` job or the `Smart Answers Supervised Training` job with the Model Base `word_en_300d_2M`, then the dimension would be 300.
* `Index file size`. Files with more documents than this will cause Milvus to build an index on this collection.
* `Metric`. The type of metric used to calculate vector similarity scores. `Inner Product` is recommended. It produces values between 0 and 1, where a higher value means higher similarity.
### Index pipeline setup
**Stages in the default "smart-answers" index pipeline**
Only one custom index stage needs to be configured in your index pipeline, the [Encode into Milvus index stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/encode-into-milvus-index-stage).
#### The Encode into Milvus Index Stage
If you are using a dynamic schema, make sure this stage is added *after* the [Solr Dynamic Field Name Mapping stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/solr-dynamic-field-name-mapping-index-stage).
The [Encode into Milvus](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/encode-into-milvus-index-stage) index stage uses the specified model to encode the `Field to Encode` and store it in Milvus in the given Milvus collection.
There are several required parameters:
* `Model ID`. The ID of the model.
* `Encoder Output Vector`. The name of the field that stores the compressed dense vectors output from the model. Default value: `vector`.
* `Field to Encode`. The text field to encode into a dense vector, such as `answer_t` or `body_t`.
* `Milvus Collection Name`. The name of the collection you created via the [Create Milvus Collection](/docs/5/fusion/reference/config-ref/jobs/create-collections-in-milvus) job, which will store the dense vectors. When creating the collection you specify the type of **Metric** to use to calculate vector similarity.
This stage can be used multiple times to encode additional fields, each into a different Milvus collection.
### Query pipeline setup
#### The Query Fields stage
The first stage is **Query Fields**. For more information see the [Query Fields stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/search-fields-query-stage).
#### The Milvus Query stage
The [Milvus Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-query-stage) stage encodes the query into a vector using the specified model. It then performs a vector similarity search against the specified Milvus collection and returns a list of the best document matches.
* `Model ID`. The ID of the model used when configuring the model training job.
* `Encoder Output Vector`. The name of the output vector from the specified model, which will contain the query encoded as a vector. Defaults to vector.
* `Milvus Collection Name`. The name of the collection that you used in the `Encode into Milvus` index stage to store the encoded vectors.
* `Milvus Results Context Key`. The name of the variable used to store the vector distances. It can be changed as needed. It will be used in the Milvus Ensemble Query Stage to calculate the query score for the document.
* `Number of Results`. The number of highest scoring results returned from Milvus.
This stage would typically be used the same number of times that the `Encode into Milvus` index stage is used, each with a different Milvus collection and a different `Milvus Results Context Key`.
#### The Milvus Ensemble Query stage
The [Milvus Ensemble Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-ensemble-query-stage) takes the results of the Milvus Query stage(s) and calculates the `ensemble score`, which is used to return the best matches.
* `Ensemble math expression`. The mathematical expression used to calculate the `ensemble score`. It should reference the value(s) variable name specified in the `Milvus Results Context Key` parameter in the Milvus Query stage.
* `Result field name`. The name of the field used to store the `ensemble score`. It defaults to `ensemble_score`.
* `Threshold`- A parameter that filters the stage results to remove items that fall below the configured score. Items with a score at, or above, the threshold will be returned.
The **Threshold** feature is only available in Fusion 5.4 and later.
#### The Milvus Response Update Query stage
The [Milvus Response Update Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-response-update-stage) stage does not need to be configured and can be skipped if desired. It inserts the Milvus values, including the `ensemble_score`, into each of the returned documents, which is particularly useful when there is more than one `Milvus Query Stage`. This stage needs to come after the `Solr Query` stage.
## Short answer extraction
By default, the question-answering query pipelines return complete documents that answer questions. Optionally, you can extract just a paragraph, a sentence, or a few words that answer the question.
## Milvus indexes
When a Milvus collection is large, creating an index can speed up the similarities search. If you do not create an index on a collection then a FLAT index is used. See [Milvus index documentation](https://www.milvus.io/docs/v0.10.2/index.md) for more details.
When you create an index on a collection, in addition to the `collection name` you will need to specify:
* `Index Type`. The index type must be one of \[`ANNOY`, `FLAT`, `HNSW`, `IVF_PQ`, `IVF_SQ8`, `IVF_SQ8H`, `IVFLAT`, `RNSG`]. We recommend you use the Fusion job default value of `HNSW`.
* `Index Parameters`. Parameters used to create an index in Milvus. These required parameters are specific to the chosen `Index Type`. For example, `nlist=4096` for IVF indexes or `M=36` and `efConstruction=500` for HNSW indexes.
If you create an index for the collection different from FLAT, index specific query parameters have to be added as Search Parameters in the Milvus Query stage. Refer to the [Milvus supported indexes](https://www.milvus.io/docs/v0.10.2/index.md#Indexes-that-Milvus-supports) for query parameters for each index type.
## Milvus management jobs
Fusion ships with several jobs for managing collections and indexes in Milvus:
* [Create Collections in Milvus](/docs/5/fusion/reference/config-ref/jobs/create-collections-in-milvus)
* [Delete Collections in Milvus](/docs/5/fusion/reference/config-ref/jobs/delete-collections-in-milvus)
* [Create Indexes in Milvus](/docs/5/fusion/reference/config-ref/jobs/create-indexes-in-milvus)
* [Delete Indexes in Milvus](/docs/5/fusion/reference/config-ref/jobs/delete-indexes-in-milvus)
## Pipeline stages
For more information about using these stages see **Configure the Smart Answers Pipelines (5.3 and later)**.
Before beginning this procedure, train a machine learning model using either the FAQ method or the cold start method.
Regardless of how you set up your model, the deployment procedure is the same:
1. [Create the `Milvus` collection](#create-the-milvus-collection).
2. [Configure the `smart-answers` index pipeline](#configure-the-index-pipeline).
3. [Configure the `smart-answers` query pipeline](#configure-the-query-pipeline).
See also [Best Practices](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/best-practices), [Advanced Model Training Configuration for Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/smart-answers-advanced-model-config), and Smart Answers Detailed Pipeline Setup.
## Create the Milvus collection
For complete details about job configuration options, see the [Create Collections in Milvus job](/docs/5/fusion/reference/config-ref/jobs/create-collections-in-milvus).
1. Navigate to **Collections** > **Jobs** > **Add +** and select **Create Collections in Milvus**.
2. Configure the job:
1. Enter an ID for this job.
2. Under **Collections**, click **Add**.
3. Enter a collection name.
4. In the **Dimension** field, enter the dimension size of vectors to store in this Milvus collection. The Dimension should match the size of the vectors returned by the encoding model. For example, the `Smart Answers Pre-trained Coldstart` models outputs vectors of 512 dimension size. Dimensionality of encoders trained by `Smart Answers Supervised Training` job depends on the provided parameters and printed in the training job logs.
3. Click **Save**.
The `Create Collections in Milvus` job can be used to create multiple collections at once. In this image, the first collection is used in the [indexing](#configure-the-index-pipeline) and [query](#configure-the-query-pipeline) steps. The other two collections are used in the [example](#pipeline-setup-example).
4. Click **Run** > **Start** to run the job.
## Configure the index pipeline
1. Open the Index Workbench.
2. Load or create your datasource using the default smart-answers index pipeline.
3. Configure the
[Encode into Milvus stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/encode-into-milvus-index-stage):
1. change the value of **Model ID** to match the model deployment name you chose when you configured the model training job.
2. Change `Field to Encode` to the document field name to be processed and encoded into dense vectors.
3. Ensure the `Encoder Output Vector` matches the output vector from the chosen model.
4. Ensure the `Milvus Collection Name` matches the collection name created via the `Create Milvus Collection` job.
To test out your settings, turn on `Fail on Error` in the `Encode into Milvus` stage and Apply the changes. This will cause an error message to display if any settings need to be changed.
4. Save the datasource.
5. Index your data.
## Configure the query pipeline
1. Open the Query Workbench.
2. Load the default smart-answers query pipeline.
3. Configure the [Milvus Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-query-stage):
1. Change the **Model ID** value to match the model deployment name you chose when you configured the model training job.
2. Ensure the `Encoder Output Vector` matches the output vector from the chosen model.
3. Ensure the `Milvus Collection Name` matches the collection name created via the `Create Milvus Collection` job.
4. `Milvus Results Context Key` can be changed as needed. It will be used in the Milvus Ensemble Query Stage to calculate the query score.
4. In the [Milvus Ensemble Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-ensemble-query-stage), update the `Ensemble math expression` as needed based on your model and the name used in the prior stage for the storing the Milvus results.
In versions 5.4 and later, you can also set the `Threshold` so that the Milvus Ensemble Query Stage will only return items with a score **greater than** or **equal to** the configured value.
5. Save the query pipeline.
## Pipeline Setup Example
### Index and retrieve the question and answer together
To show question and answer together in one document (that is, treat the question as the title and the answer as the description), you can index them together in the same document. You can still use the default `smart-answers` index and query pipelines with a few additional changes.
Prior to configuring the Smart Answers pipelines, use the `Create Milvus Collection` job to create two collections, `question_collection` and `answer_collection`, to store the encoded "questions" and the encoded "answers", respectively.
#### Index Pipeline
As shown in the pictures below, you will need two Encode into Milvus stages, named Encode Question and Encode Answer respectively.
**Encode Question (Encode Into Milvus) stage**
**Encode Answer (Encode Into Milvus) stage**
In the Encode Question stage, specify `Field to Encode` to be `title_t` and change the `Milvus Collection Name` to match the new Milvus collection, `question_collection`.
In the Encode Answer stage, specify `Field to Encode` to be `description_t` and change the `Milvus Collection Name` to match the new Milvus collection, `answer_collection`.
#### Query Pipeline
Since we have two dense vectors generated during indexing, at query time we need to compute both query to question distance and query to answer distance. This can be set up as the pictures shown below with two Milvus Query Stages, one for each of the two Milvus collections.
To store those two distances separately, the `Milvus Results Context Key` needs to be different in each of these two stages.
In the Query Questions stage, we set the `Milvus Results Context Key` to `milvus_questions` and the Milvus collection name to `question_collection`.
Query Questions (Milvus Query) stage:
In the Query Answers stage, we set the `Milvus Results Context Key` to `milvus_answers` and the Milvus collection name to `answer_collection`.
Query Answers (Milvus Query) stage:
Now we can ensemble them together with the Milvus Ensemble Query Stage with the `Ensemble math expression` combining the results from the two query stages. If we want the question scores and answer scores weighted equally, we would use: `0.5 * milvus_questions + 0.5 * milvus_answers`.
This is recommended especially when you have limited FAQ dataset and want to utilize both question and answer information.
**Milvus Ensemble Query stage**
## Evaluate the query pipeline
The [Evaluate QnA Pipeline job](/docs/5/fusion/reference/config-ref/jobs/smart-answers-evaluate-pipeline) evaluates the rankings of results from any Smart Answers pipeline and finds the best set of weights in the ensemble score.
## Detailed pipeline setup
Typically, you can use the [default pipelines](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/overview) included with Fusion AI. These pipelines now utilize Milvus to store encoded vectors and to calculate vector similarity. This topic provides information you can use to customize the Smart Answers pipelines.
| | | |
| - | - | - |
| **"smart-answers" index pipeline** | | [Encode into Milvus stage](#the-encode-into-milvus-index-stage) |
| **"smart-answers" query pipeline** | |
|
### Create the Milvus collection
Prior to indexing data, the [Create Collections in Milvus job](/docs/5/fusion/reference/config-ref/jobs/create-collections-in-milvus) can be used to create the Milvus collection(s) used by the Smart Answers pipelines (see [Milvus overview](/docs/5/fusion/intro/fusion-stack/milvus)).
* `Job ID`. A unique identifier for the job.
* `Collection Name`. A name for the Milvus collection you are creating. This name is used in both the Smart Answer Index and the Smart Answer Query pipelines.
* `Dimension`. The dimension size of the vectors to store in this Milvus collection. The Dimension should match the size of the vectors returned by the encryption model. For example, if the model was created with either the `Smart Answers Coldstart Training` job or the `Smart Answers Supervised Training` job with the Model Base `word_en_300d_2M`, then the dimension would be 300.
* `Index file size`. Files with more documents than this will cause Milvus to build an index on this collection.
* `Metric`. The type of metric used to calculate vector similarity scores. `Inner Product` is recommended. It produces values between 0 and 1, where a higher value means higher similarity.
### Index pipeline setup
**Stages in the default "smart-answers" index pipeline**
Only one custom index stage needs to be configured in your index pipeline, the [Encode into Milvus index stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/encode-into-milvus-index-stage).
#### The Encode into Milvus Index Stage
If you are using a dynamic schema, make sure this stage is added *after* the [Solr Dynamic Field Name Mapping stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/solr-dynamic-field-name-mapping-index-stage).
The [Encode into Milvus](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/encode-into-milvus-index-stage) index stage uses the specified model to encode the `Field to Encode` and store it in Milvus in the given Milvus collection.
There are several required parameters:
* `Model ID`. The ID of the model.
* `Encoder Output Vector`. The name of the field that stores the compressed dense vectors output from the model. Default value: `vector`.
* `Field to Encode`. The text field to encode into a dense vector, such as `answer_t` or `body_t`.
* `Milvus Collection Name`. The name of the collection you created via the [Create Milvus Collection](/docs/5/fusion/reference/config-ref/jobs/create-collections-in-milvus) job, which will store the dense vectors. When creating the collection you specify the type of **Metric** to use to calculate vector similarity.
This stage can be used multiple times to encode additional fields, each into a different Milvus collection.
### Query pipeline setup
#### The Query Fields stage
The first stage is **Query Fields**. For more information see the [Query Fields stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/search-fields-query-stage).
#### The Milvus Query stage
The [Milvus Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-query-stage) stage encodes the query into a vector using the specified model. It then performs a vector similarity search against the specified Milvus collection and returns a list of the best document matches.
* `Model ID`. The ID of the model used when configuring the model training job.
* `Encoder Output Vector`. The name of the output vector from the specified model, which will contain the query encoded as a vector. Defaults to vector.
* `Milvus Collection Name`. The name of the collection that you used in the `Encode into Milvus` index stage to store the encoded vectors.
* `Milvus Results Context Key`. The name of the variable used to store the vector distances. It can be changed as needed. It will be used in the Milvus Ensemble Query Stage to calculate the query score for the document.
* `Number of Results`. The number of highest scoring results returned from Milvus.
This stage would typically be used the same number of times that the `Encode into Milvus` index stage is used, each with a different Milvus collection and a different `Milvus Results Context Key`.
#### The Milvus Ensemble Query stage
The [Milvus Ensemble Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-ensemble-query-stage) takes the results of the Milvus Query stage(s) and calculates the `ensemble score`, which is used to return the best matches.
* `Ensemble math expression`. The mathematical expression used to calculate the `ensemble score`. It should reference the value(s) variable name specified in the `Milvus Results Context Key` parameter in the Milvus Query stage.
* `Result field name`. The name of the field used to store the `ensemble score`. It defaults to `ensemble_score`.
* `Threshold`- A parameter that filters the stage results to remove items that fall below the configured score. Items with a score at, or above, the threshold will be returned.
The **Threshold** feature is only available in Fusion 5.4 and later.
#### The Milvus Response Update Query stage
The [Milvus Response Update Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-response-update-stage) stage does not need to be configured and can be skipped if desired. It inserts the Milvus values, including the `ensemble_score`, into each of the returned documents, which is particularly useful when there is more than one `Milvus Query Stage`. This stage needs to come after the `Solr Query` stage.
## Short answer extraction
By default, the question-answering query pipelines return complete documents that answer questions. Optionally, you can extract just a paragraph, a sentence, or a few words that answer the question.
* The [Encode into Milvus](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/encode-into-milvus-index-stage) index pipeline stage invokes a machine learning model to encode a field and store it into Milvus.
* Query pipeline stages:
* [Milvus Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-query-stage) performs vectors similarity search in Milvus.
* [Milvus Ensemble Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-ensemble-query-stage) performs ensemble of different search results to retrieve metadata from Solr.
* [Milvus Response Update](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-response-update-stage) updates response documents with vectors similarity and ensemble scores.
## Troubleshooting
### Update a Milvus collection
**Update and delete semantics are not supported on individual items in a Milvus collection. We recommend that you use a blue/green set of collections: reindex to apply updates to an empty Milvus collection, then swap between collections to apply the updates.**
# The Fusion Stack
Source: https://doc.lucidworks.com/docs/5/fusion/intro/fusion-stack/overview
[localhost link]: http://localhost:3000/docs/5/fusion/intro/fusion-stack/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/intro/fusion-stack/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/3212
In addition to Lucidworks' proprietary technology, the Fusion platform incorporates a powerful stack of open source software.
* [Apache Solr](/docs/5/fusion/intro/fusion-stack/solr/overview) is the search engine at the core of Fusion. Solr stores your data in a primary collection for the data your users will search and auxiliary collections for signals, aggregations, and related data.
* [Apache Spark](/docs/5/fusion/intro/fusion-stack/spark/overview) is the cluster-computing framework for large-scale processing jobs. These jobs can be turned into stepwise tasks and distributed across a cluster of computers.
* [Kubernetes](/docs/5/fusion/intro/kubernetes) is used for Fusion as the deployment platform. Fusion runs as a set of microservices on Kubernetes distributions such as AKS, EKS, and GKE.
* [ZooKeeper](/docs/5/fusion/intro/fusion-stack/zookeeper) is a distributed configuration service, synchronization service, and naming registry. Fusion uses a ZooKeeper [microservice](/docs/5/fusion/reference/microservices) to configure and manage all Fusion components in a single Fusion deployment.
The course for **Fusion Architecture** focuses on the components that make Fusion a one-of-a-kind search solution.
# Seldon
Source: https://doc.lucidworks.com/docs/5/fusion/intro/fusion-stack/seldon
[localhost link]: http://localhost:3000/docs/5/fusion/intro/fusion-stack/seldon
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/intro/fusion-stack/seldon
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/buompm
Seldon is an open-source platform used to deploy, scale, and monitor machine learning (ML) models in Kubernetes environments. It is integrated into Fusion to enable model-serving capabilities for inference workflows. For more information, see [Seldon Core documentation](https://docs.seldon.io/projects/seldon-core) for more details.
## Seldon models and endpoints
Fusion Seldon models are trained on different lengths of text such as short and long sentences or phrases, as well as varied collections for B2B, B2C, and knowledge management entities.
Models are exposed via a unique endpoint, and are used in a pipeline stage or external application. Endpoints can be secured using Istio or other methods through the Fusion gateway.
Supported models include general purpose sentiment prediction, biomedical text collections, and large corpuses in a multiple languages. Based on the model, Seldon supports input formats such as `ndarray`, `tensor`, and `json`. Outputs are in a structured JSON format, typically wrapped in metadata, which supports monitoring and logging functions.
## Seldon configuration and management
This section provides links to more detailed information to configure and manage Seldon deployments for Fusion users.
* [Create Seldon Core Model deployment](/docs/5/fusion/reference/config-ref/jobs/create-seldon-core-model-deployment) in a Fusion cluster
* [Update Seldon Model deployment](#update-a-seldon-model-deployment)
* [Delete Seldon Core Model deployment](/docs/5/fusion/reference/config-ref/jobs/delete-seldon-core-model-deployment)
* **Configure Ray/Seldon vector search** in Neural Hybrid Search
* [Ray/Seldon Vectorize Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/vectorize-query-via-seldon-query-stage) to generate a vector based on a query string
* [Ray/Seldon Vectorize Field stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/vectorize-field-via-seldon-index-stage) to invoke a machine learning model that encodes a string field to a vector representation
You can use Seldon or Ray models to vectorize text for [Neural Hybrid Search](/docs/5/fusion/hybrid-search/overview).
This feature is available starting in Fusion 5.9.5 and in all subsequent Fusion 5.9 releases.
## Prerequisites
### Seldon
To vectorize text with Seldon, you first need to develop and deploy a machine learning model with Seldon.
### Ray
This feature is only available in Fusion 5.9.x for versions 5.9.12 and later.
To vectorize text with Ray, you first need to [develop and deploy a machine learning model with Ray](#develop-and-deploy-a-machine-learning-model-with-ray).
## Configure index pipeline
1. Sign into Fusion, go to **Indexing** > **Index Pipelines**, then select an existing pipeline or create a new one.
2. Click **Add a new pipeline stage**, then select **Ray/Seldon Vectorize Field**. In Fusion 5.9.11 and earlier, this stage is called **Seldon Vectorize Field**.
3. Fill in the required fields:
1. Enter a **Model ID**. This is the name of the model you developed and deployed.
2. Enter the **Model Input Field**. For example, `text`.
3. Enter the **Model Output Vector Field**. For example, `vector`.
4. Enter the **Source Field**. For example, `body_t`.
5. Enter the **Destination Field**. For example, `body_512_v`.
4. Click **Save**.
5. Make sure the **Seldon Vectorize Field** stage is ordered before the **Solr Indexer** stage.
Index data using the new pipeline. Confirm that your vector field is indexed by checking to see that the field is present in documents.
## Configure query pipeline
1. Go to **Querying** > **Query Pipelines**, then select an existing pipeline.
2. Click **Add a new pipeline stage**, then select **Ray/Seldon Vectorize Query**. In Fusion 5.9.11 and earlier, this stage is called **Seldon Vectorize Field**.
3. Fill in the required fields, making sure to search against the field into which you indexed the vectors:
1. Enter a **Model ID**. This is the name of the model you developed and deployed].
2. Enter the **Query Input**.
3. Enter the **Model Input Field**. For example, `text`.
4. Enter the **Model Output Vector Field**. For example, `vector`.
5. Enter the **Vector Context Key**. For example, `vector`.
4. Click **Save**.
5. Make sure the **Ray/Seldon Vectorize Query** stage is ordered before the **Solr Query** stage.
## Perform vector searches
After setting up the stages, you can perform vector searches via the [`knn` query parser](https://solr.apache.org/guide/solr/latest/query-guide/dense-vector-search.html#knn-query-parser) as you would with Solr. Specify the search vector and include it in the query. For example, change the `q` parameter to a `knn` query parser string.
The **Ray/Seldon Vectorize Query** stage will encode user queries using the specified model and modify the `q` parameter to use the `knn` query parser, turning the query into a vector search.
This tutorial walks you through deploying your own model to Fusion with Ray.
This feature is only available in Fusion 5.9.x for versions 5.9.12 and later.
## Prerequisites
* A Fusion instance with an app and indexed data.
* An understanding of Python and the ability to write Python code.
* [Docker](https://docs.docker.com/get-docker/) installed locally, plus a private or public Docker repository.
* Ray installed locally: `pip install ray[serve]` using the version of ray\[serve] found in the release notes for your version of Fusion.
* Code editor; you can use any editor, but Visual Studio Code is used in this example.
* Model: [intfloat/e5-small-v2](https://huggingface.co/intfloat/e5-small-v2)
* Docker image: [e5-small-v2-ray](https://hub.docker.com/r/jstrmec/e5-small-v2-ray/)
## Tips
* Always test your Python code locally before uploading to Docker and then Fusion.
This simplifies troubleshooting significantly.
* Once you’ve created your Docker you can also test locally by doing `docker run` with a specified port, like 9000, which you can then `curl` to confirm functionality in Fusion.
See the testing example below.
* If you previously deployed a model with Seldon, you can deploy the same model with Ray after making a few changes to your Docker image as explained in this topic.
To avoid conflicts, deploy the model with a different name.
When you have verified that the Ray model is working after deployment with Ray, you can delete the Seldon model using the [Delete Seldon Core Model Deployment job](/docs/5/fusion/reference/config-ref/jobs/delete-seldon-core-model-deployment).
* If you run into an issue with the model not deploying and you’re using the 'real' example, there is a very good chance you haven’t allocated enough memory or CPU in your job spec or in the Ray-Argo config.
It’s easy to increase the resources. To edit the ConfigMap, run `kubectl edit configmap argo-deploy-ray-model-workflow -n ` and then find the `ray-head` container in the artisanal escaped YAML and change the memory limit.
Exercise caution when editing because it can break the YAML.
Just delete and replace a single character at a time without changing any formatting.
* For additional guidance, see the [testing locally e5-model example](https://colab.research.google.com/drive/1ef-NTJBtP8p77aAa8Pp9SPXtQyozJEas?usp=sharing).
The course for **Intro to Machine Learning in Fusion** focuses on using machine learning to infer the goals of customers and users in order to deliver a more sophisticated search experience.
## Local testing example
1. Docker command:
```bash theme={"dark"}
docker run -p 127.0.0.1:9000:9000 DOCKER_IMAGE
```
2. Curl to hit Docker:
```bash theme={"dark"}
curl -i -X POST http://127.0.0.1:8000 -H 'Content-Type: application/json' -d '{"text": "The quick brown fox jumps over the lazy dog."}'
```
3. Curl model in Fusion:
```bash theme={"dark"}
curl -u $FUSION_USER:$FUSION_PASSWORD -X POST -H 'Content-Type: application/json' -d '{"text": "i love fusion"}' https://FUSION_HOST.com:6764/api/ai/ml-models/MODEL_NAME/prediction
```
4. See all your deployed models:
```bash theme={"dark"}
curl -u USERNAME:PASSWORD http://FUSION_HOST:FUSION_PORT/api/ai/ml-models
```
5. Check the Ray UI to see Replica State, Resources, and Logs.\
If you are getting an internal model error, the best way to see what is going on is to query via port-forwarding the model.\
The `MODEL_DEPLOYMENT` in the command below can be found with `kubectl get svc -n NAMESPACE`. It will have the same name as set in the model name in the **Create Ray Model Deployment** job.
```bash theme={"dark"}
kubectl -n NAMESPACE port-forward svc/MODEL_DEPLOYMENT-head-svc 8000:8000
```
Once port-forwarding is successful, you can use the below cURL command to see the issue.
At that point your worker logs should show helpful error messages.
```bash theme={"dark"}
curl --location 'http://127.0.0.1:8000/' \
--header 'charset: utf-8' \
--header 'Content-Type: application/json' \
--data '{"text": "i love fusion"}'
```
## Download the model
This tutorial uses the [`e5-small-v2`](https://huggingface.co/intfloat/e5-small-v2) model from Hugging Face, but any pre-trained model from [https://huggingface.co](https://huggingface.co) will work with this tutorial.
If you want to use your own model instead, you can do so, but your model must have been trained and then saved though a function similar to the PyTorch’s `torch.save(model, PATH)` function.
See [Saving and Loading Models](https://pytorch.org/tutorials/beginner/saving_loading_models.html) in the PyTorch documentation.
## Format a Python class
The next step is to format a Python class which will be invoked by Fusion to get the results from your model.
The skeleton below represents the format that you should follow.
See also [Getting Started](https://docs.ray.io/en/latest/serve/getting_started.html) in the Ray Serve documentation.
```python wrap expandable theme={"dark"}
from ray import serve
from starlette.requests import Request
# These defaults are for the ray serve deployment
# when running simply from docker. The 'Create Ray Model Deployment'
# job can override these replicas and resources if needed.
@serve.deployment(num_replicas=1, ray_actor_options={"num_cpus": 1})
class Deployment(object):
def __init__(self):
"""
Add any initialization parameters. Generally this is where you would load
your model. This method will be called once when the deployment is created.
"""
print("Initializing")
self.model = load_model() #faux code
# This can be named as any method which takes a dictionary as input and returns a dictionary
# as output. In this example, we are using the encode method to encode the
# input text into a vector.
def encode(self, input_dict: Dict[str, Any]) -> Dict[str, Any]:
"""
This method will be called when the deployment is queried. It will receive
the input data and should return the output data.
"""
text = input_dict["text"]
embeddings = self.model.encode #faux code
return { "vector": embeddings } # To use the 'Ray / Seldon Vectorize Field' stage, the output key should be `vector`, if using the 'Machine Learning' stage you must ensure the output key matches the output key in the 'Machine Learning' stage
async def __call__(self, http_request: Request) -> Dict[str, Any]:
input_dict: Dict[str, Any] = await http_request.json()
return self.encode(input_dict=input_dict) # This will be the function you defined above, in this case encode
app = Deployment.bind()
```
A real instance of this class with the `e5-small-v2` model is as follows:
This code pulls from Hugging Face. To have the model load in the image without pulling from Hugging Face or other external sources, download the model weights into a folder name and change the model name to the folder name preceded by `./`.
```python wrap expandable theme={"dark"}
import json
import sys
from time import time
from typing import Any, Dict
import torch
import torch.nn.functional as F
from ray import serve
from starlette.requests import Request
from starlette.responses import JSONResponse
from torch import Tensor
from transformers import AutoModel, AutoTokenizer
HUB_MODEL_NAME = "intfloat/e5-small-v2"
@serve.deployment(num_replicas=1, ray_actor_options={"num_cpus": 1})
class Deployment(object):
def __init__(self):
from loguru import logger
self.logger = logger
# Initializing logger
self.logger.remove()
self.logger.add(sys.stdout, level="INFO", serialize=False, colorize=True)
# Initializing model
self.logger.info("Loading model...")
self.tokenizer = AutoTokenizer.from_pretrained(HUB_MODEL_NAME)
self.model = AutoModel.from_pretrained(HUB_MODEL_NAME)
self.model.eval()
self.logger.info("Model initialization finished!")
def encode(self, input_dict: Dict[str, Any]) -> Dict[str, Any]:
_start_time = time()
# Extracting text from input
text = input_dict["text"]
# Tokenization
tokenized_texts = self.tokenizer(
text,
max_length=512,
padding=True,
truncation=True,
return_tensors="pt",
)
# Encoding
with torch.inference_mode():
# Forward pass of the model
outputs = self.model(**tokenized_texts)
# Average pooling the last hidden states
embeddings = self.average_pool(
outputs.last_hidden_state, tokenized_texts["attention_mask"]
)
# Normalizing embeddings
embeddings = F.normalize(embeddings, p=2, dim=1)
# Converting into output format
output_dict = {"vector": embeddings.squeeze().tolist()}
prediction_time = (time() - _start_time) * 1000
self.logger.info(f"Time taken to make a prediction: {prediction_time:.0f}ms")
return output_dict
async def __call__(self, http_request: Request) -> Dict[str, Any]:
try:
input_dict: Dict[str, Any] = await http_request.json()
except UnicodeDecodeError:
body_bytes = await http_request.body()
try:
decoded = body_bytes.decode("utf-8", errors="replace")
input_dict = json.loads(decoded)
except json.JSONDecodeError:
return JSONResponse({"error": "Invalid JSON"}, status_code=400)
return self.encode(input_dict=input_dict)
@staticmethod
def average_pool(last_hidden_states: Tensor, attention_mask: Tensor) -> Tensor:
last_hidden = last_hidden_states.masked_fill(
~attention_mask[..., None].bool(), 0.0
)
return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]
app = Deployment.bind()
```
In the preceding code, logging has been added for debugging purposes.
The preceding code example contains the following functions:
* `__call__`: This function is non-negotiable.
* `init`: The `init` function is where models, tokenizers, vectorizers, and the like should be set to self for invoking.
It is recommended that you include your model’s trained parameters directly into the Docker container rather than reaching out to external storage inside `init`.
* `encode`: The `encode` function is where the field or query that is passed to the model from Fusion is processed.
Alternatively, you can process it all in the `__call__` function, but it is cleaner not to.
The `encode` function can handle any text processing needed for the model to accept input invoked in its `model.predict()` or equivalent function which gets the expected model result.
If the output needs additional manipulation, that should be done before the result is returned.
For embedding models, the return value must have the shape of (1, DIM), where DIM (vector dimension) is a consistent integer, to enable Fusion to handle the vector encoding into Ray.
Use the *exact* name of the class when naming this file.
In the preceding example, the Python file is named `deployment.py` and the class name is `Deployment()`.
## Create a Dockerfile
The next step is to create a Dockerfile. The Dockerfile should follow this general outline; read the comments for additional details:
```dockerfile theme={"dark"}
#It is important that python version is 3.x-slim
FROM python:3.10-slim
# Install dependencies
RUN apt-get update && apt-get install -y wget
# Create working app directory
RUN mkdir -p /app
WORKDIR /app
# Copy the requirements file and install the dependencies
COPY requirements.txt /app
RUN pip install -r requirements.txt --no-cache-dir
# Copy source code
COPY deployment.py /app
# Expose serving port for HTTP communication with Fusion
EXPOSE 8000
# The end of the command follows module:application and the below value should be set in the RAY DEPLOYMENT IMPORT PATH field in 'Create Ray Model Deployment' job
CMD exec serve run deployment:app
```
## Create a requirements file
The `requirements.txt` file is a list of installs for the `Dockerfile` to run to ensure the Docker container has the right resources to run the model.
For the `e5-small-v2` model, the requirements are as follows:
```text theme={"dark"}
torch -f https://download.pytorch.org/whl/torch_stable.html # Make sure that we download CPU version of PyTorch
transformers
loguru
ray[serve]==2.42.1
```
Any recent ray\[serve] version should work, but the tested value and known supported version is 2.42.1.
In general, if an item was used in an `import` statement in your Python file, it should be included in the requirements file.
To populate the requirements, use the following command in the terminal, inside the directory that contains your code:
```bash theme={"dark"}
pip freeze > requirements.txt
```
## Build and push the Docker image
After creating the `MODEL_NAME.py`, `Dockerfile`, and `requirements.txt` files, you need to run a few Docker commands.
Run the following commands in order:
```bash theme={"dark"}
DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build . -t [DOCKERHUB-USERNAME]/[REPOSITORY]:[VERSION-TAG]
```
```bash theme={"dark"}
docker push [DOCKERHUB USERNAME]/[REPOSITORY]:[VERSION-TAG]
```
Using the example model, the terminal commands would be as follows:
```bash theme={"dark"}
DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build . -t jstrmec/e5-small-v2-ray:0.1
```
```bash theme={"dark"}
docker push jstrmec/e5-small-v2-ray:0.1
```
This repository is public and you can visit it here: [e5-small-v2-ray](https://hub.docker.com/r/jstrmec/e5-small-v2-ray/)
## Deploy the model in Fusion
Now you can go to Fusion to deploy your model.
When deploying your Ray model, you have two options for handling traffic:
* Use a single deployment. Deploy one model job that handles both indexing and query traffic. This is simpler to manage and requires only one deployment.
* Use separate deployments for indexing and querying. Deploy two separate model jobs: one dedicated to indexing and another for query traffic. This approach eliminates the risk of indexing workloads impacting query response times, providing better performance isolation and independent scaling control.
Choose separate deployments if query performance is critical and you want to ensure indexing operations don't compete with user-facing queries for resources.
To use separate deployments, follow the deployment steps to create two Ray model deployment jobs with different job IDs (for example, `EXAMPLE_MODEL_INDEX` and `EXAMPLE_MODEL_QUERY`). Use the index-specific model in your index pipeline stages and the query-specific model in your query pipeline stages. To keep both deployments in sync, ensure both jobs use the exact same model name, Ray deployment import path, Docker repository, and image name.
1. In Fusion, navigate to **Collections** > **Jobs**.
2. Add a job by clicking the **Add+** Button and selecting **Create Ray Model Deployment**.
3. Fill in each of the text fields:
| Parameter | Description |
| - | - |
| Job ID | A string used by the Fusion API to reference the job after its creation. |
| Model name | A name for the deployed model. This is used to generate the deployment name in Ray. It is also the name that you reference as a `model-id` when making predictions with the ML Service. |
| Model min replicas | The minimum number of load-balanced replicas of the model to deploy. |
| Model max replicas | The maximum number of load-balanced replicas of the model to deploy. Specify multiple replicas for a higher-volume intake. |
| Model CPU limit | The number of CPUs to allocate to a single model replica. |
| Model memory limit | The maximum amount of memory to allocate to a single model replica. |
| Ray Deployment Import Path | The path to your top-level Ray Serve deployment (or the same path passed to `serve run`). For example, `deployment:app` |
| Docker Repository | The public or private repository where the Docker image is located. If you’re using Docker Hub, fill in the Docker Hub username here. |
| Image name | The name of the image. For example, `e5-small-v2-ray:0.1`. |
| Kubernetes secret | If you’re using a private repository, supply the name of the Kubernetes secret used for access. |
4. Click **Advanced** to view and configure advanced details:
| | |
| - | - |
| Parameter | Description |
| Additional parameters. | This section lets you enter parameter `name:parameter` value options to be injected into the training JSON map at runtime. The values are inserted as they are entered, so you must surround string values with `"`. This is the sparkConfig field in the configuration file. |
| Write Options. | This section lets you enter parameter `name:parameter` value options to use when writing output to Solr or other sources. This is the writeOptions field in the configuration file. |
| Read Options. | This section lets you enter parameter `name:parameter` value options to use when reading input from Solr or other sources. This is the readOptions field in the configuration file. |
5. Click **Save**, then **Run** and **Start**.
When the job finishes successfully, you can proceed to the next section.
Now that the model is in Fusion, you can use it in the Machine Learning or Ray / Seldon Vectorize index and query stages.
## Configure the Fusion pipelines
Your real-world pipeline configuration depends on your use case and model, but for our example we will configure the index pipeline and then the query pipeline.
**Configure the index pipeline**
1. Create a new index pipeline or load an existing one for editing.
2. Click **Add a Stage** and then **Machine Learning**.
3. In the new stage, fill in these fields:
* The model ID
* The model input
* The model output
4. Save the stage in the pipeline and index your data with it.
**Configure the query pipeline**
1. Create a new query pipeline or load an existing one for editing.
2. Click **Add a Stage** and then **Machine Learning**
3. In the new stage, fill in these fields:
* The model ID
* The model input
* The model output
4. Save the stage and then run a query by typing a search term.
5. To verify the Ray results are correct, use the **Compare+** button to see another pipeline without the model implementation and compare the number of results.
You have now successfully uploaded a Ray model to Fusion and deployed it.
### Update a Seldon model deployment
Seldon does not support in-place updates to certain deployment properties, for example, protocol or runtime class. To apply significant updates:
1. Deploy a new version of the model using a different deployment name.
2. Execute test requests to validate the new endpoint.
3. Update the Fusion pipeline with the new endpoint information.
4. Delete the previous Seldon deployment when the migration is complete.
# Configset templates
Source: https://doc.lucidworks.com/docs/5/fusion/intro/fusion-stack/solr/configset-templates
Fusion uses Solr configset templates to create collections with predefined schemas and configurations optimized for different purposes.
When you create a new collection in Fusion, the system clones one of these templates and customizes it for your specific collection.
This ensures each collection has its own independent configuration that you can modify without affecting other collections.
## Template naming convention
All Fusion configset templates follow a naming convention that begins with `_fusion_`:
* `_fusion_default` - Primary application collections
* `_fusion_signals` - Signals collections
* `_fusion_query_rewrite` - Query rewrite collections
* Additional templates for specific Fusion features
Fusion prevents you from creating collections with names that begin with `_fusion_` to avoid conflicts with system templates.
## Template ownership and versioning
Different Fusion deployment types may use different implementations of the same template.
For example, the `_fusion_default` template in Lucidworks Search (managed Fusion) differs from the template in self-hosted Fusion deployments.
Each configset template includes a special `_fusion_owner.txt` file that identifies which Fusion component manages that template.
This file contains a single line identifying the owner, such as `fusion-solr` or `fusion-solr-managed`.
### Purpose of `_fusion_owner.txt`
The ownership file serves two critical functions:
1. **Tracks template source**: Identifies which docker image provided the configset template.
2. **Protects customizations**: Prevents Fusion upgrades from overwriting customized configsets.
## How templates work during installation and upgrades
When you install or upgrade Fusion, a bootstrap process copies configset templates from docker images into ZooKeeper, where Solr can access them.
The bootstrap script (`/opt/lucidworks/sync-configsets-with-zk.sh`) follows these rules:
1. Connects to ZooKeeper and creates a lock to prevent concurrent modifications.
2. Loops through all configset templates in the docker image.
3. For each template:
* If no configset exists in ZooKeeper with this name, the template is copied.
* If a configset already exists in ZooKeeper:
* **Matching ownership**: If the `_fusion_owner.txt` file matches, the template replaces the existing configset.
* **Mismatched ownership**: If the `_fusion_owner.txt` file doesn't match or is missing, the template is skipped and a warning is logged.
This ownership check ensures that if you've customized a configset template and removed or modified its `_fusion_owner.txt` file, Fusion upgrades won't overwrite your changes.
## Customizing configset templates
If you need to customize a Fusion configset template for your self-hosted deployment:
1. Make your modifications to the configset in ZooKeeper.
2. Either:
* **Option A**: Remove the `_fusion_owner.txt` file to prevent any future upgrades from replacing your customizations.
* **Option B**: Modify the `_fusion_owner.txt` file to contain a different value (not matching the Fusion docker image owner).
If you customize a template and preserve the original `_fusion_owner.txt` file unchanged, Fusion upgrades will replace your customizations with the new template version.
## Viewing template ownership
To check which component owns a configset template in your deployment:
1. Connect to ZooKeeper.
2. Navigate to `/configs//`.
3. Read the contents of the `_fusion_owner.txt` file.
The file contains a single line identifying the owner, which corresponds to the Fusion docker image that manages this template.
## Troubleshooting
### Warning: Configset skipped during upgrade
If you see warnings in the Fusion logs about skipped configsets during an upgrade, this indicates:
* A configset exists in ZooKeeper with a template name (starting with `_fusion_`).
* The `_fusion_owner.txt` file is either missing or contains a value that doesn't match the docker image.
This is normal behavior when you've customized templates.
Fusion preserves your customizations rather than overwriting them.
If you want Fusion to resume managing a customized template, either restore the original `_fusion_owner.txt` file or delete the configset from ZooKeeper and let the next upgrade recreate it.
### Dry run mode
To preview what the bootstrap script would do without making changes, you can run it in dry run mode by setting the `DRY_RUN` environment variable to `true`.
The script will log all actions it would take without modifying ZooKeeper.
# Apache Solr
Source: https://doc.lucidworks.com/docs/5/fusion/intro/fusion-stack/solr/overview
[localhost link]: http://localhost:3000/docs/5/fusion/intro/fusion-stack/solr/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/intro/fusion-stack/solr/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/3214
Solr is the open source search engine at the core of Fusion. When you [index your data](/docs/5/fusion/getting-data-in/indexing/overview), it is stored in a Solr collection.
## Collections
Your data is organized into collections. When you create an app, Fusion automatically creates a collection with the same name. You can create additional collections in any app.
A primary collection contains the data that your users will search. Every primary collection is associated with a set of auxiliary collections that contain related data, such as signals, aggregations, and more.
Under the hood, a Fusion collection is a distributed index in Solr, defined by a named configuration stored in ZooKeeper, with these properties:
* Number of shards\
Documents are distributed across this number of partitions.
* Document routing strategy\
How documents are assigned to shards.
* Replication factor\
How many copies of each document in the collection.
* Replica placement strategy\
Where to place replicas in the cluster.
If your data is already stored in a Solr instance or cluster, you can manage this collection
in Fusion by creating a Fusion collection that imports the existing Solr collection.
Collection names are case-insensitive, but Fusion preserves case when displaying collection names.
### Auxiliary Collections
Every primary collection is associated with a set of auxiliary collections that contain related data, such as signals, aggregations, and more.
Some auxiliary collections are created for every primary collection. Others are created only for the app’s default collection, one per app.
Auxiliary collections are described below:
| | | |
| - | - | - |
| `APP_NAME_job_reports` | Output from Fusion [experiments](/docs/5/fusion/getting-data-out/data-analytics/experiments/overview), [Ranking Metrics jobs](/docs/5/fusion/reference/config-ref/jobs/ranking-metrics), and [Head/Tail Analysis jobs](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis). | 1 per app |
| `APP_NAME_query_rewrite` |
A collection of documents to use for [rewriting queries](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting), optimized for high‑volume traffic. These documents originate from the `COLLECTION_NAME_query_rewrite_staging` collection. Certain Fusion query pipeline stages read from this collection: • [Text Tagger](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/text-tagger-query-stage) • [Apply Rules](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/query-rules-query-stage) • [Modify Response with Rules](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/rules-augment-response-query-stage)
| 1 per app |
| `APP_NAME_query_rewrite_staging` |
A collection of documents created by the Rules Editor or by certain [Fusion jobs](/docs/5/fusion/reference/config-ref/jobs/overview), not optimized for production traffic. Documents move from this collection to the `COLLECTION_NAME_query_rewrite` collection as follows: • Job output documents with high confidence contain a `review=auto` field and are moved to the `COLLECTION_NAME_query_rewrite` collection automatically. • Job output documents with low confidence contain a `review=pending` field. When these are approved by a Fusion user, Fusion copies them to the `COLLECTION_NAME_query_rewrite` collection.
| 1 per app |
| `COLLECTION_NAME_signals` | A search query logs and signals collection. | 1 per collection |
| `COLLECTION_NAME_signals_aggr` | A collection for aggregated signals. | 1 per collection |
| `APP_NAME_user_prefs` | A collection of data to support App Studio’s social features, such as user‑generated tags, bookmarks, comments, ratings, and so on. | 1 per app |
Do not create primary collections with names that end in the suffixes above; these are reserved for Fusion auxiliary collections, which are created and managed by Fusion directly.
Fusion maintains a set of Solr collections that store Fusion’s own
log files and other internal information.
These are called [System Collections](#system-collections), described below.
Do not create primary collections named "logs" or beginning with "system\_".
These names are reserved for Fusion system collections.
Fusion uses ZooKeeper to register information about all collections,
and the Fusion components and services related to a collection.
The Fusion components associated with a collection include:
* Datasources
* Pipelines
* Profiles
* Signals and aggregations
* Analytics dashboards
### System Collections
Fusion automatically creates some collections that are used for internal purposes and shared across all apps:
* **system\_autocomplete** store the content that the Fusion UI displays when you use the search bar.
* **system\_blobs** stores [blobs](/docs/5/fusion/getting-data-in/blob-storage) in Solr. This is used to store model files for the NLP components and other binary files used by Fusion components.
* **system\_history** keeps a record of configuration changes, start and stop times for services and experiments, and more.
* **system\_jobs\_history** keeps a record of Fusion [jobs](/docs/5/fusion/operations/jobs-and-scheduling/overview), including start/stop times and status.
* **system\_messages** is used by Fusion’s messaging services.
### Collection Configuration Properties
Collections have three properties that you can configure only when you are creating a collection using the
[Collections API](/api-reference/collections/get-collections-service-status).
| Property | Description | Default behavior |
| - | - | - |
| signals\* | The `signals` property determines whether to create auxiliary collections with suffixes `_signals` and `_signals_aggr`. | When you create a collection in the Fusion UI, `signals` defaults to **true**. When you create a collection using the Fusion API, this property defaults to **false**. |
| searchLogs | The `searchLogs` property determines whether to create an auxiliary search query logs collection with suffix `_logs`. | When you create a collection in the Fusion UI, this property defaults to **true**. When you create a collection using the Fusion API, this property defaults to **false**. |
\*Signals are events with timestamps that can be used to improve search results.
For more information about signals in Fusion, see [Signals](/docs/5/fusion/getting-data-out/query-enhancement/signals/overview) in the Fusion documentation.
\*\*In schemaless mode, if a document contains a field not currently in the Solr schema, Solr processes the field value to determine what the field type should be defined as, and then adds a new field to the schema with the field name and field type.
This behavior can be convenient during preliminary application development, but it is rarely appropriate in a production environment.
### Using profiles to associate collections with pipelines
Index pipelines and query pipelines are not connected to a specific collection by default. Index profiles and query profiles are configurations that create consistent endpoints for indexing and querying, each with a specific pipeline and collection.
* [Index Profiles](/docs/5/fusion/getting-data-in/indexing/index-pipelines/index-profiles) work with index pipelines for getting content into the system.
* [Query Profiles](/docs/5/fusion/getting-data-out/query-basics/query-pipelines/query-profiles) work with query pipelines for user queries.
### Field Editor UI
The Fusion UI includes a space under Collections to edit Fields. The Fusion UI includes a space under Collections to edit Fields. Descriptions for these fields can be found in the Field Type Definitions section of the [Solr Reference Guide](/docs/5/fusion/reference/solr-reference-guide) associated with your Fusion release.
Field options displayed in the UI include:
* **Dynamic** checkbox (cannot change via UI)
* **Field Name** (cannot change via UI)
* **Field Type** (a preset value is shown that can be changed using edit mode)
* Checkboxes for **Indexed**, **Stored**, **Multivalued**, **Required**
* Text field to enter a **Default Value**
* **Copy Fields** uses the plus sign to add rows (**static** can copy to `raw_content` or `text`; **dynamic** can copy to any `raw_content`/`text` or any other dynamic field)
* **Advanced** toggles checkboxes for **Doc Values**, **Omit Norms**, **Omit Positions**, **Omit term freq and positions**, **Term Vectors**, **Term Positions**, **Term Offsets**
## Automated Solr backups
You can schedule backups of Solr collections, store the backups for a configurable period of time, and restore these backups into a specified Fusion cluster when needed.
The following guide uses Google Kubernetes Engine (GKE) for examples and assumes you used the setup scripts in the [fusion-cloud-native](https://github.com/lucidworks/fusion-cloud-native) repository to install Fusion.
The standard approach of using a provider-specific Persistent Volume Claim (PVC) for storing collection backups ensures consistency in configuration. However, this method does not leverage the unique storage features offered by each cloud provider. For instance, Google Cloud provides Google Cloud Storage, which includes additional features such as access control management, various storage tiers, and other capabilities that are not available when using a PVC. To take advantage of these features, Solr instances running within Fusion require additional provider-specific information.
Refer to the Solr documentation for detailed information on the [repositories available for configuring collection backups](https://solr.apache.org/guide/solr/latest/deployment-guide/backup-restore.html#backuprestore-storage-repositories). Each repository type comes with specific configuration options and features. Generally, you will need to integrate the provider-specific configuration into the `solr.xml` configuration file and ensure that the appropriate library or module for the provider is included in the Solr classpath. This step is necessary for the repository implementation to resolve correctly at runtime.
For example, when configuring `GCSBackupRepository` to store backups in Google Cloud Storage (GCS), it is essential to include the corresponding library for the provider in the Solr classpath. Additionally, you will need to add a section to the `solr.xml` file similar to the XML example below to specify the target bucket where backups will be stored:
```xml wrap theme={"dark"}
solrBackups/local/path/to/credential/file/default/gcs/backup/location515001.510000
```
After configuring the backup provider, you can utilize the standard [Solr backup and restore APIs](https://solr.apache.org/guide/solr/latest/deployment-guide/backup-restore.html#user-managed-clusters-and-single-node-installations) to create new backups or restore from existing ones. Instead of writing to a PVC, backups will be stored in the storage solution specific to the provider.
Backups are taken using the Solr collection [BACKUP command](https://solr.apache.org/guide/solr/latest/deployment-guide/collection-management.html#backup). This requires that each Solr node has access to a shared volume or a `ReadWriteMany` volume in Kubernetes. Most cloud providers offer a simple way of creating a shared filestore and exposing it as a `PersistentVolumeClaim` within Kubernetes to mount into the Solr pods. An option is added to the **setup\_f5\_PROVIDER.sh** scripts in the **fusion-cloud-native** repository to provision these.
The backup action of the script is invoked by a Kubernetes CronJob to run the backup schedule. The backups are saved to a configurable directory with an automatically generated name: `-`.
A separate CronJob is responsible for cleanup and retention of backups. Cleanup can be disabled if not needed. Setting a series of retention periods can automatically remove backups as they become outdated.
For example, a cluster that backs up a collection every 3 hours could specify a retention policy that:
* Keeps all backups for a single day.
* Keeps a single daily backup for a week.
* Keeps a single weekly backup for a month.
* Keeps a single monthly backup for 6 months.
* Deletes all backups that are older than 6 months.
All times are configurable as part of the `configmap` for this service.
The process for restoring a collection is a manual step involving `kubectl run` to invoke the Solr `RESTORE` action pointing to the collection and the name of the backup being restored. Use the following command, replacing the placeholder values with your environment-specific values:
```bash wrap theme={"dark"}
kubectl run --namespace "${NAMESPACE}" --rm=true -it \
--image=curlimages/curl:latest solr-restore -- \
curl "http://${RELEASE}-solr-headless:8983/solr/admin/collections?action=RESTORE&name=&location=/var/solr/data/solr-backups&collection="
```
Replace the following placeholder values:
* `${NAMESPACE}` with your Kubernetes namespace (for example, `default`)
* `${RELEASE}` with your Helm release name (for example, `f5`)
* `` with the name of the backup directory (for example, `mycollection-2024-01-15T00:00:00`)
* `` with the name of the collection to restore into
The backup directory name is generated automatically at backup time in the format `-`. You can list available backups by running:
```bash wrap theme={"dark"}
kubectl exec --namespace "${NAMESPACE}" "${RELEASE}-solr-0" -- \
ls /var/solr/data/solr-backups
```
These instructions are for GKE only. For other platforms, backup and restoration involves copying the collection to the cloud and using Parallel Bulk Loader.
### Install using a PVC with GKE
The `solr-backup-runner` requires that a `ReadWriteMany` volume is mounted onto all `solr` and `backup-runner` pods so they all back up to the same filesystem.
The easiest way to install on GKE is by using a GCP Filestore as the `ReadWriteMany` volume.
1. Create the Filestore.
```bash wrap theme={"dark"}
gcloud --project "${GCLOUD_PROJECT}" filestore instances create "${NFS_NAME}" --tier=STANDARD --file-share=name="solrbackups,capacity=${SOLR_BACKUP_NFS_GB}GB" --zone="${GCLOUD_ZONE}" --network=name="${network_name}"
```
2. Fetch the IP of the Filestore.
```bash wrap theme={"dark"}
NFS_IP="$(gcloud filestore instances describe "${NFS_NAME}" --project="${GCLOUD_PROJECT}" --zone="${GCLOUD_ZONE}" --format="value(networks.ipAddresses[0])")"
```
3. Create a Persistent Volume in Kubernetes that is backed by this volume.
```yaml theme={"dark"}
cat <
# Spark Operations
Source: https://doc.lucidworks.com/docs/5/fusion/intro/fusion-stack/spark/overview
[localhost link]: http://localhost:3000/docs/5/fusion/intro/fusion-stack/spark/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/intro/fusion-stack/spark/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/189
[Apache Spark](http://spark.apache.org/) is an open source cluster-computing framework that serves as a fast and general execution engine for large-scale data processing jobs that can be decomposed into stepwise tasks, which are distributed across a cluster of networked computers.
Spark improves on previous MapReduce implementations by using resilient distributed datasets (RDDs), a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner.
These topics explain Spark administration concepts in Fusion 5:
* [Spark Job Drivers](/docs/5/fusion/intro/fusion-stack/spark/spark-job-drivers)
* [Spark Administration in Kubernetes](/docs/5/fusion/operations/survival-guide/spark-kubernetes-overview)
Spark operations include:
1. Audit all [Spark jobs](/docs/5/fusion/operations/jobs-and-scheduling/spark-jobs) for natural key support.
2. Audit [SQL Aggregation](/docs/5/fusion/reference/config-ref/jobs/aggregations/sql-aggregation) jobs for natural key usage.
3. When looking at SQL for the [BPR Recommender job](/docs/5/fusion/reference/config-ref/jobs/bpr-recommender), audit generated aggregation SQL to ensure that it’s using a natural key projection.
4. Support partitioning in all Spark jobs in accordance with config options.
5. Support external data sources for all jobs (Spark, NLP, Clustering, Recommender), including external Spark source support for NLP.
# Spark Job Drivers
Source: https://doc.lucidworks.com/docs/5/fusion/intro/fusion-stack/spark/spark-job-drivers
[localhost link]: http://localhost:3000/docs/5/fusion/intro/fusion-stack/spark/spark-job-drivers
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/intro/fusion-stack/spark/spark-job-drivers
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/188
A Spark "driver" is an application that creates a SparkContext for executing one or more jobs in the Spark cluster. The following diagram depicts the driver’s role in a Spark cluster:
In the diagram above, the `spark-master` service in Fusion is the Cluster Manager.
If your Spark job performs any collect operations, then the result of the collect (or collectAsMap) is sent back to the driver from all the executors. Consequently, if the result of the collect is too big too fit into memory, you will encounter OOM issues (or other memory related problems) when running your job.
All Fusion jobs run on Spark using a driver process started by the API service.
## Custom jobs
Fusion supports custom Spark jobs that are written in Scala, Java, or Python jobs and that are built using the [Spark Job Workbench](https://github.com/Lucidworks/fusion-spark-job-workbench), a toolkit provided by Lucidworks. See the examples in the repository for details.
## Drivers
Fusion has four types of job drivers:
* **Default driver.** Executes built-in Fusion jobs, such as a signal aggregation job or a metrics rollup job.
* **Script-job driver.** Executes custom script jobs; a separate driver is needed to isolate the classpath for custom Scala scripts.
* **Spark-shell driver.** Wrapper script provided with Fusion to launch the Spark Scala REPL shell with the correct master URL (pulled from Fusion’s API) and a shaded Fusion JAR added. Launched using `{fusion-parent-unix}/fusion/5.9{sub-version}/bin/spark-shell` (on Unix) or `{fusion-parent-windows}\var\fusion\5.9{sub-version}\bin\spark-shell.cmd` (on Windows).
* **Custom-job driver.** Executes custom jobs built using the [Spark Job Workbench](https://github.com/Lucidworks/fusion-spark-job-workbench), a toolkit provided by Lucidworks to help build custom Spark jobs using Scala, Java, or Python.
# ZooKeeper
Source: https://doc.lucidworks.com/docs/5/fusion/intro/fusion-stack/zookeeper
[localhost link]: http://localhost:3000/docs/5/fusion/intro/fusion-stack/zookeeper
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/intro/fusion-stack/zookeeper
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/3213
[Apache ZooKeeper](http://zookeeper.apache.org/) is a distributed configuration service, synchronization service, and naming registry.
Fusion uses a ZooKeeper [microservice](/docs/5/fusion/reference/microservices) to configure and manage all Fusion components in a single Fusion deployment.
## ZooKeeper Terms
At a high level, ZooKeeper services consist of the following components:
* **znode.** ZooKeeper data is organized into a hierarchal name space of data nodes called znodes.\
A znode can have data associated with it as well as child znodes.\
The data in a znode is stored in a binary format, but it is possible to import, export, and view this information as JSON data.\
Paths to znodes are always expressed as canonical, absolute, slash-separated paths; there are no relative reference.
* **ephemeral nodes.** An ephemeral node is a znode which exists only for the duration of an active session.\
When the session ends the znode is deleted. An ephemeral znode cannot have children.
* **server.** A ZooKeeper service consists of one or more machines; each machine is a server which runs in its own JVM and listens on its own set of ports.\
For testing, you can run several ZooKeeper servers at once on a single workstation by configuring the ports for each server.
* **quorum.** A quorum is a set of ZooKeeper servers. It must be an odd number. For most deployments, only 3 servers are required.
* **client.** A client is any host or process which uses a ZooKeeper service.
The Fusion UI and API handle changes to ZooKeeper’s configuration. See [the ZooKeeper Import/Export API](/api-reference/zookeeper/get-zookeeper-data).
See the official [ZooKeeper documentation](http://zookeeper.apache.org/doc/current/) for details about using and managing a ZooKeeper service.
# Indexing Data Flow
Source: https://doc.lucidworks.com/docs/5/fusion/intro/indexing-dataflow
[localhost link]: http://localhost:3000/docs/5/fusion/intro/indexing-dataflow
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/intro/indexing-dataflow
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/3211
With Fusion, your data is indexed in a set of collections in Fusion’s Solr core.
A *primary collection* holds your searchable content, such as your product catalog, knowledge base, blog articles, product reviews, and so on.
A set of *secondary collections* are associated with your primary collection to hold related data that Fusion can use to enhance the relevancy of your search results.
This topic shows you how different types of data flow through Fusion to be indexed in your primary and secondary collections.
## 1. Index your content
No matter where your content is located, Fusion can index it. The Fusion collection where your searchable content is indexed is called the primary collection. Secondary collections are automatically created to hold related data, such as signals, machine learning job output, and so on.
There are a few ways to index your searchable content to the primary collection:
* [Connectors](/docs/fusion-connectors/overview)\
Lucidworks has a wide variety of connectors for many types of data sources. [Find your connector](/docs/fusion-connectors/overview).\
Once the connector fetches your data, [parsers](/docs/5/fusion/reference/config-ref/parser-stages/overview) read it before passing it to the [index pipeline](/docs/5/fusion/getting-data-in/indexing/index-pipelines/overview).
* The gRpc remote framework is configured on the client side and only works with V2 type connectors.
* V1 classic connectors support ingesting data using IP white lists, VPN tunnels, and public channels.
* The [Parallel Bulk Loader (PBL)](/docs/5/fusion/getting-data-in/parallel-bulk-loader)\
The PBL can send your data to an index pipeline *or* directly to the primary collection, depending on whether the data requires transformation before indexing. It does not support parsers and is not recommended for production environments.
* Import Data with the REST API\
Send your content to an index profile using the Fusion REST API. Index profiles are saved configurations of parsers and index pipelines.
It is often possible to get documents into Fusion by configuring a datasource with [the appropriate connector](/docs/5/fusion/getting-data-in/indexing/connectors).
But if there are obstacles to using connectors, it can be simpler to index documents with a REST API call to an index profile or pipeline.
## Push documents to Fusion using index profiles
[Index profiles](/docs/5/fusion/getting-data-in/indexing/index-pipelines/index-profiles) allow you to send documents to a consistent endpoint (the profile alias) and change the backend index pipeline as needed. The profile is also a simple way to use one pipeline for multiple collections without any one collection "owning" the pipeline.
You can send documents directly to an index using the [Index Profiles REST API](/docs/5/fusion/getting-data-in/indexing/index-pipelines/index-profiles). The request path is:
```bash theme={"dark"}
/api/apps/APP_NAME/index/INDEX_PROFILE
```
These requests are sent as a POST request. The request header specifies the format of the contents of the request body. Create an index profile in the Fusion UI.
To send a streaming list of JSON documents, you can send the JSON file that holds these objects to the API listed above with `application/json` as the content type. If your JSON file is a list or array of many items, the endpoint operates in a streaming way and indexes the docs as necessary.
### Send data to an index profile that is part of an app
Accessing an index profile through an app lets a Fusion admin secure and manage all objects on a per-app basis. Security is then determined by whether a user can access an app. This is the recommended way to manage permissions in Fusion.
The syntax for sending documents to an index profile that is part of an app is as follows:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/json' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE --data-binary @my-json-data.json
```
Spaces in an app name become underscores. Spaces in an index profile name become hyphens.
To prevent the terminal from displaying all the data and metadata it indexes--useful if you are indexing a large file--you can optionally append `?echo=false` to the URL.
Be sure to set the content type header properly for the content being sent. Some frequently used content types are:
* Text: `application/json`, `application/xml`
* PDF documents: `application/pdf`
* MS Office:
* DOCX: `application/vnd.openxmlformats-officedocument.wordprocessingml.document`
* XLSX: `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet`
* PPTX: `application/vnd.vnd.openxmlformats-officedocument.presentationml.presentation`
* More types: [http://filext.com/faq/office\_mime\_types.php](http://filext.com/faq/office_mime_types.php)
### Example: Send JSON data to an index profile under an app
In `$FUSION_HOME/apps/solr-dist/example/exampledocs` you can find a few sample documents. This example uses one of these, `books.json`.
To push JSON data to an index profile under an app:
1. Create an index profile. In the Fusion UI, click **Indexing > Index Profiles** and follow the prompts.
2. From the directory containing `books.json`, enter the following, substituting your values for username, password, and index profile name:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/json' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE?echo=false --data-binary @books.json
```
3. Test that your data has made it into Fusion:
1. Log into the Fusion UI.
2. Navigate to the app where you sent your data.
3. Navigate to the Query Workbench.
4. Search for `*:*`.
5. Select relevant Display Fields, for example `author` and `name`.
### Example: Send JSON data without defining an app
In most cases it is best to delegate permissions on a per-app basis. But if your use case requires it, you can push data to Fusion without defining an app.
To send JSON data without app security, issue the following curl command:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/json' https://FUSION_HOST:FUSION_PORT/api/index/INDEX_PROFILE --data-binary @my-json-data.json
```
### Example: Send XML data to an index profile with an app
To send XML data to an app, use the following:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/xml' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE --data-binary @my-xml-file.xml
```
In Fusion 5, documents can be created on the fly using the [PipelineDocument](https://javadoc.lucidworks.com/fusion-pipeline-javadocs/5.9/com/lucidworks/apollo/common/pipeline/PipelineDocument.html) JSON notation.
## Remove documents
### Example 1
The following example removes content:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/vnd.lucidworks-document' https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index/INDEX_PROFILE --data-binary @del-json-data.json
```
### Example 2
A more specific example removes data from `books.json`. To delete "The Lightning Thief" and "The Sea of Monsters" from the index, use their id values in the JSON file.
The `del-json-data.json` file to delete the two books:
```json wrap theme={"dark"}
[{ "id": "978-0641723445","commands": [{"name": "delete","params": {}}]},{ "id": "978-1423103349","commands": [{"name": "delete","params": {}}, {"name": "commit","params": {}}]}]
```
You can use `?echo=false` to turn off the response to the terminal.
### Example 3
Another example to delete items using the Push API is:
```bash wrap theme={"dark"}
curl -u admin:XXX -X POST 'http://FUSION_HOST:FUSION_PORT/api/apps/APP/index/INDEX' -H 'Content-Type: application/vnd.lucidworks-document' -d '[
{
"id": "1663838589-44",
"commands":
[
{
"name": "delete",
"params":
{}
},
{
"name": "commit",
"params":
{}
}
]
}, ...
]'
```
## Send documents to an index pipeline
Although sending documents to an index profile is recommended, if your use case requires it, you can send documents directly to an index pipeline.
For more information about index pipeline REST API reference documentation, see [Fusion 5.x Index Pipelines API](/api-reference/index-pipelines-api/list-all-index-pipelines).
### Specify a parser
When you push data to a pipeline, you can specify the name of the parser by adding a parserId querystring parameter to the URL.
For example: `https://FUSION_HOST:FUSION_PORT/api/index-pipelines/INDEX_PIPELINE/collections/COLLECTION_NAME/index?parserId=PARSER`.
If you do not specify a parser, and you are indexing outside of an app (`https://FUSION_HOST:FUSION_PORT/api/index-pipelines/...`), then the `_system` parser is used.
If you do not specify a parser, and you are indexing in an app context (`https://FUSION_HOST:FUSION_PORT/api/apps/APP_NAME/index-pipelines/...`), then the parser with the same name as the app is used.
## Indexing CSV Files
In the usual case, to index a CSV or TSV file, the file is split into records, one per row, and each row is indexed as a separate document.
When using the REST API for synchronous parsing, you can send content directly to the Fusion indexing endpoint for Fusion to process. You can designate an index profile that consists of a parser, collection, and a pipeline. If you do not want to use an index profile, you can specify the parser, collection, and pipeline in the API call.
When using asynchronous parsing, you can send content directly to the Fusion indexing endpoint for Fusion to process. You can designate an index profile that consists of a parser, collection, and a pipeline. If you do not want to use an index profile, you can specify the parser, collection, and pipeline in the API call.
Asynchronous parsing only uses V2 connectors.
The [index pipeline](/docs/5/fusion/getting-data-in/indexing/index-pipelines/overview) consists of one or more configurable [index pipeline stages](/docs/5/fusion/getting-data-in/indexing/index-pipelines/index-pipeline-stages), each performing a different type of transformation on the incoming data. Each connector has a default index pipeline, but you can modify these or create new ones.
The last stage in any index pipeline should be the [Solr Indexer](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/solr-indexer-stage) stage, which submits the documents to Solr for indexing.
## 2. Index your signals
Signals are event records that provide historical data about user behavior, such as clicks, likes, purchases, and so on. You don’t need to index signals about query responses; Fusion indexes response signals automatically by default. If you are using [App Studio](/docs/5/fusion/dev-portal/appkit/overview) or [App Insights](/docs/5/fusion/getting-data-out/data-analytics/app-insights/overview), then you need to index request signals. Learn more about [signal types and required fields](/docs/5/fusion/getting-data-out/query-enhancement/signals/types-and-structures).
To index your signals, you send them to Fusion using the [Signals API](/api-reference/signal-indexing-api/index-an-event-signal), which points to the hidden index pipeline designed especially for signals.
Raw signals are indexed in a secondary collection called `COLLECTION_NAME_signals`. For example, if your primary collection is called `Products`, then the raw signals collection is `Products_signals`.
## 3. Fusion jobs that index signals-related data
When you **Enable or Disable Signals**, Fusion creates jobs and secondary collections for analyzing and aggregating your raw signals. Some of this data enables [query rewriting](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) and [automatic boosting](/docs/5/fusion/getting-data-out/query-enhancement/boosting/boost-with-signals), while other data becomes useful when you enable [recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/overview).
You can enable and disable signals using the [Fusion UI](#using-the-ui) or the [REST API](#using-the-collection-features-api).
When you disable signals, the [aggregation jobs](/docs/4/fusion-ai/concepts/signals-and-aggregations/aggregations/sql-aggregations) are deleted, but the `_signals` and `_signals_aggr` collections are not, your legacy signal data remains intact.
## Using the UI
When you create a collection using the Fusion UI, signals are enabled and a signals collection created by default. You can also enable and disable signals for existing collections using the Collections Manager.
**Enable signals for a collection**
1. In the Fusion workspace, navigate to **Collections** > **Collections Manager**.
2. Hover over the primary collection for which you want to enable signals.
3. Click **Configure** to open the drop-down menu.
4. Click **Enable Signals**.\
The **Enable Signals** window appears, with a list of collections and jobs that are created when you enable signals.
5. Click **Enable Signals**.
**Disable signals for a collection**
1. In the Fusion workspace, navigate to **Collections** > **Collections Manager**.
2. Hover over the primary collection for which you want to disable signals.
3. Click **Configure** to open the drop-down menu.
4. Click **Disable Signals**.\
The **Disable Signals** window appears, with a list of jobs that are created when you enable signals.
5. Click **Disable Signals**.\
Your `_signals` and `_signals_aggr` collections remain intact so that you can access your legacy signals data.
## Using the Collection Features API
Using the API, the [`/collections/{collection}/features/{feature}`](/api-reference/collections/get-collection-features) endpoint enables or disables signals for any collection:
**Check whether signals are enabled for a collection**
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD http://localhost:{api-port}/api/collections/COLLECTION_NAME/features/signals
```
**Enable signals for a collection**
```json wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X PUT -H "Content-type: application/json" -d '{"enabled" : true}' http://localhost:{api-port}/api/collections/COLLECTION_NAME/features/signals
```
**Disable signals for a collection**
```json wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X PUT -H "Content-type: application/json" -d '{"enabled" : false}' http://localhost:{api-port}/api/collections/COLLECTION_NAME/features/signals
```
* The [`_session_rollup` job](/docs/5/fusion/reference/config-ref/jobs/aggregations/built-in-sql-aggregation-jobs#collection-name-session-rollup) creates session signals from your raw signals and adds them to the `_signals` collection.
* [Automatic query rewrites](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) are created by [several jobs](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) and indexed in the `_query_rewrite_staging` collection for review before they are published to the `_query_rewrite` collection.
* The [`_head_tail` job](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis) sends [rewrites for underperforming queries](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) to the `_query_rewrite_staging` collection. It also sends analytics tables to the `_job_reports` collection.
* The `_signals_aggr` collection stores unique combinations of query, document ID, and filters found in your raw signals, each with a timestamp, weight, and count. These are generated by the [`_click_signals_aggregation` job](/docs/5/fusion/reference/config-ref/jobs/aggregations/overview) and the [`_user_query_history_agg` job](/docs/5/fusion/reference/config-ref/jobs/aggregations/built-in-sql-aggregation-jobs).
* The [`user_item_prefs_agg` job](/docs/5/fusion/reference/config-ref/jobs/aggregations/built-in-sql-aggregation-jobs) identifies unique pairs of document IDs and user IDs, assigns a timestamp, weight, and count to each pair, and indexes it in the `_recs_aggr` collection.
## 4. Fusion jobs that index recommendations
When you [enable recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/getting-started#enable-recommendations), another set of jobs and secondary collections is created.
The default recommendation jobs read data from different collections depending on the type of recommendations being generated:
* [Content-based recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-item) are produced by the [`_content_recs` job](/docs/5/fusion/reference/config-ref/jobs/content-based-recommender) by analyzing your primary collection.
* [Queries-for-query recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/queries-for-query) are produced by the [`_query_recs` job](/docs/5/fusion/reference/config-ref/jobs/query-to-query-session-based-similarity) by analyzing your raw signals.
* BPR-based [items-for-item](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-item#collaborative-items-for-item-recommendations) and [items-for-user](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-user) recommendations are produced by the [`_bpr_item_recs` job](/docs/5/fusion/reference/config-ref/jobs/bpr-recommender) by analyzing aggregated signals.
For more recommendations, try **Identify Trending Items or Queries**.
The Trending Recommender job analyzes signals to measure customer engagement over time. Use this job to identify spikes in popularity for specific items or queries, then display those items to your users or analyze the trends for business purposes. You can configure any time window, such as daily, weekly, or monthly.
For complete details about the job’s configuration options, see [Trending Recommender Jobs](/docs/5/fusion/reference/config-ref/jobs/trending-recommender).
**How to identify trending items or queries**
1. Navigate to **Collections** > **Jobs** > **Add +** > **Trending Recommender**.
2. Configure the job:
1. Enter an ID for this job.
2. In the **Reference Time Days** field, enter the number of days to use as a *baseline* for identifying trends, starting from today.\
For example, enter 21 days to analyze three weeks of signals data to use as a baseline.
3. In the **Target Time Days** field, enter the number of days to use as a *target* for identifying trends, starting from today.\
For example, enter 7 days to get documents or products whose popularity has spiked in the past week.
**Reference Time Days** and **Target Time Days** do not overlap. For example, with the values suggested above, a total of 28 days of signals are analyzed, and the first 21 days are compared to the last 7 days.
4. If you want to identify trending queries instead of trending items, change the value of the **Document ID Field** from `doc_id_s` to `query_s`.
This field must be present in your signals. See the required signals fields in the [Trending Recommender Jobs](/docs/5/fusion/reference/config-ref/jobs/trending-recommender) reference topic.
5. In the **Training Collection** field, enter the Solr collection or cloud path where signals are stored (the `COLLECTION_NAME_signals` collection by default).
6. In the **Output Collection** field, enter the Solr collection or cloud path where trend analysis data will be stored.
7. If you are using a format other than `solr`, enter it in the **Data Format** field.
8. In the **Solr Fields to Read** field, enter one or more field names containing text training data.
You can enter multiple field names with weights, as in `field1:weight1,field2:weight2...`.
9. In the **Event Count Field Name** field, enter the name of the event count field in your training data, usually `count_i`.
3. Click **Save**.
4. Click **Run** > **Start** to run the job.
The job outputs documents similar to this example:
```json theme={"dark"}
{
"doc_id":250,
"ref_hits":1,
"ref_rank":1,
"trgt_hits":1,
"trgt_rank":1,
"vol_diff":0.5,
"average_weekly_vol":0.5,
"hit_vol_ratio":2.0,
"combine_score":1.0,
"vol_diff_ratio":1.0,
"ref_wt_vol_diff_ratio":1.0,
"vol_diff_wt_vol_diff_ratio":0.5,
"log_diff_wt_ratio":1.3068528194400546,
"trend-type":"prds_weekly",
"id":"284f930d-d750-49a2-90ac-be4692bddda9",
"_version_":1682995654191742976
}
```
5. Configure a query pipeline to retrieve trending items from the job’s output collection for display or further analysis. For information about pipelines created when recommendations are enabled, see [enable recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/getting-started).
Search on the `log_diff_wt_ratio` field to find the top trending items in the output collection.
## What’s next?
See [Query Data Flow](/docs/5/fusion/intro/query-dataflow) to learn about querying your content and recommendations.
# Kubernetes Deployment Architecture
Source: https://doc.lucidworks.com/docs/5/fusion/intro/kubernetes
[localhost link]: http://localhost:3000/docs/5/fusion/intro/kubernetes
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/intro/kubernetes
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/3209
The figure below provides a general overview on Fusion architecture and shows how Fusion services and data flow within a deployment.
The image above is for illustrative purposes only. It depicts services available in Fusion 5.9, but your actual implementation may differ.
## Architecture and requirements
* [Which Kubernetes?](/docs/5/fusion/operations/survival-guide/platforms)
* [Multiple Zones for High Availability](/docs/5/fusion/operations/survival-guide/high-availability)
* [Fusion Microservices](/docs/5/fusion/reference/microservices)
* [Ingress and Security](/docs/5/fusion/operations/survival-guide/ingress)
* [Stateless Sessions with JWT](/docs/5/fusion/operations/survival-guide/jwt)
* [Workload Isolation with Multiple Node Pools](/docs/5/fusion/operations/survival-guide/workload-isolation)
* [High-Performance Query Processing with Auto-Scaling](/docs/5/fusion/operations/survival-guide/auto-scaling)
The learning path for **Implementing Fusion** focuses on how to be successful when starting your implementation journey to avoid delays.
## Deploy Fusion on Kubernetes
Fusion supports deployment on Google Kubernetes Engine (GKE). This topic explains how to deploy a Fusion cluster on GKE using the [`setup_f5_gke.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_gke.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository.
## Prerequisites
This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes.
### Release Name and Namespace
Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) Kubernetes namespace to install Fusion into.
Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*.
However, please *do not* install more than one Fusion release in the same namespace.
All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_
Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default.
### Install Helm
Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster.
Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform.
On MacOS, you can do:
```bash theme={"dark"}
brew install kubernetes-helm
```
If you already have helm installed, make sure you're using the latest version:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/)
The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`.
#### Helm User Permissions
If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory.
When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.:
```bash theme={"dark"}
alias k=kubectl
```
To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into:
```bash theme={"dark"}
k create namespace fusion-namespace
```
Apply the `role.yaml` and `cluster-role.yaml` files to that namespace
```bash theme={"dark"}
k apply -f cluster-role.yaml
k config set-context --current --namespace=$NAMESPACE
k apply -f role.yaml
```
Then bind the rolebinding and clusterolebinding to the install user:
```bash wrap theme={"dark"}
k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user
k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user
```
You will then be able to run the `helm install` command as the ``
### Clone fusion-cloud-native from GitHub
You should clone this repo from github as you'll need to run the scripts on your local workstation:
```bash theme={"dark"}
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts.
```bash theme={"dark"}
cd fusion-cloud-native
git pull
```
Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip).
Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory.
## Prerequisites
This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes.
### Release Name and Namespace
Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) Kubernetes namespace to install Fusion into.
Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*.
However, please *do not* install more than one Fusion release in the same namespace.
All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_
Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default.
### Install Helm
Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster.
Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform.
On MacOS, you can do:
```bash theme={"dark"}
brew install kubernetes-helm
```
If you already have helm installed, make sure you're using the latest version:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/)
The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`.
#### Helm User Permissions
If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory.
When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.:
```bash theme={"dark"}
alias k=kubectl
```
To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into:
```bash theme={"dark"}
k create namespace fusion-namespace
```
Apply the `role.yaml` and `cluster-role.yaml` files to that namespace
```bash theme={"dark"}
k apply -f cluster-role.yaml
k config set-context --current --namespace=$NAMESPACE
k apply -f role.yaml
```
Then bind the rolebinding and clusterolebinding to the install user:
```bash wrap theme={"dark"}
k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user
k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user
```
You will then be able to run the `helm install` command as the ``
### Clone fusion-cloud-native from GitHub
You should clone this repo from github as you'll need to run the scripts on your local workstation:
```bash theme={"dark"}
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts.
```bash theme={"dark"}
cd fusion-cloud-native
git pull
```
Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip).
Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory.
The [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_gke.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_gke.sh) `setup_f5_gke.sh` script provided in this repo is strictly optional.
The script is mainly to help those new to Kubernetes and/or Fusion get started quickly.
If you're already familiar with K8s, Helm, and GKE, then you can skip the script and just use Helm directly to install Fusion into an existing cluster or one you create yourself using the process described `<>`.
### Set up the Google Cloud SDK (one time only)
If you've already installed the `gcloud` command-line tools, you can skip to `<>`.
These steps set up your local Google Cloud SDK environment so that you're ready to use the command-line tools to manage your Fusion deployment.
Usually, you only need to perform these setup steps once. After that, you're ready to [create a cluster](#create-a-single-node-demo-cluster).
For a nice getting started tutorial for GKE, see: [https://cloud.google.com/kubernetes-engine/docs/deploy-app-cluster](https://cloud.google.com/kubernetes-engine/docs/deploy-app-cluster)
How to set up the Google Cloud SDK:
1. [https://console.cloud.google.com/apis/library/container.googleapis.com?q=kubernetes%20engine](https://console.cloud.google.com/apis/library/container.googleapis.com?q=kubernetes%20engine)
2. Log in to Google Cloud: `gcloud auth login`
3. Set up the Google Cloud SDK:
1. `gcloud config set compute/zone `
If you are working with regional clusters instead of zone clusters, use `gcloud config set compute/region ` instead.
2. `gcloud config set core/account `
3. *New GKE projects only:* `gcloud projects create `
If you have already created a project, for example in [https://console.cloud.google.com/](https://console.cloud.google.com/), then skip to the next step.
4. `gcloud config set project `
Make sure you install the Kubernetes command-line tool `kubectl` using:
```bash theme={"dark"}
gcloud components install kubectl
gcloud components update
```
### Create a single-node demo cluster
Run the [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_gke.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_gke.sh) `setup_f5_gke.sh` script to install Fusion 5.x in a GKE cluster. To create a new, single-node *demo* cluster and install Fusion, simply do:
```bash theme={"dark"}
./setup_f5_gke.sh -c -p --create demo
```
Use the `--help` option to see script usage. If you want the script to create a cluster for you, then you need to pass the `--create` option with either `demo` or `multi_az`. If you don't want the script to create a cluster, then you need to create a cluster before running the script and simply pass the name of the existing cluster using the `-c` parameter.
If you pass `--create demo` to the script, then we create a single node GKE cluster (defaults to using `n1-standard-8` node type). The minimum node type you'll need for a 1 node cluster is an `n1-standard-8` (on GKE) which has 8 CPU and 30 GB of memory. This is cutting it very close in terms of resources as you also need to host all of the Kubernetes system pods on this same node. Obviously, this works for kicking the tires on Fusion 5.1 but is not sufficient for production workloads.
You can change the instance type using the `-i` parameter; see: [https://cloud.google.com/compute/docs/regions-zones/#available](https://cloud.google.com/compute/docs/regions-zones/#available) for a list of which machine types are available in your desired region.
Newer Google Cloud machine families, including N4, C4, and C4A, can't attach the `pd-balanced` disks that GKE's default storage class provisions, which causes Fusion's stateful services to stall with `FailedAttachVolume` events.
If you choose one of these machine families, create a Hyperdisk storage class before installing Fusion.
See [Match the storage class to the GKE machine family](/docs/5/fusion/operations/deployment#match-the-storage-class-to-the-gke-machine-family).
If not provided the script generates a custom values file named `gke___fusion_values.yaml` which you can use to customize the Fusion chart.\_\_
\#WARNING# If using Helm V2, the `setup_f5_gke.sh` script installs Helm's `tiller` component into your GKE cluster with the cluster admin role. If you don't want this, then please upgrade to Helm v3.
If you see an error similar to the following, then wait a few seconds and try running the `setup_f5_gke.sh` script again with the same arguments as this is usually a transient issue:
```bash wrap theme={"dark"}
Error: could not get apiVersions from Kubernetes: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
```
After running the `setup_f5_gke.sh` script, proceed to the `<>` section below.
When you're ready to deploy Fusion to a production-like environment, see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
### Create a three-node regional cluster to withstand a zone outage
With a three-node regional cluster, nodes are deployed across three separate availability zones.
```bash wrap theme={"dark"}
./setup_f5_gke.sh -c -p -n --region --create multi_az
```
* `` value should be the name of a non-existent cluster; the script will create the new cluster.
* `` must match the name of an existing project in GKE. Run `gcloud config get-value project` to get this value, or see the [GKE setup instructions](#set-up-the-google-cloud-sdk-one-time-only).
* `` Kubernetes namespace to install Fusion into, defaults to `default` with release `f5`
* `` value should be the name of a GKE region, defaults to `us-west1`. Run `gcloud config get-value compute/zone` to get this value, or see the [GKE setup instructions](#set-up-the-google-cloud-sdk-one-time-only) to set the value.
In this configuration, Kubernetes deploys a ZooKeeper and Solr pod on each of the three nodes, which allows the cluster to retain ZK quorum and remain operational after losing one node, such as during an outage in one availability zone.
When running in a multi-zone cluster, each Solr node has the `solr_zone` system property set to the zone it is running in, such as `-Dsolr_zone=us-west1-a`.
After running the `setup_f5_gke.sh` script, proceed to the `<>` section below.
When you're ready to deploy Fusion to a production-like environment, see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
#### GKE Ingress and TLS
The Fusion proxy service provides authentication and serves as an API gateway for accessing all other Fusion services.
It's typical to use an Ingress for TLS termination in front of the proxy service.
The `setup_f5_gke.sh` supports creating an Ingress with a TLS cert for a domain you own by passing: `-t -h `
After the script runs, you need to create an A record in GCP's DNS service to map your domain name to the Ingress IP. Once this occurs, our script setup uses [https://letsencrypt.org/](https://letsencrypt.org/) to issue a TLS cert for your Ingress.
To see the status of the Let's Encrypt issued certificate, do:
```bash theme={"dark"}
kubectl get managedcertificates -n -o yaml
```
Please refer to the Kubernetes documentation on configuring an Ingress for GKE: [https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer](https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer)
The GCP Ingress defaults to a 30 second timeout, which can lead to false negatives for long running requests such as importing apps. To configure the timeout for the backend in kubernetes:
Create a BackendConfig object in your namespace:
```yaml theme={"dark"}
---
apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
name: backend_config_name
spec:
timeoutSec: 120
connectionDraining:
drainingTimeoutSec: 60
```
Then make sure that the following entries are in the right place in your values.yaml file:
```yaml theme={"dark"}
api-gateway:
service:
annotations:
beta.cloud.google.com/backend-config: '{"ports": {"6764":"backend_config_name"}}'
```
and upgrade your release to apply the configuration changes
#### Ingresses and externalTrafficPolicy
When running a fusion cluster behind an externally controlled LoadBalancer it can be advantageous
to configure the `externalTrafficPolicy` of the `proxy` service to `Local`. This preserves the client
source IP and avoids a second hop for LoadBalancer and NodePort type services, but risks potentially
imbalanced traffic spreading. Although when running in a cluster with a dedicated pool for spark jobs
that can scale up and down freely it can prevent unwanted request failures. This behaviour can be
altered with the `api-gateway.service.externalTrafficPolicy` value, which is set to `Local` if the example values
file is used.
**You must use `externalTrafficPolicy`=`Local` for the Trusted HTTP Realm to work correctly.**
If you are already using a custom 'values.yaml' file, create an entry for `externalTrafficPolicy` under `api-gateway` service.
```yaml theme={"dark"}
api-gateway:
service:
externalTrafficPolicy: Local
```
#### Considerations when using the nginx ingress controller
If you are using the `nginx` ingress controller to fulfil your ingress definitions there are a couple
of options that are recommended to be set in the configmap:
```bash wrap theme={"dark"}
enable-underscores-in-headers: "true" # Fusion can return some headers that have underscores, these have to be explicitly enabled in nginx
proxy-body-size: "0" # By default nginx places a maximum size on request bodies, either increase as needed or disable by setting to 0
proxy-read-timeout: "300" # Increases the timeout for potential slow queries.
```
#### Custom values
There are some example values files that can be used as a starting point for
resources, affinity and replica count configuration in the `example-values` folder.
These can be passed to the install script using the `--values` option, for example:
```bash wrap theme={"dark"}
./setup_f5_gke.sh -c -p -r -n \
--values example-values/affinity.yaml --values example-values/resources.yaml --values example-values/replicas.yaml
```
The `--values` option can be passed multiple times, if the same configuration property is contained within multiple `values` files then the values from the latest file passed as a `--values` option are used.
##### Connectors custom values
If you are using Fusion 5.9 or later, you can specify resources and replica count per connector. This allows you to set different resource limits for each connector. If you do not set custom values for a connector, that connector uses the default values.
Set each connector's resource values in the `connector-plugin` section under `pluginValues`. The `pluginValues` section is a list of plugins and its resources. The following sample shows an example.
```yaml theme={"dark"}
pluginValues:
- id: "plugin-id" <1>
resources: <2>
limits:
cpu: "2"
memory: "3Gi"
requests:
cpu: "250m"
memory: "2Gi"
replicaCount: 1 <3>
```
* `<1>` The plugin ID. The plugin ID must match the plugin ID on the plugin ZIP file. without the `lucidworks.` prefix. For example, if the plugin ID on the plugin ZIP file is `lucidworks.sharepoint-optimized`, the plugin ID is `sharepoint-optimized`.
* `<2>` The resources settings. You may specify the limits, the requests, and the CPU and memory for each.
* `<3>` The number of replicas per connector. This value is 1 by default.
After editing the `connector-plugin` section, you must reinstall the affected connector.
#### Upgrades and Ingress
If you used the `-t -h ` options when installing your cluster, our script created an additional values yaml file named `tls-values.yaml`.
To make things easier for you when upgrading, you should add the settings from this file into your main custom values yaml file, e.g.:
```yaml theme={"dark"}
api-gateway:
service:
type: "NodePort"
ingress:
enabled: true
host: ""
tls:
enabled: true
annotations:
"networking.gke.io/managed-certificates": "-managed-certificate"
"kubernetes.io/ingress.class": "gce"
```
This way you don't have to remember to pass the additional `tls-values.yaml` file when upgrading.
## Verifying the Fusion Installation
In this section, we provide some tips on how to verify the Fusion installation.
Check if the Fusion Admin UI is available at `\https://:6764/admin/`.
Let's review some useful kubectl commands.
### Enhance the K8s Command-line Experience
Here is a list of tools we found useful for improving your command-line experience with Kubernetes:
* krew (kubectl plugin mgr): [https://github.com/kubernetes-sigs/krew/](https://github.com/kubernetes-sigs/krew/)
* kube-ps1 (show current context on command line prompt): [https://github.com/jonmosco/kube-ps1](https://github.com/jonmosco/kube-ps1)
* kubectx / kubens (switch between clusters / namespaces): [https://github.com/ahmetb/kubectx](https://github.com/ahmetb/kubectx)
### Useful kubectl commands
kubectl reference: [https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands)
Set the namespace for `kubectl` if not using the default:
```bash theme={"dark"}
kubectl config set-context --current --namespace=
```
**This saves you from having to pass `-n` with every command.**
Get a list of running pods: `k get pods`
Get logs for a pod using a label: `k logs –l app.kubernetes.io/component=query-pipeline`
Get pod deployment spec and details: `k get pods -o yaml`
Get details about a pod events: `k describe po `
Port forward to a specific pod: `k port-forward 8983:8983`
SSH into a pod: `k exec -it -- /bin/bash`
CPU/Memory usage report for pods: `k top pods`
Forcefully kill a pod: `k delete po --force --grace-period 0`
Scale up (or down) a deployment: `k scale deployment.v1.apps/ --replicas=N`
Get a list of pod versions: `k get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq`
### Check Fusion Pods and Services
Once the install script completes, you can check that all pods and services are available using:
```bash theme={"dark"}
kubectl get pods
```
If all goes well, you should see a list of pods similar to:
```bash wrap theme={"dark"}
NAME READY STATUS RESTARTS AGE
seldon-controller-manager-6675874894-qxwrv 1/1 Running 0 8m45s
f5-admin-ui-74d794f4f8-m5jms 1/1 Running 0 8m45s
f5-ambassador-fd6b9b5dc-7ghf6 1/1 Running 0 8m43s
f5-api-gateway-6b9998b9c-tmchk 1/1 Running 0 8m45s
f5-auth-ui-7565564b4c-rdc74 1/1 Running 0 8m42s
f5-classic-rest-service-0 1/1 Running 3 8m44s
f5-devops-ui-77bb867ffb-fbzxd 1/1 Running 0 8m42s
f5-fusion-admin-78b8f8fc7f-4d7l8 1/1 Running 0 8m42s
f5-fusion-indexing-599c8d448-xzsvm 1/1 Running 0 8m44s
f5-insights-665fd9f6fc-g5psw 1/1 Running 0 8m43s
f5-job-launcher-84dd4c5c96-p8528 1/1 Running 0 8m44s
f5-job-rest-server-6d44d964b8-xtnxw 1/1 Running 0 8m45s
f5-logstash-0 1/1 Running 0 8m45s
f5-ml-model-service-6987dc94c9-9ppp8 2/2 Running 1 8m45s
f5-monitoring-grafana-5d499dbb58-pzw72 1/1 Running 0 10m
f5-monitoring-prometheus-kube-state-metrics-54d6678dv9h7h 1/1 Running 0 10m
f5-monitoring-prometheus-pushgateway-7d65c65b85-vwrwf 1/1 Running 0 10m
f5-monitoring-prometheus-server-0 2/2 Running 0 10m
f5-pm-ui-86cbc5bb65-nd2n8 1/1 Running 0 8m44s
f5-pulsar-bookkeeper-0 1/1 Running 0 8m45s
f5-pulsar-broker-b56cc776f-56msx 1/1 Running 0 8m45s
f5-query-pipeline-5d75d7d5f4-l2mdf 1/1 Running 0 8m43s
f5-connectors-7bb6cfc65f-7wfs2 1/1 Running 0 8m42s
f5-connectors-backend-987fdc648-dldwv 1/1 Running 0 8m45s
f5-rules-ui-6b9d55b78f-9hzzj 1/1 Running 0 8m43s
f5-solr-0 1/1 Running 0 8m44s
f5-solr-exporter-c4687c785-jsm7x 1/1 Running 0 8m45s
f5-ui-6cdbcc68c6-rj9cq 1/1 Running 0 8m45s
f5-webapps-6d6bb9bfd-hm4qx 1/1 Running 0 8m45s
f5-workflow-controller-7b66679fb7-sjbvp 1/1 Running 0 8m44s
f5-zookeeper-0 1/1 Running 0 8m45s
```
The number of pods per deployment / statefulset will vary based on your cluster size and replicaCount settings in your custom values YAML file.
Also, don't worry if you see some pods having been restarted as that just means they were too slow to come up and Kubernetes killed and restarted them.
You do want to see at least one pod running for every service. If a pod is not running after waiting a sufficient amount of time,
use `kubectl logs ` to see the logs for that pod; to see the logs for previous versions of a pod, use: `kubectl logs -p`.
You can also look at the actions Kubernetes performed on the pod using `kubectl describe po `.
To see a list of Fusion services, do:
```bash theme={"dark"}
kubectl get svc
```
For an overview of the various Fusion 5 microservices, see: [Fusion microservices](/docs/5/fusion/reference/microservices).
Once you're ready to build a Fusion cluster for production, please see see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
### Upgrading with Zero Downtime
One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. Fusion 5 allows customers to upgrade from Fusion 5.x.y to a later 5.x.z version on a live cluster with zero downtime or disruption of service.
When Kubernetes performs a rolling update to an individual microservice, there will be a mix of old and new services in the cluster concurrently (only briefly in most cases) and requests from other services will be routed to both versions. Consequently, Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same 5.x line of releases. We also ensure stored configuration remains compatible in the same 5.x release line.
Lucidworks releases minor updates to individual services frequently, so our customers can pull in those upgrades using Helm at their discretion.
To upgrade your cluster at any time, use the `--upgrade` option with our setup scripts in this repo.
The scripts in this repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.
To see what would be upgraded, you can pass the `--dry-run` option to the script.
### Grafana Dashboards
Get the initial Grafana password from a K8s secret by doing:
```bash wrap theme={"dark"}
kubectl get secret --namespace "${NAMESPACE}" ${RELEASE}-monitoring-grafana \
-o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```
With Grafana, you can either setup a temporary port-forward to a Grafana pod or expose Grafana on an external IP using a K8s LoadBalancer.
To define a LoadBalancer, do (replace \$ with your Helm release label):
```bash wrap theme={"dark"}
kubectl expose deployment ${RELEASE}-monitoring-grafana --type=LoadBalancer --name=grafana --port=3000 --target-port=3000
```
You can use `kubectl get services --namespace ` to determine when the load balancer is setup and its IP address. Direct your browser to `http://:3000` and enter the username `admin@localhost` and the password that was returned in the previous step.
This will log you into the application. It is recommended that you create another administrative user with a more desirable password.
The dashboards and datasoure will be setup for you in grafana, simply navigate to `Dashboards` -> `Manage` to view the available dashboards
## More deployment options
* How to deploy Fusion 5 on Amazon Elastic Kubernetes Service
* How to deploy Fusion 5 on Azure Kubernetes Service
* How to deploy Fusion on 5 other Kubernetes platforms
## Learn more
The course for **Deploying Fusion 5** focuses on the prerequisite software needed to deploy Fusion, the necessary setup steps, and the physical act of deployment.
Fusion supports deployment on Amazon Elastic Kubernetes Service (EKS). This topic explains how to deploy a Fusion cluster on EKS using the [`setup_f5_eks.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_eks.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository.
In addition, this topic provides information about how to [configure IAM roles for the service account](#configure-iam-roles-for-the-service-account).
## Prerequisites
This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes.
### Release Name and Namespace
Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) Kubernetes namespace to install Fusion into.
Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*.
However, please *do not* install more than one Fusion release in the same namespace.
All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_
Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default.
### Install Helm
Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster.
Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform.
On MacOS, you can do:
```bash theme={"dark"}
brew install kubernetes-helm
```
If you already have helm installed, make sure you're using the latest version:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/)
The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`.
#### Helm User Permissions
If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory.
When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.:
```bash theme={"dark"}
alias k=kubectl
```
To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into:
```bash theme={"dark"}
k create namespace fusion-namespace
```
Apply the `role.yaml` and `cluster-role.yaml` files to that namespace
```bash theme={"dark"}
k apply -f cluster-role.yaml
k config set-context --current --namespace=$NAMESPACE
k apply -f role.yaml
```
Then bind the rolebinding and clusterolebinding to the install user:
```bash wrap theme={"dark"}
k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user
k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user
```
You will then be able to run the `helm install` command as the ``
### Clone fusion-cloud-native from GitHub
You should clone this repo from github as you'll need to run the scripts on your local workstation:
```bash theme={"dark"}
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts.
```bash theme={"dark"}
cd fusion-cloud-native
git pull
```
Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip).
Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory.
The [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_eks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_eks.sh) `setup_f5_eks.sh` script provided in this repo is strictly optional.
The script is mainly to help those new to Kubernetes and/or Fusion get started quickly.
If you're already familiar with K8s, Helm, and EKS, then you use Helm directly to install Fusion into an existing cluster or one you create yourself using the process described `<>`.
If you're new to Amazon Web Services (AWS), then please visit the Amazon Web Services [https://aws.amazon.com/getting-started/](https://aws.amazon.com/getting-started/) to set up an account.
If you're new to Kubernetes and EKS, then we recommend going through Amazon's [https://eksworkshop.com/introduction/](https://eksworkshop.com/introduction/) before proceeding with Fusion.
### Set up the AWS CLI tools
Before launching an EKS cluster, you need to install and configure `kubectl`, `aws`, `eksctl`, `aws-iam-authenticator` using the links provided below:
Required AWS Command-line Tools:
* kubectl: [https://kubernetes.io/docs/tasks/tools/install-kubectl/](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
* aws: [https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html)
* eksctl: [https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html)
* aws-iam-authenticator: [https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html](https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html)
Run `aws configure` to configure a profile for authenticating to AWS. You'll use the profile name you configure in this step, which defaults to `default`, as the `-p` argument to the `setup_f5_eks.sh` script in the next section.
When working in Ubuntu, avoid using the eksctl snap version. Alternative sources can have different versions that could cause command failures. Also, always make sure you are using the latest version for each one of the required tools.
### Set up Fusion on EKS
To create a cluster in EKS the following IAM policies are required:
* AmazonEC2FullAccess
* AWSCloudFormationFullAccess
EKS Permissions:
```bash theme={"dark"}
eks:CreateCluster
eks:DeleteCluster
eks:DescribeCluster
eks:DescribeUpdate
eks:ListClusters
eks:ListUpdates
eks:UpdateClusterVersion
```
VPC Permissions:
```bash theme={"dark"}
ec2:AssociateSubnetCidrBlock
ec2:AssociateVpcCidrBlock
ec2:AttachInternetGateway
ec2:CreateInternetGateway
ec2:CreateSubnet
ec2:CreateVpc
ec2:CreateVpcEndpoint
ec2:DeleteInternetGateway
ec2:DeleteSubnet
ec2:DeleteVpc
ec2:DeleteVpcEndpoints
ec2:DescribeSubnets
ec2:DescribeVpcAttribute
ec2:DescribeVpcs
ec2:DetachInternetGateway
ec2:DisassociateSubnetCidrBlock
ec2:DisassociateVpcCidrBlock
ec2:ModifySubnetAttribute
ec2:ModifyVpcAttribute
ec2:ModifyVpcEndpoint
```
.IAM Permissions
```bash theme={"dark"}
iam:AddRoleToInstanceProfile
iam:AttachRolePolicy
iam:CreateInstanceProfile
iam:CreatePolicy
iam:CreatePolicyVersion
iam:CreateRole
iam:DeleteInstanceProfile
iam:DeletePolicy
iam:DeletePolicyVersion
iam:DeleteRole
iam:DeleteRolePolicy
iam:DetachRolePolicy
iam:GetInstanceProfile
iam:GetPolicy
iam:GetPolicyVersion
iam:GetRole
iam:GetRolePolicy
iam:ListInstanceProfiles
iam:ListInstanceProfilesForRole
iam:PassRole
iam:PutRolePolicy
iam:RemoveRoleFromInstanceProfile
iam:TagRole
iam:UntagRole
```
Download and run the [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_eks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_eks.sh) `setup_f5_eks.sh` script to install Fusion 5.x in an EKS cluster.
This script does not support multiple node pools and should not be used for production clusters.
* To create a new cluster and install Fusion, run the following command:
```
./setup_f5_eks.sh -c my-eks-cluster -p profile-name -n fusion-namespace --create demo
```
* Replace `my-eks-cluster`, `profile-name`, and `fusion-namespace` with your cluster, profile, and namespace values.
* Pass the `--create` option with either `demo` or `multi_az`.
* To use an existing cluster and install Fusion, run the following command:
```
./setup_f5_eks.sh -c cluster-name -p profile-name
```
* Replace `cluster-name` with the name of the cluster you already created.
* Replace `profile-name` with the name of your profile.
The profile is automatically set to `default` if you ran the AWS configure command without giving the profile a name.
Use the `--help` option to see full script usage.
If using Helm V2, the `setup_f5_eks.sh` script installs Helm's `tiller` component into your EKS cluster with the cluster admin role. If you don't want this, then please upgrade to Helm v3.
The `setup_f5_eks.sh` script creates a service account that provides S3 read-only permissions to the created pods.
After running the `setup_f5_eks.sh` script, proceed to the `<>` section below.
#### EKS cluster overview
The EKS cluster is created using `eksctl` ([https://eksctl.io/](https://eksctl.io/)). By default it will setup the following resources in your AWS account:
* A dedicated VPC for the EKS cluster in the specified region with CIDR: `192.168.0.0/16`
* 3 Public and 3 Private subnets within the created VPC, each with a `/19` CIDR range, along with the corresponding route tables.
* A NAT gateway in each Public subnet
* An Auto Scaling Group of the instance type specified by the script, which defaults to `m5.2xlarge`, with 3 instances spanning the public subnets.
See [https://eksctl.io/usage/vpc-networking/](https://eksctl.io/usage/vpc-networking/) for more information on the networking setup.
#### EKS Ingress
The `setup_f5_eks.sh` script exposes the Fusion proxy service on an external DNS name provided by an ELB over HTTP. This is done for demo or getting started purposes. However, you're strongly encouraged to configure a K8s Ingress with TLS termination in front of the proxy service.
See: [https://aws.amazon.com/premiumsupport/knowledge-center/terminate-https-traffic-eks-acm/](https://aws.amazon.com/premiumsupport/knowledge-center/terminate-https-traffic-eks-acm/)
Our EKS script creates a classic ELB for exposing fusion proxy service. In case you need to change this behavior and use [https://github.com/kubernetes-sigs/aws-load-balancer-controller](https://github.com/kubernetes-sigs/aws-load-balancer-controller) instead you can use the following parameters when running the `setup_f5_eks.sh` script:
```bash theme={"dark"}
--deploy-alb # Tells the script to deploy an ALB
```
By default the `kube-system` namespace is being used for installing the `aws-load-balancer-controller` because pods `priorityClassName` is set to `system-cluster-critical`.
In case you need to deploy an internal ALB you can use the `--internal-alb` option. This will create the nodes in the internal subnets. Fusion will be reachable from an AWS instance located in any of the external subnets on the same VPC. To use an ALB also an ingress with a DNS name is required, you can use the `-h` option to create an ingress with the required DNS name.
Finally, use Route 53 or your DNS provider for creating an A ALIAS DNS record for your DNS name pointing to the ingress ADRESS. You can get the address listing the ingress using the command `kubectl get ing`.
## Verifying the Fusion Installation
In this section, we provide some tips on how to verify the Fusion installation.
Check if the Fusion Admin UI is available at `\https://:6764/admin/`.
Let's review some useful kubectl commands.
### Enhance the K8s Command-line Experience
Here is a list of tools we found useful for improving your command-line experience with Kubernetes:
* krew (kubectl plugin mgr): [https://github.com/kubernetes-sigs/krew/](https://github.com/kubernetes-sigs/krew/)
* kube-ps1 (show current context on command line prompt): [https://github.com/jonmosco/kube-ps1](https://github.com/jonmosco/kube-ps1)
* kubectx / kubens (switch between clusters / namespaces): [https://github.com/ahmetb/kubectx](https://github.com/ahmetb/kubectx)
### Useful kubectl commands
kubectl reference: [https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands)
Set the namespace for `kubectl` if not using the default:
```bash theme={"dark"}
kubectl config set-context --current --namespace=
```
**This saves you from having to pass `-n` with every command.**
Get a list of running pods: `k get pods`
Get logs for a pod using a label: `k logs –l app.kubernetes.io/component=query-pipeline`
Get pod deployment spec and details: `k get pods -o yaml`
Get details about a pod events: `k describe po `
Port forward to a specific pod: `k port-forward 8983:8983`
SSH into a pod: `k exec -it -- /bin/bash`
CPU/Memory usage report for pods: `k top pods`
Forcefully kill a pod: `k delete po --force --grace-period 0`
Scale up (or down) a deployment: `k scale deployment.v1.apps/ --replicas=N`
Get a list of pod versions: `k get po -o jsonpath='{..image}'
tr -s '[[:space:]]' '\n'
sort
uniq`
### Check Fusion Pods and Services
Once the install script completes, you can check that all pods and services are available using:
```bash theme={"dark"}
kubectl get pods
```
If all goes well, you should see a list of pods similar to:
```bash wrap theme={"dark"}
NAME READY STATUS RESTARTS AGE
seldon-controller-manager-6675874894-qxwrv 1/1 Running 0 8m45s
f5-admin-ui-74d794f4f8-m5jms 1/1 Running 0 8m45s
f5-ambassador-fd6b9b5dc-7ghf6 1/1 Running 0 8m43s
f5-api-gateway-6b9998b9c-tmchk 1/1 Running 0 8m45s
f5-auth-ui-7565564b4c-rdc74 1/1 Running 0 8m42s
f5-classic-rest-service-0 1/1 Running 3 8m44s
f5-devops-ui-77bb867ffb-fbzxd 1/1 Running 0 8m42s
f5-fusion-admin-78b8f8fc7f-4d7l8 1/1 Running 0 8m42s
f5-fusion-indexing-599c8d448-xzsvm 1/1 Running 0 8m44s
f5-insights-665fd9f6fc-g5psw 1/1 Running 0 8m43s
f5-job-launcher-84dd4c5c96-p8528 1/1 Running 0 8m44s
f5-job-rest-server-6d44d964b8-xtnxw 1/1 Running 0 8m45s
f5-logstash-0 1/1 Running 0 8m45s
f5-ml-model-service-6987dc94c9-9ppp8 2/2 Running 1 8m45s
f5-monitoring-grafana-5d499dbb58-pzw72 1/1 Running 0 10m
f5-monitoring-prometheus-kube-state-metrics-54d6678dv9h7h 1/1 Running 0 10m
f5-monitoring-prometheus-pushgateway-7d65c65b85-vwrwf 1/1 Running 0 10m
f5-monitoring-prometheus-server-0 2/2 Running 0 10m
f5-pm-ui-86cbc5bb65-nd2n8 1/1 Running 0 8m44s
f5-pulsar-bookkeeper-0 1/1 Running 0 8m45s
f5-pulsar-broker-b56cc776f-56msx 1/1 Running 0 8m45s
f5-query-pipeline-5d75d7d5f4-l2mdf 1/1 Running 0 8m43s
f5-connectors-7bb6cfc65f-7wfs2 1/1 Running 0 8m42s
f5-connectors-backend-987fdc648-dldwv 1/1 Running 0 8m45s
f5-rules-ui-6b9d55b78f-9hzzj 1/1 Running 0 8m43s
f5-solr-0 1/1 Running 0 8m44s
f5-solr-exporter-c4687c785-jsm7x 1/1 Running 0 8m45s
f5-ui-6cdbcc68c6-rj9cq 1/1 Running 0 8m45s
f5-webapps-6d6bb9bfd-hm4qx 1/1 Running 0 8m45s
f5-workflow-controller-7b66679fb7-sjbvp 1/1 Running 0 8m44s
f5-zookeeper-0 1/1 Running 0 8m45s
```
The number of pods per deployment / statefulset will vary based on your cluster size and replicaCount settings in your custom values YAML file.
Also, don't worry if you see some pods having been restarted as that just means they were too slow to come up and Kubernetes killed and restarted them.
You do want to see at least one pod running for every service. If a pod is not running after waiting a sufficient amount of time,
use `kubectl logs ` to see the logs for that pod; to see the logs for previous versions of a pod, use: `kubectl logs -p`.
You can also look at the actions Kubernetes performed on the pod using `kubectl describe po `.
To see a list of Fusion services, do:
```bash theme={"dark"}
kubectl get svc
```
For an overview of the various Fusion 5 microservices, see: [Fusion microservices](/docs/5/fusion/reference/microservices).
Once you're ready to build a Fusion cluster for production, please see see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
### Upgrading with Zero Downtime
One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. Fusion 5 allows customers to upgrade from Fusion 5.x.y to a later 5.x.z version on a live cluster with zero downtime or disruption of service.
When Kubernetes performs a rolling update to an individual microservice, there will be a mix of old and new services in the cluster concurrently (only briefly in most cases) and requests from other services will be routed to both versions. Consequently, Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same 5.x line of releases. We also ensure stored configuration remains compatible in the same 5.x release line.
Lucidworks releases minor updates to individual services frequently, so our customers can pull in those upgrades using Helm at their discretion.
To upgrade your cluster at any time, use the `--upgrade` option with our setup scripts in this repo.
The scripts in this repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.
To see what would be upgraded, you can pass the `--dry-run` option to the script.
### Grafana Dashboards
Get the initial Grafana password from a K8s secret by doing:
```bash wrap theme={"dark"}
kubectl get secret --namespace "${NAMESPACE}" ${RELEASE}-monitoring-grafana \
-o jsonpath="{.data.admin-password}"
base64 --decode ; echo
```
With Grafana, you can either setup a temporary port-forward to a Grafana pod or expose Grafana on an external IP using a K8s LoadBalancer.
To define a LoadBalancer, do (replace \$ with your Helm release label):
```bash wrap theme={"dark"}
kubectl expose deployment ${RELEASE}-monitoring-grafana --type=LoadBalancer --name=grafana --port=3000 --target-port=3000
```
You can use `kubectl get services --namespace ` to determine when the load balancer is setup and its IP address. Direct your browser to `http://:3000` and enter the username `admin@localhost` and the password that was returned in the previous step.
This will log you into the application. It is recommended that you create another administrative user with a more desirable password.
The dashboards and datasoure will be setup for you in grafana, simply navigate to `Dashboards` -> `Manage` to view the available dashboards
## Configure IAM roles for the service account
Configuring IAM roles lets you utilize the Amazon Web Services Security Token Service (AWS STS) for short-term authentication credentials to access services like the Amazon S3 simple storage service.
To configure IAM roles, your user account must be granted `admin` permissions or `IAM:FullAccess`. Complete the following steps:
1. To create the OpenID Connect (OIDC) provider, run the following command:
```bash wrap theme={"dark"}
eksctl utils associate-iam-oidc-provider --cluster cluster_name --approve
```
2. To create an IAM role for the service account associated with the plugin-pod, run the following command:
```bash wrap theme={"dark"}
eksctl create iamserviceaccount --name f5-connector-plugin --namespace default --cluster cluster_name --attach-policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess --approve --override-existing-serviceaccounts
```
This command:
* Creates an IAM role and attaches the target policy.
* Updates the existing Kubernetes `f5-connector-plugin` service account and annotates it with the IAM role.
* Uses the existing `policy/AmazonS3ReadOnlyAccess` policy.
If the IAM role was already created without the command, and you want to associate the service account, run the following command:
```bash wrap theme={"dark"}
kubectl annotate serviceaccount -n default f5-connector-plugin eks.amazonaws.com/role-arn=arn:aws:iam::411271863668:role/FUS_ROLE --overwrite=true
```
To utilize this feature, create a data source with the settings in **S3 Authentication Settings > AWS Instance Credentials Authentication Settings**. For detailed installation information, see [AWS S3 Pro connector](/docs/fusion-connectors/connectors/amazon-aws-s3-pro).
For more information, see:
* [Welcome to the AWS Security Token Service API Reference](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html)
* [Temporary security credentials in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html)
* [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html)
* [What is Amazon EKS?](https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html)
* [Fine-grained IAM roles for service accounts](https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts)
## More deployment options
* How to deploy Fusion 5 in Google Kubernetes Engine
* How to deploy Fusion 5 in Azure Kubernetes Service
* How to deploy Fusion 5 on other Kubernetes platforms
## Learn more
The course for **Deploying Fusion 5** focuses on the prerequisite software needed to deploy Fusion, the necessary setup steps, and the physical act of deployment.
Fusion supports deployment on Azure Kubernetes Service (AKS). This topic explains how to deploy a Fusion cluster on AKS using the [`setup_f5_aks.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_aks.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository.
The `setup_f5_aks.sh` script is the basic foundation for getting started and proof-of-concept purposes. For information about custom values in a production-ready environment, see Custom values YAML file.
## Prerequisites
This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes.
### Release Name and Namespace
Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) namespace to install Fusion into.
Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*.
However, please *do not* install more than one Fusion release in the same namespace.
All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_
Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default.
### Install Helm
Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster.
Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform.
On MacOS, you can do:
```bash theme={"dark"}
brew install kubernetes-helm
```
If you already have helm installed, make sure you're using the latest version:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/)
The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`.
#### Helm User Permissions
If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory.
When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.:
```bash theme={"dark"}
alias k=kubectl
```
To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into:
```bash theme={"dark"}
k create namespace fusion-namespace
```
Apply the `role.yaml` and `cluster-role.yaml` files to that namespace
```bash theme={"dark"}
k apply -f cluster-role.yaml
k config set-context --current --namespace=$NAMESPACE
k apply -f role.yaml
```
Then bind the rolebinding and clusterolebinding to the install user:
```bash wrap theme={"dark"}
k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user
k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user
```
You will then be able to run the `helm install` command as the ``
### Clone fusion-cloud-native from GitHub
You should clone this repo from github as you'll need to run the scripts on your local workstation:
```bash theme={"dark"}
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts.
```bash theme={"dark"}
cd fusion-cloud-native
git pull
```
Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip).
Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory.
The [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_aks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_aks.sh) script provided in this repo is strictly optional.
The script is mainly to help those new to Kubernetes and/or Fusion get started quickly.
If you're already familiar with K8s, Helm, and AKS, then you use Helm directly to install Fusion into an existing cluster or one you create yourself using the process described `<>`.
If you're new to Azure, then please visit [https://azure.microsoft.com/en-us/free/search/](https://azure.microsoft.com/en-us/free/search/) to set up an account.
### Set up the AKS CLI tools
Before launching an AKS cluster, you need to install and configure `kubectl` and `az` using the links provided below:
Required AKS Command-line Tools:
* `kubectl`: [https://kubernetes.io/docs/tasks/tools/install-kubectl/](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
* `az`: [https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest)
To confirm your account access and command-line tools are set up correctly, run the `az login` command (`az login –help` to see available options).
### Azure Prerequisites
To launch a cluster in AKS (or pretty much do anything with Azure) you need to setup a Resource Group. Resource Groups are a way of organizing and managing related resources in Azure.
For more information about resource groups, see [https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview#resource-groups](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview#resource-groups).
You also need to choose a location where you want to spin up your AKS cluster, such as `westus2`. For a list of locations you can choose, see [https://azure.microsoft.com/en-us/global-infrastructure/locations/](https://azure.microsoft.com/en-us/global-infrastructure/locations/).
Use the Azure console in your browser to create a resource group, or simply do:
```bash theme={"dark"}
az group create -g $AZURE_RESOURCE_GROUP -l $AZURE_LOCATION
```
To recap, you should have the following requirements in place:
* Azure Account set up.
* `azure-cli` (`az`) command-line tools installed.
* `az` login working.
* Created an Azure Resource Group and selected a location to launch the cluster.
### Set up Fusion on AKS
Download and run the [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_aks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_aks.sh) to install Fusion 5.x in a AKS cluster. To create a new cluster and install Fusion, simply do:
```bash theme={"dark"}
./setup_f5_aks.sh -c -p
```
If you don't want the script to create a cluster, then you need to create a cluster before running the script and simply pass the name of the existing cluster using the `-c` parameter.
Use the `--help` option to see full script usage.
By default, our script installs Fusion into the default namespace; think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster in separate namespaces. However, please do not install more than one Fusion release in the same namespace.
You can override the namespace using the `-n` option. In addition, our script uses f5 for the Helm release name; you can customize this using the `-r` option. Helm uses the release name you provide to track a specific instance of an installation, allowing you to perform updates and rollback changes for that specific release only.
You can also pass the `--preview` option to the script, which enables soon-to-be-released features for AKS, such as deploying a multi-zone cluster across 3 availability zones for higher availability guarantees. For more information about the Availability Zone feature, see [https://docs.microsoft.com/en-us/azure/aks/availability-zones](https://docs.microsoft.com/en-us/azure/aks/availability-zones).
It takes a while for AKS to spin up the new cluster. The cluster will have three Standard\_D4\_v3 nodes which have 4 CPU cores and 16 GB of memory. Behind the scenes, our script calls the `az aks create` command.
If using Helm V2, the `setup_f5_aks.sh` script installs Helm's `tiller` component into your AKS cluster with the cluster admin role. If you don't want this, then please upgrade to Helm v3.
After running the `setup_f5_aks.sh` script, proceed to `<>`.
#### AKS Ingress
The `setup_f5_aks.sh` script exposes the Fusion proxy service on an external IP over HTTP. This is done for demo or getting started purposes. However, you're strongly encouraged to configure a K8s Ingress with TLS termination in front of the proxy service.
Use the `-t` and `-h ` options to have our script create an Ingress with a TLS certificate issued by Let's Encrypt.
#### Upgrades and Ingress
IMPORTANT: If you used the `-t -h ` options when installing your cluster, our script created an additional values yaml file named `tls-values.yaml`.
To make things easier for you when upgrading, you should add the settings from this file into your main custom values yaml file. For example:
```yaml theme={"dark"}
api-gateway:
service:
type: "NodePort"
ingress:
enabled: true
host: ""
tls:
enabled: true
annotations:
"networking.gke.io/managed-certificates": "-managed-certificate"
"kubernetes.io/ingress.class": "gce"
```
This way, you don't have to remember to pass the additional `tls-values.yaml` file when upgrading.
## Verifying the Fusion Installation
In this section, we provide some tips on how to verify the Fusion installation.
Check if the Fusion Admin UI is available at `\https://:6764/admin/`.
Let's review some useful kubectl commands.
### Enhance the K8s Command-line Experience
Here is a list of tools we found useful for improving your command-line experience with Kubernetes:
* krew (kubectl plugin mgr): [https://github.com/kubernetes-sigs/krew/](https://github.com/kubernetes-sigs/krew/)
* kube-ps1 (show current context on command line prompt): [https://github.com/jonmosco/kube-ps1](https://github.com/jonmosco/kube-ps1)
* kubectx / kubens (switch between clusters / namespaces): [https://github.com/ahmetb/kubectx](https://github.com/ahmetb/kubectx)
### Useful kubectl commands
kubectl reference: [https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands)
Set the namespace for `kubectl` if not using the default:
```bash theme={"dark"}
kubectl config set-context --current --namespace=
```
**This saves you from having to pass `-n` with every command.**
Get a list of running pods: `k get pods`
Get logs for a pod using a label: `k logs –l app.kubernetes.io/component=query-pipeline`
Get pod deployment spec and details: `k get pods -o yaml`
Get details about a pod events: `k describe po `
Port forward to a specific pod: `k port-forward 8983:8983`
SSH into a pod: `k exec -it -- /bin/bash`
CPU/Memory usage report for pods: `k top pods`
Forcefully kill a pod: `k delete po --force --grace-period 0`
Scale up (or down) a deployment: `k scale deployment.v1.apps/ --replicas=N`
Get a list of pod versions: `k get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq`
### Check Fusion Pods and Services
Once the install script completes, you can check that all pods and services are available using:
```bash theme={"dark"}
kubectl get pods
```
If all goes well, you should see a list of pods similar to:
```bash wrap theme={"dark"}
NAME READY STATUS RESTARTS AGE
seldon-controller-manager-6675874894-qxwrv 1/1 Running 0 8m45s
f5-admin-ui-74d794f4f8-m5jms 1/1 Running 0 8m45s
f5-ambassador-fd6b9b5dc-7ghf6 1/1 Running 0 8m43s
f5-api-gateway-6b9998b9c-tmchk 1/1 Running 0 8m45s
f5-auth-ui-7565564b4c-rdc74 1/1 Running 0 8m42s
f5-classic-rest-service-0 1/1 Running 3 8m44s
f5-devops-ui-77bb867ffb-fbzxd 1/1 Running 0 8m42s
f5-fusion-admin-78b8f8fc7f-4d7l8 1/1 Running 0 8m42s
f5-fusion-indexing-599c8d448-xzsvm 1/1 Running 0 8m44s
f5-insights-665fd9f6fc-g5psw 1/1 Running 0 8m43s
f5-job-launcher-84dd4c5c96-p8528 1/1 Running 0 8m44s
f5-job-rest-server-6d44d964b8-xtnxw 1/1 Running 0 8m45s
f5-logstash-0 1/1 Running 0 8m45s
f5-ml-model-service-6987dc94c9-9ppp8 2/2 Running 1 8m45s
f5-monitoring-grafana-5d499dbb58-pzw72 1/1 Running 0 10m
f5-monitoring-prometheus-kube-state-metrics-54d6678dv9h7h 1/1 Running 0 10m
f5-monitoring-prometheus-pushgateway-7d65c65b85-vwrwf 1/1 Running 0 10m
f5-monitoring-prometheus-server-0 2/2 Running 0 10m
f5-pm-ui-86cbc5bb65-nd2n8 1/1 Running 0 8m44s
f5-pulsar-bookkeeper-0 1/1 Running 0 8m45s
f5-pulsar-broker-b56cc776f-56msx 1/1 Running 0 8m45s
f5-query-pipeline-5d75d7d5f4-l2mdf 1/1 Running 0 8m43s
f5-connectors-7bb6cfc65f-7wfs2 1/1 Running 0 8m42s
f5-connectors-backend-987fdc648-dldwv 1/1 Running 0 8m45s
f5-rules-ui-6b9d55b78f-9hzzj 1/1 Running 0 8m43s
f5-solr-0 1/1 Running 0 8m44s
f5-solr-exporter-c4687c785-jsm7x 1/1 Running 0 8m45s
f5-ui-6cdbcc68c6-rj9cq 1/1 Running 0 8m45s
f5-webapps-6d6bb9bfd-hm4qx 1/1 Running 0 8m45s
f5-workflow-controller-7b66679fb7-sjbvp 1/1 Running 0 8m44s
f5-zookeeper-0 1/1 Running 0 8m45s
```
The number of pods per deployment / statefulset will vary based on your cluster size and replicaCount settings in your custom values YAML file.
Also, don't worry if you see some pods having been restarted as that just means they were too slow to come up and Kubernetes killed and restarted them.
You do want to see at least one pod running for every service. If a pod is not running after waiting a sufficient amount of time,
use `kubectl logs ` to see the logs for that pod; to see the logs for previous versions of a pod, use: `kubectl logs -p`.
You can also look at the actions Kubernetes performed on the pod using `kubectl describe po `.
To see a list of Fusion services, do:
```bash theme={"dark"}
kubectl get svc
```
For an overview of the various Fusion 5 microservices, see: [Fusion microservices](/docs/5/fusion/reference/microservices).
Once you're ready to build a Fusion cluster for production, please see see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
### Upgrading with Zero Downtime
One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. Fusion 5 allows customers to upgrade from Fusion 5.x.y to a later 5.x.z version on a live cluster with zero downtime or disruption of service.
When Kubernetes performs a rolling update to an individual microservice, there will be a mix of old and new services in the cluster concurrently (only briefly in most cases) and requests from other services will be routed to both versions. Consequently, Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same 5.x line of releases. We also ensure stored configuration remains compatible in the same 5.x release line.
Lucidworks releases minor updates to individual services frequently, so our customers can pull in those upgrades using Helm at their discretion.
To upgrade your cluster at any time, use the `--upgrade` option with our setup scripts in this repo.
The scripts in this repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.
To see what would be upgraded, you can pass the `--dry-run` option to the script.
### Grafana Dashboards
Get the initial Grafana password from a K8s secret by doing:
```bash wrap theme={"dark"}
kubectl get secret --namespace "${NAMESPACE}" ${RELEASE}-monitoring-grafana \
-o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```
With Grafana, you can either setup a temporary port-forward to a Grafana pod or expose Grafana on an external IP using a K8s LoadBalancer.
To define a LoadBalancer, do (replace \$ with your Helm release label):
```bash wrap theme={"dark"}
kubectl expose deployment ${RELEASE}-monitoring-grafana --type=LoadBalancer --name=grafana --port=3000 --target-port=3000
```
You can use `kubectl get services --namespace ` to determine when the load balancer is setup and its IP address. Direct your browser to `http://:3000` and enter the username `admin@localhost` and the password that was returned in the previous step.
This will log you into the application. It is recommended that you create another administrative user with a more desirable password.
The dashboards and datasoure will be setup for you in grafana, simply navigate to `Dashboards` -> `Manage` to view the available dashboards
## More deployment options
* How to deploy Fusion 5 in Google Kubernetes Engine
* How to deploy Fusion 5 in Amazon Elastic Kubernetes Service
* How to deploy Fusion on 5 other Kubernetes platforms
## Frequently Asked Questions
**Can the stateful database services, for example, MySQL, be supported by an Azure PaaS service?**
This option is not supported. In theory, it may be possible to implement this function.
**Is it possible to use cross-zone storage solutions rather than volumes, such as Microsoft Azure file storage, for stateful services?**
While in theory it may be possible to implement this functionality, the configuration has not been tested by Lucidworks and is not supported.
## Learn more
The course for **Deploying Fusion 5** focuses on the prerequisite software needed to deploy Fusion, the necessary setup steps, and the physical act of deployment.
The [`setup_f5_k8s.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_k8s.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository provides deployment support for any Kubernetes platform, including on-premise, private cloud, public cloud, and hybrid platforms.
This script is used by the [`setup_f5_gke.sh`](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_gke.sh), [`setup_f5_eks.sh`](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_eks.sh), and [`setup_f5_aks.sh`](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_aks.sh) scripts, which provide additional platform-specific support for Google Kubernetes Engine (GKE), Amazon Elastic Kubernetes Service (EKS), and Azure Kubernetes Service (AKS).
This topic explains how to deploy a Fusion cluster in Kubernetes using the [`setup_f5_k8s.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_k8s.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository.
If you’re deploying on-premises or using a localized repository, you’ll need to use a private repository for Docker images.
## Prerequisites
This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes.
### Release Name and Namespace
Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) Kubernetes namespace to install Fusion into.
Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*.
However, please *do not* install more than one Fusion release in the same namespace.
All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_
Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default.
### Install Helm
Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster.
Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform.
On MacOS, you can do:
```bash theme={"dark"}
brew install kubernetes-helm
```
If you already have helm installed, make sure you're using the latest version:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/)
The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`.
#### Helm User Permissions
If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory.
When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.:
```bash theme={"dark"}
alias k=kubectl
```
To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into:
```bash theme={"dark"}
k create namespace fusion-namespace
```
Apply the `role.yaml` and `cluster-role.yaml` files to that namespace
```bash theme={"dark"}
k apply -f cluster-role.yaml
k config set-context --current --namespace=$NAMESPACE
k apply -f role.yaml
```
Then bind the rolebinding and clusterolebinding to the install user:
```bash wrap theme={"dark"}
k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user
k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user
```
You will then be able to run the `helm install` command as the ``
### Clone fusion-cloud-native from GitHub
You should clone this repo from github as you'll need to run the scripts on your local workstation:
```bash theme={"dark"}
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts.
```bash theme={"dark"}
cd fusion-cloud-native
git pull
```
Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip).
Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory.
## Deployment
If you're not running on a managed K8s platform like GKE, AKS, or EKS, you can use Helm to install the Fusion chart to an existing Kubernetes cluster.
Fusion version 5.5 now includes support for the Rancher Kubernetes Engine (RKE) platform. Before deploying Fusion to RKE, you must download and install the [latest RKE software](https://rancher.com/docs/rke/latest/en/). After configuring your cluster, you can proceed with the Helm v3 installation.
You must have a working cluster configured before performing the Helm v3 installation.
### Use Helm v3 to Install Fusion
You should upgrade to the latest version of Helm v3 for working with Fusion. If you need to keep Helm V2 for other clusters,
ensure Helm V3 is ahead of Helm V2 in your working shell's PATH before proceeding.
#### Customize Fusion Chart Settings
Fusion aims to be well-configured out-of-the-box, but you can customize any of the built-in settings using a custom values `YAML` file. If you use one of our setup scripts, such as `setup_f5_gke.sh`, then it will create a custom values YAML file for you the first time you run it using [https://github.com/lucidworks/fusion-cloud-native/blob/master/customize\_fusion\_values.yaml.example](https://github.com/lucidworks/fusion-cloud-native/blob/master/customize_fusion_values.yaml.example) as a template.
If you're working with Helm directly and not using one of our setup scripts, then run the [https://github.com/lucidworks/fusion-cloud-native/blob/master/customize\_fusion\_values.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/customize_fusion_values.sh) script to create a custom values YAML file from our [https://github.com/lucidworks/fusion-cloud-native/blob/master/customize\_fusion\_values.yaml.example](https://github.com/lucidworks/fusion-cloud-native/blob/master/customize_fusion_values.yaml.example) template as a starting point:
```bash theme={"dark"}
./customize_fusion_values.sh -c -n \
--provider --num-solr 1 --node-pool ""
```
Pass `--help` for usage details.
In this example:
* `` is the K8s platform you're running on, such as `gke`
* `` is the name of your cluster
* `` is the K8s namespace where you plan to install Fusion
The `--node-pool` option specifies the node selector label for determining which nodes to run Fusion pods. You can pass `"{}"` to let Kubernetes decide which nodes to schedule pods on.
This file is referred to as `${MY_VALUES}` in the commands belo. Replace the filename with the correct filename for your environment. Keep this file handy, as you'll need it to customize Fusion settings and upgrade to a newer version.
Review the settings in the custom values YAML file to ensure the defaults are appropriate for your environment, including the number of Solr and Zookeeper replicas.
Add the Lucidworks Helm repo:
```bash theme={"dark"}
helm repo add lucidworks https://charts.lucidworks.com
```
The `customize_fusion_values.sh` script creates an upgrade script to install/upgrade Fusion into Kubernetes using Helm. Look in the directory where you ran `customize_fusion_values.sh` for a script named like:
`___upgrade_fusion.sh`. Run this script to install Fusion.
## Verifying the Fusion Installation
In this section, we provide some tips on how to verify the Fusion installation.
Check if the Fusion Admin UI is available at `\https://:6764/admin/`.
Let's review some useful kubectl commands.
### Enhance the K8s Command-line Experience
Here is a list of tools we found useful for improving your command-line experience with Kubernetes:
* krew (kubectl plugin mgr): [https://github.com/kubernetes-sigs/krew/](https://github.com/kubernetes-sigs/krew/)
* kube-ps1 (show current context on command line prompt): [https://github.com/jonmosco/kube-ps1](https://github.com/jonmosco/kube-ps1)
* kubectx / kubens (switch between clusters / namespaces): [https://github.com/ahmetb/kubectx](https://github.com/ahmetb/kubectx)
### Useful kubectl commands
kubectl reference: [https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands)
Set the namespace for `kubectl` if not using the default:
```bash theme={"dark"}
kubectl config set-context --current --namespace=
```
**This saves you from having to pass `-n` with every command.**
Get a list of running pods: `k get pods`
Get logs for a pod using a label: `k logs –l app.kubernetes.io/component=query-pipeline`
Get pod deployment spec and details: `k get pods -o yaml`
Get details about a pod events: `k describe po `
Port forward to a specific pod: `k port-forward 8983:8983`
SSH into a pod: `k exec -it -- /bin/bash`
CPU/Memory usage report for pods: `k top pods`
Forcefully kill a pod: `k delete po --force --grace-period 0`
Scale up (or down) a deployment: `k scale deployment.v1.apps/ --replicas=N`
Get a list of pod versions: `k get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq`
### Check Fusion Pods and Services
Once the install script completes, you can check that all pods and services are available using:
```bash theme={"dark"}
kubectl get pods
```
If all goes well, you should see a list of pods similar to:
```bash wrap theme={"dark"}
NAME READY STATUS RESTARTS AGE
seldon-controller-manager-6675874894-qxwrv 1/1 Running 0 8m45s
f5-admin-ui-74d794f4f8-m5jms 1/1 Running 0 8m45s
f5-ambassador-fd6b9b5dc-7ghf6 1/1 Running 0 8m43s
f5-api-gateway-6b9998b9c-tmchk 1/1 Running 0 8m45s
f5-auth-ui-7565564b4c-rdc74 1/1 Running 0 8m42s
f5-classic-rest-service-0 1/1 Running 3 8m44s
f5-devops-ui-77bb867ffb-fbzxd 1/1 Running 0 8m42s
f5-fusion-admin-78b8f8fc7f-4d7l8 1/1 Running 0 8m42s
f5-fusion-indexing-599c8d448-xzsvm 1/1 Running 0 8m44s
f5-insights-665fd9f6fc-g5psw 1/1 Running 0 8m43s
f5-job-launcher-84dd4c5c96-p8528 1/1 Running 0 8m44s
f5-job-rest-server-6d44d964b8-xtnxw 1/1 Running 0 8m45s
f5-logstash-0 1/1 Running 0 8m45s
f5-ml-model-service-6987dc94c9-9ppp8 2/2 Running 1 8m45s
f5-monitoring-grafana-5d499dbb58-pzw72 1/1 Running 0 10m
f5-monitoring-prometheus-kube-state-metrics-54d6678dv9h7h 1/1 Running 0 10m
f5-monitoring-prometheus-pushgateway-7d65c65b85-vwrwf 1/1 Running 0 10m
f5-monitoring-prometheus-server-0 2/2 Running 0 10m
f5-pm-ui-86cbc5bb65-nd2n8 1/1 Running 0 8m44s
f5-pulsar-bookkeeper-0 1/1 Running 0 8m45s
f5-pulsar-broker-b56cc776f-56msx 1/1 Running 0 8m45s
f5-query-pipeline-5d75d7d5f4-l2mdf 1/1 Running 0 8m43s
f5-connectors-7bb6cfc65f-7wfs2 1/1 Running 0 8m42s
f5-connectors-backend-987fdc648-dldwv 1/1 Running 0 8m45s
f5-rules-ui-6b9d55b78f-9hzzj 1/1 Running 0 8m43s
f5-solr-0 1/1 Running 0 8m44s
f5-solr-exporter-c4687c785-jsm7x 1/1 Running 0 8m45s
f5-ui-6cdbcc68c6-rj9cq 1/1 Running 0 8m45s
f5-webapps-6d6bb9bfd-hm4qx 1/1 Running 0 8m45s
f5-workflow-controller-7b66679fb7-sjbvp 1/1 Running 0 8m44s
f5-zookeeper-0 1/1 Running 0 8m45s
```
The number of pods per deployment / statefulset will vary based on your cluster size and replicaCount settings in your custom values YAML file.
Also, don't worry if you see some pods having been restarted as that just means they were too slow to come up and Kubernetes killed and restarted them.
You do want to see at least one pod running for every service. If a pod is not running after waiting a sufficient amount of time,
use `kubectl logs ` to see the logs for that pod; to see the logs for previous versions of a pod, use: `kubectl logs -p`.
You can also look at the actions Kubernetes performed on the pod using `kubectl describe po `.
To see a list of Fusion services, do:
```bash theme={"dark"}
kubectl get svc
```
For an overview of the various Fusion 5 microservices, see: [Fusion microservices](/docs/5/fusion/reference/microservices).
Once you're ready to build a Fusion cluster for production, please see see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
### Upgrading with Zero Downtime
One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. Fusion 5 allows customers to upgrade from Fusion 5.x.y to a later 5.x.z version on a live cluster with zero downtime or disruption of service.
When Kubernetes performs a rolling update to an individual microservice, there will be a mix of old and new services in the cluster concurrently (only briefly in most cases) and requests from other services will be routed to both versions. Consequently, Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same 5.x line of releases. We also ensure stored configuration remains compatible in the same 5.x release line.
Lucidworks releases minor updates to individual services frequently, so our customers can pull in those upgrades using Helm at their discretion.
To upgrade your cluster at any time, use the `--upgrade` option with our setup scripts in this repo.
The scripts in this repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.
To see what would be upgraded, you can pass the `--dry-run` option to the script.
### Grafana Dashboards
Get the initial Grafana password from a K8s secret by doing:
```bash wrap theme={"dark"}
kubectl get secret --namespace "${NAMESPACE}" ${RELEASE}-monitoring-grafana \
-o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```
With Grafana, you can either setup a temporary port-forward to a Grafana pod or expose Grafana on an external IP using a K8s LoadBalancer.
To define a LoadBalancer, do (replace \$ with your Helm release label):
```bash wrap theme={"dark"}
kubectl expose deployment ${RELEASE}-monitoring-grafana --type=LoadBalancer --name=grafana --port=3000 --target-port=3000
```
You can use `kubectl get services --namespace ` to determine when the load balancer is setup and its IP address. Direct your browser to `http://:3000` and enter the username `admin@localhost` and the password that was returned in the previous step.
This will log you into the application. It is recommended that you create another administrative user with a more desirable password.
The dashboards and datasoure will be setup for you in grafana, simply navigate to `Dashboards` -> `Manage` to view the available dashboards
## More deployment options
* How to deploy Fusion 5 in Google Kubernetes Engine
* How to deploy Fusion 5 in Amazon Elastic Kubernetes Service
* How to deploy Fusion 5 in Azure Kubernetes Service
## Learn more
The course for **Deploying Fusion 5** focuses on the prerequisite software needed to deploy Fusion, the necessary setup steps, and the physical act of deployment.
Before you begin, see [Fusion Server Deployment](/docs/5/fusion/operations/deployment) to understand the architecture and requirements.
This article explains how to plan and execute a Fusion deployment at the scale required for staging or production.
While the `setup_f5_*.sh` scripts are handy for getting started and proof-of-concept purposes, this article covers the planning process for building a production-ready environment.
The course for **Preparing for Fusion Implementation** focuses on the key elements for a successful implementation, defining your business requirements, preparing clean data, and involving the right personnel.
## Prerequisites
You must meet the following prerequisites before you can customize your Fusion cluster:
* A local copy of the [fusion-cloud-native repository](https://github.com/lucidworks/fusion-cloud-native). This must be up-to-date with the latest master branch.
* Any cloud provider-specific **command line tools**, such as `gcloud` or `aws`, and `kubectl`.\
See the platform-specific instructions linked above, or check with your cloud provider.
* Helm v3
* To install on a Mac:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
* For other operating systems, download from [Helm Releases](https://github.com/helm/helm/releases).
* Verify your installation:
```bash theme={"dark"}
helm version --short
v3.0.0+ge29ce2a
```
* Kubernetes namespace
* Collect the following information about your Kubernetes environment:
* **CLUSTER**: Cluster name (passed to our setup scripts using the `-c` arg)
* **NAMESPACE**: Kubernetes namespace where to install Fusion; a namespace should only contain lowercase letters (a-z), digits (0-9), or dash. No periods or underscores allowed.
* *(optional)* Clarify your organization’s DockerHub policy. The Fusion Helm chart points to public Docker images on DockerHub. Your organization may not allow Kubernetes to pull images directly from DockerHub or may require extra security scanning before loading images into production clusters.\
Consult your Kubernetes and Docker admin team to find how to get the Fusion images loaded into a registry that’s accessible to your cluster. You can update the image for each service using the [custom values YAML file](#custom-values-yaml-file).
**Kubernetes namespace tips**
* Fusion 5 service discovery requires all services for the same release be deployed in the same namespace. Moreover, you should only run one instance of Fusion in a namespace. If you need multiple instances of Fusion running in the same Kubernetes cluster, then you need to deploy them in separate namespaces.
* If your organization requires CPU / Memory quotas for namespaces, you can start with a minimum of 12 CPU and 45GB of RAM (such as 3 x n1-standard-4 on GKE), but you will need to increase the quotas once you start load testing Fusion with production workloads and real datasets.
* Fusion requires at least 3 ZooKeeper nodes and 2 Solr nodes to achieve high availability.
## Custom values YAML file
1. Clone the `fusion-cloud-native` repository: `git clone https://github.com/lucidworks/fusion-cloud-native`
2. Run the `customize_fusion_values.sh` script.
```bash theme={"dark"}
./customize_fusion_values.sh --provider -c -n \
--num-solr 3 \
--solr-disk-gb 100 \
--node-pool \
--prometheus true \
--with-resource-limits \
--with-affinity-rules
```
Pass the `--help` parameter to see script usage details.
The script creates the following files:
| File | Description |
| - | - |
| `___fusion_values.yaml` | Main custom values YAML used to override Helm chart defaults for Fusion microservices. |
| `___monitoring_values.yaml` | Custom values yaml used to configure Prometheus and Grafana. |
| `___fusion_resources.yaml` | Resource requests and limits for all Microservices. |
| `___fusion_affinity.yaml` | Pod affinity rules to ensure multiple replicas for a single service are evenly distributed across zones and nodes. |
| `___upgrade_fusion.sh` | Script used to install and/or upgrade Fusion using the aforementioned custom values YAML files. |
For an explanation of these placeholder values, see [Configuration Values](#custom-values-yaml-file) below.
3. Add the new files to version control. You will make changes to it over time as you fine-tune your Fusion installation. You will also need it to perform upgrades. If you try to upgrade your Fusion installation and don’t provide the custom values YAML, your deployment will revert to chart defaults.\
Review the `___fusion_values.yaml` file to familiarize yourself with its structure and contents. Notice it contains a separate section for each of the Fusion microservices. The example configuration of the `query-pipeline` service below illustrates some important concepts about the custom values YAML file.
```yaml highlight={1,2,3,5,6} theme={"dark"}
query-pipeline:
enabled: true
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
javaToolOptions: "..."
pod:
annotations:
prometheus.io/port: "8787"
prometheus.io/scrape: "true"
prometheus.io/path: "/actuator/prometheus"
```
4. Service-specific setting overrides under the top-level heading
5. Every Fusion service has an implicit enabled flag that defaults to true, set to false to remove this service from your cluster
6. Node selector identifies the label find nodes to schedule pods on
7. Used to pass JVM options to the service
8. Pod annotations to allow Prometheus to scrape metrics from the service
Once we go through all of the configuration topics in this topic, you’ll have a well-configured custom values YAML file for your Fusion 5 installation. You’ll then use this file during the Helm v3 installation at the end of this topic.
### Deployment-specific values
The script creates a custom values YAML file using the naming convention: `___fusion_values.yaml`. For example, `gke_search_f5_fusion_values.yaml`.
| Parameter | Description |
| - | - |
| `` | The K8s platform you’re running on, such as `gke`. |
| `` | The name of your cluster. |
| `` | The K8s namespace where you want to install Fusion. |
| `` | Specifies a `nodeSelector` label to find nodes to schedule Fusion pods on. |
Providing the correct `--node-pool ` label is very important. Using the wrong value will cause your pods to be stuck in the `pending` state. If you’re not sure about the correct value for your cluster, pass ’'\` to let Kubernetes decide which nodes to schedule Fusion pods on.
Default `nodeSelector` labels are provider-specific. The `fusion-cloud-native` scripts use the following defaults for GKE and EKS:
| Provider | Default node selector |
| - | - |
| GKE | cloud.google.com/gke-nodepool: default-pool |
| EKS | alpha.eksctl.io/nodegroup-name: standard-workers |
If you are deploying Fusion 5.9.12, add the following to your `values.yaml` file to avoid a known issue that prevents the `kuberay-operator` pod from launching successfully: `yaml kuberay-operator: crd: create: true `
### Flags
The script provides flags for additional configuration:
| Flag | Description |
| - | - |
| `--node-pool` | Add a Fusion specific label to your nodes. |
| `--with-resource-limits` | Configure resource requests/limits. |
| `--with-replicas` | Configure replica counts. |
| `--with-affinity-rules` | Configure pod affinity rules for Fusion services. |
Use `--node-pool` to add a Fusion specific label to your nodes by doing:
```bash theme={"dark"}
kubectl label fusion_node_type=
```
Then, pass `--node-pool 'fusion_node_type: '`.
## Configure Solr sizing
When you’re ready to build a production-ready setup for Fusion 5, you need to customize the Fusion Helm chart to ensure Fusion is well-configured for production workloads.
You’ll be able to scale the number of nodes for Solr up and down after building the cluster, but you need to establish the initial size of the nodes (memory and CPU) and the size and type of disks you need.
See the example config below to learn which parameters to change in the custom values YAML file.
```yaml expandable theme={"dark"}
solr:
resources: # Set resource limits for Solr to help K8s pod scheduling;
limits: # these limits are not just for the Solr process in the pod,
cpu: "7700m" # so allow ample memory for loading index files into the OS cache (mmap)
memory: "26Gi"
requests:
cpu: "7000m"
memory: "25Gi"
logLevel: WARN
nodeSelector:
fusion_node_type: search # Run this Solr StatefulSet in the "search" node pool
exporter:
enabled: true # Enable the Solr metrics exporter (for Prometheus) and
# schedule on the default node pool (system partition)
podAnnotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9983"
prometheus.io/path: "/metrics"
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
image:
tag: 8.4.1
updateStrategy:
type: "RollingUpdate"
javaMem: "-Xmx3g -Dfusion_node_type=system" # Configure memory settings for Solr
solrGcTune: "-XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UseStringDeduplication -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=150 -XX:+UseLargePages -XX:+AlwaysPreTouch"
volumeClaimTemplates:
storageSize: "100Gi" # Size of the Solr disk
replicaCount: 6 # Number of Solr pods to run in this StatefulSet
zookeeper:
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
replicaCount: 3 # Number of Zookeepers
persistence:
size: 20Gi
resources: {}
env:
ZK_HEAP_SIZE: 1G
ZOO_AUTOPURGE_PURGEINTERVAL: 1
```
To be clear, you can tune GC settings and number of replicas after the cluster is built. But changing the size of the persistent volumes is more complicated so you should try to pick a good size initially.
### Configure storage class for Solr pods (optional)
If you wish to run with a storage class other than the default you can create a storage class for your Solr pods before you install. For example, to create regional disks in GCP you can create a file called `storageClass.yaml` with the following contents:
```yaml theme={"dark"}
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: solr-gke-storage-regional
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-standard
replication-type: regional-pd
zones: us-west1-b, us-west1-c
```
and then provision into your cluster by calling:
```bash theme={"dark"}
kubectl apply -f storageClass.yaml
```
to then have Solr use the storage class by adding the following to the custom values YAML:
```yaml theme={"dark"}
solr:
volumeClaimTemplates:
storageClassName: solr-gke-storage-regional
storageSize: 250Gi
```
We’re not advocating that you must use regional disks for Solr storage, as that would be redundant with Solr replication. We’re just using this as an example of how to configure a custom storage class for Solr disks if you see the need. For instance, you could use regional disks without Solr replication for write-heavy type collections.
## Configure multiple node pools
Lucidworks recommends isolating search workloads from analytics workloads using multiple node pools. The included scripts do not do this for you; this is a manual process.
See the example script for GKE, see [create\_gke\_cluster\_node\_pools.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/additional_environments/create_gke_cluster_node_pools.sh).
In the custom values YAML file, you can add additional Solr StatefulSets by adding their names to the list under the `nodePools` property. If any property for that statefulset needs to be changed from the default set of values, then it can be set directly on the object representing the node pool, any properties that are omitted are defaulted to the base value. See the following example (additional whitespace added for display purposes only):
```yaml expandable highlight={3,4,9,17,22,31} theme={"dark"}
solr:
nodePools:
- name: ""
- name: "analytics"
javaMem: "-Xmx6g"
replicaCount: 6
storageSize: "100Gi"
nodeSelector:
fusion_node_type: analytics
resources:
requests:
cpu: 2
memory: 12Gi
limits:
cpu: 3
memory: 12Gi
- name: "search"
javaMem: "-Xms11g -Xmx11g"
replicaCount: 12
storageSize: "50Gi"
nodeSelector:
fusion_node_type: search
resources:
limits:
cpu: "7700m"
memory: "26Gi"
requests:
cpu: "7000m"
memory: "25Gi"
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
...
```
1. The empty string `""` is the suffix for the default partition.
2. Overrides the settings for the **analytics** Solr pods.
3. Assigns the **analytics** Solr pods to the node pool and attaches the label `fusion_node_type=analytics`. You can use the `fusion_node_type` property in Solr auto-scaling policies to govern replica placement during collection creation.
4. Overrides the settings for the **search** Solr pods.
5. Assigns the **search** Solr pods to the node pool and attaches the label `fusion_node_type=search`.
6. Sets the default settings for all Solr pods, if not specifically overridden in the `nodePools` section above.
Do not edit the `nodePools` value `""`.
In the example above, the **analytics** partition `replicaCount`, or number of Solr pods, is six. The **search** partition `replicaCount` is twelve.
Each nodePool is automatically be assigned the -Dfusion\_node\_type property of ``, ``, or ``. This value matches the name of the nodePool. For example, `-Dfusion_node_type=`.
The Solr pods have a `fusion_node_type` system property, as shown below:
## Solr auto-scaling policy
Use [replica placement plugins](https://solr.apache.org/guide/solr/latest/configuration-guide/replica-placement-plugins.html) to control how replicas are placed in Solr.
## Pod network policy
A Kubernetes network policy governs how groups of pods communicate with each other and other network endpoints. With Fusion, all incoming traffic flows through the API Gateway service. All Fusion services in the same namespace expect an internal JWT, which is supplied by the Gateway, as part of the request. As a result, Fusion services enforce a basic level of API security and don’t need an additional network policy to protect them from other pods in the cluster.
To install the network policy for Fusion services, pass `--set global.networkPolicyEnabled=true` when installing the Fusion Helm chart.
## On-premises private Docker registries
For on-premises Kubernetes deployments, your organization may not allow Kubernetes to pull [Fusion’s Docker images from DockerHub](https://hub.docker.com/u/lucidworks/). See the instructions below for details on using a private Docker registry with Fusion. These are general instructions that may need to be adapted to work within your organization’s security policies:
1. Transfer the public images from DockerHub to your private Docker registry.
2. Establish a workstation that has access to [DockerHub](https://hub.docker.com). This workstation must connect to your internal Docker registry, most likely via VPN connection. In this example, the workstation is referred to as `envoy`.
3. Install Docker on `envoy`. You need at least 100GB of free disk for Docker.
4. Pull all of the images from DockerHub to `envoy`’s local registry. For example, to pull the query pipeline image, run `docker pull lucidworks/query-pipeline:5.9.0`. See `docker pull --help` for more information about pulling Docker images.
5. Establish a connection from `envoy` to the private Docker registry, most likely via a VPN connection. In this example, the private Docker registry is referred to as ``.
6. Push the images from `envoy`’s Docker registry to the private registry. This will take a long time.
1. You’ll need to re-tag all images for the internal registry. For example, to tag the query-pipeline image, run:
```bash theme={"dark"}
docker tag lucidworks/query-pipeline:5.9.0 /query-pipeline:5.9.0
```
2. Push each image to the internal repo:
```bash theme={"dark"}
docker push /query-pipeline:5.9.0
```
7. Install the Docker registry secret in Kubernetes. Create the Docker registry secret in the Kubernetes namespace where you want to install Fusion:
```bash theme={"dark"}
SECRET_NAME=
REPO=
kubectl create secret docker-registry "${SECRET_NAME}" \
--namespace "${NAMESPACE}" \
--docker-server="${REPO}" \
--docker-username=${REPO_USER} \
--docker-password=${REPO_PASS} \
--docker-email=${REPO_USER}
```
For details, see the Kubernetes article [Pull an Image from a Private Registry](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/).
8. Update the custom values YAML for your cluster to point to your private registry and secret to allow Kubernetes to pull images. For example:
```yaml theme={"dark"}
query-pipeline:
image:
imagePullSecrets:
- name:
repository:
```
Repeat the process for all Fusion services.
### Customize Helm Chart
Every Fusion service allows you to override the `imagePullSecrets` setting using custom values YAML. However, other 3rd party services--including Zookeeper, Pulsar, Prometheus, and Grafana--don’t allow you to supply the pull secret using the custom values YAML.
To patch the default service account for your namespace and add the pull secret, run the following:
```bash theme={"dark"}
kubectl patch sa default -n $NAMESPACE \
-p '"imagePullSecrets": [{"name": "" }]'
```
In Windows using PowerShell or another CLI, you might have to escape the double quotes with a backslash (`\`) or reverse the order of single and double quotes:
```bash theme={"dark"}
kubectl patch sa default -n $NAMESPACE \
-p "'imagePullSecrets': [{'name': ''}]"
```
Replace `` with the name of the secret you created in the steps above.
This allows the default service account to pull images from the private registry without specifying the pull secret on the resources directly.
## Add additional trusted certificate(s) to Fusion’s indexing and querying services *(optional)*
You can add custom trusted certificates to support Fusion’s indexing and querying services. You may want to use custom trusted certificates if, for example, you have specific security requirements for data handling or need to support an existing infrastructure and its security needs. This method involves updating your Helm chart.
If you want to add custom trusted certificates for both the indexing and querying services, follow these instructions twice: once for the indexing service, and once for the querying service. To add different certificates to the indexing and querying services, create one YAML file with the indexing service certificates and one YAML file for the querying service certificates before following these instructions.
You may use the same YAML file if you want to use the same certificates for both services.
To add custom trusted certificates:
1. Create a new YAML file for your custom trusted certificates. The `customcerts.yaml` file is the example file in these instructions.
2. Add the custom certificate(s) in the YAML file created in the previous step. For example:
```yaml theme={"dark"}
trustedCertificates:
enabled: true
files:
some.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE-----
other.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE---------
```
3. Update the indexing or querying service by running the following Helm command. Replace `EXAMPLE-VALUES-FILE.yaml` with your previous values file.
```bash theme={"dark"}
helm upgrade --install --namespace ${EXAMPLE-NAMESPACE} ${HELM-RELEASE} ${HELM-CHART-PATH} --values EXAMPLE-VALUES-FILE.yaml --values customcerts.yaml
```
4. Verify the indexing or querying pod has a new `init-container` with the name `import-certs`.
## Add additional trusted certificate(s) for connectors to allow crawling of web resources with SSL/TLS enabled *(optional)*
To crawl a datasource which for some reason is using a self-signed certificate, add arbitrary certificates to connectors. For example:
```yaml wrap expandable theme={"dark"}
classic-rest-service:
trustedCertificates:
enabled: true
files:
some.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE-----
other.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE---------
connector-plugin:
trustedCertificates:
enabled: true
files:
some.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE-----
other.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE---------
```
### Generating the certificate on linux command line
Use the following command to generate a `.crt` file in `$fusion_home/apps/jetty/connectors/etc/yourcertname.crt`:
```bash theme={"dark"}
openssl s_client -servername remote.server.net -connect remote.server.net:443 $fusion_home/apps/jetty/connectors/etc/yourcertname.crt
```
### Generating the certificate using Firefox web browser
1. Navigate to the SharePoint host.
2. Click the in the address bar, then click the icon.
3. Next, navigate to **More Information** > **View Certificate** > **Export**.\
Save the file to the following folder:
`$fusion_home/apps/jetty/connectors/etc/yourcertname.crt`
### Generating the certificate using Chrome web browser
1. Navigate to **Chrome menu** > **More Tools** > **Developer Tools** > **Security Tab**.
This will display the **Security overview**.
2. Click the **View certificate** button.
3. Save the file to the following folder:
`$fusion_home/apps/jetty/connectors/etc/yourcertname.crt`
### Generating the certificate using powershell
Use the following script to generate a `.crt`` file in `\$fusion\_home\apps\jetty\connectors\etc\yourcertname.crt\`\`:
```bash theme={"dark"}
$fusion_home = c:\your\fusion\install\directory
$webRequest = [Net.WebRequest]::Create("https://your-hostname")
try { $webRequest.GetResponse() } catch {}
$cert = $webRequest.ServicePoint.Certificate
$bytes = $cert.Export([Security.Cryptography.X509Certificates.X509ContentType]::Cert)
set-content -value $bytes -encoding byte -path "$fusion_home\apps\jetty\connectors\etc\yourcertname.binary.crt"
certutil -encode "$fusion_home\apps\jetty\connectors\etc\yourcertname.binary.crt" "$fusion_home\apps\jetty\connectors\etc\yourcertname.crt"
rm "$fusion_home\apps\jetty\connectors\etc\yourcertname.binary.crt" -f
```
## Install Fusion 5 on Kubernetes
At this point, you’re ready to install Fusion 5 using the custom values YAML files and upgrade script. If you used the `customize_fusion_values.sh` script, run it using BASH:
```bash theme={"dark"}
./gke_search_f5_upgrade_fusion.sh
```
Once the installation is complete, verify your Fusion installation is running correctly.
## Monitoring Fusion with Prometheus and Grafana
Lucidworks recommends using Prometheus and Grafana for monitoring the performance and health of your Fusion cluster. Your operations team may already have these services installed. If not, install them into the Fusion namespace.
The [Custom values YAML file shown above](#custom-values-yaml-file) activates the Solr metrics exporter service and adds pod annotations so Prometheus can scrape metrics from Fusion services.
1. Run the `customize_fusion_values.sh` script with the `--prometheus true` option. This creates an extra custom values YAML file for installing Prometheus and Grafana, `___monitoring_values.yaml`. For example: `gke_search_f5_monitoring_values.yaml`.
2. Commit the YAML file to version control.
3. Review its contents to ensure that the settings suit your needs. For example, decide how long you want to keep metrics. The default is 36 hours.\
See the [Prometheus documentation](https://github.com/helm/charts/tree/master/stable/prometheus) and [Grafana documentation](https://github.com/helm/charts/tree/master/stable/grafana) for details.
4. Run the `install_prom.sh` script to install Prometheus & Grafana in your namespace. Include the provider, cluster name, namespace, and helm release as in the example below:
```bash theme={"dark"}
./install_prom.sh --provider gke -c search -n f5 -r 5-5-1
```
Pass the `--help` parameter to see script usage details.
The Grafana dashboards from [monitoring/grafana](https://github.com/lucidworks/fusion-cloud-native/tree/master/monitoring/grafana) are installed automatically by the `install_prom.sh` script.
## Upgrade Fusion on Kubernetes
Upgrade information about Kubernetes is included in the **Fusion 5 Upgrades** topic.
This guide describes how to perform Fusion 5 upgrades.
Before upgrading, be aware of changes by checking for [Deprecations and Removals](/docs/5/fusion/deprecations-and-removals) between versions.
Lucidworks recommends upgrading to the next minor version only. For example, you should upgrade from Fusion 5.6.1 to Fusion 5.7.1 before upgrading to Fusion 5.8.0.
The [general upgrade process](#general-upgrade-process) is described in this article. However, the specific upgrade procedures may vary depending on your upgrade path. For the most accurate instructions, please refer to the upgrade article specific to your upgrade.
## General upgrade process
Fusion natively supports deployments on supported Kubernetes platforms, including AKS, EKS, and GKE.
Fusion includes an upgrade script for AKS, EKS, and GKE. This script is not generated for other Kubernetes deployments.
Upgrades differ from platform to platform. See below for more information about upgrading on your platform of choice.
Whenever you upgrade Fusion, you must also update your [remote connectors](/docs/fusion-connectors/developers/remote-v2-connectors), if you are running any.
You can download the latest files at [V2 Connectors Downloads](/docs/fusion-connectors/downloads/v2-connectors-downloads).
### Natively supported deployment upgrades
| Deployment type | Platform |
| - | - |
| **Azure Kubernetes Service (AKS)** | `aks` |
| **Amazon Elastic Kubernetes Service (EKS)** | `eks` |
| **Google Kubernetes Engine (GKE)** | `gke` |
Fusion includes upgrade scripts for natively supported deployment types. To upgrade:
1. Open the `___upgrade_fusion.sh` upgrade script file for editing.
2. Update the `CHART_VERSION` to your target Fusion version, and save your changes.
3. Run the `___upgrade_fusion.sh` script. The `` value is the same as your namespace, unless you overrode the default value using the `-r` option.
After running the upgrade, use `kubectl get pods` to see the changes applied to your cluster. It may take several minutes to perform the upgrade, as new Docker images are pulled from DockerHub. To see the versions of running pods, do:
```
kubectl get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq
```
### Other Kubernetes deployment upgrades
To update an existing installation, do:
```
RELEASE=f5
NAMESPACE=default
helm repo update
helm upgrade ${RELEASE} "lucidworks/fusion" --namespace "${NAMESPACE}" --values "${MY_VALUES}"
```
Except for ZooKeeper, all K8s deployments and statefulsets use a `RollingUpdate` update policy:
```yaml theme={"dark"}
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
```
ZooKeeper instances use `OnDelete` to avoid changing critical stateful pods in the Fusion deployment. To apply changes to Zookeeper after performing the upgrade (uncommon), you need to manually delete the pods. For example:
```
kubectl delete pod f5-zookeeper-0
```
Delete one pod at a time. Verify the new pod is healthy and serving traffic, before deleting the next healthy pod.
You can also set the `updateStrategy` under the `zookeeper` section in your `"${MY_VALUES}"` file:
```yaml theme={"dark"}
solr:
...
zookeeper:
updateStrategy:
type: "RollingUpdate"
```
#### Upgrades with Helm v3
One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. For example, Fusion 5 allows customers to upgrade from Fusion 5.1.0 to a later 5.x.y version on a live cluster with zero downtime or disruption of service.
When Kubernetes performs a rolling update to an individual microservice, there is a mix of old and new services in the cluster. Requests from other services route to both versions.
Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same minor release version (5.x). We also ensure that the stored configuration remains compatible in the same minor release version.
Lucidworks releases minor updates to individual services frequently. Pull in those upgrades using Helm at your discretion.
**How to upgrade Fusion**
1. Clone the [**fusion-cloud-native** repo](https://github.com/lucidworks/fusion-cloud-native), if you haven’t already.
2. Locate the `setup_f5_.sh` script that matches your Kubernetes platform.
3. Run the script with the `--upgrade` option.
To see what would be upgraded, pass the `--dry-run` option to the script.
The scripts in the **fusion-cloud-native** repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.
#### Helm upgrade script
Once you deploy a working cluster, use the upgrade script created by the `customize_fusion_values.sh` script. The upgrade script hard-codes the parameters and eases the need to remember which parameters to pass to the script. This is helpful when working with multiple K8s clusters. Make sure you check the script into version control alongside your custom values YAML files.
Whenever you change the custom values YAML files for your cluster, you need to run the upgrade script to apply the changes. The script calls `helm upgrade` with the correct parameters and `--values` options.
If you run `helm upgrade` without passing the custom values YAML files, the deployment will revert to using chart defaults, which you never want to do.The script assumes your `kubeconfig` is pointing to the correct cluster and you’re using Heml v3. If not, the upgrade fails. Select the correct `kubeconfig` before running the script.
# Machine Learning Models in Fusion
Source: https://doc.lucidworks.com/docs/5/fusion/intro/machine-learning/machine-learning-models
[localhost link]: http://localhost:3000/docs/5/fusion/intro/machine-learning/machine-learning-models
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/intro/machine-learning/machine-learning-models
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/492
For information about machine learning models, see **Develop and Deploy a Machine Learning Model**.
This tutorial walks you through deploying your own model to Fusion with Seldon Core.
## Prerequisites
* A Fusion instance with an app and indexed data
* An understanding of Python and the ability to write Python code
* [Docker](https://docs.docker.com/get-docker/) installed locally, plus a private or public Docker repository
* Seldon-core installed locally: `pip install seldon-core`
* Code editor; you can use any editor, but Visual Studio Code is used in the example
* Model: [paraphrase-multilingual-MiniLM-L12-v2 from Hugging Face](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2)
* Docker image: [example\_sbert\_model](https://hub.docker.com/r/jstrmec/example_sbert_model/)
## Tips
* Always test your Python code locally before uploading to Docker and then Fusion.
This simplifies troubleshooting significantly.
* Once you’ve created your Docker you can also test locally by doing `docker run` with a specified port, like 9000, which you can then `curl` to confirm functionality in Fusion.
See the testing example below.
The course for **Intro to Machine Learning in Fusion** focuses on using machine learning to infer the goals of customers and users in order to deliver a more sophisticated search experience.
## Local testing example
The examples in this section use the following models:
* [multilingual miniLM model](https://colab.research.google.com/drive/1LeCGl57_Evn_PCL25NJAt1MzP6GYH4j2)
* [e5-model](https://colab.research.google.com/drive/15WOHqqMBng5abUsWJYdElxgVLdAoJdn2)
1. Docker command:
```bash theme={"dark"}
docker run -p 127.0.0.1:9000:9000
```
2. Curl to hit Docker:
```bash theme={"dark"}
curl -X POST -H 'Content-Type: application/json' -d '{"data": { "ndarray": ["Sentence to test"], "names":["text"]} }' https://localhost:9000/api/v1.0/predictions
```
3. Curl model in Fusion:
```bash theme={"dark"}
curl -u $FUSION_USER:$FUSION_PASSWORD -X POST -H 'Content-Type: application/json' -d '{"text": "i love fusion"}' https://.lucidworks.com:6764/api/ai/ml-models//prediction
```
4. See all your deployed models:
```bash theme={"dark"}
curl -u USERNAME:PASSWORD http://FUSION_HOST:FUSION_PORT/api/ai/ml-models
```
## Download the model
This tutorial uses the [paraphrase-multilingual-MiniLM-L12-v2](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2) model from Hugging Face, but any pre-trained model from [https://huggingface.co](https://huggingface.co) will work with this tutorial.
If you want to use your own model instead, you can do so, but your model must have been trained and then saved though a function similar to the PyTorch’s `torch.save(model, PATH)` function.
See [Saving and Loading Models](https://pytorch.org/tutorials/beginner/saving_loading_models.html) in the PyTorch documentation.
## Format a Python class
The next step is to format a Python class which will be invoked by Fusion to get the results from your model.
The skeleton below represents the format that you should follow.
See also [Packaging a Python model for Seldon Core using Docker](https://docs.seldon.ai/seldon-core-1/configuration/wrappers-and-sdks/python-language-wrapper/python_wrapping_docker) in the Seldon Core documentation.
```python wrap expandable theme={"dark"}
class MyModel(object):
"""
Model template. You can load your model parameters in __init__ from a
location accessible at runtime
"""
def __init__(self):
"""
Add any initialization parameters. These will be passed at runtime
from the graph definition parameters defined in your seldondeployment
kubernetes resource manifest.
"""
print("Initializing")
def predict(self,X,features_names,**kwargs):
"""
Return a prediction.
Parameters
----------
X : array-like
feature_names : array of feature names (optional)
"""
print("Predict called - will run identity function")
return X
def class_names(self):
return ["X_name"]
```
A real instance of this class with the Paraphrase Multilingual MiniLM L12 v2 model is as follows:
```python wrap expandable theme={"dark"}
import logging
import os
from transformers import AutoTokenizer, AutoModel
from torch.nn import functional as F
from typing import Iterable
import numpy as np
import torch
log = logging.getLogger()
class mini():
def __init__(self):
self.tokenizer = AutoTokenizer.from_pretrained('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')
self.model= AutoModel.from_pretrained('sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2')
#Mean Pooling
def mean_pooling(self, model_output, attention_mask):
token_embeddings = model_output[0] #First element of model_output contains all token embeddings
input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
def predict(self, X:np.ndarray, names=None, **kwargs):
# In Fusion there are several variables passed in the numpy array with the Milvus Query stage,
# Encode to Milvus index stage, and Vectorize Seldon index and query stage:
# [pipeline, bool, and text]. Text is what variable will be encoded, so that is what will be set to 'text'
# When using the Machine Learning stage, the input map keys should match what what is in this file.
model_input = dict(zip(names, X))
text = model_input["text"]
with torch.inference_mode(): # Allows torch to run more quickly
# Tokenize sentences
encoded_input = self.tokenizer(text, padding=True, truncation=True, return_tensors='pt')
log.debug('encoded input',str(encoded_input))
model_output = self.model(**encoded_input)
log.debug('model output',str(model_output))
# Perform pooling. In this case, max pooling.
sentence_embeddings = self.mean_pooling(model_output, encoded_input['attention_mask'])
# Normalize embeddings, because Fusion likes it that way.
sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=-1)
# Fixing the shape of the emebbedings to match (1, 384).
final = [sentence_embeddings.squeeze().cpu().detach().numpy().tolist()]
return final
def class_names(self) -> Iterable[str]:
return ["vector"]
```
In the above code, an additional function has been added in the class; this is completely fine to do.
Logging has also been added for debugging purposes.
Two functions are non-negotiable:
* `init`: The `init` function is where models, tokenizers, vectorizers, and the like should be set to self for invoking.\
It is recommended that you include your model’s trained parameters directly into the Docker container rather than reaching out to external storage inside `init`.
* `predict`: The `predict` function processes the field or query that Fusion passes to the model.\
The `predict` function must be able to handle any text processing needed for the model to accept input invoked in its `model.evaluate()`, `model.predict()`, or equivalent function to get the expected model result.
If the output needs additional manipulation, that should be done before the result is returned.\
For embedding models the return value must have the shape of (1, DIM), where DIM (dimension) is a consistent integer, to enable Fusion to handle the vector encoding into Milvus or Solr.
Use the *exact* name of the class when naming this file.
For the example, above the Python file is named `mini.py` and the class name is `mini()`.
## Create a Dockerfile
The next step is to create a Dockerfile. The Dockerfile should follow this general outline; read the comments for additional details:
```dockerfile wrap expandable theme={"dark"}
#It is important that python version is 3.x-slim for seldon-core
FROM python:3.10-slim
# Whatever directory(folder)the python file for your python class, Dockerfile, and
# requirements.txt is in should be copied then denoted as the work directory.
COPY . /app
WORKDIR /app
# The requirements file for the Docker container
COPY requirements.txt requirements.txt
RUN pip install --upgrade pip && pip install --no-cache-dir -r requirements.txt
# Copy source code
COPY . .
# GRPC - Allows Fusion to do a Remote Procedure Call
EXPOSE 5000
# Define environment variable for seldon-core
# !!!MODEL_NAME must be the EXACT same as the python file & python class name!!!
ENV MODEL_NAME=mini
ENV SERVICE_TYPE=MODEL
ENV PERSISTENCE=0
# Changing active directory folder (same one as above on lines 5 & 6) to default user, required for Fusion
RUN chown -R 8888 /app
# Command to wrap python class with seldon-core to allow it to be usable in Fusion
CMD ["sh", "-c", "seldon-core-microservice $MODEL_NAME --service-type $SERVICE_TYPE --persistence $PERSISTENCE"]
# You can use the following if You need shell features like environment variable expansion or
# You need to use shell constructs like pipes, redirects, etc.
# See https://docs.docker.com/reference/dockerfile/#cmd for more details.
# CMD exec seldon-core-microservice $MODEL_NAME --service-type $SERVICE_TYPE --persistence $PERSISTENCE
```
## Create a requirements file
The `requirements.txt` file is a list of installs for the `Dockerfile` to run to ensure the Docker container has the right resources to run the model.\
For the Paraphrase Multilingual MiniLM L12 v2 model, the requirements are as follows:
```text theme={"dark"}
seldon-core
torch
transformers
numpy
```
In general, if an item was used in an `import` statement in your Python file, it should be included in the requirements file.
An easy way to populate the requirements is by using in the following command in the terminal, inside the directory that contains your code:
```bash theme={"dark"}
pip freeze > requirements.txt
```
If you use `pip freeze`, you must manually add `seldon-core` to the requirements file because it is not invoked in the Python file but is required for containerization.
## Build and push the Docker image
After creating the `.py`, `Dockerfile`, and `requirements.txt` files, you need to run a few Docker commands.
Run the commands below in order:
```bash theme={"dark"}
DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build . -t [DOCKERHUB-USERNAME]/[REPOSITORY]:[VERSION-TAG]
```
```bash theme={"dark"}
docker push [DOCKERHUB USERNAME]/[REPOSITORY]:[VERSION-TAG]
```
Using the example model, the terminal commands would be as follows:
```bash theme={"dark"}
DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build . -t jstrmec/example_sbert_model:0.14; docker push jstrmec/example_sbert_model:0.14
```
This repository is public and you can visit it here: [example\_sbert\_model](https://hub.docker.com/r/jstrmec/example_sbert_model/)
## Deploy the model in Fusion
Now you can go to Fusion to deploy your model.
1. In Fusion, navigate to **Collections** > **Jobs**.
2. Add a job by clicking the **Add+** Button and selecting **Create Seldon Core Model Deployment**.
3. Fill in each of the text fields:
| Parameter | Description |
| - | - |
| Job ID | A string used by the Fusion API to reference the job after its creation. |
| Model name | A name for the deployed model. This is used to generate the deployment name in Seldon Core. It is also the name that you reference as a `model-id` when making predictions with the ML Service. |
| Model replicas | The number of load-balanced replicas of the model to deploy; specify multiple replicas for a higher-volume intake. |
| Docker Repository | The public or private repository where the Docker image is located. If you’re using Docker Hub, fill in the Docker Hub username here. |
| Image name | The name of the image with an optional tag. If no tag is given, `latest` is used. |
| Kubernetes secret | If you’re using a private repository, supply the name of the Kubernetes secret used for access. |
| Output columns | A list of column names that the model’s `predict` method returns. |
4. Click **Save**, then **Run** and **Start**.
When the job finishes successfully, you can proceed to the next section.
Now that the model is in Fusion, it can be utilized in either index or query pipelines, depending on the model’s purpose.
In this case the model is a word vectorizer or semantic vector search implementation, so both pipelines must invoke the model.
## Apply an API key to the deployment
These steps are only needed if your model utilizes any kind of secret, such as an API key.
If not, skip this section and proceed to the next.
1. Create and modify a `_sdep.yaml` file.\
In the first line, `kubectl get sdep` gets the details for the currently running Seldon Deployment job and saves those details to a YAML file. `kubectl apply -f open_sdep.yaml` adds the key to the Seldon Deployment job the next time it launches.
```yaml theme={"dark"}
kubectl get sdep -o yaml > _sdep.yaml
# Modify _sdep.yaml to add
- env:
- name: API_KEY
value: "your-api-key-here"
kubectl apply -f _sdep.yaml
```
2. Delete `sdep` before redeploying the model. The currently running Seldon Deployment job does not have the key applied to it. Delete it before redeploying and the new job will have the key.
```bash theme={"dark"}
kubectl delete sdep
```
3. Lastly, you can [encode into Milvus](/docs/5/fusion/intro/fusion-stack/milvus).
## Create a Milvus collection
1. In Fusion, navigate to **Collections** > **Jobs**.
2. Click the **Add+** Button and select **Create Collections in Milvus**.\
This job creates a collection in Milvus for storing the vectors sent to it.
The job is needed because a collection does not automatically spawn at indexing or query time if it does not already exist.
3. Name the job and the collection.
4. Click **Add** on the right side of the job panel.\
The key to creating the collection is the **Dimension** text field; this must exactly match the shape value your output prediction has.\
In our example the shape is (1,384), so 384 will be in the collections **Dimension** field:
The **Metric** field should typically be left at the default of `Inner Product`, but this also depends on use case and model type.
5. Click **Save**, then **Run** and **Start**.
## Configure the Fusion pipelines
Your real-world pipeline configuration depends on your use case and model, but for our example we will configure the index pipeline and then the query pipeline.
**Configure the index pipeline**
1. Create a new index pipeline or load an existing one for editing.
2. Click **Add a Stage** and then **Encode to Milvus**.
3. In the new stage, fill in these fields:
* The name of your model
* The output name you have for your model job
* The field you’d like to encode
* The collection name
4. Save the stage in the pipeline and index your data with it.
**Configure the query pipeline**
1. Create a new query pipeline or load an existing one for editing.
2. Click **Add a Stage** and then **Milvus Query**.
3. Fill in the configuration fields, then save the stage.
4. Add a **Milvus Ensemble Query** stage.\
This stage is necessary to have the Milvus collection scores taken into account in ranking and to weight multiple collections.
The **Milvus Results Context Key** from the Milvus Query Stage is used in this stage to preform math on the Milvus result scores.
One (1) is a typical multiplier for the Milvus results but any number can be used.
5. Save the stage and then run a query by typing a search term.
6. To verify the Milvus results are correct, use the **Compare+** button to see another pipeline without the model implementation and compare the number of results.
You have now successfully uploaded a Seldon Core model to Fusion and deployed it.
The course for **Machine Learning Demystified** focuses on key concepts in machine learning, plus the available Fusion machine learning jobs and pipeline stages.
# Machine Learning Jobs
Source: https://doc.lucidworks.com/docs/5/fusion/intro/machine-learning/ml-jobs
[localhost link]: http://localhost:3000/docs/5/fusion/intro/machine-learning/ml-jobs
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/intro/machine-learning/ml-jobs
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/494
Fusion provides these job types to perform machine learning tasks.
The course for **Machine Learning in Fusion** focuses on the basics about machine learning jobs in Fusion.
## Signals analysis
These jobs analyze a collection of signals in order to perform [query rewriting](#query-rewriting), [signals aggregation](#signals-aggregation), or [experiment analysis](#experiment-analysis).
* [Ground Truth](/docs/5/fusion/reference/config-ref/jobs/ground-truth)\
Ground truth or gold standard datasets are used in the ground truth jobs and query relevance metrics to define a specific set of documents.
Ground truth jobs estimate ground truth queries using click signals and query signals, with document relevance per query determined using a click/skip formula.
Use this job along with the [Ranking Metrics job](/docs/5/fusion/reference/config-ref/jobs/ranking-metrics) to calculate relevance metrics, such as Normalized Discounted Cumulative Gain (nDCG).
To create a ground truth job, sign in to Fusion and click **Collections > Jobs**. Then click **Add+** and in the Experiment Evaluation Jobs section, select **Ground Truth**. You can enter basic and advanced parameters to configure the job. If the field has a default value, it is populated when you click to add the job.
## Basic parameters
To enter advanced parameters in the UI, click **Advanced**. Those parameters are described in [the advanced parameters section](#advanced-parameters).
* **Spark job ID.** The unique ID for the Spark job that references this job in the API. This is the `id` field in the configuration file. Required field.
* **Input/Output Parameters.** This section includes the **Signals collection** field, which is the Solr collection that contains click signals and its associated search log identifier. This is the `signalsCollection` field in the configuration file. Required field.
## Advanced parameters
If you click the **Advanced** toggle, the following optional fields are displayed in the UI.
* **Spark Settings.** This section lets you enter `parameter name:parameter value` options to use in this job. This is the `sparkConfig` field in the configuration file.
* **Additional Options.** This section includes the following options:
* **Search logs pipeline.** The pipeline ID associated with search log entries. This is the `searchLogsPipeline` field in the configuration file.
* **Join key (query signals).** The common key that joins the query signals in the signals collection. This is the `joinKeySignals` field in the configuration file.
* **Join key (click signals).** The common key that joins the click signals in the signals collection. This is the `joinKeySignals` field in the configuration file.
* **Search logs and options.** This section lets you enter `property name:property value` options to when loading the search logs collection. This is the `searchLogsAddOpts` field in the configuration file.
* **Additional signals options.** This section lets you enter `property name:property value` options when loading the signals collection. This is the `signalsAddOpts` field in the configuration file.
* **Filter queries.** The `array[string]` filter query to apply when selecting top queries from the query signals in the signals collection. This is the `filterQueries` field in the configuration file.
* **Top queries limit.** The total number of queries to select for ground truth calculations when this job is run. This is the `topQueriesLimit` field in the configuration file.
For more information, see [Ground truth query rewrite API configurations](/api-reference/experiments-api/get-ground-truth-results).
### Query rewriting
These jobs produce data that can be used for query rewriting or to inform updates to the [synonyms.txt file](/docs/5/fusion/getting-data-out/query-enhancement/synonyms/overview).
These jobs are deprecated in Fusion 5.9.15 and will be removed in a future release.
Lucidworks recommends using [Neural Hybrid Search](/docs/5/fusion/hybrid-search/overview), which achieves superior relevance compared to legacy machine learning methods.
* [Head/Tail Analysis](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis)\
Perform head/tail analysis of queries from collections of raw or aggregated signals, to identify underperforming queries and the reasons. This information is valuable for improving overall conversions, Solr configurations, auto-suggest, product catalogs, and SEO/SEM strategies, in order to improve conversion rates.
* [Phrase Extraction](/docs/5/fusion/reference/config-ref/jobs/phrase-extraction)\
Identify multi-word phrases in signals.
* [Synonym Detection Jobs](/docs/5/fusion/reference/config-ref/jobs/synonym-detection)\
Use this job to generate pairs of synonyms and pairs of similar queries. Two words are considered potential synonyms when they are used in a similar context in similar queries.
* [Token and Phrase Spell Correction](/docs/5/fusion/reference/config-ref/jobs/token-and-phrase-spell-correction)\
Detect misspellings in queries or documents using the numbers of occurrences of words and phrases.
### Signals aggregation
* [SQL Aggregation](/docs/5/fusion/reference/config-ref/jobs/aggregations/sql-aggregation)\
A Spark SQL aggregation job where user-defined parameters are injected into a built-in SQL template at runtime.
### Experiment analysis
* [Ranking Metrics](/docs/5/fusion/reference/config-ref/jobs/ranking-metrics)\
Use this job to calculate relevance metrics by replaying [ground truth queries](/docs/5/fusion/reference/config-ref/jobs/ground-truth) against catalog data using variants from an [experiment](/docs/5/fusion/getting-data-out/data-analytics/experiments/overview). Metrics include Normalized Discounted Cumulative Gain (nDCG) and others.
To create a Ranking Metrics job, sign in to Fusion and click **Collections > Jobs**. Then click **Add+** and in the Experiment Evaluation Jobs section, select **Ranking Metrics**. You can enter basic and advanced parameters to configure the job. If the field has a default value, it is populated when you click to add the job.
## Basic parameters
To enter advanced parameters in the UI, click **Advanced**. Those parameters are described in [the advanced parameters section](#advanced-parameters).
* **Spark job ID.** The unique ID for the Spark job that references this job in the API. This is the `id` field in the configuration file. Required field.
* **Output collection.** The Solr collection where the job output is stored. The job will write the output to this collection. This is the `outputCollection` field in the configuration file. Required field.
* **Ground Truth Parameters.** This section includes this parameter:
* **Ground truth input collection.** The collection that stores the ground truth dataset this job accesses. This is the `inputCollection` field in the configuration file. Required field.
* **Ranking Experiment Parameters.** This section includes the following parameters:
* **Ranking experiment input collection.** The collection that stores the experiment data this job accesses. This is the `rankingExperimentConfig` `inputCollection` field in the configuration file. Optional field.
* **Experiment ID.** The identifier for the experiment that stores the variants this job uses to calculate ranking metrics. This is the `rankingExperimentConfig` `experimentId` field in the configuration file. Optional field.
* **Experiment metric name.** The name of the purpose (objective) of the experiment this job accesses to calculate ranking metrics. This is the `rankingExperimentConfig` `experimentObjectiveName` field in the configuration file. Optional field.
* **Default query profile.** The name of the query profile this job defaults to if the value is not specified in the experiment variants. This is the `rankingExperimentConfig` `defaultProfile` field in the configuration file. Optional field.
## Advanced parameters
If you click the **Advanced** toggle, the following optional fields are displayed in the UI.
* **Spark Settings.** This section lets you enter `parameter name:parameter value` options to use in this job. This is the `sparkConfig` field in the configuration file.
* **Ranking position @K.** The number of returned or recommended items that are ranked (based on the relevancy rating) that are used for metrics calculation. This is the `rankingPositionK` field in the configuration file.
* **Calculate metrics per query.** If this checkbox is selected (set to `true`), the job calculates the ranking metrics per query in the ground truth dataset, and saves the metrics data to the **Output collection** designated for this job. This is the `metricsPerQuery` field in the configuration file.
* **Ground Truth Parameters.** The advanced option adds these parameters:
* **Filter queries.** The Solr filter queries this job applies against the ground truth collection to calculate ranking metrics. This is the `groundTruthConfig` `filterQueries` field in the configuration file.
* **Query field.** The query field in the ground truth collection. This is the `groundTruthConfig` `queryField` field in the configuration file.
* **Doc ID field.** This field contains the ranked document IDs in the collection. This is the `groundTruthConfig` `docIdField` field in the configuration file.
* **Weight field.** This field contains the weight of the document as it relates to the query. This is the `groundTruthConfig` `weightField` field in the configuration file.
* **Ranking Experiment Parameters.** The advanced option adds these parameters:
* **Query pipelines.** These are the query pipelines for the experiment that stores the variants this job uses to calculate ranking metrics. This is the `rankingExperimentConfig` `queryPipelines` field in the configuration file.
* **Doc ID field.** This field contains the values (that match the ground truth data) this job uses to calculate ranking metrics. This is the `rankingExperimentConfig` `docIdField` field in the configuration file.
## Collaborative recommenders
These jobs analyze signals and generate matrices used to provide [collaborative recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods).
* [BPR Recommender](/docs/5/fusion/reference/config-ref/jobs/bpr-recommender)\
Use this job when you want to compute user recommendations or item similarities using a Bayesian Personalized Ranking (BPR) recommender algorithm.
* [Query-to-Query Session-Based Similarity jobs](/docs/5/fusion/reference/config-ref/jobs/query-to-query-session-based-similarity)\
This recommender is based on co-occurrence of queries in the context of clicked documents and sessions. It is useful when your data shows that users tend to search for similar items in a single search session. This method of generating query-to-query recommendations is faster and more reliable than the Query-to-Query Similarity recommender job, and is session-based unlike the similar queries previously generated as part of the [Synonym Detection job](/docs/5/fusion/reference/config-ref/jobs/synonym-detection).
## Content-based recommenders
[Content-based recommenders](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods) create matrices of similar items based on their content.
* [Content-Based Recommender](/docs/5/fusion/reference/config-ref/jobs/content-based-recommender)\
Use this job when you want to compute item similarities based on their content, such as product descriptions.
## Content analysis
* [Cluster Labeling](/docs/5/fusion/reference/config-ref/jobs/cluster-labeling)\
Cluster labeling jobs are run against your data collections, and are used:
* When clusters or well-defined document categories already exist
* When you want to discover and attach keywords to see representative words within existing clusters
To create a cluster labeling job, sign in to Fusion and click **Collections > Jobs**. Then click **Add+** and in the Clustering and Outlier Analysis Jobs section, select **Cluster Labeling**. You can enter basic and advanced parameters to configure the job. If the field has a default value, it is populated when you click to add the job.
## Basic parameters
To enter advanced parameters in the UI, click **Advanced**. Those parameters are described in [the advanced parameters section](#advanced-parameters).
* **Spark job ID.** The unique ID for the Spark job that references this job in the API. This is the `id` field in the configuration file. Required field.
* **Input/Output Parameters.** This section includes these parameters:
* **Training collection.** The Solr collection that contains documents associated with defined categories or clusters. The job will be run against this information. This is the `trainingCollection` field in the configuration file. Required field.
* **Output collection.** The Solr collection where the job output is stored. The job will write the output to this collection. This is the `outputCollection` field in the configuration file. Required field.
* **Data format.** The format that contains training data. The format must be compatible with Spark and options include `solr`, `parquet`, and `orc`. Required field.
* **Field Parameters.** This section includes these parameters:
* **Field to detect keywords from.** The field that contains the data that the job will use to discover keywords for the cluster. This is the `fieldToVectorize` field in the configuration file. Required field.
* **Existing document category field.** The field that contains existing cluster IDs or document categories. This is the `clusterIdField` field in the configuration file. Required field.
* **Top frequent terms field name.** The field where the job output stores top frequent terms in each cluster. Terms may overlap with other clusters. This is the `freqTermField` field in the configuration file. Optional field.
* **Top unique terms field name.** The field where the job output stores the top frequent terms that, for the most part, are unique in each cluster. This is the `clusterLabelField` field in the configuration file. Optional field.
* **Model Tuning Parameters.** This section includes these parameters:
* **Max doc support.** The maximum number of documents that can contain the term. Values that are `<1.0` indicate a percentage, `1.0` is 100 percent, and `>1.0` indicates the exact number. This is the `maxDF` field in the configuration file. Optional field.
* **Min doc support.** The minimum of documents that must contain the term. Values that are `<1.0` indicate a percentage, `1.0` is 100 percent, and `>1.0` indicates the exact number. This is the `minDF` field in the configuration file. Optional field.
* **Number of keywords for each cluster.** The number of keywords required to label each cluster. This is the `numKeywordsPerLabel` field in the configuration file. Optional field.
* **Featurization Parameters.** This section includes this parameter:
* **Lucene analyzer schema.** This is the JSON-encoded Lucene text analyzer schema used for tokenization. This is the `analyzerConfig` field in the configuration file. Optional field.
## Advanced parameters
If you click the **Advanced** toggle, the following optional fields are displayed in the UI.
* **Spark Settings.** The Spark configuration settings include:
* **Spark SQL filter query.** This field contains the Spark SQL query that filters your input data. For example, `SELECT * from spark_input` registers the input data as `spark_input`. This is the `sparkSQL` field in the configuration file.
* **Data output format.** The format for the job output. The format must be compatible with Spark and options include `solr` and `parquet`. This is the `dataOutputFormat` field in the configuration file.
* **Partition fields.** If the job output is written to non-Solr sources, this field contains a comma-delimited list of column names that partition the dataframe before the external output is written. This is the `partitionCols` field in the configuration file.
* **Read Options.** This section lets you enter `parameter name:parameter value` options to use when reading input from Solr or other sources. This is the `readOptions` field in the configuration file.
* **Write Options.** This section lets you enter `parameter name:parameter value` options to use when writing output to Solr or other sources. This is the `writeOptions` field in the configuration file.
* **Dataframe config options.** This section includes these parameters:
* **Property name:property value.** Each entry defines an additional Spark dataframe loading configuration option. This is the `trainingDataFrameConfigOptions` field in the configuration file.
* **Training data sampling fraction.** This is the fractional amount of the training data the job will use. This is the `trainingDataSamplingFraction` field in the configuration file.
* **Random seed.** This value is used in any deterministic pseudorandom number generation to group documents into clusters based on similarities in their content. This is the `randomSeed` field in the configuration file.
* **Field Parameters.** The advanced option adds this parameter:
* **Fields to load.** This field contains a comma-delimited list of Solr fields to load. If blank, the job selects the required fields to load at runtime. This is the `sourceFields` field in the configuration file.
* **Miscellaneous Parameters.** This section includes this parameter:
* **Model ID.** The unique identifier for the model to be trained. If no value is entered, the `Spark Job ID` is used. This is the `modelId` field in the configuration file.
To create new clusters, use the [Document Clustering job](/docs/5/fusion/reference/config-ref/jobs/document-clustering).
* [Document Clustering](/docs/5/fusion/reference/config-ref/jobs/document-clustering)\
The Document Clustering job uses an unsupervised machine learning algorithm to group documents into clusters based on similarities in their content. You can enable more efficient document exploration by using these clusters as facets, high-level summaries or themes, or to recommend other documents from the same cluster. The job can automatically group similar documents in all kinds of content, such as clinical trials, legal documents, book reviews, blogs, scientific papers, and products.
* [Classification job](/docs/5/fusion/reference/config-ref/jobs/classification)\
This job analyzes how your existing documents are categorized and produces a classification model that can be used to predict the categories of new documents at index time.
* [Outlier Detection](/docs/5/fusion/reference/config-ref/jobs/outlier-detection)\
Outlier detection jobs are run against your data collections, and also perform the following actions:
* Identify information that significantly differs from other data in the collection
* Attach labels to designate each outlier group
To create an Outlier Detection job, sign in to Fusion and click **Collections > Jobs**. Then click **Add+** and in the Clustering and Outlier Analysis Jobs section, select **Outlier Detection**. You can enter basic and advanced parameters to configure the job. If the field has a default value, it is populated when you click to add the job.
## Basic parameters
To enter advanced parameters in the UI, click **Advanced**. Those parameters are described in [the advanced parameters section](#advanced-parameters).
* **Spark job ID.** The unique ID for the Spark job that references this job in the API. This is the `id` field in the configuration file. Required field.
* **Input/Output Parameters.** This section includes these parameters:
* **Training collection.** The Solr collection that contains documents that will be clustered. The job will be run against this information. This is the `trainingCollection` field in the configuration file. Required field.
* **Output collection.** The Solr collection where the job output is stored. The job will write the output to this collection. This is the `outputCollection` field in the configuration file. Required field.
* **Data format.** The format that contains training data. The format must be compatible with Spark and options include `solr`, `parquet`, and `orc`. This is the `dataFormat` field in the configuration file. Required field.
* **Only save outliers?** If this checkbox is selected (set to `true`), only outliers are saved in the job’s output collection. If not selected (set to `false`), the entire dataset is saved in the job’s output collection. This is the `outputOutliersOnly` field in the configuration file. Optional field.
* **Field Parameters.** This section includes these parameters:
* **Field to vectorize.** The Solr field that contains text training data. To combine data from multiple fields with different weights, enter `field1:weight1`,`field2:weight2`, etc. This is the `fieldToVectorize` field in the configuration file. Required field.
* **ID field name.** The unique ID for each document. This is the `uidField` field in the configuration file. Required field.
* **Output field name for outlier group ID.** The field that contains the ID for the outlier group. This is the `outlierGroupIdField` field in the configuration file. Optional field.
* **Top unique terms field name.** The field where the job output stores the top frequent terms that, for the most part, are unique for each outlier group. The information is computed based on term frequency-inverse document frequency (TF-IDF) and group ID. This is the `outlierGroupLabelField` field in the configuration file. Optional field.
* **Top frequent terms field name.** The field where the job output stores top frequent terms in each cluster. Terms may overlap with other clusters. This is the `freqTermField` field in the configuration file. Optional field.
* **Output field name for doc distance to its corresponding cluster center.** The field that contains the document’s distance from the center of its cluster. This is based on the arithmetic mean of all of the documents in the cluster. This denotes how representative the document is in the cluster. This is the `distToCenterField` field in the configuration file. Optional field.
* **Model Tuning Parameters.** This section includes these parameters:
* **Max doc support.** The maximum number of documents that can contain the term. Values that are `<1.0` indicate a percentage, `1.0` is 100 percent, and `>1.0` indicates the exact number. This is the `maxDF` field in the configuration file. Optional field.
* **Min doc support.** The minimum number of documents that must contain the term. Values that are `<1.0` indicate a percentage, `1.0` is 100 percent, and `>1.0` indicates the exact number. This is the `minDF` field in the configuration file. Optional field.
* **Number of keywords for each cluster.** The number of keywords required to label each cluster. This is the `numKeywordsPerLabel` field in the configuration file. Optional field.
* **Featurization Parameters.** This section includes the following parameter:
* **Lucene analyzer schema.** This is the JSON-encoded Lucene text analyzer schema used for tokenization. This is the `analyzerConfig` field in the configuration file. Optional field.
## Advanced parameters
If you click the **Advanced** toggle, the following optional fields are displayed in the UI.
* **Spark Settings.** The Spark configuration settings include the following:
* **Spark SQL filter query.** This field contains the Spark SQL query that filters your input data. For example, `SELECT * from spark_input` registers the input data as `spark_input`. This is the `sparkSQL` field in the configuration file.
* **Data output format.** The format for the job output. The format must be compatible with Spark and options include `solr` and `parquet`. This is the `dataOutputFormat` field in the configuration file.
* **Partition fields.** If the job output is written to non-Solr sources, this field contains a comma-delimited list of column names that partition the dataframe before the external output is written. This is the `partitionCols` field in the configuration file.
* **Input/Output Parameters.** This advanced option adds these parameters:
* **Training data filter query.** If Solr is used, this field contains the Solr query executed to load training data. This is the `trainingDataFilterQuery` field in the configuration file.
* **Read Options.** This section lets you enter `parameter name:parameter value` options to use when reading input from Solr or other sources. This is the `readOptions` field in the configuration file.
* **Write Options.** This section lets you enter `parameter name:parameter value` options to use when writing output to Solr or other sources. This is the `writeOptions` field in the configuration file.
* **Dataframe config options.** This section includes these parameters:
* **Property name:property value.** Each entry defines an additional Spark dataframe loading configuration option. This is the `trainingDataFrameConfigOptions` field in the configuration file.
* **Training data sampling fraction.** This is the fractional amount of the training data the job will use. This is the `trainingDataSamplingFraction` field in the configuration file.
* **Random seed.** This value is used in any deterministic pseudorandom number generation to group documents into clusters based on similarities in their content. This is the `randomSeed` field in the configuration file.
* **Field Parameters.** The advanced option adds this parameter:
* **Fields to load.** This field contains a comma-delimited list of Solr fields to load. If blank, the job selects the required fields to load at runtime. This is the `sourceFields` field in the configuration file.
* **Model Tuning Parameters.** The advanced option adds these parameters:
* **Number of outlier groups.** The number of clusters to help find outliers. This is the `outlierK` field in the configuration file.
* **Outlier cutoff.** The fraction out of the total documents to designate as an outlier group. Values that are `<1.0` indicate a percentage, `1.0` is 100 percent, and `>1.0` indicates the exact number. This is the `outlierThreshold` field in the configuration file.
* **Vector normalization.** The p-norm value used to normalize vectors. A value of `-1` turns off normalization. This is the `norm` field in the configuration file.
* **Miscellaneous Parameters.** This section includes this parameter:
* **Model ID.** The unique identifier for the model to be trained. If no value is entered, the `Spark Job ID` is used. This is the `modelId` field in the configuration file.
## Data ingest
The Parallel Bulk Loader (PBL) job enables bulk ingestion of structured and semi-structured data from big data systems, NoSQL databases, and common file formats like Parquet and Avro.
Datasources the PBL uses include not only common file formats, but Solr databases, JDBC-compliant databases, MongoDB databases and more.
In addition, the PBL distributes the load across the Fusion Spark cluster to optimize performance. And because no parsing is needed, indexing performance is also maximized by writing directly to Solr.
For more information about:
* Available datasources and key features of the Parallel Bulk Loader, see [Parallel Bulk Loader concepts](/docs/5/fusion/getting-data-in/parallel-bulk-loader).
* Usage and detailed configuration, see [Parallel Bulk Loader configuration reference](/docs/5/fusion/reference/config-ref/jobs/parallel-bulk-loader).
# Machine Learning
Source: https://doc.lucidworks.com/docs/5/fusion/intro/machine-learning/overview
[localhost link]: http://localhost:3000/docs/5/fusion/intro/machine-learning/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/intro/machine-learning/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/495
## Machine learning with Spark
[Apache Spark](http://spark.apache.org/) is an open source cluster-computing framework that serves as a fast and general execution engine for large-scale data processing jobs that can be decomposed into stepwise tasks, which are distributed across a cluster of networked computers.
Spark improves on previous MapReduce implementations by using resilient distributed datasets (RDDs), a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner.
Fusion manages a Spark cluster that is used for all [signal aggregation](/docs/5/fusion/reference/config-ref/jobs/aggregations/overview) processes.
With a Fusion license, you can also use the Spark cluster to [train and compile machine learning models](/docs/5/fusion/intro/machine-learning/machine-learning-models), as well as to run experiments via the [Fusion UI](/docs/5/fusion/getting-data-out/data-analytics/experiments/overview) or the [Spark Jobs API](/api-reference/spark-job-controller-api/list-all-job-runs).
See [Machine Learning Jobs](/docs/5/fusion/intro/machine-learning/ml-jobs) for details about each pre-defined machine learning job in Fusion.
## More information
* [Apache Spark Key Terms, Explained](https://databricks.com/blog/2016/06/22/apache-spark-key-terms-explained.html)
* [Apache Spark on Wikipedia](https://en.wikipedia.org/wiki/Apache_Spark)
* [Machine Learning Models in Fusion](/docs/5/fusion/intro/machine-learning/machine-learning-models)
* [Machine Learning Jobs](/docs/5/fusion/intro/machine-learning/ml-jobs)
* [Spark Administration in Kubernetes](/docs/5/fusion/operations/survival-guide/spark-kubernetes-overview)
The course for **Intro to Machine Learning in Fusion** focuses on using machine learning to infer the goals of customers and users in order to deliver a more sophisticated search experience.
# Semantic vector search
Source: https://doc.lucidworks.com/docs/5/fusion/intro/machine-learning/vector-search
[localhost link]: http://localhost:3000/docs/5/fusion/intro/machine-learning/vector-search
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/intro/machine-learning/vector-search
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/80xtou
Semantic vector search uses machine learning models to determine context and meaning behind search queries and deliver relevant results.
## What is semantic vector search?
Semantic vector search (SVS) is the process of interpreting the intent and contextual meaning of queries and content. This is unique from the lexical approach to search, which is driven by keyword matching.
SVS uses deep metric learning to train a semantic model. The model learns to organize products and queries in a vector space such that similar things are placed near each other and dissimilar things are pushed far apart. At query time, SVS uses the semantic model to embed the query in the vector space and then find nearest neighbor products. This is *searching by meaning*.
**Semantic Vector Search advantages:**
1. Rich semantic capabilities for better relevancy than lexical search
2. Fixes problems:
1. Misspelled queries, synonyms, and phrases
2. Underperforming and zero-result queries
3. Doesn’t require signals to start
1. Uses general pre-trained models for cold-start solution
4. Collected signals are used to improve relevancy
1. Models are directly trained to rank items higher when users are more likely to interact with them
For more information, see **Set Up Semantic Vector Search**
These instructions are for Milvus and do not work with Solr SVS.
These are the suggested steps to get started with Semantic Vector Search (SVS).
1. Craft training data.
The training data fed to SVS models creates connections between consumer queries and products from a catalog. Crafting the training data is the biggest influence in terms of the performance and quality a SVS model can deliver.
2. Diagnose Zero Search Result themes.
Before create training data pairs, first identify the common themes across top zero result queries to craft the right kinds of training data appropriate for the kinds of problem themes.
\
This step involves looking at a report of the top zero search result queries and grouping them into categories such as:
* In Stock/Out of Stock
* Product Not Carried
* Foreign Vocabulary
* Misspelling
3. Create Zero Search Result training pairs.
A Zero Search Result Query (ZRQ) is the most obvious indicator that a search engine is struggling to retrieve relevant results, so create training data pairs geared toward learning what consumers were looking for when submitting such terms. Zero Search Result (ZSR) training pairs are created by looking at what a consumer did after getting to a zero results page. More specifically, look at what products a consumer adds to cart following a zero results page. For a ZSR query, a subset will try to search another way, either through using different search terms or by browsing category pages. This is referred to as "Learning from the Persistent Shopper." In the end, pairs of data are created which look like `ZSR Query, Next Item Added-to-Cart`.
4. Create Abandoned Search training pairs.
Another indicator for troublesome queries or poor relevance is Abandoned Searches (AS). An AS is defined when a consumer submits a query and receives search results but does not click on any of those results. Look at what a consumer adds to their cart following an AS. The pairs of this type follow the format of `AS query, Next Item Added-to-Cart`.
5. Create Converted Search training pairs.
The two prior types of training data outline a circumventive consumer path to find relevant results. These circumventive paths are essential to enhance the coverage of the SVS models in terms of edge cases and understanding foreign vernacular to the product catalog. The final type of training data, Converted Search (CS), outlines the 'happy path' a consumer takes. A happy path occurs when a consumer submits a search, finds exactly what they are looking for in the search results, then adds that item to their cart.
This type of training data is essential in enhancing the precision and accuracy of the SVS model by providing positively converting pairs where there is a direct correlation between the query submitted and the search results returned. This is also the simplest type of training data to create, as it is just an aggregation of already collected `Search Query, Item Added-to-Cart` pairs.
6. Apply a filter.
The last step of the training data crafting process is to apply a filter to the aggregated training data pairs collection to remove noisy data pairs for ZRQ and AS types. A noisy data pair is defined by a pair in which the outlined search query has no connection to the next item added to cart. Noisy signals can manifest in several different ways, such as shopping from a list of items and not finding a product. Instead of being persistent and using different methods to find that or a similar product, a shopper might move on to the next, unrelated, item on their list. To remove noisy pairs from the training data collection, look at the pairs with low aggregation counts and selectively remove them.
7. Train the SVS models.
After validating that the training data pairs were created appropriately, utilize this training data to build SVS deep learning models using supervised learning methodology. This means the pre-trained models are used as a base for understanding relationships between items in the product catalog, then the crafted training data is used to refine the baseline relationships specific to a business and its consumers. To maximize the effectiveness of these SVS models, conduct model trials (or iterations) to find the best configuration and training data combination.
8. Deploy model.
Upon completion of a Supervised Training Job, a model is deployed using Seldon-Core, an open source technology used to deploy and manage models within a Kubernetes cluster. This Seldon-Core model deployment is used both in the index and query pipeline procedures.
9. Index product catalog.
Part of the implementation of SVS includes encoding the product catalog into a vector space by creating a vector for each product within the catalog and storing them in an appropriate collection. Milvus is an open source vector search similarity engine and is used to store the encoded vectors from the product catalog and to perform vector similarity searches in query pipelines.
10. Integrate with an index pipeline.\
To populate a Milvus collection with vectors from the catalog, add **Encode into Milvus** stage to the index pipeline. This **Encode into Milvus** stage vectorizes each product within the catalog and stores it inside a Milvus collection as the products are being indexed into a Solr collection. Each vector in a Milvus collection has a corresponding `milvus_id` stored as part of the product record in the Solr collection.
11. Set up a [**Create Collections in Milvus**](/docs/5/fusion/reference/config-ref/jobs/create-collections-in-milvus) job.\
Give the job and collection a name and fill in the **Dimension** parameter. The **Dimension** parameter is always 2x the value input for the **RNN Function Units List** parameter in the Supervised Training job. For example, if the RNN Function Units List parameter is 128, then the Dimension parameter is 256.
12. Refresh the trained model.\
The trained model also needs a periodic refresh to account for changes in consumer behavior, such as seasonal changes. To refresh the SVS models, a new model is trained using the most recent consumer training data and the catalog is reindexed using the freshly created model and queries are switched to also be encoded using the new model.
13. Integrate with a query pipeline.\
Prepare the SVS solution for load testing and production by integrating vector search with query pipelines. There are three main components to this integration:
* Ensure that the Vector Search component of the pipeline is only triggered when zero results are returned by the main query pipeline.
* Ensure that the results returned by vector search maintain the functionality of results from the traditional pipeline, such as sorting and faceting.
* Ensure that searches involving the vector search components are tagged appropriately in the generated signals for analytics purposes.
14. Conduct load testing.\
Test load based on needs, for example against a 20% null result rate at peak QPS plus comfort room (20% of 1200 QPS + comfort room = \~300 QPS for vector search). If the initial vector search configuration does not perform well against such benchmarks, consider configuration changes:
* Optimization of resource allocation
* Turning off Solr spell check within the vector search pipeline. Removing spell check is the main configuration change needed to drive the query response times down to acceptable levels.
# Query Data Flow
Source: https://doc.lucidworks.com/docs/5/fusion/intro/query-dataflow
[localhost link]: http://localhost:3000/docs/5/fusion/intro/query-dataflow
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/intro/query-dataflow
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/3215
[Indexing Data Flow](/docs/5/fusion/intro/indexing-dataflow) shows you how your content, signals, and recommendations are indexed in Fusion. This topic explains how to query your indexed data.
## 1. Query your content
When you query your content using the default query pipeline *and* you have [signals enabled](/docs/5/fusion/intro/indexing-dataflow), you also get [query rewrites](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) and [automatic boosting](/docs/5/fusion/getting-data-out/query-enhancement/boosting/boost-with-signals) to enhance the relevancy of your search results. In addition to matches from your primary collection, you’ll get results produced by Fusion’s machine learning jobs, based on [signals](/docs/5/fusion/getting-data-out/query-enhancement/signals/overview) from your users.
[Query profiles](/docs/5/fusion/getting-data-out/query-basics/query-pipelines/query-profiles) give you a stable endpoint that can be associated with any [query pipeline](/docs/5/fusion/getting-data-out/query-basics/query-pipelines/overview). Just specify the query profile when sending queries to the [Query Profiles API](/api-reference/query-profiles-api/get-the-service-status) endpoint. You can always reconfigure the query profile to point to a different pipeline, or to an [experiment](/docs/5/fusion/getting-data-out/data-analytics/experiments/overview).
* Several stages perform [query rewriting](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) and [response rewriting](/docs/5/fusion/getting-data-out/query-enhancement/response-rewriting) by reading from the `_query_rewrite` collection:
* The [Text Tagger stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/text-tagger-query-stage) finds [misspellings](/docs/5/fusion/getting-data-out/query-enhancement/misspelling-detection), [phrases](/docs/5/fusion/getting-data-out/query-enhancement/phrase-detection), [underperforming query strings](/docs/5/fusion/getting-data-out/query-enhancement/underperforming-queries), and [synonyms](/docs/5/fusion/reference/config-ref/jobs/synonym-detection) in the incoming query.
* The [Apply Rules stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/query-rules-query-stage) applies your [business rules](/docs/5/fusion/getting-data-out/query-enhancement/business-rules/overview) to the incoming query.
* The [Modify Response with Rules stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/rules-augment-response-query-stage)
* The [Boost with Signals stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommendation-boosting-query-stage) reads aggregated signals to boost search results that have been popular with users.
* The [Solr Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/solr-query-stage) fetches the final set of relevant search results from the primary collection.
## 2. Query for recommendations
To get even more relevant results based on AI-powered analysis of your signals data, your query pipeline can perform any number of secondary queries to the collections where the output from machine learning jobs is indexed. Your secondary queries can provide additional search results based on Fusion’s analysis of your signals.
* Use the [Recommend Items for Item stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommend-items-for-item-query-stage) to query for [collaborative items-for-item recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-item#collaborative-items-for-item-recommendations).
* The [Recommend Items for User stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommend-items-for-user-query-stage) queries for [items-for-user recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/items-for-user).
* The [Recommend Similar Queries stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/recommend-similar-queries-query-stage) provides [queries-for-query recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/queries-for-query).
# Fusion Apps
Source: https://doc.lucidworks.com/docs/5/fusion/intro/ui-tour/fusion-apps
[localhost link]: http://localhost:3000/docs/5/fusion/intro/ui-tour/fusion-apps
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/intro/ui-tour/fusion-apps
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/179
*Fusion apps* provide tailored search functionality to specific groups of users.
An app is a named set of linked *objects*, including collections, datasources, index and query pipelines, index and query profiles, parsers, and more.
You can define security on a per-app basis using [roles](/docs/5/fusion/operations/security/access-control/roles) and
[security realms](/docs/5/fusion/operations/security/access-control/security-realms).
The **Create App** modal can be accessed by clicking Create App. When naming an app, the name must start with a letter and can include letters, numbers, dashes (`-`), or underscores (`_`), with a maximum length of 128 characters.
**To edit an app, access the **Edit App** modal by clicking on the settings cog icon on the app tile. Click on the pencil icon by the app name in the modal to edit the app name.**
Generally, you will create multiple apps for different purposes. You can view all of your apps in the launcher when you log in to the Fusion UI:
To enter the Fusion workspace for any app, click the app name.
In the Fusion workspace, you can hover over to switch to another app, create a new app, or return to the launcher:
The course for **Fusion Applications and Collections** focuses on the features of Fusion apps and collections and best practices when working with them.
## Learn more
You can create, modify, and delete apps in the Fusion UI or with the Apps API.
## Manage apps in the Fusion UI
### Create an app
Fusion prompts you to create an app the first time you log in. You can create additional apps at anytime from Fusion’s home page.
1. Log into Fusion and navigate to the home screen.
2. Click **Create new app**.
3. Enter a **Name**, **Description**, and select an **App tile color**.
4. Click **Create App**.
### Modify or delete an app
1. Log into Fusion and navigate to the home screen.
2. Click the **Configure** icon on the app you want to modify or delete.
3. To modify the app, click the edit icon .
1. Edit the **Name** or **Description**.
2. Click **OK**.
4. To delete the app, click **Delete app**.
1. A dialog box appears where you can choose which associated objects to delete or retain. Only objects not associated with other objects are displayed. All objects are selected by default.
2. Click **Confirm**.
## Manage apps with the Apps API
### Create an app
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'Content-type: application/json' -d '{"id": "MyNewApp", "name":"My new app", "description":"A really really great new app"}' https://FUSION_HOST/api/apps
```
### Delete an app
Run the following command, replacing `{id}` with the name of your app. This operation take about a minute to complete.
```bash theme={"dark"}
curl -u USERNAME:PASSWORD -X DELETE https://FUSION_HOST/api/apps/{id}
```
To export a Fusion app, you can use either the Fusion UI or the Objects API.
To upgrade from one Fusion version to a 5.x version, see Fusion 5 Upgrades to migrate objects.
## Export an app with the Fusion UI
1. How to export an app with the Fusion UI
2. Navigate to the launcher.
3. Hover over the app you want to export and click the Configure icon:
4. In the app config window, click **Export app to zip**:
See import for information to import the downloaded zip file into other instances of Fusion 5.x.
## Export an app with the Objects API
The examples below show how to export one or more apps:
**Export all apps**
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD http://localhost:6764/api/objects/export?type=app > all-apps.zip
```
**Get all app IDs, then export one app by ID**
```bash expandable theme={"dark"}
> curl -u USERNAME:PASSWORD http://localhost:6764/api/apps
[ {
"id" : "movies",
"name" : "Movies",
"description" : "Search the movielens database.",
"dataUri" : "/App-Tile-01-460x160.png",
"properties" : {
"headerImageName" : "headerImage1",
"tileColor" : "apps-darkblue",
"previousCollectionId" : "movies"
}
}, {
"id" : "tech-pubs",
"name" : "TechPubs",
"description" : "Search the documentation.",
"dataUri" : "/App-Tile-02-460x160.png",
"properties" : {
"headerImageName" : "headerImage2",
"tileColor" : "apps-darkblue",
"previousCollectionId" : "tech-pubs"
}
} ]
> curl -u USERNAME:PASSWORD http://localhost:6764/api/objects/export?app.ids=tech-pubs > techpubs.zip
```
When you export an app that includes objects that are shared with other apps, then all apps linked to that object are also exported.
**Export two apps by ID**
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD http://localhost:6764/api/objects/export?app.ids=tech-pubs,movies > two-apps.zip
```
For the `app` object type, the `deep` parameter is not used. Linked objects are always included when exporting apps.
# Index Workbench
Source: https://doc.lucidworks.com/docs/5/fusion/intro/ui-tour/index-workbench
[localhost link]: http://localhost:3000/docs/5/fusion/intro/ui-tour/index-workbench
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/intro/ui-tour/index-workbench
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/158
The Index Workbench is a powerful tool that combines key aspects of the data indexing configuration process into one user-friendly panel. It guides the user through the workflow for configuring datasources, parsers and index pipelines, and then running a datasource job to index your data.
## Datasources
A datasource is a complete data ingest configuration that consists of the following components:
* **Connector.** Connectors are out-of-the-box components for pulling your data into Fusion. Lucidworks provides a wide variety of connectors, each specialized for a particular data type. For a list of connectors, see [Fusion Connectors](/docs/fusion-connectors/overview).
* **Parser.** Parsers provide fine-grained configuration for inbound data.
* For Fusion 5.x, see [Parsers](/docs/5/fusion/reference/config-ref/parser-stages/overview) for more information.
* For Fusion 4.x, see [Parsers](/docs/4/fusion-server/reference/parser-stages/overview) for more information.
* **Index pipeline.** Index pipelines transform incoming data for indexing by Fusion.
* For Fusion 5.x, see [Index Pipelines](/docs/5/fusion/getting-data-in/indexing/index-pipelines/overview) for more information.
* For Fusion 4.x, see [Index Pipelines](/docs/4/fusion-server/concepts/indexing/datasources/index-pipelines) for more information.
To set up a new datasource in the Index Workbench, click **New**. The **Add A New Datasource** panel appears, offering three simplified methods of configuring how Fusion fetches your data:
* By URL
* By local file upload
* By selecting an installed connector to configure\
See [Connectors Configuration Reference](/docs/fusion-connectors/connectors/overview) for a complete list of available connectors. Some are installed by default, and others require separate download and installation.
## Parsers
[Parsers](/docs/5/fusion/reference/config-ref/parser-stages/overview) are a configurable component of the indexing workflow, for flexibility and specificity when parsing inbound data.
A parser consists of an ordered list of parser stages that is completely customizable. The same parser stage can be added to a given configuration multiple times if the different specified settings within those stages best suits the parsing of the data. There is no limit to the number of parser stages that can be included in a parser, and the order in which they run is also completely flexible.
In a parser, after all of the doctype-specific parser stages have run, the Tika and Fallback stages are useful catch-all stages that can attempt to parse anything that has not yet been matched. Tika is used for parsing many types of unstructured documents like PDFs, DOCX, and many more. If all of the other stages in the parser fail to completely parse the data, the Fallback stage can copy the raw bytes directly to Solr.
## Index pipelines
An [Index Pipeline](/docs/5/fusion/getting-data-in/indexing/index-pipelines/overview) transforms incoming data into a document suitable for indexing by Solr via a series of modularized operations called stages. Fusion provides a variety of specialized index stages to index data effectively. Stages can be selected, configured, and enabled or disabled in the Index Pipeline section of the Index Workbench.
Once you finish configuring a datasource using the Index Workbench, you can move on to setting up queries using the [Query Workbench](/docs/5/fusion/intro/ui-tour/query-workbench), which provides a similar workflow for configuring and previewing search results.
### Adding a pipeline stage
Click **Add a Stage** to add index pipeline stages that can perform data filtering, transformations, and more.
The [Solr Indexer stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/solr-indexer-stage)
is the only pipeline stage that is required to complete the indexing process, and therefore exists in every query pipeline. It is always the last stage in a series.
### Re-ordering pipeline stages
The order of the pipeline stages matters, because the output from one stage becomes the input to the next stage. For example, the
[Solr Indexer stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/solr-indexer-stage)
must always come last in the sequence, so that data is indexed only after it has been processed by all other stages. Putting this stage first in the sequence means that subsequent stages have no effect on the indexed data.
Click and drag any stage in the pipeline to move it up or down in the sequence of stages. The preview panel automatically updates the results to reflect the output of the new sequence.
### Enabling and disabling pipeline stages
By default, every stage in an index pipeline is enabled. While working with a pipeline, it can be helpful to disable stages without removing them completely. This allows you to preserve a stage’s configuration while observing how the search results change in its absence. You can re-enable the stage at any time. When you save a query pipeline, the enabled/disabled state of each stage is also saved.
Click the circle next to any stage in order to enable or disable it.
# Object Explorer
Source: https://doc.lucidworks.com/docs/5/fusion/intro/ui-tour/object-explorer
[localhost link]: http://localhost:3000/docs/5/fusion/intro/ui-tour/object-explorer
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/intro/ui-tour/object-explorer
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/121
Object Explorer lets you easily find and manage Fusion *objects* within a specific app and across multiple apps in an entire Fusion deployment.
The quick learning for **Object Explorer Quick Learning** focuses on a brief overview of the Object Explorer.
The course for **The Object Explorer** focuses on how to view and manage objects such as collections, datasources, index and query pipelines, index and query profiles, and parsers within apps and share them to other applications.
## Fusion apps and objects
*Fusion apps* provide tailored search functionality to specific groups of users.
An app is a named set of linked *objects*, including collections, datasources, index and query pipelines, index and query profiles, parsers, and more.
You can define security on a per-app basis using [roles](/docs/5/fusion/operations/security/access-control/roles) and
[security realms](/docs/5/fusion/operations/security/access-control/security-realms).
The **Create App** modal can be accessed by clicking Create App. When naming an app, the name must start with a letter and can include letters, numbers, dashes (`-`), or underscores (`_`), with a maximum length of 128 characters.
**To edit an app, access the **Edit App** modal by clicking on the settings cog icon on the app tile. Click on the pencil icon by the app name in the modal to edit the app name.**
Generally, you will create multiple apps for different purposes. You can view all of your apps in the launcher when you log in to the Fusion UI:
To enter the Fusion workspace for any app, click the app name.
In the Fusion workspace, you can hover over to switch to another app, create a new app, or return to the launcher:
An object is something that has a specific function in a Fusion app, such as an index pipeline or Spark job. It is often something that you configure. An app is an object. Several object types are *groups* that group other objects. For more information about object types, see [Object types](#object-types).
## What you can do in Object Explorer
In Object Explorer, you can explore objects.
Which objects you can explore depends on the [permissions](/docs/5/fusion/operations/security/access-control/permissions) that your security realm and user definition grant you. As the user `admin`, you can explore all objects supported by Object Explorer.
* **Determine which objects are in which apps.** You can determine which objects are in the current app, in any app, in no apps, and in any app or no apps. When you navigate to an object, you can display the apps that the object is in.
* **View and navigate between related objects.** In Object Explorer, related objects are connected by lines. Clicking an object moves it to the center of the Object Explorer window and displays related objects around it.
* **View related objects.** When you select an object, Object Explorer centers and displays related objects around it, connected by lines, for example:
* **Perform some management tasks.** You can perform some management tasks inside Object Explorer. For example, you can schedule jobs and tasks from Object Explorer. For other management tasks, Object Explorer takes you to where you need to be.
* **Add and remove objects** (*Advanced feature*) - When you create an app, it consists of objects. When you create additional apps, Fusion can *share* some objects across apps. Shared objects are in the `_shared` system group, and they have multiple dots on them, one for each app they are in.\
In some cases, you might want to remove an object from an app (undo sharing) or add an object to an app. For example, after configuring a parser in one app, you could share the parser with other apps, *without having to share things you do not want to share*, for example, the collection.
## Object types
You can explore these types of objects with Object Explorer:
| |
| - |
| App |
| Spark job or task |
| Collection |
| Datasource |
| Index pipeline |
| Parser |
| Query pipeline |
| A generic icon that is currently used for query profiles, index profiles, blobs, and experiments |
| Recommendations group |
| System group |
| User-defined group or Fusion-defined group that is not a system group |
# Fusion UI Tour
Source: https://doc.lucidworks.com/docs/5/fusion/intro/ui-tour/overview
[localhost link]: http://localhost:3000/docs/5/fusion/intro/ui-tour/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/intro/ui-tour/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/116
The Fusion UI gives you easy access to all Fusion features.
The UI service utilizes the API service, so each UI feature has an API equivalent. For programmatic access to the features you see in the UI, see the [API Reference Guide](/docs/5/fusion/dev-portal/rest-apis).
The course for **Navigating the Fusion UI** focuses on navigating through the Fusion application.
## Logging in
The Proxy service runs on port 6764 by default. Access the Fusion UI through the Proxy service. To log in, point your browser to this port on the Fusion host, for example, `http://FUSION_HOST:FUSION_PORT/`.
When you are the *first user* to log in:
Fusion prompts you to set the password for the "admin" account and agree to the license terms. All other accounts must be created by an admin.
The first time you log in:
Fusion opens the **Quickstart**, to give you a demonstration of Fusion’s most basic features. You can click **Exit the Quickstart** to close it and go to the Fusion launcher.
The Quickstart is a wizard that lets you explore some of Fusion’s core capabilities:
* Creating or selecting an app
* Selecting a collection (where data is stored)
* Indexing data
* Searching
The course for **Using The Quickstart Wizard** focuses on using the wizard to practice using the key functions of Fusion.
## Step through the Quickstart wizard
If it is not already open, open the Quickstart wizard. In the Fusion launcher, click **New here? Get started...**.
1. Click **Continue**.
On the **Select an App** screen, you can select or create an [app](/docs/4/fusion-server/concepts/overview) for your quickstart data.
If you click **Create new app**, Fusion prompts you to enter an app name (and optionally a description), then click **Create App** to return to the Quickstart. Your new app is selected by default.
2. After selecting an app, click **Continue**.
On the **Select a Collection** screen, you can select or create a [collection](/docs/4/fusion-server/concepts/overview) for your quickstart data. A collection with the same name as your app is created automatically:
3. Click **Continue**.\
On the **Index Data** screen, you can either select one of the built-in sample datasets or click **Use my data** to upload your own:
4. Click **Continue**.\
On the **Query Data** screen, you can see all search results and enter your own search queries to test the indexed dataset. You can also select the display fields or view the parsed documents:
5. Click **Continue**.
From here, you can do the following:
* Open the Index Workbench to change the index pipeline.
* [Fusion 4.x Index Workbench](/docs/4/fusion-server/concepts/indexing/datasources/index-workbench)
* [Fusion 5.x Index Workbench](/docs/5/fusion/intro/ui-tour/index-workbench)
* Open the Query Workbench to change the query pipeline.
* [Fusion 4.x Query Workbench](/docs/4/fusion-server/concepts/querying/query-workbench)
* [Fusion 5.x Query Workbench](/docs/5/fusion/getting-data-out/query-basics/query-workbench)
The workbenches are essentials tools in the Fusion workflow.
* Open [App Studio](/docs/4/app-studio/overview) to create a user interface for searching this collection.
App Studio is only available in Fusion 4.1 and 4.2.
You can open the Quickstart at any time by navigating to the Fusion launcher and clicking **New here? Get started…**.
The brief tour below shows you how to find your way around the interface. For a more detailed tour, try the **Getting Started with Fusion Server** tutorial.
This tutorial takes you from installation to application-ready search data in four easy parts, using a [MovieLens](https://grouplens.org/datasets/movielens/) dataset.
* Part 1: Run Fusion and Create an App
Download, install Fusion, and run Fusion, then create a Movie Search app.
* Part 2: Get Data In
Use the Index Workbench to configure an index pipeline, preview the results, and get data into the Movie Search app in a format that is useful for search.
* Part 3: Get Data Out
Use Query Workbench to get data out of the Movie Search app, explore the role of query pipeline stages, configure faceting, and preview search results.
* Part 4: Improve Relevancy
Use signals and boosting to make search results more relevant.
## A brief tour
* The *launcher* is where you can view all apps in your Fusion environment and create new ones.
* The *Fusion workspace* is where you configure how an app ingests, indexes, queries, and analyzes data.
### The Fusion launcher
When you log in to Fusion, the first page is the Fusion launcher. This is where you create and launch apps:
To return to the Fusion workspace, hover over , and then click **Return to launcher**.
See details about **Create an App**.
To create an app in App Studio:
1. Navigate to the launcher and click **Create new app**.
2. Enter an app name.
Optionally, you can also enter a description and select the app tile color.
If you want to choose the name of the first collection in the app, then deselect **Create new app default collection**. When you do this and click **Create app** (in the next step), Fusion Server prompts you to enter a name for the first collection. Otherwise, Fusion Server creates a default collection whose name is the same as the app name.
3. Click **Create App**.
The Fusion workspace appears, and you are ready to work with your app’s first collection.
### The Fusion workspace
You can reach the Fusion workspace by clicking on any app in the launcher. When you enter the Fusion workspace, you find all the tools you need to configure the selected app:
* On the left side are the [app tool menus](#app-tool-menus), where you can find tools for working with your app.
* [System-wide tools](#system-wide) are at the top of the UI. With the exception of the **Collections** tool, which allows you to switch between collections for your app, these tools do not concern your app specifically.
#### App tool menus
On the left side of the screen are six menus where you can find tools for working with your app:
1. **Indexing** (getting data in)\
The [Index Workbench](/docs/5/fusion/intro/ui-tour/index-workbench) is the primary tool for getting data into your app through the Fusion UI. It provides an all-in-one view of the components of a datasource. You can also access those components individually through this menu.
2. **Querying** (getting data out)\
For getting data out, the [Query Workbench](/docs/5/fusion/intro/ui-tour/query-workbench) is the main tool. It shows you a live preview of search results while you configure and fine tune your query pipelines.\
See also [Getting Data Out](/docs/5/fusion/getting-data-out/query-basics/overview).
3. **Collections**\
Manage your data collections and configure their features.\
See also [Collection Management](/docs/5/fusion/getting-data-in/indexing/collections/overview).
4. **Relevance**\
Add or edit rules, recommendations, and signals.
5. **Analytics Hub**\
View App Insights, run Experiments, or view your Dashboard.
6. **System**\
Manage your system, including access control and logs.
#### System-wide
On the right are menus for system-wide actions:
1. **Apps**\
Switch between apps or return to the app launcher.
2. **Collections**\
View and navigate to other collections.
3. **Microservices**\
View the status of all Fusion microservices.
4. **Search**\
Click the search icon and enter a query to find areas of the Fusion UI, objects, and related documentation. Clicking an object opens the [Object Explorer](/docs/5/fusion/intro/ui-tour/object-explorer).
5. **Help**\
Access the documentation.
6. **Alerts**\
New alerts are briefly displayed in the foreground. Open this menu to see past alerts, or to clear the alert history.
7. **Account Settings**\
Edit your profile, log out, or manage your Fusion license.
# Query Workbench
Source: https://doc.lucidworks.com/docs/5/fusion/intro/ui-tour/query-workbench
[localhost link]: http://localhost:3000/docs/5/fusion/intro/ui-tour/query-workbench
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/intro/ui-tour/query-workbench
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/167
The Query Workbench is where you edit query pipeline stages, results parameters, and more, then preview the impacts of the changes in real time before saving them. Relevancy tools such as boosting and blocking can be accessed and altered through the Query Workbench. The Query Workbench streamlines the process of querying indexed results and fine-tuning Query Pipelines to surface the results that best satisfy the end user’s needs.
See the [Query Language Cheat Sheet](/docs/5/fusion/getting-data-out/query-basics/query-language-cheat-sheet) for help constructing queries.
With this sandbox paradigm, you can experiment with query pipelines without consequence because Fusion uses a copy of a pipeline to simulate the new results, and it is not permanently modified in the collection until you click **Save**.
## Query pipelines and pipeline stages
[Query pipelines](/docs/5/fusion/getting-data-out/query-basics/query-pipelines/overview) work by processing search requests and returning an ordered list of matching documents. Each pipeline consists of a series of query stages that can be added, ordered, and configured using the Query Workbench.
Select any stage in the pipeline to open its configuration panel:
## Boosting and blocking
As you search your data and inspect the results, you can manipulate the rankings of individual documents. Boosting a document raises its ranking, while blocking a document removes it from search results.
Boosting and blocking affects the results for specific queries. For example, when you search for "citizen" and boost the document for "Citizen Kane", your end users will see that title boosted to the top when they also search for "citizen", but not necessarily when they search for "kane".
The Query Workbench provides convenient buttons for instantly blocking or boosting the documents that you see in the preview panel. Clicking **Boost** automatically adds the [Boost Documents stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/boost-documents-query-stage) to your query pipeline, with a boost rule that matches your query and the document you clicked. Likewise, clicking **Block** adds and configures the [Block Documents stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/block-documents-query-stage).
## Compare mode
Compare mode allows you to compare search results using two different query pipelines, or compare search results from the original query pipeline and the copy being modified in the Query Workbench. On the right is the working pipeline, which you can edit. On the left is one of your saved pipelines.
# Advanced Query Enhancement
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/advanced-query-enhancement/overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/advanced-query-enhancement/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/advanced-query-enhancement/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/3233
In addition to the basic query enhancement features, Fusion has some advanced query enhancement features that are best configured by data scientists or the Lucidworks Professional Services team:
* [Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/overview) analyzes your question-and-answer content to produce data that can power virtual assistants or chatbots.
* Fusion includes pre-trained [Sentiment Analysis and Prediction](/docs/5/fusion/getting-data-out/advanced-query-enhancement/sentiment-analysis) models to assist with the interpretation of queries or content to determine some of their subjective aspects, such as emotions, intentions, opinions, and so on, using machine learning.
# Sentiment Analysis and Prediction
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/advanced-query-enhancement/sentiment-analysis
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/advanced-query-enhancement/sentiment-analysis
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/advanced-query-enhancement/sentiment-analysis
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/453
One major part of understanding intent is understanding sentiment. Fusion has done the heavy lifting by integrating state-of-the-art, pre-trained sentiment analysis models directly into the platform. Lucidworks makes these models readily available for search developers to embed sentiment scoring into the query and indexing pipelines as stages.
## What are sentiment analysis and sentiment prediction?
Sentiment analysis is the interpretation of queries or content to determine some of their subjective aspects, such as emotions, intentions, opinions, and so on, using machine learning. Sentiment analysis produces a sentiment model that Fusion can use to perform sentiment prediction.
Since stronger sentiments can be positive or negative while a weaker sentiment is more neutral, sentiment analysis uses the positive/negative polarity to measure a sentiment’s strength:
```txt theme={"dark"}
Strongly Strongly
Negative Neutral Positive
-2 -1 0 1 2
```
For example, a query on a customer support site can express negative emotions, as in "How can I fix this frustrating upgrade problem?" Similarly, a shopper may imply a positive or negative intention to purchase an item, or a positive or negative opinion of that item.
## Intelligent applications with sentiment analysis and prediction
When you assign sentiment values to incoming queries or to documents at indexing time, you can leverage that data to deliver more insight to your business applications. Understanding the intent and sentiment of customers or employees can help determine which information you deliver or how you treat subsequent interactions in the workflow. This kind of intelligence is valuable for e-commerce, call centers, social media, or surveys.
## Pre-trained sentiment models
Lucidworks provides two pre-trained sentiment models that you can deploy to get started:
| sentiment-general:v1.0 | sentiment-reviews:v1.0 |
| - | - |
| A general-purpose model, trained on short sentences.
Suitable for short texts and for intent prediction.
See **Deploy the sentiment-general Model** for instructions. | A model trained on a variety of customer reviews.
Optimized for longer texts. It also supports highlighting the tokens that provide stronger sentiment.
See **Deploy the sentiment-reviews Model** for instructions. |
This topic explains how to deploy the sentiment-general pre-trained [sentiment prediction](/docs/5/fusion/getting-data-out/advanced-query-enhancement/sentiment-analysis) model. This is a general-purpose sentiment prediction model, trained on short sentences. It is suitable for short texts and for intent prediction.
## Install the model in Fusion
1. Navigate to **Collections** > **Jobs**.
2. Select **New** > **Create Seldon Core Model Deployment**.
3. Configure the job as follows:
* **Job ID.** The ID for this job, such as `deploy-sentiment-general`.
* **Model Name.** The model name of the Seldon Core deployment that will be referenced in the Machine Learning pipeline stage configurations, such as `sentiment-general`.
* **Docker Repository.** The value is `lucidworks`.
* **Image Name.** The value is `sentiment-general:v1.0`.
* **Kubernetes Secret Name for Model Repo.** This value is left empty.
* **Output Column Names for Model.** The value is `[label, score]`.
4. Click **Save**.
5. Click **Run > Start**.
If you are running a private Docker repository that virtualizes external images, enter your private registry into the **Docker Repository** field and `lucidworks/sentiment-general:v1.0` into the **Image Name** field.
## Configure the Machine Learning pipeline stages
In your index or query pipelines add Machine Learning stage and specify sentiment-general in the Model ID field (or a custom model name that was used during deployment).
### Configure the Machine Learning index stage
1. In your index pipeline, click **Add a Stage** > **Machine Learning**.
2. In the **Model ID** field, enter the model name you configured above, such as `sentiment-general`.
3. In the **Model input transformation script** field, enter the following:
```js theme={"dark"}
var modelInput = new java.util.HashMap()
modelInput.put("text", doc.getFirstFieldValue("text"))
modelInput
```
4. In the **Model output transformation script** field, enter the following:
```js theme={"dark"}
doc.addField("sentiment_label_s", modelOutput.get("label")[0])
doc.addField("sentiment_score_d", modelOutput.get("score")[0])
```
5. Save the pipeline.
### Configure the Machine Learning query stage
1. In your query pipeline, click **Add a Stage** > **Machine Learning**.
2. In the **Model ID** field, enter the model name you configured above, such as sentiment-general.
3. In the **Model input transformation script** field, enter the following:
```js theme={"dark"}
var modelInput = new java.util.HashMap()
modelInput.put("text", request.getFirstParam("q"))
modelInput
```
4. In the **Model output transformation script** field, enter the following:
```js theme={"dark"}
{/* // To put into request */}
request.putSingleParam("sentiment_label", modelOutput.get("label")[0])
request.putSingleParam("sentiment_score", modelOutput.get("score")[0])
{/* // To put into query context */}
context.put("sentiment_label", modelOutput.get("label")[0])
context.put("sentiment_score", modelOutput.get("score")[0])
{/* // To put into response documents. NOTE: This can be done only after Solr Query stage */}
var docs = response.get().getInnerResponse().getDocuments();
var ndocs = new java.util.ArrayList();
for (var i=0; i
This topic explains how to deploy the sentiment-reviews pre-trained [sentiment prediction](/docs/5/fusion/getting-data-out/advanced-query-enhancement/sentiment-analysis) model. This model is trained on a variety of customer reviews and optimized for longer texts. It also supports attention weights output that can be used for highlighting the tokens that provide stronger sentiment; see [Model output](#model-output) below for an example.
## Install the model in Fusion
1. Navigate to **Collections** > **Jobs**.
2. Select **New** > **Create Seldon Core Model Deployment**.
3. Configure the job as follows:
* **Job ID.** The ID for this job, such as `deploy-sentiment-reviews`
* **Model Name.** The model name of the Seldon Core deployment that will be referenced in the Machine Learning pipeline stage configurations, such as `sentiment-reviews`.
* **Docker Repository.** The value is `lucidworks`.
* **Image Name.** The value is `sentiment-reviews:v1.0`.
* **Kubernetes Secret Name for Model Repo.** The value is left empty.
* **Output Column Names for Model.** The value is `[label, score, tokens, attention_weights]`.
4. Click **Save**.
5. Click **Run** > **Start**.
## Configure the Machine Learning pipeline stages
You can put your sentiment prediction model to work using the Machine Learning index stage or Machine Learning query stage. You will specify the same **Model Name** that you used when you installed the model above.
Generally, you only need to apply the model in the index pipeline, in order to perform sentiment prediction on your content. Optionally, you can configure the query pipeline in a similar way, to perform sentiment prediction on incoming queries and outgoing responses and apply special treatment depending on the prediction.
### Configure the Machine Learning *index* stage
1. In your index pipeline, click **Add a Stage** > **Machine Learning**.
2. In the **Model ID** field, enter the model name you configured above, such as `sentiment-reviews`.
3. In the **Model input transformation script** field, enter one of the following, depending on whether you want to output attention weights:\
Without attention weights:
```js theme={"dark"}
var modelInput = new java.util.HashMap()
modelInput.put("text", doc.getFirstFieldValue("text"))
modelInput
```
With attention weights:
```js theme={"dark"}
var modelInput = new java.util.HashMap()
modelInput.put("text", doc.getFirstFieldValue("text"))
modelInput.put("attention_output", "true")
modelInput
```
4. In the **Model output transformation script** field, enter the following:\
Without attention weights:
```js theme={"dark"}
doc.addField("sentiment_label_s", modelOutput.get("label")[0])
doc.addField("sentiment_score_d", modelOutput.get("score")[0])
```
With attention weights:
```js theme={"dark"}
doc.addField("sentiment_label_s", modelOutput.get("label")[0])
doc.addField("sentiment_score_d", modelOutput.get("score")[0])
doc.addField("sentiment_attention_tokens_ss", modelOutput.get("tokens"))
doc.addField("sentiment_attention_weights_ds", modelOutput.get("attention_weights"))
```
5. Save the pipeline.
### Optional: Configure the Machine Learning *query* stage
1. In your query pipeline, click **Add a Stage** > **Machine Learning**.
2. In the **Model ID** field, enter the model name you configured above, such as sentiment-reviews.
3. In the **Model input transformation script** field, enter the following:\
Without attention weights:
```js theme={"dark"}
var modelInput = new java.util.HashMap()
modelInput.put("text", request.getFirstParam("q"))
modelInput
```
With attention weights:
```js theme={"dark"}
var modelInput = new java.util.HashMap()
modelInput.put("text", request.getFirstParam("q"))
modelInput.put("attention_output", "true")
modelInput
```
4. In the **Model output transformation script** field, enter the following, noting the sections that need to be uncommented if you are using attention weights:
```js theme={"dark"}
{/* // To put into request */}
request.putSingleParam("sentiment_label", modelOutput.get("label")[0])
request.putSingleParam("sentiment_score", modelOutput.get("score")[0])
{/* // With attention output also uncomment this */}
{/* // request.putSingleParam("sentiment_attention_tokens", modelOutput.get("tokens")) */}
{/* // request.putSingleParam("sentiment_attention_weights", modelOutput.get("attention_weights")) */}
{/* // To put into query context */}
context.put("sentiment_label", modelOutput.get("label")[0])
context.put("sentiment_score", modelOutput.get("score")[0])
{/* // With attention output also uncomment this */}
{/* // context.put("sentiment_attention_tokens", modelOutput.get("tokens")) */}
{/* // context.put("sentiment_attention_weights", modelOutput.get("attention_weights")) */}
{/* // To put into response documents (can be done only after Solr Query stage) */}
var docs = response.get().getInnerResponse().getDocuments();
var ndocs = new java.util.ArrayList();
var attention_tokens = modelOutput.get("tokens")
var attention_weights = modelOutput.get("attention_weights")
var attention_tokens_arr = new java.util.ArrayList(attention_tokens.size());
var attention_weights_arr = new java.util.ArrayList(attention_weights.size());
for ( i = 0; i < attention_tokens.size(); i++) {
attention_tokens_arr.add(attention_tokens[i])
attention_weights_arr.add(attention_weights[i])
}
for (var i=0; i
These models support the English language only.
## Model output
Both of the pre-trained models output the following:
* a **label**, `negative` or `positive`
* a **score** from `-2` to `2`
The sentiment-reviews:v1.0 model also optionally outputs tokens and their corresponding attention weights, that is, the weight that each token carries in the sentiment prediction. The total of all attention weights is always 1, that is, each value represents a percentage of the total weight. In the example below, "awesome" has the highest weight because it expresses the strongest sentiment compared to other tokens in the string:
```js theme={"dark"}
// Input
text = "That is awesome!"
// Output
sentiment_label = ‘positive’
sentiment_score = 1.998
sentiment_attention_tokens = ['That', "'", 's', 'awesome', '!']
sentiment_attention_weights = [0.154, 0.078, 0.069, 0.444, 0.255]
```
# Best Practices
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/best-practices
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/best-practices
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/best-practices
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/462
Below is a collection of useful tips and tricks that can help improve results.
## How to begin
### Use cases for chatbots, virtual assistants, QnA, and FAQ search
In most cases, Lucidworks recommends you begin with [coldstart models](#coldstart-models).
* The coldstart model is only suitable for English. And the large coldstart model increases query time. So if a moderate query time is acceptable, use a [coldstart model](#coldstart-models).
* For multilingual support or fast query time, use a multilingual model.
#### Coldstart models
Pre-trained cold start models are:
* Robust out-of-the-box models.
* Easily configured and do not require training data.
* Used to establish a solid baseline solution to compare when running supervised training.
Lucidworks provides these pre-trained cold start models for [Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/overview):
* `qna-coldstart-large` - this is a large model trained on variety of corpuses and tasks.
* `qna-coldstart-multilingual` - covers 16 languages. List of supported languages: Arabic, Chinese-simplified, Chinese-traditional, English, French, German, Italian, Japanese, Korean, Dutch, Polish, Portuguese, Spanish, Thai, Turkish, Russian.
When you use these models, you do not need to run the model training job. Instead, you run a job that deploys the model into Fusion. The [Create Seldon Core Model Deployment job](/docs/5/fusion/reference/config-ref/jobs/create-seldon-core-model-deployment) deploys your model as a Docker image in Kubernetes, which you can scale up or down like other Fusion services.
These models are a good basis for a cold start solution if your data does not contain much domain-specific terminology. Otherwise, consider training a model using your existing content.
Dimension size of vectors for both models is **512**. You might need this information when creating collections in Milvus.
## Deploy a pre-trained cold-start model into Fusion
The pre-trained cold-start models are deployed using a Fusion job called [Create Seldon Core Model Deployment](/docs/5/fusion/reference/config-ref/jobs/create-seldon-core-model-deployment). This job downloads the selected pre-trained model and installs it in Fusion.
1. Navigate to **Collections** > **Jobs**.
2. Select **Add** > **Create Seldon Core Model Deployment**.
3. Enter a **Job ID**, such as `deploy-qna-coldstart-multilingual` or `deploy-qna-coldstart-large`.
4. Enter the **Model Name**, one of the following:
* `qna-coldstart-multilingual`
* `qna-coldstart-large`
5. In the **Docker Repository** field, enter `lucidworks`.
6. In the **Image Name** field, enter one of the following:
* `qna-coldstart-multilingual:v1.1`
* `qna-coldstart-large:v1.1`
7. Leave the **Kubernetes Secret Name for Model Repo** field empty.
8. In the **Output Column Names for Model** field, enter one of the following:
* `qna-coldstart-multilingual:[vector]`
* `qna-coldstart-large:[vector, compressed_vector]`
9. Click **Save**.
10. Click **Run** > **Start** to start the deployment job.
## Next steps
1. Configure The Smart Answers Pipelines
2. [Evaluate a Smart Answers Query Pipeline](#how-to-evaluate-results)
### Use cases for never null and eCommerce
For these use cases, the preferred method is to build a training set and use a supervised job. It is usually possible to set up a training dataset. However, if that is not possible, set up an initial [coldstart pre-trained model](#coldstart-models).
The quality of model and results may vary because models are designed to use natural language questions and not short, less defined queries. To enhance the results:
1. Collect signal data (if it does not exist).
2. Build a training set.
## Obtain training data
There are multiple ways to obtain training data. The supervised training job uses pairs of questions and answers (or queries and product names) to train deep learning encoders.
### Use cases for chatbots, virtual assistants, QnA, and FAQ search
For these use cases:
* Base training data typically consists of existing FAQ pair information.
* After the coldstart solution is deployed, training data can also be derived from sources such as search logs and customer support correspondence. The data can be constructed and expanded manually.
* Expanded training data can be obtained using the augmentation techniques in the [Data Augmentation job](/docs/5/fusion/reference/config-ref/jobs/data-augmentation). For natural language scenarios, backtranslation is effective to provide paraphrased versions of the original questions and answers that improve vocabulary and semantic coverage.
### Use cases for never null and eCommerce
Training data for these use cases is primarily collected signals data.
There are different ways to construct training data, which determines how the resulting trained model behaves based on the:
* Available types of signals.
* Quantity of signals.
* Needs of your business.
**Examples include:**
1. Query and product pair click signals:
* Help increase the click-through rate (CTR).
* Are usually high volume, but low quality.
2. Add-to-cart and purchase complete signals:
* Help increase sales.
* Have a very strong relationship between query and products.
* Are high quality, but low volume.
* Produce a resulting trained model that ranks products most likely to be purchased higher than it does other products (in the results generated by the query).
3. Null-search signals result from products selected after a zero-result search or an abandoned search.
4. Different training datasets can be combined to train one model to rule them all or results from the different models can be combined at query time.
## How to evaluate results
The supervised training job incorporates the evaluation mechanism. A hold out validation set:
* Is constructed automatically from unique questions and queries.
* Can be sized in the job configuration.
* Provides the ability to:
* Evaluate how well models generalize to new, unseen queries.
* Prevent the occurrence of overfitting.
You can also evaluate a Smart Answers query pipeline to:
* Ensure configuration is set correctly.
* Compare different models or pipelines.
* Perform an end-to-end evaluation of the entire query pipeline.
The [Smart Answers Evaluate Pipeline job](/docs/5/fusion/reference/config-ref/jobs/smart-answers-evaluate-pipeline) evaluates the rankings of results from any [Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/overview) pipeline and finds the best set of weights in the ensemble score. This topic explains how to set up the job.
Before beginning this procedure, prepare a machine learning model using either the Supervised method or the Cold start method, or by selecting one of the pre-trained cold start models, then Configure your pipelines.
The input for this job is a set of test queries and the text or ID of the correct responses. At least 100 entries are needed to obtain useful results. The job compares the test data with Fusion’s actual results and computes variety of the ranking metrics to provide insights of how well the pipeline works. It is also useful to use to compare with other setups or pipelines.
## Prepare test data
1. Format your test data as query/response pairs, that is, a query and its corresponding answer in each row.
You can do this in any format that Fusion support, but parquet file would be preferable to reduce the amount of possible encoding issues.
The response value can be either the document ID of the correct answer in your Fusion index (preferable), or the text of the correct answer.
If you use answer text instead of an ID, make sure that the answer text in the evaluation file is formatted identically to the answer text in Fusion.
If there are multiple possible answers for a unique question, then repeat the questions and put the pair into different rows to make sure each row has exactly one query and one response.
2. If you wish to index test data into Fusion, create a collection for your test data, such as `sa_test_input` and index the test data into that collection.
## Configure the evaluation job
1. If you wish to save the job output in Fusion, create a collection for your evaluation data such as `sa_test_output`.
2. Navigate to **Collections** > **Jobs**.
3. Select **New** > **Smart Answers Evaluate Pipeline** (**Evaluate QnA Pipeline** in Fusion 5.1 and 5.2).
4. Enter a **Job ID**, such as `sa-pipeline-evaluator`.
5. Enter the name of your test data collection (such as `sa_test_input`) in the **Input Evaluation Collection** field.
6. Enter the name of your output collection (such as `sa_test_output`) in the **Output Evaluation Collection** field.
In Fusion 5.3 and later, you can also configure this job to read from or write to cloud storage.
7. Enter the name of the **Test Question Field** in the input collection.
8. Enter the name of the answer field as the **Ground Truth Field**.
9. Enter the **App Name** of the Fusion app where the main Smart Answers content is indexed.
10. In the **Main Collection** field, enter the name of the Fusion collection that contains your Smart Answers content.
11. In the **Fusion Query Pipeline** field, enter the name of the Smart Answers query pipeline you want to evaluate.
12. In the **Answer Or ID Field In Fusion** field, enter the name of the field that Fusion will return containing the answer text or answer ID.
13. Optionally, you can configure the **Return Fields** to pass from Smart Answers collection into the evaluation output.
Check the Query Workbench to see which fields are available to be returned.
14. Configure the **Metrics** parameters:
* **Solr Scale Function**\
Specify the function used in the Compute Mathematical Expression stage of the query pipeline, one of the following:
* `max`
* `log10`
* `pow0.5`
* **List of Ranking Scores For Ensemble**\
To find the best weights for different ranking scores, list the names of the ranking score fields, separated by commas. Different ranking scores might include Solr score, query-to-question distance, or query-to-answer distance from the Compute Mathematical Expression pipeline stage.
* **Target Metric To Use For Weight Selection**\
The target ranking metric to optimize during weights selection. The default is `mrr@3`.
15. Optionally, [read about the advanced parameters](/docs/5/fusion/reference/config-ref/jobs/smart-answers-evaluate-pipeline) and consider whether to configure them as well.
For example, **Sampling proportion** and **Sampling seed** provide a way to run the job only on a sample of the test data.
16. Click **Save**.
17\. Click **Run** > **Start**.
## Examine the output
The job provides a variety of metrics (controlled by the **Metrics list** advanced parameter) at different positions (controlled by the **Metrics\@k list** advanced parameter) for the chosen final ranking score (specified in **Ranking score** parameter).
**Example: Pipeline evaluation metrics**
**Example: recall\@1,3,5 for different weights and distances**
In addition to metrics, a results evaluation file is indexed to the specified output evaluation collection. It provides the correct answer position for each test question as well as the top returned results for each field specified in **Return fields** parameter.
## How to choose a training model
Select the model that best fits your training needs.
### RNN models
RNN models support most use cases, including digital commerce and heavy query traffic domains. The models also provide:
* Easily-understood training.
* Extremely fast query time.
While several model cases are available, Lucidworks recommends that in most cases, begin with `word_en_300d_2M`. If the case requires other language support, use `bpe_multilingual` or one of the specific `bpe` models for CJK languages.
### BERT models
BERT models:
* Are *only* recommended if you have GPU access. Transformer-based models can *only* use a single GPU. Even if multiple GPUs are available, this type of model does *not* scale to multiple GPUs.
* Are suitable for natural language queries, chatbots, virtual assistants, and low query traffic domains.
* Provide better semantic quality results, especially if only a small amount of training data is available.
* Are more expensive to use for training.
* Run more slowly at query time than RNN models.
## Model vector size
Pre-trained models produce vectors with a dimension of 512.
Vector dimensions for supervised job trained models are highlighted at the end of the training logs, or can be derived from the model and parameters:
* BERT-based model vectors have a dimension of 768.
* RNN-based model vectors have a dimension of two times the size of the last specified layer.
## Hyperparameters to tune the training job
Most of the parameters in the supervised training job contain effective default values. Some of the parameters are automatically determined based on the dataset statistics.
Specific parameters to review are listed below, but see [Advanced Model Training Configuration for Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/smart-answers-advanced-model-config) for detailed information about tuning the training job.
### General encoder parameters
These parameters are common to BERT and RNN models. The most significant parameter in this section is **Fine-tune Token Embeddings**. The parameter is disabled by default to prevent overfitting. However, enable the parameter to improve queries with multiple variations or jargon. For example, queries in the eCommerce domain.
### RNN encoder parameters
**Dropout Ratio:** Tune in the \[0.1, 0.5] range to improve regularization and prevent overfitting. The customary value is 0.15 or 0.3.
**RNN Function (Units) List:** The parameters determine the number of layers to use and the output vector size.
One layer is typically sufficient, and Lucidworks does *not* recommend using over two layers. Adding a smaller second layer produces a smaller vector size that reduces the vector index size and might reduce query time.
For example, the current configuration is a one-layer network with 128 units (256 vector size) that provides good results. To improve QPS and index a very large collection without losing quality, add a small layer with 64 units (128 vector size). Instead of adding one small layer, you can add two small layers. The two additional small layers slightly increases encoding time, but the vector size is two times smaller in memory, which may improve search time.
### Training and evaluation parameters
To reduce training time, especially when training on a CPU, lower the value in following parameters:
* **Number of Epochs**
* **Monitor Patience**
To increase training time, increase the value in the parameters.
The **Training Batch Size** parameter impacts the timeframe of an epoch and how often validation is performed. Lucidworks recommends setting the following:
* At least 100 batches per training epoch
* Use bigger batches with larger training datasets
* Values of:
* \[32, 64] for small training sets
* \[128, 256] for medium training sets
* \[512, 1024, 2048] for large training sets
Due to the size of BERT-based models, the training batch size values recommended are \[8, 16, 32].
## How to choose a Milvus index
Milvus supports multiple indexes. See the following Milvus documentation:
* [Vector index](https://milvus.io/docs/v1.0.0/index.md#Vector-index) for detailed descriptions of the indexes.
* [Performance FAQ](https://milvus.io/docs/v1.0.0/performance_faq.md) for performance information.
### Default index
The default (FLAT) index provides:
* The best possible quality (typically 100% of the model quality), but is the least effective in terms of query time.
* A reasonable query time for low traffic use cases under 250 QPS.
### HNSW index
The HNSW index:
* Supports higher QPS.
* Supports collections over a few million documents.
* Needs to be configured with the parameter values detailed in [Create Indexes in Milvus Jobs](/docs/5/fusion/reference/config-ref/jobs/create-indexes-in-milvus).
# Cold Start Solution
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/cold-start
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/cold-start
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/cold-start
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/463
When no training data exists to use the Supervised job, the cold start solution uses general pre-trained Deep Learning models or provides a possibility to train custom word embeddings for specific domains.
Over time, we suggest capturing [signals](/docs/5/fusion/getting-data-out/query-enhancement/signals/overview) from document clicks, likes, and downloads.
These signals can be used to construct training dataset. After cumulating at least 3000 pairs, that feedback can be used as training data for the [Supervised solution](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/faq-solution).
Like the Supervised solution, the Cold start solution has two parts:
## 1. Prepare a model
There are two ways to do this:
* Smart Answers comes with two pre-trained cold-start models. These models can serve as a strong baseline if your data does not have too much domain-specific terms. See **Set Up a Pre-Trained Cold Start Model for Smart Answers**.
* You can train your own cold start model by configuring a job that analyzes your existing content. See **Train a Smart Answers Cold Start Model**.
Lucidworks provides these pre-trained cold start models for [Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/overview):
* `qna-coldstart-large` - this is a large model trained on variety of corpuses and tasks.
* `qna-coldstart-multilingual` - covers 16 languages. List of supported languages: Arabic, Chinese-simplified, Chinese-traditional, English, French, German, Italian, Japanese, Korean, Dutch, Polish, Portuguese, Spanish, Thai, Turkish, Russian.
When you use these models, you do not need to run the model training job. Instead, you run a job that deploys the model into Fusion. The [Create Seldon Core Model Deployment job](/docs/5/fusion/reference/config-ref/jobs/create-seldon-core-model-deployment) deploys your model as a Docker image in Kubernetes, which you can scale up or down like other Fusion services.
These models are a good basis for a cold start solution if your data does not contain much domain-specific terminology. Otherwise, consider training a model using your existing content.
Dimension size of vectors for both models is **512**. You might need this information when creating collections in Milvus.
## Deploy a pre-trained cold-start model into Fusion
The pre-trained cold-start models are deployed using a Fusion job called [Create Seldon Core Model Deployment](/docs/5/fusion/reference/config-ref/jobs/create-seldon-core-model-deployment). This job downloads the selected pre-trained model and installs it in Fusion.
1. Navigate to **Collections** > **Jobs**.
2. Select **Add** > **Create Seldon Core Model Deployment**.
3. Enter a **Job ID**, such as `deploy-qna-coldstart-multilingual` or `deploy-qna-coldstart-large`.
4. Enter the **Model Name**, one of the following:
* `qna-coldstart-multilingual`
* `qna-coldstart-large`
5. In the **Docker Repository** field, enter `lucidworks`.
6. In the **Image Name** field, enter one of the following:
* `qna-coldstart-multilingual:v1.1`
* `qna-coldstart-large:v1.1`
7. Leave the **Kubernetes Secret Name for Model Repo** field empty.
8. In the **Output Column Names for Model** field, enter one of the following:
* `qna-coldstart-multilingual:[vector]`
* `qna-coldstart-large:[vector, compressed_vector]`
9. Click **Save**.
10. Click **Run** > **Start** to start the deployment job.
## Next steps
1. [Configure The Smart Answers Pipelines](#3-configure-the-pipelines)
2. [Evaluate a Smart Answers Query Pipeline](#evaluate-the-query-pipeline)
The Smart Answers Cold Start Training job is deprecated in Fusion 5.12.
The [cold start solution for Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/cold-start) begins with training a model using your existing content. To do this, you run the [Smart Answers Coldstart Training](/docs/5/fusion/reference/config-ref/jobs/smart-answers-coldstart-training) job. This job uses variety of word embeddings, including custom via Word2Vec training, to learn about the vocabulary that you want to search against.
Smart Answers comes with two pre-trained cold-start models. If your data does not have many domain-specific words, then consider using a pre-trained model.
During a cold start, we suggest capturing user feedback such as document clicks, likes, and downloads on the Web site [App Studio](/docs/4/app-studio/overview) can help you get started). After accumulating feedback data and at least 3,000 query/response pairs, the feedback can be used to \[train a model using the Supervised method.
## Configure the training job
1. In Fusion, navigate to **Collections** > **Jobs**.
2. Select **Add** > **Smart Answer Coldstart Training**.
3. In the **Training Collection** field, specify the collection that contains the content that can be used to answer questions.
You can also configure this job to read from or write to cloud storage.
4. Enter the name of the **Field which contains the content documents**.
5. Enter a **Model Deployment Name**.
The new machine learning model will be saved in the blob store with this name. You will reference it later when you configure your pipelines.
6. Configure the **Model base**.
There are several pre-trained word and [BPE](https://nlp.h-its.org/bpemb) embeddings for different languages, as well as a few pre-trained BERT models.
If you want to train custom embeddings, please select `word_custom` or `bpe_custom`.
This trains Word2vec on the data and fields specified in **Training collection** and **Field which contains the content documents**. It might be useful in cases when your content includes unusual or domain-specific vocabulary.
When you use the pre-trained embeddings, the log shows the percentage of processed vocabulary words. If this value is high, then try using custom embeddings.
During the training job analyzes the content data to select weights for each of the words. The result model performs the weighted average of word embeddings to obtain final single dense vector for the content.
7. Click **Save**.
If using solr as the training data source ensure that the source collection contains the `random_*` dynamic field defined in its `managed-schema.xml`. This field is required for sampling the data. If it is not present, add the following entry to the `managed-schema.xml` alongside other dynamic fields `` and `` alongside other field types.
8. Click **Run** > **Start**.
After training is finished the model is deployed into the cluster and can be used in index and query pipelines.
## Next steps
1. [Configure The Smart Answers Pipelines](#3-configure-the-pipelines)
2. [Evaluate a Smart Answers Query Pipeline](#evaluate-the-query-pipeline)
## 2. Create collections in Milvus
In order to use `{app_name}-smart-answers` pipelines, you need to create collections in Milvus. Please refer to the [Milvus](/docs/5/fusion/intro/fusion-stack/milvus) documentation page.
## 3. Configure the pipelines
The trained model should be used at both index and query time in order to perform dense vector search.
* At index time, we provide the `{app_name}-smart-answers` index pipeline to help generate a dense vector representation of answers.
* At query time, we provide a `{app_name}-smart-answers` query pipeline to conduct run-time neural search. This pipeline transforms the incoming query into a dense vector using the trained model, then compares it with indexed answer dense vectors by computing the cosine distance between them. You can also use a query stage to combine Solr and document vector similarity scores at query time.
See **Configure the Smart Answers Pipelines**. Once your pipelines are configured, see **Evaluate a Smart Answers Query Pipeline** to test its effectiveness so that you can fine-tune the configuration.
Before beginning this procedure, train a machine learning model using either the FAQ method or the cold start method.
Regardless of how you set up your model, the deployment procedure is the same:
1. Create the `Milvus` collection.
2. Configure the `smart-answers` index pipeline.
3. Configure the `smart-answers` query pipeline.
See also [Best Practices](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/best-practices), [Advanced Model Training Configuration for Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/smart-answers-advanced-model-config), and Smart Answers Detailed Pipeline Setup.
## Create the Milvus collection
For complete details about job configuration options, see the [Create Collections in Milvus job](/docs/5/fusion/reference/config-ref/jobs/create-collections-in-milvus).
1. Navigate to **Collections** > **Jobs** > **Add +** and select **Create Collections in Milvus**.
2. Configure the job:
1. Enter an ID for this job.
2. Under **Collections**, click **Add**.
3. Enter a collection name.
4. In the **Dimension** field, enter the dimension size of vectors to store in this Milvus collection. The Dimension should match the size of the vectors returned by the encoding model. For example, the `Smart Answers Pre-trained Coldstart` models outputs vectors of 512 dimension size. Dimensionality of encoders trained by `Smart Answers Supervised Training` job depends on the provided parameters and printed in the training job logs.
3. Click **Save**.
The `Create Collections in Milvus` job can be used to create multiple collections at once. In this image, the first collection is used in the [indexing](#configure-the-index-pipeline) and [query](#configure-the-query-pipeline) steps. The other two collections are used in the [example](#pipeline-setup-example).
4. Click **Run** > **Start** to run the job.
## Configure the index pipeline
1. Open the Index Workbench.
2. Load or create your datasource using the default smart-answers index pipeline.
3. Configure the
[Encode into Milvus stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/encode-into-milvus-index-stage):
1. change the value of **Model ID** to match the model deployment name you chose when you configured the model training job.
2. Change `Field to Encode` to the document field name to be processed and encoded into dense vectors.
3. Ensure the `Encoder Output Vector` matches the output vector from the chosen model.
4. Ensure the `Milvus Collection Name` matches the collection name created via the `Create Milvus Collection` job.
To test out your settings, turn on `Fail on Error` in the `Encode into Milvus` stage and Apply the changes. This will cause an error message to display if any settings need to be changed.
4. Save the datasource.
5. Index your data.
## Configure the query pipeline
1. Open the Query Workbench.
2. Load the default smart-answers query pipeline.
3. Configure the [Milvus Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-query-stage):
1. Change the **Model ID** value to match the model deployment name you chose when you configured the model training job.
2. Ensure the `Encoder Output Vector` matches the output vector from the chosen model.
3. Ensure the `Milvus Collection Name` matches the collection name created via the `Create Milvus Collection` job.
4. `Milvus Results Context Key` can be changed as needed. It will be used in the Milvus Ensemble Query Stage to calculate the query score.
4. In the [Milvus Ensemble Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-ensemble-query-stage), update the `Ensemble math expression` as needed based on your model and the name used in the prior stage for the storing the Milvus results.
In versions 5.4 and later, you can also set the `Threshold` so that the Milvus Ensemble Query Stage will only return items with a score **greater than** or **equal to** the configured value.
5. Save the query pipeline.
## Pipeline Setup Example
### Index and retrieve the question and answer together
To show question and answer together in one document (that is, treat the question as the title and the answer as the description), you can index them together in the same document. You can still use the default `smart-answers` index and query pipelines with a few additional changes.
Prior to configuring the Smart Answers pipelines, use the `Create Milvus Collection` job to create two collections, `question_collection` and `answer_collection`, to store the encoded "questions" and the encoded "answers", respectively.
#### Index Pipeline
As shown in the pictures below, you will need two Encode into Milvus stages, named Encode Question and Encode Answer respectively.
**Encode Question (Encode Into Milvus) stage**
**Encode Answer (Encode Into Milvus) stage**
In the Encode Question stage, specify `Field to Encode` to be `title_t` and change the `Milvus Collection Name` to match the new Milvus collection, `question_collection`.
In the Encode Answer stage, specify `Field to Encode` to be `description_t` and change the `Milvus Collection Name` to match the new Milvus collection, `answer_collection`.
#### Query Pipeline
Since we have two dense vectors generated during indexing, at query time we need to compute both query to question distance and query to answer distance. This can be set up as the pictures shown below with two Milvus Query Stages, one for each of the two Milvus collections.
To store those two distances separately, the `Milvus Results Context Key` needs to be different in each of these two stages.
In the Query Questions stage, we set the `Milvus Results Context Key` to `milvus_questions` and the Milvus collection name to `question_collection`.
Query Questions (Milvus Query) stage:
In the Query Answers stage, we set the `Milvus Results Context Key` to `milvus_answers` and the Milvus collection name to `answer_collection`.
Query Answers (Milvus Query) stage:
Now we can ensemble them together with the Milvus Ensemble Query Stage with the `Ensemble math expression` combining the results from the two query stages. If we want the question scores and answer scores weighted equally, we would use: `0.5 * milvus_questions + 0.5 * milvus_answers`.
This is recommended especially when you have limited FAQ dataset and want to utilize both question and answer information.
**Milvus Ensemble Query stage**
## Evaluate the query pipeline
The [Evaluate QnA Pipeline job](/docs/5/fusion/reference/config-ref/jobs/smart-answers-evaluate-pipeline) evaluates the rankings of results from any Smart Answers pipeline and finds the best set of weights in the ensemble score.
## Detailed pipeline setup
Typically, you can use the [default pipelines](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/overview) included with Fusion AI. These pipelines now utilize Milvus to store encoded vectors and to calculate vector similarity. This topic provides information you can use to customize the Smart Answers pipelines.
| | | |
| - | - | - |
| **"smart-answers" index pipeline** | | [Encode into Milvus stage](#the-encode-into-milvus-index-stage) |
| **"smart-answers" query pipeline** | |
|
### Create the Milvus collection
Prior to indexing data, the [Create Collections in Milvus job](/docs/5/fusion/reference/config-ref/jobs/create-collections-in-milvus) can be used to create the Milvus collection(s) used by the Smart Answers pipelines (see [Milvus overview](/docs/5/fusion/intro/fusion-stack/milvus)).
* `Job ID`. A unique identifier for the job.
* `Collection Name`. A name for the Milvus collection you are creating. This name is used in both the Smart Answer Index and the Smart Answer Query pipelines.
* `Dimension`. The dimension size of the vectors to store in this Milvus collection. The Dimension should match the size of the vectors returned by the encryption model. For example, if the model was created with either the `Smart Answers Coldstart Training` job or the `Smart Answers Supervised Training` job with the Model Base `word_en_300d_2M`, then the dimension would be 300.
* `Index file size`. Files with more documents than this will cause Milvus to build an index on this collection.
* `Metric`. The type of metric used to calculate vector similarity scores. `Inner Product` is recommended. It produces values between 0 and 1, where a higher value means higher similarity.
### Index pipeline setup
**Stages in the default "smart-answers" index pipeline**
Only one custom index stage needs to be configured in your index pipeline, the [Encode into Milvus index stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/encode-into-milvus-index-stage).
#### The Encode into Milvus Index Stage
If you are using a dynamic schema, make sure this stage is added *after* the [Solr Dynamic Field Name Mapping stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/solr-dynamic-field-name-mapping-index-stage).
The [Encode into Milvus](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/encode-into-milvus-index-stage) index stage uses the specified model to encode the `Field to Encode` and store it in Milvus in the given Milvus collection.
There are several required parameters:
* `Model ID`. The ID of the model.
* `Encoder Output Vector`. The name of the field that stores the compressed dense vectors output from the model. Default value: `vector`.
* `Field to Encode`. The text field to encode into a dense vector, such as `answer_t` or `body_t`.
* `Milvus Collection Name`. The name of the collection you created via the [Create Milvus Collection](/docs/5/fusion/reference/config-ref/jobs/create-collections-in-milvus) job, which will store the dense vectors. When creating the collection you specify the type of **Metric** to use to calculate vector similarity.
This stage can be used multiple times to encode additional fields, each into a different Milvus collection.
### Query pipeline setup
#### The Query Fields stage
The first stage is **Query Fields**. For more information see the [Query Fields stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/search-fields-query-stage).
#### The Milvus Query stage
The [Milvus Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-query-stage) stage encodes the query into a vector using the specified model. It then performs a vector similarity search against the specified Milvus collection and returns a list of the best document matches.
* `Model ID`. The ID of the model used when configuring the model training job.
* `Encoder Output Vector`. The name of the output vector from the specified model, which will contain the query encoded as a vector. Defaults to vector.
* `Milvus Collection Name`. The name of the collection that you used in the `Encode into Milvus` index stage to store the encoded vectors.
* `Milvus Results Context Key`. The name of the variable used to store the vector distances. It can be changed as needed. It will be used in the Milvus Ensemble Query Stage to calculate the query score for the document.
* `Number of Results`. The number of highest scoring results returned from Milvus.
This stage would typically be used the same number of times that the `Encode into Milvus` index stage is used, each with a different Milvus collection and a different `Milvus Results Context Key`.
#### The Milvus Ensemble Query stage
The [Milvus Ensemble Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-ensemble-query-stage) takes the results of the Milvus Query stage(s) and calculates the `ensemble score`, which is used to return the best matches.
* `Ensemble math expression`. The mathematical expression used to calculate the `ensemble score`. It should reference the value(s) variable name specified in the `Milvus Results Context Key` parameter in the Milvus Query stage.
* `Result field name`. The name of the field used to store the `ensemble score`. It defaults to `ensemble_score`.
* `Threshold`- A parameter that filters the stage results to remove items that fall below the configured score. Items with a score at, or above, the threshold will be returned.
The **Threshold** feature is only available in Fusion 5.4 and later.
#### The Milvus Response Update Query stage
The [Milvus Response Update Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-response-update-stage) stage does not need to be configured and can be skipped if desired. It inserts the Milvus values, including the `ensemble_score`, into each of the returned documents, which is particularly useful when there is more than one `Milvus Query Stage`. This stage needs to come after the `Solr Query` stage.
## Short answer extraction
By default, the question-answering query pipelines return complete documents that answer questions. Optionally, you can extract just a paragraph, a sentence, or a few words that answer the question.
The [Smart Answers Evaluate Pipeline job](/docs/5/fusion/reference/config-ref/jobs/smart-answers-evaluate-pipeline) evaluates the rankings of results from any [Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/overview) pipeline and finds the best set of weights in the ensemble score. This topic explains how to set up the job.
Before beginning this procedure, prepare a machine learning model using either the Supervised method or the Cold start method, or by selecting one of the pre-trained cold start models, then Configure your pipelines.
The input for this job is a set of test queries and the text or ID of the correct responses. At least 100 entries are needed to obtain useful results. The job compares the test data with Fusion’s actual results and computes variety of the ranking metrics to provide insights of how well the pipeline works. It is also useful to use to compare with other setups or pipelines.
## Prepare test data
1. Format your test data as query/response pairs, that is, a query and its corresponding answer in each row.
You can do this in any format that Fusion support, but parquet file would be preferable to reduce the amount of possible encoding issues.
The response value can be either the document ID of the correct answer in your Fusion index (preferable), or the text of the correct answer.
If you use answer text instead of an ID, make sure that the answer text in the evaluation file is formatted identically to the answer text in Fusion.
If there are multiple possible answers for a unique question, then repeat the questions and put the pair into different rows to make sure each row has exactly one query and one response.
2. If you wish to index test data into Fusion, create a collection for your test data, such as `sa_test_input` and index the test data into that collection.
## Configure the evaluation job
1. If you wish to save the job output in Fusion, create a collection for your evaluation data such as `sa_test_output`.
2. Navigate to **Collections** > **Jobs**.
3. Select **New** > **Smart Answers Evaluate Pipeline** (**Evaluate QnA Pipeline** in Fusion 5.1 and 5.2).
4. Enter a **Job ID**, such as `sa-pipeline-evaluator`.
5. Enter the name of your test data collection (such as `sa_test_input`) in the **Input Evaluation Collection** field.
6. Enter the name of your output collection (such as `sa_test_output`) in the **Output Evaluation Collection** field.
You can also configure this job to read from or write to cloud storage.
7. Enter the name of the **Test Question Field** in the input collection.
8. Enter the name of the answer field as the **Ground Truth Field**.
9. Enter the **App Name** of the Fusion app where the main Smart Answers content is indexed.
10. In the **Main Collection** field, enter the name of the Fusion collection that contains your Smart Answers content.
11. In the **Fusion Query Pipeline** field, enter the name of the Smart Answers query pipeline you want to evaluate.
12. In the **Answer Or ID Field In Fusion** field, enter the name of the field that Fusion will return containing the answer text or answer ID.
13. Optionally, you can configure the **Return Fields** to pass from Smart Answers collection into the evaluation output.
Check the Query Workbench to see which fields are available to be returned.
14. Configure the **Metrics** parameters:
* **Solr Scale Function**
Specify the function used in the Compute Mathematical Expression stage of the query pipeline, one of the following:
* `max`
* `log10`
* `pow0.5`
* **List of Ranking Scores For Ensemble**
To find the best weights for different ranking scores, list the names of the ranking score fields, separated by commas. Different ranking scores might include Solr score, query-to-question distance, or query-to-answer distance from the Compute Mathematical Expression pipeline stage.
* **Target Metric To Use For Weight Selection**
The target ranking metric to optimize during weights selection. The default is `mrr@3`.
15. Optionally, [read about the advanced parameters](/docs/5/fusion/reference/config-ref/jobs/smart-answers-evaluate-pipeline) and consider whether to configure them as well.
For example, **Sampling proportion** and **Sampling seed** provide a way to run the job only on a sample of the test data.
16. Click **Save**.
17\. Click **Run** > **Start**.
## Examine the output
The job provides a variety of metrics (controlled by the **Metrics list** advanced parameter) at different positions (controlled by the **Metrics\@k list** advanced parameter) for the chosen final ranking score (specified in **Ranking score** parameter).
**Example: Pipeline evaluation metrics**
**Example: recall\@1,3,5 for different weights and distances**
In addition to metrics, a results evaluation file is indexed to the specified output evaluation collection. It provides the correct answer position for each test question as well as the top returned results for each field specified in **Return fields** parameter.
# FAQ Solution
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/faq-solution
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/faq-solution
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/faq-solution
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/464
The Supervised solution consists of two parts.
## 1. Train a model
Train a deep learning model using existing query/response pairs by configuring the Smart Answers Supervised Training job. See **Train a Smart Answers Supervised Model**.
The job includes an auto-tuning feature that seeks to identify the optimal configuration for your RNN-based model. Transformer-based models like BERT are not used in the auto-tuning.
The [Supervised solution for Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/faq-solution) begins with training a model using your existing data and the [Smart Answers Supervised Training job](/docs/5/fusion/reference/config-ref/jobs/smart-answers-supervised-training), as explained in this topic. The job includes an auto-tune feature that you can use instead of manually tuning the configuration.
See also [Advanced Model Training Configuration for Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/smart-answers-advanced-model-config).
## Training job requirements
**Storage**
150GB plus 2.5 times the total input data size.
**Processor and memory**
The memory requirements depend on whether you choose GPU or CPU processing:
| GPU | CPU |
| - | - |
|
one core
11GB RAM
|
32 cores
32GB RAM
|
If your training data contains more than 1 million entries, use GPU.
## Prepare the input data
1. Format your input data as question/answer pairs, that is, a query and its corresponding response in each row.
You can do this in any format that Fusion supports.
If there are multiple possible answers for a unique question, then repeat the questions and put the pair into different rows to make sure each row has one question and one answer, as in the example JSON below:
```json wrap theme={"dark"}
[{"question":"How to transfer personal auto lease to business auto lease?","answer":"I would approach the lender that you are getting the lease from..."}
{"question":"How to transfer personal auto lease to business auto lease?","answer":"See what the contract says about transfers or subleases..."}]
```
2. Index the input data in Fusion.
If you wish to have the training data in Fusion, index it into a separate collection for training data such as `model_training_input`. Otherwise you can use it directly from the cloud storage.
## Configure the training job
1. In Fusion, navigate to **Collections** > **Jobs**.
2. Select **Add** > **Smart Answers Supervised Training**:
3. In the **Training Collection** field, specify the input data collection that you created when you [prepared the input data](#prepare-the-input-data).
You can also configure this job to read from or write to cloud storage.
4. Enter the names of the **Question Field** and the **Answer Field** in the training collection.
5. Enter a **Model Deployment Name**.
The new machine learning model will be saved in the blob store with this name. You will reference it later when you configure your pipelines.
6. Configure the **Model base**.
There are several pre-trained word and [BPE](https://nlp.h-its.org/bpemb) embeddings for different languages, as well as a few pre-trained BERT models.
If you want to train custom embeddings, select `word_custom` or `bpe_custom`. This trains Word2vec on the provided data and specified fields. It might be useful in cases when your content includes unusual or domain-specific vocabulary.
If you have content in addition to the query/response pairs that can be used to train the model, then specify it in the **Texts Data Path**.
When you use the pre-trained embeddings, the log shows the percentage of processed vocabulary words. If this value is high, then try using custom embeddings.
The job trains a few (configurable) RNN layers on top of word embeddings or fine-tunes a BERT model on the provided training data. The result model uses an attention mechanism to average word embeddings to obtain the final single dense vector for the content.
Dimension size of vectors for Transformer-based models is 768. For RNN-based models it is 2 times the number units of the last layer. To find the dimension size: download the model, expand the zip, open the log and search for `Encoder output dim size:` line. You might need this information when creating collections in Milvus.
7. *Optional:* Check **Perform auto hyperparameter tuning** to use auto-tune.
Although training module tries to select the most optimal default parameters based on the training data statistics, auto-tune can extend it by automatically finding even better training configuration through hyper-parameter search.
Although this is a resource-intensive operation, it can be useful to identify the best possible RNN-based configuration. Transformer-based models like BERT are not used during auto hyperparameter tuning as they usually perform better yet they are much more expensive on both training and inference time.
8. Click **Save**.
If using solr as the training data source ensure that the source collection contains the `random_*` dynamic field defined in its `managed-schema.xml`. This field is required for sampling the data. If it is not present, add the following entry to the `managed-schema.xml` alongside other dynamic fields `` and `` alongside other field types.
9. Click **Run** > **Start**.
After training is finished the model is deployed into the cluster and can be used in index and query pipelines.
## Next steps
1. See A Smart Answers Supervised Job’s Status and Output
2. [Configure The Smart Answers Pipelines](#3-configure-the-pipelines)
3. [Evaluate a Smart Answers Query Pipeline](#evaluate-the-query-pipeline)
## 2. Create collections in Milvus
In order to use `{app_name}-smart-answers` pipelines, you need to create collections in Milvus. Please, refer to the [Milvus](/docs/5/fusion/intro/fusion-stack/milvus) documentation page.
## 3. Configure the pipelines
The trained model should be used at both index and query time in order to perform dense vector search.
* At index time, we provide the `{app_name}-smart-answers` index pipeline to help generate a dense vector representation of answers.
* At query time, we provide a `{app_name}-smart-answers` query pipeline to conduct run-time neural search. This pipeline transforms the incoming query into a dense vector using the trained model, then compares it with indexed answer dense vectors by computing the cosine distance between them. You can also use a query stage to combine Solr and document vector similarity scores at query time.
See **Configure the Smart Answers Pipelines**. Once your pipelines are configured, see **Evaluate a Smart Answers Query Pipeline** to test its effectiveness so that you can fine-tune the configuration.
Before beginning this procedure, train a machine learning model using either the FAQ method or the cold start method.
Regardless of how you set up your model, the deployment procedure is the same:
1. [Create the `Milvus` collection](#create-the-milvus-collection).
2. [Configure the `smart-answers` index pipeline](#configure-the-index-pipeline).
3. [Configure the `smart-answers` query pipeline](#configure-the-query-pipeline).
See also [Best Practices](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/best-practices), [Advanced Model Training Configuration for Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/smart-answers-advanced-model-config), and Smart Answers Detailed Pipeline Setup.
## Create the Milvus collection
For complete details about job configuration options, see the [Create Collections in Milvus job](/docs/5/fusion/reference/config-ref/jobs/create-collections-in-milvus).
1. Navigate to **Collections** > **Jobs** > **Add +** and select **Create Collections in Milvus**.
2. Configure the job:
1. Enter an ID for this job.
2. Under **Collections**, click **Add**.
3. Enter a collection name.
4. In the **Dimension** field, enter the dimension size of vectors to store in this Milvus collection. The Dimension should match the size of the vectors returned by the encoding model. For example, the `Smart Answers Pre-trained Coldstart` models outputs vectors of 512 dimension size. Dimensionality of encoders trained by `Smart Answers Supervised Training` job depends on the provided parameters and printed in the training job logs.
3. Click **Save**.
The `Create Collections in Milvus` job can be used to create multiple collections at once. In this image, the first collection is used in the [indexing](#configure-the-index-pipeline) and [query](#configure-the-query-pipeline) steps. The other two collections are used in the [example](#pipeline-setup-example).
4. Click **Run** > **Start** to run the job.
## Configure the index pipeline
1. Open the Index Workbench.
2. Load or create your datasource using the default smart-answers index pipeline.
3. Configure the
[Encode into Milvus stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/encode-into-milvus-index-stage):
1. change the value of **Model ID** to match the model deployment name you chose when you configured the model training job.
2. Change `Field to Encode` to the document field name to be processed and encoded into dense vectors.
3. Ensure the `Encoder Output Vector` matches the output vector from the chosen model.
4. Ensure the `Milvus Collection Name` matches the collection name created via the `Create Milvus Collection` job.
To test out your settings, turn on `Fail on Error` in the `Encode into Milvus` stage and Apply the changes. This will cause an error message to display if any settings need to be changed.
4. Save the datasource.
5. Index your data.
## Configure the query pipeline
1. Open the Query Workbench.
2. Load the default smart-answers query pipeline.
3. Configure the [Milvus Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-query-stage):
1. Change the **Model ID** value to match the model deployment name you chose when you configured the model training job.
2. Ensure the `Encoder Output Vector` matches the output vector from the chosen model.
3. Ensure the `Milvus Collection Name` matches the collection name created via the `Create Milvus Collection` job.
4. `Milvus Results Context Key` can be changed as needed. It will be used in the Milvus Ensemble Query Stage to calculate the query score.
4. In the [Milvus Ensemble Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-ensemble-query-stage), update the `Ensemble math expression` as needed based on your model and the name used in the prior stage for the storing the Milvus results.
You can also set the `Threshold` so that the Milvus Ensemble Query Stage will only return items with a score **greater than** or **equal to** the configured value.
5. Save the query pipeline.
## Pipeline Setup Example
### Index and retrieve the question and answer together
To show question and answer together in one document (that is, treat the question as the title and the answer as the description), you can index them together in the same document. You can still use the default `smart-answers` index and query pipelines with a few additional changes.
Prior to configuring the Smart Answers pipelines, use the `Create Milvus Collection` job to create two collections, `question_collection` and `answer_collection`, to store the encoded "questions" and the encoded "answers", respectively.
#### Index Pipeline
As shown in the pictures below, you will need two Encode into Milvus stages, named Encode Question and Encode Answer respectively.
**Encode Question (Encode Into Milvus) stage**
**Encode Answer (Encode Into Milvus) stage**
In the Encode Question stage, specify `Field to Encode` to be `title_t` and change the `Milvus Collection Name` to match the new Milvus collection, `question_collection`.
In the Encode Answer stage, specify `Field to Encode` to be `description_t` and change the `Milvus Collection Name` to match the new Milvus collection, `answer_collection`.
#### Query Pipeline
Since we have two dense vectors generated during indexing, at query time we need to compute both query to question distance and query to answer distance. This can be set up as the pictures shown below with two Milvus Query Stages, one for each of the two Milvus collections.
To store those two distances separately, the `Milvus Results Context Key` needs to be different in each of these two stages.
In the Query Questions stage, we set the `Milvus Results Context Key` to `milvus_questions` and the Milvus collection name to `question_collection`.
Query Questions (Milvus Query) stage:
In the Query Answers stage, we set the `Milvus Results Context Key` to `milvus_answers` and the Milvus collection name to `answer_collection`.
Query Answers (Milvus Query) stage:
Now we can ensemble them together with the Milvus Ensemble Query Stage with the `Ensemble math expression` combining the results from the two query stages. If we want the question scores and answer scores weighted equally, we would use: `0.5 * milvus_questions + 0.5 * milvus_answers`.
This is recommended especially when you have limited FAQ dataset and want to utilize both question and answer information.
**Milvus Ensemble Query stage**
## Evaluate the query pipeline
The [Evaluate QnA Pipeline job](/docs/5/fusion/reference/config-ref/jobs/smart-answers-evaluate-pipeline) evaluates the rankings of results from any Smart Answers pipeline and finds the best set of weights in the ensemble score.
## Detailed pipeline setup
Typically, you can use the [default pipelines](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/overview) included with Fusion AI. These pipelines now utilize Milvus to store encoded vectors and to calculate vector similarity. This topic provides information you can use to customize the Smart Answers pipelines.
| | | |
| - | - | - |
| **"smart-answers" index pipeline** | | [Encode into Milvus stage](#the-encode-into-milvus-index-stage) |
| **"smart-answers" query pipeline** | |
|
### Create the Milvus collection
Prior to indexing data, the [Create Collections in Milvus job](/docs/5/fusion/reference/config-ref/jobs/create-collections-in-milvus) can be used to create the Milvus collection(s) used by the Smart Answers pipelines (see [Milvus overview](/docs/5/fusion/intro/fusion-stack/milvus)).
* `Job ID`. A unique identifier for the job.
* `Collection Name`. A name for the Milvus collection you are creating. This name is used in both the Smart Answer Index and the Smart Answer Query pipelines.
* `Dimension`. The dimension size of the vectors to store in this Milvus collection. The Dimension should match the size of the vectors returned by the encryption model. For example, if the model was created with either the `Smart Answers Coldstart Training` job or the `Smart Answers Supervised Training` job with the Model Base `word_en_300d_2M`, then the dimension would be 300.
* `Index file size`. Files with more documents than this will cause Milvus to build an index on this collection.
* `Metric`. The type of metric used to calculate vector similarity scores. `Inner Product` is recommended. It produces values between 0 and 1, where a higher value means higher similarity.
### Index pipeline setup
**Stages in the default "smart-answers" index pipeline**
Only one custom index stage needs to be configured in your index pipeline, the [Encode into Milvus index stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/encode-into-milvus-index-stage).
#### The Encode into Milvus Index Stage
If you are using a dynamic schema, make sure this stage is added *after* the [Solr Dynamic Field Name Mapping stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/solr-dynamic-field-name-mapping-index-stage).
The [Encode into Milvus](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/encode-into-milvus-index-stage) index stage uses the specified model to encode the `Field to Encode` and store it in Milvus in the given Milvus collection.
There are several required parameters:
* `Model ID`. The ID of the model.
* `Encoder Output Vector`. The name of the field that stores the compressed dense vectors output from the model. Default value: `vector`.
* `Field to Encode`. The text field to encode into a dense vector, such as `answer_t` or `body_t`.
* `Milvus Collection Name`. The name of the collection you created via the [Create Milvus Collection](/docs/5/fusion/reference/config-ref/jobs/create-collections-in-milvus) job, which will store the dense vectors. When creating the collection you specify the type of **Metric** to use to calculate vector similarity.
This stage can be used multiple times to encode additional fields, each into a different Milvus collection.
### Query pipeline setup
#### The Query Fields stage
The first stage is **Query Fields**. For more information see the [Query Fields stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/search-fields-query-stage).
#### The Milvus Query stage
The [Milvus Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-query-stage) stage encodes the query into a vector using the specified model. It then performs a vector similarity search against the specified Milvus collection and returns a list of the best document matches.
* `Model ID`. The ID of the model used when configuring the model training job.
* `Encoder Output Vector`. The name of the output vector from the specified model, which will contain the query encoded as a vector. Defaults to vector.
* `Milvus Collection Name`. The name of the collection that you used in the `Encode into Milvus` index stage to store the encoded vectors.
* `Milvus Results Context Key`. The name of the variable used to store the vector distances. It can be changed as needed. It will be used in the Milvus Ensemble Query Stage to calculate the query score for the document.
* `Number of Results`. The number of highest scoring results returned from Milvus.
This stage would typically be used the same number of times that the `Encode into Milvus` index stage is used, each with a different Milvus collection and a different `Milvus Results Context Key`.
#### The Milvus Ensemble Query stage
The [Milvus Ensemble Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-ensemble-query-stage) takes the results of the Milvus Query stage(s) and calculates the `ensemble score`, which is used to return the best matches.
* `Ensemble math expression`. The mathematical expression used to calculate the `ensemble score`. It should reference the value(s) variable name specified in the `Milvus Results Context Key` parameter in the Milvus Query stage.
* `Result field name`. The name of the field used to store the `ensemble score`. It defaults to `ensemble_score`.
* `Threshold`- A parameter that filters the stage results to remove items that fall below the configured score. Items with a score at, or above, the threshold will be returned.
The **Threshold** feature is only available in Fusion 5.4 and later.
#### The Milvus Response Update Query stage
The [Milvus Response Update Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-response-update-stage) stage does not need to be configured and can be skipped if desired. It inserts the Milvus values, including the `ensemble_score`, into each of the returned documents, which is particularly useful when there is more than one `Milvus Query Stage`. This stage needs to come after the `Solr Query` stage.
## Short answer extraction
By default, the question-answering query pipelines return complete documents that answer questions. Optionally, you can extract just a paragraph, a sentence, or a few words that answer the question.
The [Smart Answers Evaluate Pipeline job](/docs/5/fusion/reference/config-ref/jobs/smart-answers-evaluate-pipeline) evaluates the rankings of results from any [Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/overview) pipeline and finds the best set of weights in the ensemble score. This topic explains how to set up the job.
Before beginning this procedure, prepare a machine learning model using either the Supervised method or the Cold start method, or by selecting one of the pre-trained cold start models, then Configure your pipelines.
The input for this job is a set of test queries and the text or ID of the correct responses. At least 100 entries are needed to obtain useful results. The job compares the test data with Fusion’s actual results and computes variety of the ranking metrics to provide insights of how well the pipeline works. It is also useful to use to compare with other setups or pipelines.
## Prepare test data
1. Format your test data as query/response pairs, that is, a query and its corresponding answer in each row.
You can do this in any format that Fusion support, but parquet file would be preferable to reduce the amount of possible encoding issues.
The response value can be either the document ID of the correct answer in your Fusion index (preferable), or the text of the correct answer.
If you use answer text instead of an ID, make sure that the answer text in the evaluation file is formatted identically to the answer text in Fusion.
If there are multiple possible answers for a unique question, then repeat the questions and put the pair into different rows to make sure each row has exactly one query and one response.
2. If you wish to index test data into Fusion, create a collection for your test data, such as `sa_test_input` and index the test data into that collection.
## Configure the evaluation job
1. If you wish to save the job output in Fusion, create a collection for your evaluation data such as `sa_test_output`.
2. Navigate to **Collections** > **Jobs**.
3. Select **New** > **Smart Answers Evaluate Pipeline** (**Evaluate QnA Pipeline** in Fusion 5.1 and 5.2).
4. Enter a **Job ID**, such as `sa-pipeline-evaluator`.
5. Enter the name of your test data collection (such as `sa_test_input`) in the **Input Evaluation Collection** field.
6. Enter the name of your output collection (such as `sa_test_output`) in the **Output Evaluation Collection** field.
You can also configure this job to read from or write to cloud storage.
7. Enter the name of the **Test Question Field** in the input collection.
8. Enter the name of the answer field as the **Ground Truth Field**.
9. Enter the **App Name** of the Fusion app where the main Smart Answers content is indexed.
10. In the **Main Collection** field, enter the name of the Fusion collection that contains your Smart Answers content.
11. In the **Fusion Query Pipeline** field, enter the name of the Smart Answers query pipeline you want to evaluate.
12. In the **Answer Or ID Field In Fusion** field, enter the name of the field that Fusion will return containing the answer text or answer ID.
13. Optionally, you can configure the **Return Fields** to pass from Smart Answers collection into the evaluation output.
Check the Query Workbench to see which fields are available to be returned.
14. Configure the **Metrics** parameters:
* **Solr Scale Function**
Specify the function used in the Compute Mathematical Expression stage of the query pipeline, one of the following:
* `max`
* `log10`
* `pow0.5`
* **List of Ranking Scores For Ensemble**
To find the best weights for different ranking scores, list the names of the ranking score fields, separated by commas. Different ranking scores might include Solr score, query-to-question distance, or query-to-answer distance from the Compute Mathematical Expression pipeline stage.
* **Target Metric To Use For Weight Selection**
The target ranking metric to optimize during weights selection. The default is `mrr@3`.
15. Optionally, [read about the advanced parameters](/docs/5/fusion/reference/config-ref/jobs/smart-answers-evaluate-pipeline) and consider whether to configure them as well.
For example, **Sampling proportion** and **Sampling seed** provide a way to run the job only on a sample of the test data.
16. Click **Save**.
17\. Click **Run** > **Start**.
## Examine the output
The job provides a variety of metrics (controlled by the **Metrics list** advanced parameter) at different positions (controlled by the **Metrics\@k list** advanced parameter) for the chosen final ranking score (specified in **Ranking score** parameter).
**Example: Pipeline evaluation metrics**
**Example: recall\@1,3,5 for different weights and distances**
In addition to metrics, a results evaluation file is indexed to the specified output evaluation collection. It provides the correct answer position for each test question as well as the top returned results for each field specified in **Return fields** parameter.
# Smart Answers
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/461
Fusion Smart Answers brings the benefits of a versatile, scalable Semantic Search platform to provide cutting edge relevancy for your applications.
This system makes use of advanced Deep Learning techniques to provide the power of Neural dense vectors search.
Semantically rich models encode queries and documents into vectors in the same digital vector space in such way that query and the most relevant information are located near each other.
It pushes search boundaries beyond classical token-based matching mechanisms by leveraging the contextual information and query understanding.
Virtual assistants and Chatbots can incorporate Smart Answers to enable self-service for employees and customers and save cost on answering incoming queries.
E-commerce can utilize Smart Answers to handle zero-result search queries and improve the overall relevancy by recommending products that are most likely of users interest.
Training data can be explicitly provided as query/response pairs or constructed from the collected signals data.
Even if you do not have an existing set of recorded interactions, you can rely on Smart Answers cold start solution. It uses various pre-trained models to get out-of-the-box (OOTB) semantic capabilities as well as provides a possibility to train unsupervised models for specific domains.
These features bring traditional search relevancy development and Data Science together into an easy-to-use configuration framework that leverages Fusion’s indexing and querying pipelines.
The course for **Smart Answers** focuses on how to utilize Smart Answers to go beyond classical token-based matching mechanisms by leveraging contextual information and query understanding.
## Example business use cases
#### Call center or IT support
* Embed this system as a self-help feature on your Help page or Contact Us page to reduce the call center load.
* Make it available to your customer support team to find the answers to already-solved problems.
#### Zero-result search problem in E-commerce:
Queries that lead to zero-results is a huge problem for E-commerce that leads to income loss and decrease of the overall user experience. Semantic search does not have this problem as it does not operate on the exact token matches as classical search. Instead search is done in the vector space which can find relevant products even if there is no token overlapping.
#### Questions about products for E-commerce:
E-commerce websites can use this system to search "how to" content, product user manuals, or existing product questions. For example, amazon.com provides a search function on questions about each product.
#### Search in Slack, email conversations, or SharePoint FAQ docs
You can achieve fast knowledge extraction by applying this solution to these types of knowledge repositories.
#### Improve search for long queries
As the solution utilizes state of the art Deep Learning techniques, it is able to capture semantic and contextual information into query understanding. This makes it very suitable to work with long queries or natural language questions.
## Get started with Smart Answers
To get started, you need a trained machine learning model. There are two methods for building a model, depending on the kind of data you already have.
Each method requires a slightly different model training procedure, but the model deployment procedure is the same for both.
To implement Smart Answers, follow this workflow:
1. Train or install a machine learning model. This process differs depending on whether you use the Smart Answers supervised solution or the Smart Answers cold start solution.
* Use the supervised solution when you already have a collection of query/response pairs or if you can construct such a dataset from the signals data. The input is a dataset of query/response pairs. See **Train a Smart Answers Supervised Model**.
The [Supervised solution for Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/faq-solution) begins with training a model using your existing data and the [Smart Answers Supervised Training job](/docs/5/fusion/reference/config-ref/jobs/smart-answers-supervised-training), as explained in this topic. The job includes an auto-tune feature that you can use instead of manually tuning the configuration.
See also [Advanced Model Training Configuration for Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/smart-answers-advanced-model-config).
## Training job requirements
**Storage**
150GB plus 2.5 times the total input data size.
**Processor and memory**
The memory requirements depend on whether you choose GPU or CPU processing:
| GPU | CPU |
| - | - |
|
one core
11GB RAM
|
32 cores
32GB RAM
|
If your training data contains more than 1 million entries, use GPU.
## Prepare the input data
1. Format your input data as question/answer pairs, that is, a query and its corresponding response in each row.
You can do this in any format that Fusion supports.
If there are multiple possible answers for a unique question, then repeat the questions and put the pair into different rows to make sure each row has one question and one answer, as in the example JSON below:
```json wrap theme={"dark"}
[{"question":"How to transfer personal auto lease to business auto lease?","answer":"I would approach the lender that you are getting the lease from..."}
{"question":"How to transfer personal auto lease to business auto lease?","answer":"See what the contract says about transfers or subleases..."}]
```
2. Index the input data in Fusion.
If you wish to have the training data in Fusion, index it into a separate collection for training data such as `model_training_input`. Otherwise you can use it directly from the cloud storage.
## Configure the training job
1. In Fusion, navigate to **Collections** > **Jobs**.
2. Select **Add** > **Smart Answers Supervised Training**:
3. In the **Training Collection** field, specify the input data collection that you created when you [prepared the input data](#prepare-the-input-data).
You can also configure this job to read from or write to cloud storage.
4. Enter the names of the **Question Field** and the **Answer Field** in the training collection.
5. Enter a **Model Deployment Name**.
The new machine learning model will be saved in the blob store with this name. You will reference it later when you configure your pipelines.
6. Configure the **Model base**.
There are several pre-trained word and [BPE](https://nlp.h-its.org/bpemb) embeddings for different languages, as well as a few pre-trained BERT models.
If you want to train custom embeddings, select `word_custom` or `bpe_custom`. This trains Word2vec on the provided data and specified fields. It might be useful in cases when your content includes unusual or domain-specific vocabulary.
If you have content in addition to the query/response pairs that can be used to train the model, then specify it in the **Texts Data Path**.
When you use the pre-trained embeddings, the log shows the percentage of processed vocabulary words. If this value is high, then try using custom embeddings.
The job trains a few (configurable) RNN layers on top of word embeddings or fine-tunes a BERT model on the provided training data. The result model uses an attention mechanism to average word embeddings to obtain the final single dense vector for the content.
Dimension size of vectors for Transformer-based models is 768. For RNN-based models it is 2 times the number units of the last layer. To find the dimension size: download the model, expand the zip, open the log and search for `Encoder output dim size:` line. You might need this information when creating collections in Milvus.
7. *Optional:* Check **Perform auto hyperparameter tuning** to use auto-tune.
Although training module tries to select the most optimal default parameters based on the training data statistics, auto-tune can extend it by automatically finding even better training configuration through hyper-parameter search.
Although this is a resource-intensive operation, it can be useful to identify the best possible RNN-based configuration. Transformer-based models like BERT are not used during auto hyperparameter tuning as they usually perform better yet they are much more expensive on both training and inference time.
8. Click **Save**.
If using solr as the training data source ensure that the source collection contains the `random_*` dynamic field defined in its `managed-schema.xml`. This field is required for sampling the data. If it is not present, add the following entry to the `managed-schema.xml` alongside other dynamic fields `` and \ alongside other field types.
9. Click **Run** > **Start**.
After training is finished the model is deployed into the cluster and can be used in index and query pipelines.
## Next steps
1. See A Smart Answers Supervised Job’s Status and Output
2. Configure The Smart Answers Pipelines
3. Evaluate a Smart Answers Query Pipeline
* Use the cold start solution when you have no historical training data or fewer than 200 query/response pairs. Each method requires a slightly different model training procedure, but the model deployment procedure is the same for both.
* If you have a pre-trained cold start model, see **Set Up a Pre-Trained Cold Start Model for Smart Answers**, which deploys the existing model into Fusion.
* If you have a body of content that can be used for unsupervised training, see **Train a Smart Answers Cold Start Model**.
Lucidworks provides these pre-trained cold start models for [Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/overview):
* `qna-coldstart-large` - this is a large model trained on variety of corpuses and tasks.
* `qna-coldstart-multilingual` - covers 16 languages. List of supported languages: Arabic, Chinese-simplified, Chinese-traditional, English, French, German, Italian, Japanese, Korean, Dutch, Polish, Portuguese, Spanish, Thai, Turkish, Russian.
When you use these models, you do not need to run the model training job. Instead, you run a job that deploys the model into Fusion. The [Create Seldon Core Model Deployment job](/docs/5/fusion/reference/config-ref/jobs/create-seldon-core-model-deployment) deploys your model as a Docker image in Kubernetes, which you can scale up or down like other Fusion services.
These models are a good basis for a cold start solution if your data does not contain much domain-specific terminology. Otherwise, consider training a model using your existing content.
Dimension size of vectors for both models is **512**. You might need this information when creating collections in Milvus.
## Deploy a pre-trained cold-start model into Fusion
The pre-trained cold-start models are deployed using a Fusion job called [Create Seldon Core Model Deployment](/docs/5/fusion/reference/config-ref/jobs/create-seldon-core-model-deployment). This job downloads the selected pre-trained model and installs it in Fusion.
1. Navigate to **Collections** > **Jobs**.
2. Select **Add** > **Create Seldon Core Model Deployment**.
3. Enter a **Job ID**, such as `deploy-qna-coldstart-multilingual` or `deploy-qna-coldstart-large`.
4. Enter the **Model Name**, one of the following:
* `qna-coldstart-multilingual`
* `qna-coldstart-large`
5. In the **Docker Repository** field, enter `lucidworks`.
6. In the **Image Name** field, enter one of the following:
* `qna-coldstart-multilingual:v1.1`
* `qna-coldstart-large:v1.1`
7. Leave the **Kubernetes Secret Name for Model Repo** field empty.
8. In the **Output Column Names for Model** field, enter one of the following:
* `qna-coldstart-multilingual:[vector]`
* `qna-coldstart-large:[vector, compressed_vector]`
9. Click **Save**.
10. Click **Run** > **Start** to start the deployment job.
## Next steps
1. Configure The Smart Answers Pipelines
2. Evaluate a Smart Answers Query Pipeline
The Smart Answers Cold Start Training job is deprecated in Fusion 5.12.
The [cold start solution for Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/cold-start) begins with training a model using your existing content. To do this, you run the [Smart Answers Coldstart Training](/docs/5/fusion/reference/config-ref/jobs/smart-answers-coldstart-training) job. This job uses variety of word embeddings, including custom via Word2Vec training, to learn about the vocabulary that you want to search against.
Smart Answers comes with two pre-trained cold-start models. If your data does not have many domain-specific words, then consider using a pre-trained model.
During a cold start, we suggest capturing user feedback such as document clicks, likes, and downloads on the Web site [App Studio](/docs/4/app-studio/overview) can help you get started. After accumulating feedback data and at least 3,000 query/response pairs, the feedback can be used to train a model using the Supervised method.
## Configure the training job
1. In Fusion, navigate to **Collections** > **Jobs**.
2. Select **Add** > **Smart Answer Coldstart Training**.
3. In the **Training Collection** field, specify the collection that contains the content that can be used to answer questions.
You can also configure this job to read from or write to cloud storage.
4. Enter the name of the **Field which contains the content documents**.
5. Enter a **Model Deployment Name**.
The new machine learning model will be saved in the blob store with this name. You will reference it later when you configure your pipelines.
6. Configure the **Model base**.
There are several pre-trained word and [BPE](https://nlp.h-its.org/bpemb) embeddings for different languages, as well as a few pre-trained BERT models.
If you want to train custom embeddings, please select `word_custom` or `bpe_custom`.
This trains Word2vec on the data and fields specified in **Training collection** and **Field which contains the content documents**. It might be useful in cases when your content includes unusual or domain-specific vocabulary.
When you use the pre-trained embeddings, the log shows the percentage of processed vocabulary words. If this value is high, then try using custom embeddings.
During the training job analyzes the content data to select weights for each of the words. The result model performs the weighted average of word embeddings to obtain final single dense vector for the content.
7. Click **Save**.
If using solr as the training data source ensure that the source collection contains the `random_*` dynamic field defined in its `managed-schema.xml`. This field is required for sampling the data. If it is not present, add the following entry to the `managed-schema.xml` alongside other dynamic fields `` and `` alongside other field types.
8. Click **Run** > **Start**.
After training is finished the model is deployed into the cluster and can be used in index and query pipelines.
## Next steps
1. Configure The Smart Answers Pipelines
2. Evaluate a Smart Answers Query Pipeline
2. Configure the index and query pipelines.\
Fusion includes default pipelines to get you started. See **Configure the Smart Answers Pipelines**.
Before beginning this procedure, train a machine learning model using either the FAQ method or the cold start method.
Regardless of how you set up your model, the deployment procedure is the same:
1. Create the `Milvus` collection.
2. Configure the `smart-answers` index pipeline.
3. Configure the `smart-answers` query pipeline.
See also [Best Practices](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/best-practices) and [Advanced Model Training Configuration for Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/smart-answers-advanced-model-config).
## Create the Milvus collection
For complete details about job configuration options, see the [Create Collections in Milvus job](/docs/5/fusion/reference/config-ref/jobs/create-collections-in-milvus).
1. Navigate to **Collections** > **Jobs** > **Add +** and select **Create Collections in Milvus**.
2. Configure the job:
1. Enter an ID for this job.
2. Under **Collections**, click **Add**.
3. Enter a collection name.
4. In the **Dimension** field, enter the dimension size of vectors to store in this Milvus collection. The Dimension should match the size of the vectors returned by the encoding model. For example, the `Smart Answers Pre-trained Coldstart` models outputs vectors of 512 dimension size. Dimensionality of encoders trained by `Smart Answers Supervised Training` job depends on the provided parameters and printed in the training job logs.
3. Click **Save**.
The `Create Collections in Milvus` job can be used to create multiple collections at once. In this image, the first collection is used in the [indexing](#configure-the-index-pipeline) and [query](#configure-the-query-pipeline) steps. The other two collections are used in the [example](#pipeline-setup-example).
4. Click **Run** > **Start** to run the job.
## Configure the index pipeline
1. Open the Index Workbench.
2. Load or create your datasource using the default smart-answers index pipeline.
3. Configure the
[Encode into Milvus stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/encode-into-milvus-index-stage):
1. change the value of **Model ID** to match the model deployment name you chose when you configured the model training job.
2. Change `Field to Encode` to the document field name to be processed and encoded into dense vectors.
3. Ensure the `Encoder Output Vector` matches the output vector from the chosen model.
4. Ensure the `Milvus Collection Name` matches the collection name created via the `Create Milvus Collection` job.
To test out your settings, turn on `Fail on Error` in the `Encode into Milvus` stage and Apply the changes. This will cause an error message to display if any settings need to be changed.
4. Save the datasource.
5. Index your data.
## Configure the query pipeline
1. Open the Query Workbench.
2. Load the default smart-answers query pipeline.
3. Configure the [Milvus Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-query-stage):
1. Change the **Model ID** value to match the model deployment name you chose when you configured the model training job.
2. Ensure the `Encoder Output Vector` matches the output vector from the chosen model.
3. Ensure the `Milvus Collection Name` matches the collection name created via the `Create Milvus Collection` job.
4. `Milvus Results Context Key` can be changed as needed. It will be used in the Milvus Ensemble Query Stage to calculate the query score.
4. In the [Milvus Ensemble Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-ensemble-query-stage), update the `Ensemble math expression` as needed based on your model and the name used in the prior stage for the storing the Milvus results.
In versions 5.4 and later, you can also set the `Threshold` so that the Milvus Ensemble Query Stage will only return items with a score **greater than** or **equal to** the configured value.
5. Save the query pipeline.
## Pipeline Setup Example
### Index and retrieve the question and answer together
To show question and answer together in one document (that is, treat the question as the title and the answer as the description), you can index them together in the same document. You can still use the default `smart-answers` index and query pipelines with a few additional changes.
Prior to configuring the Smart Answers pipelines, use the `Create Milvus Collection` job to create two collections, `question_collection` and `answer_collection`, to store the encoded "questions" and the encoded "answers", respectively.
#### Index Pipeline
As shown in the pictures below, you will need two Encode into Milvus stages, named Encode Question and Encode Answer respectively.
**Encode Question (Encode Into Milvus) stage**
**Encode Answer (Encode Into Milvus) stage**
In the Encode Question stage, specify `Field to Encode` to be `title_t` and change the `Milvus Collection Name` to match the new Milvus collection, `question_collection`.
In the Encode Answer stage, specify `Field to Encode` to be `description_t` and change the `Milvus Collection Name` to match the new Milvus collection, `answer_collection`.
#### Query Pipeline
Since we have two dense vectors generated during indexing, at query time we need to compute both query to question distance and query to answer distance. This can be set up as the pictures shown below with two Milvus Query Stages, one for each of the two Milvus collections.
To store those two distances separately, the `Milvus Results Context Key` needs to be different in each of these two stages.
In the Query Questions stage, we set the `Milvus Results Context Key` to `milvus_questions` and the Milvus collection name to `question_collection`.
Query Questions (Milvus Query) stage:
In the Query Answers stage, we set the `Milvus Results Context Key` to `milvus_answers` and the Milvus collection name to `answer_collection`.
Query Answers (Milvus Query) stage:
Now we can ensemble them together with the Milvus Ensemble Query Stage with the `Ensemble math expression` combining the results from the two query stages. If we want the question scores and answer scores weighted equally, we would use: `0.5 * milvus_questions + 0.5 * milvus_answers`.
This is recommended especially when you have limited FAQ dataset and want to utilize both question and answer information.
**Milvus Ensemble Query stage**
## Evaluate the query pipeline
The [Evaluate QnA Pipeline job](/docs/5/fusion/reference/config-ref/jobs/smart-answers-evaluate-pipeline) evaluates the rankings of results from any Smart Answers pipeline and finds the best set of weights in the ensemble score.
## Detailed pipeline setup
Typically, you can use the [default pipelines](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/overview) included with Fusion AI. These pipelines now utilize Milvus to store encoded vectors and to calculate vector similarity. This topic provides information you can use to customize the Smart Answers pipelines.
| | | |
| - | - | - |
| **"smart-answers" index pipeline** | | [Encode into Milvus stage](#the-encode-into-milvus-index-stage) |
| **"smart-answers" query pipeline** | |
|
### Create the Milvus collection
Prior to indexing data, the [Create Collections in Milvus job](/docs/5/fusion/reference/config-ref/jobs/create-collections-in-milvus) can be used to create the Milvus collection(s) used by the Smart Answers pipelines (see [Milvus overview](/docs/5/fusion/intro/fusion-stack/milvus)).
* `Job ID`. A unique identifier for the job.
* `Collection Name`. A name for the Milvus collection you are creating. This name is used in both the Smart Answer Index and the Smart Answer Query pipelines.
* `Dimension`. The dimension size of the vectors to store in this Milvus collection. The Dimension should match the size of the vectors returned by the encryption model. For example, if the model was created with either the `Smart Answers Coldstart Training` job or the `Smart Answers Supervised Training` job with the Model Base `word_en_300d_2M`, then the dimension would be 300.
* `Index file size`. Files with more documents than this will cause Milvus to build an index on this collection.
* `Metric`. The type of metric used to calculate vector similarity scores. `Inner Product` is recommended. It produces values between 0 and 1, where a higher value means higher similarity.
### Index pipeline setup
**Stages in the default "smart-answers" index pipeline**
Only one custom index stage needs to be configured in your index pipeline, the [Encode into Milvus index stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/encode-into-milvus-index-stage).
#### The Encode into Milvus Index Stage
If you are using a dynamic schema, make sure this stage is added *after* the [Solr Dynamic Field Name Mapping stage](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/solr-dynamic-field-name-mapping-index-stage).
The [Encode into Milvus](/docs/5/fusion/reference/config-ref/pipeline-stages/index-stages/encode-into-milvus-index-stage) index stage uses the specified model to encode the `Field to Encode` and store it in Milvus in the given Milvus collection.
There are several required parameters:
* `Model ID`. The ID of the model.
* `Encoder Output Vector`. The name of the field that stores the compressed dense vectors output from the model. Default value: `vector`.
* `Field to Encode`. The text field to encode into a dense vector, such as `answer_t` or `body_t`.
* `Milvus Collection Name`. The name of the collection you created via the [Create Milvus Collection](/docs/5/fusion/reference/config-ref/jobs/create-collections-in-milvus) job, which will store the dense vectors. When creating the collection you specify the type of **Metric** to use to calculate vector similarity.
This stage can be used multiple times to encode additional fields, each into a different Milvus collection.
### Query pipeline setup
#### The Query Fields stage
The first stage is **Query Fields**. For more information see the [Query Fields stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/search-fields-query-stage).
#### The Milvus Query stage
The [Milvus Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-query-stage) stage encodes the query into a vector using the specified model. It then performs a vector similarity search against the specified Milvus collection and returns a list of the best document matches.
* `Model ID`. The ID of the model used when configuring the model training job.
* `Encoder Output Vector`. The name of the output vector from the specified model, which will contain the query encoded as a vector. Defaults to vector.
* `Milvus Collection Name`. The name of the collection that you used in the `Encode into Milvus` index stage to store the encoded vectors.
* `Milvus Results Context Key`. The name of the variable used to store the vector distances. It can be changed as needed. It will be used in the Milvus Ensemble Query Stage to calculate the query score for the document.
* `Number of Results`. The number of highest scoring results returned from Milvus.
This stage would typically be used the same number of times that the `Encode into Milvus` index stage is used, each with a different Milvus collection and a different `Milvus Results Context Key`.
#### The Milvus Ensemble Query stage
The [Milvus Ensemble Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-ensemble-query-stage) takes the results of the Milvus Query stage(s) and calculates the `ensemble score`, which is used to return the best matches.
* `Ensemble math expression`. The mathematical expression used to calculate the `ensemble score`. It should reference the value(s) variable name specified in the `Milvus Results Context Key` parameter in the Milvus Query stage.
* `Result field name`. The name of the field used to store the `ensemble score`. It defaults to `ensemble_score`.
* `Threshold`- A parameter that filters the stage results to remove items that fall below the configured score. Items with a score at, or above, the threshold will be returned.
The **Threshold** feature is only available in Fusion 5.4 and later.
#### The Milvus Response Update Query stage
The [Milvus Response Update Query](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/milvus-response-update-stage) stage does not need to be configured and can be skipped if desired. It inserts the Milvus values, including the `ensemble_score`, into each of the returned documents, which is particularly useful when there is more than one `Milvus Query Stage`. This stage needs to come after the `Solr Query` stage.
## Short answer extraction
By default, the question-answering query pipelines return complete documents that answer questions. Optionally, you can extract just a paragraph, a sentence, or a few words that answer the question.
3. Evaluate the query pipeline’s effectiveness.\
Fine tune your query pipeline configuration by running a job that analyzes its effectiveness. See **Evaluate a Smart Answers Query Pipeline**.
The [Smart Answers Evaluate Pipeline job](/docs/5/fusion/reference/config-ref/jobs/smart-answers-evaluate-pipeline) evaluates the rankings of results from any [Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/overview) pipeline and finds the best set of weights in the ensemble score. This topic explains how to set up the job.
Before beginning this procedure, prepare a machine learning model using either the Supervised method or the Cold start method, or by selecting one of the pre-trained cold start models, then Configure your pipelines.
The input for this job is a set of test queries and the text or ID of the correct responses. At least 100 entries are needed to obtain useful results. The job compares the test data with Fusion’s actual results and computes variety of the ranking metrics to provide insights of how well the pipeline works. It is also useful to use to compare with other setups or pipelines.
## Prepare test data
1. Format your test data as query/response pairs, that is, a query and its corresponding answer in each row.
You can do this in any format that Fusion support, but parquet file would be preferable to reduce the amount of possible encoding issues.
The response value can be either the document ID of the correct answer in your Fusion index (preferable), or the text of the correct answer.
If you use answer text instead of an ID, make sure that the answer text in the evaluation file is formatted identically to the answer text in Fusion.
If there are multiple possible answers for a unique question, then repeat the questions and put the pair into different rows to make sure each row has exactly one query and one response.
2. If you wish to index test data into Fusion, create a collection for your test data, such as `sa_test_input` and index the test data into that collection.
## Configure the evaluation job
1. If you wish to save the job output in Fusion, create a collection for your evaluation data such as `sa_test_output`.
2. Navigate to **Collections** > **Jobs**.
3. Select **New** > **Smart Answers Evaluate Pipeline** (**Evaluate QnA Pipeline** in Fusion 5.1 and 5.2).
4. Enter a **Job ID**, such as `sa-pipeline-evaluator`.
5. Enter the name of your test data collection (such as `sa_test_input`) in the **Input Evaluation Collection** field.
6. Enter the name of your output collection (such as `sa_test_output`) in the **Output Evaluation Collection** field.
In Fusion 5.3 and later, you can also configure this job to read from or write to cloud storage.
7. Enter the name of the **Test Question Field** in the input collection.
8. Enter the name of the answer field as the **Ground Truth Field**.
9. Enter the **App Name** of the Fusion app where the main Smart Answers content is indexed.
10. In the **Main Collection** field, enter the name of the Fusion collection that contains your Smart Answers content.
11. In the **Fusion Query Pipeline** field, enter the name of the Smart Answers query pipeline you want to evaluate.
12. In the **Answer Or ID Field In Fusion** field, enter the name of the field that Fusion will return containing the answer text or answer ID.
13. Optionally, you can configure the **Return Fields** to pass from Smart Answers collection into the evaluation output.
Check the Query Workbench to see which fields are available to be returned.
14. Configure the **Metrics** parameters:
* **Solr Scale Function**\
Specify the function used in the Compute Mathematical Expression stage of the query pipeline, one of the following:
* `max`
* `log10`
* `pow0.5`
* **List of Ranking Scores For Ensemble**\
To find the best weights for different ranking scores, list the names of the ranking score fields, separated by commas. Different ranking scores might include Solr score, query-to-question distance, or query-to-answer distance from the Compute Mathematical Expression pipeline stage.
* **Target Metric To Use For Weight Selection**\
The target ranking metric to optimize during weights selection. The default is `mrr@3`.
15. Optionally, [read about the advanced parameters](/docs/5/fusion/reference/config-ref/jobs/smart-answers-evaluate-pipeline) and consider whether to configure them as well.
For example, **Sampling proportion** and **Sampling seed** provide a way to run the job only on a sample of the test data.
16\. Click **Save**.
17\. Click **Run** > **Start**.
## Examine the output
The job provides a variety of metrics (controlled by the **Metrics list** advanced parameter) at different positions (controlled by the **Metrics\@k list** advanced parameter) for the chosen final ranking score (specified in **Ranking score** parameter).
**Example: Pipeline evaluation metrics**
**Example: recall\@1,3,5 for different weights and distances**
In addition to metrics, a results evaluation file is indexed to the specified output evaluation collection. It provides the correct answer position for each test question as well as the top returned results for each field specified in **Return fields** parameter.
4. Check the job status and output.
When your job has completed, you can check the job's status through the ML Model service logs. See **View a Smart Answers (QnA) Job's Status and Output**.
This topic explains how you can track the running steps of the following jobs:
* Smart Answers Supervised Training
* Smart Answers Coldstart Training
* Smart Answers Evaluate Pipeline
The ML Model service logs provide information on data pre-processing, training steps, evaluations, and model generation for those jobs. Follow the instructions below to access the logs.
1. Find the name of the ML Model service pod:
```bash theme={"dark"}
kubectl get pods -n | grep ml-model-service-ui
```
The pod name looks like `-ml-model-service-ui--`, as in `f5-ml-model-service-ui-547dd78d6-p9d6q`.
2. Set up port forwarding:
```bash theme={"dark"}
kubectl port-forward -ml-model-service-ui-- 8001:8001 -n
```
3. In the Fusion UI, navigate to **Collections** > **Jobs**.
4. Select your Smart Answers (QnA) model training job.
5. Click **Job History**.
6. Find the ML Model workflow ID.
7. In the ML Model service UI, locate instances of the ID in the logs.
## Smart Answers jobs
These jobs provide the machine learning features that drive Smart Answers:
* Machine learning model training:
* [Smart Answers Supervised Training](/docs/5/fusion/reference/config-ref/jobs/smart-answers-supervised-training) job.
Fusion includes default pipelines to get you started. See **Configure the Smart Answers Pipelines**.
* [Smart Answers Coldstart Training](/docs/5/fusion/reference/config-ref/jobs/smart-answers-coldstart-training) job.
* Machine learning model deployment:
* [Create Seldon Core Model Deployment](/docs/5/fusion/reference/config-ref/jobs/create-seldon-core-model-deployment)
* [Delete Seldon Core Model Deployment](/docs/5/fusion/reference/config-ref/jobs/delete-seldon-core-model-deployment)
* [Smart Answers Evaluate Pipeline](/docs/5/fusion/reference/config-ref/jobs/smart-answers-evaluate-pipeline)
This job analyzes your configured Smart Answers query pipeline to provide insights about its effectiveness so that you can fine-tune your configuration for the best possible results.
## Smart Answers pipelines and stages
Once you have trained and deployed your model, you can use one of the default pipelines that are automatically created with your Fusion app. Both pipelines are called `APP_NAME-smart-answers`.
See **Configure the Smart Answers Pipelines** for more information.
| Pipeline | Stages |
| - | - |
| **"-smart-answers" index pipeline** | |
| **"-smart-answers" query pipeline** | |
## Short answer extraction
By default, the question-answering query pipelines return complete documents that answer questions. Optionally, you can extract just a paragraph, a sentence, or a few words that answer the question. See **Extract Short Answers from Longer Documents**.
This topic explains how to deploy and configure the transformer-based deep learning model for short answer extraction with [Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/overview). This model is useful for analyzing long documents and extracting just a paragraph, a sentence, or a few words that answer the question.
This model is trained on the [SQuAD2.0 dataset](https://rajpurkar.github.io/SQuAD-explorer) which consists of questions about Wikipedia articles and answers gleaned from those articles. Therefore, this model is most effective with Wikipedia-like content and may produce uneven results when applied to more informal content such as message boards.The out-of-the-box (OOTB) model only supports English content.
## Hardware recommendations
When creating a nodePool to perform short answer extraction, use a configuration that meets these guidelines in order to achieve the best performance:
* It is strongly recommended to use the latest possible Intel CPU architecture; Intel CascadeLake or higher architectures are recommended.
* Large core count is also recommended: 12-16 cores with 32G of RAM.
## Deploy the model in Fusion
1. Navigate to **Collections** > **Jobs**.
2. Select **New** > **Create Seldon Core Model Deployment**.
3. Configure the job as follows:
* **Job ID.** the ID for this job, such as `deploy-answer-extractor`.
* **Model Name.** model name of the Seldon Core deployment that will be referenced in the Machine Learning pipeline stage configurations, such as `answer-extractor`.
* **Docker Repository.** `lucidworks`
* **Image Name.** `answer-extractor:v1.1`
* **Kubernetes Secret Name for Model Repo.** (empty)
* **Output Column Names for Model.** `[answer,score,start,end]`
4. Click **Save**.
5. Click **Run** > **Start**.
## Configure the Machine Learning query stage
This model provides the best results when used with one of the [question-answering query pipelines](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/overview). The default query pipeline is called `APP_NAME-smart-answers`.
Starting with one of those pipelines, add a new Machine Learning stage to the end of the pipeline and configure it as described below.
**How to configure short answer extraction in the query pipeline**
1. Make sure you have performed the basic configuration of your query pipeline.
2. In the query pipeline, click **Add a Stage** > **Machine Learning**.
3. In the **Model ID** field, enter the model name you configured above, such as answer-extractor.
4. In the **Model input transformation script** field, enter the following:
```js expandable theme={"dark"}
var textFieldToExtract = "answer_t"
var numDocsToExtract = 3
responses = new java.util.ArrayList();
var docs = response.get().getInnerResponse().getDocuments();
for (var i=0; i Make sure that the question is provided as it was originally entered by user. If you have previous stages that augments question (like stopwords removing or synonyms expansion), it is better to copy original question and use it for the answer extraction without additional modifications.
* `context` (Required). A string or list of contexts; by default this is the first `num_docs_to_extract` documents in the output of the previous stage in the pipeline.
If only one question is present with multiple contexts, that question will be applied to every context and vice versa for 1 context and multiple questions. If a list of questions and contexts is passed, a 1:1 mapping of questions and contexts will be created in the order in which they’re passed.
* `topk`. The number of answers to return (will be chosen by order of likelihood). Default: 1
* `handle_impossible_answer`. Whether or not to deal with a question that has no answer in the context.
If true, an empty string is returned. If false, the most probable (`topk`) answer(s) are returned regardless of how low the probability score is. Default: True
Experiment with this parameter to see what value returns the most acceptable answers.
For advanced use cases, you can add the following parameters to the script to override their defaults:
* `batch_size`. How many samples to process at a time. Reducing this number will reduce memory usage but increase execution time, while increasing it will increase memory usage and decrease execution time to a certain extent. Default: 8
* `max_context_len`. If set to greater than 0, truncate contexts to this length in characters. Default: 5000
* `max_answer_len`. The maximum length of predicted answers (for example, only answers with a shorter length are considered). Default: 15
* `max_question_len`. The maximum length of the question after tokenization. It will be truncated if needed. Default: 64
* `doc_stride`. If the context is too long to fit with the question for the model, it will be split in several chunks with some overlap. This argument controls the size of that overlap. Default: 128
* `max_seq_len`. The maximum length of the total sentence (context + question) after tokenization. The context will be split in several chunks (using doc\_stride) if needed. Default: 384
5. In the **Model output transformation script** field, enter the following:
```js expandable theme={"dark"}
{/* // Parse raw output from model */}
var jsonOutput = JSON.parse(modelOutput.get("_rawJsonResponse"))
var parsedOutput = {};
for (var i=0; i
## Recreate a Milvus collection
If a Milvus collection is lost, you can recreate the collection. You can also use these steps to create a Milvus collection for a different field or a Milvus collection that uses a different encryption model.
These steps assume that:
* You used the Smart Answers pipeline to index data, which created the Milvus and Solr collections.
* The created Solr collection still exists, which is used as the Source Collection.
1. Create the Milvus collection. See [Create Collections in Milvus](/docs/5/fusion/reference/config-ref/jobs/create-collections-in-milvus) for more information.
If the Milvus collection still exists and you want to overwrite it, select the **Override Collections** checkbox. This option deletes the current collection, allowing you to create a new collection.
2. Edit the Smart Answers Index Pipeline as follows:
* Disable the Solr Indexer stage.
* If you are creating a Milvus collection for a different field, or one that uses a different encryption model, access the **Encode into Milvus** stage and:
* Specify the **model**, the **Encoder Output Vector**, and the **Field to Encode**.
* Verify the **Milvus Collection Name** refers to the new Milvus collection.
3. Navigate to the Datasources panel.
4. Select **Add a Solr connector** and enter the following information:
* **Pipeline ID.** Smart Answers pipeline
* **Solr Connection Type.** SolrCloud
* **SolrCloud Zookeeper Host String.** Execute the following curl command. The response displays the SolrCloud Zookeeper Host String value. In the following `RESPONSE` example, the value you would set for the SolrCloud Zookeeper Host String is `ns-zookeeper-0.ns-zookeeper-headless:2181`.
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD https://FUSION_HOST:FUSION_PORT/api/searchCluster/default
```
**RESPONSE**
```json theme={"dark"}
{
"id": "default",
"connectString": "ns-zookeeper-0.ns-zookeeper-headless:2181",
"zkClientTimeout": 30000,
"zkConnectTimeout": 60000,
"cloud": true,
"bufferFlushInterval": 1000,
"bufferSize": 100,
"concurrency": 10,
"authConfig": {
"authType": "none"
},
"validateCluster": true
}
```
* **Source Collection.** Original collection from the initial Smart Answers pipeline indexing job
5. Navigate to Index Workbench and load the new solr-index datasource.
6. Select **Start Job** to run the Solr datasource through the updated Smart Answers pipeline.
7. When the data is reindexed, edit the Smart Answers pipeline and re-enable the Solr Indexer stage.
## Learn more
When you create a new Fusion app, Fusion also creates the default Smart Answers index and query pipelines for that app.
If you have an existing app and you want to add the latest Smart Answers default pipelines, follow the instructions below.
Next, you can configure the Smart Answers pipelines.
## Create the new Smart Answers Index Pipeline
To add the new Smart Answers index pipeline to an existing app, execute the following curl command, substituting for the `fusion_host` and the `app_name` in the url:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/json' -d '{"id":"smart-answers","stages":[{"type":"field-mapping"},{"type":"solr-dynamic-field-name-mapping"},{"type":"model-to-milvus","modelId":"smart-answers-encoder","modelOutputVector":"vector","milvusCollection":"milvus_collection","fieldToEncode":"change-me"},{"type":"solr-index"}]}' http://{fusion_host}/api/apps/{app_name}/index-pipelines
```
## Create the new Smart Answers Query Pipeline
To add the new Smart Answers query pipeline to an existing app, execute the following curl command, substituting for the `fusion_host` and the `app_name` in the url:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST -H 'content-type: application/json' -d '{"id":"smart-answers","stages":[{"type":"search-fields"},{"type":"milvus-query-stage","modelId":"smart-answers-encoder","modelOutputVector":"vector","milvusResultsContextKey":"milvus_results","milvusVectorsCollection":"milvus_collection","milvusNumResults":10},{"type":"milvus-ensemble-query-stage","ensembleExpression":"1.0*milvus_results","resultFieldName":"ensemble_score"},{"type":"solr-query"},{"type":"milvus-response-update-stage"}]}' http://{fusion_host}/api/apps/{app_name}/query-pipelines
```
This topic explains how you can track the running steps of the following jobs:
* Smart Answers Supervised Training
* Smart Answers Coldstart Training
* Smart Answers Evaluate Pipeline
The ML Model service logs provide information on data pre-processing, training steps, evaluations, and model generation for those jobs. Follow the instructions below to access the logs.
1. Find the name of the ML Model service pod:
```bash theme={"dark"}
kubectl get pods -n | grep ml-model-service-ui
```
The pod name looks like `-ml-model-service-ui--`, as in `f5-ml-model-service-ui-547dd78d6-p9d6q`.
2. Set up port forwarding:
```bash wrap theme={"dark"}
kubectl port-forward -ml-model-service-ui-- 8001:8001 -n
```
3. In the Fusion UI, navigate to **Collections** > **Jobs**.
4. Select your Smart Answers (QnA) model training job.
5. Click **Job History**.
6. Find the ML Model workflow ID.
7. In the ML Model service UI, locate instances of the ID in the logs.
You can implement a voice chatbot using Smart Answers to integrate your search platform with a voice interface application. **This procedure uses [Google Dialogflow](https://cloud.google.com/dialogflow) as an example**, but you can use other voice interface platforms that support the following:
* Speech-to-text translation\
Translate the customer’s speech into a text-based query for Fusion.
* Workflow routing\
Route queries to your Smart Answers query profile.
* Text-to speech translation\
Translate Fusion’s text response into speech for the customer.
The high-level steps will be similar for any voice interface application:
1. Enable Smart Answers on your Fusion platform.
2. Configure your voice interface application to perform speech-to-text on voice queries.
3. Configure your voice interface application to route queries to the Fusion query profile associated with your Smart Answers query pipeline.
4. Configure your voice interface application to perform text-to-speech on Fusion’s responses.
In this example procedure, we provide Javascript code that you can copy and customize, which performs the routing to your Fusion cluster.
## Enable Smart Answers
To enable Smart Answers, you need a trained machine learning model, a question-and-answer body of content, and a query pipeline configured to apply the machine learning model to the content.
## Perform speech-to-text on voice queries
In the case of Google Dialogflow, this functionality is enabled by default whenever you create a Dialogflow agent. See your voice interface platform’s documentation to find out how to enable speech-to-text for your application.
## Route voice queries to Fusion
The procedure below shows you how to configure a Google Dialogflow agent as an example to demonstrate one possible configuration for routing voice queries to Fusion. For other voice interface platforms, check your documentation.
In this example, we will configure the agent so that all intents use a webhook, for which we provide a code sample.
1. Make sure you have created a [billing account](https://cloud.google.com/dialogflow/docs/support/billing-questions) for Google Dialogflow, in order to enable all the required features.
2. [Download the `voice.js.zip` file](/assets/attachments/voice.js.zip), unzip it, and update the variables below to match your Fusion environment:
```js theme={"dark"}
const FUSION_APP_NAME = 'YourAppName';
const FUSION_QUERY_PROFILE_NAME = 'QueryProfileName';
const FUSION_ADMIN_URL = 'http://FusionAdminUrl.com';
const FUSION_ADMIN_USERNAME = 'username';
const FUSION_ADMIN_PASSWORD = 'password';
const FUSION_CREDENTIALS = 'Basic xxxxxxxsssssssssssss';
```
The `FUSION_ADMIN_URL = 'http://FusionAdminUrl.com'` variable is also referred to as `FUSION_ADMIN_URL = 'http://FUSION_HOST:FUSION_PORT'`.\
If your Fusion response field is not called `answer_t`, then you also need to update the field name in this line of the `voice.js` file:
```js theme={"dark"}
agent.add(result.data.response.docs[0].answer_t);
```
3. In your Google Dialogflow agent interface, navigate to **Intents**.\
A newly-created Google Dialogflow agent contains two default intents:
* Default Fallback Intent
* Default Welcome Intent\
The `voice.js` code includes a handler for each of the default intents, plus "Try again" and "Yes" intents (which you will create below). If your application includes additional intents, you may need to add those to the code.
4. Navigate to **Intents** > **Default Fallback Intent** and configure it as follows:
1. Delete all of the items in the **Context**, **Events**, **Training Phrases**, **Actions**, and **Response** sections.
2. Under **Fulfillment**, select **Enable webhook call for this intent**.\\
3. Click **Save**.
5. Click **Default Welcome Intent** and configure it as follows:
1. Delete all of the items in the **Training Phrases** and **Responses** sections.
2. Under **Fulfillment**, select **Enable webhook call for this intent**.
3. Click **Save**.
6. Navigate to **Intents** > **Create Intent**, call the new intent "Try again" (case-sensitive), and configure it as follows:
1. Delete all of the items in the **Context**, **Events**, **Actions**, and **Response** sections.
2. Under **Training Phrases**, add "Try again" (case-sensitive).
3. Under **Fulfillment**, select **Enable webhook call for this intent**.
4. Click **Save**.
7. Navigate to **Intents** > **Create Intent**, call the new intent "Yes" (case-sensitive), and configure it as follows:
1. Delete all of the items in the **Context**, **Events**, **Actions**, and **Response** sections.
2. Under **Training Phrases**, add the following phrases (case-sensitive):
* "Try again"
* "Yep"
* "Perfect"
3. Under **Fulfillment**, select **Enable webhook call for this intent**.
4. Click **Save**.
8. Navigate to **fulfillment** and configure it as follows:
1. Select **Enable Inline Editor**.
2. Overwrite the default code with the `voice.js` code that you customized in step 2 above.\\
3. Click **Deploy**.
In order to enable external API calls to Fusion, you must upgrade your [Google Firebase](https://console.firebase.google.com/) billing account to the "Pay as you go" plan. Google Firebase is also where you can view the console logs for these API calls.
## Perform text-to-speech on Fusion's responses
1. In your Google Dialogflow agent interface, click the gear icon next to your agent’s name.
2. Under **Automatic Spell Correction**, enable **Allow ML to correct spelling of query during request processing**.
3. Under **Text to speech**, select **Enable Automatic Text to Speech**.
4. Click **Save**.
## Test your configuration
1. In your Google Dialogflow agent interface, click the **Google Assistant** link in the right-hand sidebar.\\
2. Click **Talk to ``**.
3. Click the microphone and ask a question.\
If you do not receive a response that makes sense, check the logs in [Google Firebase](https://console.firebase.google.com/) for information you can use to troubleshoot your configuration.
Once your voice chatbot is working, you can click **Integrations** to access a pre-built widget that you can paste into the code for your front-end application.
See the [Google Dialogflow documentation](https://cloud.google.com/dialogflow/docs/) for more details.
# Advanced Model Training Configuration for Smart Answers
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/smart-answers-advanced-model-config
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/smart-answers-advanced-model-config
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/smart-answers-advanced-model-config
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/724
This topic provides tips for training your [Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/overview) deep learning model.
## Model Base
There are several types of model bases that can be used for training and fine-tuning:
* **word\_en\_300d\_2M** are general pre-trained word embeddings. It is a good default choice to start with for English language.
* **bpe\_\{language}\_\{dim\_size}\_\{vocab\_size}** are general pre-trained [BPE](https://nlp.h-its.org/bpemb) embeddings that are available for different languages, including CJK languages and multilingual. Also useful in scenarios when vocabulary is very big or when the data might have a lot of misspellings.
* **word\_custom** or **bpe\_custom** specifies that custom embeddings should be trained on users data via Word2Vec algorithm. It might be useful when your domain has a very unusual specific vocabulary.
* **transformer** based models such as **distilbert\_\{language}** and **biobert**. Much bigger and expensive models that might provide even better quality for FAQ, Chatbot and virtual Assistance use-cases. Also useful when the training data is limited.
If **word** or **bpe** based models are used, one or more RNN layers are added on top of the embeddings to be trained to capture contextual and semantic information. It is configurable in the **RNN Encoder Parameters** section. If you wish to use embeddings initialized on your data, refer to the **Custom Embeddings Initialization** to configure Word2Vec algorithm.
Transformer-based models already have specified fixed model architecture which is fine-tuned during the training procedure.
Dimension size of vectors for Transformer-based models is 768. For RNN-based models it is 2 times the number units of the last layer. To find the dimension size: download the model, expand the zip, open the log and search for `Encoder output dim size:` line. You might need this information when creating collections in Milvus.
We recommend to use Transformer-based models only if you can allocate GPU for the training job as these models are very computationally expensive.
At the end, Attention mechanism to aggregate output into final single vector for all models.
## Auto hyperparameter tuning
By default, training module tries to select the most optimal parameter values (for those left as blank) based on the training data statistics. Auto-tune can extend it by automatically finding even better training configuration through hyper-parameter search.
If **Perform auto hyperparameter tuning** is enabled, multiple models will be trained across several stages. On each stage the most impactful parameters are tuned to find the best configuration. All other parameters are used with default values or those specified on UI.
Although this is a resource-intensive operation, it can be useful to identify better RNN-based configuration. Transformer-based models are not used during auto hyperparameter tuning as they have a fixed architecture. They usually perform better on Q\&A tasks yet they are much more expensive on both training and inference time.
## Input/Output parameters
Here you can specify the input data that should be used for training with possibility to filter or sample it.
You can also configure this job to read from or write to cloud storage. See **Configure An Argo-Based Job to Access GCS** and **Configure An Argo-Based Job to Access S3**.
Some jobs can be configured to read from or write to Google Cloud Storage (GCS).
You can configure a combination of Solr and cloud-based input or output, that is, you can read from GCS and then write to Solr or vice versa.
However, you cannot configure multiple storage sources for input or multiple storage targets for output; only one can be configured for each.
## Supported jobs
This procedure applies to these Argo jobs:
* Content based Recommender
* BPR Recommender
* Classification
* Evaluate QnA Pipeline
* QnA Coldstart Training
* QnA Supervised Training
For Spark jobs, see Configure A Spark-Based Job to Access Cloud Storage.
## How to configure a job to access GCS
1. Gather the access key for your GCS account.\
See the [GCS documentation](https://cloud.google.com/iam/docs/creating-managing-service-account-keys).
2. Create a Kubernetes secret:
```
kubectl create secret generic my-gcs-serviceaccount-key --from-file=.json --namespace
```
3. In the job’s **Cloud storage secret name** field, enter the name of the secret for the GCS target as mounted in the Kubernetes namespace.\
This is the name you specified in the previous step. In the example above, the secret name is `my-gcs-serviceaccount-key`.
You can also find this name using `kubectl get secret -n `.
4. In the job’s **Additional Parameters**, add this parameter:
* Parameter name: `google.cloud.auth.service.account.json.keyfile`
* Parameter value: ``
The file name may be different than the secret name. You can check using `kubectl get secret -n -o yaml`.
Some jobs can be configured to read from or write to Amazon S3 (S3).
You can configure a combination of Solr and cloud-based input or output, that is, you can read from S3 and then write to Solr or vice versa.
However, you cannot configure multiple storage sources for input or multiple storage targets for output; only one can be configured for each.
## Supported jobs
This procedure applies to these Argo jobs:
* Content based Recommender
* BPR Recommender
* Classification
* Evaluate QnA Pipeline
* QnA Coldstart Training
* QnA Supervised Training
For Spark jobs, see Configure A Spark-Based Job to Access Cloud Storage.
## How to configure a job to access S3
1. Gather the access key and secret key for your S3 account.\
See the [AWS documentation](https://aws.amazon.com/premiumsupport/knowledge-center/create-access-key).
2. Create a Kubernetes secret:
```
kubectl create secret generic aws-secret --from-literal=my-aws-key-file='' --from-literal=my-aws-secret-path='' --namespace
```
3. In the job’s **Cloud storage secret name** field, enter the name of the secret for the S3 target as mounted in the Kubernetes namespace.\
This is the name you specified in the previous step. In the example above, the secret name is `aws-secret`.
You can also find this name using `kubectl get secret -n `.
4. In the job’s **Additional Parameters**, add these two parameters:
* Param name: `fs.s3a.access.keyPath`\
Param value: ``
* Param name: `fs.s3a.secret.keyPath`\
Param value: ``
The file name may be different than the secret name. You can check using `kubectl get secret -n -o yaml`.
If you have additional text data that can be used for **custom embeddings initialization** or to learn and capture bigger vocabulary when **word\_en\_300d\_2M** is used, please provide it in the **Texts Data Path** field.
**Model Replicas** parameter allows to specify how many replicas of the model should be deployed. Auto-balancing mechanism is used to distribute queries between model replicas, so more replicas might provide faster indexing as well as higher QPS.
If you use aggregated signals data for training or have weights for each training pair, you can also specify **Weight Field**. It will be used for sampling positive answers for a particular query if there are more than one possible. It is useful for eCommerce use-cases when for one unique query there might be a lot of different paired products.
**Weight Field** will not be used if **Use Labelling Resolution** is set on. These parameters are mutually exclusive.
## Data pre-processing parameters
**Labeling Resolution** allows to find missing query/response pairs in the training data which helps in the training. When set on, a graph of all pairs connections is built. Then connected components are obtained to match missing query/response pairs. For example if there are three existing pairs: `q1-a1`, `q2-a1` and `q2-a2`. Then Labelling Resolution will match `q1-a2` as additional pair through `q2-a1` connection.
This is useful in Q\&A use-cases when there are not a lot of answers per unique question, otherwise too big connected components will be found. If you have data when for one query there might be a lot of different responses, like in eCommerce, it is better to leave it off.
If **Use Labelling Resolution** is set on, **Weight Field** is ignored. These parameters are mutually exclusive.
The **Maximum vocabulary size**, **Lower case all words** and **Apply unidecode decoding** parameters impact the vocabulary size if `word_en_300d_2M`, `word_custom` or `bpe_custom` model bases are used. Otherwise these parameters are ignored and model specific pre-processing is used. Default values should work in most cases, given enough RAM and time to train.
If you want to train custom embeddings for languages like CJK, disable **Apply unidecode decoding**.
If you see an out-of-memory error, try reducing the vocabulary size and/or the training batch size. The **Minimum number of words** and **Maximum number of words** parameters can help trim problematic documents.
## Custom embeddings initialization parameters
If **word\_custom** or **bpe\_custom** model bases are chosen, then custom embeddings will be trained on the provided data.
If you want to use addition dataset to train custom embeddings, please specify **Texts Data Path** and **Text Fields** in the **Input/Output parameters**.
Additionally, commonly-used Word2vec training parameters are **Word2Vec Training Epochs**, **Size of Word Vectors** and **Word2Vec Window Size**. Default values should work in most cases.
Smaller word vectors size makes models smaller and more robust to overfitting. However, dimensions smaller than 100 may impact the quality.
## Evaluation parameters
**Validation Sample Size** controls how much *unique* queries should be hold-out and used for validation. It is a fraction if the value below 1.0 or specific number of queries if it is integer value higher than 1.
During evaluation, all responses/answers are used. They form an index which is queried by unique validation queries. **Eval ANN Index** parameter controls should it be ANN index or brute-force search with auto value by default. If you notice that evaluation takes a lot of time, try to enable ANN index or reduce the number of evaluation queries.
Generally, this evaluation setup is similar to how it will work in index and query pipelines, so the evaluation results should provide good approximation of the quality. To evaluate the configured pipelines on the test data, please use **Evaluate a Smart Answers Query Pipeline**.
The [Smart Answers Evaluate Pipeline job](/docs/5/fusion/reference/config-ref/jobs/smart-answers-evaluate-pipeline) evaluates the rankings of results from any [Smart Answers](/docs/5/fusion/getting-data-out/advanced-query-enhancement/smart-answers/overview) pipeline and finds the best set of weights in the ensemble score. This topic explains how to set up the job.
Before beginning this procedure, prepare a machine learning model using either the Supervised method or the Cold start method, or by selecting one of the pre-trained cold start models, then Configure your pipelines.
The input for this job is a set of test queries and the text or ID of the correct responses. At least 100 entries are needed to obtain useful results. The job compares the test data with Fusion’s actual results and computes variety of the ranking metrics to provide insights of how well the pipeline works. It is also useful to use to compare with other setups or pipelines.
## Prepare test data
1. Format your test data as query/response pairs, that is, a query and its corresponding answer in each row.
You can do this in any format that Fusion support, but parquet file would be preferable to reduce the amount of possible encoding issues.
The response value can be either the document ID of the correct answer in your Fusion index (preferable), or the text of the correct answer.
If you use answer text instead of an ID, make sure that the answer text in the evaluation file is formatted identically to the answer text in Fusion.
If there are multiple possible answers for a unique question, then repeat the questions and put the pair into different rows to make sure each row has exactly one query and one response.
2. If you wish to index test data into Fusion, create a collection for your test data, such as `sa_test_input` and index the test data into that collection.
## Configure the evaluation job
1. If you wish to save the job output in Fusion, create a collection for your evaluation data such as `sa_test_output`.
2. Navigate to **Collections** > **Jobs**.
3. Select **New** > **Smart Answers Evaluate Pipeline** (**Evaluate QnA Pipeline** in Fusion 5.1 and 5.2).
4. Enter a **Job ID**, such as `sa-pipeline-evaluator`.
5. Enter the name of your test data collection (such as `sa_test_input`) in the **Input Evaluation Collection** field.
6. Enter the name of your output collection (such as `sa_test_output`) in the **Output Evaluation Collection** field.
You can also configure this job to read from or write to cloud storage.
7. Enter the name of the **Test Question Field** in the input collection.
8. Enter the name of the answer field as the **Ground Truth Field**.
9. Enter the **App Name** of the Fusion app where the main Smart Answers content is indexed.
10. In the **Main Collection** field, enter the name of the Fusion collection that contains your Smart Answers content.
11. In the **Fusion Query Pipeline** field, enter the name of the Smart Answers query pipeline you want to evaluate.
12. In the **Answer Or ID Field In Fusion** field, enter the name of the field that Fusion will return containing the answer text or answer ID.
13. Optionally, you can configure the **Return Fields** to pass from Smart Answers collection into the evaluation output.
Check the Query Workbench to see which fields are available to be returned.
14. Configure the **Metrics** parameters:
* **Solr Scale Function**
Specify the function used in the Compute Mathematical Expression stage of the query pipeline, one of the following:
* `max`
* `log10`
* `pow0.5`
* **List of Ranking Scores For Ensemble**
To find the best weights for different ranking scores, list the names of the ranking score fields, separated by commas. Different ranking scores might include Solr score, query-to-question distance, or query-to-answer distance from the Compute Mathematical Expression pipeline stage.
* **Target Metric To Use For Weight Selection**
The target ranking metric to optimize during weights selection. The default is `mrr@3`.
15. Optionally, [read about the advanced parameters](/docs/5/fusion/reference/config-ref/jobs/smart-answers-evaluate-pipeline) and consider whether to configure them as well.
For example, **Sampling proportion** and **Sampling seed** provide a way to run the job only on a sample of the test data.
16. Click **Save**.
17\. Click **Run** > **Start**.
## Examine the output
The job provides a variety of metrics (controlled by the **Metrics list** advanced parameter) at different positions (controlled by the **Metrics\@k list** advanced parameter) for the chosen final ranking score (specified in **Ranking score** parameter).
**Example: Pipeline evaluation metrics**
**Example: recall\@1,3,5 for different weights and distances**
In addition to metrics, a results evaluation file is indexed to the specified output evaluation collection. It provides the correct answer position for each test question as well as the top returned results for each field specified in **Return fields** parameter.
A list of evaluation metrics is provided to monitor the training process and measure the quality of the final model:
* Mean Average Precision (MAP)
* Mean Reciprocal Rank (MRR)
* Recall
You can choose from the list in the **Metrics list** parameter. It uses all metrics by default.
You can also specify measuring the ranking position for each metric. For example, if you specify **Metrics\@k list** as `[1,3]`, with **Metrics list** `[“map”,”mrr”,”recall”]`, then the metrics `map@1`, `map@3`, `mrr@1`, `mrr@3`, `recall@1`, and `recall@3` will be logged for each training epoch and final model.
You can choose a particular metric at a particular `k` (controlled by the **Monitoring metric** parameter) to help decide when to stop training. Specifically, when there is no increase in the Monitoring metric value for a particular number of epochs (controlled by the **Patience during monitoring** parameter), then training stops.
During the training we evaluate the result using similar cold-start model (weighted average of word vectors) as a baseline. Look for the *Cold-start encoder validation evaluation* section of the logs, it is printed before first training epoch.
## General Encoder parameters
Note that the following parameters are common across all model bases including RNN and Transformer architectures.
* **Fine-tune Token Embeddings** will allow to fine-tune embeddings (word vectors) layer to be updated during the training alongside with all other layers. It is disabled by default as it is usually one of the biggest layer in the network and updating it might lead to overfitting. It is useful to enable if your data have a lot of specific or misspelled words.
* **Max Length** controls the maximum context window that model can process. Texts longer than this value will be trimmed. The default value is the max value between three times the STD of question lengths and two times the STD of answer lengths.\
The longer the context the longer and harder it takes for model to process. This parameter is especially important for Transformer-based models as it affects training and inference time. Note that the maximum supported length for Transformer models is 512 tokens, so you can specify any value up to that.
* **Global Pool Type** specifies how token vectors should be aggregated to obtain final content vector. The default mechanism is self-attention which provides the best quality in most cases.
* **Number of clusters** and **Top K of clusters to return** are deprecated since 5.3 and will be removed in the following releases. There is no practical need to use them after Milvus vectors similarity search integration.
## RNN Encoder parameters
We use [RNN-based deep learning](https://en.wikipedia.org/wiki/Recurrent_neural_network) architecture for `word` and `bpe` model bases, with the flexibility to choose between `LSTM` and `GRU` layers with more than one layer. We don’t recommend using more than three layers. The layers and layer sizes are controlled by the **RNN function list** and **RNN function units list** parameters.
**Dropout ratio** parameters provides regularization effect and is applied between embeddings layer and the first RNN layer.
## Training parameters
These parameters controls the training procedure. Most of them are left blank so the robust default values can be determined by the training module based on the dataset statistics.
The learning rate scheduler has 3 stages. Firstly, it linearly increases the LR from **Minimum Learning Rate** value to **Base Learning Rate** value over **Number of Warm-Up epochs**. Then it stays consistent for **Number of Flat epochs**. And at the last stage Cosine Annealing is used for the remain number of epochs.
**Use Mixed Precision** parameter can enable mixed precision during the training for Transformer-based models if modern GPU are used (Turing and later). It helps to get more VRAM so bigger batch size can be used. As well as provides some performance boost in training time.
**Cross-Batch Memory** parameters allow to re-use encoded representations from the previous batches during loss computation, so loss function can process more positive and negative examples for the model update. It works well with Transformer-based models that consumes more VRAM and can be used with only with limited batch size. This is not necessary when the training batch size is large. When configured, this number needs to be greater than or equal to the training batch size.
# Supported Browsers
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/experience-optimizer/eo-browser-support
Experience Optimizer
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/curating-your-search-experience/experience-optimizer/eo-browser-support
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/experience-optimizer/eo-browser-support
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/az23pr
Currently, Experience Optimizer is officially supported in Google Chrome *only*. Although Experience Optimizer is available in unsupported browsers, features and functionality may be limited.
| Browser | Supported |
| - | - |
| Google Chrome | ✅ |
| Firefox | ✘ |
| Safari | ✘ |
| Opera | ✘ |
| Edge | ✘ |
# Document Grouping
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/experience-optimizer/eo-document-grouping
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/curating-your-search-experience/experience-optimizer/eo-document-grouping
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/experience-optimizer/eo-document-grouping
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/4jzskx
Experience Optimizer makes it easy to group documents together so they can be presented to the user. This can be useful when you want to show several documents that are relevant to each other, for example.
Document groups can be boosted or blocked. If you want to promote a document group that is new, for example, you can boost the group to the top of the results.
You can still apply rules individually to documents within a group. This allows you to boost a document based on a specific user’s search behavior. If a user that is typically interested in ballet searches for "dance" content, you can boost ballet-related articles to the top of the group.
## Best practices
These parameters group documents, and are defined in the Additional Query Parameters Stage:
| Parameter Name | Parameter Value | Update Policy |
| - | - | - |
| group | true | default |
| group.format | grouped | default |
| group.ngroups | true | default |
| group.field | style\_id\_s | default
NOTE: You must configure `groupField=` if you use [App Studio](/docs/5/fusion/dev-portal/appkit/overview) to create your search application. |
For more information about adding a query pipeline and additional query parameters, see:
* **Add a Query Pipeline Stage**
* [Additional Query Parameters Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/set-query-params-stage)
In the Query Workbench, click **Add a Stage** to add query pipeline stages that can perform query setup, results relevancy, troubleshooting, and more.
The Solr Query stage is the only pipeline stage that is required for querying processes to complete, and therefore exists in every query pipeline. It is always the last stage in a series.
**See these pages for more information:**
Fusion 5.x:
* [Query Workbench](/docs/5/fusion/intro/ui-tour/query-workbench)
* [query pipeline stages](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/overview)
* [Solr Query stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/solr-query-stage)
Fusion 4.x:
* [Query Workbench](/docs/4/fusion-server/concepts/querying/query-workbench)
* [query pipeline stages](/docs/4/fusion-server/reference/pipeline-stages/query/overview)
* [Solr Query stage](/docs/4/fusion-server/reference/pipeline-stages/query/solr-query-stage)
### Collapse/Expand parser grouping
The Collapse/Expand parser allows grouping with the following parameters:
| Parameter Name | Parameter Value |
| - | - |
| expand | true |
| enableElevation | true |
| group | false |
See [Collapse and Expand Results](https://solr.apache.org/guide/collapse-and-expand-results.html) for more information.
# Search Rewrites
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/experience-optimizer/eo-search-rewrites
Experience Optimizer
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/curating-your-search-experience/experience-optimizer/eo-search-rewrites
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/experience-optimizer/eo-search-rewrites
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/orstay
Search rewriting is a way of modifying search queries so that they more accurately reflect the intentions of your customers. You can create a misspelling search rewrite, for example, to correct a spelling mistake that commonly occurs when searching for a particular document. The misspelled query will then produce the same results as a correctly spelled search.
There are four rewrite types:
* **Head/Tail** - improves poorly performing searches.
* **Misspelling detection** - corrects common spelling mistakes.
* **Phrase detection** - identifies products with matching phrases.
* **Synonym detection** - includes alternative words with the same meaning.
You can create query rewrite rules in Experience Optimizer. This is similar to using the Rules Editor to create query rewrite rules, except there are additional options in Experience Optimizer.
1. In Fusion, navigate to **Relevance** > **Rules** > **Optimizer**.
2. Enter a search term or phrase in the search bar.
3. Click **Start Task**.
4. Hover your cursor next to the query. A **+** button will appear:
5. Click the **+** button. A list of query rewrites options will appear: [Head/Tail](#head-tail), [Misspelling](#misspelling), [Phrase](#phrase), [Synonym](#synonym), and [Remove Words](#remove-words).
6. Click on the type of rule you’d like to configure.
The course for **Experience Optimizer: Rewrites Manager** focuses on how to create search rewrites that optimize your user's search results.
## Head/Tail
You can create a Head/Tail rewrite to improve search results using methods other than correcting for misspellings or synonym expansion. When a poorly defined search term is identified, the original term is replaced by an improved search term.
For example, a search for `benefits enrollment` could be improved by using the search term `benefits enrollment+year:"2021"` (in this case making use of the year field in the data).
Most Head/Tail rewrites will typically be created automatically via machine learning. However, if desired, custom rewrites can be manually created using the following steps.
1. From the list of query rewrite options, select **Head/Tail**. A form will appear:
This form contains the following fields:
| | | |
| - | - | - |
| Parameter | Description | Example Value |
| Tail Query | The tail query itself. | `benefits enrollment` |
| Improved Query | The query that will replace the tail query phrase. | `benefits enrollment+year:"2021"` |
| Tags | Optional metadata tags that can be used to identify and organize rewrites. | `enrollmentpacket` |
2. Enter one or more improved search terms in the **Improved Query** field.
3. Click the **Save** button.
The search is re-run using the improved search term, and a Search box information panel drops down to show that the new query rewrite has fired:
## Misspelling
You can create a misspelling query rewrite to detect and correct common spelling mistakes. When a customer enters a search term containing a known misspelling, the incorrect spelling is replaced with the spelling correction.
For example, if your clients frequently misspell or mistype the word `questionnaire` as `questionare`, you can set up a query rewrite to automatically correct it.
1. From the list of query rewrite options, select **Misspelling**. A form will appear:
This form contains the following fields:
| | | |
| - | - | - |
| Parameter | Description | Example Value |
| Misspelled Term | The phrase itself. | `questionare` |
| Corrected Term | The term that will replace the misspelled term. | `questionnaire` |
| Action | Action to perform. | `expand` |
| Confidence | Confidence score from the phrase job. A confidence level of `1` represents 100% confidence. For rules created automatically via machine learning, the confidence level will reflect the output from the machine learning model. | `1` |
| Tags | Optional metadata tags that can be used to identify and organize rewrites. | `enrollmentpacket` |
2. Enter one or more spelling corrections in the **Corrected Term** field.
3. Click the **Save** button.
The search is re-run using the corrected spelling, and a Search box information panel drops down to show that the new query rewrite has fired:
## Phrase
You can use query rewriting to identify phrases used in search terms so that products with matching phrases are boosted in the search results. This is helpful when users do not use quotation marks to identify phrases in their search terms.
For example, without phrase detection a search for the words `background check` would show results for both `background` and `check`. With phrase detection, this search would correctly boost results for `"background check"`.
1. From the list of query rewrite options, select **Phrase**. A form will appear:
This form contains the following fields:
| | | |
| - | - | - |
| Parameter | Description | Example Value |
| Surface Form | The phrase itself. | `background check` |
| Word Count | Indicates how many words are included in the phrase. | `2` |
| Confidence | Confidence score from the phrase job. A confidence level of `1` represents 100% confidence. For rules created automatically via machine learning, the confidence level will reflect the output from the machine learning model. | `1` |
| Tags | Optional metadata tags that can be used to identify and organize rewrites. | `enrollmentpacket` |
| Phrase Count | Denotes how many times this phrase was found in the source. This value is automatically set via machine learning. It does not need to be set manually. | `5` |
| Boost Factor | The factor to use to boost this phrase in matching queries. | `2.0` |
| Slop Factor | Phrase slop, or the distance between the terms of the query while still considering it a phrase match. | `10` |
2. Enter the number of words in the phrase in the **Word Count** field.
3. Click the **Save** button.
The search is re-run, with your search words identified as a phrase, and a Search box information panel drops down to show that the new query rewrite has fired:
## Synonym
You can specify synonyms for a specified search term so that alternative words with the same meaning are automatically used in the search query. When a customer enters a search term with a synonym match, the alternative words are used instead of, or in addition to, the original search term.
For example, the search term `questionnaire` could have the synonyms `application` and `survey`.
1. From the list of query rewrite options, select **Synonym**. A form will appear:
This form contains the following fields:
| | | |
| - | - | - |
| Parameter | Description | Example Value |
| Surface Form | The term that has synonyms. | `questionnaire` |
| Direction | With a **oneway** search, the original search term is replaced by the synonym. In the example above, `questionnaire` would be replaced by the alternative words `application` and `survey`. With a **symmetric** search, the search query is expanded to include the original term and the synonyms, resulting in a greater number of potential hits. In the example above, this time the query would include `questionnaire`, `application`, and `survey`. | `symmetric` |
| Synonym Mappings | Synonyms for the surface form. | `application, `survey\` |
| Confidence | Confidence score from the phrase job. A confidence level of `1` represents 100% confidence. For rules created automatically via machine learning, the confidence level will reflect the output from the machine learning model. | `1` |
| Tags | Optional metadata tags that can be used to identify and organize rewrites. | `enrollmentpacket` |
| Count | How many times this term occurred in the signal data when it was discovered. This value is optional when a rewrite is being defined manually. | `5` |
2. Choose whether the direction is **oneway** or **symmetric**.
3. Enter one or more alternative words in the **Synonym Mappings** field.
4. Click the **Save** button.
The search is re-run using the synonyms, and a Search box information panel drops down to show that the new query rewrite has fired:
## Remove Words
The Remove Words feature is available in Fusion 5.4 and later.
You can create a Remove Words query rewrite to remove words from a query. This query rewrite is helpful when a word in the search query does not add value to the search results.
For example, you can rewrite a search query for `case study examples` to remove `examples` and then display results for `case study`.
1. From the list of query rewrite options, select Remove Words. A form appears:
2. This form contains the following fields:
| Parameter | Description | Example Value |
| - | - | - |
| Phrase to remove | The words to remove from the trigger phrase. | `examples` |
| Trigger phrases | The query that prompts the removal of the phrase. The trigger phrase is not necessarily a complete query. If the query contains the trigger phrase, then Fusion removes the phrase in the Phrase to Remove field. | `case study examples` |
1. Enter a phrase to remove and a trigger phrase. Note that the phrase to remove is auto-populated with the query.
2. Click **Save**.
The search is re-run, and a Search box information panel drops down to show that the new query rewrite has fired:
In Experience Optimizer, you can create search rewrites manually. You can also edit, test, review and publish the search rewrites that are generated automatically in Fusion from signals data.
The course for **Experience Optimizer: Rewrites Manager** focuses on how to create search rewrites that optimize your user’s search results.
# Optimizing Zero Results
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/experience-optimizer/optimizing-zero-results
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/curating-your-search-experience/experience-optimizer/optimizing-zero-results
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/experience-optimizer/optimizing-zero-results
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/qxpasi
As a search manager, search queries that generate zero results can drive customers away from the site. Experience Optimizer allows you to ensure this does not happen. Instead of display zero results, you can choose to display certain documents when a customer reaches a zero results page.
Learn how to use the **Optimizing Zero Results** feature.
1. Enter a search query the generates 0 results:
2. Click the **Add** button:
3. Select documents or [document groups](/docs/5/fusion/getting-data-out/curating-your-search-experience/experience-optimizer/eo-document-grouping) that you want to associate with the 0 results query:
4. Click the **Insert** button. The selected products are now displayed when the search query is used, resolving the 0 results query:
# Experience Optimizer
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/experience-optimizer/overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/curating-your-search-experience/experience-optimizer/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/experience-optimizer/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/62cqt4
Experience Optimizer is an AI-powered search tool that optimizes results for enterprise assets.
The course for **Experience Optimizer: Getting Started** focuses on the core features and benefits of implementing Experience Optimizer.
## Use cases
### Reduce administration overhead
Experience Optimizer is a way for internal admin to build out pages necessary to find information such as benefits enrollment, hot links for new hires, or topics/docs other employees are looking for. At benefits enrollment time, HR can take over the top zone of the page to reduce repeated requests about links, where to go, and what to do, thus saving time and resources. All such zones and templates can be built without the need of a developer.
### Content importance
You can build a zone for blog posts at the bottom of a page and set a pipeline to only pull in blog posts and sort by date.
### Trending articles
Use signals or click throughs to show trending content for all users. Trending articles shows what’s important while also phasing out information that becomes irrelevant over time.
## Licensing
Experience Optimizer is integrated with Fusion and requires a valid license.
To verify your Experience Optimizer license, click **Account Settings** and select **License Details**.
If you have a valid Experience Optimizer license, an asterisk `*` displays after **Fusion AI Eo**.
To upload a valid license, click **Upload**.
## Access Experience Optimizer
To access Experience Optimizer:
1. Sign in to Fusion and click **Relevance > Rules**.
2. Click the **Optimizer** icon.
## Learn more
You can block a document so that it does not appear in the search results when your customers run a particular query. This is helpful if a document has been removed from your catalog, or it is not yet available, for example.
When you block a document, a new rule is created automatically.
1. Hover your cursor over a document. A menu overlay will appear.
2. Hover your cursor over the **Actions** button. A dropdown menu will appear.
3. Click **Block**.
You can also block multiple products at once by clicking the checkbox in the top-left corner of the document tiles and selecting the **Block** button above the top row of products.
You can boost a document so that it occupies a higher position in the search results when your customers run a particular query. This can be helpful if you have new products in your catalog, or large stocks of some products, for example.
Boosting a document increases the document’s position in the results, but takes other factors into consideration while doing so. As a result, it does not guarantee the document will occupy a specific position in the list. Pinning a document, however, places the document at a specified position regardless of other factors.
When you boost a document, a new rule is created automatically.
1. Hover your cursor over a document. A menu overlay will appear.
2. Hover your cursor over the **Actions** button. A dropdown menu will appear.
3. Click **Boost**.
You can also boost multiple products at once by clicking the checkbox in the top-left corner of the product tiles and selecting the **Boost** button above the top row of products.
You can bury a document so that it occupies a lower position in the search results when your customers run a particular query. This can be helpful if some of the products in your catalog are temporarily out of stock, for example.
Burying a document typically moves it to the end of the list. However, its exact position will depend on multiple factors, including its relevance score and user signals in relation to other products.
When you bury a document, a new rule is created automatically.
1. Hover your cursor over a document. A menu overlay will appear.
2. Hover your cursor over the **Actions** button. A dropdown menu will appear.
3. Select the **Bury** button:
You can also bury multiple products at once by clicking the checkbox in the top-left corner of the document tiles and selecting the **Bury** button above the top row of products.
You can pin a document so that it occupies a fixed position in the search results when your customers run a particular query. This can be helpful if you have bestsellers or popular items that you want customers to see first, for example.
Boosting a document increases the document’s position in the results, but takes other factors into consideration while doing so. As a result, it does not guarantee the document will occupy a specific position in the list. Pinning a document, however, places the document at a specified position regardless of other factors.
When you pin a document, either a new rule is created automatically or an existing pin rule is modified.
## Use Actions
1. Hover your cursor over a document. A menu overlay will appear.
2. Hover your cursor over the **Actions** button. A dropdown menu will appear.
3. Select the **Pin** button:
## Drag and Pin a document
1. Hover your cursor over a document. A menu overlay will appear.
2. Click-and-hold the drag bar.
3. Move the document to the desired location.
4. Release the click.
## Insert and Pin a document
You can search for and insert an alternative document from your catalog into a list of search results, and pin it at a fixed position. This is helpful if:
* You want to include a document that is not included in the results list generated by the query.
* The results list is long and you need to search for a particular document to locate and pin it.
* You want to pin multiple products simultaneously.
When you pin a document, either a new rule is created automatically or an existing pin rule is modified.
1. Hover your cursor between two products. A **+** button will appear:
2. Click the **+** button.
3. Locate and select one or more products you want to pin.
4. Click **Insert**.
## Add a Facet
You can create rules that add facets by interacting directly with the Facet List in Interactive Mode. You can add facets that are set unconditionally, or facets that are only set when your customers run a particular query. For example, you might want to set a music genre facet when a customer navigates to the music section of your catalog.
Adding a facet automatically creates a new rule if a default facet rule does not already exist. If a default facet rule does exist, adding a facet updates that rule with the new facet.
### Enable the Facet Zone
Facets appear in the zone on the left panel of the Experience Optimizer, once enabled. Follow the steps below to turn on the facets zone:
1. Click **Start Task**.\
The button appears.
2. In the left panel, click **+Add** and enter the desired parameters.
3. Click **Done** when you’ve finished.
You can also add facets that are not dependent on rules being triggered. To do this, edit the Facet Zone widget.
### Edit the Facet Zone widget
1. In the facet zone, click **+Add**. The **Edit Rule** dialog box appears.
2. Click the **Facet Field** field, and enter the desired facet parameter.
You can add multiple facet parameters by clicking the **+ Add** button:
3. Click the **Save** button to add your facet:
### Add Additional Facets
1. Hover your cursor above or below an existing facet. A **+** button appears:
2. Click the **+** button.
## Move a Facet
1. Hover your cursor over a facet. A menu overlay appears at the top.
2. Click-and-hold the drag bar:
3. Move the facet to the desired location.
4. Release the click.
## Remove a Facet
1. Hover your cursor over a facet. A menu overlay appears at the top.
2. Click the **Remove** button:
## Block a Facet Item
1. Hover your cursor over a facet item. A menu overlay appears.
2. Click the **Block** button:
## Bury a Facet Item
1. Hover your cursor over a facet item. A menu overlay appears.
2. Click the **Bury** button:
## Boost a Facet Item
1. Hover your cursor over a facet item. A menu overlay appears.
2. Click the **Boost** button:
### Reorder Boosted Facet Items
1. Hover over a boosted facet item. A drag bar appears.
2. Click and hold the drag bar:
3. Move the facet item to the desired location.
4. Release the click.
## Start a task
1. Click the **Start Task** button:
This enables Interactive Mode. In Interactive Mode, you can interact directly with some of the widgets on the Experience Optimizer board, creating and managing business rules as you simulate your customers' searches. You can:
* Pin, block, boost, and bury documents.
* Set additional facets.
* Create search rewrites from poorly performing search terms.
* See which rules and search rewrites are triggered by a search, then edit or delete them as needed.
* Sort search results by ascending or descending order, or by relevance.
* Check product information.
## Undo changes in a task
1. Click the **Undo All** button: . A confirmation dialog box will appear.
2. Select **Undo** to confirm.
## Exit a task
Exiting a task will
1. Click the **Exit** button:
## Publish a task
You can publish any changes you make to rules back to the Fusion data source where they can impact live queries.
1. Click the **Publish** button: . A confirmation dialog box will appear.
2. Select **Publish** to confirm.
Experience Optimizer counts unpublished changes in the top navigation bar:
# Rule Conflict Resolution
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/experience-optimizer/rule-conflict-resolution
Experience Optimizer
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/curating-your-search-experience/experience-optimizer/rule-conflict-resolution
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/experience-optimizer/rule-conflict-resolution
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/e9f420
Sometimes, rules created in the [Rules Editor](/docs/5/fusion/getting-data-out/query-enhancement/rules-editor) and Experience Optimizer conflict with each other. For example, if you have 3 rules to boost, bury, and block a document, Experience Optimizer highlights these conflicting rules so you can address them.
To highlight conflicting rules, the **Rules Fired** section will display a red warning icon next to the dominant rule and a red flag on the conflicting rules.
By clicking the **Edit** button, you can view the details of the conflicting rules so the conflict can be resolved:
The details of the conflicting rules affecting specific products are found in the document details panel. Click the **Start Task** button
Hover over a document, and click the **Document Details** button.
# Templates UI Configuration Options
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/templates-config
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/templates-config
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/curating-your-search-experience/predictive-merchandiser/templates-config
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/8857
The Templates screen in the Rules Editor allows you to design, test, and implement a wide variety of search experiences. Results are conditionally delivered from multiple different query pipelines, enabling you to use Fusion’s search, browse, and AI-driven functionality throughout your site.
## Configuration options
### Templates
When creating or editing a template, the status, such as active or stale, is indicated after the template name.
The active status can hold the following values:
| Tag | New | Saved | Published | Active |
| - | - | - | - | - |
| **Editing** | ✅ | | | |
| **Stale** | | ✅ | | |
| (nothing) | | | ✅ | ✅ |
Configuration options for templates:
| Option | Description |
| - | - |
| Precedence | If multiple templates are triggered, this setting determines which template is displayed. Lower numbers are considered higher precedence. For example, a template with precedence of `999` will trigger before a template with precedence of `1`. |
| Display Name | The name of the template. |
| Type | The type of template. This is an arbitrary value classifying the template. It has no effect on triggering/rendering logic unless the value is `Typeahead`. If `Typeahead` is assigned, the template will be excluded from `/trigger` and `/render` endpoints. This value is read-only and for informational purposes only. |
| Zones | Determines which zones are loaded when the template is triggered. Zones are displayed in the order they appear. |
| Triggering | The conditions that must be met for the template to display • **Date Range.** Define a date range to trigger the display of the template. For example, set to display during a Black Friday sales event. • **Trigger.** Define triggers to display the template. Triggers are added with the **Trigger** button • **Search Term.** Search terms required in the query for this trigger to match. Configured analysis will tokenize and process these terms. • **Match Mode.** Defines search term matching criteria: • `phrase` requires the search tokens of this trigger to exist as a phrase in the same order. • `all` requires the search tokens to exist in any order. • `exact` requires search tokens to match exactly in the same order, applying configured analysis (e.g. stemming). • **Filters.** `fq` values required in the query for this trigger to match. Additional filters may be present and still satisfy this trigger. • **Exact.** If **On**, the nav state must match this trigger’s criteria exactly (no extra filters or terms if none defined). |
| Matches All Locations | If set to **On**, this template is always displayed, and triggers are ignored. |
#### Triggering
The date range trigger is activated alongside triggers with an `AND` logical operator. Conversely, triggers are activated alongside other triggers with an `OR` logical operator. For example, consider the following scenarios:
| Scenario 1 | | Scenario 2 | |
| - | - | - | - |
| Date Range | *none* | Date Range | `November 25, 2020 - December 5, 2020` |
| Search Term | `cyber monday` | Search Term | `cyber monday` |
| Search Term | `sales event` | Search Term | `sales event` |
In **Scenario 1**, no date range is configured. The template is triggered if:
* The user searches the term `cyber monday` *or* `sales event`.
In **Scenario 2**, a date range is configured. The template is triggered if:
* Today’s date is within the date range `November 25, 2020 - December 5, 2020`, *and* the user searches the term `cyber monday`
* Today’s date is within the date range `November 25, 2020 - December 5, 2020`, *and* the user searches the term `sales event`
The template is *not* triggered if:
* Today’s date is not within the date range `November 25, 2020 - December 5, 2020`
* Today’s date is within the date range `November 25, 2020 - December 5, 2020`, but the user does not search the term `cyber monday` *or* `sales event`
### Zones
| Option | Description |
| - | - |
| Display Name | The name of the zone. |
| Type | An arbitrary value classifying the zone. It has no effect on triggering/rendering logic. Intended only to be used by the PM UI. This value is read-only and for informational purposes only. |
| Query Profile | Determines query profile the zone will fetch results from. |
| Tags | Tags are passed to the query profile when rendering. They are combined with tags found in the request parameters of the render request. |
| Omit Filters from Query | If set to **On**, the filters (`fq`) are ignored. For example, if you want the zone to display items with type `sunglasses` even though the user has filtered the results to `shorts`, enabling this setting allows the zone to display sunglasses. |
| UI Treatment | Arbitrary text that is included in the response when rendering the zone. This is generally used by the frontend to determine how to display the zone. UI Treatment tags accept a color value to help distinguish them in the Predictive Merchandiser screen: |
| Results Layout | Select grid or list view. |
| Display Fields | Add field names and how they are displayed (type of field). |
| Product Image | Field name for the product image. |
| Product Title | Field name for the product title. |
| Product Description | Field name for the product description. |
Product Image, Product Title, and Product Description are only used for rendering products in Predictive Merchandiser. Your search application will determine what fields are used in production.
# Analytics
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/data-analytics/app-insights/analytics
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/data-analytics/app-insights/analytics
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/data-analytics/app-insights/analytics
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/459
Analytics include a standard set of reports, plus the ability to create custom reports. Custom reports can also be defined on the [Dashboards](/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/overview) and [Events](/docs/5/fusion/getting-data-out/data-analytics/app-insights/events) pages. Report data can be filtered by time or by free text search, and reports for tabular data can be exported by clicking **Export table**.
## Standard reports
To view the standard reports, click **Analytics**, then select one of the standard reports:
* Facets used
* Facet filters applied
* Application servers
* Applications
* Response times
* URLs clicked
* Type of query
* Search pages
* Search platforms
* Types of response
* Visitor countries
* Visitor cities
* Browsers
* Operating System
* Device types
* Websites people are coming from
* Users
* User domains
* Types of event
* Head Tail analysis
# Dashboards in App Insights
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/data-analytics/app-insights/dashboards
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/data-analytics/app-insights/dashboards
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/data-analytics/app-insights/dashboards
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/458
When you open the Dashboards page, it displays graphs and tables for attributes that are common to all signals. For more specific charts, you can open the **All Dashboard Categories** menu at the top and select a category:
## Dashboard filtering
You can filter by timeframe using the **All Time** menu:
To filter by the content of the data, enter a string in the **Filter this dashboard…** text box:
For more information, see [dashboards](/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/overview).
# Events
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/data-analytics/app-insights/events
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/data-analytics/app-insights/events
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/data-analytics/app-insights/events
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/457
Click **Events** to view histograms and timelines about events, filtered by:
import from '/snippets/LwTemplate.jsx';
* event type
* users
* request/response information
# Experiment Results
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/data-analytics/app-insights/experiment-results
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/data-analytics/app-insights/experiment-results
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/data-analytics/app-insights/experiment-results
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/455
If you have configured an [experiment](/docs/5/fusion/getting-data-out/data-analytics/experiments/overview), you can see visualizations about the results here.
This example shows a 92% confidence index for the difference in click-through rate (CTR). Conversions use the same signal type (`click`), so the result is identical.
# App Insights
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/data-analytics/app-insights/overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/data-analytics/app-insights/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/data-analytics/app-insights/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/456
**Important**
App Insights is deprecated in Fusion 5.12.
App Insights provides detailed, real-time, searchable reports and visualizations derived from your [signals data](/docs/5/fusion/getting-data-out/query-enhancement/signals/overview). It also provides alerts and triggers to notify you when specific events occur.
* To open App Insights from the Fusion workspace, navigate to **Analytics** > **Insights**.
* To switch apps while in App Insights, hover over , and then click a different app.
* To exit App Insights, hover over , and then click **Return to Fusion**.
## App Insights pages
When App Insights is open, you can hover over the left navigation panel to reach these pages:
* [Dashboards](/docs/5/fusion/getting-data-out/data-analytics/app-insights/dashboards)\
View graphs and tables about:
* requests
* queries
* results
* clicks
* users
* sessions\
You can filter by timeframe or by the content of the data, and create custom reports.
* [Events](/docs/5/fusion/getting-data-out/data-analytics/app-insights/events)\
View histograms and timelines about events, filtered by:
* event type
* users
* request/response information\
You can also create custom reports about events here.
* [Sessions](/docs/5/fusion/getting-data-out/data-analytics/app-insights/sessions)\
View charts and timelines about search events, filtered by:
* user
* subject
* event duration
* events per session
* [Analytics](/docs/5/fusion/getting-data-out/data-analytics/app-insights/analytics)\
A variety of standard reports are available here. You can also define custom reports to suit your needs.
* [Experiment results](/docs/5/fusion/getting-data-out/data-analytics/app-insights/experiment-results)\
If you have configured an [experiment](/docs/5/fusion/getting-data-out/data-analytics/experiments/overview), you can see visualizations about the results here.
## Reports
App Insights provides a standard set of reports, plus the ability to **Create Custom Reports**. Standard reports are located on the [Analytics](/docs/5/fusion/getting-data-out/data-analytics/app-insights/analytics) page, while custom reports can be defined on the [Dashboard](/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/overview), [Events](/docs/5/fusion/getting-data-out/data-analytics/app-insights/events), or Analytics pages. Report data can be filtered by time or by free text search, and reports can be exported by clicking **Export table**.
There are several ways to create a custom report:
**On the Analytics page for App Insights**
1. Apply one or more filters.
2. Click **+ New custom report**.
3. Define the report’s parameters.
4. Click **Create and launch report**.
**On the Dashboard page**
1. Filter any dashboard.
2. Scroll to the bottom.
3. Click **\ events match your criteria. View Now.**
This takes you to the [Events](/docs/4/fusion-ai/concepts/insights/events) page, where you can navigate to **Analytics** > **+New Custom Report**. Define additional parameters for the report, then click **Create and launch report**.
# Requirements for Reports and Dashboards
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/data-analytics/app-insights/report-requirements
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/data-analytics/app-insights/report-requirements
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/data-analytics/app-insights/report-requirements
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/507
Although there are many fields that are relevant to [signals](/docs/5/fusion/getting-data-out/query-enhancement/signals/overview), this topic describes which fields are **required** for each [type of signal](/docs/5/fusion/getting-data-out/query-enhancement/signals/types-and-structures). Without these fields, reports and dashboards may not function as expected.
| Report | Signal Types Used | Required Field | Type | Description | Example |
| - | - | - | - | - | - |
| Search queries | ✘ Annotation ✅ Click ✘ Login ✅ Request ✅ Response | `query` | string | Terms of the query. | `physics` |
| Facets used | ✘ Annotation ✘ Click ✘ Login ✅ Request ✘ Response | `filter_field` | array of string | List of filters associated with the query, which in turn is associated with signal. | `["categories_s"]` |
| Facet filters applied | ✘ Annotation ✘ Click ✘ Login ✅ Request ✘ Response | `filter` | array of string | List of **applied** filters associated with the query, which in turn is associated with signal. | `["categories_s/math.OA"]` |
| Application servers | ✘ Annotation ✅ Click ✘ Login ✅ Request ✘ Response | `host` | string | Host name of the server that is hosting the app, which in turn is generating the signal. | `lucidworks` |
| Response times | ✘ Annotation ✘ Click ✘ Login ✘ Request ✅ Response | `time` | number | Response time (in milliseconds). | `17` |
| URLs clicked | ✘ Annotation ✅ Click ✘ Login ✘ Request ✘ Response | `url` | string | URL of the page that was selected from the search results. This typically results from a click signal. | `http://arxiv.org/pdf/astro-ph/0611688v1` |
| Search pages | ✘ Annotation ✘ Click ✘ Login ✅ Request ✘ Response | `page_title` | string | Title of the page that the signal originated from. | `Search Preview` |
| Search platforms | ✘ Annotation ✅ Click ✘ Login ✅ Request ✅ Response | `platform` | string | Name of the search platform. | `fusion` |
| Visitor countries | ✘ Annotation ✅ Click ✘ Login ✅ Request ✘ Response | `geo_country_name` | string | Originating country of the user. | `United States` |
| Visitor cities | ✘ Annotation ✅ Click ✘ Login ✅ Request ✘ Response | `geo_city_name` | string | Originating city of the user. | `San Francisco` |
| Browsers | ✘ Annotation ✅ Click ✘ Login ✅ Request ✘ Response | `browser_name` | string | Name of the browser that generated the signal. | `Firefox` |
| Operating System | ✘ Annotation ✅ Click ✘ Login ✅ Request ✘ Response | `os_name` | string | Name of the operating system used by the user. | `Ubuntu` |
| Device types | ✘ Annotation ✅ Click ✘ Login ✅ Request ✘ Response | `os_device` | string | Type of the device used by the user. | `Computer` |
| Websites people are coming from | ✘ Annotation ✅ Click ✘ Login ✘ Request ✘ Response | `referrer_domain` | string | URL of the page that the signal originated from. | `http://www.google.com` |
| Users | ✘ Annotation ✅ Click ✘ Login ✅ Request ✘ Response | `user_id` | string | Unique ID for the user that generated the signal. | `admin` |
| User domains | N/A | `domain` | string | Domain of the user that generated the signal. | `lucidworks.com` |
| Types of event | ✅ Annotation ✅ Click ✅ Login ✅ Request ✅ Response | `type` | string | Type of the signal. | `click` |
Head Tail analysis uses different fields depending on signal type.
This report is populated by running a head tail job in Lucidworks Search. With a collection called `foo`, for example, run the `foo_head_tail` job.
| Signal Types Used | Required Field | Type | |
| - | - | - | - |
| ✘ Annotation | | | |
| ✅ Click | type | string | |
| ✅ Click | query | string | |
| ✘ Login | | | |
| ✘ Request | | | |
| ✅ Response | `count_i` | number | |
The fields mentioned above are defined as follows:
| Field Name | Example Value | Description |
| - | - | - |
| `count_i` | 1 | Number of times an interaction event occurred with this item |
| `doc_id` | NMDDV | Product ID or Item ID |
| `query` | xwearabletech | A query string from the user |
| `session_id` | 91aa66d11af44b6c90ccef44d055cf9a | Id for session in which user generated the signal |
| `type` | quick\_view\_click | Type of session the user used to interact with the platform |
| `user_id` | 11506893 | ID of user during the session |
| `timestamp_tdt` | 2018-11-20T17:58:57.650Z | Time when signal was generated |
Some [signal types](/docs/5/fusion/getting-data-out/query-enhancement/signals/overview), including [custom signal types](/docs/5/fusion/getting-data-out/query-enhancement/signals/types-and-structures#custom-signal-types), may include additional fields.
# Sessions
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/data-analytics/app-insights/sessions
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/data-analytics/app-insights/sessions
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/data-analytics/app-insights/sessions
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/454
Click **Sessions** to view charts and timelines about search events, filtered by:
import from '/snippets/LwTemplate.jsx';
* user
* subject
* event duration
* events per session
Charts are located on the left, below the filters. Scroll down or minimize the filters to view the charts:
Hover over the bars on any chart to filter the timeline by event duration or events per session:
# Evaluation Metrics
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/data-analytics/experiments/evaluation-metrics
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/data-analytics/experiments/evaluation-metrics
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/data-analytics/experiments/evaluation-metrics
[old doc.lw link]: none
Evaluation metrics are computed over a set with known query/document relevancy. Also known as offline metrics, search evaluation metrics help measure how effective a system is at returning relevant and well-ranked results. Many of these metrics are evaluated `@k`, meaning they consider only the top k results returned by the search engine.
Compare to online metrics, which include Average Order Value (AOV) and Click-Through Rate (CTR).
## Computing evaluation metrics
When a metric is computed `@k`, it measures performance using only the top k ranked results for a given query. Once `@k` metrics are computed for each query, they are averaged, etc.
Examples:
* `Precision@5` considers only the top 5 results.
* `Recall@10` checks how many relevant documents are retrieved in the top 10.
* `nDCG@3` rewards ranking quality within the top 3 results.
Default values for `@k` include `@1`, `@3`, `@5`, or `@10`, depending on how deeply users typically browse the result list.
## Evaluation Metrics
Each evaluation metric has different formulas and use cases.
| Metric | Measures | Rank-Aware | Uses Graded Relevance | Interpretable As |
| :- | :- | :- | :- | :- |
| Recall\@k | Coverage of relevant documents | ❌ | ❌ | Percentage of relevant docs retrieved |
| F1\@k | Balance between precision and recall | ❌ | ❌ | Unified classification score |
| Hit\@k | Presence of any relevant result | ❌ | ❌ | Binary success or failure |
| MRR\@k | Position of 1st relevant result | ✅ | ❌ | Early relevance ranking |
| nDCG\@k | Overall ranking quality | ✅ | ✅ (optional) | Best results high in list |
| MAP\@k | Aggregate relevance and ranking | ✅ | ❌ | Average quality over queries |
### Use cases
| Short Description | Use Case | Recommended Metrics |
| :- | :- | :- |
| Quick success checks | Site-search, Knowledge Management, Ecommerce | Hit\@k, MRR\@k |
| Early relevance (fast find-ability) | Knowledge Management, Ecommerce | MRR\@k, nDCG\@k |
| Ranking quality | Ecommerce | nDCG\@k, MRR\@k, MAP\@k |
| Multi-relevant answers | Ecommerce, Knowledge Management | nDCG\@k, MAP\@k, Recall\@k |
| Classification | Classification | F1\@k, Recall\@k |
### Precision\@k
`Precision@k` is a straightforward metric that measures the proportion of relevant items among the top `k` results in a ranked list. This metric is useful for measuring overall system performance and relevance.
* What it is: The number of relevant items in the top `k` results, divided by `k`
* Formula: `(Number of relevant items in the top k) / k`
### Hit\@k
`Hit@k` is a straightforward but powerful metric that mimics accuracy by checking whether the system succeeded in returning something relevant at all. This metric is useful as a quick success check and intuitive to interpret, as the answer is either `0` or `1`.
* What it is: A binary indicator that states whether any relevant document present in the top k results
* Formula: `Hit@k = 1` if any relevant document is in top k, else `Hit@k = 0`
### Mean Average Precision (MAP\@k)
`MAP@k` evaluates how many results out of `k` results are relevant. It uses both relevance and ranking quality across the full result list, rewarding systems that consistently return relevant results near the top. This metric is especially useful in ecommerce and multi-relevant-answer scenarios, where it's expected for each query to have many relevant documents or products.
* What it is: The mean of average precision scores across all queries, computed at cutoff k
* Formula (per query): `Average Precision@k = Mean of precision values at each rank i ≤ k where document_i is relevant`
* Then: `MAP@k = Mean of Average Precision@k over all queries`
### Mean Reciprocal Rank (MRR\@k)
`MRR@k` captures how high in the search rankings a user finds a relevant result. It rewards systems that surface correct answers earlier in the result list.
* What it is: The average reciprocal of the rank at which the first relevant document appears, up to position k
* Formula: `MRR@k = 1 / rank of first relevant doc` (if rank ≤ k, else 0)
### Normalized Discounted Cumulative Gain (nDCG\@k)
`nDCG@k` is order-aware and optionally weight-aware (graded-relevance). It penalizes systems that bury relevant results and rewards those that put the most relevant documents near the top. If graded relevance scores are unavailable, binary nDCG is computed (relevant vs. not relevant, where all relevant documents have a weight/grade of 1).
* What it is: Measures both the presence and position of relevant documents in the top k, while optionally considering graded relevance (that is, some results are more relevant than others)
* Formula (simplified): `DCG@k = Σ (2^relevance_i / log2(i + 1)) for i = 1 to k`
* `nDCG@k = DCG@k / IDCG@k (ideal DCG)`
### Recall\@k
`Recall@k` captures completeness. Even if many relevant results exist, if they aren't surfaced early, users may never find them. While `Precision@k` measures the proportion of relevant documents out of a specified `k` results, `Recall@k` measures the proportion of relevant documents that show up early in the results out of the total relevant documents.
Recall\@k is bounded by `k / total_relevant_docs`. For example, if you compute `Recall@5` for a query that has 10 relevant documents, the `Recall@5` value cannot exceed 0.5 even if all 5 results are relevant.
* What it is: The proportion of all relevant documents that appear in the top k
* Formula: `Recall@k = (Relevant documents in top k) / (Total relevant documents)`
### F1\@k
`F1@k` balances both the accuracy (precision) and coverage (recall) of search results, giving a holistic performance score at depth k. This is a classification-specific metric.
* What it is: The harmonic mean of `Precision@k` and `Recall@k`
* Formula: `F1@k = 2 * (Precision@k * Recall@k) / (Precision@k + Recall@k)`
# Experiment Metrics
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/data-analytics/experiments/experiment-metrics
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/data-analytics/experiments/experiment-metrics
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/data-analytics/experiments/experiment-metrics
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/3229
This section describes metrics available for experiments.
## Click-Through Rate
The Click-Through Rate (CTR) metric provides the rate of clicks per query for a variant. The CTR is a number between 0 and 1, that is, what proportion of queries lead to clicks. Variants with a CTR closer to 1 perform better than variants with a lower rate.
CTR is *cumulative*, that is, each time it is calculated, it is calculated from the beginning of the experiment. After each variant has reached a stable level, you should not see large day-to-day fluctuations in the CTR.
The automatically-created job that generates the Click-Through Rate metrics is named `-`, for example, `Experiment-CTR`.
## Conversion Rate
The Conversion Rate metric provides the rate of some type of signal per variant, that is, what proportion of queries lead to some type of signal, such as `cart`, `purchase` or `like` signals. (These signal types are not predefined.)
For example, if you are interested in how many queries convert into `cart` signals, specify the `cart` signal type in the conversion rate metric.
The Click-Through Rate metric is a conversion rate for `click` signals.
The automatically-created job that generates the Conversion Rate metrics is named `-`, for example, `Experiment-Conversion`.
## Mean Reciprocal Rank (MRR)
The Mean Reciprocal Rank (MRR) metric measures the position of documents that were clicked on in ranked results. It ranges from 0 (at the very bottom) to 1 (at the very top). MRR penalizes clicks that occur further down in the results, which indicate a ranking issue where relevant documents are not ranked high enough. Variants with an MRR closer to 1 indicate that users are clicking on documents that have higher ranks.
The automatically-created job that generates the Mean Reciprocal Rank metrics is named `-`, for example, `Experiment-MRR`.
## Response Time
The Response Time metric computes the named statistic (for example, `mean`, `variance` or `max`) from response-time data. The default statistic is `avg` (average, the same as `mean`).
You can use the Response Time metric to evaluate the impact of adding additional stages to a query pipeline, for example, a recommendation or machine learning stage.
The response time is the end-to-end processing time from when a query pipeline receives a query to when the pipeline supplies a response:
* **No Experiment stage.** If a query pipeline does not have an Experiment stage, then there is no experiment-processing overhead in the response times.
* **Experiment stage.** If a query pipeline includes an Experiment stage, then processing by that stage is included in the response times.
The automatically-created job that generates the Response Time metrics is named `-`, for example, `Experiment-Response_time`.
### Supported functions
When adding the Response Time metric to an experiment, specify one of these Spark SQL function names or aliases for the **Statistic**.
| Function name or alias | Description |
| - | - |
| `avg` | Mean response time |
| `kurtosis` | Kurtosis of the response times |
| `max` | Maximum response time |
| `mean` | Mean response time |
| `median` | Median response time. This is an alias for `percentile(query_time,0.5)`. |
| `min` | Minimum response time |
| `percentile_N` | Nth percentile of the response times, that is, the value at or closest to the percentile. `N` is an integer between 1 and 100. This is an alias for the function `percentile(query_time,N/100)`. |
| `skewness` | Skewness of the response times |
| `sum` | Sum of the response times |
| `stddev` | Standard deviation of the response times |
| `variance` | Variance of the response times |
For more information about these functions, see the documentation for [Spark SQL Built-in Functions](https://spark.apache.org/docs/latest/api/sql/index.html).
## Custom SQL
Under the covers, Fusion computes all experiment metrics using Fusion’s [SQL aggregation](/docs/5/fusion/reference/config-ref/jobs/aggregations/sql-aggregation) engine.
The Custom SQL metric lets you define your own SQL to compute a metric per variant. The SQL must project these three columns in the final output, and perform a GROUP BY on `variant_id`:
* `value`.\* A double field that represents the metric provided by this custom SQL
* `count`.\* The number of rows used to compute the value for a variant, that is, how many signals contributed to this value
* `variant_id`. The unique identifier of the variant
An internal view named `variant_queries` is built into the experiment job framework. This view is transient and is not defined in the table catalog; it only exists for the duration of the metrics job. The `variant_queries` view exposes all response signals for a given variant ID. The `variant_queries` view exposes the following fields pulled from response signals:
| Field | Description |
| - | - |
| `id` | Response signal ID set by a query pipeline and returned to the client application using the `x-fusion-query-id` response header |
| `variant_id` | Experiment variant this response signal is associated with |
| `query_doc_ids` | Comma-delimited list of document IDs returned in the response, in ranked order |
| `query_timestamp` | ISO-8601 timestamp for the time when Fusion executed the query |
| `query_user_id` | User associated with the query. The front-end application must supply this. |
| `query_rows` | Number of rows returned for this query, that is, the page size |
| `query_hits` | Total number of documents that match this query, that is, the number of documents that were found |
| `query_offset` | Page offset |
| `query_time` | Total time to execute the query (in milliseconds) |
You can use the `fusion_query_id` field to join the `variant_signals` view with other signal types such as `click`. For example, if you want to get a count of clicks per variant, you would use:
```sql wrap theme={"dark"}
1: SELECT COUNT(1) AS value, COUNT(1) AS count, vq.variant_id as variant_id
2: FROM ${inputCollection} c
3: INNER JOIN variant_queries vq ON c.fusion_query_id = vq.id
4: WHERE c.type = 'click'
5: GROUP BY variant_id
```
In this SQL:
* At line 1, we project the required `value`, `count`, and `variant_id` columns as the output for our custom SQL; this is required for all custom SQL metrics.
* At line 2, we use a built-in macro that represents the input collection for our metrics job. The SQL engine replaces the `${inputCollection}` variable with the correct collection name at runtime, which is typically a signals collection.
* At line 3, we use the `fusion_query_id` column to join `click` signals with the `id` column of the `variant_queries` view. This illustrates how the `variant_queries` view helps simplify the SQL you have to write to build a custom metric.
* At line 4, we filter signals to only include `click` signals. Behind the scenes, Fusion will send a query to Solr with `fq=type:click`.
* At line 5, we group by the variant\_id to compute the aggregated metrics for each variant; all Custom SQL must perform a group by variant\_id.
To illustrate the power of Custom SQL metrics for experiments, let us build the SQL to compute the average page depth of clicks for each variant, to indicate if users are having to navigate beyond the first page to find results. The intuition behind this metric is that variants having a higher average page depth might indicate a ranking problem. Users are not finding relevant documents on the first page of results.
Specifically, to build our query, we need the `query_offset` and `query_rows` columns associated with each click in a variant:
```sql wrap theme={"dark"}
SELECT AVG((vq.query_offset/vq.query_rows)+1) as value,
COUNT(1) as count,
vq.variant_id as variant_id
FROM ${inputCollection} c
INNER JOIN variant_queries vq ON c.fusion_query_id = vq.id
WHERE c.type = 'click'
GROUP BY variant_id
```
In practice, MRR is a better metric for determining the ranked position of clicks, but this SQL gives a basic illustration of how to build Custom SQL metrics.
Lastly, when building Custom SQL metrics, you have the full power of Spark SQL functions, see: [https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.functions\$](https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.functions\$).
The automatically-created job that generates the Custom SQL metrics is named `-`, for example, `Experiment-SQL`.
## Query Relevance
The Query Relevance metric calculates the performance of queries against a "gold standard" or "ground truth" dataset that lists which documents should be returned for each query. You can either predetermine the queries that will be used and the documents that should be returned, and place them in a Solr collection in the correct format, or let the `groundTruth` job use historical click signals to generate the ground truth data automatically.
Note that the Query Relevance metric *does not* calculate metrics based on live traffic. Instead, it issues the queries specified in the ground truth collection against each variant, and calculates the performance of the queries.
The jobs that generate the Query Relevance metrics are named `-groundTruth-` and `-rankingMetrics-`, for example, `Experiment-groundTruth-QR` and `Experiment-rankingMetrics-QR`.
**Important**
You must run the `groundTruth` job by hand the first time. Query Relevance `rankingMetrics` jobs that run before the `groundTruth` job runs do not produce metrics. Subsequently, the `groundTruth` job runs once a month.
### Ground Truth Queries
Query relevance metrics rely on having a set of queries and a list of documents that should be returned for those queries in ranked order. Specifically, a ground truth dataset contains tuples of query + document ID + weight, such as the following data for a fictitious Home Improvement search application:
| Query | Document ID | Weight |
| - | - | - |
| hammer | 123 | 0.9 |
| hammer | 456 | 0.8 |
| hammer | 789 | 0.7 |
| masking tape | 234 | 0.85 |
| masking tape | 567 | 0.82 |
| masking tape | 890 | 0.76 |
Typically, the queries included in the ground truth set represent important queries for a given search application. The weight assigned to each document is used to determine the expected ranking order for the query. Ideally, your ground truth dataset should specify the same number of documents per query, for example. 10. But this is not required technically for computing query relevance metrics. In other words, one query can have 10 documents specified and another query can only specify 5.
In Fusion, you can either load a curated ground truth dataset into a Fusion collection or use Fusion’s ground truth job to build a ground truth dataset using signals. If you use the ground truth job, Fusion looks at click/skip behavior for documents by analyzing response and click signals. It follows that you need a sufficient number of signals to generate an accurate ground truth dataset.
The basic intuition behind the ground truth job is that for queries that occur frequently in your search application, whether a user clicks or skips over a document serves as a relevance judgement of a document for a given query. With a sufficient sample size per query, Fusion can decide which documents are relevant and which are not for any given query. It is important to note, however, that, because the ground truth dataset is generated from your click signals, if you have relevant documents that are never clicked (maybe because they are on the second page of results), then they will never appear in your ground truth set.
### Calculating Performance vs. Ground Truth
After you have a ground truth dataset loaded into Fusion, the Query Relevance metric will calculate all of the following metrics:
**Precision**
Precision is the fraction of returned documents that are relevant to the query (that is, how many of the documents returned by this variant exist in the ground truth dataset).
**Recall**
Recall is the fraction of total relevant docs that are returned by this query (that is, how many of the documents in the ground truth set appear in the result set for this variant).
**Normalized Discounted Cumulative Gain (nDGC)**
The Normalized Discounted Cumulative Gain (nDCG) indicates whether a variant is returning highly relevant documents near the top of results.
The nDCG has a value between 0 and 1. Larger values indicate that more highly relevant documents occur earlier in the results for a query. Conversely, if a variant returns highly relevant documents lower in the results, then its nDCG score will be lower, penalizing the ranking strategy used by the variant for returning highly relevant documents lower in the results. For more details on nDCG, see [https://en.wikipedia.org/wiki/Discounted\_cumulative\_gain](https://en.wikipedia.org/wiki/Discounted_cumulative_gain).
**F1**
The F1 score is the harmonic mean between precision and recall at a given depth (10 by default). The F1 score ranges between 0 and 1, with larger values indicating that a variant is achieving a better balance of precision and recall than variants with lower F1 scores. For more details, see [https://en.wikipedia.org/wiki/F1\_score](https://en.wikipedia.org/wiki/F1_score).
**Mean Average Precision (MAP)**
The Mean Average Precision (MAP) metric indicates how many documents returned for a query, down to a specific depth, are considered relevant to a query averaged over all queries in the ground truth dataset. MAP is a value between 0 and 1. Larger values mean that the variant returns more relevant than non-relevant documents. For example, if the relevance judgement for a result set containing 3 documents is: 1, 0, 1, then the average precision for that query will be 1/1, 0, ⅔ \~ 0.834 (1.667/2).
# Experiments
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/data-analytics/experiments/overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/data-analytics/experiments/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/data-analytics/experiments/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/460
When making changes to a query pipeline or query parameters that will affect users' search experience, it is often a good idea to run an experiment in order to verify that the results are what you intended. Fusion lets you create and run experiments that take care of dividing traffic between variants and calculating the results of each variant with respect to configurable objectives such as purchases, click-through rate, or search relevance.
There are two ways that a search application might interact with an experiment:
* Preferred: Using a query profile
* Alternative: Using an Experiment query pipeline stage
If a query profile is configured to use an experiment, then a search app sends queries and signals to the query profile endpoint. If the experiment is active, then Fusion routes each query through one of the experiment variants. The search app will also send subsequent signal data relating to that query — clicks, purchases, "likes", or whatever is relevant to the application — to that same query profile, and Fusion will record it along with information about the experiment variant that the user was exposed to.
Fusion generates and stores the data that metrics calculations use. It also automatically creates jobs that periodically calculate the metrics. After metrics have been calculated, they are available in [App Insights](/docs/5/fusion/getting-data-out/data-analytics/app-insights/overview).
This topic explains the experiment workflow and basic concepts. These additional topics provide details about how to implement experiments to improve the user experience:
* **Plan an Experiment**
* **Set Up an Experiment using a Query Profile**
* **Run an Experiment**
* **Analyze Experiment Results**
From a planning standpoint, an experiment has these parts:
* **A baseline control.** One of the experiment variants will be the control. This is "how we are doing things today." If you are experimenting from the start, choose the simplest variant as the control.
* **Experiment variants.** Experiment variants other than the control are attempts to improve the user’s extended search experience. Which relevancy strategy works best for your search app and your users?
* **Metrics.** This is how you know whether the search variants produce differences in user interactions, and whether the differences are statistically significant.
In the remainder of this topic, you will make decisions about these broad areas, as well as about experiment details.
## Plan what you want to vary
Identify different relevancy strategies, where each represents a hypothesis about which user experience will drive more click-throughs, purchases, and so on. Use the [Query Workbench](/docs/5/fusion/getting-data-out/query-basics/query-workbench) to explore how to produce different search results and recommendations using different query pipelines, and evaluate which ones might engage your users most effectively.
## Plan what you want to measure
Metrics compare the control against other variants pairwise. For example, if the variants are `experiment`, `B`, `C`, and `D`, and you choose `experiment` as the control, then the comparisons for which metrics are generated will be `experiment/B`, `experiment/C`, and `experiment/D`.
You can [learn more about metrics](/docs/5/fusion/getting-data-out/data-analytics/experiments/experiment-metrics).
## Design the experiment
When designing an experiment, you must make these decisions:
* How users are identified
* Percentage of total traffic to send through the experiment
* Number of variants and how they differ
* Metrics to generate
In many cases identifying users is straightforward, using an existing user ID or session ID if the application has one. In other cases, you may need to generate an identifier of some sort to send in on queries. It is important to send in some kind of identifier with each query so that the experiment can route the query to a variant, and to send that same identifier with any subsequent signals that resulted from that query. Queries without a user ID will not be routed through the experiment.
The percentage of total traffic to send through the experiment is the one variable that can change over the course of the experiment. It is often a good practice to start out sending only a small percentage of search traffic through a new experiment, in order to verify that each of the variants are functioning properly. Then, once you have established that the behavior is as intended, you can increase the percentage of traffic through the experiment to the desired level.
With modest usage and for a possibly small effect, or when testing multiple variants at the same time, you might want to send 100% of users through the experiment and let it run longer. For high usage and an effect that is expected to be larger, and with only two variants, you might not need to send all users through the experiment and the experiment will not take as long.
## Choose traffic weights
Fusion AI uses *traffic weights* to apportion search traffic among the variants. This allows you to send a different percentage of traffic through each variant if desired.
### Automatic traffic weights (multi-armed bandit)
Automatic traffic weights are only available in Fusion 4.2+.
The **Automatically Adjust Weights Between Variants** configuration option enables multi-armed bandits and eliminates the need to specify a traffic weight for each variant.
In multi-arm bandit mode, [metrics](/docs/4/fusion-ai/reference/metrics) jobs are created automatically once the experiment starts. In Fusion 4.2.x, schedules are also automatically created. In Fusion 5.x, however, the jobs must be scheduled manually. It is recommended that you schedule the metrics jobs to run on an hourly basis. The weights between variants only change after the metrics jobs run.
Fusion’s multi-arm bandit implementation uses a variation of [Thompson Sampling](https://en.wikipedia.org/wiki/Thompson_sampling) (sometimes called Bayesian Bandits). This algorithm uses the current count of successes versus failures to build a beta distribution that represents the level of confidence in the primary metric value for each variant. It then samples a random number from each variant’s distribution, and picks the highest number.
This type of implementation has three effects:
* It weights better-performing variants higher.
Since the beta distribution of each variant is centered around the primary metric value for that variant, a random number selected from a higher-performing variant is likely to be higher than a random number picked from a lower-performing variant.
* Lower-performing variants remain in play
Picking a random number from each distribution preserves the chance that Fusion will try a lower-performing variant, as long as there is still a chance that it is better.
* The more confident the measurements, the narrower the beta distributions become.
The more uncertain the measurements, the wider the distributions will be, and thus the more likely that Fusion will choose variants that appear to be performing more poorly.
Since Fusion adjusts the weights between variants each time the metrics jobs run, users can still get different results on subsequent visits. For example, if variant A is getting 80% of traffic, but after recalculating metrics it is only getting 50% of traffic, then some users who were previously assigned to variant A will be assigned to variant B. However, only the bare minimum of users will be reassigned to a new variant. Most users will see no changes. Once the experiment has been running for some time, the changes between the variants should be fairly small, so relatively few users should be affected.
### Manually specifying traffic weights
The formula for variant A is:
```
Proportion~A~ = (Traffic weight~A~)/(Sum of traffic weights for all variants)
```
For example:
| Variant traffic weights | Sum of traffic weights | Variant proportions |
| - | - | - |
| 1.0 1.0 | 2 | 0.5 0.5 |
| 1.0 1.0 2.0 | 4 | 0.25 0.25 0.5 |
| 0.5 1.0 1.0 2.5 | 5 | 0.1 0.2 0.2 0.5 |
## Calculate sample sizes
Fusion will calculate the required sample size to detect a statistically significant result based on the results at runtime. The "confidence level" metric has this minimum sample size factored in, so that confidence is always low for experiments that have not yet reached their required sample size.
However, if you would like to use different power or significance level in evaluating your experiment (Fusion will use 0.08 and 0.05), or if you would like to establish your own sample size based on a desired minimum detectable effect, you may do so.
## Choose an implementation approach
You can construct an experiment in either of two ways:
* **Experiment and query profile.** (recommended) For most cases, you will want to create additional query pipelines that return different search results. A query profile directs traffic through the query pipelines in accordance with the traffic weights of experiment variants.
* **Experiment stage in a query pipeline.** If you want to use parts of a single query pipeline in all experiment variants, you can add an Experiment stage to that pipeline (the pipeline that receives search queries). The app can direct queries to the endpoint of a query profile that references the pipeline (recommended) or to the endpoint of the query pipeline. If used, the query profile *does not* reference an experiment.
## Next step
You have planned the experiment. Next, you will set it up using either a query profile or an Experiment stage. This guide includes both options.
Perform the tasks needed to set up an experiment. This topic describes how to set up an experiment using a query profile that references the experiment.
## Before you begin
Before you set up an experiment, you must already have:
* **A search app.** The key aspect of the search app for experiments is that the search app identifies users in some way. A user ID might be associated with users persistently (this is best) or only for the current session. Searches and subsequent actions by anonymous users are not sent through an experiment.
- **A Fusion app.** The Fusion app provides the search functionality you want to provide. Below, you will modify the app to include an experiment.
- **Data for users to search.** The app should have data that users can search and search results that users can interact with. Typically, users will search a single dataset in the sense that different users are not given search results from different datasets. But in an experiment, different experiment variants *can* use data in different collections.
- **Results for users to interact with.** Experiment metrics depend on users interacting with search results, for example, clicking on them. A search app uses signals to report the interactions to Fusion.
- **A plan for the experiment.** This plan includes which control and variants to compare, the projected traffic, sample sizes, experiment duration, metrics, and decision criteria.
Basically, you need a working system in some environment on which you want to perform experiments, and a plan for experiment variants and for deciding which results are best.
## Create query pipelines
The primary approach for experiment variants is to have multiple query pipelines. You must create additional query pipelines as needed *before* creating the experiment.
You can also vary the collection that is searched and query parameters, which you do when creating the experiment. In fact, if that is all you are varying, you can define an experiment that uses a single query pipeline. (And you can skip this section.)
You will need the following pipelines:
* **Pipeline for the control variant.** The first variant in an experiment is the control. The A/B/n experiment metrics compare the control variant pairwise with the other variants. The control variant should probably use the default query pipeline. Here, we assume that is the case.
You could modify the configuration of the default query pipeline (which the control variant will use) before the experiment. However, doing so has two disadvantages. First, pairwise comparisons in the experiment will not be against the status quo. Second, Fusion AI will not have a baseline of historical signal data to compare against (that comparison lets an experiment reach higher confidence numbers more quickly).
* **Pipelines for other variants.** Other variants in an experiment can use other query pipelines, each with different stages and/or that are configured differently. These are "how you might do things differently."
**How to create query pipelines for non-control variants**
Create and configure the additional query pipelines you need for non-control variants. Repeat these steps for each pipeline.
1. Navigate to Querying > **Query Pipelines**.
2. Click **Add**.
3. Enter a **Pipeline ID** (arbitrary name) for the pipeline, and then click **Save**.
4\. Modify the pipeline as desired, for example, by adding, configuring, and reordering stages.
5\. Click **Save**.
## Create the query profile and the experiment
In the Fusion UI, you can use either of these equivalent approaches to set up an experiment:
* **[Create the experiment in the Query Profiles UI](#option-a-create-the-experiment-in-the-query-profiles-ui).** Choose an existing query profile or create a new one. Then create the experiment in the **Querying** > **Query Profiles** UI.
* **[Create the experiment in the Experiments UI](#option-b-create-the-experiment-in-the-experiments-ui).** Create an experiment in the **Analytics** > **Experiments** UI, and then reference the experiment in the **Querying** > **Query Profiles** UI.
### Option A: Create the experiment in the Query Profiles UI
1. **Choose or create a query profile.** Choose an existing query profile (for example, the default query profile) or create a new query profile for the experiment.
**How to choose an existing query profile:**
1. Navigate to Querying > **Query Profiles**.
2. In the left pane, click the query profile you want to use.
3. Verify that the pipeline, search handler, collection ID, and parameters are correct.
4. (Optional) Click **New params** and specify URL parameters to add to all queries for this query profile.
**How to create a new query profile:**
1. Navigate to Querying > **Query Profiles**.
2. Click **New**.
3. Enter a **Query Profile ID** (an arbitrary name for the query profile).
4. Choose a pipeline and a collection.
5. (Optional) Click **New params** and specify URL parameters to add to all queries for this query profile.
2. **Enable experimentation and specify experiment details in the query profile**:
1. Click the checkbox in front of **Enable experimentation**.
2. Specify the percent of traffic to include in the experiment.
3. Click the , and then click **Add Experiment**.
4. Enter an arbitrary **ID** (name) for the experiment.
5. Verify that the **unique ID parameter** is correct. It is the parameter that uniquely identifies each user. The default is `userId`. Correct the parameter if necessary, for example by specifying the session ID field instead.
6. Choose the base collection for signals. Signals resulting from requests that flow through the experiment are stored in the `_signals` collection associated with this collection.
7. (Optional) Enter a description for the experiment.
8. (Optional in Fusion 4.2+) To use a multi-armed bandit, select **Automatically Adjust Weights Between Variants**.
9. Add variants. Click **Add Variant** to add each non-control variant in your experiment.
10. For each variant:
11. Enter an arbitrary name. For the first variant, which is the control, Fusion uses the name `control`. You can change that name if you wish.
12. Click **Specify what varies** and specify what varies. Items you select are visible in the variant UI and have a green check mark in the dropdown menu. You can vary the query pipeline, query parameters (URL parameters to add to the query), and/or the collection.
13. (*For query parameters*) Click **New params**. In the dialog box, specify the **Parameter Name**, **Parameter Value**, and **Update Policy** for each parameter (append, default, remove, or replace).
3. Add metrics. For each metric:
4. Click **Add Metric** and select the type of metric.
5. Fill in information for the metric.
6. Click **Save** to save the experiment.
### Option B: Create the experiment in the Experiments UI
You can create the experiment first and reference it from a query profile.
1. **Create an experiment.** The experiment defines variants and metrics, as well as the user ID parameter and the base collection for signals:
2. Navigate to Analytics > **Experiments**.
3. Click **New**.
4. Enter an arbitrary **ID** (name) for the experiment.
5. Verify that the **unique ID parameter** is correct. It is the parameter that uniquely identifies each user. The default is `userId`. Correct the parameter if necessary, for example by specifying the session ID field instead.
6. Choose the base collection for signals. Signals resulting from requests that flow through the experiment are stored in the `_signals` collection associated with this collection.
7. (Optional) Enter a description for the experiment.
8. (Optional in Fusion 4.2+) To use a multi-armed bandit, select **Automatically Adjust Weights Between Variants**.
9. Add variants. Click **Add Variant** to add each non-control variant in your experiment.
10. For each variant:
11. Enter an arbitrary name. For the first variant, which is the control, Fusion uses the name `control`. You can change that name if you wish.
12. Click **Specify what varies** and specify what varies. Items you select are visible in the variant UI and have a green check mark in the dropdown menu. You can vary the query pipeline, query parameters (URL parameters to add to the query), and/or the collection.
13. (*For query parameters*) Click **New params**. In the dialog box, specify the **Parameter Name**, **Parameter Value**, and **Update Policy** for each parameter (append, default, remove, or replace).
14. Add metrics. For each metric:
15. Click **Add Metric** and select the type of metric.
16. Fill in information for the metric.
17. Click **Save** to save the experiment.
18. **Reference the experiment from the query profile.** Open the query profile you want to use for the experiment.
1. Navigate to Querying > **Query Profiles**.
2. Click the query profile you want to use for the experiment.
3. (Optional) If necessary, modify the query profile ID, default pipeline, and/or the search handler. These modifications are not related to experiments and are probably not required.
4. (Optional) Click **New params** and specify URL parameters to add to all queries for this query profile.
5. Click the checkbox in front of **Enable experimentation**.
6. Specify the percent of traffic to include in the experiment.
7. Click the , and then click the experiment name.
8\. Click **Save** to save the query profile.
## Next step
You have set up the experiment. Alternatively, you will set it up using an Experiment stage. If you chose to use this article for setting up your experiment, you can skip the next article.
Now that you have set up an experiment, you can run it. While an experiment is running:
* Fusion receives queries from the search app.
* For the queries that identify a user, Fusion routes the specified percentage of traffic through the experiment. Fusion apportions the queries from different users among the variants in accordance with traffic weights.
* Fusion records what users do after receiving search results, for example, what links they click.
We recommend that you only run one experiment at a time.
Running an experiment involves three steps:
1. **Activate the experiment.** Activating an experiment turns on the logic that sends queries through the experiment.
2. **Users submit queries and interact with results.** Queries and signals about interactions with results come from a search app. In this topic, we explain how to verify that data is flowing through the experiment correctly.
3. **Deactivate the experiment.** When enough data have been collected for metrics to be significant, you can deactivate the experiment. To run the experiment again, just activate it again. With each activation, you get new metrics.
Metrics will calculate periodically while an experiment is running. You can also generate metrics for a running experiment manually. Just run the metrics job(s) for the metrics you want to generate.
## Activate an experiment
You can activate an experiment in either of these ways. The outcome is identical.
**From the experiment**
1. With the app open, navigate to Analytics > **Experiments**.
2. In the left pane, click the experiment you want to activate. In the upper right, click **Activate**.
**From the query profile**
1. With the app open, navigate to Querying > **Experiments**. In the left pane, click the experiment you want to activate. In the upper right, click **Activate Experiment**.
If you stop Fusion while an experiment is running, then Fusion restarts the experiment automatically the next time you start Fusion.
## Verify that data is coming into the experiment
Signals resulting from requests that flow through the experiment are stored in the `_signals` collection associated with the primary collection.
You can use the Query Workbench or App Insights (if available) to examine this collection to verify that requests are being distributed among your experiment’s query pipelines.
Do not modify a running experiment. If you need to make a change, stop the experiment, make the modifications, and then start a new experiment that uses the modified object.
## Deactivate an experiment
You can deactivate an experiment in either of these ways. In order to stop an experiment from dividing traffic, you must remove it from the query profile.
**From the experiment**
1. With the app open, navigate to Analytics > **Experiments**.
2. In the left pane, click the experiment you want to deactivate. In the upper right, click **Deactivate**.
**From the query profile**
1. With the app open, navigate to Querying > **Experiments**. In the left pane, click the experiment you want to deactivate. In the upper right, click **Deactivate Experiment**.
## Next step
You have run the experiment. Next, you will analyze the experimental results.
After you have run an experiment in Fusion, you can analyze the results. When you stop an experiment, Fusion runs jobs that calculate metrics for the data that were collected. All jobs associated with an experiment are prefixed with the name of the experiment, that is, `-`.
To populate information in experiment metrics such as the Query Relevance metric, you *must* run both the `-groundTruth-` job and the `-rankingMetrics-` job.
You can also have Fusion generate metrics while an experiment is still running, by running metrics jobs by hand.
## Default schedules for metrics jobs
When you activate an experiment, Fusion AI schedules metrics jobs for the experiment. These are the default schedules for metrics jobs:
Ground Truth (used for the Query Relevance metric):
* **First run.** *Not scheduled*. The first time, you must run the Ground Truth job by hand.
* **Subsequent runs.** Every month until the experiment is stopped (by default; you can specify a different schedule)
All other metrics jobs:
* **First run.** 20 minutes after the experiment starts
* **Subsequent runs.** Every 24 hours until the experiment is stopped (by default; you can specify a different schedule)
* **Last run.** Immediately after the experiment is stopped
## Modify metrics jobs schedules (optional)
You can modify the default schedule as follows:
**How to modify the schedules of periodic metrics jobs**
1. Navigate to the experiment: Analytics > **Experiments**.
2. Next to each metric, find the **Processing Schedule** link. This link is active even if the experiment is running.
3. Edit the schedule as desired.
4. Click **Save**.
Periodic runs of metrics jobs are intended to give you up-to-date metrics. The metrics are always calculated from the beginning of the experiment.
Even with periodically updated metrics, we recommend that you let an experiment run its course before drawing conclusions and taking action.
## Check the last time metrics jobs ran
When you view experiment metrics and statistics, that information reflects the experiment’s state the last time the metrics jobs ran. When you stop an experiment, it is especially important that you verify that the end-of-experiment metrics jobs have run.
**How to check the last time metrics jobs ran**
1. Navigate to **Collections** > **Jobs**.
2. In the **Filter** field, enter the experiment name.
This displays only the experiment jobs.
3. Examine the **Last run** value below each job name.
## Consider the metrics produced (Fusion 4.x)
After metrics jobs run, you can view the metrics that they have produced in App Insights. For more information about the metrics, [read this topic](/docs/4/fusion-server/reference/system-metrics).
## Statistical significance
[Statistical significance](https://en.wikipedia.org/wiki/Statistical_significance) calculations inform you whether differences among experiment variants are likely to result from random chance, as opposed to real causal effects.
Fusion AI provides two measures of statistical significance:
* **Confidence index.** The [confidence index](#confidence-index) expresses the confidence that the experiment results are statistically significant. It takes into account the current sample size of the experiment, the required sample size to accurately establish statistical significance, as well as the calculated p-value.
* **Percent chance of beating.** The [percent chance of beating](#percent-chance-of-beating) uses a [Bayesian](https://en.wikipedia.org/wiki/Bayesian_statistics) algorithm to calculate the percent chance that another variant performs better than the control.
### Confidence index
The confidence index expresses the confidence that the experiment results are statistically significant. It gives you a gauge for whether the differences between variants are due to a causal effect (as opposed to random chance). The confidence index combines two concepts: the minimum sample size, and the p-value. If the number of samples is lower than the minimum sample size, then the confidence index is based entirely on the percentage of sample size. If the number of samples is above the minimum sample size, then the confidence index directly related to the p-value generated using Welch’s t-test [Welch’s t-test](https://en.wikipedia.org/wiki/Welch%27s_t-test), which is a variation of the [Student’s t-test](https://en.wikipedia.org/wiki/Student%27s_t-test). Welch’s t-test is better than the Student’s t-test when samples have unequal variances and/or sample sizes.
The test is a pairwise test, with each comparison being two-tailed (there is no *a priori* assumption that the difference will be in a specific direction). Fusion AI compares each variant against the first variant (the control), and generates a p-value for the comparison. The **confidence index** score is based on the *lowest* p-value amongst the variants.
The confidence index is this, rounded to the nearest whole number:
```
CI = 100 * (1-p)
```
You can recover two digits of the p-value from the confidence index as follows:
```
p = 1 - CI/100
```
### Percent chance of beating
The percent chance of beating uses a [Bayesian](https://en.wikipedia.org/wiki/Bayesian_statistics) algorithm to calculate the percent chance that another variant than the control does better than the control.
When calculating the percent chance of beating, Fusion AI uses up to 30 days of historical signal data to establish a baseline to compare against. The baseline is useful but not required. If the historical data is available, an experiment can reach higher confidence numbers more quickly.
Fusion AI calculates historical metrics one time and stores them, so subsequent runs of the metrics calculation jobs will not need to recalculate them.
Percent chance of beating is only accessible through the Fusion AI API, not through App Insights. Use the metrics endpoint `_hostname_:/api/experiments/_ experiment-name_/metrics`. In Fusion 5.x, the API port is `6764`. In prior versions, the API port is `8764`.
### Best practices
Note the following best practices regarding statistical significance:
* **If you peek, do not act.** P-values only reach significant levels when there is enough data. This leads to the problem of *peeking* (when people look at experiment results too early and make incorrect decisions). Wait until an experiment is over before making decisions based on the experiment. The confidence index is intended to encourage this practice.
* **Do not modify running experiments.** To modify an experiment, you have to stop it, and data collection for the experiment stops. This is nice and clean and as it should be. You could, however, modify some object that the experiment uses (for example, you could modify a query pipeline) *while the experiment is running*. But this makes it unclear what you have been testing. We recommend against this practice. Instead, stop the first experiment, make the modifications, and then activate (start) an experiment that uses the modified object.
The course for **Experiments** focuses on understanding and running experiments in Fusion.
## Run an Experiment Tutorial
The **Run an Experiment** tutorial takes you through the steps needed to run an A/B experiment to compare metrics such as click-through rate (CTR) and query relevance for two differently configured query pipelines. You plan the experiment, create a Fusion app, index a datasource, and create a query profile that includes the configuration data needed for experiments. In Fusion, you start and stop the experiment. A search app uses the query profile for Fusion queries. Different users get different search results, but they are blissfully unaware that an experiment is going on.
Now that you have set up an experiment, you can run it. While an experiment is running:
* Fusion receives queries from the search app.
* For the queries that identify a user, Fusion routes the specified percentage of traffic through the experiment. Fusion apportions the queries from different users among the variants in accordance with traffic weights.
* Fusion records what users do after receiving search results, for example, what links they click.
We recommend that you only run one experiment at a time.
Running an experiment involves three steps:
1. **Activate the experiment.** Activating an experiment turns on the logic that sends queries through the experiment.
2. **Users submit queries and interact with results.** Queries and signals about interactions with results come from a search app. In this topic, we explain how to verify that data is flowing through the experiment correctly.
3. **Deactivate the experiment.** When enough data have been collected for metrics to be significant, you can deactivate the experiment. To run the experiment again, just activate it again. With each activation, you get new metrics.
Metrics will calculate periodically while an experiment is running. You can also generate metrics for a running experiment manually. Just run the metrics job(s) for the metrics you want to generate.
## Activate an experiment
You can activate an experiment in either of these ways. The outcome is identical.
**From the experiment**
1. With the app open, navigate to Analytics > **Experiments**.
2. In the left pane, click the experiment you want to activate. In the upper right, click **Activate**.
**From the query profile**
1. With the app open, navigate to Querying > **Experiments**. In the left pane, click the experiment you want to activate. In the upper right, click **Activate Experiment**.
If you stop Fusion while an experiment is running, then Fusion restarts the experiment automatically the next time you start Fusion.
## Verify that data is coming into the experiment
Signals resulting from requests that flow through the experiment are stored in the `_signals` collection associated with the primary collection.
You can use the Query Workbench or App Insights (if available) to examine this collection to verify that requests are being distributed among your experiment’s query pipelines.
Do not modify a running experiment. If you need to make a change, stop the experiment, make the modifications, and then start a new experiment that uses the modified object.
## Deactivate an experiment
You can deactivate an experiment in either of these ways. In order to stop an experiment from dividing traffic, you must remove it from the query profile.
**From the experiment**
1. With the app open, navigate to Analytics > **Experiments**.
2. In the left pane, click the experiment you want to deactivate. In the upper right, click **Deactivate**.
**From the query profile**
1. With the app open, navigate to Querying > **Experiments**. In the left pane, click the experiment you want to deactivate. In the upper right, click **Deactivate Experiment**.
## Next step
You have run the experiment. Next, you will analyze the experimental results.
## A/B/n experiments
Fusion’s experiments feature set implements [A/B/n experiments](https://en.wikipedia.org/wiki/A/B_testing), also called A/B experiments or A/B tests, where A and B are *experiment groups* with one or more *variants*.
Fusion’s implementation of an A/B experiment uses consistent hashing on a unique ID field (typically `userId`), concatenated with the experiment’s name, to assign each request to one of the experiment groups. Any future requests with that hash are assigned to the same group, guaranteeing user "stickiness".
If you prefer "stickiness" only at the session level, you can send a session ID instead of a user ID.
If you send no ID at all, the request is not assigned to a variant since there is no way to consistently assign it to the same one. In that case, the request uses the "default" configuration of the query profile or experiment stage.
## Example
The following experiment is an example of an A/B/n experiment with three variants:
* **Variant 1 (control).** Use the default query pipeline with no modifications. Each experiment should have a "control" variant as the first variant; the other variants will be compared against this one.
* **Variant 2 (content-based filtering with a Solr MoreLikeThis stage).** Content-based filtering uses data about a user’s search results, browsing history, and/or purchase history to determine which content to serve to the user. The filtering is non-collaborative.
* **Variant 3 (collaborative filtering with a Recommend Items for User stage).** Collaborative filtering takes advantage of knowledge about the behavior of many individuals. It makes serendipitous discovery possible—a user is presented with items that other users deem relevant, for example, socks when buying shoes.
## High-level workflow
In an experiment:
1. A Fusion administrator defines the experiment. An experiment has *variants* with differences in query pipelines, query pipeline stages, collections, and/or query parameters.
2. The Fusion administrator assigns the experiment to a query profile.
3. A user searches using that query profile.
4. If the experiment is running, Fusion assigns the user to one of the experiment variants, in accordance with **Plan an Experiment**. Assignment to a variant is persistent. The next time the user searches, Fusion assigns the same variant.
5. Different experiment variants return different search results to users.
6. Users interact with the search results, for example, viewing them, possibly clicking on specific results, possibly buying things, and so forth.
7. Based on the interactions, the search app backend sends signals to the signals endpoint of the query profile for the experiment.
8. Using signal data, an automatically-created Spark job periodically computes metrics for each experiment variant and writes the metrics to the `job_reports` collection.
9. In the Fusion UI, an administrator can view reports about the experiment.
10. Once the results of the experiment are conclusive, the Fusion administrator can stop the experiment and change the query profile to use the winning variant, or start a new experiment.
From a planning standpoint, an experiment has these parts:
* **A baseline control.** One of the experiment variants will be the control. This is "how we are doing things today." If you are experimenting from the start, choose the simplest variant as the control.
* **Experiment variants.** Experiment variants other than the control are attempts to improve the user’s extended search experience. Which relevancy strategy works best for your search app and your users?
* **Metrics.** This is how you know whether the search variants produce differences in user interactions, and whether the differences are statistically significant.
In the remainder of this topic, you will make decisions about these broad areas, as well as about experiment details.
## Plan what you want to vary
Identify different relevancy strategies, where each represents a hypothesis about which user experience will drive more click-throughs, purchases, and so on. Use the [Query Workbench](/docs/5/fusion/getting-data-out/query-basics/query-workbench) to explore how to produce different search results and recommendations using different query pipelines, and evaluate which ones might engage your users most effectively.
## Plan what you want to measure
Metrics compare the control against other variants pairwise. For example, if the variants are `experiment`, `B`, `C`, and `D`, and you choose `experiment` as the control, then the comparisons for which metrics are generated will be `experiment/B`, `experiment/C`, and `experiment/D`.
You can [learn more about metrics](/docs/5/fusion/getting-data-out/data-analytics/experiments/experiment-metrics).
## Design the experiment
When designing an experiment, you must make these decisions:
* How users are identified
* Percentage of total traffic to send through the experiment
* Number of variants and how they differ
* Metrics to generate
In many cases identifying users is straightforward, using an existing user ID or session ID if the application has one. In other cases, you may need to generate an identifier of some sort to send in on queries. It is important to send in some kind of identifier with each query so that the experiment can route the query to a variant, and to send that same identifier with any subsequent signals that resulted from that query. Queries without a user ID will not be routed through the experiment.
The percentage of total traffic to send through the experiment is the one variable that can change over the course of the experiment. It is often a good practice to start out sending only a small percentage of search traffic through a new experiment, in order to verify that each of the variants are functioning properly. Then, once you have established that the behavior is as intended, you can increase the percentage of traffic through the experiment to the desired level.
With modest usage and for a possibly small effect, or when testing multiple variants at the same time, you might want to send 100% of users through the experiment and let it run longer. For high usage and an effect that is expected to be larger, and with only two variants, you might not need to send all users through the experiment and the experiment will not take as long.
## Choose traffic weights
Fusion AI uses *traffic weights* to apportion search traffic among the variants. This allows you to send a different percentage of traffic through each variant if desired.
### Automatic traffic weights (multi-armed bandit)
Automatic traffic weights are only available in Fusion 4.2+.
The **Automatically Adjust Weights Between Variants** configuration option enables multi-armed bandits and eliminates the need to specify a traffic weight for each variant.
In multi-arm bandit mode, [metrics](/docs/4/fusion-ai/reference/metrics) jobs are created automatically once the experiment starts. In Fusion 4.2.x, schedules are also automatically created. In Fusion 5.x, however, the jobs must be scheduled manually. It is recommended that you schedule the metrics jobs to run on an hourly basis. The weights between variants only change after the metrics jobs run.
Fusion’s multi-arm bandit implementation uses a variation of [Thompson Sampling](https://en.wikipedia.org/wiki/Thompson_sampling) (sometimes called Bayesian Bandits). This algorithm uses the current count of successes versus failures to build a beta distribution that represents the level of confidence in the primary metric value for each variant. It then samples a random number from each variant’s distribution, and picks the highest number.
This type of implementation has three effects:
* It weights better-performing variants higher.
Since the beta distribution of each variant is centered around the primary metric value for that variant, a random number selected from a higher-performing variant is likely to be higher than a random number picked from a lower-performing variant.
* Lower-performing variants remain in play
Picking a random number from each distribution preserves the chance that Fusion will try a lower-performing variant, as long as there is still a chance that it is better.
* The more confident the measurements, the narrower the beta distributions become.
The more uncertain the measurements, the wider the distributions will be, and thus the more likely that Fusion will choose variants that appear to be performing more poorly.
Since Fusion adjusts the weights between variants each time the metrics jobs run, users can still get different results on subsequent visits. For example, if variant A is getting 80% of traffic, but after recalculating metrics it is only getting 50% of traffic, then some users who were previously assigned to variant A will be assigned to variant B. However, only the bare minimum of users will be reassigned to a new variant. Most users will see no changes. Once the experiment has been running for some time, the changes between the variants should be fairly small, so relatively few users should be affected.
### Manually specifying traffic weights
The formula for variant A is:
```
Proportion~A~ = (Traffic weight~A~)/(Sum of traffic weights for all variants)
```
For example:
| Variant traffic weights | Sum of traffic weights | Variant proportions |
| - | - | - |
| 1.0 1.0 | 2 | 0.5 0.5 |
| 1.0 1.0 2.0 | 4 | 0.25 0.25 0.5 |
| 0.5 1.0 1.0 2.5 | 5 | 0.1 0.2 0.2 0.5 |
## Calculate sample sizes
Fusion will calculate the required sample size to detect a statistically significant result based on the results at runtime. The "confidence level" metric has this minimum sample size factored in, so that confidence is always low for experiments that have not yet reached their required sample size.
However, if you would like to use different power or significance level in evaluating your experiment (Fusion will use 0.08 and 0.05), or if you would like to establish your own sample size based on a desired minimum detectable effect, you may do so.
## Choose an implementation approach
You can construct an experiment in either of two ways:
* **Experiment and query profile.** (recommended) For most cases, you will want to create additional query pipelines that return different search results. A query profile directs traffic through the query pipelines in accordance with the traffic weights of experiment variants.
* **Experiment stage in a query pipeline.** If you want to use parts of a single query pipeline in all experiment variants, you can add an Experiment stage to that pipeline (the pipeline that receives search queries). The app can direct queries to the endpoint of a query profile that references the pipeline (recommended) or to the endpoint of the query pipeline. If used, the query profile *does not* reference an experiment.
## Next step
You have planned the experiment. Next, you will set it up using either a query profile or an Experiment stage. This guide includes both options.
## Information flow
This diagram illustrates information flow through an experiment. Numbers correspond to explanations below the diagram.
1. A user searches in a search app. For example, the user might search for `shirt`.
2. The search app backend appends a `userId` or other unique ID that identifies the user, for example, `userId=123`, to the query and sends the query to the query profile endpoint for the experiment.
3. Using information in the query profile and the value of the unique ID, Fusion routes the query through one of the experiment’s variants. In this example, Fusion routes the query through query pipeline 1.
4. A query pipeline adds a `x-fusion-query-id` to the response header, for example, `x-fusion-query-id=abc`.
5. Based on the query, Fusion obtains a search result from the index, which is stored in the primary collection. Fusion sends the search result back to the search app.
6. Fusion sends a response signal to the signals collection.
7. A different user might be routed through the other experiment variant shown here, and through query pipeline 2. This query pipeline has an enabled Boost with Signals stage, unlike query pipeline 1.
8. The search user interacts with the search results, viewing them, possibly clicking on specific results, possibly buying things, and so forth. For example, the user might click the document with `docId=757`.
9. Based on the interactions, the search app backend sends click signals to the signals endpoint for the query profile. Signals include the same query ID so Fusion can associate the signals with the experiment. Specifically, the click signal *must* include a field named `fusion_query_id` in the `params` object of the raw click signal whose value was returned in the response object in a header named `x-fusion-query-id`. If you are tracking queries and responses with App Studio, the `fusion_query_id` parameter will be passed with the click signal as long as you specify the appropriate response attribute in your `track:clicks` tag.
10. Using information in the query profile, Fusion routes the signals to the `_signals_ingest` pipeline.
11. The `_signals_ingest` pipeline stores signals in the `_signals` collection. Signals include the collection ID of the primary collection and experiment tracking information.
## Metrics generation
This diagram illustrates metrics generation:
1. A Fusion administrator can configure which metrics are relevant for a given experiment and the frequency with which experiment metrics are generated. They can also generate metrics on demand.
2. Using signal data, an automatically-created Spark job periodically runs in the background. It obtains signal data from the `COLLECTION_NAME_signals` collection, computes metrics for each experiment variant, and writes the metrics to the collection used for aggregated signals (`_signals_aggr`).
3. In the Fusion UI, a Fusion administrator can view experiment metrics.
4. These calculated metrics are used to display reports about the experiment.
## Learn more
When making changes to a query pipeline or query parameters that will affect users' search experience, it is often a good idea to run an experiment in order to verify that the results are what you intended. Fusion lets you create and run experiments that take care of dividing traffic between variants and calculating the results of each variant with respect to configurable objectives such as purchases, click-through rate, or search relevance.
There are two ways that a search application might interact with an experiment:
* Preferred: Using a query profile
* Alternative: Using an Experiment query pipeline stage
This guide takes you through the process of planning, setting up, running, and analyzing an experiment.
Here we describe how to set up an experiment that uses an Experiment query pipeline stage. Using an Experiment query pipeline stage is a bit more complicated.
This approach is an alternative to setting up an experiment and a query profile that references it. If you have used the query profile method, you can skip this article.
## How an Experiment stage works
An Experiment stage applies the experiment actions in-line in the query pipeline (wherever it is located in the pipeline), instead of performing the actions before passing queries to a query pipeline or pipelines.
An Experiment stage that apportions traffic among query pipelines is similar to a Call Pipeline stage, but the processing is *conditional*. Queries that lack a user ID parameter are not sent processed by the stage and are not sent to other pipelines (if that is what the stage does).
For the primary pipeline to process queries that do not include a user identifier, it *must* contain a Solr Query stage as the last stage. If the Experiment query stage references other pipelines, then there are two options:
* **Solr Query stage as the last stage in the variant pipelines.** The variant pipelines send queries to Solr. Control does not return to the primary pipeline. In the primary pipeline, the Experiment stage must be the second-to-last stage. The last stage must be the Solr Query stage.
* **No Solr Query stage in the variant pipelines.** The variant pipelines do not send queries to Solr. Control returns from the variant pipelines to the primary pipeline. In the primary pipeline, the Experiment stage can be in any position except the last.
## Before you begin
Before you set up an experiment, you must already have:
* **A search app.** The key aspect of the search app for experiments is that the search app identifies users in some way. A user ID might be associated with users persistently (this is best) or only for the current session. Searches and subsequent actions by anonymous users are not sent through an experiment.
* **A Fusion app.** The Fusion app provides the search functionality you want to provide. Below, you will modify the app to include an experiment.
* **Data for users to search.** The app should have data that users can search and search results that users can interact with. Typically, users will search a single dataset in the sense that different users are not given search results from different datasets. But in an experiment, different experiment variants *can* use data in different collections.
* **Results for users to interact with.** Experiment metrics depend on users interacting with search results, for example, clicking on them. A search app uses signals to report the interactions to Fusion.
* **A plan for the experiment.** This plan includes which control and variants to compare, the projected traffic, sample sizes, experiment duration, metrics, and decision criteria.
Basically, you need a working system in some environment on which you want to perform experiments, and a plan for experiment variants and for deciding which results are best.
## Create an experiment
Create an experiment. The experiment defines variants and metrics, as well as the user ID parameter and the base collection for signals:
1. Navigate to **Analytics** > **Experiments**.
2. Click **New**.
3. Enter an arbitrary **ID** (name) for the experiment.
4. Verify that the **unique ID parameter** is correct. It is the parameter that uniquely identifies each user. The default is `userId`. Correct the parameter if necessary, for example by specifying the session ID field instead.
5. Choose the base collection for signals. Signals resulting from requests that flow through the experiment are stored in the `_signals` collection associated with this collection.
6. (Optional) Enter a description for the experiment.
7. To use a multi-armed bandit, select **Automatically Adjust Weights Between Variants**.
8. Add variants. Click **Add Variant** to add each non-control variant in your experiment.
9. For each variant:
1. Enter an arbitrary name. For the first variant, which is the control, Fusion uses the name `control`. You can change that name if you wish.
2. Click **Specify what varies** and specify what varies. Items you select are visible in the variant UI and have a green check mark in the dropdown menu. You can vary the query pipeline, query parameters (URL parameters to add to the query), and/or the collection.
3. (*For query parameters*) Click **New params**. In the dialog box, specify the **Parameter Name**, **Parameter Value**, and **Update Policy** for each parameter (append, default, remove, or replace).
10. Add metrics. For each metric:
1. Click **Add Metric** and select the type of metric.
2. Fill in information for the metric.
11. Click **Save** to save the experiment.
## Set up an Experiment stage
If part or all of what you will vary in the experiment is encompassed by differences in query pipelines, create additional pipelines for experiment variants. You *cannot* use the default query pipeline (the pipeline to which you are adding the Experiment stage) as one of the variants. That pipeline will be a part of all variants. Fusion directs traffic that does not identify users through the default pipeline but not through the experiment.
**How to set up an experiment stage**
1. Navigate to Querying > **Query Pipelines**.
2. Click the name of the pipeline to which you want to add the Experiment stage.
3. Click **Add a new pipeline stage** and select **Experiment stage**.
4. (Optional) Specify a label for the stage.
5. (Optional) Specify a condition that must be satisfied for queries to pass through the experiment.
6. Under **Experiment ID**, choose the experiment.
7. (Optional) Specify the percent of traffic to include in the experiment.
8. Click **Save**.
9. Drag the stage to where you want it in the pipeline.
10. Click **Save**.
You control the experiment in **Analytics** > **Experiments**.
# Data Analytics
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/data-analytics/overview
[localhost link]: http://localhost:3000/docs/5/fusion/getting-data-out/data-analytics/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/data-analytics/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/3230
* [Experiments](/docs/5/fusion/getting-data-out/data-analytics/experiments/overview) provide A/B testing and [metrics](/docs/5/fusion/getting-data-out/data-analytics/experiments/experiment-metrics).
* You can run and schedule [analytics jobs](/docs/5/fusion/getting-data-out/data-analytics/app-insights/analytics) to keep your analytics up to date.
* [App Insights](/docs/5/fusion/getting-data-out/data-analytics/app-insights/overview) provides visualizations about your signals and experiments.
The course for **Query Analytics** focuses on how Fusion provides query analytics to detect and improve underperforming queries.
# Export Data from ZooKeeper
Source: https://doc.lucidworks.com/docs/5/fusion/getting-data-out/zookeeper
While the [ZooKeeper Import/Export API](/api-reference/zookeeper/get-zookeeper-data) lets you export information from Fusion’s ZooKeeper service, the Fusion distribution provides a script you can use to import or export ZooKeeper data through the ZKImportExport service.
The `zkImportExport.sh` script is located in the top-level Fusion `scripts` directory.
## Usage
The script takes the following command-line arguments:
```sh wrap theme={"dark"}
-c,--cmd Command, one of: 'export', 'import', 'update', 'delete'.
-e,--encode Type of encoding for znodes. Valid options:
'none', 'utf-8', 'base64', default is 'base64'.
Option 'none' will not return any data from the znodes.
-ep,--exclude Exclude znode paths, followed by list of paths.
Can only be used to exclude nodes one level below the root node.
-eph,--ephemeral Include ephemeral nodes while exporting znodes, boolean, default false.
-f,--filename Name of file containing import/export data.
-h,--help Display help page.
-ip,--include Include znode paths to include, followed by a list of paths.
Can only be used to include nodes one level below the root node.
-nr,--non-recursive Do not perform recursive operations on znodes.
-o,--overwrite Overwrite data for existing znodes. Valid only with 'update' command.
-p,--path Path from ZooKeeper root node, e.g. '/lucid/query-pipelines'.
-r,--recursive Perform recursive operations on znodes.
-z,--zkhost ZooKeeper Connect string, required.
```
Required arguments are:
* -c, --cmd : operation to perform.
* -z, --zkhost : the ZooKeeper Connect string.
## Examples
*Export all data from a local single-node ZooKeeper service, save data to a file:*
```sh wrap theme={"dark"}
zkImportExport.sh -zkhost FUSION_HOST:9983 -cmd export -path / -filename znode_dump.json
```
*Export all Fusion configurations from a local single-node ZooKeeper service, save data to a file:*
```sh wrap theme={"dark"}
zkImportExport.sh -zkhost FUSION_HOST:9983 -cmd export -path /lucid -filename znode_lucid_dump.json
```
*Export Fusion user databases, groups, roles, and realms configurations from a local single-node ZooKeeper service, save data to a file:*
```sh wrap theme={"dark"}
zkImportExport.sh -zkhost FUSION_HOST:9983 -cmd export -path /lucid-apollo-admin -filename znode_lucid_admin_dump.json
```
*Initial import of saved Fusion configuration into a new ZooKeeper:*
```sh wrap theme={"dark"}
zkImportExport.sh -zkhost FUSION_HOST:9983 -cmd import -filename znode_lucid_dump.json
```
Note that the above command will fail if there is conflict between existing znode structures
or contents between the ZooKeeper service and the dump file.
*Update information for Fusion’s ZooKeeper service:*
```sh wrap theme={"dark"}
zkImportExport.sh -zkhost FUSION_HOST:9983 -cmd update -filename znode_lucid_dump.json
```
*Remove a znode from Fusion’s ZooKeeper service:*
```sh wrap theme={"dark"}
zkImportExport.sh -zkhost FUSION_HOST:9983 -cmd delete -path /lucid/test
```
# Commit management
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/commits-management
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/ozi0ub
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/commits-management
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/commits-management
ConfigSync uses a GitHub repository to store the configuration files.
It pulls the latest changes from the repository every 30 seconds by default, and applies them to the Fusion cluster.
You can configure the polling interval; see the [ConfigSync configuration reference](/docs/5/fusion/operations/config-sync/settings).
At the same time, it listens to the changes in the Fusion cluster and pushes them to the GitHub repository.
Fusion cluster changes have a higher priority than GitHub repository changes in case of conflicts.
## Latest synced commit
The latest synced commit is the commit that was last applied by ConfigSync.
ConfigSync tracks it as `latest-synced-commit` whose value is the Git commit hash.
It uses this value to determine which commits to apply.
ConfigSync applies all commits that are newer than `latest-synced-commit` and sets `latest-synced-commit` to the latest commit.
For example, imagine the following Git commit history:
```mermaid theme={"dark"}
%%{init: { 'gitGraph': {'showBranches': true} } }%%
gitGraph
accTitle: ConfigSync commit history: initial state with latest synced commit
accDescr: A git graph showing commits 0000000, 1111111, and latest-synced-commit highlighted on the project/customer/environment branch
commit id: " "
branch project/customer/environment
checkout project/customer/environment
commit id: "0000000"
commit id: "1111111"
commit id: "latest-synced-commit" type: HIGHLIGHT
```
When ConfigSync performs a `git pull` and finds a new commit, it applies it and sets `latest-synced-commit` to the latest commit.
```mermaid theme={"dark"}
%%{init: { 'gitGraph': {'showBranches': true} } }%%
gitGraph
accTitle: ConfigSync commit history: new commit available after git pull
accDescr: A git graph showing commits up to latest-synced-commit with a new latest-git-commit available to be applied
commit id: " "
branch project/customer/environment
checkout project/customer/environment
commit id: "0000000"
commit id: "1111111"
commit id: "latest-synced-commit" type: HIGHLIGHT
commit id: "latest-git-commit"
```
ConfigSync calculates the difference between `latest-synced-commit` and `latest-git-commit` and applies all commits within that range.
Then it sets `latest-synced-commit` to `latest-git-commit`.
```mermaid theme={"dark"}
%%{init: { 'gitGraph': {'showBranches': true} } }%%
gitGraph
accTitle: ConfigSync commit history: latest synced commit advanced after sync
accDescr: A git graph showing commits 0000000, 1111111, 2222222, and latest-synced-commit after ConfigSync applied the new commit
commit id: " "
branch project/customer/environment
checkout project/customer/environment
commit id: "0000000"
commit id: "1111111"
commit id: "2222222"
commit id: "latest-synced-commit" type: HIGHLIGHT
```
If needed, you can set `latest-synced-commit` manually:
```bash theme={"dark"}
PUT /api/properties/latest-synced-commit
Content-Type: text/plain
1111111
```
This sets `latest-synced-commit` pointer to `1111111`.
```mermaid theme={"dark"}
%%{init: { 'gitGraph': {'showBranches': true} } }%%
gitGraph
accTitle: ConfigSync commit history: latest synced commit manually reset
accDescr: A git graph showing latest-synced-commit manually set to an earlier position, with newer commits 2222222 and latest-git-commit ahead of it
commit id: " "
branch project/customer/environment
checkout project/customer/environment
commit id: "0000000"
commit id: "latest-synced-commit" type: HIGHLIGHT
commit id: "2222222"
commit id: "latest-git-commit"
```
After that, ConfigSync reapplies all commits that are newer than `latest-synced-commit` and set `latest-synced-commit` to the latest commit.
## Locked commit
ConfigSync can lock the configuration to a specific commit.
This means that ConfigSync :
* Does not apply any commits that are newer than the locked commit
* Does not update changes from the Fusion cluster to the GitHub repository
```mermaid theme={"dark"}
%%{init: { 'gitGraph': {'showBranches': true} } }%%
gitGraph
accTitle: ConfigSync commit history: initial state with latest synced commit
accDescr: A git graph showing commits 0000000, 1111111, and latest-synced-commit highlighted on the project/customer/environment branch
commit id: " "
branch project/customer/environment
checkout project/customer/environment
commit id: "0000000"
commit id: "1111111"
commit id: "latest-synced-commit" type: HIGHLIGHT
```
You can set `locked-commit` using the API:
```bash theme={"dark"}
PUT /api/properties/locked-commit
Content-Type: text/plain
2222222
```
If `locked-commit` precedes `latest-synced-commit`, ConfigSync reverts all changes to the `locked-commit` and sets `latest-synced-commit` to `locked-commit`.
```mermaid theme={"dark"}
%%{init: { 'gitGraph': {'showBranches': true} } }%%
gitGraph
accTitle: ConfigSync commit history: locked commit set
accDescr: A git graph showing the locked-commit branch with a locked-commit commit highlighted, branching from the project/customer/environment branch
commit id: " "
branch project/customer/environment
checkout project/customer/environment
commit id: "0000000"
commit id: "1111111"
branch locked-commit
checkout locked-commit
commit id: "locked-commit" type: HIGHLIGHT
```
All commits that are newer than `locked-commit` are ignored, as well as changes from the Fusion cluster.
```mermaid theme={"dark"}
%%{init: { 'gitGraph': {'showBranches': true} } }%%
gitGraph
accTitle: ConfigSync commit history: changes blocked by locked commit
accDescr: A git graph showing git-changes accumulating on the locked-commit branch and fusion-changes on the project branch while the locked commit is active
commit id: " "
branch project/customer/environment
checkout project/customer/environment
commit id: "0000000"
commit id: "1111111"
branch locked-commit
checkout locked-commit
commit id: "locked-commit" type: HIGHLIGHT
commit id: "git-changes"
checkout project/customer/environment
commit id: "fusion-changes"
```
After we remove the `locked-commit`, ConfigSync commits all Fusion cluster changes and pulls the latest changes from the GitHub repository:
```bash theme={"dark"}
DELETE /api/properties/locked-commit
```
That triggers a merge between `git-changes` and `fusion-changes` where `fusion-changes` has a higher priority in case of conflicts.
```mermaid theme={"dark"}
%%{init: { 'gitGraph': {'showBranches': true} } }%%
gitGraph
accTitle: ConfigSync commit history: after removing locked commit
accDescr: A git graph showing the merge of git-changes from the locked-commit branch into the project branch, with latest-synced-commit set to the merge commit
commit id: " "
branch project/customer/environment
checkout project/customer/environment
commit id: "0000000"
commit id: "1111111"
branch locked-commit
checkout locked-commit
commit id: "locked-commit" type: HIGHLIGHT
commit id: "git-changes"
checkout project/customer/environment
commit id: "fusion-changes"
merge locked-commit id: "latest-synced-commit" type: HIGHLIGHT
```
# ConfigSync deployment
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/deployment
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/5gj79e
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/deployment
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/deployment
ConfigSync is disabled by default because it requires preparation of a GitHub repository.
This topic provides an overview of the deployment process.
## 1. Prepare a GitHub repository
The `config-sync` service uses a GitHub repository to store the configuration files.
See **Set up a GitHub repo for ConfigSync** to learn how to prepare a repository for use with ConfigSync.
[ConfigSync](/docs/5/fusion/operations/config-sync/overview) uses a GitHub repository to store the configuration files.
The repository should be created independently and shared with the ConfigSync service via the Helm chart values.
This document describes the process of setting up a GitHub repository for Fusion config files.
## Create a new repository
Create a new repository in the github organization where the ConfigSync should store the config files.
## Create a deploy key
Create a deploy key for the new repository.
The deploy key should have write access to the repository.
See the [GitHub documentation](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/managing-deploy-keys#set-up-deploy-keys) for instructions.
## Add the deploy key and repository information to the ConfigSync values
### Option 1: Add the deploy key in the Helm chart
```yaml theme={"dark"}
fusion-config-sync:
# other values
sshKey:
cfg:
github:
repo: # use the ssh url e.g. git@github.com:OWNER/REPO.git
branch: # you can leave it as is
mainBranch: # most probably it's `main`
```
If outbound port 22 is blocked, you can use SSH over HTTPS with a URL like `ssh://git@ssh.github.com:443/OWNER/REPO.git`.
See the [GitHub documentation](https://docs.github.com/en/authentication/troubleshooting-ssh/using-ssh-over-the-https-port) for details.
You still need a deploy key for this option.
If port 443 is also blocked, you can use HTTPS with a [Personal Access Token (PAT)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens).
To encode the private key to base64 you can use the following command:
```bash theme={"dark"}
cat ~/.ssh/id_rsa | base64
```
### Option 2: Add the deploy key as an external secret
Alternatively, you can store the deploy key as an external secret and reference it in the ConfigSync values.
#### Create a secret with the deploy key:
```yaml theme={"dark"}
apiVersion: v1
kind: Secret
metadata:
name: config-sync-ssh
type: Opaque
data:
config: SG9zdCAqCiAgU3RyaWN0SG9zdEtleUNoZWNraW5nIG5v
id_rsa:
```
To encode the private key to base64 you can use the following command:
```bash theme={"dark"}
cat ~/.ssh/id_rsa | base64
```
#### Update the ConfigSync values to reference the secret with the deploy key:
```yaml theme={"dark"}
fusion-config-sync:
# other values
sshKey: null
additionalVolumeMounts:
- name: config-sync-ssh-key
mountPath: /root/.ssh
readOnly: true
additionalVolumes:
- name: config-sync-ssh-key
secret:
secretName: config-sync-ssh
defaultMode: 256
```
## Additional configuration
`github.*` settings control the remote GitHub repo, branch, and path within the repo.
```yaml theme={"dark"}
github connection properties, common for pub/sub modes
cfg:
github:
repo: https://github.com/lucidworks/cloud-config-sync.git # git repo, mandatory
branch: dev # git branch, mandatory
mainBranch: main # main branch, mandatory
dir: "/tmp/config-sync-repo" # repo dir, default "/tmp/config-sync-repo"
path: / # configs path in the repo dir, default "/"
username: "fusion-config-sync" # git user name, default "fusion-config-sync"
email: "fusion-config-sync@lucidworks.com" # git user email, mandatory, default "fusion-config-sync@lucidworks.com"
```
## Update the ConfigSync deployment
Update the ConfigSync deployment with the new values.
[Repo organization](/docs/5/fusion/operations/config-sync/repo-org) explains what you’ll find in the repository when ConfigSync is enabled.
## 2. Enable ConfigSync
**To enable ConfigSync, you need to update your Helm chart configuration.**
See **Enable ConfigSync** for instructions.
To enable [ConfigSync](/docs/5/fusion/operations/config-sync/overview), add the `config-sync` values to your Helm chart configuration, similar to the example below:
```yaml theme={"dark"}
fusion-config-sync:
enabled: true
# other config-sync specific values
sshKey:
cfg:
github:
repo: # use the ssh url e.g. git@github.com:my-fusion/config-sync-repo.git
branch: # you can leave it as is
mainBranch: # most probably it's `main`
```
If outbound port 22 is blocked, you can use SSH over HTTPS with a URL like `ssh://git@ssh.github.com:443/OWNER/REPO.git`.
See the [GitHub documentation](https://docs.github.com/en/authentication/troubleshooting-ssh/using-ssh-over-the-https-port) for details.
You still need a deploy key for this option.
If port 443 is also blocked, you can use HTTPS with a [Personal Access Token (PAT)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens).
Next, you can configure the ConfigSync mode to select Publisher mode, Subscriber mode, or Pub/Sub mode.
For additional options, see the [ConfigSync configuration reference](/docs/5/fusion/operations/config-sync/settings).
## 3. Configure the synchronization mode
You can select [Publisher](/docs/5/fusion/operations/config-sync/modes/publisher-mode), [Subscriber](/docs/5/fusion/operations/config-sync/modes/subscriber-mode), or [PubSub](/docs/5/fusion/operations/config-sync/modes/pubsub-mode) mode.
See **Configure the ConfigSync mode** for instructions.
[ConfigSync](/docs/5/fusion/operations/config-sync/overview) can operate in different modes:
* [Publisher mode](/docs/5/fusion/operations/config-sync/modes/publisher-mode): Monitors Fusion for changes and pushes them to a GitHub repository.
* [Subscriber mode](/docs/5/fusion/operations/config-sync/modes/subscriber-mode): Polls a GitHub repository for changes and applies them to Fusion.
* [Pub/Sub mode](/docs/5/fusion/operations/config-sync/modes/pubsub-mode): Combines the features of both Publisher and Subscriber modes.
You can configure the synchronization mode by adding the appropriate profile to the `sync` configuration in the `fusion-config-sync` section of the Helm chart configuration, as shown below.
* Publisher mode:
```yaml theme={"dark"}
fusion-config-sync:
springProfilesOverride: "kubernetes,jwt,fusion,pub"
```
* Subscriber mode:
```yaml theme={"dark"}
fusion-config-sync:
springProfilesOverride: "kubernetes,jwt,fusion,sub"
```
* PubSub mode:
```yaml theme={"dark"}
fusion-config-sync:
springProfilesOverride: "kubernetes,jwt,fusion,pub,sub"
```
For additional options, see [Filters](/docs/5/fusion/operations/config-sync/filters/overview) and the [ConfigSync configuration reference](/docs/5/fusion/operations/config-sync/settings).
## Additional configuration options
For additional configuration options, see the [ConfigSync configuration reference](/docs/5/fusion/operations/config-sync/settings).
## Disable ConfigSync
To disable ConfigSync, set the `enabled` flag to `false` in your Helm chart configuration:
```yaml theme={"dark"}
fusion-config-sync:
enabled: false
```
# Data restoration
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/disaster-recovery
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/0iq2cv
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/disaster-recovery
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/disaster-recovery
ConfigSync restore operation allows you to restore the Fusion config to a specific state.
To learn more about the ConfigSync repository and branches, see [Repo organization](/docs/5/fusion/operations/config-sync/repo-org).
## Before you begin
Please make sure that ConfigSync pod is up and running. You can check it by running the following command:
```sh wrap theme={"dark"}
kubectl get pods | grep config-sync
```
## Restore from the specific branch
To restore from the specific branch, use the `branch=` request parameter.
### HTTP request
```http wrap theme={"dark"}
PUT https:///config-sync/restore?branch=
Authorization: Basic
```
### cURL request
```bash wrap theme={"dark"}
curl -I -X PUT -u 'admin:' \
'https:///config-sync/restore?branch='
```
## Restore from a specific commit
First, get the commit SHA from the commit history.
To find the commit you want to restore from:
* Go to the Config Sync GitHub repository.
* Select the branch you want to restore from.
* Click on the `commits` link.
* Find the commit you want to restore from and click on the commit message to view the commit details.
* Copy the commit SHA from the commit details page or from the URL. Also, you could copy it from the commit history page.
To restore from a specific commit, use the `commit=` request parameter. For example:
### HTTP request
```http wrap theme={"dark"}
PUT https:///config-sync/restore?commit=
Authorization: Basic
```
### cURL request
```bash wrap theme={"dark"}
curl -I -X PUT -u 'admin:' \
'https:///config-sync/restore?commit='
```
## Restore from a specific date
To restore from a specific date, use the `date=` request parameter. The date should be in the `YYYY-MM-DDTHH:MM:SSZ` UTC format.
For example: `2021-01-01T00:00:00Z`.
In this case ConfigSync restores the config to the state it was at the specified date.
Actually, it restores the config to the state it was at the last commit before the specified date.
### HTTP request
```http wrap theme={"dark"}
PUT https:///config-sync/restore?date=
Authorization: Basic
```
### cURL request
```bash wrap theme={"dark"}
curl -I -X -u 'admin:' \
'https:///config-sync/restore?date='
```
## Restore specific apps
To restore specific apps, use the `apps=` request parameter. For example:
```bash wrap theme={"dark"}
curl -I -X PUT -u 'admin:' \
'https:///config-sync/restore?branch=&apps=app1,app2'
```
or
```bash wrap theme={"dark"}
curl -I -X PUT -u 'admin:' \
'https:///config-sync/restore?commit=&apps=app1,app2'
```
## Track the restore progress
Each restore operation is asynchronous. You can track the restore progress by checking logs in the ConfigSync pod.
# Apps filter
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/filters/apps-filter
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/7p74hf
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/filters/apps-filter
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/filters/apps-filter
You can configure which apps to synchronize using the `sync.apps` config setting in the `fusion-config-sync` section of the Helm chart.
Apps can either be excluded or included.
For instance, to exclude application `app1` from synchronization, use the following configuration:
import from '/snippets/LwTemplate.jsx';
```yaml wrap theme={"dark"}
cfg:
sync:
apps: "-app1"
```
To include only `app1` and `app2` for synchronization, use the following
```yaml wrap theme={"dark"}
cfg:
sync:
apps: "app1,app2"
```
To include all apps for synchronization, use the following
```yaml wrap theme={"dark"}
cfg:
sync:
apps: "*"
```
# Object filters
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/filters/object-filter
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/ztfcnr
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/filters/object-filter
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/filters/object-filter
You can configure which objects to synchronize using the `objects` config setting in the `fusion-config-sync` section of the Helm chart.
## Enabled/disabled objects
Each object type can be disabled for synchronization using the `cfg.sub.objects..enabled` setting.
For instance, to disable synchronization of the `users` object type, use the following configuration:
```yaml wrap theme={"dark"}
cfg:
sub:
objects:
users:
enabled: false
```
By default, all object types are enabled for synchronization.
## Name filters
Each object type can have a name filter regex (`cfg.sub.objects..name-filter` setting) that filters out objects by their name.
The global name filter regex for all object types is `cfg.sub.objects.name-filter` setting.
For instance, to exclude all objects with names starting with `test` for the `users` object type, use the following configuration:
```yaml wrap theme={"dark"}
cfg:
sub:
objects:
users:
name-filter: "^test.*"
```
## Prop filters
Each object type can have a prop filter list (`cfg.pub.objects..prop-filter` setting) that filters out object properties.
[Subscriber mode](/docs/5/fusion/operations/config-sync/modes/subscriber-mode) does not support prop filters.
The prop filter list is a comma-separated list of property JSON Pointer paths.
Read more about JSON Pointer syntax [here](https://tools.ietf.org/html/rfc6901).
The global prop filter list for all object types is `cfg.pub.objects.prop-filter` setting.
For instance, to exclude the `password` property for the `users` object type, use the following configuration:
```yaml wrap theme={"dark"}
cfg:
pub:
objects:
users:
prop-filter: "/password"
```
# Filters
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/filters/overview
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/a8nbty
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/filters/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/filters/overview
ConfigSync supports [App filters](/docs/5/fusion/operations/config-sync/filters/apps-filter) and [Object filters](/docs/5/fusion/operations/config-sync/filters/object-filter).
Below is an example configuration in the `fusion-config-sync` section of the Helm chart that synchronizes all apps but only specific objects:
```yaml wrap expandable theme={"dark"}
pub:
apps: "*"
objects: &objects
# filter tmp file names
name-filter: ".*_lw_tmp.*"
# possible modes: all, any
json-matchers-mode: "any"
# filter audit properties by json pointer
prop-filter: >
/created,
/modified,
/createdAt,
/updatedAt,
/updates
solr-configs:
name-filter: "^(system_|_default).*"
solr-collections:
name-filter: "^system_.*"
apps:
prop-filter: >
/properties/previousCollectionId
configurations:
name-filter: "^config-sync.+"
datasources-v2:
prop-filter: >
/_id
blobs:
name-filter: "^quickstart/.+"
json-matchers: >
/contentType=.+
collections:
name-filter: "^system_.+"
spark-jobs:
# data-migration jobs
name-filter: "^lw_(import|export)_.+"
tasks:
# data-migration tasks
name-filter: "^lw_export_finalizer_.+"
index-pipelines:
prop-filter: >
/properties/secretSourcePipelineId,
/stages/secretSourceStageId,
/stages/licensed
query-pipelines:
prop-filter: >
/properties/secretSourcePipelineId,
/stages/secretSourceStageId,
/stages/licensed
users:
json-matchers: >
/username!=admin
roles:
json-matchers: >
/name!=admin
groups:
json-matchers-mode: "any"
json-matchers: >
/id=_shared,
/isSystem=false
```
# Branch locks
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/locks
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/srm7fz
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/locks
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/locks
ConfigSync uses locks to prevent concurrent access to the same repository from multiple ConfigSync instances.
The lock is a file named `.lock` in the root of the repository.
The lock file contains a UUID that is generated when the lock is acquired.
Each time on startup, ConfigSync checks if the lock file exists in the repository and the file contents match the UUID
in the `config-sync.branch-lock` Fusion configuration property (stored in ZooKeeper).
* If the lock file does not exist, ConfigSync creates it and writes the UUID to the file.
* If the Fusion configuration property is not set, ConfigSync sets the UUID in the `config-sync.branch-lock` property to the UUID in the lock file.
* If the lock file exists and the UUID in the file matches the UUID in the `config-sync.branch-lock` Fusion configuration, ConfigSync continues to work with the repository; otherwise it fails to start.
If you encounter an error like `App lock doesn’t match git lock `, see **Delete the ConfigSync lock file**.
If you recreate a Fusion cluster, the lock file remains in the repository and prevents ConfigSync from starting.
You might see the following error in the logs:
```log theme={"dark"}
Caused by: java.lang.IllegalStateException: App lock doesn't match git lock
at com.lucidworks.cloud.config.repo.AbstractSubscriberGitRepo.checkLock(AbstractSubscri
```
To resolve this issue:
1. Delete the lock file.
The lock is a file named `.lock` in the root of the repository.
2. Restart the `config-sync` service.
## Learn more
ConfigSync can lock the configuration to a specific commit.
This means that ConfigSync :
* Does not apply any commits that are newer than the locked commit
* Does not update changes from the Fusion cluster to the Git repository
You can set `locked-commit` using the API:
```http request theme={"dark"}
PUT /api/properties/locked-commit
Content-Type: text/plain
COMMIT_SHA1
```
Replace `COMMIT_SHA1` with the commit SHA1 to which you want to lock ConfigSync.
If `locked-commit` precedes `latest-synced-commit`, ConfigSync reverts all changes to the `locked-commit` and sets `latest-synced-commit` to `locked-commit`.
# ConfigSync metrics and annotations
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/metrics
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/c7r9h8
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/metrics
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/metrics
The metrics endpoint is available at `/actuator/prometheus` on the ConfigSync service. All custom metrics use the `configsync_` prefix for easy discovery and to avoid collisions with other applications.
The following metrics are exposed by the ConfigSync service:
* `configsync_resource_update_errors_total` (`counter`) - The total number of errors encountered while updating resources.
* `configsync_out_of_sync` (`gauge`) - The number of resources that are out of sync. Only in `sub` mode.
* `configsync_sync_seconds` (`summary`) - The time taken to sync resources.
* `configsync_sync_seconds_max` (`gauge`) - The maximum time taken to sync resources.
## Annotations
Annotations are visual markers that appear on Grafana time-series graphs to highlight when specific events occurred. They provide context by overlaying event information directly onto your graphs, making it easier to correlate Fusion system behavior with ConfigSync activities.
When monitoring ConfigSync in Grafana graphs, annotations help you:
* **Identify the cause of metric changes** - Correlate spikes or dips in metrics with specific ConfigSync events like Git commits or Fusion configuration updates
* **Track deployment timelines** - Visualize when configuration changes were applied to your Fusion cluster
* **Debug synchronization issues** - Pinpoint when files changed in Git versus when they were applied to Fusion
Annotations in Grafana are created using Prometheus queries that trigger when certain conditions are met. The annotation appears as a vertical line or marker on your dashboard at the exact time the event occurred. You can hover over the annotation to see additional details about the event.
The metrics available to use in annotations are:
* `configsync_git_file_changes_total` (`counter`) - The number of files changes in Git from Fusion.
* `configsync_fusion_updates_total` (`counter`) - The number of operations applied to Fusion from Git.
* `configsync_git_commits_total` (`counter`) - The number of commits by type. You can use labels to filter by the type of commit.
* `updates-from-fusion` - Updates created by ConfigSync
* `explicit` - Updates created by users and applied to Fusion
## Example Grafana annotation queries
The following example queries can be used as annotation sources in Grafana.
### Annotate when Git files changed
`increase(configsync_git_file_changes_total[1m]) > 0`
### Annotate when Fusion was updated
`increase(configsync_fusion_updates_total[1m]) > 0`
### Annotate explicit user commits
`increase(configsync_git_commits_total{commit_type="explicit"}[1m]) > 0`
# Synchronization modes
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/modes/overview
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/twkik3
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/modes/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/modes/overview
[ConfigSync](/docs/5/fusion/operations/config-sync/overview) can operate in different modes:
import from '/snippets/LwTemplate.jsx';
* [Publisher mode](/docs/5/fusion/operations/config-sync/modes/publisher-mode): Monitors Fusion for changes and pushes them to a GitHub repository.
* [Subscriber mode](/docs/5/fusion/operations/config-sync/modes/subscriber-mode): Polls a GitHub repository for changes and applies them to Fusion.
* [Pub/Sub mode](/docs/5/fusion/operations/config-sync/modes/pubsub-mode): Combines the features of both Publisher and Subscriber modes.
See **Configure the ConfigSync mode** for steps to configure the synchronization mode.
You can configure the synchronization mode by adding the appropriate profile to the `sync` configuration in the `fusion-config-sync` section of the Helm chart configuration, as shown below.
* Publisher mode:
```yaml theme={"dark"}
fusion-config-sync:
springProfilesOverride: "kubernetes,jwt,fusion,pub"
```
* Subscriber mode:
```yaml theme={"dark"}
fusion-config-sync:
springProfilesOverride: "kubernetes,jwt,fusion,sub"
```
* PubSub mode:
```yaml theme={"dark"}
fusion-config-sync:
springProfilesOverride: "kubernetes,jwt,fusion,pub,sub"
```
For additional options, see [Filters](/docs/5/fusion/operations/config-sync/filters/overview) and the [ConfigSync configuration reference](/docs/5/fusion/operations/config-sync/settings).
# Publisher mode
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/modes/publisher-mode
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/b1rtue
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/modes/publisher-mode
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/modes/publisher-mode
In Publisher mode, ConfigSync watches Zookeeper and polls Solr collections for changes in Fusion and pushes them to a GitHub repository.
Once you deploy ConfigSync in the publisher mode, any changes to the app in the Fusion admin UI (or API) are picked and committed to the branch in GitHub.
See **Configure the ConfigSync mode** for steps to configure Publisher mode.
[ConfigSync](/docs/5/fusion/operations/config-sync/overview) can operate in different modes:
* [Publisher mode](/docs/5/fusion/operations/config-sync/modes/publisher-mode): Monitors Fusion for changes and pushes them to a GitHub repository.
* [Subscriber mode](/docs/5/fusion/operations/config-sync/modes/subscriber-mode): Polls a GitHub repository for changes and applies them to Fusion.
* [Pub/Sub mode](/docs/5/fusion/operations/config-sync/modes/pubsub-mode): Combines the features of both Publisher and Subscriber modes.
You can configure the synchronization mode by adding the appropriate profile to the `sync` configuration in the `fusion-config-sync` section of the Helm chart configuration, as shown below.
* Publisher mode:
```yaml theme={"dark"}
fusion-config-sync:
springProfilesOverride: "kubernetes,jwt,fusion,pub"
```
* Subscriber mode:
```yaml theme={"dark"}
fusion-config-sync:
springProfilesOverride: "kubernetes,jwt,fusion,sub"
```
* PubSub mode:
```yaml theme={"dark"}
fusion-config-sync:
springProfilesOverride: "kubernetes,jwt,fusion,pub,sub"
```
For additional options, see [Filters](/docs/5/fusion/operations/config-sync/filters/overview) and the [ConfigSync configuration reference](/docs/5/fusion/operations/config-sync/settings).
## Watch Zookeeper
The publisher watches Zookeeper for changes in Fusion objects.
It uses Fusion’s [Links API](/api-reference/links/get-the-links-service-status) to determine app association for Fusion objects.
The publisher does not track links in GitHub as it uses the directory structure to maintain app association for an object.
## Poll Solr collections
Not all Fusion objects are stored in Zookeeper.
Some objects are stored in Solr collections, such as blobs, rules, etc.
The publisher polls Solr collections for changes in these objects and pushes them to the GitHub repository.
### Blobs
**Blobs are stored in Solr (`system_blobs` collection) and not Zookeeper, consequently there is no ZK watch event fired when a blob has been added, deleted, or updated.**
Consequently, the publisher runs a background task that queries the blob store for changes every 20 seconds (configurable via `cfg.pub.checkForBlobUpdatesEveryMs`).
If blob changes are detected, the publisher stores the blob in the local repo and normally pushes it to GitHub.
### Rules
Rules are stored in Solr (`*_query_rewrite` and `*_query_rewrite_staging` collections) and not Zookeeper, consequently there is no ZK watch event fired when a rule has been added, deleted, or updated.
Consequently, the publisher runs a background task that queries the rule store for changes every 300 seconds (configurable via `cfg.pub.checkForRuleUpdatesEveryMs`).
Rules are stored in the local repo as JSON lines files and normally pushed to GitHub.
Rules sync isn’t a cheap operation, as it requires a full scan of the `*_query_rewrite` and `*_query_rewrite_staging` collections.
## Pushing changes
By default, the publisher pushes pending commits in the local repo every 30 secs (see `cfg.sync.syncEvery`); a PUSH operation is typically slow and can take several seconds to complete.
PUSH is synchronized so that only one thread is sending updates to GitHub at a time.
## Repository structure
Read more about the repository structure in the [Repo organization](/docs/5/fusion/operations/config-sync/repo-org) section.
## Watched objects
Check the [Supported objects](/docs/5/fusion/operations/config-sync/objects/overview) section for a list of objects that are watched by the publisher.
## Filters
Publisher mode supports [App filters](/docs/5/fusion/operations/config-sync/filters/apps-filter) and [Object filters](/docs/5/fusion/operations/config-sync/filters/object-filter).
# PubSub mode
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/modes/pubsub-mode
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/r0xnd3
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/modes/pubsub-mode
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/modes/pubsub-mode
The ConfigSync service can work in a mixed mode called PubSub. This mode is a combination of the [Publisher mode](/docs/5/fusion/operations/config-sync/modes/publisher-mode) and [Subscriber mode](/docs/5/fusion/operations/config-sync/modes/subscriber-mode).
In this mode, the changes from Fusion are propagated to GitHub (like in the publisher mode), but you can also restore Fusion applications from GitHub (like in the Subscriber mode).
This mode is more flexible than the Publisher and Subscriber modes, but it requires more resources and is more complex.
See **Configure the ConfigSync mode** for steps to configure PubSub mode.
[ConfigSync](/docs/5/fusion/operations/config-sync/overview) can operate in different modes:
* [Publisher mode](/docs/5/fusion/operations/config-sync/modes/publisher-mode): Monitors Fusion for changes and pushes them to a GitHub repository.
* [Subscriber mode](/docs/5/fusion/operations/config-sync/modes/subscriber-mode): Polls a GitHub repository for changes and applies them to Fusion.
* [Pub/Sub mode](/docs/5/fusion/operations/config-sync/modes/pubsub-mode): Combines the features of both Publisher and Subscriber modes.
You can configure the synchronization mode by adding the appropriate profile to the `sync` configuration in the `fusion-config-sync` section of the Helm chart configuration, as shown below.
* Publisher mode:
```yaml theme={"dark"}
fusion-config-sync:
springProfilesOverride: "kubernetes,jwt,fusion,pub"
```
* Subscriber mode:
```yaml theme={"dark"}
fusion-config-sync:
springProfilesOverride: "kubernetes,jwt,fusion,sub"
```
* PubSub mode:
```yaml theme={"dark"}
fusion-config-sync:
springProfilesOverride: "kubernetes,jwt,fusion,pub,sub"
```
For additional options, see [Filters](/docs/5/fusion/operations/config-sync/filters/overview) and the [ConfigSync configuration reference](/docs/5/fusion/operations/config-sync/settings).
Here’s how PubSub mode works:
1. [`pub:`](https://git-scm.com/docs/merge-strategies#Documentation/merge-strategies.txt-resolve) The first job reads all the configurations from Fusion, adds all the new or changed files to Git, and then performs a `git pull` from the remote GitHub repository using \[the `resolve` merge strategy^]
2. `[pub]:` In case of conflicts during the pull command, the changes from Fusion always take precedence (which should be a rare occurrence).
3. `[sub]:` After that, all the changes in the GitHub files since the latest commit SHA1 are resolved and applied to Fusion.\
If the latest commit SHA1 is not set in the Fusion configuration, it is set to the latest commit SHA1 in the GitHub repository.\
That means that the first time the PubSub mode is enabled, ConfigSync does not apply any changes from GitHub to Fusion.
4. `[sub]:` The current commit SHA1 is set as the latest commit SHA1 in the Fusion configuration.
5. `[pub/sub]:` The next job reads all the configurations from Fusion and compares them with the latest commit SHA1 in the GitHub repository. If there are any differences, the changes from GitHub are applied to Fusion.
6. `[pub/sub]:` The process is repeated in a loop.
## Filters
PubSub mode supports [App filters](/docs/5/fusion/operations/config-sync/filters/apps-filter) and [Object filters](/docs/5/fusion/operations/config-sync/filters/object-filter).
# Subscriber Mode
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/modes/subscriber-mode
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/t51vvb
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/modes/subscriber-mode
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/modes/subscriber-mode
Subscriber mode is a long-running process that watches a Git branch for changes.
You can use this mode to make GitHub the source of truth for the Fusion configuration.
When a change is detected, the subscriber pulls the latest changes from the branch and applies them to the Fusion cluster.
The subscriber is designed to be safe and idempotent, meaning it can be run multiple times without causing issues.
Changes are applied using REST API calls based on the type of object.
For example, a query pipeline update is applied using a `PUT` call to the query pipeline API endpoint.
* All the appropriate validation occurs before the update is applied
* Avoids having to keep track of link objects outside Fusion
* Lets Fusion trigger all the necessary dependent changes as needed
* Better error handling if something goes wrong, the API may be able to correct itself
If the ConfigSync subscriber is down when an update to the subscriber branch is applied, then the subscriber applies the changes when it restarts.
The subscriber keeps track of the latest Git SHA1 hash it has applied to the target environment (stored in ZooKeeper).
Thus, if the subscriber branch changes while the service is offline, when it re-initializes, it compares the stored Git SHA1 with the latest it gets from a `git pull` operation during startup.
See **Configure the ConfigSync mode** for steps to configure Subscriber mode.
[ConfigSync](/docs/5/fusion/operations/config-sync/overview) can operate in different modes:
* [Publisher mode](/docs/5/fusion/operations/config-sync/modes/publisher-mode): Monitors Fusion for changes and pushes them to a GitHub repository.
* [Subscriber mode](/docs/5/fusion/operations/config-sync/modes/subscriber-mode): Polls a GitHub repository for changes and applies them to Fusion.
* [Pub/Sub mode](/docs/5/fusion/operations/config-sync/modes/pubsub-mode): Combines the features of both Publisher and Subscriber modes.
You can configure the synchronization mode by adding the appropriate profile to the `sync` configuration in the `fusion-config-sync` section of the Helm chart configuration, as shown below.
* Publisher mode:
```yaml theme={"dark"}
fusion-config-sync:
springProfilesOverride: "kubernetes,jwt,fusion,pub"
```
* Subscriber mode:
```yaml theme={"dark"}
fusion-config-sync:
springProfilesOverride: "kubernetes,jwt,fusion,sub"
```
* PubSub mode:
```yaml theme={"dark"}
fusion-config-sync:
springProfilesOverride: "kubernetes,jwt,fusion,pub,sub"
```
For additional options, see [Filters](/docs/5/fusion/operations/config-sync/filters/overview) and the [ConfigSync configuration reference](/docs/5/fusion/operations/config-sync/settings).
## Disaster recovery support
ConfigSync (in subscriber mode) can restore a Fusion cluster from source control in the event of a disaster.
You can also use it to restore the Fusion configuration to a specific state by specifying a branch, commit, or date.
Read more about [Data restoration](/docs/5/fusion/operations/config-sync/disaster-recovery)
## Filters
Subscriber mode supports [App filters](/docs/5/fusion/operations/config-sync/filters/apps-filter) and [Object filters](/docs/5/fusion/operations/config-sync/filters/object-filter).
## Ordered updates
The subscriber orders updates of Fusion objects based on the update type (`DELETE`, `CREATE`, `UPDATE`) and resource type (such as changes to index pipelines applied before index profiles).
For example, a query profile can refer to a query pipeline and optionally an experiment.
Thus, operations are performed on query pipelines and experiments before query profiles.
Read more about ordered updates in [Supported objects](/docs/5/fusion/operations/config-sync/objects/overview).
## Rollback
The subscriber pulls updates from the GitHub branch.
Thus, to "rollback", a Fusion admin needs to push a change to the branch to set the desired state for Fusion objects.
In other words, the subscriber doesn’t have the concept of rollback but applies the changes based on the current HEAD of the branch it watches.
## Collections Deletion
ConfigSync (in subscriber mode) does not delete collections.
This is because deleting a collection is a destructive operation that can lead to data loss.
## Environment specific Substitution Variables
One of the challenges of promoting config changes from one environment to another is the need to apply environment specific config changes.
For instance, the JDBC database URL would be different for staging and production.
ConfigSync provides a very simple mechanism to parameterize environment specific settings.
In subscriber mode, before applying changes to the target environment, ConfigSync matches substitution variables from four possible variables files:
```bash theme={"dark"}
REPO_ROOT
|__ vars.json
|__
|__ vars.json
```
Project level `vars.json` takes precedence over the root level `vars.json`
Each variable has an optional [glob path](https://javapapers.com/java/glob-with-java-nio/) matcher and [JSON Path](https://www.baeldung.com/guide-to-jayway-jsonpath) component.
For instance, the following variable sets any stage property named `secret` to value `STAGE_SECRET` for the query pipeline with ID `lab4`:
```json wrap theme={"dark"}
{
"key": "/queryPipelines/lab4:$.stages.*.secret",
"value": "STAGE_SECRET"
}
```
Here’s an example of a variable that does not have a path component, which means it is evaluated for every object processed by the subscriber before it makes the API call:
```json wrap theme={"dark"}
{
"key": "$.stages.*.paramToTag",
"value": "QUERY"
}
```
Any stage `paramToTag` property is updated to the value `QUERY`.
### Secret Variables
Secret variables are a special type of variable that are sourced from a file, java system property, or environment variable.
This is useful for sensitive information like passwords, API keys, etc.
The secret variable is a JSON object with the following properties:
#### File
```json wrap theme={"dark"}
{
"key": "$.stages.*.dbPassword",
"type": "secret",
"secretPath": "/etc/secret-volume/db-password"
}
```
In this example, the `dbPassword` setting for any stage is populated from a file located at `/etc/secret-volume/db-password`.
#### Env or System Property
```json wrap theme={"dark"}
{
"key": "$.stages.*.dbPassword",
"type": "secret",
"secretEnv": "DB_PASSWORD"
}
```
In this example, the `dbPassword` setting for any stage is populated from an environment variable named `DB_PASSWORD`.
If the environment variable is not set, the java system property is checked as a fallback.
# Blobs
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/blobs
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/kslobm
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/objects/blobs
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/blobs
When you add a blob to the [blob store](/docs/5/fusion/getting-data-in/blob-storage) in Fusion, extra information is needed:
* **Resource type**: This tells Fusion what kind of file or data you’re adding.
* **Content type**: This is about the file format, such as JPEG, JavaScript, and so on.\
You can find more about these types in the [Blob Store API](/api-reference/blobs/get-blob-store-service-status).
## How ConfigSync handles blob metadata
ConfigSync handles this extra information with a special `.metadata` file.
This file has the same name as your blob but is stored in a different location.
For example, if your blob’s path is `app/blobs/some-folder/blob.txt`, then your metadata file’s path should be `APP/.metadata/blobs/FOLDER_NAME/blob.txt`.
## Metadata file structure
This `.metadata` file is a JSON file that looks like this:
```json wrap theme={"dark"}
{
"contentType": "text/javascript",
"resourceType": "file:js-index"
}
```
### Rules for using meta-files
* You only need to include the metadata file when adding new blobs.\
Existing blobs should already have metadata files.
* If you ever need to change the `resourceType` or `contentType` of a blob, update and commit the meta-file.\
No need to re-upload the blob unless it has also changed.
### Metadata for system blobs
The `_lw_system` folder stores custom connectors or JDBC drivers which do not require separate metadata files.
# Connector plugins
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/connector-plugins
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/gxge5x
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/objects/connector-plugins
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/connector-plugins
ConfigSync enhances Fusion’s capabilities by allowing for the efficient management of both custom and standard connector plugins.
This functionality is pivotal for extending the data integration and processing capabilities of Fusion.
## Custom connectors
Custom connectors are identified based on their presence and version compatibility with the connector repository.
A connector is deemed custom if it:
* Is not listed in [Connector Configuration Reference](/docs/fusion-connectors/overview).
* Has a version that differs from what is available in the repository.\
Furthermore, only connectors following the version 2 (v2) architecture, also known as non-classic connectors, are recognized as custom.
### Connector repository
Connectors such as "kaltura (v2)" and "solr (classic)" are available in the repository with specific versions and SDK compatibility.
You can see this information at the `/api/connectors/repository` endpoint.
```json wrap theme={"dark"}
[
{
"id": "lucidworks.kaltura",
"displayName": "kaltura (v2)",
"version": "0.0.1",
"sdkVersion": "4.1.3",
"location": "4.1.3/lucidworks.connector.kaltura/1.0.0/lucidworks.connector.kaltura-1.0.0.zip"
},
{
"id": "lucid.solr",
"displayName": "solr (classic)",
"version": "5.5.1",
"sdkVersion": "5.5.1",
"location": "/classic/apps/connectors/repository-plugins/lucidworks.solr-5.5.1.zip"
}
]
```
### Deployed connectors
Examples include `lucidworks.workday`, `lucidworks.kaltura`, and `lucid.solr` marked as `DEPLOYED` in Fusion’s connectors list.
You can see this information at the `/api/connectors/plugins` endpoint.
```json wrap theme={"dark"}
[
{
"type": "lucidworks.workday",
"state": "DEPLOYED"
},
{
"type": "lucidworks.kaltura",
"state": "DEPLOYED"
},
{
"type": "lucid.solr",
"state": "DEPLOYED"
}
]
```
### Custom connector criteria
* `lucidworks.kaltura` is custom due to its unique developmental version `0.1.0-dev.565.uncommitted+kaltura.5.4.nwm.1217913` not matching the repository’s version `0.0.1`.
* `lucidworks.workday` qualifies as custom because it is absent in the repository.
* `lucid.solr` is not considered custom as it is a classic connector.
* [https://fusion-host/api/connectors/plugins/lucidworks.kaltura](https://fusion-host/api/connectors/plugins/lucidworks.kaltura)
```json wrap theme={"dark"}
{
"id": "EC0oe4dWRX",
"name": "lucidworks.kaltura",
"pluginVersion": "0.1.0-dev.565.uncommitted+kaltura.5.4.nwm.1217913",
"sdkVersion": "4.0.0",
"schema": {}
}
```
### Storage of custom connectors
Custom connectors are systematically stored in the `_lw_system/connectors-plugins` folder within the config-sync repository.
## Standard connectors
ConfigSync facilitates the automatic installation of non-custom connectors.
When creating or migrating a data source associated with a non-custom connector that is not yet installed, ConfigSync employs Fusion’s default mechanism to install the latest version from the repository.
This ensures seamless data source integration.
If a commit introduces a data source linked to an uninstalled non-custom connector, ConfigSync automatically proceeds with its installation.
# Index stage plugins
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/index-stage-plugins
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/dnjquf
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/objects/index-stage-plugins
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/index-stage-plugins
Similar to [query stage plugins](/docs/5/fusion/operations/config-sync/objects/query-stage-plugins), these are binary blobs that enhance or modify the indexing process in Fusion.
They’re specifically targeted by ConfigSync for seamless integration.
For a deeper understanding, see the [Index Stage SDK](/docs/5/fusion/dev-portal/index-stage-sdk).
These plugins are kept in `_lw_system/index-stage-plugins`, ensuring they’re correctly managed and deployed by ConfigSync.
# JDBC drivers
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/jdbc-drivers
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/3dussz
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/objects/jdbc-drivers
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/jdbc-drivers
JDBC drivers allow Fusion to connect to various databases, enabling data ingestion from a wide range of sources.
These drivers are also managed by ConfigSync, ensuring that Fusion has the necessary drivers to connect to your databases.
For more information, see **Upload a JDBC Driver to Fusion**.
The JDBC V2 connector is supported, and fetches documents from a relational database via SQL queries. Under the hood, this connector implements the Solr [DataImportHandler (DIH)](https://wiki.apache.org/solr/DataImportHandler) plugin.
Fusion stores JDBC drivers in the blob store. You can upload a driver using the Fusion UI or the Blob Store API.
## How to upload a JDBC driver using the Fusion UI
1. In the Fusion UI, navigate to **System** > **Blobs**.
2. Click **Add**.
3. Select **JDBC Driver**.\
The "New 'JDBC Driver' Upload" panel appears.
4. Click **Choose File** and select the .jar file from your file system.
5. Click **Upload**.\
The new driver’s blob manifest appears.
From this screen you can also delete or replace the driver.
## How to install a JDBC driver using the API
1. Upload the JAR file to Fusion’s blob store using the [`/blobs/{id}` endpoint](/api-reference/blobs/upload-a-blob).
Specify an arbitrary blob ID, and a `resourceType` value of `plugin:connector`, as in this example:
```bash theme={"dark"}
curl -u USERNAME:PASSWORD -H "content-type:application/java-archive" -H "content-length:707261" -X PUT --data-binary @postgresql-42.0.0.jar http://localhost:8764/api/blobs/mydriver?resourceType=driver:jdbc
```
Success response:
```json theme={"dark"}
{
"name" : "mydriver",
"contentType" : "application/java-archive",
"size" : 707261,
"modifiedTime" : "2017-06-09T19:00:48.919Z",
"version" : 0,
"md5" : "c67163ca764bfe632f28229c142131b5",
"metadata" : {
"subtype" : "driver:jdbc",
"drivers" : "org.postgresql.Driver",
"resourceType" : "driver:jdbc"
}
}
```
Fusion automatically publishes the event to the cluster, and the listeners perform the driver installation process on each node.
If the blob ID is identical to an existing one, the old driver will be uninstalled and the new driver will installed in its place. To get the list of existing blob IDs, run: `curl -u USERNAME:PASSWORD https://FUSION_HOST:FUSION_PORT/api/blobs`
2. To verify the uploaded driver, run:
```bash theme={"dark"}
curl -u USERNAME:PASSWORD https://FUSION_HOST:FUSION_PORT/api/blobs/BLOB_ID/manifest
```
Where the `BLOB_ID` is the name specified during upload, such as "mydriver" above. A success response looks like this:
```json theme={"dark"}
{
"name" : "mydriver",
"contentType" : "application/java-archive",
"size" : 707261,
"modifiedTime" : "2017-06-09T19:05:17.897Z",
"version" : 1569755095787110400,
"md5" : "c67163ca764bfe632f28229c142131b5",
"metadata" : {
"subtype" : "driver:jdbc",
"drivers" : "org.postgresql.Driver",
"resourceType" : "driver:jdbc"
}
}
```
Store your JDBC drivers in `_lw_system/jdbc-drivers`.
ConfigSync handles the rest, making sure they’re available for Fusion to use.
# JS-based stages in pipelines
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/js-pipeline-stages
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/jjg5pb
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/objects/js-pipeline-stages
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/js-pipeline-stages
When working with Fusion’s pipelines, you might come across stages that use JavaScript (JS) for custom logic.
These are known as JS-based stages and are identified by their type, which includes either `javascript-query` or `javascript-index`.
To make life easier when dealing with these stages, ConfigSync adopts a smart approach.
## Storing JS scripts separately
Instead of embedding JS scripts directly within the pipeline configuration, ConfigSync saves the script content to separate files.
This setup not only makes it easier to view and edit the scripts but also keeps your pipeline configurations clean and manageable.
## Where JS files are stored
* All JS files related to pipelines are stored in a `/js` folder within the specific pipeline folder.\
For index pipelines, this would be `/index-pipelines/js`, and for query pipelines, `/query-pipelines/js`.
* Within the `/js` folder, there’s a subfolder named after the pipeline to which the scripts belong.\
This helps keep everything organized and easy to find.
* The name of the JS file can vary and is typically based on the stage label used within ConfigSync.\
However, it’s crucial that these files are correctly linked to their corresponding pipeline stages.
## Linking JS files to pipeline stages
This linking is achieved through a meta-file, which maps each stage ID to its JS filename.
Here’s how it works:
* If your pipeline configuration is stored at `app/index-pipelines/pipeline.json`,
* Then your meta-file is located at `app/.metadata/index-pipelines/pipeline.json`.\
The pattern here is simple: just add `/.metadata` before the pipeline folder in the path.
* This applies to both index and query pipelines.
## The meta-file structure
The meta-file itself is a JSON file that contains mappings from stage IDs to JS filenames.
For example:
```json wrap theme={"dark"}
{
"js": {
"stage-id": "JS-Script-Name.js",
...
}
}
```
Basically, ConfigSync manages this file but in some rare cases such as if you want to rename the JS file you have to also reflect that in the mappings meta-file.
You can make changes in both pipeline-JSON or JS files but if both pipeline-JSON and JS files have been changed JS file changes always win.
This structure ensures that each JS script is correctly associated with its stage in the pipeline, facilitating easy updates and management.
## Managing changes
ConfigSync is designed to handle updates to these mappings, but there might be times when you need to manually adjust them, like if you’re renaming a JS file.
In such cases, remember to update the meta-file to reflect the changes.
When making edits, it’s possible to update both the pipeline configuration (the JSON file) and the JS files.
If there are conflicting changes, the updates made to the JS files take precedence.
This ensures that your custom logic in JS always reflects your latest intentions.
# Supported objects
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/overview
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/u8n1i5
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/objects/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/overview
ConfigSync supports a wide range of objects that can be synchronized.
This topic lists the supported objects in order of processing.
* System objects are processed before app objects.
* Within each group (system or app), objects are processed in a specific order.
* ConfigSync applies updates in the following order:
1. `DELETE`
2. `CREATE`
3. `UPDATE`
The table below lists the object types supported by the ConfigSync service in order of processing by group:
| Object Type | Group | Path | Description |
| - | - | - | - |
| Configurations | sys: 0 | `_lw_system/configurations` | Custom system settings |
| Solr config-sets | sys: 0 | `_lw_system/solr-configs` | [Solr config-sets](/docs/5/fusion/operations/config-sync/objects/solr-configs), such as `solrconfig.xml`, `managed-schema.xml`, and so on. |
| Solr collections | sys: 1 | `_lw_system/solr-collections` | [Solr collections](/docs/5/fusion/operations/config-sync/objects/solr-collections) metadata |
| Rules | | `_lw_system/collections-data` | Query rewrite [rules](/docs/5/fusion/operations/config-sync/objects/rules) |
| Users | sys: 2 | `_lw_system/users` | [User](/docs/5/fusion/operations/config-sync/objects/users) accounts and permissions |
| Roles | sys: 2 | `_lw_system/roles` | Roles and access levels |
| Realms | sys: 2 | `_lw_system/realms` | Security realms and configurations |
| API keys | sys: 2 | `_lw_system/api-keys` | API keys for Fusion API access |
| Connectors plugins | sys: 2 | `_lw_system/connectors-plugins` | [Connector plugins](/docs/5/fusion/operations/config-sync/objects/connector-plugins) for data connectivity |
| Query stage plugins | sys: 2 | `_lw_system/query-stage-plugins` | [Query stage plugins](/docs/5/fusion/operations/config-sync/objects/query-stage-plugins) for query pipelines |
| Index stage plugins | sys: 2 | `_lw_system/index-stage-plugins` | [Index stage plugins](/docs/5/fusion/operations/config-sync/objects/index-stage-plugins) for index pipelines |
| JDBC drivers | sys: 2 | `_lw_system/jdbc-drivers` | [JDBC drivers](/docs/5/fusion/operations/config-sync/objects/jdbc-drivers) for data connectivity |
| Apps | app: 0 | `/apps` | Fusion Applications |
| Blobs | app: 1 | `/blobs` | [Objects stored in the `system_blobs` collection](/docs/5/fusion/operations/config-sync/objects/blobs) |
| Fusion collections | app: 1 | `/collections` | Metadata for Solr collections in Fusion |
| Data models | app: 1 | `/data-models` | Data models |
| Zones | app: 1 | `/zones` | Predictive Merchandiser config |
| Datasources | app: 2 | `/datasources` | V1 Connectors config |
| Datasources V2 | app: 2 | `/datasources-v2` | V2 Connectors config |
| Query pipelines | app: 2 | `/query-pipelines` | [Query pipelines](/docs/5/fusion/operations/config-sync/objects/js-pipeline-stages) |
| Query profiles | app: 2 | `/query-profiles` | Query profiles |
| Experiments | app: 2 | `/experiments` | Query pipeline experiments |
| Parsers | app: 2 | `/parsers` | Index pipeline parsers |
| Index profiles | app: 2 | `/index-profiles` | Index profiles |
| Index pipelines | app: 2 | `/index-pipelines` | [Index pipelines](/docs/5/fusion/operations/config-sync/objects/js-pipeline-stages) |
| Spark jobs | app: 2 | `/spark-jobs` | [Spark jobs config](/docs/5/fusion/operations/config-sync/objects/scripts) |
| Tasks | app: 2 | `/tasks` | Tasks config |
| Subscriptions | app: 2 | `/subscriptions` | Subscriptions config |
| Groups | app: 2 | `/groups` | Object groups, including non-system and `_shared` groups |
| Links | app: 2 | `/links` | Fusion object links, supported only for non-system groups |
| Templates | app: 3 | `/templates` | Predictive Merchandiser config |
| Schedules | app: 3 | `/schedules` | For data-sources, Spark jobs and tasks |
Replace `/*` in the folder column with your Fusion app name.
The `_lw_system/*` folders are system objects folders.
# Query stage plugins
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/query-stage-plugins
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/l1mi38
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/objects/query-stage-plugins
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/query-stage-plugins
These are binary blobs used by Fusion to extend or customize the behavior of query processing stages.
ConfigSync specifically manages these plugins to ensure they’re correctly integrated into Fusion.
Check out the [Query Stage SDK](/docs/5/fusion/dev-portal/query-stage-sdk/overview) for details on how these plugins work.
You’ll store these plugins in `_lw_system/query-stage-plugins` within your repo.
This special folder is where ConfigSync looks to manage and synchronize query stage plugins.
**See also [Index stage plugins](/docs/5/fusion/operations/config-sync/objects/index-stage-plugins).**
# Rules
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/rules
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/mt9mtl
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/objects/rules
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/rules
ConfigSync treats rule collections distinctively, aligning more with configurations than data, due to their integral role in the operational logic of Fusion applications.
This unique approach to handling rule collections is outlined below, emphasizing their storage, synchronization, and selective inclusion criteria.
## Storage location
Rule collections are strategically stored within the `_lw_system/collections-data` repository folder.
This placement underscores the perception of rules as configurations pivotal to the Fusion ecosystem, necessitating careful version control and management.
## Format
Rule collections are stored as JSON lines files, a format that is conducive to version control and change tracking.
This format ensures that rule configurations are stored in a human-readable and version-friendly manner, facilitating collaboration and change management.
## Periodic change capture
ConfigSync actively monitors and captures changes to rule collections at 60-minute intervals, a frequency that is configurable.
This ensures that any modifications are promptly recorded, maintaining an up-to-date configuration state.
## Handling of specific rule collections
### `_query_rewrite` collections
Collections named with the `_query_rewrite` suffix are stored in their entirety.
This means all associated data within these collections is preserved in ConfigSync, providing a comprehensive snapshot of the rule configurations.
### `_query_rewrite_staging` collections
Staging collections, identified by the `_query_rewrite_staging` suffix, undergo a selective storage process.
Only records marked as deployed (filtered by `deployed:true`) are stored.
This selective inclusion is strategic, addressing the potential for these staging collections to amass significant volumes of data, possibly spanning several gigabytes.
Storing such vast amounts of data, especially on platforms like GitHub, could be impractical, even when leveraging Large File Storage ([Git Large File Storage](https://git-lfs.com/)) capabilities for rules.
### Commerce Studio collections
Collections with the `_em` suffix are automatically created when Commerce Studio is enabled for an application.
These collections follow the naming pattern `_query_rewrite_em` and contain Commerce Studio configuration data. Starting with Fusion 5.17.0, ConfigSync automatically backs up these collections in their entirety, similar to standard `_query_rewrite` collections.
# Tips for scripts and code fields
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/scripts
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/o72zno
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/objects/scripts
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/scripts
ConfigSync can be set up to automatically detect and separate fields containing scripts or code.
By moving these bits of code into separate files, it makes them much easier to view, edit, and manage.
## Setting up inline fields
Setup is done through a configuration that specifies which fields to inline based on their type and location.
Here are the basics:
* **Selectors**: These are like filters that tell ConfigSync which types of objects and fields it should look for.\
For example, you might have a selector for SQL templates within spark jobs.
* **Extensions**: This tells ConfigSync what file extension to use when it creates a new file for the inlined content, like `.sql` for SQL scripts or `.scala` for Scala scripts.
* **Properties**: These are the specific fields within your JSON objects that contain the script or code you want to inline.
## Default configuration
It can be found in the `application.yml` file.
## How it works
For every piece of content that matches a selector, ConfigSync creates a new file in a subfolder named after the object ID and the property name, with the specified extension.
This means your scripts get their own dedicated file, making them much easier to work with.
## Example in action
Imagine you have a JSON file for a spark job that includes a Scala script.
ConfigSync can automatically detect this script and move it into a separate `.scala` file, based on your configuration.
* **Original JSON file**: `spark-jobs/export_aad_acls.json` might contain a Scala script under the script property.
* **Inline setup**: According to the config, since it’s a type=script, ConfigSync creates a new file at `spark-jobs/scala/export_aad_acls/script.scala`, containing just the script.
* **The benefit**: You can now edit the Scala script directly in its own file, which is much simpler than editing inline JSON.
## Managing changes
You can make changes either directly in the new script file or in the original JSON.
However, if there are changes in both places, ConfigSync prioritizes the changes made in the separate script file.
# Solr collections
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/solr-collections
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/011kpa
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/objects/solr-collections
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/solr-collections
ConfigSync offers a streamlined way to handle Solr collections directly from your repository.
Here’s a practical guide on creating, modifying, and potentially deleting Solr collections through ConfigSync.
## Configuration storage
Store your Solr collection configurations in the `_lw_system/solr-collections` directory within your repository.
## Creating Solr collections
### Format
The JSON file for a Solr collection should follow the Solr V2 API format as detailed in the [Solr Guide for Collection Management](https://solr.apache.org/guide/solr/latest/deployment-guide/collection-management.html#create).
### Example
Here’s a sample JSON configuration for creating a Solr collection with an alias:
```json wrap theme={"dark"}
{
"pullReplicas" : "0",
"replicationFactor" : "1",
"router" : {
"name" : "compositeId"
},
"maxShardsPerNode" : "1",
"autoAddReplicas" : "false",
"nrtReplicas" : "1",
"tlogReplicas" : "0",
"name" : "collection_live",
"numShards" : 1,
"alias" : "solr_collection_b",
"config" : "RH"
}
```
### Alias creation
If an alias is specified in the JSON, ConfigSync also creates this alias alongside the collection.
An alias cannot have the same name as an existing collection. Ensure your alias name is unique and does not match any collection name.
## Modifying Solr collections
### Modification constraints
While it’s possible to modify the Solr collection JSON file, only certain properties can be changed as per the [Solr Guide for Modifying Collections](https://solr.apache.org/guide/solr/latest/deployment-guide/collection-management.html#modifycollection).
### Properties you can modify
These include `alias`, `config`, `replicationFactor`, `maxShardsPerNode`, `readOnly`, `perReplicaState`, and `autoAddReplicas`.
### Alias modification/deletion
Modifying the collection configuration can also modify or delete an existing alias, if specified.
## Deleting Solr collections
### Safety first
By default, ConfigSync is designed not to delete Solr collections to prevent accidental data loss.
### Force deletion
If you’re certain about your action, you can force delete a Solr collection by prefixing your commit message with `force:` when you remove collection files from the repo.
This action can also remove any specified alias.
# Solr configs
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/solr-configs
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/0ksttl
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/objects/solr-configs
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/solr-configs
Solr configuration files are kept in the `_lw_system/solr-configs` directory.
Any changes to Solr configs trigger a reload of the corresponding collections to ensure they’re up-to-date with the latest settings.
# Users
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/users
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/ce5jc6
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/objects/users
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/objects/users
User data files are stored within the directory `_lw_system/users`.
This designated location consolidates all user-related information, facilitating efficient data management and access.
## File naming convention
The naming convention for user files is structured as `---`.
This format is carefully chosen to ensure uniqueness and organization within the system.
It allows for easy identification and retrieval of user-specific data by linking the username directly to their associated security realm.
## Components
* **Username**: This is a unique identifier for each user.\
It is a string that distinguishes one user from another within the system.
* **Realm**: The realm refers to the security domain within which the user operates.\
It is a critical component of the system’s security model, segregating users based on their access permissions and operational scope.
## Reference documentation
Detailed information about user management, including the specifics of user file naming and location, can be found in [Users](/docs/5/fusion/operations/security/access-control/users).
# ConfigSync
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/overview
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/j47yes
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/overview
ConfigSync keeps your Fusion configuration settings consistent and up-to-date across different environments by automatically synchronizing configuration data between your GitHub repository and your Fusion cluster.
It follows [GitOps](https://github.blog/enterprise-software/devops/applying-gitops-principles-to-your-operations/) principles by using GitHub as the single source of truth for configuration management.
ConfigSync requires using GitHub. Other Git providers are not supported.
ConfigSync supports TLS in Fusion 5.9.16 and later. For earlier versions, TLS must be disabled.
## Features
ConfigSync supports these features:
* Configuration storage:
ConfigSync stores and synchronizes the objects that make up your Fusion configuration.
See [Supported Objects](/docs/5/fusion/operations/config-sync/objects/overview) for the complete list, with details about how each object type is handled.
It works like this:
* GitHub Repository: Stores the configuration files.
* Fusion Cluster: Applies the configuration files to the Fusion cluster.
* Kubernetes Secret: Stores the Fusion secrets.
* Synchronization mechanisms
* Polling:
* Periodically pulls the latest changes from the GitHub repository and applies them to the Fusion cluster.
* Periodically pulls the latest changes from some solr collections (blobs, rules) and pushes them to the GitHub repository.
* Event-driven:
* Listens for changes in the Fusion cluster and pushes them to the GitHub repository.
* Listens for changes in the Kubernetes secret and syncs them with the Fusion cluster.
* Synchronization modes
* [Publisher mode](/docs/5/fusion/operations/config-sync/modes/publisher-mode): Monitors Fusion for changes and pushes them to a GitHub repository.
* [Subscriber mode](/docs/5/fusion/operations/config-sync/modes/subscriber-mode): Polls a GitHub repository for changes and applies them to Fusion.
* [Pub/Sub mode](/docs/5/fusion/operations/config-sync/modes/pubsub-mode): Combines the features of both Publisher and Subscriber modes.
* [Secret management](/docs/5/fusion/operations/config-sync/secrets-management-used-in-config-sync)
* Supports storing Fusion secrets in a Kubernetes secret and syncing them with the GitHub repository using the legacy ZooKeeper encryption format.
* Watches for changes in the Kubernetes secret and syncs them with the Fusion cluster.
Config Sync uses the [Secrets Management using ZooKeeper](/docs/5/fusion/operations/security/zookeeper) method. For new deployments, see [Secrets Management using Fusion Vault](/docs/5/fusion/operations/security/external-vault).
* [Version control](/docs/5/fusion/operations/config-sync/commits-management)
* Maintains the `latest-synced-commit` to track the last applied commit.
* Supports locking the configuration to a specific commit using `locked-commit`.
* Restoring configuration
* Allows rollback to a previous state.
* Supports restoring the configurations to a specific commit, branch or date.
* Metrics
* Exposes metrics such as `resource_update_errors_total`, `out_of_sync`, `sync_seconds`, and `sync_seconds_max` at the `/actuator/prometheus `endpoint.
## Setup
See [ConfigSync deployment](/docs/5/fusion/operations/config-sync/deployment) to set up ConfigSync on your Fusion cluster.
The course for **ConfigSync** focuses on how to synchronize your Fusion configuration with a GitHub repository.
## ConfigSync and S3BackupRepository
If you are using the S3BackupRepository plugin for Solr, note that changes from a restoration using the plugin are not pushed through ConfigSync, because backups and restores on the primary cluster using S3BackupRepository do not run through the update handler.
Backup and restore is an independent process.
You must backup and restore in both locations.
# Repo organization
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/repo-org
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/a6m3du
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/repo-org
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/repo-org
This topic explains what you’ll find in the GitHub repository used by ConfigSync.
**See also **Set up a GitHub repo for ConfigSync**.**
[ConfigSync](/docs/5/fusion/operations/config-sync/overview) uses a GitHub repository to store the configuration files.
The repository should be created independently and shared with the ConfigSync service via the Helm chart values.
This document describes the process of setting up a GitHub repository for Fusion config files.
## Create a new repository
Create a new repository in the github organization where the ConfigSync should store the config files.
## Create a deploy key
Create a deploy key for the new repository.
The deploy key should have write access to the repository.
See the [GitHub documentation](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/managing-deploy-keys#set-up-deploy-keys) for instructions.
## Add the deploy key and repository information to the ConfigSync values
### Option 1: Add the deploy key in the Helm chart
```yaml theme={"dark"}
fusion-config-sync:
# other values
sshKey:
cfg:
github:
repo: # use the ssh url e.g. git@github.com:OWNER/REPO.git
branch: # you can leave it as is
mainBranch: # most probably it's `main`
```
If outbound port 22 is blocked, you can use SSH over HTTPS with a URL like `ssh://git@ssh.github.com:443/OWNER/REPO.git`.
See the [GitHub documentation](https://docs.github.com/en/authentication/troubleshooting-ssh/using-ssh-over-the-https-port) for details.
You still need a deploy key for this option.
If port 443 is also blocked, you can use HTTPS with a [Personal Access Token (PAT)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens).
To encode the private key to base64 you can use the following command:
```bash theme={"dark"}
cat ~/.ssh/id_rsa | base64
```
### Option 2: Add the deploy key as an external secret
Alternatively, you can store the deploy key as an external secret and reference it in the ConfigSync values.
#### Create a secret with the deploy key:
```yaml theme={"dark"}
apiVersion: v1
kind: Secret
metadata:
name: config-sync-ssh
type: Opaque
data:
config: SG9zdCAqCiAgU3RyaWN0SG9zdEtleUNoZWNraW5nIG5v
id_rsa:
```
To encode the private key to base64 you can use the following command:
```bash theme={"dark"}
cat ~/.ssh/id_rsa | base64
```
#### Update the ConfigSync values to reference the secret with the deploy key:
```yaml theme={"dark"}
fusion-config-sync:
# other values
sshKey: null
additionalVolumeMounts:
- name: config-sync-ssh-key
mountPath: /root/.ssh
readOnly: true
additionalVolumes:
- name: config-sync-ssh-key
secret:
secretName: config-sync-ssh
defaultMode: 256
```
## Additional configuration
`github.*` settings control the remote GitHub repo, branch, and path within the repo.
```yaml theme={"dark"}
github connection properties, common for pub/sub modes
cfg:
github:
repo: https://github.com/lucidworks/cloud-config-sync.git # git repo, mandatory
branch: dev # git branch, mandatory
mainBranch: main # main branch, mandatory
dir: "/tmp/config-sync-repo" # repo dir, default "/tmp/config-sync-repo"
path: / # configs path in the repo dir, default "/"
username: "fusion-config-sync" # git user name, default "fusion-config-sync"
email: "fusion-config-sync@lucidworks.com" # git user email, mandatory, default "fusion-config-sync@lucidworks.com"
```
## Update the ConfigSync deployment
Update the ConfigSync deployment with the new values.
## Branch and Path
If you need to share the same GitHub repo for different clusters or environments,
you can use the `cfg.github.branch` setting to specify the branch in the GitHub repo where the config files are stored.
Additionally (or alternatively) ConfigSync supports a `cfg.github.path` (referred to as `root` in the future)
setting to specify the path in the GitHub repo where the config files are stored.
By default, ConfigSync stores the config files in the root of the GitHub repo.
## Apps and Object Types
Under the `root` directory, there is a directory for each app, such as `app1`, `app2`.
Under each app directory, there are directories for each object type, such as `query-pipelines`, `blobs`, etc.
## System Objects
ConfigSync uses `_lw_system` as the app name for system objects.
This directory is always present in the repo and contains system objects like users, roles, solr configs, etc.
## Reserved Files
Additionally, ConfigSync has set of reserved files in the `root` directory:
* `.lock` - A lock file to prevent concurrent updates
* `.gitattributes` - Used to list LFS files
* `.gitignore` - Used to ignore files that should not be stored in the repo
**Important**
These files should not be modified or deleted.
## Example repo structure
```bash theme={"dark"}
|-- root
| |-- _lw_system
| | |-- users
| | | |-- user1.json
| | | |-- user2.json
| | |-- roles
| | | |-- role1.json
| | | |-- role2.json
| | |-- solr-configs
| | | |-- config1
| | | | |-- solrconfig.xml
| | | | |-- managed-schema.xml
| |-- app1
| | |-- query-pipelines
| | | |-- pipeline1.json
| | | |-- pipeline2.json
| | |-- blobs
| | | |-- blob1.json
| | | |-- blob2.json
| |-- app2
| | |-- query-pipelines
| | | |-- pipeline3.json
| | | |-- pipeline4.json
| | |-- blobs
| | | |-- blob3.json
| | | |-- blob4.json
|-- .lock
|-- .gitattributes
|-- .gitignore
```
ConfigSync uses the Fusion links API to determine app association for
Fusion objects. However, it does not track links in GitHub as it uses
the directory structure to maintain app association for an object.
# Secrets management in Config Sync
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/secrets-management-used-in-config-sync
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/2vrxsr
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/secrets-management-used-in-config-sync
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/secrets-management-used-in-config-sync
Secrets are sensitive data that should be protected. This data can be passwords, API keys, tokens, and so on.
Config Sync secrets management uses the **legacy ZooKeeper-based encryption format** (`default@encoded-secret`) detailed in [Secrets management using ZooKeeper](/docs/5/fusion/operations/security/zookeeper).
This feature is separate from the [Secrets management using Fusion Vault](/docs/5/fusion/operations/security/external-vault) introduced in Fusion 5.18, which integrates with Kubernetes External Secrets Operator and external vault providers.
## Secret definition
Config Sync doesn’t have a specific secret concept.
Secrets are defined as part of the Fusion configuration files.
Fusion uses `default@encoded-secret` as the secret value in the configuration files and ConfigSync rely on this value to manage secrets.
```json wrap theme={"dark"}
{
"id": "ds-name",
"properties": {
"password": "default@encoded-secret"
}
}
```
You could find the secret fields from the [Fusion codebase](https://github.com/search?q=org%3Alucidworks+%40StringSchema%28encrypted+%3D+true%29\&type=code\&p=1).
Basically these are data-source password fields.
## Options
Config Sync supports the following options for managing secrets:
* Keep secrets decoded in the repository.
* Keep secrets encoded in the repository and the actual secret stored in a Kubernetes secret.
## Keep secrets decoded in the repository
In this approach, the actual secret is stored in the repository in plain text.
### Pros
* Easy to manage.
* No need to manage multiple secrets.
### Cons
* Security risk.
* Anyone with access to the repository can manage the secret.
## Keep secrets encoded in the repository and the actual secret stored in a Kubernetes secret
In this approach, the actual secret is stored in a Kubernetes secret and the encoded secret is stored in the repository.
Encoded secret value is the same as in Fusion: `xXx-Redacted-xXx`.
Config Sync matches the encoded secret in the repository with the actual secret in the Kubernetes secret using the secret data and labels:
* labels:
* `app.kubernetes.io/part-of=config-sync`
* data:
* `file`: The file name of the file in the repository (such as `app/datasources-v2/ds-name.json`)
* `path`: JSON pointer to the secret in the file (such as `/properties/password`)
* `value`: The encoded secret value
## Syncing secrets (encoded secret) between Fusion and Kubernetes secret
Config Sync supports both way sync for secrets.
It can sync the actual secret from the Kubernetes secret to Fusion and vice versa.
### Initial sync
When Config Sync is started, it iterates over all the Fusion configuration files and check if there is any secret in the configuration file.
For each secret, it creates or updates the Kubernetes secret with the encoded secret value.
After that, it starts watching the Kubernetes secrets and any changes in the Kubernetes secret is synced to Fusion.
### Syncing from Kubernetes secret to Fusion
If the actual secret is updated in the Kubernetes secret, Config Sync syncs the updated secret to Fusion.
It’s important that Config Sync is up and running to sync the updated secret to Fusion.
### Syncing from Fusion to Kubernetes secret
If the encoded secret is updated in the Fusion, Config Sync syncs the updated secret to the Kubernetes secret.
# ConfigSync configuration reference
Source: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/settings
[old doc.lw link]: https//doc.lucidworks.com/fusion/5.9/ifv6st
[localhost link]: http://localhost:3000/docs/5/fusion/operations/config-sync/settings
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/config-sync/settings
This topic provides a complete reference to the ConfigSync configuration options.
```yaml wrap theme={"dark"}
fusion-config-sync:
# pub mode
springProfilesOverride: "kubernetes,jwt,fusion,pub"
# sub mode
#springProfilesOverride: "kubernetes,jwt,fusion,sub"
# pubsub mode
#springProfilesOverride: "kubernetes,jwt,fusion,pub,sub"
cfg:
sync:
apps: -cloud-platform # track all the apps except cloud-platform
# apps: "*" # track all the apps (default)
# apps: "app1,app2" # track only app1 and app2 apps
syncEvery: 30000 # this setting controls how often we're running pub/sub job, 30000 milliseconds is a default
# github connection properties, common for pub/sub modes
github:
repo: "https://github.com/mycompany/cloud-config-sync.git" # git repo, mandatory
branch: "rh/stg" # git branch, mandatory
dir: "/tmp/config-sync-repo" # repo dir, default "/tmp/config-sync-repo"
path: / # configs path in the repo dir, default "/"
username: "fusion-config-sync" # git user name, default "fusion-config-sync"
email: "fusion-config-sync@mycompany.com" # git user email, mandatory, default "fusion-config-sync@lucidworks.com"
secretPath: "/etc/secrets/github_oauth_token" # alternative location for github oauth token as a file
githubToken: "token" # alternative way to specify github oauth token explicitly
pub:
checkForBlobUpdatesEveryMs: 20000 # blob aren't stored in zk but in solr so this setting controls how often should we check blob updates from solr, 20000 is a default
checkForRuleUpdatesEveryMs: 300000 # rules aren't stored in zk but in solr so this setting controls how often should we check rule updates from solr, 300000 is a default
objects:
decode-secrets: false # shall ConfigSync not decode secrets like s3 private keys, JDBC passwords etc.
name-filter: ".*_lw_tmp.*" # filter tmp file names
# filter audit properties by json pointer
prop-filter: >
/created,
/modified,
/createdAt,
/updatedAt,
/updates
apps:
prop-filter: >
/properties/previousCollectionId
blobs:
name-filter: "^(quickstart/|prefs-).+"
collections:
name-filter: "^system_.+"
spark-jobs:
# data-migration jobs
name-filter: "^(import|export)_.+"
tasks:
# data-migration tasks
name-filter: "^export_finalizer_.+"
index-pipelines:
prop-filter: >
/properties/secretSourcePipelineId,
/stages/secretSourceStageId,
/stages/licensed
query-pipelines:
prop-filter: >
/properties/secretSourcePipelineId,
/stages/secretSourceStageId,
/stages/licensed
sub:
objects:
name-filter: ".*_lw_tmp.*" # filter tmp file names
# add filters and other settings for the subscriber mode here
```
# Kubernetes Deployment
Source: https://doc.lucidworks.com/docs/5/fusion/operations/deployment
[localhost link]: http://localhost:3000/docs/5/fusion/operations/deployment
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/deployment
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/142
The figure below provides a general overview on Fusion architecture and shows how Fusion services and data flow within a deployment.
The image above is for illustrative purposes only. It depicts services available in Fusion 5.9, but your actual implementation may differ.
## Architecture and requirements
* [Which Kubernetes?](/docs/5/fusion/operations/survival-guide/platforms)
* [Multiple Zones for High Availability](/docs/5/fusion/operations/survival-guide/high-availability)
* [Fusion Microservices](/docs/5/fusion/reference/microservices)
* [Ingress and Security](/docs/5/fusion/operations/survival-guide/ingress)
* [Stateless Sessions with JWT](/docs/5/fusion/operations/survival-guide/jwt)
* [Workload Isolation with Multiple Node Pools](/docs/5/fusion/operations/survival-guide/workload-isolation)
* [High-Performance Query Processing with Auto-Scaling](/docs/5/fusion/operations/survival-guide/auto-scaling)
The learning path for **Implementing Fusion** focuses on how to be successful when starting your implementation journey to avoid delays.
## Deploy Fusion on Kubernetes
* **Deploy Fusion 5 on Google Kubernetes Engine (GKE)**
* **Deploy Fusion 5 on Amazon Elastic Kubernetes Service (EKS)**
* **Deploy Fusion 5 on Azure Kubernetes Service (AKS)**
* **Deploy Fusion 5 on Other Kubernetes Platforms**
* **Deploy Fusion at Scale**
Fusion supports deployment on Google Kubernetes Engine (GKE). This topic explains how to deploy a Fusion cluster on GKE using the [`setup_f5_gke.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_gke.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository.
## Prerequisites
This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes.
### Release Name and Namespace
Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) Kubernetes namespace to install Fusion into.
Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*.
However, please *do not* install more than one Fusion release in the same namespace.
All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_
Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default.
### Install Helm
Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster.
Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform.
On MacOS, you can do:
```bash theme={"dark"}
brew install kubernetes-helm
```
If you already have helm installed, make sure you're using the latest version:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/)
The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`.
#### Helm User Permissions
If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory.
When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.:
```bash theme={"dark"}
alias k=kubectl
```
To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into:
```bash theme={"dark"}
k create namespace fusion-namespace
```
Apply the `role.yaml` and `cluster-role.yaml` files to that namespace
```bash theme={"dark"}
k apply -f cluster-role.yaml
k config set-context --current --namespace=$NAMESPACE
k apply -f role.yaml
```
Then bind the rolebinding and clusterolebinding to the install user:
```bash wrap theme={"dark"}
k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user
k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user
```
You will then be able to run the `helm install` command as the ``
### Clone fusion-cloud-native from GitHub
You should clone this repo from github as you'll need to run the scripts on your local workstation:
```bash theme={"dark"}
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts.
```bash theme={"dark"}
cd fusion-cloud-native
git pull
```
Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip).
Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory.
## Prerequisites
This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes.
### Release Name and Namespace
Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) Kubernetes namespace to install Fusion into.
Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*.
However, please *do not* install more than one Fusion release in the same namespace.
All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_
Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default.
### Install Helm
Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster.
Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform.
On MacOS, you can do:
```bash theme={"dark"}
brew install kubernetes-helm
```
If you already have helm installed, make sure you're using the latest version:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/)
The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`.
#### Helm User Permissions
If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory.
When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.:
```bash theme={"dark"}
alias k=kubectl
```
To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into:
```bash theme={"dark"}
k create namespace fusion-namespace
```
Apply the `role.yaml` and `cluster-role.yaml` files to that namespace
```bash theme={"dark"}
k apply -f cluster-role.yaml
k config set-context --current --namespace=$NAMESPACE
k apply -f role.yaml
```
Then bind the rolebinding and clusterolebinding to the install user:
```bash wrap theme={"dark"}
k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user
k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user
```
You will then be able to run the `helm install` command as the ``
### Clone fusion-cloud-native from GitHub
You should clone this repo from github as you'll need to run the scripts on your local workstation:
```bash theme={"dark"}
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts.
```bash theme={"dark"}
cd fusion-cloud-native
git pull
```
Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip).
Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory.
The [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_gke.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_gke.sh) `setup_f5_gke.sh` script provided in this repo is strictly optional.
The script is mainly to help those new to Kubernetes and/or Fusion get started quickly.
If you're already familiar with K8s, Helm, and GKE, then you can skip the script and just use Helm directly to install Fusion into an existing cluster or one you create yourself using the process described `<>`.
### Set up the Google Cloud SDK (one time only)
If you've already installed the `gcloud` command-line tools, you can skip to `<>`.
These steps set up your local Google Cloud SDK environment so that you're ready to use the command-line tools to manage your Fusion deployment.
Usually, you only need to perform these setup steps once. After that, you're ready to [create a cluster](#create-a-single-node-demo-cluster).
For a nice getting started tutorial for GKE, see: [https://cloud.google.com/kubernetes-engine/docs/deploy-app-cluster](https://cloud.google.com/kubernetes-engine/docs/deploy-app-cluster)
How to set up the Google Cloud SDK:
1. [https://console.cloud.google.com/apis/library/container.googleapis.com?q=kubernetes%20engine](https://console.cloud.google.com/apis/library/container.googleapis.com?q=kubernetes%20engine)
2. Log in to Google Cloud: `gcloud auth login`
3. Set up the Google Cloud SDK:
1. `gcloud config set compute/zone `
If you are working with regional clusters instead of zone clusters, use `gcloud config set compute/region ` instead.
2. `gcloud config set core/account `
3. *New GKE projects only:* `gcloud projects create `
If you have already created a project, for example in [https://console.cloud.google.com/](https://console.cloud.google.com/), then skip to the next step.
4. `gcloud config set project `
Make sure you install the Kubernetes command-line tool `kubectl` using:
```bash theme={"dark"}
gcloud components install kubectl
gcloud components update
```
### Create a single-node demo cluster
Run the [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_gke.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_gke.sh) `setup_f5_gke.sh` script to install Fusion 5.x in a GKE cluster. To create a new, single-node *demo* cluster and install Fusion, simply do:
```bash theme={"dark"}
./setup_f5_gke.sh -c -p --create demo
```
Use the `--help` option to see script usage. If you want the script to create a cluster for you, then you need to pass the `--create` option with either `demo` or `multi_az`. If you don't want the script to create a cluster, then you need to create a cluster before running the script and simply pass the name of the existing cluster using the `-c` parameter.
If you pass `--create demo` to the script, then we create a single node GKE cluster (defaults to using `n1-standard-8` node type). The minimum node type you'll need for a 1 node cluster is an `n1-standard-8` (on GKE) which has 8 CPU and 30 GB of memory. This is cutting it very close in terms of resources as you also need to host all of the Kubernetes system pods on this same node. Obviously, this works for kicking the tires on Fusion 5.1 but is not sufficient for production workloads.
You can change the instance type using the `-i` parameter; see: [https://cloud.google.com/compute/docs/regions-zones/#available](https://cloud.google.com/compute/docs/regions-zones/#available) for a list of which machine types are available in your desired region.
If not provided the script generates a custom values file named `gke___fusion_values.yaml` which you can use to customize the Fusion chart.\_\_
\#WARNING# If using Helm V2, the `setup_f5_gke.sh` script installs Helm's `tiller` component into your GKE cluster with the cluster admin role. If you don't want this, then please upgrade to Helm v3.
If you see an error similar to the following, then wait a few seconds and try running the `setup_f5_gke.sh` script again with the same arguments as this is usually a transient issue:
```bash wrap theme={"dark"}
Error: could not get apiVersions from Kubernetes: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
```
After running the `setup_f5_gke.sh` script, proceed to the `<>` section below.
When you're ready to deploy Fusion to a production-like environment, see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
### Create a three-node regional cluster to withstand a zone outage
With a three-node regional cluster, nodes are deployed across three separate availability zones.
```bash wrap theme={"dark"}
./setup_f5_gke.sh -c -p -n --region --create multi_az
```
* `` value should be the name of a non-existent cluster; the script will create the new cluster.
* `` must match the name of an existing project in GKE. Run `gcloud config get-value project` to get this value, or see the [GKE setup instructions](#set-up-the-google-cloud-sdk-one-time-only).
* `` Kubernetes namespace to install Fusion into, defaults to `default` with release `f5`
* `` value should be the name of a GKE region, defaults to `us-west1`. Run `gcloud config get-value compute/zone` to get this value, or see the [GKE setup instructions](#set-up-the-google-cloud-sdk-one-time-only) to set the value.
In this configuration, Kubernetes deploys a ZooKeeper and Solr pod on each of the three nodes, which allows the cluster to retain ZK quorum and remain operational after losing one node, such as during an outage in one availability zone.
When running in a multi-zone cluster, each Solr node has the `solr_zone` system property set to the zone it is running in, such as `-Dsolr_zone=us-west1-a`.
After running the `setup_f5_gke.sh` script, proceed to the `<>` section below.
When you're ready to deploy Fusion to a production-like environment, see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
#### GKE Ingress and TLS
The Fusion proxy service provides authentication and serves as an API gateway for accessing all other Fusion services.
It's typical to use an Ingress for TLS termination in front of the proxy service.
The `setup_f5_gke.sh` supports creating an Ingress with a TLS cert for a domain you own by passing: `-t -h `
After the script runs, you need to create an A record in GCP's DNS service to map your domain name to the Ingress IP. Once this occurs, our script setup uses [https://letsencrypt.org/](https://letsencrypt.org/) to issue a TLS cert for your Ingress.
To see the status of the Let's Encrypt issued certificate, do:
```bash theme={"dark"}
kubectl get managedcertificates -n -o yaml
```
Please refer to the Kubernetes documentation on configuring an Ingress for GKE: [https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer](https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer)
The GCP Ingress defaults to a 30 second timeout, which can lead to false negatives for long running requests such as importing apps. To configure the timeout for the backend in kubernetes:
Create a BackendConfig object in your namespace:
```yaml theme={"dark"}
---
apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
name: backend_config_name
spec:
timeoutSec: 120
connectionDraining:
drainingTimeoutSec: 60
```
Then make sure that the following entries are in the right place in your values.yaml file:
```yaml theme={"dark"}
api-gateway:
service:
annotations:
beta.cloud.google.com/backend-config: '{"ports": {"6764":"backend_config_name"}}'
```
and upgrade your release to apply the configuration changes
#### Ingresses and externalTrafficPolicy
When running a fusion cluster behind an externally controlled LoadBalancer it can be advantageous
to configure the `externalTrafficPolicy` of the `proxy` service to `Local`. This preserves the client
source IP and avoids a second hop for LoadBalancer and NodePort type services, but risks potentially
imbalanced traffic spreading. Although when running in a cluster with a dedicated pool for spark jobs
that can scale up and down freely it can prevent unwanted request failures. This behaviour can be
altered with the `api-gateway.service.externalTrafficPolicy` value, which is set to `Local` if the example values
file is used.
**You must use `externalTrafficPolicy`=`Local` for the Trusted HTTP Realm to work correctly.**
If you are already using a custom 'values.yaml' file, create an entry for `externalTrafficPolicy` under `api-gateway` service.
```yaml theme={"dark"}
api-gateway:
service:
externalTrafficPolicy: Local
```
#### Considerations when using the nginx ingress controller
If you are using the `nginx` ingress controller to fulfil your ingress definitions there are a couple
of options that are recommended to be set in the configmap:
```bash wrap theme={"dark"}
enable-underscores-in-headers: "true" # Fusion can return some headers that have underscores, these have to be explicitly enabled in nginx
proxy-body-size: "0" # By default nginx places a maximum size on request bodies, either increase as needed or disable by setting to 0
proxy-read-timeout: "300" # Increases the timeout for potential slow queries.
```
#### Custom values
There are some example values files that can be used as a starting point for
resources, affinity and replica count configuration in the `example-values` folder.
These can be passed to the install script using the `--values` option, for example:
```bash wrap theme={"dark"}
./setup_f5_gke.sh -c -p -r -n \
--values example-values/affinity.yaml --values example-values/resources.yaml --values example-values/replicas.yaml
```
The `--values` option can be passed multiple times, if the same configuration property is contained within multiple `values` files then the values from the latest file passed as a `--values` option are used.
##### Connectors custom values
If you are using Fusion 5.9 or later, you can specify resources and replica count per connector. This allows you to set different resource limits for each connector. If you do not set custom values for a connector, that connector uses the default values.
Set each connector's resource values in the `connector-plugin` section under `pluginValues`. The `pluginValues` section is a list of plugins and its resources. The following sample shows an example.
```yaml theme={"dark"}
pluginValues:
- id: "plugin-id" <1>
resources: <2>
limits:
cpu: "2"
memory: "3Gi"
requests:
cpu: "250m"
memory: "2Gi"
replicaCount: 1 <3>
```
* `<1>` The plugin ID. The plugin ID must match the plugin ID on the plugin ZIP file. without the `lucidworks.` prefix. For example, if the plugin ID on the plugin ZIP file is `lucidworks.sharepoint-optimized`, the plugin ID is `sharepoint-optimized`.
* `<2>` The resources settings. You may specify the limits, the requests, and the CPU and memory for each.
* `<3>` The number of replicas per connector. This value is 1 by default.
After editing the `connector-plugin` section, you must reinstall the affected connector.
#### Upgrades and Ingress
If you used the `-t -h ` options when installing your cluster, our script created an additional values yaml file named `tls-values.yaml`.
To make things easier for you when upgrading, you should add the settings from this file into your main custom values yaml file, e.g.:
```yaml theme={"dark"}
api-gateway:
service:
type: "NodePort"
ingress:
enabled: true
host: ""
tls:
enabled: true
annotations:
"networking.gke.io/managed-certificates": "-managed-certificate"
"kubernetes.io/ingress.class": "gce"
```
This way you don't have to remember to pass the additional `tls-values.yaml` file when upgrading.
## Verifying the Fusion Installation
In this section, we provide some tips on how to verify the Fusion installation.
Check if the Fusion Admin UI is available at `\https://:6764/admin/`.
Let's review some useful kubectl commands.
### Enhance the K8s Command-line Experience
Here is a list of tools we found useful for improving your command-line experience with Kubernetes:
* krew (kubectl plugin mgr): [https://github.com/kubernetes-sigs/krew/](https://github.com/kubernetes-sigs/krew/)
* kube-ps1 (show current context on command line prompt): [https://github.com/jonmosco/kube-ps1](https://github.com/jonmosco/kube-ps1)
* kubectx / kubens (switch between clusters / namespaces): [https://github.com/ahmetb/kubectx](https://github.com/ahmetb/kubectx)
### Useful kubectl commands
kubectl reference: [https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands)
Set the namespace for `kubectl` if not using the default:
```bash theme={"dark"}
kubectl config set-context --current --namespace=
```
**This saves you from having to pass `-n` with every command.**
Get a list of running pods: `k get pods`
Get logs for a pod using a label: `k logs –l app.kubernetes.io/component=query-pipeline`
Get pod deployment spec and details: `k get pods -o yaml`
Get details about a pod events: `k describe po `
Port forward to a specific pod: `k port-forward 8983:8983`
SSH into a pod: `k exec -it -- /bin/bash`
CPU/Memory usage report for pods: `k top pods`
Forcefully kill a pod: `k delete po --force --grace-period 0`
Scale up (or down) a deployment: `k scale deployment.v1.apps/ --replicas=N`
Get a list of pod versions: `k get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq`
### Check Fusion Pods and Services
Once the install script completes, you can check that all pods and services are available using:
```bash theme={"dark"}
kubectl get pods
```
If all goes well, you should see a list of pods similar to:
```bash wrap theme={"dark"}
NAME READY STATUS RESTARTS AGE
seldon-controller-manager-6675874894-qxwrv 1/1 Running 0 8m45s
f5-admin-ui-74d794f4f8-m5jms 1/1 Running 0 8m45s
f5-ambassador-fd6b9b5dc-7ghf6 1/1 Running 0 8m43s
f5-api-gateway-6b9998b9c-tmchk 1/1 Running 0 8m45s
f5-auth-ui-7565564b4c-rdc74 1/1 Running 0 8m42s
f5-classic-rest-service-0 1/1 Running 3 8m44s
f5-devops-ui-77bb867ffb-fbzxd 1/1 Running 0 8m42s
f5-fusion-admin-78b8f8fc7f-4d7l8 1/1 Running 0 8m42s
f5-fusion-indexing-599c8d448-xzsvm 1/1 Running 0 8m44s
f5-insights-665fd9f6fc-g5psw 1/1 Running 0 8m43s
f5-job-launcher-84dd4c5c96-p8528 1/1 Running 0 8m44s
f5-job-rest-server-6d44d964b8-xtnxw 1/1 Running 0 8m45s
f5-logstash-0 1/1 Running 0 8m45s
f5-ml-model-service-6987dc94c9-9ppp8 2/2 Running 1 8m45s
f5-monitoring-grafana-5d499dbb58-pzw72 1/1 Running 0 10m
f5-monitoring-prometheus-kube-state-metrics-54d6678dv9h7h 1/1 Running 0 10m
f5-monitoring-prometheus-pushgateway-7d65c65b85-vwrwf 1/1 Running 0 10m
f5-monitoring-prometheus-server-0 2/2 Running 0 10m
f5-pm-ui-86cbc5bb65-nd2n8 1/1 Running 0 8m44s
f5-pulsar-bookkeeper-0 1/1 Running 0 8m45s
f5-pulsar-broker-b56cc776f-56msx 1/1 Running 0 8m45s
f5-query-pipeline-5d75d7d5f4-l2mdf 1/1 Running 0 8m43s
f5-connectors-7bb6cfc65f-7wfs2 1/1 Running 0 8m42s
f5-connectors-backend-987fdc648-dldwv 1/1 Running 0 8m45s
f5-rules-ui-6b9d55b78f-9hzzj 1/1 Running 0 8m43s
f5-solr-0 1/1 Running 0 8m44s
f5-solr-exporter-c4687c785-jsm7x 1/1 Running 0 8m45s
f5-ui-6cdbcc68c6-rj9cq 1/1 Running 0 8m45s
f5-webapps-6d6bb9bfd-hm4qx 1/1 Running 0 8m45s
f5-workflow-controller-7b66679fb7-sjbvp 1/1 Running 0 8m44s
f5-zookeeper-0 1/1 Running 0 8m45s
```
The number of pods per deployment / statefulset will vary based on your cluster size and replicaCount settings in your custom values YAML file.
Also, don't worry if you see some pods having been restarted as that just means they were too slow to come up and Kubernetes killed and restarted them.
You do want to see at least one pod running for every service. If a pod is not running after waiting a sufficient amount of time,
use `kubectl logs ` to see the logs for that pod; to see the logs for previous versions of a pod, use: `kubectl logs -p`.
You can also look at the actions Kubernetes performed on the pod using `kubectl describe po `.
To see a list of Fusion services, do:
```bash theme={"dark"}
kubectl get svc
```
For an overview of the various Fusion 5 microservices, see: [Fusion microservices](/docs/5/fusion/reference/microservices).
Once you're ready to build a Fusion cluster for production, please see see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
### Upgrading with Zero Downtime
One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. Fusion 5 allows customers to upgrade from Fusion 5.x.y to a later 5.x.z version on a live cluster with zero downtime or disruption of service.
When Kubernetes performs a rolling update to an individual microservice, there will be a mix of old and new services in the cluster concurrently (only briefly in most cases) and requests from other services will be routed to both versions. Consequently, Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same 5.x line of releases. We also ensure stored configuration remains compatible in the same 5.x release line.
Lucidworks releases minor updates to individual services frequently, so our customers can pull in those upgrades using Helm at their discretion.
To upgrade your cluster at any time, use the `--upgrade` option with our setup scripts in this repo.
The scripts in this repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.
To see what would be upgraded, you can pass the `--dry-run` option to the script.
### Grafana Dashboards
Get the initial Grafana password from a K8s secret by doing:
```bash wrap theme={"dark"}
kubectl get secret --namespace "${NAMESPACE}" ${RELEASE}-monitoring-grafana \
-o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```
With Grafana, you can either setup a temporary port-forward to a Grafana pod or expose Grafana on an external IP using a K8s LoadBalancer.
To define a LoadBalancer, do (replace \$ with your Helm release label):
```bash wrap theme={"dark"}
kubectl expose deployment ${RELEASE}-monitoring-grafana --type=LoadBalancer --name=grafana --port=3000 --target-port=3000
```
You can use `kubectl get services --namespace ` to determine when the load balancer is setup and its IP address. Direct your browser to `http://:3000` and enter the username `admin@localhost` and the password that was returned in the previous step.
This will log you into the application. It is recommended that you create another administrative user with a more desirable password.
The dashboards and datasoure will be setup for you in grafana, simply navigate to `Dashboards` -> `Manage` to view the available dashboards
## Learn more
The course for **Deploying Fusion 5** focuses on the prerequisite software needed to deploy Fusion, the necessary setup steps, and the physical act of deployment.
Fusion supports deployment on Amazon Elastic Kubernetes Service (EKS). This topic explains how to deploy a Fusion cluster on EKS using the [`setup_f5_eks.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_eks.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository.
In addition, this topic provides information about how to [configure IAM roles for the service account](#configure-iam-roles-for-the-service-account).
## Prerequisites
This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes.
### Release Name and Namespace
Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) Kubernetes namespace to install Fusion into.
Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*.
However, please *do not* install more than one Fusion release in the same namespace.
All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_
Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default.
### Install Helm
Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster.
Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform.
On MacOS, you can do:
```bash theme={"dark"}
brew install kubernetes-helm
```
If you already have helm installed, make sure you're using the latest version:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/)
The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`.
#### Helm User Permissions
If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory.
When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.:
```bash theme={"dark"}
alias k=kubectl
```
To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into:
```bash theme={"dark"}
k create namespace fusion-namespace
```
Apply the `role.yaml` and `cluster-role.yaml` files to that namespace
```bash theme={"dark"}
k apply -f cluster-role.yaml
k config set-context --current --namespace=$NAMESPACE
k apply -f role.yaml
```
Then bind the rolebinding and clusterolebinding to the install user:
```bash wrap theme={"dark"}
k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user
k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user
```
You will then be able to run the `helm install` command as the ``
### Clone fusion-cloud-native from GitHub
You should clone this repo from github as you'll need to run the scripts on your local workstation:
```bash theme={"dark"}
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts.
```bash theme={"dark"}
cd fusion-cloud-native
git pull
```
Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip).
Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory.
The [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_eks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_eks.sh) `setup_f5_eks.sh` script provided in this repo is strictly optional.
The script is mainly to help those new to Kubernetes and/or Fusion get started quickly.
If you're already familiar with K8s, Helm, and EKS, then you use Helm directly to install Fusion into an existing cluster or one you create yourself using the process described `<>`.
If you're new to Amazon Web Services (AWS), then please visit the Amazon Web Services [https://aws.amazon.com/getting-started/](https://aws.amazon.com/getting-started/) to set up an account.
If you're new to Kubernetes and EKS, then we recommend going through Amazon's [https://eksworkshop.com/introduction/](https://eksworkshop.com/introduction/) before proceeding with Fusion.
### Set up the AWS CLI tools
Before launching an EKS cluster, you need to install and configure `kubectl`, `aws`, `eksctl`, `aws-iam-authenticator` using the links provided below:
Required AWS Command-line Tools:
* kubectl: [https://kubernetes.io/docs/tasks/tools/install-kubectl/](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
* aws: [https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html)
* eksctl: [https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html)
* aws-iam-authenticator: [https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html](https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html)
Run `aws configure` to configure a profile for authenticating to AWS. You'll use the profile name you configure in this step, which defaults to `default`, as the `-p` argument to the `setup_f5_eks.sh` script in the next section.
When working in Ubuntu, avoid using the eksctl snap version. Alternative sources can have different versions that could cause command failures. Also, always make sure you are using the latest version for each one of the required tools.
### Set up Fusion on EKS
To create a cluster in EKS the following IAM policies are required:
* AmazonEC2FullAccess
* AWSCloudFormationFullAccess
EKS Permissions:
```bash theme={"dark"}
eks:CreateCluster
eks:DeleteCluster
eks:DescribeCluster
eks:DescribeUpdate
eks:ListClusters
eks:ListUpdates
eks:UpdateClusterVersion
```
VPC Permissions:
```bash theme={"dark"}
ec2:AssociateSubnetCidrBlock
ec2:AssociateVpcCidrBlock
ec2:AttachInternetGateway
ec2:CreateInternetGateway
ec2:CreateSubnet
ec2:CreateVpc
ec2:CreateVpcEndpoint
ec2:DeleteInternetGateway
ec2:DeleteSubnet
ec2:DeleteVpc
ec2:DeleteVpcEndpoints
ec2:DescribeSubnets
ec2:DescribeVpcAttribute
ec2:DescribeVpcs
ec2:DetachInternetGateway
ec2:DisassociateSubnetCidrBlock
ec2:DisassociateVpcCidrBlock
ec2:ModifySubnetAttribute
ec2:ModifyVpcAttribute
ec2:ModifyVpcEndpoint
```
.IAM Permissions
```bash theme={"dark"}
iam:AddRoleToInstanceProfile
iam:AttachRolePolicy
iam:CreateInstanceProfile
iam:CreatePolicy
iam:CreatePolicyVersion
iam:CreateRole
iam:DeleteInstanceProfile
iam:DeletePolicy
iam:DeletePolicyVersion
iam:DeleteRole
iam:DeleteRolePolicy
iam:DetachRolePolicy
iam:GetInstanceProfile
iam:GetPolicy
iam:GetPolicyVersion
iam:GetRole
iam:GetRolePolicy
iam:ListInstanceProfiles
iam:ListInstanceProfilesForRole
iam:PassRole
iam:PutRolePolicy
iam:RemoveRoleFromInstanceProfile
iam:TagRole
iam:UntagRole
```
Download and run the [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_eks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_eks.sh) `setup_f5_eks.sh` script to install Fusion 5.x in an EKS cluster.
This script does not support multiple node pools and should not be used for production clusters.
* To create a new cluster and install Fusion, run the following command:
```
./setup_f5_eks.sh -c my-eks-cluster -p profile-name -n fusion-namespace --create demo
```
* Replace `my-eks-cluster`, `profile-name`, and `fusion-namespace` with your cluster, profile, and namespace values.
* Pass the `--create` option with either `demo` or `multi_az`.
* To use an existing cluster and install Fusion, run the following command:
```
./setup_f5_eks.sh -c cluster-name -p profile-name
```
* Replace `cluster-name` with the name of the cluster you already created.
* Replace `profile-name` with the name of your profile.
The profile is automatically set to `default` if you ran the AWS configure command without giving the profile a name.
Use the `--help` option to see full script usage.
If using Helm V2, the `setup_f5_eks.sh` script installs Helm's `tiller` component into your EKS cluster with the cluster admin role. If you don't want this, then please upgrade to Helm v3.
The `setup_f5_eks.sh` script creates a service account that provides S3 read-only permissions to the created pods.
After running the `setup_f5_eks.sh` script, proceed to the `<>` section below.
#### EKS cluster overview
The EKS cluster is created using `eksctl` ([https://eksctl.io/](https://eksctl.io/)). By default it will setup the following resources in your AWS account:
* A dedicated VPC for the EKS cluster in the specified region with CIDR: `192.168.0.0/16`
* 3 Public and 3 Private subnets within the created VPC, each with a `/19` CIDR range, along with the corresponding route tables.
* A NAT gateway in each Public subnet
* An Auto Scaling Group of the instance type specified by the script, which defaults to `m5.2xlarge`, with 3 instances spanning the public subnets.
See [https://eksctl.io/usage/vpc-networking/](https://eksctl.io/usage/vpc-networking/) for more information on the networking setup.
#### EKS Ingress
The `setup_f5_eks.sh` script exposes the Fusion proxy service on an external DNS name provided by an ELB over HTTP. This is done for demo or getting started purposes. However, you're strongly encouraged to configure a K8s Ingress with TLS termination in front of the proxy service.
See: [https://aws.amazon.com/premiumsupport/knowledge-center/terminate-https-traffic-eks-acm/](https://aws.amazon.com/premiumsupport/knowledge-center/terminate-https-traffic-eks-acm/)
Our EKS script creates a classic ELB for exposing fusion proxy service. In case you need to change this behavior and use [https://github.com/kubernetes-sigs/aws-load-balancer-controller](https://github.com/kubernetes-sigs/aws-load-balancer-controller) instead you can use the following parameters when running the `setup_f5_eks.sh` script:
```bash theme={"dark"}
--deploy-alb # Tells the script to deploy an ALB
```
By default the `kube-system` namespace is being used for installing the `aws-load-balancer-controller` because pods `priorityClassName` is set to `system-cluster-critical`.
In case you need to deploy an internal ALB you can use the `--internal-alb` option. This will create the nodes in the internal subnets. Fusion will be reachable from an AWS instance located in any of the external subnets on the same VPC. To use an ALB also an ingress with a DNS name is required, you can use the `-h` option to create an ingress with the required DNS name.
Finally, use Route 53 or your DNS provider for creating an A ALIAS DNS record for your DNS name pointing to the ingress ADRESS. You can get the address listing the ingress using the command `kubectl get ing`.
## Verifying the Fusion Installation
In this section, we provide some tips on how to verify the Fusion installation.
Check if the Fusion Admin UI is available at `\https://:6764/admin/`.
Let's review some useful kubectl commands.
### Enhance the K8s Command-line Experience
Here is a list of tools we found useful for improving your command-line experience with Kubernetes:
* krew (kubectl plugin mgr): [https://github.com/kubernetes-sigs/krew/](https://github.com/kubernetes-sigs/krew/)
* kube-ps1 (show current context on command line prompt): [https://github.com/jonmosco/kube-ps1](https://github.com/jonmosco/kube-ps1)
* kubectx / kubens (switch between clusters / namespaces): [https://github.com/ahmetb/kubectx](https://github.com/ahmetb/kubectx)
### Useful kubectl commands
kubectl reference: [https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands)
Set the namespace for `kubectl` if not using the default:
```bash theme={"dark"}
kubectl config set-context --current --namespace=
```
**This saves you from having to pass `-n` with every command.**
Get a list of running pods: `k get pods`
Get logs for a pod using a label: `k logs –l app.kubernetes.io/component=query-pipeline`
Get pod deployment spec and details: `k get pods -o yaml`
Get details about a pod events: `k describe po `
Port forward to a specific pod: `k port-forward 8983:8983`
SSH into a pod: `k exec -it -- /bin/bash`
CPU/Memory usage report for pods: `k top pods`
Forcefully kill a pod: `k delete po --force --grace-period 0`
Scale up (or down) a deployment: `k scale deployment.v1.apps/ --replicas=N`
Get a list of pod versions: `k get po -o jsonpath='{..image}'
tr -s '[[:space:]]' '\n'
sort
uniq`
### Check Fusion Pods and Services
Once the install script completes, you can check that all pods and services are available using:
```bash theme={"dark"}
kubectl get pods
```
If all goes well, you should see a list of pods similar to:
```bash wrap theme={"dark"}
NAME READY STATUS RESTARTS AGE
seldon-controller-manager-6675874894-qxwrv 1/1 Running 0 8m45s
f5-admin-ui-74d794f4f8-m5jms 1/1 Running 0 8m45s
f5-ambassador-fd6b9b5dc-7ghf6 1/1 Running 0 8m43s
f5-api-gateway-6b9998b9c-tmchk 1/1 Running 0 8m45s
f5-auth-ui-7565564b4c-rdc74 1/1 Running 0 8m42s
f5-classic-rest-service-0 1/1 Running 3 8m44s
f5-devops-ui-77bb867ffb-fbzxd 1/1 Running 0 8m42s
f5-fusion-admin-78b8f8fc7f-4d7l8 1/1 Running 0 8m42s
f5-fusion-indexing-599c8d448-xzsvm 1/1 Running 0 8m44s
f5-insights-665fd9f6fc-g5psw 1/1 Running 0 8m43s
f5-job-launcher-84dd4c5c96-p8528 1/1 Running 0 8m44s
f5-job-rest-server-6d44d964b8-xtnxw 1/1 Running 0 8m45s
f5-logstash-0 1/1 Running 0 8m45s
f5-ml-model-service-6987dc94c9-9ppp8 2/2 Running 1 8m45s
f5-monitoring-grafana-5d499dbb58-pzw72 1/1 Running 0 10m
f5-monitoring-prometheus-kube-state-metrics-54d6678dv9h7h 1/1 Running 0 10m
f5-monitoring-prometheus-pushgateway-7d65c65b85-vwrwf 1/1 Running 0 10m
f5-monitoring-prometheus-server-0 2/2 Running 0 10m
f5-pm-ui-86cbc5bb65-nd2n8 1/1 Running 0 8m44s
f5-pulsar-bookkeeper-0 1/1 Running 0 8m45s
f5-pulsar-broker-b56cc776f-56msx 1/1 Running 0 8m45s
f5-query-pipeline-5d75d7d5f4-l2mdf 1/1 Running 0 8m43s
f5-connectors-7bb6cfc65f-7wfs2 1/1 Running 0 8m42s
f5-connectors-backend-987fdc648-dldwv 1/1 Running 0 8m45s
f5-rules-ui-6b9d55b78f-9hzzj 1/1 Running 0 8m43s
f5-solr-0 1/1 Running 0 8m44s
f5-solr-exporter-c4687c785-jsm7x 1/1 Running 0 8m45s
f5-ui-6cdbcc68c6-rj9cq 1/1 Running 0 8m45s
f5-webapps-6d6bb9bfd-hm4qx 1/1 Running 0 8m45s
f5-workflow-controller-7b66679fb7-sjbvp 1/1 Running 0 8m44s
f5-zookeeper-0 1/1 Running 0 8m45s
```
The number of pods per deployment / statefulset will vary based on your cluster size and replicaCount settings in your custom values YAML file.
Also, don't worry if you see some pods having been restarted as that just means they were too slow to come up and Kubernetes killed and restarted them.
You do want to see at least one pod running for every service. If a pod is not running after waiting a sufficient amount of time,
use `kubectl logs ` to see the logs for that pod; to see the logs for previous versions of a pod, use: `kubectl logs -p`.
You can also look at the actions Kubernetes performed on the pod using `kubectl describe po `.
To see a list of Fusion services, do:
```bash theme={"dark"}
kubectl get svc
```
For an overview of the various Fusion 5 microservices, see: [Fusion microservices](/docs/5/fusion/reference/microservices).
Once you're ready to build a Fusion cluster for production, please see see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
### Upgrading with Zero Downtime
One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. Fusion 5 allows customers to upgrade from Fusion 5.x.y to a later 5.x.z version on a live cluster with zero downtime or disruption of service.
When Kubernetes performs a rolling update to an individual microservice, there will be a mix of old and new services in the cluster concurrently (only briefly in most cases) and requests from other services will be routed to both versions. Consequently, Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same 5.x line of releases. We also ensure stored configuration remains compatible in the same 5.x release line.
Lucidworks releases minor updates to individual services frequently, so our customers can pull in those upgrades using Helm at their discretion.
To upgrade your cluster at any time, use the `--upgrade` option with our setup scripts in this repo.
The scripts in this repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.
To see what would be upgraded, you can pass the `--dry-run` option to the script.
### Grafana Dashboards
Get the initial Grafana password from a K8s secret by doing:
```bash wrap theme={"dark"}
kubectl get secret --namespace "${NAMESPACE}" ${RELEASE}-monitoring-grafana \
-o jsonpath="{.data.admin-password}"
base64 --decode ; echo
```
With Grafana, you can either setup a temporary port-forward to a Grafana pod or expose Grafana on an external IP using a K8s LoadBalancer.
To define a LoadBalancer, do (replace \$ with your Helm release label):
```bash wrap theme={"dark"}
kubectl expose deployment ${RELEASE}-monitoring-grafana --type=LoadBalancer --name=grafana --port=3000 --target-port=3000
```
You can use `kubectl get services --namespace ` to determine when the load balancer is setup and its IP address. Direct your browser to `http://:3000` and enter the username `admin@localhost` and the password that was returned in the previous step.
This will log you into the application. It is recommended that you create another administrative user with a more desirable password.
The dashboards and datasoure will be setup for you in grafana, simply navigate to `Dashboards` -> `Manage` to view the available dashboards
## Configure IAM roles for the service account
Configuring IAM roles lets you utilize the Amazon Web Services Security Token Service (AWS STS) for short-term authentication credentials to access services like the Amazon S3 simple storage service.
To configure IAM roles, your user account must be granted `admin` permissions or `IAM:FullAccess`. Complete the following steps:
1. To create the OpenID Connect (OIDC) provider, run the following command:
```bash wrap theme={"dark"}
eksctl utils associate-iam-oidc-provider --cluster cluster_name --approve
```
2. To create an IAM role for the service account associated with the plugin-pod, run the following command:
```bash wrap theme={"dark"}
eksctl create iamserviceaccount --name f5-connector-plugin --namespace default --cluster cluster_name --attach-policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess --approve --override-existing-serviceaccounts
```
This command:
* Creates an IAM role and attaches the target policy.
* Updates the existing Kubernetes `f5-connector-plugin` service account and annotates it with the IAM role.
* Uses the existing `policy/AmazonS3ReadOnlyAccess` policy.
If the IAM role was already created without the command, and you want to associate the service account, run the following command:
```bash wrap theme={"dark"}
kubectl annotate serviceaccount -n default f5-connector-plugin eks.amazonaws.com/role-arn=arn:aws:iam::411271863668:role/FUS_ROLE --overwrite=true
```
To utilize this feature, create a data source with the settings in **S3 Authentication Settings > AWS Instance Credentials Authentication Settings**. For detailed installation information, see [AWS S3 Pro connector](/docs/fusion-connectors/connectors/amazon-aws-s3-pro).
For more information, see:
* [Welcome to the AWS Security Token Service API Reference](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html)
* [Temporary security credentials in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html)
* [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html)
* [What is Amazon EKS?](https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html)
* [Fine-grained IAM roles for service accounts](https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts)
## Learn more
The course for **Deploying Fusion 5** focuses on the prerequisite software needed to deploy Fusion, the necessary setup steps, and the physical act of deployment.
Fusion supports deployment on Azure Kubernetes Service (AKS). This topic explains how to deploy a Fusion cluster on AKS using the [`setup_f5_aks.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_aks.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository.
The `setup_f5_aks.sh` script is the basic foundation for getting started and proof-of-concept purposes. For information about custom values in a production-ready environment, see Custom values YAML file.
## Prerequisites
This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes.
### Release Name and Namespace
Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) namespace to install Fusion into.
Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*.
However, please *do not* install more than one Fusion release in the same namespace.
All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_
Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default.
### Install Helm
Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster.
Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform.
On MacOS, you can do:
```bash theme={"dark"}
brew install kubernetes-helm
```
If you already have helm installed, make sure you're using the latest version:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/)
The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`.
#### Helm User Permissions
If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory.
When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.:
```bash theme={"dark"}
alias k=kubectl
```
To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into:
```bash theme={"dark"}
k create namespace fusion-namespace
```
Apply the `role.yaml` and `cluster-role.yaml` files to that namespace
```bash theme={"dark"}
k apply -f cluster-role.yaml
k config set-context --current --namespace=$NAMESPACE
k apply -f role.yaml
```
Then bind the rolebinding and clusterolebinding to the install user:
```bash wrap theme={"dark"}
k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user
k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user
```
You will then be able to run the `helm install` command as the ``
### Clone fusion-cloud-native from GitHub
You should clone this repo from github as you'll need to run the scripts on your local workstation:
```bash theme={"dark"}
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts.
```bash theme={"dark"}
cd fusion-cloud-native
git pull
```
Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip).
Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory.
The [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_aks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_aks.sh) script provided in this repo is strictly optional.
The script is mainly to help those new to Kubernetes and/or Fusion get started quickly.
If you're already familiar with K8s, Helm, and AKS, then you use Helm directly to install Fusion into an existing cluster or one you create yourself using the process described `<>`.
If you're new to Azure, then please visit [https://azure.microsoft.com/en-us/free/search/](https://azure.microsoft.com/en-us/free/search/) to set up an account.
### Set up the AKS CLI tools
Before launching an AKS cluster, you need to install and configure `kubectl` and `az` using the links provided below:
Required AKS Command-line Tools:
* `kubectl`: [https://kubernetes.io/docs/tasks/tools/install-kubectl/](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
* `az`: [https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest)
To confirm your account access and command-line tools are set up correctly, run the `az login` command (`az login –help` to see available options).
### Azure Prerequisites
To launch a cluster in AKS (or pretty much do anything with Azure) you need to setup a Resource Group. Resource Groups are a way of organizing and managing related resources in Azure.
For more information about resource groups, see [https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview#resource-groups](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview#resource-groups).
You also need to choose a location where you want to spin up your AKS cluster, such as `westus2`. For a list of locations you can choose, see [https://azure.microsoft.com/en-us/global-infrastructure/locations/](https://azure.microsoft.com/en-us/global-infrastructure/locations/).
Use the Azure console in your browser to create a resource group, or simply do:
```bash theme={"dark"}
az group create -g $AZURE_RESOURCE_GROUP -l $AZURE_LOCATION
```
To recap, you should have the following requirements in place:
* Azure Account set up.
* `azure-cli` (`az`) command-line tools installed.
* `az` login working.
* Created an Azure Resource Group and selected a location to launch the cluster.
### Set up Fusion on AKS
Download and run the [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_aks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_aks.sh) to install Fusion 5.x in a AKS cluster. To create a new cluster and install Fusion, simply do:
```bash theme={"dark"}
./setup_f5_aks.sh -c -p
```
If you don't want the script to create a cluster, then you need to create a cluster before running the script and simply pass the name of the existing cluster using the `-c` parameter.
Use the `--help` option to see full script usage.
By default, our script installs Fusion into the default namespace; think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster in separate namespaces. However, please do not install more than one Fusion release in the same namespace.
You can override the namespace using the `-n` option. In addition, our script uses f5 for the Helm release name; you can customize this using the `-r` option. Helm uses the release name you provide to track a specific instance of an installation, allowing you to perform updates and rollback changes for that specific release only.
You can also pass the `--preview` option to the script, which enables soon-to-be-released features for AKS, such as deploying a multi-zone cluster across 3 availability zones for higher availability guarantees. For more information about the Availability Zone feature, see [https://docs.microsoft.com/en-us/azure/aks/availability-zones](https://docs.microsoft.com/en-us/azure/aks/availability-zones).
It takes a while for AKS to spin up the new cluster. The cluster will have three Standard\_D4\_v3 nodes which have 4 CPU cores and 16 GB of memory. Behind the scenes, our script calls the `az aks create` command.
If using Helm V2, the `setup_f5_aks.sh` script installs Helm's `tiller` component into your AKS cluster with the cluster admin role. If you don't want this, then please upgrade to Helm v3.
After running the `setup_f5_aks.sh` script, proceed to `<>`.
#### AKS Ingress
The `setup_f5_aks.sh` script exposes the Fusion proxy service on an external IP over HTTP. This is done for demo or getting started purposes. However, you're strongly encouraged to configure a K8s Ingress with TLS termination in front of the proxy service.
Use the `-t` and `-h ` options to have our script create an Ingress with a TLS certificate issued by Let's Encrypt.
#### Upgrades and Ingress
IMPORTANT: If you used the `-t -h ` options when installing your cluster, our script created an additional values yaml file named `tls-values.yaml`.
To make things easier for you when upgrading, you should add the settings from this file into your main custom values yaml file. For example:
```yaml theme={"dark"}
api-gateway:
service:
type: "NodePort"
ingress:
enabled: true
host: ""
tls:
enabled: true
annotations:
"networking.gke.io/managed-certificates": "-managed-certificate"
"kubernetes.io/ingress.class": "gce"
```
This way, you don't have to remember to pass the additional `tls-values.yaml` file when upgrading.
## Verifying the Fusion Installation
In this section, we provide some tips on how to verify the Fusion installation.
Check if the Fusion Admin UI is available at `\https://:6764/admin/`.
Let's review some useful kubectl commands.
### Enhance the K8s Command-line Experience
Here is a list of tools we found useful for improving your command-line experience with Kubernetes:
* krew (kubectl plugin mgr): [https://github.com/kubernetes-sigs/krew/](https://github.com/kubernetes-sigs/krew/)
* kube-ps1 (show current context on command line prompt): [https://github.com/jonmosco/kube-ps1](https://github.com/jonmosco/kube-ps1)
* kubectx / kubens (switch between clusters / namespaces): [https://github.com/ahmetb/kubectx](https://github.com/ahmetb/kubectx)
### Useful kubectl commands
kubectl reference: [https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands)
Set the namespace for `kubectl` if not using the default:
```bash theme={"dark"}
kubectl config set-context --current --namespace=
```
**This saves you from having to pass `-n` with every command.**
Get a list of running pods: `k get pods`
Get logs for a pod using a label: `k logs –l app.kubernetes.io/component=query-pipeline`
Get pod deployment spec and details: `k get pods -o yaml`
Get details about a pod events: `k describe po `
Port forward to a specific pod: `k port-forward 8983:8983`
SSH into a pod: `k exec -it -- /bin/bash`
CPU/Memory usage report for pods: `k top pods`
Forcefully kill a pod: `k delete po --force --grace-period 0`
Scale up (or down) a deployment: `k scale deployment.v1.apps/ --replicas=N`
Get a list of pod versions: `k get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq`
### Check Fusion Pods and Services
Once the install script completes, you can check that all pods and services are available using:
```bash theme={"dark"}
kubectl get pods
```
If all goes well, you should see a list of pods similar to:
```bash wrap theme={"dark"}
NAME READY STATUS RESTARTS AGE
seldon-controller-manager-6675874894-qxwrv 1/1 Running 0 8m45s
f5-admin-ui-74d794f4f8-m5jms 1/1 Running 0 8m45s
f5-ambassador-fd6b9b5dc-7ghf6 1/1 Running 0 8m43s
f5-api-gateway-6b9998b9c-tmchk 1/1 Running 0 8m45s
f5-auth-ui-7565564b4c-rdc74 1/1 Running 0 8m42s
f5-classic-rest-service-0 1/1 Running 3 8m44s
f5-devops-ui-77bb867ffb-fbzxd 1/1 Running 0 8m42s
f5-fusion-admin-78b8f8fc7f-4d7l8 1/1 Running 0 8m42s
f5-fusion-indexing-599c8d448-xzsvm 1/1 Running 0 8m44s
f5-insights-665fd9f6fc-g5psw 1/1 Running 0 8m43s
f5-job-launcher-84dd4c5c96-p8528 1/1 Running 0 8m44s
f5-job-rest-server-6d44d964b8-xtnxw 1/1 Running 0 8m45s
f5-logstash-0 1/1 Running 0 8m45s
f5-ml-model-service-6987dc94c9-9ppp8 2/2 Running 1 8m45s
f5-monitoring-grafana-5d499dbb58-pzw72 1/1 Running 0 10m
f5-monitoring-prometheus-kube-state-metrics-54d6678dv9h7h 1/1 Running 0 10m
f5-monitoring-prometheus-pushgateway-7d65c65b85-vwrwf 1/1 Running 0 10m
f5-monitoring-prometheus-server-0 2/2 Running 0 10m
f5-pm-ui-86cbc5bb65-nd2n8 1/1 Running 0 8m44s
f5-pulsar-bookkeeper-0 1/1 Running 0 8m45s
f5-pulsar-broker-b56cc776f-56msx 1/1 Running 0 8m45s
f5-query-pipeline-5d75d7d5f4-l2mdf 1/1 Running 0 8m43s
f5-connectors-7bb6cfc65f-7wfs2 1/1 Running 0 8m42s
f5-connectors-backend-987fdc648-dldwv 1/1 Running 0 8m45s
f5-rules-ui-6b9d55b78f-9hzzj 1/1 Running 0 8m43s
f5-solr-0 1/1 Running 0 8m44s
f5-solr-exporter-c4687c785-jsm7x 1/1 Running 0 8m45s
f5-ui-6cdbcc68c6-rj9cq 1/1 Running 0 8m45s
f5-webapps-6d6bb9bfd-hm4qx 1/1 Running 0 8m45s
f5-workflow-controller-7b66679fb7-sjbvp 1/1 Running 0 8m44s
f5-zookeeper-0 1/1 Running 0 8m45s
```
The number of pods per deployment / statefulset will vary based on your cluster size and replicaCount settings in your custom values YAML file.
Also, don't worry if you see some pods having been restarted as that just means they were too slow to come up and Kubernetes killed and restarted them.
You do want to see at least one pod running for every service. If a pod is not running after waiting a sufficient amount of time,
use `kubectl logs ` to see the logs for that pod; to see the logs for previous versions of a pod, use: `kubectl logs -p`.
You can also look at the actions Kubernetes performed on the pod using `kubectl describe po `.
To see a list of Fusion services, do:
```bash theme={"dark"}
kubectl get svc
```
For an overview of the various Fusion 5 microservices, see: [Fusion microservices](/docs/5/fusion/reference/microservices).
Once you're ready to build a Fusion cluster for production, please see see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
### Upgrading with Zero Downtime
One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. Fusion 5 allows customers to upgrade from Fusion 5.x.y to a later 5.x.z version on a live cluster with zero downtime or disruption of service.
When Kubernetes performs a rolling update to an individual microservice, there will be a mix of old and new services in the cluster concurrently (only briefly in most cases) and requests from other services will be routed to both versions. Consequently, Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same 5.x line of releases. We also ensure stored configuration remains compatible in the same 5.x release line.
Lucidworks releases minor updates to individual services frequently, so our customers can pull in those upgrades using Helm at their discretion.
To upgrade your cluster at any time, use the `--upgrade` option with our setup scripts in this repo.
The scripts in this repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.
To see what would be upgraded, you can pass the `--dry-run` option to the script.
### Grafana Dashboards
Get the initial Grafana password from a K8s secret by doing:
```bash wrap theme={"dark"}
kubectl get secret --namespace "${NAMESPACE}" ${RELEASE}-monitoring-grafana \
-o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```
With Grafana, you can either setup a temporary port-forward to a Grafana pod or expose Grafana on an external IP using a K8s LoadBalancer.
To define a LoadBalancer, do (replace \$ with your Helm release label):
```bash wrap theme={"dark"}
kubectl expose deployment ${RELEASE}-monitoring-grafana --type=LoadBalancer --name=grafana --port=3000 --target-port=3000
```
You can use `kubectl get services --namespace ` to determine when the load balancer is setup and its IP address. Direct your browser to `http://:3000` and enter the username `admin@localhost` and the password that was returned in the previous step.
This will log you into the application. It is recommended that you create another administrative user with a more desirable password.
The dashboards and datasoure will be setup for you in grafana, simply navigate to `Dashboards` -> `Manage` to view the available dashboards
## Frequently Asked Questions
**Can the stateful database services, for example, MySQL, be supported by an Azure PaaS service?**
This option is not supported. In theory, it may be possible to implement this function.
**Is it possible to use cross-zone storage solutions rather than volumes, such as Microsoft Azure file storage, for stateful services?**
While in theory it may be possible to implement this functionality, the configuration has not been tested by Lucidworks and is not supported.
## Learn more
The course for **Deploying Fusion 5** focuses on the prerequisite software needed to deploy Fusion, the necessary setup steps, and the physical act of deployment.
The [`setup_f5_k8s.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_k8s.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository provides deployment support for any Kubernetes platform, including on-premise, private cloud, public cloud, and hybrid platforms.
This script is used by the [`setup_f5_gke.sh`](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_gke.sh), [`setup_f5_eks.sh`](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_eks.sh), and [`setup_f5_aks.sh`](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_aks.sh) scripts, which provide additional platform-specific support for Google Kubernetes Engine (GKE), Amazon Elastic Kubernetes Service (EKS), and Azure Kubernetes Service (AKS).
This topic explains how to deploy a Fusion cluster in Kubernetes using the [`setup_f5_k8s.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_k8s.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository.
If you’re deploying on-premises or using a localized repository, you’ll need to use a private repository for Docker images.
## Prerequisites
This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes.
### Release Name and Namespace
Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) Kubernetes namespace to install Fusion into.
Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*.
However, please *do not* install more than one Fusion release in the same namespace.
All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_
Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default.
### Install Helm
Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster.
Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform.
On MacOS, you can do:
```bash theme={"dark"}
brew install kubernetes-helm
```
If you already have helm installed, make sure you're using the latest version:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/)
The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`.
#### Helm User Permissions
If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory.
When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.:
```bash theme={"dark"}
alias k=kubectl
```
To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into:
```bash theme={"dark"}
k create namespace fusion-namespace
```
Apply the `role.yaml` and `cluster-role.yaml` files to that namespace
```bash theme={"dark"}
k apply -f cluster-role.yaml
k config set-context --current --namespace=$NAMESPACE
k apply -f role.yaml
```
Then bind the rolebinding and clusterolebinding to the install user:
```bash wrap theme={"dark"}
k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user
k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user
```
You will then be able to run the `helm install` command as the ``
### Clone fusion-cloud-native from GitHub
You should clone this repo from github as you'll need to run the scripts on your local workstation:
```bash theme={"dark"}
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts.
```bash theme={"dark"}
cd fusion-cloud-native
git pull
```
Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip).
Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory.
## Deployment
If you're not running on a managed K8s platform like GKE, AKS, or EKS, you can use Helm to install the Fusion chart to an existing Kubernetes cluster.
Fusion version 5.5 now includes support for the Rancher Kubernetes Engine (RKE) platform. Before deploying Fusion to RKE, you must download and install the [latest RKE software](https://rancher.com/docs/rke/latest/en/). After configuring your cluster, you can proceed with the Helm v3 installation.
You must have a working cluster configured before performing the Helm v3 installation.
### Use Helm v3 to Install Fusion
You should upgrade to the latest version of Helm v3 for working with Fusion. If you need to keep Helm V2 for other clusters,
ensure Helm V3 is ahead of Helm V2 in your working shell's PATH before proceeding.
#### Customize Fusion Chart Settings
Fusion aims to be well-configured out-of-the-box, but you can customize any of the built-in settings using a custom values `YAML` file. If you use one of our setup scripts, such as `setup_f5_gke.sh`, then it will create a custom values YAML file for you the first time you run it using [https://github.com/lucidworks/fusion-cloud-native/blob/master/customize\_fusion\_values.yaml.example](https://github.com/lucidworks/fusion-cloud-native/blob/master/customize_fusion_values.yaml.example) as a template.
If you're working with Helm directly and not using one of our setup scripts, then run the [https://github.com/lucidworks/fusion-cloud-native/blob/master/customize\_fusion\_values.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/customize_fusion_values.sh) script to create a custom values YAML file from our [https://github.com/lucidworks/fusion-cloud-native/blob/master/customize\_fusion\_values.yaml.example](https://github.com/lucidworks/fusion-cloud-native/blob/master/customize_fusion_values.yaml.example) template as a starting point:
```bash theme={"dark"}
./customize_fusion_values.sh -c -n \
--provider --num-solr 1 --node-pool ""
```
Pass `--help` for usage details.
In this example:
* `` is the K8s platform you're running on, such as `gke`
* `` is the name of your cluster
* `` is the K8s namespace where you plan to install Fusion
The `--node-pool` option specifies the node selector label for determining which nodes to run Fusion pods. You can pass `"{}"` to let Kubernetes decide which nodes to schedule pods on.
This file is referred to as `${MY_VALUES}` in the commands belo. Replace the filename with the correct filename for your environment. Keep this file handy, as you'll need it to customize Fusion settings and upgrade to a newer version.
Review the settings in the custom values YAML file to ensure the defaults are appropriate for your environment, including the number of Solr and Zookeeper replicas.
Add the Lucidworks Helm repo:
```bash theme={"dark"}
helm repo add lucidworks https://charts.lucidworks.com
```
The `customize_fusion_values.sh` script creates an upgrade script to install/upgrade Fusion into Kubernetes using Helm. Look in the directory where you ran `customize_fusion_values.sh` for a script named like:
`___upgrade_fusion.sh`. Run this script to install Fusion.
## Verifying the Fusion Installation
In this section, we provide some tips on how to verify the Fusion installation.
Check if the Fusion Admin UI is available at `\https://:6764/admin/`.
Let's review some useful kubectl commands.
### Enhance the K8s Command-line Experience
Here is a list of tools we found useful for improving your command-line experience with Kubernetes:
* krew (kubectl plugin mgr): [https://github.com/kubernetes-sigs/krew/](https://github.com/kubernetes-sigs/krew/)
* kube-ps1 (show current context on command line prompt): [https://github.com/jonmosco/kube-ps1](https://github.com/jonmosco/kube-ps1)
* kubectx / kubens (switch between clusters / namespaces): [https://github.com/ahmetb/kubectx](https://github.com/ahmetb/kubectx)
### Useful kubectl commands
kubectl reference: [https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands)
Set the namespace for `kubectl` if not using the default:
```bash theme={"dark"}
kubectl config set-context --current --namespace=
```
**This saves you from having to pass `-n` with every command.**
Get a list of running pods: `k get pods`
Get logs for a pod using a label: `k logs –l app.kubernetes.io/component=query-pipeline`
Get pod deployment spec and details: `k get pods -o yaml`
Get details about a pod events: `k describe po `
Port forward to a specific pod: `k port-forward 8983:8983`
SSH into a pod: `k exec -it -- /bin/bash`
CPU/Memory usage report for pods: `k top pods`
Forcefully kill a pod: `k delete po --force --grace-period 0`
Scale up (or down) a deployment: `k scale deployment.v1.apps/ --replicas=N`
Get a list of pod versions: `k get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq`
### Check Fusion Pods and Services
Once the install script completes, you can check that all pods and services are available using:
```bash theme={"dark"}
kubectl get pods
```
If all goes well, you should see a list of pods similar to:
```bash wrap theme={"dark"}
NAME READY STATUS RESTARTS AGE
seldon-controller-manager-6675874894-qxwrv 1/1 Running 0 8m45s
f5-admin-ui-74d794f4f8-m5jms 1/1 Running 0 8m45s
f5-ambassador-fd6b9b5dc-7ghf6 1/1 Running 0 8m43s
f5-api-gateway-6b9998b9c-tmchk 1/1 Running 0 8m45s
f5-auth-ui-7565564b4c-rdc74 1/1 Running 0 8m42s
f5-classic-rest-service-0 1/1 Running 3 8m44s
f5-devops-ui-77bb867ffb-fbzxd 1/1 Running 0 8m42s
f5-fusion-admin-78b8f8fc7f-4d7l8 1/1 Running 0 8m42s
f5-fusion-indexing-599c8d448-xzsvm 1/1 Running 0 8m44s
f5-insights-665fd9f6fc-g5psw 1/1 Running 0 8m43s
f5-job-launcher-84dd4c5c96-p8528 1/1 Running 0 8m44s
f5-job-rest-server-6d44d964b8-xtnxw 1/1 Running 0 8m45s
f5-logstash-0 1/1 Running 0 8m45s
f5-ml-model-service-6987dc94c9-9ppp8 2/2 Running 1 8m45s
f5-monitoring-grafana-5d499dbb58-pzw72 1/1 Running 0 10m
f5-monitoring-prometheus-kube-state-metrics-54d6678dv9h7h 1/1 Running 0 10m
f5-monitoring-prometheus-pushgateway-7d65c65b85-vwrwf 1/1 Running 0 10m
f5-monitoring-prometheus-server-0 2/2 Running 0 10m
f5-pm-ui-86cbc5bb65-nd2n8 1/1 Running 0 8m44s
f5-pulsar-bookkeeper-0 1/1 Running 0 8m45s
f5-pulsar-broker-b56cc776f-56msx 1/1 Running 0 8m45s
f5-query-pipeline-5d75d7d5f4-l2mdf 1/1 Running 0 8m43s
f5-connectors-7bb6cfc65f-7wfs2 1/1 Running 0 8m42s
f5-connectors-backend-987fdc648-dldwv 1/1 Running 0 8m45s
f5-rules-ui-6b9d55b78f-9hzzj 1/1 Running 0 8m43s
f5-solr-0 1/1 Running 0 8m44s
f5-solr-exporter-c4687c785-jsm7x 1/1 Running 0 8m45s
f5-ui-6cdbcc68c6-rj9cq 1/1 Running 0 8m45s
f5-webapps-6d6bb9bfd-hm4qx 1/1 Running 0 8m45s
f5-workflow-controller-7b66679fb7-sjbvp 1/1 Running 0 8m44s
f5-zookeeper-0 1/1 Running 0 8m45s
```
The number of pods per deployment / statefulset will vary based on your cluster size and replicaCount settings in your custom values YAML file.
Also, don't worry if you see some pods having been restarted as that just means they were too slow to come up and Kubernetes killed and restarted them.
You do want to see at least one pod running for every service. If a pod is not running after waiting a sufficient amount of time,
use `kubectl logs ` to see the logs for that pod; to see the logs for previous versions of a pod, use: `kubectl logs -p`.
You can also look at the actions Kubernetes performed on the pod using `kubectl describe po `.
To see a list of Fusion services, do:
```bash theme={"dark"}
kubectl get svc
```
For an overview of the various Fusion 5 microservices, see: [Fusion microservices](/docs/5/fusion/reference/microservices).
Once you're ready to build a Fusion cluster for production, please see see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
### Upgrading with Zero Downtime
One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. Fusion 5 allows customers to upgrade from Fusion 5.x.y to a later 5.x.z version on a live cluster with zero downtime or disruption of service.
When Kubernetes performs a rolling update to an individual microservice, there will be a mix of old and new services in the cluster concurrently (only briefly in most cases) and requests from other services will be routed to both versions. Consequently, Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same 5.x line of releases. We also ensure stored configuration remains compatible in the same 5.x release line.
Lucidworks releases minor updates to individual services frequently, so our customers can pull in those upgrades using Helm at their discretion.
To upgrade your cluster at any time, use the `--upgrade` option with our setup scripts in this repo.
The scripts in this repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.
To see what would be upgraded, you can pass the `--dry-run` option to the script.
### Grafana Dashboards
Get the initial Grafana password from a K8s secret by doing:
```bash wrap theme={"dark"}
kubectl get secret --namespace "${NAMESPACE}" ${RELEASE}-monitoring-grafana \
-o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```
With Grafana, you can either setup a temporary port-forward to a Grafana pod or expose Grafana on an external IP using a K8s LoadBalancer.
To define a LoadBalancer, do (replace \$ with your Helm release label):
```bash wrap theme={"dark"}
kubectl expose deployment ${RELEASE}-monitoring-grafana --type=LoadBalancer --name=grafana --port=3000 --target-port=3000
```
You can use `kubectl get services --namespace ` to determine when the load balancer is setup and its IP address. Direct your browser to `http://:3000` and enter the username `admin@localhost` and the password that was returned in the previous step.
This will log you into the application. It is recommended that you create another administrative user with a more desirable password.
The dashboards and datasoure will be setup for you in grafana, simply navigate to `Dashboards` -> `Manage` to view the available dashboards
## Learn more
The course for **Deploying Fusion 5** focuses on the prerequisite software needed to deploy Fusion, the necessary setup steps, and the physical act of deployment.
Before you begin, see [Fusion Server Deployment](/docs/5/fusion/operations/deployment) to understand the architecture and requirements.
This article explains how to plan and execute a Fusion deployment at the scale required for staging or production.
While the `setup_f5_*.sh` scripts are handy for getting started and proof-of-concept purposes, this article covers the planning process for building a production-ready environment.
The course for **Preparing for Fusion Implementation** focuses on the key elements for a successful implementation, defining your business requirements, preparing clean data, and involving the right personnel.
## Prerequisites
You must meet the following prerequisites before you can customize your Fusion cluster:
* A local copy of the [fusion-cloud-native repository](https://github.com/lucidworks/fusion-cloud-native). This must be up-to-date with the latest master branch.
* Any cloud provider-specific **command line tools**, such as `gcloud` or `aws`, and `kubectl`.\
See the platform-specific instructions linked above, or check with your cloud provider.
* Helm v3
* To install on a Mac:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
* For other operating systems, download from [Helm Releases](https://github.com/helm/helm/releases).
* Verify your installation:
```bash theme={"dark"}
helm version --short
v3.0.0+ge29ce2a
```
* Kubernetes namespace
* Collect the following information about your Kubernetes environment:
* **CLUSTER**: Cluster name (passed to our setup scripts using the `-c` arg)
* **NAMESPACE**: Kubernetes namespace where to install Fusion; a namespace should only contain lowercase letters (a-z), digits (0-9), or dash. No periods or underscores allowed.
* *(optional)* Clarify your organization’s DockerHub policy. The Fusion Helm chart points to public Docker images on DockerHub. Your organization may not allow Kubernetes to pull images directly from DockerHub or may require extra security scanning before loading images into production clusters.\
Consult your Kubernetes and Docker admin team to find how to get the Fusion images loaded into a registry that’s accessible to your cluster. You can update the image for each service using the [custom values YAML file](#custom-values-yaml-file).
**Kubernetes namespace tips**
* Fusion 5 service discovery requires all services for the same release be deployed in the same namespace. Moreover, you should only run one instance of Fusion in a namespace. If you need multiple instances of Fusion running in the same Kubernetes cluster, then you need to deploy them in separate namespaces.
* If your organization requires CPU / Memory quotas for namespaces, you can start with a minimum of 12 CPU and 45GB of RAM (such as 3 x n1-standard-4 on GKE), but you will need to increase the quotas once you start load testing Fusion with production workloads and real datasets.
* Fusion requires at least 3 ZooKeeper nodes and 2 Solr nodes to achieve high availability.
## Custom values YAML file
1. Clone the `fusion-cloud-native` repository: `git clone https://github.com/lucidworks/fusion-cloud-native`
2. Run the `customize_fusion_values.sh` script.
```bash theme={"dark"}
./customize_fusion_values.sh --provider -c -n \
--num-solr 3 \
--solr-disk-gb 100 \
--node-pool \
--prometheus true \
--with-resource-limits \
--with-affinity-rules
```
Pass the `--help` parameter to see script usage details.
The script creates the following files:
| File | Description |
| - | - |
| `___fusion_values.yaml` | Main custom values YAML used to override Helm chart defaults for Fusion microservices. |
| `___monitoring_values.yaml` | Custom values yaml used to configure Prometheus and Grafana. |
| `___fusion_resources.yaml` | Resource requests and limits for all Microservices. |
| `___fusion_affinity.yaml` | Pod affinity rules to ensure multiple replicas for a single service are evenly distributed across zones and nodes. |
| `___upgrade_fusion.sh` | Script used to install and/or upgrade Fusion using the aforementioned custom values YAML files. |
For an explanation of these placeholder values, see [Configuration Values](#custom-values-yaml-file) below.
3. Add the new files to version control. You will make changes to it over time as you fine-tune your Fusion installation. You will also need it to perform upgrades. If you try to upgrade your Fusion installation and don’t provide the custom values YAML, your deployment will revert to chart defaults.\
Review the `___fusion_values.yaml` file to familiarize yourself with its structure and contents. Notice it contains a separate section for each of the Fusion microservices. The example configuration of the `query-pipeline` service below illustrates some important concepts about the custom values YAML file.
```yaml highlight={1,2,3,5,6} theme={"dark"}
query-pipeline:
enabled: true
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
javaToolOptions: "..."
pod:
annotations:
prometheus.io/port: "8787"
prometheus.io/scrape: "true"
prometheus.io/path: "/actuator/prometheus"
```
4. Service-specific setting overrides under the top-level heading
5. Every Fusion service has an implicit enabled flag that defaults to true, set to false to remove this service from your cluster
6. Node selector identifies the label find nodes to schedule pods on
7. Used to pass JVM options to the service
8. Pod annotations to allow Prometheus to scrape metrics from the service
Once we go through all of the configuration topics in this topic, you'll have a well-configured custom values YAML file for your Fusion 5 installation. You'll then use this file during the Helm v3 installation at the end of this topic.
#### Custom labels and annotations
Starting with Fusion 5.9.16 and later, you can configure custom labels and annotations for all Kubernetes resources through values files without modifying chart templates. This eliminates upgrade complexity and maintenance overhead.
Configure global values that apply to all Fusion services:
```yaml theme={"dark"}
global:
labels:
owner: search-platform
annotations:
cost-center: fin-1234
```
Configure service-specific values that apply only to individual charts:
```yaml theme={"dark"}
fusion-indexing:
labels:
team: platform-search
annotations:
monitoring: enabled
```
The chart automatically merges and renders these values into Kubernetes resource metadata. Common use cases include cost allocation labels, ownership labels, monitoring annotations, and service mesh annotations.
### Deployment-specific values
The script creates a custom values YAML file using the naming convention: `___fusion_values.yaml`. For example, `gke_search_f5_fusion_values.yaml`.
| Parameter | Description |
| - | - |
| `` | The K8s platform you’re running on, such as `gke`. |
| `` | The name of your cluster. |
| `` | The K8s namespace where you want to install Fusion. |
| `` | Specifies a `nodeSelector` label to find nodes to schedule Fusion pods on. |
Providing the correct `--node-pool ` label is very important. Using the wrong value will cause your pods to be stuck in the `pending` state. If you’re not sure about the correct value for your cluster, pass ’'\` to let Kubernetes decide which nodes to schedule Fusion pods on.
Default `nodeSelector` labels are provider-specific. The `fusion-cloud-native` scripts use the following defaults for GKE and EKS:
| Provider | Default node selector |
| - | - |
| GKE | cloud.google.com/gke-nodepool: default-pool |
| EKS | alpha.eksctl.io/nodegroup-name: standard-workers |
If you are deploying Fusion 5.9.12, add the following to your `values.yaml` file to avoid a known issue that prevents the `kuberay-operator` pod from launching successfully: `yaml kuberay-operator: crd: create: true `
### Flags
The script provides flags for additional configuration:
| Flag | Description |
| - | - |
| `--node-pool` | Add a Fusion specific label to your nodes. |
| `--with-resource-limits` | Configure resource requests/limits. |
| `--with-replicas` | Configure replica counts. |
| `--with-affinity-rules` | Configure pod affinity rules for Fusion services. |
Use `--node-pool` to add a Fusion specific label to your nodes by doing:
```bash theme={"dark"}
kubectl label fusion_node_type=
```
Then, pass `--node-pool 'fusion_node_type: '`.
## Configure Solr sizing
When you’re ready to build a production-ready setup for Fusion 5, you need to customize the Fusion Helm chart to ensure Fusion is well-configured for production workloads.
You’ll be able to scale the number of nodes for Solr up and down after building the cluster, but you need to establish the initial size of the nodes (memory and CPU) and the size and type of disks you need.
See the example config below to learn which parameters to change in the custom values YAML file.
```yaml expandable theme={"dark"}
solr:
resources: # Set resource limits for Solr to help K8s pod scheduling;
limits: # these limits are not just for the Solr process in the pod,
cpu: "7700m" # so allow ample memory for loading index files into the OS cache (mmap)
memory: "26Gi"
requests:
cpu: "7000m"
memory: "25Gi"
logLevel: WARN
nodeSelector:
fusion_node_type: search # Run this Solr StatefulSet in the "search" node pool
exporter:
enabled: true # Enable the Solr metrics exporter (for Prometheus) and
# schedule on the default node pool (system partition)
podAnnotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9983"
prometheus.io/path: "/metrics"
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
image:
tag: 8.4.1
updateStrategy:
type: "RollingUpdate"
javaMem: "-Xmx3g -Dfusion_node_type=system" # Configure memory settings for Solr
solrGcTune: "-XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UseStringDeduplication -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=150 -XX:+UseLargePages -XX:+AlwaysPreTouch"
volumeClaimTemplates:
storageSize: "100Gi" # Size of the Solr disk
replicaCount: 6 # Number of Solr pods to run in this StatefulSet
zookeeper:
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
replicaCount: 3 # Number of Zookeepers
persistence:
size: 20Gi
resources: {}
env:
ZK_HEAP_SIZE: 1G
ZOO_AUTOPURGE_PURGEINTERVAL: 1
```
To be clear, you can tune GC settings and number of replicas after the cluster is built. But changing the size of the persistent volumes is more complicated so you should try to pick a good size initially.
On a fresh Fusion 5.17.x installation, if services loop at startup with `lookup zookeeper: no such host` in their `check-zk` init containers even though ZooKeeper is healthy, add `global.zkReplicaCount` to your custom values file and set it to the same value as `zookeeper.replicaCount`:
```yaml theme={"dark"}
global:
zkReplicaCount: 3
```
Without this value, the chart renders an empty ZooKeeper connection string for dependent services.
This setting is also required when your Helm release name differs from the Kubernetes namespace name.
### Configure storage class for Solr pods (optional)
If you wish to run with a storage class other than the default you can create a storage class for your Solr pods before you install. For example, to create regional disks in GCP you can create a file called `storageClass.yaml` with the following contents:
```yaml theme={"dark"}
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: solr-gke-storage-regional
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-standard
replication-type: regional-pd
zones: us-west1-b, us-west1-c
```
and then provision into your cluster by calling:
```bash theme={"dark"}
kubectl apply -f storageClass.yaml
```
to then have Solr use the storage class by adding the following to the custom values YAML:
```yaml theme={"dark"}
solr:
volumeClaimTemplates:
storageClassName: solr-gke-storage-regional
storageSize: 250Gi
```
We’re not advocating that you must use regional disks for Solr storage, as that would be redundant with Solr replication. We’re just using this as an example of how to configure a custom storage class for Solr disks if you see the need. For instance, you could use regional disks without Solr replication for write-heavy type collections.
### Match the storage class to the GKE machine family
Newer Google Cloud machine families, including N4, C4, and C4A, support only Hyperdisk volumes and can't attach the `pd-balanced` disks that GKE's default `standard-rwo` storage class provisions.
On clusters that use these machine families, Fusion's stateful services such as Solr, ZooKeeper, Kafka, and MySQL stall at startup with `FailedAttachVolume` events.
Before installing Fusion on nodes from these machine families, save the following manifest as `hyperdisk-storage-class.yaml`, create the Hyperdisk storage class, and set it as the cluster default:
```yaml theme={"dark"}
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: hyperdisk-balanced-rwo
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: pd.csi.storage.gke.io
parameters:
type: hyperdisk-balanced
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
```
Remove the default annotation from the previous default storage class first, because only one storage class can be the default:
```bash theme={"dark"}
kubectl patch storageclass CURRENT_DEFAULT_STORAGE_CLASS -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
kubectl apply -f hyperdisk-storage-class.yaml
```
Alternatively, set `storageClassName` explicitly for each stateful service in your custom values file, as shown for Solr in the previous section.
See [Machine series support for Hyperdisk](https://cloud.google.com/compute/docs/disks/hyperdisks) in the Google Cloud documentation for the current compatibility list.
## Configure multiple node pools
Lucidworks recommends isolating search workloads from analytics workloads using multiple node pools. The included scripts do not do this for you; this is a manual process.
See the example script for GKE, see [create\_gke\_cluster\_node\_pools.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/additional_environments/create_gke_cluster_node_pools.sh).
In the custom values YAML file, you can add additional Solr StatefulSets by adding their names to the list under the `nodePools` property. If any property for that statefulset needs to be changed from the default set of values, then it can be set directly on the object representing the node pool, any properties that are omitted are defaulted to the base value. See the following example (additional whitespace added for display purposes only):
```yaml expandable highlight={3,4,9,17,22,31} theme={"dark"}
solr:
nodePools:
- name: ""
- name: "analytics"
javaMem: "-Xmx6g"
replicaCount: 6
storageSize: "100Gi"
nodeSelector:
fusion_node_type: analytics
resources:
requests:
cpu: 2
memory: 12Gi
limits:
cpu: 3
memory: 12Gi
- name: "search"
javaMem: "-Xms11g -Xmx11g"
replicaCount: 12
storageSize: "50Gi"
nodeSelector:
fusion_node_type: search
resources:
limits:
cpu: "7700m"
memory: "26Gi"
requests:
cpu: "7000m"
memory: "25Gi"
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
...
```
1. The empty string `""` is the suffix for the default partition.
2. Overrides the settings for the **analytics** Solr pods.
3. Assigns the **analytics** Solr pods to the node pool and attaches the label `fusion_node_type=analytics`. You can use the `fusion_node_type` property in Solr auto-scaling policies to govern replica placement during collection creation.
4. Overrides the settings for the **search** Solr pods.
5. Assigns the **search** Solr pods to the node pool and attaches the label `fusion_node_type=search`.
6. Sets the default settings for all Solr pods, if not specifically overridden in the `nodePools` section above.
Do not edit the `nodePools` value `""`.
In the example above, the **analytics** partition `replicaCount`, or number of Solr pods, is six. The **search** partition `replicaCount` is twelve.
Each nodePool is automatically be assigned the -Dfusion\_node\_type property of ``, ``, or ``. This value matches the name of the nodePool. For example, `-Dfusion_node_type=`.
The Solr pods have a `fusion_node_type` system property, as shown below:
## Solr auto-scaling policy
Use [replica placement plugins](https://solr.apache.org/guide/solr/latest/configuration-guide/replica-placement-plugins.html) to control how replicas are placed in Solr.
## Pod network policy
A Kubernetes network policy governs how groups of pods communicate with each other and other network endpoints. With Fusion, all incoming traffic flows through the API Gateway service. All Fusion services in the same namespace expect an internal JWT, which is supplied by the Gateway, as part of the request. As a result, Fusion services enforce a basic level of API security and don’t need an additional network policy to protect them from other pods in the cluster.
To install the network policy for Fusion services, pass `--set global.networkPolicyEnabled=true` when installing the Fusion Helm chart.
## On-premises private Docker registries
For on-premises Kubernetes deployments, your organization may not allow Kubernetes to pull [Fusion’s Docker images from DockerHub](https://hub.docker.com/u/lucidworks/). See the instructions below for details on using a private Docker registry with Fusion. These are general instructions that may need to be adapted to work within your organization’s security policies:
1. Transfer the public images from DockerHub to your private Docker registry.
2. Establish a workstation that has access to [DockerHub](https://hub.docker.com). This workstation must connect to your internal Docker registry, most likely via VPN connection. In this example, the workstation is referred to as `envoy`.
3. Install Docker on `envoy`. You need at least 100GB of free disk for Docker.
4. Pull all of the images from DockerHub to `envoy`’s local registry. For example, to pull the query pipeline image, run `docker pull lucidworks/query-pipeline:5.9.0`. See `docker pull --help` for more information about pulling Docker images.
5. Establish a connection from `envoy` to the private Docker registry, most likely via a VPN connection. In this example, the private Docker registry is referred to as ``.
6. Push the images from `envoy`’s Docker registry to the private registry. This will take a long time.
1. You’ll need to re-tag all images for the internal registry. For example, to tag the query-pipeline image, run:
```bash theme={"dark"}
docker tag lucidworks/query-pipeline:5.9.0 /query-pipeline:5.9.0
```
2. Push each image to the internal repo:
```bash theme={"dark"}
docker push /query-pipeline:5.9.0
```
7. Install the Docker registry secret in Kubernetes. Create the Docker registry secret in the Kubernetes namespace where you want to install Fusion:
```bash theme={"dark"}
SECRET_NAME=
REPO=
kubectl create secret docker-registry "${SECRET_NAME}" \
--namespace "${NAMESPACE}" \
--docker-server="${REPO}" \
--docker-username=${REPO_USER} \
--docker-password=${REPO_PASS} \
--docker-email=${REPO_USER}
```
For details, see the Kubernetes article [Pull an Image from a Private Registry](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/).
8. Update the custom values YAML for your cluster to point to your private registry and secret to allow Kubernetes to pull images. For example:
```yaml theme={"dark"}
query-pipeline:
image:
imagePullSecrets:
- name:
repository:
```
Repeat the process for all Fusion services.
### Customize Helm Chart
Every Fusion service allows you to override the `imagePullSecrets` setting using custom values YAML. However, other 3rd party services--including Zookeeper, Pulsar, Prometheus, and Grafana--don’t allow you to supply the pull secret using the custom values YAML.
To patch the default service account for your namespace and add the pull secret, run the following:
```bash theme={"dark"}
kubectl patch sa default -n $NAMESPACE \
-p '"imagePullSecrets": [{"name": "" }]'
```
In Windows using PowerShell or another CLI, you might have to escape the double quotes with a backslash (`\`) or reverse the order of single and double quotes:
```bash theme={"dark"}
kubectl patch sa default -n $NAMESPACE \
-p "'imagePullSecrets': [{'name': ''}]"
```
Replace `` with the name of the secret you created in the steps above.
This allows the default service account to pull images from the private registry without specifying the pull secret on the resources directly.
## Add additional trusted certificate(s) to Fusion’s indexing and querying services *(optional)*
You can add custom trusted certificates to support Fusion’s indexing and querying services. You may want to use custom trusted certificates if, for example, you have specific security requirements for data handling or need to support an existing infrastructure and its security needs. This method involves updating your Helm chart.
If you want to add custom trusted certificates for both the indexing and querying services, follow these instructions twice: once for the indexing service, and once for the querying service. To add different certificates to the indexing and querying services, create one YAML file with the indexing service certificates and one YAML file for the querying service certificates before following these instructions.
You may use the same YAML file if you want to use the same certificates for both services.
To add custom trusted certificates:
1. Create a new YAML file for your custom trusted certificates. The `customcerts.yaml` file is the example file in these instructions.
2. Add the custom certificate(s) in the YAML file created in the previous step. For example:
```yaml theme={"dark"}
trustedCertificates:
enabled: true
files:
some.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE-----
other.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE---------
```
3. Update the indexing or querying service by running the following Helm command. Replace `EXAMPLE-VALUES-FILE.yaml` with your previous values file.
```bash theme={"dark"}
helm upgrade --install --namespace ${EXAMPLE-NAMESPACE} ${HELM-RELEASE} ${HELM-CHART-PATH} --values EXAMPLE-VALUES-FILE.yaml --values customcerts.yaml
```
4. Verify the indexing or querying pod has a new `init-container` with the name `import-certs`.
## Add additional trusted certificate(s) for connectors to allow crawling of web resources with SSL/TLS enabled *(optional)*
To crawl a datasource which for some reason is using a self-signed certificate, add arbitrary certificates to connectors. For example:
```yaml wrap expandable theme={"dark"}
classic-rest-service:
trustedCertificates:
enabled: true
files:
some.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE-----
other.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE---------
connector-plugin:
trustedCertificates:
enabled: true
files:
some.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE-----
other.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE---------
```
### Generating the certificate on linux command line
Use the following command to generate a `.crt` file in `$fusion_home/apps/jetty/connectors/etc/yourcertname.crt`:
```bash theme={"dark"}
openssl s_client -servername remote.server.net -connect remote.server.net:443 $fusion_home/apps/jetty/connectors/etc/yourcertname.crt
```
### Generating the certificate using Firefox web browser
1. Navigate to the SharePoint host.
2. Click the in the address bar, then click the icon.
3. Next, navigate to **More Information** > **View Certificate** > **Export**.\
Save the file to the following folder:
`$fusion_home/apps/jetty/connectors/etc/yourcertname.crt`
### Generating the certificate using Chrome web browser
1. Navigate to **Chrome menu** > **More Tools** > **Developer Tools** > **Security Tab**.
This will display the **Security overview**.
2. Click the **View certificate** button.
3. Save the file to the following folder:
`$fusion_home/apps/jetty/connectors/etc/yourcertname.crt`
### Generating the certificate using powershell
Use the following script to generate a `.crt`` file in `\$fusion\_home\apps\jetty\connectors\etc\yourcertname.crt\`\`:
```bash theme={"dark"}
$fusion_home = c:\your\fusion\install\directory
$webRequest = [Net.WebRequest]::Create("https://your-hostname")
try { $webRequest.GetResponse() } catch {}
$cert = $webRequest.ServicePoint.Certificate
$bytes = $cert.Export([Security.Cryptography.X509Certificates.X509ContentType]::Cert)
set-content -value $bytes -encoding byte -path "$fusion_home\apps\jetty\connectors\etc\yourcertname.binary.crt"
certutil -encode "$fusion_home\apps\jetty\connectors\etc\yourcertname.binary.crt" "$fusion_home\apps\jetty\connectors\etc\yourcertname.crt"
rm "$fusion_home\apps\jetty\connectors\etc\yourcertname.binary.crt" -f
```
## Install Fusion 5 on Kubernetes
At this point, you’re ready to install Fusion 5 using the custom values YAML files and upgrade script. If you used the `customize_fusion_values.sh` script, run it using BASH:
```bash theme={"dark"}
./gke_search_f5_upgrade_fusion.sh
```
Once the installation is complete, verify your Fusion installation is running correctly.
## Monitoring Fusion with Prometheus and Grafana
Lucidworks recommends using Prometheus and Grafana for monitoring the performance and health of your Fusion cluster. Your operations team may already have these services installed. If not, install them into the Fusion namespace.
The [Custom values YAML file shown above](#custom-values-yaml-file) activates the Solr metrics exporter service and adds pod annotations so Prometheus can scrape metrics from Fusion services.
1. Run the `customize_fusion_values.sh` script with the `--prometheus true` option. This creates an extra custom values YAML file for installing Prometheus and Grafana, `___monitoring_values.yaml`. For example: `gke_search_f5_monitoring_values.yaml`.
2. Commit the YAML file to version control.
3. Review its contents to ensure that the settings suit your needs. For example, decide how long you want to keep metrics. The default is 36 hours.\
See the [Prometheus documentation](https://github.com/helm/charts/tree/master/stable/prometheus) and [Grafana documentation](https://github.com/helm/charts/tree/master/stable/grafana) for details.
4. Run the `install_prom.sh` script to install Prometheus & Grafana in your namespace. Include the provider, cluster name, namespace, and helm release as in the example below:
```bash theme={"dark"}
./install_prom.sh --provider gke -c search -n f5 -r 5-5-1
```
Pass the `--help` parameter to see script usage details.
The Grafana dashboards from [monitoring/grafana](https://github.com/lucidworks/fusion-cloud-native/tree/master/monitoring/grafana) are installed automatically by the `install_prom.sh` script.
## Upgrade Fusion on Kubernetes
Upgrade information about Kubernetes is included in the **Fusion 5 Upgrades** topic.
This guide describes how to perform Fusion 5 upgrades.
Before upgrading, be aware of changes by checking for [Deprecations and Removals](/docs/5/fusion/deprecations-and-removals) between versions.
Lucidworks recommends upgrading to the next minor version only. For example, you should upgrade from Fusion 5.6.1 to Fusion 5.7.1 before upgrading to Fusion 5.8.0.
The [general upgrade process](#general-upgrade-process) is described in this article. However, the specific upgrade procedures may vary depending on your upgrade path. For the most accurate instructions, please refer to the upgrade article specific to your upgrade.
## General upgrade process
Fusion natively supports deployments on supported Kubernetes platforms, including AKS, EKS, and GKE.
Fusion includes an upgrade script for AKS, EKS, and GKE. This script is not generated for other Kubernetes deployments.
Upgrades differ from platform to platform. See below for more information about upgrading on your platform of choice.
Whenever you upgrade Fusion, you must also update your [remote connectors](/docs/fusion-connectors/developers/remote-v2-connectors), if you are running any.
You can download the latest files at [V2 Connectors Downloads](/docs/fusion-connectors/downloads/v2-connectors-downloads).
### Natively supported deployment upgrades
| Deployment type | Platform |
| - | - |
| **Azure Kubernetes Service (AKS)** | `aks` |
| **Amazon Elastic Kubernetes Service (EKS)** | `eks` |
| **Google Kubernetes Engine (GKE)** | `gke` |
Fusion includes upgrade scripts for natively supported deployment types. To upgrade:
1. Open the `___upgrade_fusion.sh` upgrade script file for editing.
2. Update the `CHART_VERSION` to your target Fusion version, and save your changes.
3. Run the `___upgrade_fusion.sh` script. The `` value is the same as your namespace, unless you overrode the default value using the `-r` option.
After running the upgrade, use `kubectl get pods` to see the changes applied to your cluster. It may take several minutes to perform the upgrade, as new Docker images are pulled from DockerHub. To see the versions of running pods, do:
```
kubectl get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq
```
### Other Kubernetes deployment upgrades
To update an existing installation, do:
```
RELEASE=f5
NAMESPACE=default
helm repo update
helm upgrade ${RELEASE} "lucidworks/fusion" --namespace "${NAMESPACE}" --values "${MY_VALUES}"
```
Except for ZooKeeper, all K8s deployments and statefulsets use a `RollingUpdate` update policy:
```yaml theme={"dark"}
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
```
ZooKeeper instances use `OnDelete` to avoid changing critical stateful pods in the Fusion deployment. To apply changes to Zookeeper after performing the upgrade (uncommon), you need to manually delete the pods. For example:
```
kubectl delete pod f5-zookeeper-0
```
Delete one pod at a time. Verify the new pod is healthy and serving traffic, before deleting the next healthy pod.
You can also set the `updateStrategy` under the `zookeeper` section in your `"${MY_VALUES}"` file:
```yaml theme={"dark"}
solr:
...
zookeeper:
updateStrategy:
type: "RollingUpdate"
```
#### Upgrades with Helm v3
One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. For example, Fusion 5 allows customers to upgrade from Fusion 5.1.0 to a later 5.x.y version on a live cluster with zero downtime or disruption of service.
When Kubernetes performs a rolling update to an individual microservice, there is a mix of old and new services in the cluster. Requests from other services route to both versions.
Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same minor release version (5.x). We also ensure that the stored configuration remains compatible in the same minor release version.
Lucidworks releases minor updates to individual services frequently. Pull in those upgrades using Helm at your discretion.
**How to upgrade Fusion**
1. Clone the [**fusion-cloud-native** repo](https://github.com/lucidworks/fusion-cloud-native), if you haven’t already.
2. Locate the `setup_f5_.sh` script that matches your Kubernetes platform.
3. Run the script with the `--upgrade` option.
To see what would be upgraded, pass the `--dry-run` option to the script.
The scripts in the **fusion-cloud-native** repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.
#### Helm upgrade script
Once you deploy a working cluster, use the upgrade script created by the `customize_fusion_values.sh` script. The upgrade script hard-codes the parameters and eases the need to remember which parameters to pass to the script. This is helpful when working with multiple K8s clusters. Make sure you check the script into version control alongside your custom values YAML files.
Whenever you change the custom values YAML files for your cluster, you need to run the upgrade script to apply the changes. The script calls `helm upgrade` with the correct parameters and `--values` options.
If you run `helm upgrade` without passing the custom values YAML files, the deployment will revert to using chart defaults, which you never want to do.The script assumes your `kubeconfig` is pointing to the correct cluster and you’re using Heml v3. If not, the upgrade fails. Select the correct `kubeconfig` before running the script.
## Learn more
1. Log in to the Fusion UI.
2. In the upper right, open the profile menu and select **License Details**:
The License Details window appears:
3. Click **Choose License** and select your license file.
4. Click **Upload**.
Lucidworks recommends isolating search workloads from analytics workloads using multiple node pools. The included scripts do not do this for you; this is a manual process.
1. Run the `customize_fusion_values.sh` script with the following arguments:
```bash highlight={2,3,5} theme={"dark"}
./customize_fusion_values.sh \
-c demo \
-n fusion-lw \
--provider k8s \
--node-pool {} \
--with-resource-limits \
--with-affinity-rules \
--with-replicas
```
1. The name of your cluster. This example uses `demo`.
2. The name of your namespace.
3. By using the empty value `{}` as the value for the `--node-pool` argument, the script leaves the `nodePools` values unassigned. Assign these values in a later step.\
The example above outputs the following:
```bash theme={"dark"}
Created Fusion custom values yaml: k8s_ demo_fusion-lw_fusion_values.yaml
Created Monitoring custom values yaml: k8s_ demo_fusion-lw_monitoring_values.yaml. Keep this file handy as you'll need it to customize your Monitoring installation.
Create k8s_ demo_fusion-lw_upgrade_fusion.sh for upgrading you Fusion cluster. Please keep this script along with your custom values yaml file(s) in version control.
Update nodeSelector values for k8s_ demo_fusion-lw_fusion_values.yaml and update the solr to use nodepool
```
2. Update Solr in `k8s_demo_fusion-lw_fusion_values.yaml` to add `nodePools`. The `fusion_node_type` value connects the Solr `nodePools` values to the corresponding node pool.
```yaml highlight={3,4,9,17,22,31} theme={"dark"}
solr:
nodePools:
- name: ""
- name: "analytics"
javaMem: "-Xmx6g"
replicaCount: 6
storageSize: "100Gi"
nodeSelector:
fusion_node_type: analytics
resources:
requests:
cpu: 2
memory: 12Gi
limits:
cpu: 3
memory: 12Gi
- name: "search"
javaMem: "-Xms11g -Xmx11g"
replicaCount: 12
storageSize: "50Gi"
nodeSelector:
fusion_node_type: search
resources:
limits:
cpu: "7700m"
memory: "26Gi"
requests:
cpu: "7000m"
memory: "25Gi"
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
...
```
1. The empty string `""` is the suffix for the default partition.
2. Overrides the settings for the **analytics** Solr pods.
3. Assigns the **analytics** Solr pods to the node pool and attaches the label `fusion_node_type=analytics`. You can use the `fusion_node_type` property in Solr auto-scaling policies to govern replica placement during collection creation.
4. Overrides the settings for the **search** Solr pods
5. Assigns the **search** Solr pods to the node pool and attaches the label `fusion_node_type=search`.
6. Sets the default settings for all Solr pods, if not specifically overridden in the `nodePools` section above.
Do not edit the `nodePools` value `""`.Only use a single `fusion_node_type` value per Solr `nodeSelector` value.
1. Update `k8s_demo_fusion-lw_fusion_values.yaml` to add appropriate `nodeSelector` values for all desired services.
2. Run the `upgrade_fusion.sh` script:
```bash theme={"dark"}
./k8s_demo_fusion-lw_upgrade_fusion.sh
```
The output resembles the following:
```bash theme={"dark"}
namespace/fusion-lw created
Created namespace fusion-lw with owner label
Hang tight while we grab the latest from your chart repositories...
...
NAME: fusion-lw
LAST DEPLOYED: Wed Sep 15 18:25:55 2021
NAMESPACE: fusion-lw
STATUS: deployed
REVISION: 1
...
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
fusion-lw fusion-lw 1 2021-09-15 18:25:55.1470127 -0500 CDT deployed fusion-5.4.2 5.4.2
```
The `...` indicates some parts of the output are removed to shorten the example response.
3. Install Prometheus and Grafana so you can monitor Fusion metrics by running the following:
```bash theme={"dark"}
./install_prom.sh --provider k8s -c demo -n fusion-lw
```
The output resembles the following:
```bash theme={"dark"}
Adding the stable chart repo to helm repo list
"stable" has been added to your repositories
Installing Prometheus and Grafana for monitoring Fusion metrics ... this can take a few minutes.
...
NAME: fusion-lw-monitoring
LAST DEPLOYED: Wed Sep 15 22:59:28 2021
NAMESPACE: fusion-lw
STATUS: deployed
REVISION: 1
NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
fusion-lw-monitoring-prometheus-server.fusion-lw.svc.cluster.local
...
Successfully installed Prometheus and Grafana into the fusion-lw namespace.
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
fusion-lw fusion-lw 1 2021-09-15 18:25:55.1470127 -0500 CDT deployed fusion-5.4.2 5.4.2
fusion-lw-monitoring fusion-lw 1 2021-09-15 22:59:28.9054416 -0500 CDT deployed fusion-monitoring-1.0.0 1.0.0
```
4. Update the Solr collections to use specific nodes. Apply the Solr policy and run `update_app_coll_layout.sh`. This script deletes the collections and recreates them with the new policy. Check the scripts for details on collection names.\
\\
# Fusion 5 Frequently Asked Questions
Source: https://doc.lucidworks.com/docs/5/fusion/operations/faq
[localhost link]: http://localhost:3000/docs/5/fusion/operations/faq
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/faq
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/oa0map
**Does Fusion use Custom Resource Definitions?**
Yes, Fusion 5 does utilize custom resource definitions. These CRDs are part of the install and are critical to the application. All CRDs are namespaced, following Kubernetes best practices for utilizing CRDs in a multi-tenant environment.
**Are any services installed at the cluster level?**
Yes, two services, Argo and Seldon-core are installed at the cluster level. These services, which are part of the ML/AI service infrastructure for Fusion, use web hooks, which must be installed at the cluster level. These web hooks are namespaced, following best practices for utilizing web hooks in a multi-tenant environment.
**Can I install Fusion 5 without using your Helm chart?**
Yes, but we strongly recommend that you use the Helm chart to install if you want to leverage Lucidworks support.
**Are there backup jobs used in disaster recovery situations for stateful services deployed to Kubernetes?**
There is no out-of-the-box backup solution for stateful services provided with Fusion. Lucidworks recommends having a plan in place to reindex data from its primary source as part of an overall disaster recovery strategy.
## Other questions?
If you have a question about a resource not included here, please reach out to [Lucidworks support](https://support.lucidworks.com/)
# Fusion 5 Upgrades
Source: https://doc.lucidworks.com/docs/5/fusion/operations/fusion-5-upgrades
[localhost link]: http://localhost:3000/docs/5/fusion/operations/fusion-5-upgrades
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/fusion-5-upgrades
This guide describes how to perform Fusion 5 upgrades.
The [general upgrade process](#general-upgrade-process) is described in this article. However, the specific upgrade procedures may vary depending on your upgrade path. For the most accurate instructions, please refer to the upgrade article specific to your upgrade.
Fusion 5.9.x must be upgraded one maintenance release at a time. If you are on version 5.9.12 and want to reach 5.9.14, you are required to upgrade to 5.9.13 first, then upgrade to 5.9.14.
Fusion 5.9.5 introduced changes that affect Spark jobs. If you are upgrading to Fusion 5.9.5, follow the [Fusion 5.9.5 upgrade instructions](#to-5-9-5).
Before upgrading, be aware of changes by checking for [Deprecations and Removals](/docs/5/fusion/deprecations-and-removals) between versions.
## General upgrade process
Fusion natively supports deployments on supported Kubernetes platforms, including AKS, EKS, and GKE.
Fusion includes an upgrade script for AKS, EKS, and GKE. This script is not generated for other Kubernetes deployments.
Upgrades differ from platform to platform. See below for more information about upgrading on your platform of choice.
Whenever you upgrade Fusion, you must also update your [remote connectors](/docs/fusion-connectors/developers/remote-v2-connectors), if you are running any.
You can download the latest files at [V2 Connectors Downloads](/docs/fusion-connectors/downloads/v2-connectors-downloads).
Check for any running jobs prior to upgrading to maintain data integrity and prevent job state corruption.
1. Verify job status with the Connectors API:
```bash theme={"dark"}
curl GET https://FUSION_HOST/api/connectors/jobs?state=RUNNING
```
2. Complete document processing.
1. Wait for all pending documents to be indexed.
2. Verify no retry operations are pending.
3. Confirm index pipeline statistics show completion.
### Natively supported deployment upgrades
| Deployment type | Platform |
| - | - |
| **Azure Kubernetes Service (AKS)** | `aks` |
| **Amazon Elastic Kubernetes Service (EKS)** | `eks` |
| **Google Kubernetes Engine (GKE)** | `gke` |
Fusion includes upgrade scripts for natively supported deployment types. To upgrade:
1. Open the `___upgrade_fusion.sh` upgrade script file for editing.
2. Update the `CHART_VERSION` to your target Fusion version, and save your changes.
3. Run the `___upgrade_fusion.sh` script. The `` value is the same as your namespace, unless you overrode the default value using the `-r` option.
After running the upgrade, use `kubectl get pods` to see the changes applied to your cluster. It may take several minutes to perform the upgrade, as new Docker images are pulled from DockerHub. To see the versions of running pods, do:
```
kubectl get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq
```
#### Configset template upgrades
Introduced in Fusion 5.9.12
During upgrades, Fusion automatically updates Solr configset templates (system templates with names beginning with `_fusion_`) to ensure they match the version you're upgrading to.
The upgrade process uses an ownership file (`_fusion_owner.txt`) in each configset to determine whether it's safe to replace the template.
If you've customized a configset template and want to preserve your changes during upgrades, you have two options:
* Remove the `_fusion_owner.txt` file from the configset in ZooKeeper.
* Modify the `_fusion_owner.txt` file to contain a different value than the Fusion docker image owner.
When Fusion encounters a configset template with a missing or mismatched `_fusion_owner.txt` file, it skips that template and logs a warning, preserving your customizations.
For complete details about configset templates and how they work, see [Configset templates](/docs/5/fusion/intro/fusion-stack/solr/configset-templates).
### Other Kubernetes deployment upgrades
To update an existing installation, do:
```
RELEASE=f5
NAMESPACE=default
helm repo update
helm upgrade ${RELEASE} "lucidworks/fusion" --namespace "${NAMESPACE}" --values "${MY_VALUES}"
```
Except for ZooKeeper, all K8s deployments and statefulsets use a `RollingUpdate` update policy:
```yaml theme={"dark"}
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
```
ZooKeeper instances use `OnDelete` to avoid changing critical stateful pods in the Fusion deployment. To apply changes to Zookeeper after performing the upgrade (uncommon), you need to manually delete the pods. For example:
```
kubectl delete pod f5-zookeeper-0
```
Delete one pod at a time. Verify the new pod is healthy and serving traffic, before deleting the next healthy pod.
You can also set the `updateStrategy` under the `zookeeper` section in your `"${MY_VALUES}"` file:
```yaml theme={"dark"}
solr:
...
zookeeper:
updateStrategy:
type: "RollingUpdate"
```
#### Upgrades with Helm v3
One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. For example, Fusion 5 allows customers to upgrade from Fusion 5.1.0 to a later 5.x.y version on a live cluster with zero downtime or disruption of service.
When Kubernetes performs a rolling update to an individual microservice, there is a mix of old and new services in the cluster. Requests from other services route to both versions.
Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same minor release version (5.x). We also ensure that the stored configuration remains compatible in the same minor release version.
Lucidworks releases minor updates to individual services frequently. Pull in those upgrades using Helm at your discretion.
**How to upgrade Fusion**
1. Clone the [**fusion-cloud-native** repo](https://github.com/lucidworks/fusion-cloud-native), if you haven’t already.
2. Locate the `setup_f5_.sh` script that matches your Kubernetes platform.
3. Run the script with the `--upgrade` option.
To see what would be upgraded, pass the `--dry-run` option to the script.
The scripts in the **fusion-cloud-native** repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.
#### Helm upgrade script
Once you deploy a working cluster, use the upgrade script created by the `customize_fusion_values.sh` script. The upgrade script hard-codes the parameters and eases the need to remember which parameters to pass to the script. This is helpful when working with multiple K8s clusters. Make sure you check the script into version control alongside your custom values YAML files.
Whenever you change the custom values YAML files for your cluster, you need to run the upgrade script to apply the changes. The script calls `helm upgrade` with the correct parameters and `--values` options.
If you run `helm upgrade` without passing the custom values YAML files, the deployment will revert to using chart defaults, which you never want to do.The script assumes your `kubeconfig` is pointing to the correct cluster and you’re using Heml v3. If not, the upgrade fails. Select the correct `kubeconfig` before running the script.
## Learn more
This article includes instructions for upgrading Fusion from one version to another. In some cases, the instructions do not vary. Other upgrades require special instructions. Start by checking upgrade details for your target version before continuing to the [General upgrade process](#general-upgrade-process).
Remember, upgrade instructions may vary between deployment types too.
Whenever you upgrade Fusion, you must also update your remote connectors, if you are running any.
You can download the latest files at [V2 Connectors Downloads](https://doc.lucidworks.com/fusion-connectors/n0s114/v-2-connectors-downloads).
Fusion values change between releases. Check the [example values](https://github.com/lucidworks/fusion-cloud-native/blob/master/customize_fusion_values.yaml.example) and update values as needed.
## Upgrades from 5.8.x
### to 5.9.y
Upgrading from 5.8.x to 5.9.y involves using the [General upgrade process](#general-upgrade-process).
## General upgrade process
Fusion natively supports deployments on supported Kubernetes platforms, including AKS, EKS, and GKE.
Fusion includes an upgrade script for AKS, EKS, and GKE. This script is not generated for other Kubernetes deployments.
Upgrades differ from platform to platform. See below for more information about upgrading on your platform of choice.
Whenever you upgrade Fusion, you must also update your remote connectors.
You can download the latest files at [V2 Connectors Downloads](https://doc.lucidworks.com/fusion-connectors/n0s114/v-2-connectors-downloads).
### Natively supported deployment upgrades
| Deployment type | Platform |
| - | - |
| **Azure Kubernetes Service (AKS)** | `aks` |
| **Amazon Elastic Kubernetes Service (EKS)** | `eks` |
| **Google Kubernetes Engine (GKE)** | `gke` |
Fusion includes upgrade scripts for natively supported deployment types. To upgrade:
1. Open the `___upgrade_fusion.sh` upgrade script file for editing.
2. Update the `CHART_VERSION` to your target Fusion version, and save your changes.
3. Run the `___upgrade_fusion.sh` script. The `` value is the same as your namespace, unless you overrode the default value using the `-r` option.
After running the upgrade, use `kubectl get pods` to see the changes applied to your cluster. It may take several minutes to perform the upgrade, as new Docker images are pulled from DockerHub. To see the versions of running pods, do:
```bash wrap theme={"dark"}
kubectl get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq
```
This article includes instructions for upgrading Fusion from one version to another. Some upgrades require special instructions. Start by checking upgrade details for your target version before continuing to the [General upgrade process](#general-upgrade-process).
Fusion 5.9.x must be upgraded one maintenance release at a time. If you are on version 5.9.12 and want to reach 5.9.14, you are required to upgrade to 5.9.13 first, then upgrade to 5.9.14.
When you upgrade Fusion, you must also update your [remote connectors](/docs/fusion-connectors/developers/remote-v2-connectors), if you are running any.
You can download the latest files at [V2 Connectors Downloads](/docs/fusion-connectors/downloads/v2-connectors-downloads).
Fusion values change between releases. Check the [example values](https://github.com/lucidworks/fusion-cloud-native/blob/master/customize_fusion_values.yaml.example) and update values as needed.
## Upgrades from 5.9.x
### to 5.9.y
Upgrading from 5.9.x to 5.9.y involves using the [General upgrade process](#general-upgrade-process).
If you are using horizontal pod autoscaling, follow the [steps in the Fusion 5.8.1 release notes](/docs/5/fusion/release-notes/more/5.8.1-release-notes). If you have already done this as part of a previous upgrade, you can skip this process.
### to 5.9.16
If you use Seldon models or Smart Answers, which depends on Seldon, and override `JAVA_TOOL_OPTIONS` or `javaToolOptions` in your Helm values for `ml-model-service`, `fusion-query`, or `fusion-indexing`, the required protobuf flag is lost when you upgrade to Fusion 5.9.16. Without this flag, Seldon model serving and Smart Answers will fail.
To avoid this:
1. Locate where you set `JAVA_TOOL_OPTIONS` or `javaToolOptions` in your Helm values.
2. Add `-Dcom.google.protobuf.use_unsafe_pre22_gencode` to your Java options.
3. Validate Seldon models or Smart Answers after upgrade.
### to 5.9.12
When upgrading to Fusion 5.9.12, add the following to your `values.yaml` file to avoid a known issue that prevents the `kuberay-operator` pod from launching successfully:
```yaml theme={"dark"}
kuberay-operator:
crd:
create: true
```
### to 5.9.5
Upgrading to Fusion 5.9.5 has special considerations, due to changes introduced with Solr 9.6.1 and Lucene 9.10.0. All upgrades are susceptible to issues from these changes. Follow the upgrade procedures closely to avoid issues with the upgrade.
See the following sections for an overview of the issues, or [skip to the upgrade process](#upgrade-process).
#### Solr 9.6.1 changes
Prior to Fusion 5.9.5, Fusion utilized Solr 9.1.1 or earlier. Due to changes in Solr 9.3, some Solr configuration and collection configurations are no longer compatible. As Fusion 5.9.5 leverages Solr 9.6.1, it’s imperative to address these compatibility issues during the upgrade process.
To address Solr and collection configuration issues, a Docker utility called `fm-upgrade-apps-to-solr-9`, also known as the Fusion migration script, is included in the Fusion 5.9.5 release. This utility performs the following tasks:
* Removes the unused configuration, ``, from `solrconfig.xml`. Solr no longer supports this configuration.
* Removes the query response writer of class `solr.XSLTResponseWriter`.
* Comments out processors of type `solr.StatelessScriptUpdateProcessorFactory`.
* Removes `` element from request handler.
* Changes cache class attribute of elements `"filterCache"`, `"cache"`, `"documentCache"`, `"queryResultCache"` to `solr.search.CaffeineCache`.
* Removes `keepShortTerm` attribute from filter of class `solr.NGramFilterFactory`.
* Updates collection configurations, as needed.
#### Lucene 9.10.0 changes
A Lucene update to 9.10.0 in Fusion 5.9.5 may cause issues with certain collections in Solr. The change to the FST posting format codec (from `Lucene90PostingsWriterDoc` to `Lucene99PostingsWriterDoc`) in Lucene is incompatible with Solr in Fusion. As a result, Solr will not open a new searcher for collections using the `FST50` postings format.
To identify collections potentially affected by the Lucene codec change, examine the field definitions within your Solr schema. Look for fields that specify the `postingsFormat` attribute with a value of `FST50`. Collections containing such fields may experience compatibility issues. For example:
```xml theme={"dark"}
```
The following log excerpt demonstrates a typical error message encountered when an upgrade is impacted by the codec change:
```java wrap expandable theme={"dark"}
Caused by: org.apache.lucene.index.CorruptIndexException: codec mismatch: actual codec=Lucene90PostingsWriterDoc vs expected codec=Lucene99PostingsWriterDoc (resource=ByteBufferIndexInput(path="/var/solr/data/acme_query_rewrite_staging_shard1_replica_t9/data/index/_cn_FST50_0.doc"))
at org.apache.lucene.codecs.CodecUtil.checkHeaderNoMagic(CodecUtil.java:205) ~[?:?]
at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:194) ~[?:?]
at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:254) ~[?:?]
at org.apache.lucene.codecs.lucene99.Lucene99PostingsReader.(Lucene99PostingsReader.java:80) ~[?:?]
at org.apache.lucene.codecs.memory.FSTPostingsFormat.fieldsProducer(FSTPostingsFormat.java:60) ~[?:?]
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.(PerFieldPostingsFormat.java:330) ~[?:?]
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:392) ~[?:?]
at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:119) ~[?:?]
at org.apache.lucene.index.SegmentReader.(SegmentReader.java:96) ~[?:?]
at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:178) ~[?:?]
at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:220) ~[?:?]
at org.apache.lucene.index.IndexWriter.lambda$getReader$0(IndexWriter.java:542) ~[?:?]
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:138) ~[?:?]
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:604) ~[?:?]
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:112) ~[?:?]
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:91) ~[?:?]
at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:38) ~[?:?]
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2399) ~[?:?]
```
To account for the `postingsFormat="FST50"` codec issue, a Docker utility called `run-the-fm-upgrade-query-rewrite-docker-utility` is provided alongside the Fusion 5.9.5 release. You can pull this image from Docker using `docker pull lucidworks/run-the-fm-upgrade-query-rewrite-docker-utility:2.x`.
This utility performs two actions: `prepare` and `restore`. Use the `prepare` action before the Fusion 5.9.5 upgrade begins. At a high level, the `prepare` action performs the following actions:
1. Removes `postingsFormat="FST50"` from all collections in the environment.
2. Re-indexes documents to new, temporary collections.
3. Compares the original collections to the new, temporary collections to ensure data integrity.
Use the `restore` action after the Fusion 5.9.5 upgrade finishes, which must include the Solr 9.6.1 upgrade. The `restore` action performs the following actions:
1. Restores `postingsFormat="FST50"` to all collections in the environment that were changed with the `prepare` action.
2. Re-indexes documents to new, permanent collections. These collections match the original collections that were in place prior to the `prepare` action.
3. Compares the restored collections to the temporary collections to ensure data integrity.
#### Upgrade process
This section provides a high-level overview of the steps involved in upgrading to Fusion 5.9.5. Follow each step in the order given:
1. [Create a full backup of all Fusion collections](#back-up-your-solr-collections). These backups are intended as an emergency failsafe only.
2. [Run the `fm-upgrade-apps-to-solr-9` Docker utility](#run-the-fm-upgrade-apps-to-solr-9-docker-utility). This updates the Solr configuration and collections for compatibility with Solr 9.6.1.
3. [Run the `run-the-fm-upgrade-query-rewrite-docker-utility` Docker utility](#run-the-fm-upgrade-query-rewrite-docker-utility). Use the `prepare` action to address potential collection compatibility issues with Lucene 9.10.0 codecs.
4. [Upgrade your Fusion environment to version 5.9.5](#general-upgrade-process). Use the upgrade scripts or your own upgrade process.
5. [Re-run the `run-the-fm-upgrade-query-rewrite-docker-utility` Docker utility](#re-run-the-fm-upgrade-query-rewrite-docker-utility). Use the `restore` action to restore collections to their original state.
6. [Validate the upgrade was successful](#validate-the-upgrade). In addition to your usual validations, there are some extra things to check.
To mitigate potential upgrade issues, adhere to the following procedures.
#### Back up your Solr collections
Back up all Solr collections in each environment before continuing with the upgrade. For this upgrade, backups are intended as an emergency failsafe.
Performing a rollback after encountering the issue described is a difficult, time-consuming process and is not recommended.
#### Run the `fm-upgrade-apps-to-solr-9` Docker utility
Use the `fm-upgrade-apps-to-solr-9` Docker utility to mitigate issues related to the change from Solr 9.1.1 and earlier to Solr 9.6.1. To begin, run the `fm-upgrade-apps-to-solr-9` Docker utility using the `DRY_RUN` environmental variable:
```bash theme={"dark"}
docker run --rm -v $(pwd):/upgrade-work -e ZK_HOST=zk:2181 -e DRY_RUN=1 lucidworks/fm-upgrade-apps-to-solr-9:1.2.0
```
The `DRY_RUN` variable prints the changes that would occur to the console without performing those actions. Review the changes thoroughly.
If the changes look correct, run the `fm-upgrade-apps-to-solr-9` Docker utility again without using the `DRY_RUN` environmental variable. The updated config files are saved to the `/upgrade-work/updated-configs` directory. The utility also creates backups for all configs in the `/upgrade-work/backup-configs`.
The `fm-upgrade-apps-to-solr-9` Docker utility has another environmental variable, `REVERT`, that allows you to revert any changes you made. To revert your changes, run:
```bash theme={"dark"}
docker run --rm -v $(pwd):/upgrade-work -e ZK_HOST=zk:2181 -e REVERT=1 lucidworks/fm-upgrade-apps-to-solr-9:1.2.0
```
#### Run the `fm-upgrade-query-rewrite` Docker utility
Next, mitigate codec issues related to the Lucene 9.10.0 update. Run the `run-the-fm-upgrade-query-rewrite-docker-utility` Docker utility `prepare` action:
```bash theme={"dark"}
kubectl run \
--image="lucidworks/fm-upgrade-query-rewrite:2.x" \
--restart=Never \
--env="HELM_RELEASE=FUSION_NAMESPACE" \
--env="TARGET_SOLR_VERSION=9.6.1" \
--env="ACTION=prepare" prepare-595-upgrade
--namespace=FUSION_NAMESPACE
```
Change `FUSION_NAMESPACE` to the name of your application namespace for the Fusion installation. You can find this value using `helm list` against your Fusion namespace. Locate the release using the `fusion` chart, and find the value in the `name` column. Typically, the release name is the same as your namespace name.
Including `--namespace=FUSION_NAMESPACE` lets the update pod runs in the correct application namespace.
The `prepare` action removes `postingsFormat="FST50"` from all collections in the environment before re-indexing data to temporary collections. When the `prepare-595-upgrade` pod shows the status `Completed`, the process is finished.
#### Upgrade your Fusion environment
Upgrade Fusion to version 5.9.5. Before beginning, ensure the Fusion admin is running and all collections are healthy. Then, [complete the General upgrade process](#general-upgrade-process) before returning to the [next step in the process](#re-run-the-fm-upgrade-query-rewrite-docker-utility).
Alternatively, your organization may use a custom upgrade process. In either case, ensure you have successfully upgraded to Solr 9.6.1 as part of the Fusion upgrade.
**Do not** make changes to the `signals` collection with the Rules Editor during the upgrade process. For production clusters, upgrade during a maintenance window.
#### Re-run the `fm-upgrade-query-rewrite` Docker utility
Use the `run-the-fm-upgrade-query-rewrite-docker-utility` utility’s `restore` action to restore the data from the temporary collections created by the `prepare` action. Before you begin, verify all collections appended with `_temp_fix` are online and healthy.
```bash theme={"dark"}
kubectl run \
--image="lucidworks/fm-upgrade-query-rewrite:2.x" \
--restart=Never \
--env="HELM_RELEASE=FUSION_NAMESPACE" \
--env="TARGET_SOLR_VERSION=9.6.1" \
--env="ACTION=restore" restore-595-upgrade
--namespace=FUSION_NAMESPACE
```
Change `FUSION_NAMESPACE` to the name of your application namespace for the Fusion installation. You can find this value using `helm list` against your Fusion namespace. Locate the release using the `fusion` chart, and find the value in the `name` column. Typically, the release name is the same as your namespace name.
Including `--namespace=FUSION_NAMESPACE` lets the update pod runs in the correct application namespace.
When the `restore-595-upgrade` pod shows the status `Completed`, the process is finished.
For a complete summary of what this action does, refer to [Upgrade Utility](#run-the-fm-upgrade-query-rewrite-docker-utility).
#### Validate the upgrade
In addition to your typical validation process, ensure Solr collections are healthy:
1. Log into Fusion as the admin.
2. Access the Solr Admin UI at `https://FUSION_HOST:FUSION_PORT/api/solrAdmin/default/#/`.
3. Watch for error messages. For example, the following message reports errors for the query rewrite staging collections Acme1, Acme2, Acme3, and Acme4:
4. Navigate to the **Cloud graph** screen at `https://FUSION_HOST:FUSION_PORT/api/solrAdmin/default/#/~cloud?view=graph`.
5. Review each collection for errors.
After you have completed validations, delete the temporary `prepare` and `restore` pods that were created by the upgrade utility:
```bash theme={"dark"}
kubectl delete po prepare-595-upgrade
kubectl delete po restore-595-upgrade
```
#### Resolving post-upgrade issues
If you followed the previous instructions, and you are still experiencing issues with the codec, you need to re-feed the affected data. Contact [Lucidworks Support](https://support.lucidworks.com/hc/en-us) for further support.
# Datasource Jobs
Source: https://doc.lucidworks.com/docs/5/fusion/operations/jobs-and-scheduling/datasource-jobs
[localhost link]: http://localhost:3000/docs/5/fusion/operations/jobs-and-scheduling/datasource-jobs
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/jobs-and-scheduling/datasource-jobs
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/186
A datasource job specifies the origin of the data imported and indexed into Fusion. Datasources include databases, files, and data obtained from websites or applications.
Sources of data can be uploaded to Fusion in **System > Blobs**.
In addition, you can create and configure the datasource job using the following methods:
* The [Index Workbench](/docs/5/fusion/intro/ui-tour/index-workbench). Navigate to **Indexing > Index Workbench > Configure Datasource**. You can upload a particular ZIP file or select a datasource type which includes supported Fusion connectors. The list is divided datasources already installed and datasources not yet installed. When you select an uninstalled connector from the list, the system installs the connector.
* [Datasources](/docs/fusion-connectors/concepts/datasources). Navigate to **Indexing** > **Datasources**. If you click **Add**, you can select from a list of supported Fusion connectors. The list is divided into connectors already installed and connectors not yet installed. When you select an uninstalled connector from the list, the system installs the connector.
* The [Connector Datasources API](/api-reference/datasource-configuration-v2-api/list-all-v2-datasources). You can also use these endpoints to manage the crawl database and view connector schema.
Running the datasource job obtains and indexes the data in Fusion based on configuration parameters including the ability to:
* Generate diagnostic logs
* Limit document crawl levels or the number of documents obtained and indexed
* Exclude or include files based on file extension, text or patterns in the document, and authentication methods
* Set recrawl rules
* Enter links where you want the crawl to begin
## Learn more
1. Create a datasource.
2. Navigate to **System** > **Scheduler**.
3. Select the datasource from the job list.
4. Click **New Schedule**.
5. Select and configure a trigger:
* **After Another Job Completes**\
Enter the job ID and job result that trigger this one.
* **Cron String**\
Enter a Quartz cron expression. See [the Quartz documentation](http://www.quartz-scheduler.org/api/2.3.0/org/quartz/CronExpression.html) for details.
* **Start + Interval**\
Enter a start date/time, an interval, and the interval units.
To start a datasource job, complete the following:
1. Navigate to **Collections > Jobs**.
2. Select the job you want to run and click **Run > Start**.
3. To view the job status information and its result, click **Job History**.
To stop a datasource job, complete the following:
1. After you start the job, you can click **Stop** in the Run window to stop the job.
If you want to stop the job using the API, execute the following command in the Jobs API:
```json wrap theme={"dark"}
curl -u USERNAME:PASSWORD \
-X POST \
-H "Content-Type: application/json" \
https://FUSION_HOST:FUSION_PORT/api/jobs/datasource:DATASOURCE_ID/actions \
-d '{"action": "abort"}'
```
For more information, see [Start/pause/abort a job](/api-reference/jobs/startpauseabort-a-job).
Parsing and indexing of documents emitted prior to clicking stop are not stopped, so if you need to stop all ingestion activities, you must also complete the rest of the steps in this procedure.
2. To stop parsing, execute the command to delete the applicable parser in the Parsers CRUD API:
```json wrap theme={"dark"}
curl -u USERNAME:PASSWORD \
-X DELETE \
https://FUSION_HOST:FUSION_PORT/api/parser/datasource/DATASOURCE_ID
```
For more information, see [Delete a parser](/api-reference/parsers-crud-api/delete-a-parser).
3. To cancel Lucidworks AI requests, execute the following command in the Index Pipelines API:
```json wrap theme={"dark"}
curl -u USERNAME:PASSWORD \
-X POST \
https://FUSION_HOST:FUSION_PORT/api/index-pipelines/INDEX_PROFILE_ID/async-enrichment/skip-pending
```
For more information, see [Skip pending async requests](/api-reference/index-pipelines-api/skip-pending-async-requests).
4. To reset the index subscription, execute the following command in the Subscriptions API:
```json wrap theme={"dark"}
curl -u USERNAME:PASSWORD \
-X POST \
https://FUSION_HOST:FUSION_PORT/api/subscriptions/SUBSCRIPTION_ID/refresh?action=reset
```
For more information, see [Refresh a subscription](/api-reference/subscriptions-api/refresh-a-subscription).
# Job History
Source: https://doc.lucidworks.com/docs/5/fusion/operations/jobs-and-scheduling/job-history
[localhost link]: http://localhost:3000/docs/5/fusion/operations/jobs-and-scheduling/job-history
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/jobs-and-scheduling/job-history
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/182
## Checking a job status
### With the API
You can see the current job states of all datasources for the app `myApp` by calling the Jobs API:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD http://proxy-url:6764/api/apps/myApp/jobs?type=datasource
```
If a job has never run, a Jobs API call returns the following:
```json wrap theme={"dark"}
[
{
"resource": "datasource:myDatasource",
"enabled": true,
"startedBy": "09765f1d-2f1c-4c3b-ba22-887e3886d8aa",
"status": "ready",
"extra": {}
}
]
```
The value **`datasource:myDatasource`** indicates a datasource, **`myDatasource`**, is present. The line **`"status": "ready"`** indicates the current state of the job, which has never run.
After a job successfully completes, the following is returned:
```json wrap theme={"dark"}
[
{
"resource": "datasource:myDatasource",
"enabled": true,
"startedBy": "09765f1d-2f1c-4c3b-ba22-887e3886d8aa",
"status": "success",
"extra": {
"counter.failed": 0,
"counter.other.pipeline.in": 58242,
"counter.stage.field-mapping::d3ebd921-6065-479e-9d45-0bfed819346b.processed": 58241,
"counter.deleted": 0,
"counter.stage.solr-dynamic-field-name-mapping::e6aa9c51-2867-40a0-97fd-34d909098a20.processed": 58241,
"counter.new": 2268,
"counter.output": 58241,
"counter.other.pipeline.complete": 58242,
"datasourceId": "myDatasource",
"counter.input": 60435,
"startTime": 1586298855843,
"endTime": 1586299313433,
"counter.skipped": 2194,
"counter.stage.solr-index::01e1c561-0440-44b2-afb4-d6ff471f646c.processed": 58241,
"counter.other.pipeline.out": 58242
},
"lastStartTime": "2020-04-07T22:34:15.852Z",
"lastEndTime": "2020-04-07T22:41:59.271Z"
}
]
```
To see the complete job history, use the following Jobs API call:
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD http://proxy-url:6764/api/jobs/datasource:myDatasource/history
```
The following is returned:
```json wrap theme={"dark"}
[
{
"resource": "datasource:myDatasource",
"runId": "fXfKTQNSKM",
"startTime": "2020-04-07T22:34:15.852Z",
"endTime": "2020-04-07T22:41:59.271Z",
"status": "success",
"extra": {
"counter.failed": 0,
"counter.other.pipeline.in": 58242,
"counter.stage.field-mapping::d3ebd921-6065-479e-9d45-0bfed819346b.processed": 58241,
"counter.deleted": 0,
"counter.stage.solr-dynamic-field-name-mapping::e6aa9c51-2867-40a0-97fd-34d909098a20.processed": 58241,
"counter.new": 2268,
"counter.output": 58241,
"counter.other.pipeline.complete": 58242,
"datasourceId": "myDatasource",
"counter.input": 60435,
"startTime": 1586298855843,
"endTime": 1586299313433,
"counter.skipped": 2194,
"counter.stage.solr-index::01e1c561-0440-44b2-afb4-d6ff471f646c.processed": 58241,
"counter.other.pipeline.out": 58242
},
"startedBy": "09765f1d-2f1c-4c3b-ba22-887e3886d8aa"
},
{
"resource": "datasource:myDatasource",
"runId": "XxrxochVOB",
"startTime": "2020-04-07T20:06:25.819Z",
"endTime": "2020-04-07T20:14:01.686Z",
"status": "success",
"extra": {
"counter.failed": 0,
"counter.other.pipeline.in": 59090,
"counter.stage.field-mapping::d3ebd921-6065-479e-9d45-0bfed819346b.processed": 59089,
"counter.deleted": 0,
"counter.stage.solr-dynamic-field-name-mapping::e6aa9c51-2867-40a0-97fd-34d909098a20.processed": 59089,
"counter.new": 2268,
"counter.output": 59089,
"counter.other.pipeline.complete": 59090,
"datasourceId": "myDatasource",
"counter.input": 61283,
"startTime": 1586289985809,
"endTime": 1586290440433,
"counter.skipped": 2194,
"counter.stage.solr-index::01e1c561-0440-44b2-afb4-d6ff471f646c.processed": 59089,
"counter.other.pipeline.out": 59090
},
"startedBy": "09765f1d-2f1c-4c3b-ba22-887e3886d8aa"
},
{
"resource": "datasource:myDatasource",
"runId": "2jy3UemLTp",
"startTime": "2020-04-07T19:23:39.576Z",
"endTime": "2020-04-07T19:31:26.694Z",
"status": "success",
"extra": {
"counter.failed": 0,
"counter.other.pipeline.in": 58906,
"counter.stage.field-mapping::d3ebd921-6065-479e-9d45-0bfed819346b.processed": 58905,
"counter.deleted": 0,
"counter.stage.solr-dynamic-field-name-mapping::e6aa9c51-2867-40a0-97fd-34d909098a20.processed": 58905,
"counter.new": 2268,
"counter.output": 58905,
"counter.other.pipeline.complete": 58906,
"datasourceId": "myDatasource",
"counter.input": 61099,
"startTime": 1586287419556,
"endTime": 1586287883825,
"counter.skipped": 2194,
"counter.stage.solr-index::01e1c561-0440-44b2-afb4-d6ff471f646c.processed": 58905,
"counter.other.pipeline.out": 58906
},
"startedBy": "09765f1d-2f1c-4c3b-ba22-887e3886d8aa"
}
]
```
The section [How does the Jobs API work?](#how-does-the-jobs-api-work) below gives additional details on how the Jobs API retrieves information from Fusion. See the [Jobs API reference](/api-reference/get-job-histories/get-a-jobs-history) for more information.
#### Job history results field definitions
Job history counters vary between V1 (classic) connectors and V2 (plugin) connectors. V1 connectors use the `counter.*` prefix, while V2 connectors use different counter names without the prefix.
To determine if a datasource uses a V1 or V2 connector, check the `connectorType` field in the job history response. V1 connectors use types like `lucid.anda` or `lucid.fs`. V2 connectors use types like `lucidworks.*`, such as `lucidworks.file-upload` or `lucidworks.jdbc`.
**V1 Connector Counters**
| Field | Definition |
| - | - |
| **counter.deleted** | Total number of documents deleted or removed from the Solr index. |
| **counter.failed** | Total number of documents not indexed due to failure. The errors are written to the log. |
| **counter.input** | Total number of fetched documents passed to the index pipeline. |
| **counter.new** | Total number of documents fetched for the first time. |
| **counter.output** | Documents successfully sent to the Fusion indexing pipeline. |
| **counter.skipped** | Total number of documents skipped due to rules such as max, min, and file size. |
| **counter.other.pipeline.in** | Total number of documents entering the index pipeline. This includes parsed items. |
| **counter.other.pipeline.out** | Total number of documents exiting the index pipeline. |
| **counter.other.pipeline.complete** | Total number of documents that completed processing in the pipeline. |
| **endTime** | Time the job concluded, aborted, or was manually stopped. |
| **resource: datasource** | Name of the datasource. |
| **startTime** | Time the job started. |
| **status** | Job status such as SUCCESS, RUNNING, or ABORTED. |
| **runId** | Unique identifier for the specific job run. |
| **datasourceId** | The datasource configuration ID. |
**V2 Connector Counters**
V2 connectors (Java SDK-based) use a different counter format with more granular metrics. In addition, V2 connectors may include additional connector-specific counters depending on the connector type and implementation.
Pro connectors use the same framework as V2 connectors. For more information, see [What are Pro connectors?](/docs/fusion-connectors/latest-changes#what-are-pro-connectors).
The following fields apply to V2 connectors and Pro connectors.
| Field | Definition |
| - | - |
| **fetch.request** | Total number of fetch requests initiated by the connector. |
| **fetch.plugin-response** | Total number of fetch responses received from the connector plugin. |
| **fetch.plugin-response.document** | Total number of documents returned in fetch responses. |
| **pipeline.in** | Total number of documents entering the index pipeline. |
| **pipeline.out** | Total number of documents exiting the index pipeline. |
| **pipeline.complete** | Total number of documents that completed processing in the pipeline. |
| **pipeline.stages.STAGE\_ID.processed** | Number of documents processed by a specific pipeline stage, where `STAGE_ID` is the stage identifier. |
| **content-indexer.document.received.count** | Total number of documents received by the content indexer. |
| **content-indexer.completed.count** | Total number of documents that completed indexing. |
| **content-indexer.result.success.counter** | Total number of documents successfully indexed. |
| **start.request** | Indicates the job start was requested. |
| **start.plugin-response** | Indicates the connector plugin acknowledged the start request. |
| **stop.request** | Indicates the job stop was requested. |
| **stop.plugin-response** | Indicates the connector plugin acknowledged the stop request. |
| **endTime** | Time the job concluded, aborted, or was manually stopped. |
| **resource: datasource** | Name of the datasource. |
| **startTime** | Time the job started. |
| **state** | Job state such as FINISHED, RUNNING, or ABORTED. |
| **runId** | Unique identifier for the specific job run. |
| **configId** | The datasource configuration ID. |
The specific counters available in job history depend on the connector type, version, and configuration. Some connectors may provide additional metrics. Use the [Connector Jobs API](/api-reference/datasource-jobs-api/get-the-connector-job-status) to inspect the actual counters for your datasource.
### With the UI
In the Fusion UI, navigate to **Indexing** > **Datasources**. Click a datasource to open the datasource panel.
Note that the **Last run start** and **Last run stop** values are equivalent to the `lastStartTime` and `lastEndTime` values from the Jobs API call. (These values are only present if a job has been run at least once.)
| Icon | Description |
| - | - |
| | The **Run** button opens a panel containing various run options and information. |
| | The **Start** button starts a job. |
| | The **New Schedule** button displays a dropdown with several scheduling options: • **cron.** Schedule a job based on a Crontab expression. See [Cron Trigger documentation](http://www.quartz-scheduler.org). • **interval.** Schedule a job to run on an interval after a set starting point. For example, schedule a job to run every day at 1:00 AM. • **job\_completion.** Schedule a job to run after another job has completed. You can schedule the job to run if the job succeeds, fails, or regardless of success. |
| | The **Stop** button stops a running job. This results in a status of `"status": "aborted"`. |
| | The **Job History** button opens the [Job History panel](#with-the-ui). Click one of the job entries to see additional details. Expand the job details for more information. Click **Open Logs in Dashboard** for complete information. |
| | The **Clear Datasource** button removes documents from your collection that are indexed from that datasource. The job history will *not* be deleted. |
In the job history panel, there is an icon on each datasource indicating its latest run history. The possible statuses are:
| Icon | Description |
| - | - |
| Never Run | The job has been created but has not been run. |
| Running | The job is currently running. Depending on the number of documents being processed, the job can take a long time to complete. Stop the job with the **Stop** button. |
| Success | The job was completed successfully. |
| Failed | The job was not able to successfully run. Expand the job details for more information. Click **Open Logs in Dashboard** for complete information. |
| Aborted | The job was manually stopped before it could finish. |
The image above shows all possible job statuses.
## How does the Jobs API work?
In [Checking a job status with the API](#with-the-api) above, two Jobs API endpoints were used to call the datasource `myDatasource`:
* `myApp/jobs?type=datasource` - Retrieves the current datasource job states for *all* datasources of a specified app. In this case, the app is specified as `myApp`.\
You can also use `type=spark` to retrieve all Spark jobs and `type=task` for task jobs.
* `/jobs/datasource:myDatasource/history` - Retrieves a specific object’s job history. In this case, the object type is a datasource, as selected with `datasource:`.\
Datasources are not the only object type for jobs. There are also Spark jobs (`spark:`) and task jobs (`task:`).
Both of the API calls above send a request to Fusion’s API application, which serves the Jobs API. The API application contains two main components related to job history data, `JobController` and `SolrJobHistoryStore`.
### The Job Controller
The Job Controller is responsible for getting any active job’s run status, if an active job is running.
In the case of V1 connectors classic datasource jobs, it will reach out to Fusion’s connectors-classic application’s connectors API.
When the API application calls the connectors-classic Jobs API, it will first use Zookeeper to identify the IP address of the connector-classic node that is assigned to this datasource, if one exists.
When the connectors-classic node IP assigned to the job is identified, the API application will call out to the connectors-classic node via `http://:8984/connectors/v1/connectors/jobs/myDatasource`. The call reaches the component `ConnectorsManagerController` to query the job status of currently running jobs with the ID `myDatasource`.
### The Solr Job History Store
`SolrJobHistoryStore` reads and writes a job’s history data to and from Solr. Fusion’s API application creates a special system Solr collection, `system_jobs_history`, to store the history.
You can also query the job history from Solr directly, if you want:
# Jobs and Scheduling
Source: https://doc.lucidworks.com/docs/5/fusion/operations/jobs-and-scheduling/overview
[localhost link]: http://localhost:3000/docs/5/fusion/operations/jobs-and-scheduling/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/jobs-and-scheduling/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/184
A *job* is a runnable Fusion object that performs a specific activity. Datasource jobs, Spark jobs, and tasks are all types of jobs. Examples of the activities include:
* Aggregating signal data
* Training a shallow neural model and projecting each document onto the resulting vector embedding space
* Loading data into a data source
## Job types
Each job type is a type of Fusion object that you can run or schedule to be run.
| | |
| - | - |
| `datasource` | A job to ingest data according to the specified datasource configuration, such as `datasource:movie-db`. Datasources are created using the [Connector Datasources API](/api-reference/datasource-configuration-v2-api/list-all-v2-datasources) or the Fusion UI. See [Datasource Jobs](/docs/5/fusion/operations/jobs-and-scheduling/datasource-jobs). |
| `spark` | A Spark job to process data, such as `spark:dailyMetricsRollup-counters`. Spark jobs are created using the [Spark Jobs API](/api-reference/spark-job-controller-api/list-all-job-runs) or the Fusion UI. See [Spark Jobs](/docs/5/fusion/operations/jobs-and-scheduling/spark-jobs). |
| `task` | A job to perform an HTTP call or log cleanup, such as `task:delete-old-system-logs`. Tasks are created using the Fusion UI. See [Tasks](/docs/5/fusion/operations/jobs-and-scheduling/tasks). |
## The Jobs manager
The Jobs manager, available in the Fusion workspace at **Collections** > **Jobs**, provides a simple interface for viewing and scheduling jobs. You can also create tasks and Spark jobs here.
You cannot create, run, or schedule datasource jobs in the Jobs manager. You must create them on the Datasources page, in the Index Workbench, or in the Quickstart. To run a datasource job, use the [scheduler](/docs/5/fusion/operations/jobs-and-scheduling/schedules).
## Learn more
The quick learning for **Managing and Scheduling Jobs** focuses on how to create, configure, and schedule jobs using the Fusion UI.
The course for **Jobs and Scheduling** focuses on some of the most common jobs in Fusion.
1. In the Fusion workspace, navigate to **Collections > Jobs**.
2. Click **Add** and select the job type. The New Job Configuration panel appears. This panel is different for each job type.
3. Configure the new job as needed.
4. Click **Save**.
# Fusion Schedules
Source: https://doc.lucidworks.com/docs/5/fusion/operations/jobs-and-scheduling/schedules
[localhost link]: http://localhost:3000/docs/5/fusion/operations/jobs-and-scheduling/schedules
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/jobs-and-scheduling/schedules
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/185
A *schedule* defines when Fusion will perform a job.
You can schedule a job using `cron` notation, ISO-8601 interval definitions, or triggers that depend on the results of other jobs.
You can also run jobs on demand.
The [Jobs API](/api-reference/get-job-histories/get-a-jobs-history) lets you view jobs and their run histories, configure their schedules, and control them directly.
## The Scheduler
The Scheduler, available in the Fusion workspace at **System** > **Scheduler**, provides an interface for scheduling jobs.
You cannot create jobs in the Scheduler. To create new jobs, use the [Jobs manager](/docs/5/fusion/operations/jobs-and-scheduling/overview) or the REST API for each [job type](/docs/5/fusion/operations/jobs-and-scheduling/overview).
# Spark Jobs
Source: https://doc.lucidworks.com/docs/5/fusion/operations/jobs-and-scheduling/spark-jobs
[localhost link]: http://localhost:3000/docs/5/fusion/operations/jobs-and-scheduling/spark-jobs
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/jobs-and-scheduling/spark-jobs
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/187
Apache Spark can power a wide variety of data analysis jobs. In Fusion, Spark jobs are especially useful for generating [recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods).
## Spark job subtypes
For the Spark [job type](/docs/5/fusion/operations/jobs-and-scheduling/overview), the available subtypes are listed below.
* [SQL Aggregation job](/docs/5/fusion/reference/config-ref/jobs/aggregations/sql-aggregation)
A Spark SQL aggregation job where user-defined parameters are injected into a built-in SQL template at runtime.
* [Custom Python job](/docs/5/fusion/reference/config-ref/jobs/custom-python)
The Custom Python job provides user the ability to run Python code via Fusion. Python version requirements depend on your Spark version (Python 3.10 for Spark 3.4.1 and Python 3.7.3 for Spark 3.2.2).
* [Script](/docs/5/fusion/reference/config-ref/jobs/script)
This job lets you run a custom Scala script in Fusion.
## Spark version configuration
Fusion releases 5.9.10 and later use Spark 3.4.1 by default. In Fusion 5.9.12 and later, you can configure Spark to use version 3.2.2 for compatibility with legacy Python and Scala environments. See [Configure the Spark version](/docs/5/fusion/operations/survival-guide/spark-kubernetes-overview#configure-the-spark-version) for details.
See [Additional Spark jobs](/docs/5/fusion/reference/config-ref/jobs/overview) for more information.
## Spark job configuration
Spark jobs can be created and modified using the [Fusion UI](/docs/5/fusion/operations/jobs-and-scheduling/overview) or the [Spark Jobs API](/api-reference/spark-job-controller-api/list-all-job-runs). They can be scheduled using the Fusion UI or the [Spark Jobs API](/api-reference/spark-job-controller-api/list-all-job-runs).
For the complete list of configuration parameters for all Spark job subtypes, see the [Jobs Configuration Reference](/docs/5/fusion/reference/config-ref/jobs/overview).
## Machine learning jobs
Fusion provides these job types to perform machine learning tasks.
The course for **Machine Learning in Fusion** focuses on the basics about machine learning jobs in Fusion.
### Signals analysis
These jobs analyze a collection of signals in order to perform [query rewriting](#query-rewriting), [signals aggregation](#signals-aggregation), or [experiment analysis](#experiment-analysis).
* [Ground Truth](/docs/5/fusion/reference/config-ref/jobs/ground-truth)\
Ground truth or gold standard datasets are used in the ground truth jobs and query relevance metrics to define a specific set of documents.
Ground truth jobs estimate ground truth queries using click signals and query signals, with document relevance per query determined using a click/skip formula.
Use this job along with the [Ranking Metrics job](/docs/5/fusion/reference/config-ref/jobs/ranking-metrics) to calculate relevance metrics, such as Normalized Discounted Cumulative Gain (nDCG).
To create a ground truth job, sign in to Fusion and click **Collections > Jobs**. Then click **Add+** and in the Experiment Evaluation Jobs section, select **Ground Truth**. You can enter basic and advanced parameters to configure the job. If the field has a default value, it is populated when you click to add the job.
#### Query rewriting
These jobs produce data that can be used for query rewriting or to inform updates to the [synonyms.txt file](/docs/5/fusion/getting-data-out/query-enhancement/synonyms/overview).
These jobs are deprecated in Fusion 5.9.15 and will be removed in a future release.
Lucidworks recommends using [Neural Hybrid Search](/docs/5/fusion/hybrid-search/overview), which achieves superior relevance compared to legacy machine learning methods.
* [Head/Tail Analysis](/docs/5/fusion/reference/config-ref/jobs/head-tail-analysis)\
Perform head/tail analysis of queries from collections of raw or aggregated signals, to identify underperforming queries and the reasons. This information is valuable for improving overall conversions, Solr configurations, auto-suggest, product catalogs, and SEO/SEM strategies, in order to improve conversion rates.
* [Phrase Extraction](/docs/5/fusion/reference/config-ref/jobs/phrase-extraction)\
Identify multi-word phrases in signals.
* [Synonym Detection Jobs](/docs/5/fusion/reference/config-ref/jobs/synonym-detection)\
Use this job to generate pairs of synonyms and pairs of similar queries. Two words are considered potential synonyms when they are used in a similar context in similar queries.
* [Token and Phrase Spell Correction](/docs/5/fusion/reference/config-ref/jobs/token-and-phrase-spell-correction)\
Detect misspellings in queries or documents using the numbers of occurrences of words and phrases.
#### Signals aggregation
* [SQL Aggregation](/docs/5/fusion/reference/config-ref/jobs/aggregations/sql-aggregation)\
A Spark SQL aggregation job where user-defined parameters are injected into a built-in SQL template at runtime.
#### Experiment analysis
* [Ranking Metrics](/docs/5/fusion/reference/config-ref/jobs/ranking-metrics)\
Use this job to calculate relevance metrics by replaying [ground truth queries](/docs/5/fusion/reference/config-ref/jobs/ground-truth) against catalog data using variants from an [experiment](/docs/5/fusion/getting-data-out/data-analytics/experiments/overview). Metrics include Normalized Discounted Cumulative Gain (nDCG) and others.
### Collaborative recommenders
These jobs analyze signals and generate matrices used to provide [collaborative recommendations](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods).
* [BPR Recommender](/docs/5/fusion/reference/config-ref/jobs/bpr-recommender)\
Use this job when you want to compute user recommendations or item similarities using a Bayesian Personalized Ranking (BPR) recommender algorithm.
* [Query-to-Query Session-Based Similarity jobs](/docs/5/fusion/reference/config-ref/jobs/query-to-query-session-based-similarity)\
This recommender is based on co-occurrence of queries in the context of clicked documents and sessions. It is useful when your data shows that users tend to search for similar items in a single search session. This method of generating query-to-query recommendations is faster and more reliable than the Query-to-Query Similarity recommender job, and is session-based unlike the similar queries previously generated as part of the [Synonym Detection job](/docs/5/fusion/reference/config-ref/jobs/synonym-detection).
### Content-based recommenders
[Content-based recommenders](/docs/5/fusion/getting-data-out/query-enhancement/recommendations/recommendation-methods) create matrices of similar items based on their content.
* [Content-Based Recommender](/docs/5/fusion/reference/config-ref/jobs/content-based-recommender)\
Use this job when you want to compute item similarities based on their content, such as product descriptions.
### Content analysis
* [Cluster Labeling](/docs/5/fusion/reference/config-ref/jobs/cluster-labeling)\
Cluster labeling jobs are run against your data collections, and are used:
* When clusters or well-defined document categories already exist
* When you want to discover and attach keywords to see representative words within existing clusters
* [Document Clustering](/docs/5/fusion/reference/config-ref/jobs/document-clustering)\
The Document Clustering job uses an unsupervised machine learning algorithm to group documents into clusters based on similarities in their content. You can enable more efficient document exploration by using these clusters as facets, high-level summaries or themes, or to recommend other documents from the same cluster. The job can automatically group similar documents in all kinds of content, such as clinical trials, legal documents, book reviews, blogs, scientific papers, and products.
* [Classification job](/docs/5/fusion/reference/config-ref/jobs/classification)\
This job analyzes how your existing documents are categorized and produces a classification model that can be used to predict the categories of new documents at index time.
* [Outlier Detection](/docs/5/fusion/reference/config-ref/jobs/outlier-detection)\
Outlier detection jobs are run against your data collections, and also perform the following actions:
* Identify information that significantly differs from other data in the collection
* Attach labels to designate each outlier group
### Data ingest
The Parallel Bulk Loader (PBL) job enables bulk ingestion of structured and semi-structured data from big data systems, NoSQL databases, and common file formats like Parquet and Avro.
Datasources the PBL uses include not only common file formats, but Solr databases, JDBC-compliant databases, MongoDB databases and more.
In addition, the PBL distributes the load across the Fusion Spark cluster to optimize performance. And because no parsing is needed, indexing performance is also maximized by writing directly to Solr.
For more information about:
* Available datasources and key features of the Parallel Bulk Loader, see [Parallel Bulk Loader concepts](/docs/5/fusion/getting-data-in/parallel-bulk-loader).
* Usage and detailed configuration, see [Parallel Bulk Loader configuration reference](/docs/5/fusion/reference/config-ref/jobs/parallel-bulk-loader).
## Learn more
For log commands, Spark UI access, and pod cleanup, see [Spark Administration in Kubernetes](/docs/5/fusion/operations/survival-guide/spark-kubernetes-overview).
# Tasks
Source: https://doc.lucidworks.com/docs/5/fusion/operations/jobs-and-scheduling/tasks
[localhost link]: http://localhost:3000/docs/5/fusion/operations/jobs-and-scheduling/tasks
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/jobs-and-scheduling/tasks
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/183
Tasks are a flexible job type that can be used to clean up old logs or run any REST call.
## Task subtypes
The "task" job type has the following subtypes:
* [Log Cleanup](/docs/5/fusion/reference/config-ref/jobs/log-cleanup-jobs)\
Log cleanup jobs are run against your system logs collection to delete log messages that were created prior to the number of days specified in the `Delete logs older than "n" days` field.
To create a log cleanup job, sign in to Fusion and click **Collections > Jobs**. Then click **Add+** and in the Custom and Other Jobs section, select **Log Cleanup**. You can enter parameters to configure the job. If the field has a default value, it is populated when you click to add the job.
## Parameters
* **ID.** The unique ID for this job in the API. This is the `id` field in the configuration file. Required field.
* **Logs Collection.** The collection that contains the documents that are deleted when the job is run. The allowed value is **system\_logs**. This is the `collection` field in the configuration file. Required field.
* **Delete logs older than "n" days.** The number of days to retain system log messages. When this job is run, it deletes all system log messages older than the number of days entered in this field. This is the `days` field in the configuration file. Optional field.
* **Query to match docs to delete.** The query syntax to identify the documents to be deleted. The default value of `-logger_class_s:"com.lucidworks.cloud.gateway.audit"` excludes audit log entries to preserve audit information such as login and logout events. This is the `query` field in the configuration file. Optional field.
* [REST Call](/docs/5/fusion/reference/config-ref/jobs/rest-call)\
A versatile job type that runs an arbitrary REST/HTTP/Solr command.
REST jobs can call external systems using:
* any HTTP or HTTPS request: `://`; requires JWT header.
* a Solr request: `solr://COLLECTION_NAME/…`; authentication is handled.
* a Fusion service request: `extservice://SERVICE_NAME/`; authentication is handled.
To create a REST Call job, sign in to Fusion and click **Collections > Jobs**. Then click **Add+** and in the Custom and Other Jobs section, select **REST Call**. You can enter parameters to configure the job. If the field has a default value, it is populated when you click to add the job.
The parameters for this job are:
* **ID.** The user-defined name for this job. This is the `id` field in the configuration file. Required field.
* **Endpoint URI.** The name or location of the data this job searches. This is the `uri` field in the configuration file. Required field.
* **Call method.** The operation submitted in the request. Values are GET, POST, PUT, or DELETE. This is the `method` field in the configuration file. Required field.
* **Query parameters.** This section lets you enter `property name:property value` options submitted in the query. This is the `queryParams` field in the configuration file. Optional field.
* **Request protocol headers.** This section lets you enter `property name:property value` protocol headers submitted in the query. This is the `headers` field in the configuration file. Optional field.
* **Request entity.** The request string to submit for the query. This is the `entity` field in the configuration file. Optional field.
## Task configuration
Tasks can be created and modified using the [Fusion UI](/docs/5/fusion/operations/jobs-and-scheduling/overview). They can be scheduled using the Fusion UI or the [Jobs API](/api-reference/get-job-histories/get-a-jobs-history).
**See also [Jobs Configuration Reference](/docs/5/fusion/reference/config-ref/jobs/overview).**
# Autoscaling with KEDA
Source: https://doc.lucidworks.com/docs/5/fusion/operations/keda
[KEDA (Kubernetes Event-Driven Autoscaling)](https://keda.sh/) provides flexible, event-driven autoscaling for Fusion workloads as an alternative to the default Kubernetes Horizontal Pod Autoscaler (HPA).
Traditional HPA-based autoscaling relies primarily on CPU and memory utilization. KEDA extends this by enabling you to scale workloads based on events, schedules, and business metrics, allowing you to align infrastructure capacity more closely with actual demand.
## Benefits
KEDA provides these advantages for Fusion workloads:
* **Event-driven autoscaling** - Scale in response to external signals such as Prometheus metrics, queue depth, or pipeline execution load.
* **Scheduled scaling** - Automatically scale workloads based on predictable traffic patterns, such as business hours or peak usage windows.
* **Scale-to-zero capability** - Reduce infrastructure costs by scaling workloads down to zero during off-hours or periods of inactivity.
* **Improved operational efficiency** - Align infrastructure capacity with real business demand instead of relying solely on CPU or memory thresholds.
## Supported services
The following Fusion services support KEDA autoscaling:
* `api-gateway`
* `query-pipeline`
* `fusion-indexing`
## Before you begin
Before configuring KEDA for Fusion services, ensure you have:
* KEDA version 2.18.3 or later installed in your Kubernetes cluster
* Access to the Fusion Helm charts
* Ability to provide a custom values file for your Fusion deployment
If KEDA isn't already installed in your cluster, install it using Helm:
```bash theme={"dark"}
# Add the KEDA Helm repository
helm repo add kedacore https://kedacore.github.io/charts
# Update your local Helm chart repository cache
helm repo update
# Install KEDA
helm install keda kedacore/keda --version 2.18.3 --namespace keda --create-namespace
```
For more information, see the [official KEDA documentation](https://keda.sh/docs/latest/deploy/).
## How KEDA autoscaling works
Fusion Helm charts support two mutually exclusive autoscaling mechanisms per workload:
1. **Kubernetes HPA** (default) - Scales based on CPU and memory metrics.
2. **KEDA ScaledObject** - Scales based on events, schedules, and custom metrics.
Only one autoscaling mechanism can be active per workload at a time. You can't use both HPA and KEDA for the same service.
However, you can mix autoscaling approaches across different services. For example, use HPA for `api-gateway` and KEDA for `query-pipeline`.
### Autoscaling parameters
Each Fusion service has three key autoscaling parameters:
| Parameter | Description | Default |
| - | - | - |
| `autoscaling.enabled` | Master switch to enable autoscaling | `false` |
| `autoscaling.hpa.enabled` | Enables HPA autoscaling | `true` |
| `autoscaling.keda.enabled` | Enables KEDA autoscaling | `false` |
### Default behavior
When `autoscaling.enabled` is `true` and you don't modify other settings, Fusion creates an HPA by default. This preserves backward compatibility with existing deployments.
### Mutual exclusion
The Helm chart enforces these rules:
* If `autoscaling.enabled` is `false` → No autoscaling (neither HPA nor KEDA)
* If `hpa.enabled` is `true` and `keda.enabled` is `false` → HPA autoscaling
* If `hpa.enabled` is `false` and `keda.enabled` is `true` → KEDA autoscaling
* If both `hpa.enabled` and `keda.enabled` are `true` → No autoscaling (conflict resolution)
When KEDA creates a ScaledObject with certain trigger types, it may create its own HPA prefixed with `keda-hpa-`. This is expected behavior and indicates that KEDA is managing autoscaling correctly.
## Enable KEDA
You can enable KEDA for the `api-gateway`, `query-pipeline`, or `fusion-indexing` services by editing the Fusion values file before a Fusion deployment or upgrade, as explained in the next section below.
### Update the Fusion values file
The configuration for each Fusion service is the same, as shown below:
```yaml api-gateway theme={"dark"}
api-gateway:
autoscaling:
enabled: true
hpa:
enabled: false # Disable HPA
keda:
enabled: true # Enable KEDA
```
```yaml query-pipeline theme={"dark"}
query-pipeline:
autoscaling:
enabled: true
hpa:
enabled: false # Disable HPA
keda:
enabled: true # Enable KEDA
```
```yaml fusion-indexing theme={"dark"}
fusion-indexing:
autoscaling:
enabled: true
hpa:
enabled: false # Disable HPA
keda:
enabled: true # Enable KEDA
```
### Deploy or upgrade Fusion
After preparing your custom values file (for example, `fusion-values.yaml`), deploy or upgrade Fusion:
```bash theme={"dark"}
helm upgrade --install -f fusion-values.yaml
```
Replace `` and `` with your specific values.
To find your existing Fusion release name, run `helm list -n `. The chart path can be a repository reference (for example, `lucidworks/fusion`) or a local path to the chart directory.
### Verify the configuration
After deployment, verify that KEDA is active for the configured services.
Check for the ScaledObject:
```bash theme={"dark"}
kubectl get scaledobject -n api-gateway
```
The `api-gateway` ScaledObject should appear in the output.
Verify that no conflicting HPA exists:
```bash theme={"dark"}
kubectl get hpa -n api-gateway
```
No HPA should exist for `api-gateway`, except those prefixed with `keda-hpa-` (which are managed by KEDA).
Check for the ScaledObject:
```bash theme={"dark"}
kubectl get scaledobject -n query-pipeline
```
The `query-pipeline` ScaledObject should appear in the output.
Verify that no conflicting HPA exists:
```bash theme={"dark"}
kubectl get hpa -n query-pipeline
```
No HPA should exist for `query-pipeline`, except those prefixed with `keda-hpa-` (which are managed by KEDA).
Check for the ScaledObject:
```bash theme={"dark"}
kubectl get scaledobject -n fusion-indexing
```
The `fusion-indexing` ScaledObject should appear in the output.
Verify that no conflicting HPA exists:
```bash theme={"dark"}
kubectl get hpa -n fusion-indexing
```
No HPA should exist for `fusion-indexing`, except those prefixed with `keda-hpa-` (which are managed by KEDA).
KEDA may create an HPA prefixed with `keda-hpa-` followed by your release name and service name. For example: `keda-hpa-fusion-query-pipeline`. This is expected when KEDA uses certain trigger types.
### Custom configuration parameters
You can customize the scaling behavior, metadata, and the triggers KEDA monitors using the parameters in the table below.
For the `api-gateway`, `query-pipeline`, or `fusion-indexing` services, replace `[service-name]` with the name of the service.
| Parameter | Description | Default |
| - | - | - |
| `[service-name].autoscaling.keda.labels` | Labels to add to the ScaledObject. | `{}` |
| `[service-name].autoscaling.keda.annotations` | Annotations to add to the ScaledObject. | `{}` |
| `[service-name].autoscaling.keda.pollingInterval` | Interval in seconds at which KEDA checks triggers. | `30` |
| `[service-name].autoscaling.keda.cooldownPeriod` | Time in seconds KEDA waits after scale-down before re-evaluating. | `300` |
| `[service-name].autoscaling.keda.initialCooldownPeriod` | Time in seconds KEDA waits after scale-out before re-evaluating. | `0` |
| `[service-name].autoscaling.keda.idleReplicaCount` | Replicas to maintain when triggers are inactive (enables scale-to-zero). | Disabled |
| `[service-name].autoscaling.keda.minReplicas` | Minimum number of replicas. | `1` |
| `[service-name].autoscaling.keda.maxReplicas` | Maximum number of replicas. | `5` |
| `[service-name].autoscaling.keda.advanced` | Advanced HPA behavior configuration. | `{}` |
| `[service-name].autoscaling.keda.fallback` | Fallback HPA configuration if KEDA encounters errors. | `{}` |
| `[service-name].autoscaling.keda.triggers` | List of KEDA triggers (required). | `[]` |
At least one trigger is required when KEDA is enabled.
Without triggers, KEDA can't determine when to scale your workload.
This example demonstrates a custom KEDA configuration for `query-pipeline` with these components:
* **Cron trigger** - Scales to 6 replicas during business hours (Monday-Friday, 7:00 AM - 6:00 PM Central Time).
* **CPU trigger** - Scales up to 15 replicas when CPU utilization exceeds 60%.
```yaml expandable theme={"dark"}
query-pipeline:
autoscaling:
enabled: true
hpa:
enabled: false
keda:
enabled: true
pollingInterval: 40 # Check triggers every 40 seconds
minReplicas: 2
maxReplicas: 15
triggers:
# Scheduled scaling
- type: cron
metadata:
timezone: "America/Chicago"
start: "0 7 * * 1-5" # 7:00 AM weekdays
end: "0 18 * * 1-5" # 6:00 PM weekdays
desiredReplicas: "6"
# Reactive scaling
- type: cpu
metricType: Utilization
metadata:
value: "60" # Scale when CPU exceeds 60%
```
For more information about available KEDA triggers, see the [KEDA scalers documentation](https://keda.sh/docs/latest/scalers/).
## Troubleshooting
This section provides troubleshooting steps for some common issues with KEDA configuration.
1. Verify that `autoscaling.enabled` is `true`.
2. Verify that `autoscaling.keda.enabled` is `true`.
3. Verify that `autoscaling.hpa.enabled` is `false`.
4. Check that KEDA is installed: `kubectl get pods -n keda`.
5. Review Helm deployment logs for errors.
1. Check your configuration for conflicting settings.
2. Ensure that both `hpa.enabled` and `keda.enabled` aren't set to `true`.
3. Delete the unwanted resource manually if necessary.
1. Verify that triggers are configured correctly.
2. Check KEDA operator logs: `kubectl logs -n keda -l app=keda-operator`
3. Describe the ScaledObject to see its status: `kubectl describe scaledobject -n `
4. Verify that the trigger source (metrics endpoint, queue, and so on) is accessible.
# Built-in Dashboards
Source: https://doc.lucidworks.com/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/built-in-dashboards
[localhost link]: http://localhost:3000/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/built-in-dashboards
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/built-in-dashboards
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/140
Fusion ships with built-in dashboards. To access these dashboards from the Fusion UI, click **Analytics > Dashboards**.
## Service Logs dashboard
The Service Logs dashboard (`lucidworks-service-logs.json`) is also the **default** dashboard (`default.json`).
You can modify the default dashboard to suit your needs. To set a different dashboard as the browser default, click the dashboard you want to set as the default, then click **Save > Set as Browser Default**.
To open the default dashboard from the Fusion workspace, click Analytics > **Dashboards** or System > **Log Viewer**. Alternatively, on any dashboard, click **Goto saved default** .
## Combined Logs dashboard
The Combined Logs dashboard (`lucidworks-combined-logs.json`) lets you analyze all logs from the `system_logs` collection.
## Blank dashboard
An entirely blank dashboard (`blank.json`) is available. To open this dashboard from the default dashboard, click **Blank Dashboard** at the top right.
## Search Analytics dashboard
The Fusion Search Analytics dashboard (`lucidworks-searchanalytics.json`) displays search analytics derived from collection logs and from queries to Solr (`/api/solr`).
To open the Search Analytics dashboard from the default dashboard, click **Search Analytics** at the top right.
## Fusion Signals dashboard
The Fusion Signals dashboard (`lucidworks-signals.json`) is a time-series dashboard that you can use to monitor signals collections. It uses the signal timestamp as the time field.
To open the Fusion Signals dashboard from the default dashboard, click **Lucidworks Fusion Signals** at the top right.
## Default Non-Time-Series dashboard
Fusion includes a default non-time-series dashboard (`default-nts.json`). This is an example:
To open the default non-time-series dashboard, at the top right of any dashboard, click **New > Non-time-series dashboard**.
## Default Time-Series dashboard
Fusion includes a default time-series dashboard (`default-ts.json`). This is an example:
To open the default time-series dashboard, at the top right of any dashboard, click **New > Time-series dashboard**.
# Display Panels
Source: https://doc.lucidworks.com/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/display-panels
[localhost link]: http://localhost:3000/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/display-panels
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/display-panels
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/137
Display panels on a dashboard display information about the data in a single collection. Fusion has these types of display panels:
* **Layout.** Use these display panels to organize the panels on a dashboard.
* **Textual Information.** Use these panels to display textual information. For example, use a Table panel to displays the values of fields in a set of records.
* **Graphical Visualization.** Use these panels to help users to visualize data. For example, display category data in a bar chart and geographical data on a map.
## What Data is Displayed
These things determine which data Fusion displays in a display panel:
* Where the query is sent (to a Fusion query pipeline or to Solr)
* The collection that contains the data. You can specify this when you create a dashboard, or permit the user to choose the collection.
* Global query parameters (optional)
* Input panel configuration settings (possibly including a panel query)
* A user’s selection of the collection (if permitted)
* A user’s query
* A user’s interactions with the data
## Layout Panels
These layout panels can help you organize a dashboard:
* **Column.** Lay out panels in a column within a row or part of a row. (Instead of the usual left-to-right layout within rows.)
* **Text.** Add text to a dashboard, for example, to instruct the user regarding how to use the dashboard, or to describe the data.
## Textual Information Panels
These panels display textual information:
* **Doc Viewer.** Display documents as single pages of information, and let the user page through documents. Paging through documents does not affect the data displayed in other panels.
* **Full Text Search.** Provide full text search capability.
**Note**: Make a Full Text Search panel wide. Start with Span = 12. Depending on the content, you might be able to make it narrower. If you make a Full Text Search panel too narrow, content and controls can be inaccessible.
* **Hits.** A Hits panel displays statistical information about the hits returned by the query, including all filters that are applied to the query. The default information is the count (number of hits). You can display the count, minimum, mean, maximum, sum, standard deviation, the sum of squares, and/or the number of hits that lack a value for a field.
* **Table.** Display data in a table.
**Note**: Make a Table panel wide. Start with Span = 12. Depending on the content, you might be able to make it narrower. If you make a Table panel too narrow, content and controls can be inaccessible.
## Graphical Visualization Panels
Most display panels are graphical visualization panels. They visually present the data requested in the input panel(s), subject to configuration of the display panel and to any global filtering. Panels for visualizing information graphically are:
* **Bar.** Graph data as frequencies in a bar chart.
* **Better Map.** Display geolocated points in clustered groups on a map. As you zoom out, points are clustered into fewer groups. As you zoom in, points are clustered into fewer groups (and at some point are not clustered). The Better Map panel *does not* use the geospatial search capabilities of Solr. It transfers more data than the Map panel and generally requires more computation, while showing less data. If you have a time filter, the panel will show the most recent points in your search, up to your defined limit. This panel is best used after filtering the results through other queries and filter queries, or when you want to inspect a recent sample of points.
* **Heat Map.** Display a heat map, that is, a graphical representation of data along two facet axes.
* **Histogram.** Display a histogram. When used in `count` mode, this is a bucketed chart of the current query, including all applied time and non-time filters. When used in `values` mode, the histogram plots the value of a specific field over time, and lets the user group the values based on the values of a second field.
* **Map.** Display a map of shaded regions using a field that contains a 2-letter country code or U.S. state code. Regions with more hits are shaded darker. Instead of a count, you can choose to shade regions based on the minimum, maximum, mean, or sum. The map panel uses facets, so it is important that you set field values to the appropriate 2-letter codes at index time.
* **Multi-series.** Graph multiple fields of the same type that vary with time in the same graph. The X-axis is used for the time series field. Values in the other fields are graphed along the Y-axis.
* **Range Facet.** Display a histogram of a numeric field. The Range Facet panel is similar to the time series histogram. It lets you select ranges and zooming in/out to the desired numeric range. Range selections in the panel are reflected across the entire dashboard. With multiple search boxes in a Query panel, the Rangefacet panel can display multiple data sets, as shown in this example:
* **Scatter Plot.** Display a scatter plot between two variables or four variables.
* **Sunburst.** Display a sunburst plot based on facets.
* **Tag Cloud.** Display a tag cloud of the top N words in a field.
* **Terms.** Display the count, minimum, maximum, mean, or sum for a Solr facet as a bar chart, pie chart, or table.
# Input Panels
Source: https://doc.lucidworks.com/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/input-panels
[localhost link]: http://localhost:3000/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/input-panels
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/input-panels
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/138
Input panels let you control which data output panels display. The values you specify in input panels become parts of the Solr queries that output panels use to obtain data.
You can use these input panels:
* **Query Panel.** Enter a freeform query (one or more query terms) in a search bar. Add additional search bars to a query panel to submit separate queries. Some visualization panels (for example, Rangefacet) keep the data separate so you can compare it. This is an example of a Query panel with two search boxes. The parameters for query strings are `Primary_Specialty:Urology` and `Primary_Specialty:Psychiatry`. Here is the Query Panel and the resulting Rangefacet panel:
* **Time Picker Panel.** Apply a time range to time-series data. The time range can be:
* Relative.\* A time range starting now and reaching backward in time, for example, the last 15 minutes or one hour
* Absolute.\* A specific time-and-date range, for example, from 06/01/2017 00:00:00 to 06/30/17 23:59:59
* Since.\* A time range since a specific date and time, for example, since 01/01/2017 00:00:00.
* **Facet.** A Facet panel can facet any data field.
* **Filtering.** A Filtering panel lets you apply field-based filters to the hits returned by the query. The filters apply to all display panels.
**Important**
You must use a Filtering panel somewhere on your dashboard, so that all panels work correctly when you interact with data.
## Query Syntax
Enter a search term or phrase in the search box of the Query panel. Autocompletion provides a list of possibly related prior searches. Finish typing your search query or select an autocompleted query, and then click **Search**.
Enter queries in a Query panel using [Apache Lucene Query Parser syntax](https://lucene.apache.org/solr/guide/the-standard-query-parser.html#the-standard-query-parser). You enter the parameter for the query string (for example, `Susan` or `Gender:F`), not the query string (for example, `q=Susan` or `q=Gender:F`).
### Rules for the Simplest Cases
Here are some syntax rules for the simplest cases:
* A single term is either `field:value` or `value`. With `value`, the search is over all fields.
* For an exact-case match, you must specify the field name.
* Surround a term that contains spaces in *double quotation marks* (" ").
* You cannot specify `*:value` or `*:"value"` to search over all fields; that syntax does not work.
* To retrieve all records, use the search term `*:*`.
* For OR logic, enter `OR` between the terms, or just use spaces. For AND logic, enter `AND` between the terms.
### Examples
These are examples of the query syntax:
| Goal | Syntax and example |
| - | - |
| Single term in any field; no blanks in term | `term` (matches any case) |
| Single term in any field; blanks in term | `"term"` (matches any case) |
| Multiple terms, each in any field; with OR logic; no blanks in terms | `term1` `term2` … |
| Multiple terms, each in any field; with OR logic; blanks in terms | `"term1"` `"term2"` … (matches any case) |
| Multiple terms, each in any field; with AND logic (in the same record); no blanks in terms | `term1 AND term2` … |
| Multiple terms, each in any field; with AND logic (in the same record); blanks in terms | `"term1" AND "term2"` … (matches any case) |
| Single term in a specific field; no blanks in term | `field:term` (matches any case) |
| Single term in a specific field; blanks in term | `"field:term"` (matches *exact* case) |
| Multiple terms, each in a specific field; with OR logic; no blanks in terms | `field1:term1` `field2:term2` … |
| Multiple terms, each in a specific field; with OR logic; blanks in terms | `"field1:term1"` `"field2:term2"` … (matches *exact* case) |
| Multiple terms, each in each in a specific field; with AND logic (in the same record); no blanks in terms | `field1:term1 AND field2:term2` … |
| Multiple terms, each in a specific field; with AND logic (in the same record); blanks in terms | `"field1:term1" AND "field2:term2"` … (matches *exact* case) |
For more information about the query syntax, see [Standard Query Parser Parameters](https://lucene.apache.org/solr/guide/the-standard-query-parser.html#the-standard-query-parser).
You can use a Text panel to advise the user regarding the syntax of search terms. Also, if you want the dashboard user to explore subsets of the data, use a Filtering panel or a Facet panel to achieve that. do not expect users to enter complex search expressions. For example, add the field `gender` as a facet, instead of expecting the user to add `AND gender:male` to a search expression.
## Inspect a Panel Query
You cannot inspect the panel query in the Query panel. You can inspect the panel query in other panels, for example, in a Histogram or Heatmap panel. You can see the contributions that the different parts of the query make. In this example, there are no global query parameters.
Query:
```txt theme={"dark"}
start_station_name:"Broadway and E 14 St" AND gender:Male
```
Panel query for a Histogram panel:
The part of the query from the Query panel is the first part, from `q=` through `Male`.
```bash theme={"dark"}
q=start_station_name%3A%22Broadway%20and%20E%2014%20St%22%20AND%20gender%3AMale&wt=json&rows=0&fq=start_time:[2014-01-28T20:16:59.000Z%20TO%202014-03-15T05:36:55.000Z]&facet=true&facet.range=start_time&facet.range.start=2014-01-28T20:16:59.000Z&facet.range.end=2014-03-15T05:36:55.000Z&facet.range.gap=%2B12HOUR
```
# Banana Dashboards
Source: https://doc.lucidworks.com/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/overview
[localhost link]: http://localhost:3000/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/139
Dashboards are Fusion’s built-in analytics component. Most of the components of dashboards, called *panels*, are for data visualization. Some panels display textual information; for example, records in a table or documents.
Dashboards permit near-real-time views of data. For some types of panels, you can configure how frequently the panel gets new data.
Dashboards also permit users to interact with data. For example, users can submit queries, select facets, and zoom in and out on histograms.
A Fusion dashboard contains one or more controls for search query inputs and one or more quantitive displays over the results for that query.
Dashboards run as a client-side application in a web browser, using JavaScript components for HTML5. Client-side JavaScript provides a dynamic, responsive browsing experience. Solr facets provide the quantifications required for visualizations, for example, charts, graphs, tables, and maps (for geospatial data).
Dashboards can also have tabular displays for drilling down to the individual documents in a results set.
The underlying browser application is Banana, an open source visualization tool
based on [Kibana 3](http://www.elastic.co/guide/en/kibana/3.0/index.html).
Where Kibana communicates with Elasticsearch, Banana communicates with Solr.
Banana can be downloaded from: [https://github.com/LucidWorks/banana/](https://github.com/LucidWorks/banana/).
Fusion dashboards were first developed for log file analysis.
Log file entries contain a timestamp plus some amount of additional event information.
Fusion’s date/time processing aggregates these timestamps at different levels of granularity
allowing for visualization of historical trends in system and user behaviors.
See these additional topics for details:
* **Create Dashboards**
* **Use Dashboards in Fusion**
* **Manage Dashboards**
* [Input Panels](/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/input-panels)
* [Display Panels](/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/display-panels)
Create dashboards that meet the needs of users.
If you configure the dashboard to use existing data and add a Filtering panel, the panels in the dashboard work *as you add panels and configure them*. You do not need to wait until you have finished configuring the entire dashboard before viewing the panels recently added or edited.
## Requirements for a dashboard
Requirements for a dashboard are:
* **Input panel(s).** You must add one or more input panels that let you specify which data the dashboard will display.
* **Filtering panel.** You must add a filtering panel to ensure that the dashboard functions correctly.
* **Output panel(s).** You must add one or more output panels to display the data.
## Steps to create a dashboard
To create a dashboard, perform these actions in the order specified:
* [Choose the best starting point](#choose-a-starting-point)
* [Configure dashboard settings](#configure-dashboard-settings)
* [Lay out the dashboard](#lay-out-the-dashboard)
* [Configure the panels](#configure-panels)
* [Save the dashboard](#save-the-dashboard)
* [Test the dashboard](#test-the-dashboard)
### Choose a starting point
You can create a new dashboard from scratch or modify an existing dashboard.
To open the dashboard, complete the following:
1. From the Fusion workspace, click **Analytics** > **Dashboards**, or click **System** > **Log Viewer**.
2. In the upper right corner of the default dashboard, click the link for the dashboard you want to use as a starting point.
### Configure dashboard settings
The dashboard settings affect all the panels on a dashboard. To configure settings, click **Configure dashboard**.
| Option | Notes |
| - | - |
| Define where Fusion send queries from the dashboard | Click the **Solr** tab.
To use a Fusion query pipeline, select **Use Fusion**.
To send queries to Solr, deselect **Use Fusion**.
|
| Set a default collection that Fusion queries when the dashboard opens | Click the **Solr** tab and specify the name of the collection to query. To use the same dashboard for multiple collections, let the dashboard user choose a collection. On the **Controls** tab, select **Show Collections Picker**. |
| Specify optional global query parameters to append to all dashboard queries | Click the **Solr** tab and enter **Global Query Parameters**. For example, to set the default search field to the field `message_t`, specify the global query parameter `&df=message_t`. |
### Lay out the dashboard
A dashboard consists of rows and panels, and possibly columns (if you use Column panels). The table provides the available options.
| Option | Notes |
| - | - |
| Add a row | Add a new row to the dashboard. |
| Configure a row | Configure the selected row. |
| Add panel to an empty row | 1. Click **Add panel to an empty row**. 2. Click **Add panel**. 3. Select **Panel Type**. 4. Configure the panel. 5. Click **Add Panel**. |
| Add a panel to a row that already contains panels | 1. Click **Add a panel to this row**. 2. Click **Add panel**. 3. Select **Panel Type**. 4. Configure the panel. 5. Click **Add Panel**. |
| Row Full | An indicator on the left side of a row that is full. |
| Resize a panel | 1. Click **Configure**. 2. Click the **General** tab. 3. In the **Span** field, specify a width between 1 (one twelfth) and 12 (twelve twelfths). |
| Drag a panel to a different location | 1. In the upper right corner of the panel, click-and-hold the panel type. 2. Drag the panel to the new location and release the panel. If you drag the panel on top of a different panel, the existing panel is shifted to another location. |
| Remove Panel | Remove a specific panel from the dashboard. |
| Place panels in a column | Configure panels in a column format. |
| Add text | Add text to the dashboard, such as instructions to use the dashboard or descriptions of the content in the dashboard. |
### Configure panels
This section describes general tasks that apply to all panels and tasks that apply to specific types of panels.
As a best practice, we recommend you configure each panel as you add it. Then save and test to verify it is set correctly before you add another panel.
| Option | Notes |
| - | - |
| Configure | The tabs that are available based on panel types are described in this section. Not all tabs are present for all panel types.
**General.** The name and width of the panel, whether it is editable, and whether the panel will let users inspect the queries.
**Panel.** The primary tab for configuring the panel. This configuration affects *how* data is displayed, but also in some cases, *which* data is displayed.
**Info.** Optionally, add a help message that Fusion displays when a user inspects the queries for a panel.
**Fields.** Add and remove fields in a Table panel.
**Paging.** Control pagination of a Table panel.
**Queries.** View the panel query. Optionally, add one or more custom Solr query parameters. Fusion adds these to the panel query.
|
| View and configure Histogram and Range Facet panels | Click the **View** up arrow to view and modify the configuration. Click the down arrow to hide the configuration. |
| Help | Display help information for a setting. |
| Inspect | View the query that a panel sends to Solr. Fusion displays both the panel query and the custom query if it exists. |
| Make a panel editable | You must first remove, then add the panel back. 1. Click **Remove Panel**. 2. Click either **Add panel to an empty row** or **Add Panel**. 3. Enter the appropriate values and click the **Editable** check box. 4. Click **Add Panel**. |
| Make a panel non-editable | 1. Click **Configure**. 2. On the **General** tab, deselect the **Editable** check box. 3. Save the changes. |
#### Configure input panel settings
The elements on input panels determine the information displayed and is based on:
* Panel configuration
* Panel contributions to the query
* User interactions with the panels
| Option | Notes |
| - | - |
| Add a custom query for a panel | Certain panel types let you specify a custom query to add to the panel query. The combined query is sent to the Fusion query pipeline or to Solr. |
| Filter | Filters select subsets of the data in a collection. For example, records in a time range or in which the field Gender has the value `Female`. A filtering panel is required for data interactivity to work. The filtering panel displays the filters that are in effect. You can also use a Filtering panel to add, remove, edit, and toggle filters. |
| Facet | Use a Facet panel to choose data in one or more facets. For example, if the facets Gender and Graduation Year are defined, a user could choose the Gender `Male` and the Graduation Year `2010`. |
#### Configure specific types of panels
| Option | Notes |
| - | - |
| Add Search Box | Click to add a query-entry field to a Query panel. |
| Delete Search Box | Click to remove a search box from a Query panel. |
| Move a column | Click the right or left arrow, respectively, to shift the column one space in that direction. |
### Save the dashboard
You can save a dashboard to Solr, a file, or a GitHub gist. We recommend that you always save dashboards to Solr (even if you also save them as files or gists).
| Option | Notes |
| - | - |
| Save in Solr | 1. Click **Save**. 2. Under **Solr**, enter a name for the dashboard. 3. Check or deselect the **Make Public** checkbox. If this checkbox is selected, the dashboard is shared with everyone. 4. Click **Save**. |
| Export | Click to enter the number of rows to export and select the destination file type. For example, CSV, XML, or JSON. Fusion downloads the dashboard as a file in the browser’s download directory. The filename is named `**_dashboard\_name_**-**_large\_number_**`. For example, `Storefront-1502124761166`. The file is a JSON file. It lacks the file type `.json`, which is not really needed (you could add it). If you want to place the file in a different location, move it. |
| Save as a GitHub gist | When you save a dashboard as a GitHub gist, you do not need to sign in to GitHub first. Fusion stores the gist as an anonymous gist. You must configure the dashboard correctly to save as a GitHub gist. To configure the dashboard: 1. Click **Configure dashboard**. 2. Click the **Controls** tab. 3. In the **Allow saving to** field, select **Gist**. 4. At the top right of the dashboard, click **Save**. 5. Under **Gist**, enter a name for the dashboard. The name becomes the title of the dashboard, which is stored in the JSON file as the value of `"title"`. 6. Click **Save Gist** . Fusion displays the URL at which the gist resides. If you save the dashboard again, you get a different anonymous gist and a different URL. IMPORTANT: Navigate to the URL *immediately*. Save the URL. Bookmark it. Write it down. You cannot find the gist by searching for it, and Gist identifiers do not persist for long in the Fusion UI. (The last gist saved is available in the Load dialog for the duration of the browser session.) |
### Test the dashboard
View and interact with the dashboard, and verify each panel displays the expected data. If necessary, adjust the placement and settings of the panels based on user feedback.
This topic details the options available to use Fusion dashboards.
## Dashboards main page
When you open the Fusion dashboards, your organization’s dashboards display, as well as the EXPLORE DASHBOARDS section that provides options to create or display specific types of dashboards.
You can click the icons in the top right corner of the screen to perform the following functions:
| Option | Notes |
| - | - |
| Goto saved default | Open the saved default dashboard. |
| New | Create a new dashboard. Options are:
Time-series dashboard
Non time-series dashboard
|
| Load | Load a dashboard using one of the following options:
Saved dashboards are listed at the bottom of the dialog box. Click a dashboard to open it. Filter dashboards by entering an exact-match string in the Filter box. If there are more that 20 saved dashboards, use the pagination controls to find a dashboard. To view the next set of pages, click the forward arrows. To view the prior set of pages, click the back arrows. Or click a page number to view the dashboards on that page.
Enter the gist URL or number (the last part of the URL)
**Enter value > click Get** A link that opens the dashboard appears below a thin line. The link displays the dashboard name. Click the link to open the dashboard.
Type to filter.
Enter dashboard type to filter and display if the dashboard matches.
|
| Save | Save options include:
Locally
Export to File
Set as Browser Default. Click to set the current dashboard as the default.
Clear Browser Default. Click to clear the current dashboard. The default dashboard for the browser becomes the system default dashboard `default.json` (the Fusion Logs dashboard).
Solr
Make Public. If this checkbox is selected, the dashboard is shared with everyone.
Service Logs. Enter a value and \*click **Save** to save the log.
|
| Configure Dashboard | Click to configure tabs for General, Rows, Control, and Solr dashboard settings. |
| Delete a dashboard saved in Solr |
Click **Load**.
Identify the file to delete and click **Delete**.
Click **OK** to confirm the deletion.
|
| Delete a dashboard saved in a file | Navigate to the file and delete it. |
| Delete a dashboard saved as a GitHub gist | Open the gist URL and click **Delete**. |
For more information, see [Fusion 5.x Built-in Dashboards](/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/built-in-dashboards).
## Interact with data
The table describes options to interact with data displayed in certain dashboards.
| Option | Notes |
| - | - |
| Collections | Choose the collection from which to obtain data. If a dashboard is configured to let users choose among collections, select the collection from the **Collections** drop-down list at the top right of the dashboard. |
| Search | Enter the value to search.
To search for a specific value, enter a term or phrase and click **Search**.
To retrieve all data, enter `*.*` and click **Search**.
|
| Clear search box | Click to clear the search box and return the most general query, as if you entered `*.*`. |
| Waiting for data | Indicates the query is waiting for data to be returned. If the indicator spins for an extended time while other panels display data, the panel is not configured correctly. |
| Select item | Click a table row. In a graphical-visualization panel, click the data item. NOTE: Not all panel types let you select data in this way. |
| Export data | Click to enter the number of rows to export and select the destination file type. For example, CSV, XML, or JSON. |
| Hide row | Click to hide the row from the display. |
| Expand row | Click the row to display detailed information. |
## Interact with specific types of panels
The table describes options for specific types of panels.
### Table and text panels
| Option | Notes |
| - | - |
| Show or hide the field list | Click the right or left arrow, respectively, at the top of the panel. |
| Sort a table column | Click the up arrow at the top of the column to sort in ascending order. Click the down arrow at the top of the column to sort in descending order. |
| View next or previous page of hits | Click the right or left arrow, respectively, at the bottom of the page of hits. |
| View first page | Click to view the first page of hits for that particular panel. |
| Show record details | Click the table row and then select the display format of Table, JSON, or Raw. |
| Hide details | Click the up arrow at the top right of the details. |
### Histogram and Range Facet panels
The table describes options for the Histogram and Range Facet panels.
| Option | Notes |
| - | - |
| Zoom in on a subset of data | Click the area you want to enlarge, then drag across the range of data to zoom in. |
| Zoom out | Click to zoom out. |
### Filtering panels
The table describes filter options for panels.
| Option | Notes |
| - | - |
| Edit filter | Click to edit a panel filter. |
| Toggle filter | Click to toggle the filter to the opposite setting (on or off). |
| Remove filter | Click to remove the current filter. CAUTION: To display correctly, specific filters must be set for certain types of panels. Fusion also creates filters based on your interactions with data. You can add a filter back, but there is no undo. Before you remove a filter, toggle off the filter and verify the results. If the results are accurate, it is safe to remove it. |
### Heat Map panels
The table describes the option for heat map panels.
| Option | Notes |
| - | - |
| Transpose rows and columns | Click to reverse the display of rows and columns. |
## Filter by Fusion request ID
To only display information about log entries with a specific Fusion request ID, you can filter the contents of a logs dashboard by the Fusion request ID.
Filter the contents in the following ways:
* [Click a link in the error message or toast](#click-a-link-in-the-error-message-or-toast)
* [Filter in a logs dashboard](#append-a-query-filter-parameter-to-the-dashboard-url)
* [Append a query filter parameter to the dashboard URL](#append-a-query-filter-parameter-to-the-dashboard-url)
### Click a link in the error message or toast
To view the dashboard for a Fusion UI API error or API error toast, click **View logs dashboard**. The system displays the dashboard using a filter for that request ID.
API error with link in Fusion UI:
API error toast with link:
Filtered Service Logs dashboard:
Click **View logs dashboard** to open the Service Logs dashboard with filters by the Fusion request ID. For example, the filtering query for the `mdc_requestId_s` field displays the `fusion-request-id` of `BWQSjPPYgB`.
### Append a query filter parameter to the dashboard URL
As an alternative to constructing a query filter in the dashboard, you can append a query filter parameter to the dashboard URL. In the query filter, use the field name `mdc_requestID_s`.
Example 1: Default dashboard (Service Logs)
Example 2: Combined Logs dashboard
## Open the Service Logs dashboard from a job history
1. Open the job history.
* For most jobs, click **Collections** > **Jobs**, select a job, and then click **Job History**.
* For datasource jobs, click **Indexing** > **Datasources**, select a datasource, and then click **Job History**.
In the job history, you will see a link named `runId`:
2. Click the link to view the Service Logs dashboard filtered by the `runId`:
## Set Permissions
Users who manage or use dashboards need specific [permissions](/docs/5/fusion/operations/security/access-control/permissions) to do so.
### Permissions to Manage Dashboards
The built-in roles `admin` and `developer` both have sufficient permissions to manage dashboards.
For users who manage dashboards, set these permissions in [roles](/docs/5/fusion/operations/security/access-control/roles), and then assign the roles to users:
```bash theme={"dark"}
GET:/solr/{id}/\*:id=collection-name
GET:/solr/{id}/admin/luke:id=collection-name
GET,POST,PUT,DELETE:/solr/system_blobs/\*
GET,POST,PUT,DELETE:/collections/system_blobs
```
### Permissions to Use Dashboards
The built-in role `search` *does not* have sufficient permissions to use dashboards.
For users who manage dashboards, set these permissions in [roles](/docs/5/fusion/operations/security/access-control/roles), and then assign the roles to users:
```bash theme={"dark"}
GET:/solr/{id}/\*:id=collection-name
GET:/solr/{id}/admin/luke:id=collection-name
GET:/solr/system_blobs/\*
GET:/collections/system_blobs
```
## Manage Dashboards
This topic details the options available to use Fusion dashboards.
## Dashboards main page
When you open the Fusion dashboards, your organization’s dashboards display, as well as the EXPLORE DASHBOARDS section that provides options to create or display specific types of dashboards.
You can click the icons in the top right corner of the screen to perform the following functions:
| Option | Notes |
| - | - |
| Goto saved default | Open the saved default dashboard. |
| New | Create a new dashboard. Options are: ● Time-series dashboard ● Non time-series dashboard |
| Load | Load a dashboard using one of the following options: ●Load dashboard JSON layout from file Click **Local File > Choose File > select file > Open**. Saved dashboards are listed at the bottom of the dialog box. Click a dashboard to open it. Filter dashboards by entering an exact-match string in the Filter box. If there are more that 20 saved dashboards, use the pagination controls to find a dashboard. To view the next set of pages, click the forward arrows. To view the prior set of pages, click the back arrows. Or click a page number to view the dashboards on that page. ● Enter the gist URL or number (the last part of the URL) **Enter value > click Get** . A link that opens the dashboard appears below a thin line. The link displays the dashboard name. Click the link to open the dashboard. ● Type to filter. Enter dashboard type to filter and display if the dashboard matches. |
| Save | Save options include: ●Locally **Export to File** Set as Browser Default. Click to set the current dashboard as the default. **Clear Browser Default**. Click to clear the current dashboard. The default dashboard for the browser becomes the system default dashboard `default.json` (the Fusion Logs dashboard). ●Solr **Make Public**. If this checkbox is selected, the dashboard is shared with everyone. **Service Logs**. Enter a value and click **Save** to save the log. |
| Configure Dashboard | Click to configure tabs for General, Rows, Control, and Solr dashboard settings. |
| Delete a dashboard saved in Solr | 1. Click **Load**. 2. Identify the file to delete and click **Delete**. 3. Click **OK** to confirm the deletion. |
| Delete a dashboard saved in a file | Navigate to the file and delete it. |
| Delete a dashboard saved as a GitHub gist | Open the gist URL and click **Delete**. |
For more information, see [Fusion 5.x Built-in Dashboards](/docs/5/fusion/operations/monitoring-and-reporting/banana-dashboards/built-in-dashboards).
## Interact with data
The table describes options to interact with data displayed in certain dashboards.
| Option | Notes |
| - | - |
| Collections | Choose the collection from which to obtain data. If a dashboard is configured to let users choose among collections, select the collection from the **Collections** drop-down list at the top right of the dashboard. |
| Search | Enter the value to search. ●To search for a specific value, enter a term or phrase and click **Search**. ●To retrieve all data, enter `\*.\*` and click **Search**. |
| Clear search box | Click to clear the search box and return the most general query, as if you entered `\*.\*`. |
| Waiting for data | Indicates the query is waiting for data to be returned. If the indicator spins for an extended time while other panels display data, the panel is not configured correctly. |
| Select item | Click a table row. In a graphical-visualization panel, click the data item. NOTE: Not all panel types let you select data in this way. |
| Export data | Click to enter the number of rows to export and select the destination file type. For example, CSV, XML, or JSON. |
| Hide row | Click to hide the row from the display. |
| Expand row | Click the row to display detailed information. |
## Interact with specific types of panels
The table describes options for specific types of panels.
### Table and text panels
| Option | Notes |
| - | - |
| Show or hide the field list | Click the right or left arrow, respectively, at the top of the panel. |
| Sort a table column | Click the up arrow at the top of the column to sort in ascending order. Click the down arrow at the top of the column to sort in descending order. |
| View next or previous page of hits | Click the right or left arrow, respectively, at the bottom of the page of hits. |
| View first page | Click to view the first page of hits for that particular panel. |
| Show record details | Click the table row and then select the display format of Table, JSON, or Raw. |
| Hide details | Click the up arrow at the top right of the details. |
### Histogram and Range Facet panels
The table describes options for the Histogram and Range Facet panels.
| Option | Notes |
| - | - |
| Zoom in on a subset of data | Click the area you want to enlarge, then drag across the range of data to zoom in. |
| Zoom out | Click to zoom out. |
### Filtering panels
The table describes filter options for panels.
| Option | Notes |
| - | - |
| Edit filter | Click to edit a panel filter. |
| Toggle filter | Click to toggle the filter to the opposite setting (on or off). |
| Remove filter | Click to remove the current filter. CAUTION: To display correctly, specific filters must be set for certain types of panels. Fusion also creates filters based on your interactions with data. You can add a filter back, but there is no undo. Before you remove a filter, toggle off the filter and verify the results. If the results are accurate, it is safe to remove it. |
### Heat Map panels
The table describes the option for heat map panels.
| Option | Notes |
| - | - |
| Transpose rows and columns | Click to reverse the display of rows and columns. |
# Consumption Dashboard
Source: https://doc.lucidworks.com/docs/5/fusion/operations/monitoring-and-reporting/consumption
[localhost link]: http://localhost:3000/docs/5/fusion/operations/monitoring-and-reporting/consumption
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/monitoring-and-reporting/consumption
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/0m5ml4
The Consumption Dashboard displays your request and record usage against the allotted consumption in your Fusion license. This ensures you understand how much of your Fusion entitlements you are using and whether you are reaching your consumption limit.
You can also access this information through the [Apps Manager API](/api-reference/license/get-the-fusion-license).
To access the Consumption Dashboard, navigate to **Analytics Hub > Consumption** in Fusion.
You can access the Consumption Dashboard from any app, but the data in the dashboard is collected across all your Fusion apps. All consumption dashboard data updates daily at midnight Coordinated Universal Time (UTC).
**Important**
Contact your Lucidworks representative if you have questions about your Fusion license consumption entitlements.
## Requests
Request means any GET or POST call against the Fusion [Query Profiles API](/api-reference/query-profiles-api/get-the-service-status) or Fusion [Query Pipelines API](/api-reference/query-pipelines-api/get-all-query-pipelines). Requests may be initiated by user actions such as typing in the search bar, executing a search, or clicking a category or facet. Requests may also be triggered by programmatic sources such as web crawlers or loading of pages. If a user interface or program generates multiple requests against the Fusion Query Profiles API or Fusion Query Pipelines API, each individual call will be counted against the requests per year and the max queries per second (or hour).
## Download reports
You can download your data from the consumption dashboard. The data is downloaded as CSV files and displays your data by day for the selected date range.
You can export the detailed from the selected date range, all data, or the summary section only.
The ZIP file returns eight files:
* A CSV file with the requests report per day, which lists the date and the number of requests made that day.
* A CSV file with the records report per day, which lists the date and the maximum number of records stored in Fusion that day.
* A CSV file with the events that occurred for requests. This file reports only signature mismatches, which indicates that data was modified. Otherwise this file contains only the date and event title columns.
* A CSV file with the events that occurred for records. This file reports only signature mismatches, which indicates that data was modified. Otherwise this file contains only the date and event title columns.
* A signature file to accompany each of the CSV files.
## Date range
You may view your consumption over the past week, month,
or year.
You may also select a custom date range.
## Requests chart
Requests are the HTTP requests sent to Fusion. Fusion receives requests through the API and user searches. The requests chart displays the number of requests that Fusion has processed each day over the given time interval.
To view the requests per pipeline, point your mouse over **Requests across pipelines** and select **Requests per pipeline**.
## Documents chart
Records are the basic items that Fusion searches. They are also called documents in Fusion. The records chart displays the total number of records that your instance of Fusion has stored and indexed over the selected time period.
To view the documents per collection, point your mouse over **Documents across collections** and select **Documents* per collection*\*.
# Grafana Dashboards
Source: https://doc.lucidworks.com/docs/5/fusion/operations/monitoring-and-reporting/grafana-dashboards
[localhost link]: http://localhost:3000/docs/5/fusion/operations/monitoring-and-reporting/grafana-dashboards
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/monitoring-and-reporting/grafana-dashboards
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/136
Lucidworks provides several Grafana dashboards designed for Fusion. For more information, see **Configure Grafana, Prometheus, Promtail, and Loki in Fusion**.
Before you perform these installation instructions, you must delete any existing persistent volume claims (PVCs) related to Prometheus, Grafana, Promtail, and Loki on your namespace.
## Clone the `fusion-cloud-native` repository
Open a terminal window and run following command:
```
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
## Install Grafana
1. In your local `fusion-cloud-native` repository, run the following command for your `` and ``:
```
./install-prom.sh -c -n
```
The following is a sample output. The errors are related to resource limits on the sample cluster, and can be ignored. Similar errors may display for your cluster, and do not impact Grafana logging.
```
Adding the stable chart repo to helm repo list
"prometheus-community" already exists with the same configuration, skipping
"grafana" already exists with the same configuration, skipping
Installing Prometheus and Grafana for monitoring Fusion metrics ... this can take a few minutes.
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "ckotzbauer" chart repository
...Successfully got an update from the "lucidworks" chart repository
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "prometheus-community" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 2 charts
Downloading prometheus from repo https://prometheus-community.github.io/helm-charts
Downloading grafana from repo https://grafana.github.io/helm-charts
Deleting outdated charts
Release "fe-foundry-monitoring" does not exist. Installing it now.
Error: context deadline exceeded
Successfully installed Prometheus and Grafana into the fe-foundry namespace.
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
fe-foundry fe-foundry 11 2023-08-07 15:10:55.373825 -0700 PDT deployed fusion-5.8.0 5.8.0
fe-foundry-jupyter fe-foundry 2 2023-07-20 11:29:38.481329 -0700 PDT deployed fusion-jupyter-0.2.5 1.0
fe-foundry-monitoring fe-foundry 1 2023-08-10 11:41:06.113257 -0700 PDT failed fusion-monitoring-1.0.1 1.0.1
```
2. Using the Grafana service endpoint in the newly-installed Grafana helm release, run the following command:
```
kubectl get services
```
The following is a sample output.
```
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
grafana LoadBalancer 3000:32589/TCP 87m
```
If the output does not display, run the following command to expose Grafana, including an `EXTERNAL_IP` for your Grafana LoadBalancer service:
```
kubectl expose deployment --type=LoadBalancer --name=grafana --port=3000 --target-port=3000
```
## Install Loki
To obtain Loki from the helm chart repository, run the following command for the unique `` for your cluster:
```
helm upgrade --install --namespace= grafana/loki-stack
```
If you do no enter the `` correctly, an error similar to the following displays:
```
Error: rendered manifests contain a resource that already exists. Unable to continue with install: PodSecurityPolicy "loki" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-namespace" must equal "fe-foundry": current value is "ps-intl".
```
If the helm upgrade is successful, the following is a sample output.
```
Release "fe-foundry-loki" does not exist. Installing it now.
W0810 11:47:07.890370 39624 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0810 11:47:09.396246 39624 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME: fe-foundry-loki
LAST DEPLOYED: Thu Aug 10 11:47:07 2023
NAMESPACE: fe-foundry
STATUS: deployed
REVISION: 1
NOTES:
The Loki stack has been deployed to your cluster. Loki can now be added as a datasource in Grafana.
See http://docs.grafana.org/features/datasources/loki/ for more detail.
```
## Obtain Admin credentials for Grafana
1. After you validate Grafana is running by accessing `:3000`, run the following command to obtain an `` for your Grafana instance:
```
kubectl get secret --namespace -monitoring-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```
2. Sign in to Grafana and change the password for security purposes.
3. Run the following command to display the promtail pods that are running:
```
kubectl get pods | grep -i promtail | nl
```
Promtail pods must match the number of Kubernetes nodes since an instance of Promtail runs on each node.
## Add the Loki datsource
1. Sign in to Grafana and in the toolbar, click the arrow below Home to display all of the options.
2. In the Configuration section, click **Data sources**.
3. Click **Add new data source**.
4. In the search bar for the data source, enter **Loki**.
5. In the URL field on the Settings screen, enter your unique ``. The default port for Loki is `3100`.
If you encounter issues with the `` information, open a terminal and run `kubectl get services | grep loki` to display a list of every service with a name that contains `loki` along with its associated IP address and port.
6. Complete the other fields and click **Save & test**.
## Dashboards
### Pipeline and Stage Metrics
Both the Indexing and Query service provide per-pipeline, per-stage execution times for the 99th, 95th, 75th, and 50th percentiles. These are commonly used to determine where there may be a performance bottleneck when pipelines are executing slowly.
#### Indexing Service Dashboard
The Indexing Service has its own Grafana dashboard, which highlights key performance metrics for understanding the behavior of the Fusion 5 Indexing pipelines. It provides per-pipeline and per-stage execution time metrics, along with metrics for document indexing.
#### Query Service Dashboard
The Query Service also has its own Grafana dashboard that shows the performance of the pipelines and query stages that are being run. It shows the global and per-pod request metrics, in addition to per-pipeline query stage execution times.
### Gateway Metrics Dashboard
The Gateway Metrics provides visibility into the behavior of the API Gateway, which stands in front of Fusion 5 services.
It includes global request rate gauge, in addition to per-handler and per-route request metrics. This dashboard can be used for locating service slowdowns, as it can highlight particular routes or handlers that are responding slowly.
### JVM Metrics Dashboard
The JVM Metrics Dashboard provides information about the CPU and memory utilization of Fusion 5 services, along with some GC metrics. These can be used to assess the JVM performance of Fusion 5 microservices and may help locate bottlenecks, such as being CPU-pegged or hitting frequent garbage collection events that pause the application. Issues relating to garbage collection can often be solved by modifying the JVM options of the process and can then be monitored with this dashboard.
### Solr Dashboards
Solr has its metrics split into three distinct dashboards, each covering a separate area of concern for monitoring Solr performance.
##### Core Dashboard
The Solr Core dashboard contains metrics by the associated Solr core, including per-core metrics for requests, file system utilization, document processing and access, and handler usage.
##### Node Dashboard
The Solr Node dashboard contains per-node metrics including information about requests, errors, cores, thread pool utilization, and connections. Per-node metrics are useful for tracking down issues with a particular Solr node, especially when these issues do not affect all nodes.
##### System Dashboard
The Solr System Dashboard shows high-level system stats, including requests/response statistics, JVM metrics, and common operating system metrics, such as memory usage, file descriptor usage, and cpu utilization.
### Kubernetes Dashboards
The set of Kubernetes performance dashboards requires the installation of additional daemonsets to collect performance metrics from Kube.
Ensure that at least one namespace in your cluster uses the following configuration in the Prometheus `values.yaml` file for the `stable/prometheus` Helm chart.
```yaml wrap theme={"dark"}
kubeStateMetrics:
enabled: true
nodeExporter:
enabled: true
```
Additionally you will want to install cadvisor (likely into an infrastructure/system namespace):
```sh wrap theme={"dark"}
helm repo add ckotzbauer https://ckotzbauer.github.io/helm-charts
helm repo update
helm install cadvisor ckotzbauer/cadvisor
```
#### Kube Metrics Dashboard (replaced by Kube Node Dashboard)
There are Kubernetes metrics that can be queried through Grafana. However, these require installing a daemonset when setting up Prometheus, which displays information about running pods, memory allocation, and CRON jobs.
#### Kube Node Dashboard
There are Kubernetes metrics that can be queried through Grafana. However, these require installing a daemonset when setting up Prometheus, which displays information about running pods, load, network traffic, memory allocation and more. Some dashboards may not be rendered until you drill down into one or more specific nodes/instances at the top of the dashboard.
#### Kubernetes Persistence Volumes Dashboard
This dashboard shows you all of your Kubernetes PVCs, filterable by namespace and/or node. It provides you with information about disk utilization so you can monitor for PVCs that are nearing capacity.
#### Docker Host and Container Overview Dashboard
This dashboard provides you a great view of your running containers, filterable by namespace, node, and/or container name. It provides information about number of running containers, network traffic, memory utilization, disk IO, and load metrics.
## Downloads
* [Indexing Service](https://raw.githubusercontent.com/lucidworks/fusion-cloud-native/master/monitoring/helm/fusion-monitoring-stack/dashboards/dashboard_indexing_metrics.json)
* [Query Service](https://raw.githubusercontent.com/lucidworks/fusion-cloud-native/master/monitoring/helm/fusion-monitoring-stack/dashboards/dashboard_query_pipeline.json)
* [Gateway Metrics](https://raw.githubusercontent.com/lucidworks/fusion-cloud-native/master/monitoring/helm/fusion-monitoring-stack/dashboards/dashboard_gateway_metrics.json)
* [JVM Metrics](https://raw.githubusercontent.com/lucidworks/fusion-cloud-native/master/monitoring/helm/fusion-monitoring-stack/dashboards/dashboard_jvm_metrics.json)
* [Core](https://raw.githubusercontent.com/lucidworks/fusion-cloud-native/master/monitoring/helm/fusion-monitoring-stack/dashboards/dashboard_solr_core.json)
* [Node](https://raw.githubusercontent.com/lucidworks/fusion-cloud-native/master/monitoring/helm/fusion-monitoring-stack/dashboards/dashboard_solr_node.json)
* [System](https://raw.githubusercontent.com/lucidworks/fusion-cloud-native/master/monitoring/helm/fusion-monitoring-stack/dashboards/dashboard_solr_system.json)
* [Dashboard Kafka Overview](https://raw.githubusercontent.com/lucidworks/fusion-cloud-native/master/monitoring/helm/fusion-monitoring-stack/dashboards/dashboard_kafka_overview.json)
* [Dashboard Kube Node](https://raw.githubusercontent.com/lucidworks/fusion-cloud-native/master/monitoring/helm/fusion-monitoring-stack/dashboards/dashboard_kube_node.json)
* [Dashboard PVC Metrics](https://raw.githubusercontent.com/lucidworks/fusion-cloud-native/master/monitoring/helm/fusion-monitoring-stack/dashboards/dashboard_pvc_metrics.json)
* [Dashboard Templating](https://raw.githubusercontent.com/lucidworks/fusion-cloud-native/master/monitoring/helm/fusion-monitoring-stack/dashboards/dashboard_templating.json)
# Grafana Loki for log exploration
Source: https://doc.lucidworks.com/docs/5/fusion/operations/monitoring-and-reporting/grafana-loki
[localhost link]: http://localhost:3000/docs/5/fusion/operations/monitoring-and-reporting/grafana-loki
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/monitoring-and-reporting/grafana-loki
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/0b1k93
Grafana Loki is a log aggregation system designed to store and query logs. Loki is available for Fusion deployments as part of the Prometheus and Grafana integration. It is a powerful tool that can be adapted for your unique logging requirements.
With Grafana’s Explore UI and Loki, you can build queries to explore Fusion logs and build dashboards, visualizations, and alerts. To begin:
1. Log into your Grafana environment.
2. Navigate to Explore.
3. Change your data source to Loki.
The name of your Loki data source may be different in your deployment.
## Log exploration
Build queries in Explore using labels and operations. Labels are similar to facets or filters. Adding a label defines the stream of logs. Operations filter the stream for specific information. There are many types of operations, including aggregations, range functions, and filters. For example, you can apply a line filter to the log stream to ensure lines contain a string or match a RegEx pattern.
You can further customize the output using the Options menu.
See the following examples for common log queries you can adapt for your unique requirements. For more information on Loki queries, see [Grafana’s documentation on the Loki query editor](https://grafana.com/docs/grafana/latest/datasources/loki/query-editor/).
**Try the Code tool**
The examples in this article use the Builder tool. Alternatively, you can use the Code tool, which allows you to build queries using LogQL. For more information, see [Grafana’s documentation on LogQL](https://grafana.com/docs/loki/latest/logql/).
### Example workflow
These examples illustrate how to build a query that drills down to your area of interest. The values are for example purposes only and may not reflect your own environmental values.
#### Query all logs in a namespace
Add a label to filter for logs in the production namespace, `prd-a`. This focuses your stream to return all logs, regardless of context.
##### Configuration
| Type | Field | Value |
| - | - | - |
| **Label** | `namespace` | `prd-a` |
##### Screenshot
#### Query a specific service within the namespace
Add a label to examine logs for the Fusion query pipeline service. This focuses your stream on query pipeline operations and excludes other service operations.
##### Configuration
| Type | Field | Value |
| - | - | - |
| **Label** | `namespace` | `prd-a` |
| **Label** | `component` | `query-pipeline` |
Logs for some services may require you to query different components, depending on what information you are trying to discover. For example, the `fusion-indexing`, `connector-plugin`, and `connectors` components provide logs related to the indexing service.
**FAQ**
When should I use the `component` label instead of `container`?
The values available for the `component` label are also found for `container`, and the logs are identical. However, `component` values can be used for examining logs *and* metrics.
In general, prefer using the `component` label, so you can reuse the query for viewing metrics. Use the `container` label when you need logs for a value not associated with the `component` label.
##### Screenshot
#### Query a specific host for the query pipeline service
Add a label to focus the stream on a specific query pipeline service host by creating the pod label and finding the host in the list.
##### Configuration
| Type | Field | Value |
| - | - | - |
| **Label** | `namespace` | `prd-a` |
| **Label** | `component` | `query-pipeline` |
| **Label** | `pod` | `prd-a-query-pipeline-f9b9494c4-ws6nm` |
##### Screenshot
#### Basic query for errors
Look for errors with the query pipeline service by adding an operation that checks for lines containing the keyword "ERROR".
##### Configuration
| Type | Field | Value |
| - | - | - |
| **Label** | `namespace` | `prd-a` |
| **Label** | `component` | `query-pipeline` |
| **Label** | `pod` | `prd-a-query-pipeline-f9b9494c4-ws6nm` |
| **Operations** | `Line contains` | `ERROR` |
##### Screenshot
**Matches are case-sensitive**
The **Line contains** operation is case-sensitive and matches the value exactly. If you want `ERROR` to match the term "error" as well, use the **Line contains regex match** operation, and prefix the query term with `(?i)`, which ignores the case. For example, `(?i)ERROR` will match "ERROR", "error", "Error", and other variants.
#### Remove some logs from the stream
From the logs discovered so far, you may find a frequent error that you have confirmed is not the issue. That frequent error contains the text `.query_rewrite.sidecar`. Remove those errors from the log stream by adding a **Line does not contain** operation.
##### Configuration
| Type | Field | Value |
| - | - | - |
| **Label** | `namespace` | `prd-a` |
| **Label** | `component` | `query-pipeline` |
| **Label** | `pod` | `prd-a-query-pipeline-f9b9494c4-ws6nm` |
| **Operations** | `Line contains` | `ERROR` |
| **Operations** | `Line does not contain` | `.query_rewrite.sidecar` |
##### Screenshot
You have now focused your log stream on a specific namespace, service, service host, and keyword, and you have removed logs irrelevant to your interests.
### Analyzing logs
Once you have created a log stream and filtered it to show the logs you are interested in, you can explore individual logs.
1. Locate an individual log record in the list of logs.
2. Point to the log record, and a **Show context** option appears.
3. Click **Show context** to launch the context modal and view more messages from the log.
4. Close the modal and return to the list of logs.
5. Click a log record to expand key details.
From this view, you can select:
**Labels**
1. **Ad-hoc statistics.** View how frequently the label is found in the list of logs.
2. **Filter for value.** Add the label field and value to your query.
3. **Filter out value.** Remove the label field and value from your query.
**Detected fields**
1. **Ad-hoc statistics.** View how frequently the label is found in the list of logs.
2. **Show this field instead of the message.** This multi-select option replaces the title of individual logs with the selected values.
## Saving Loki queries
### Starred queries
View your query history by clicking the **Query history** button. By default, Loki stores your queries in Query History for 2 weeks. However, you can save queries indefinitely by clicking the star icon next to your query. These queries are found under Starred.
For more information, see [Grafana’s documentation on Query history](https://grafana.com/docs/grafana/latest/explore/query-management/#query-history).
### Custom dashboards
Lucidworks Search offers read-only access for Grafana. If you are a Lucidworks Search customer, you cannot create custom dashboards.
From your Loki queries, you have the option to create custom dashboards. Dashboards display data for your environment visually. To create a dashboard:
1. Define and run your query in the Explore UI, then click the **Add to dashboard** button.
2. Choose the **New dashboard** option, then click the **Open dashboard** button. Alternatively, add your query to an existing dashboard.\
A new dashboard appears containing the panel for your query.
3. Optional: Click the new panel title, then select **Edit**. Edit the panel and save your changes.
4. Click the **Save dashboard** button.
## Learn more
## Prerequisites
Before you view logs, complete the procedures in Configure Grafana, Prometheus, Promtail, and Loki in Fusion.
## View logs
Use the following steps to view the logs.
1. Log in to Grafana.
2. Select **Explore** from the icon menu.\\
3. Click on **Log Browser** to display the advanced query options.\\
4. Use the **Select labels to search in** options to display the desired filtering criteria.\\
5. Click **Show logs**.
1. A histogram shows for the timeline.\\
2. You also see logs.\\
3. If there were any errors, the stack trace shows.\\
# MCP Server Monitoring
Source: https://doc.lucidworks.com/docs/5/fusion/operations/monitoring-and-reporting/mcp-monitoring
Scrape the Fusion API gateway's per-route metrics and import a ready-made Grafana dashboard to visualize Lucidworks MCP Server usage and latency.
You can track MCP server usage using metrics exposed by the Fusion API gateway.
You only need to scrape a metrics endpoint and import the provided dashboard.
New to Lucidworks MCP?
See the [MCP Server overview](/docs/5/fusion/getting-data-out/mcp) for setup instructions and configuration details.
## How MCP usage is measured
All MCP traffic enters Fusion through the API gateway at the `/mcp` route.
The gateway records a per-route latency histogram, `gateway_request_time_secs`, on every request.
The label `routeId="mcp-server"` isolates MCP traffic from all other gateway routes, providing a clean MCP-only view without touching the MCP server itself.
The histogram carries these labels:
| Label | Meaning |
| - | - |
| `routeId` | Gateway route; `mcp-server` = MCP traffic |
| `httpMethod` | HTTP method (the MCP server uses `POST`) |
| `httpStatusCode` | Response status code (`200`, `202`, `404`, …) |
| `outcome` | `SUCCESSFUL`, `CLIENT_ERROR`, `SERVER_ERROR` |
| `status` | Spring HTTP status enum (`OK`, `ACCEPTED`, …) |
The histogram exposes three series families:
* `gateway_request_time_secs_count` — total number of requests
* `gateway_request_time_secs_sum` — sum of all request durations in seconds
* `gateway_request_time_secs_bucket` — histogram buckets that power the latency percentiles
## Prerequisites
* **Fusion 5.17 or later** with Lucidworks MCP Server deployed and serving traffic
* **Prometheus** (or any Prometheus-compatible TSDB) able to scrape your Fusion cluster
* **Grafana** with your Prometheus configured as a data source
The API gateway exposes Prometheus metrics at port **6764**, path **/actuator/prometheus**.
## Set up MCP monitoring
Configure Prometheus to scrape the gateway's `/actuator/prometheus` endpoint on port `6764`.
Use whichever method matches your setup:
Create a `ServiceMonitor` resource:
```yaml theme={"dark"}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: fusion-api-gateway
namespace:
labels:
release: prometheus # match your Prometheus Operator's serviceMonitorSelector
spec:
namespaceSelector:
matchNames:
-
selector:
matchLabels:
app.kubernetes.io/component: api-gateway
endpoints:
- targetPort: 6764
path: /actuator/prometheus
interval: 30s
```
If your Prometheus Operator selects `PodMonitor` resources instead of services, use a `PodMonitor` with the same selector and a `podMetricsEndpoints` entry on port `6764`.
If your Prometheus discovers targets via pod annotations, add these to the api-gateway pods (Helm `values.yaml` of the gateway):
```yaml theme={"dark"}
prometheus.io/scrape: "true"
prometheus.io/port: "6764"
prometheus.io/path: "/actuator/prometheus"
```
```yaml theme={"dark"}
scrape_configs:
- job_name: fusion-api-gateway
metrics_path: /actuator/prometheus
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_component]
action: keep
regex: api-gateway
- source_labels: [__address__]
action: replace
regex: ([^:]+)(?::\d+)?
replacement: $1:6764
target_label: __address__
```
**Cardinality note**: `gateway_request_time_secs` is a histogram across all gateway routes.
If you only want MCP metrics and wish to limit series, add a `metric_relabel_configs` `keep` rule for `gateway_request_time_secs.*` where `routeId="mcp-server"`.
Otherwise you will also collect percentiles for every other gateway route.
In Prometheus (or Grafana **Explore**), run:
```promql theme={"dark"}
sum by (routeId) (gateway_request_time_secs_count)
```
You should see a series with `routeId="mcp-server"` once the MCP server has received traffic.
If it's absent, confirm the scrape target is `UP` and that MCP requests have actually been made.
1. Download [mcp-usage-dashboard-portable.json](/assets/downloads/mcp-usage-dashboard-portable.json).
2. In Grafana, go to **Dashboards** → **New** → **Import**.
3. Upload the JSON file.
4. When prompted, select your **Prometheus data source**.
5. Click **Save**.
The dashboard has a **data source** dropdown and a **Namespace** filter at the top so you can scope it to your environment.
Alternatively, build your own dashboard using the reference PromQL queries below.
## Dashboard panels
The imported dashboard includes panels organized into three sections: overview metrics, latency analysis, and traffic breakdown.
### Overview metrics
| Panel | Description |
| - | - |
| **Total MCP Requests** | Total MCP calls over the selected range |
| **Request Rate** | Current throughput in requests per second |
| **Error Rate** | Percent of MCP requests that failed with 4xx or 5xx status codes; green (less than 1%), yellow (1-5%), red (greater than 5%) |
| **Avg Latency** | Average MCP response time in milliseconds |
### Latency metrics
| Panel | Description |
| - | - |
| **MCP Latency Percentiles** | p50, p95, and p99 latency distribution from the histogram |
| **Average Latency** | Average response time over time |
### Traffic metrics and errors
| Panel | Description |
| - | - |
| **MCP Request Rate by Status Code** | Throughput split by HTTP status code |
| **Error Rate** | Failure percentage over time |
| **Requests by Outcome** | Success versus client and server error breakdown |
| **Total MCP Requests Over Time** | Single-line graph showing total request volume trend |
## PromQL query examples
Use these queries to build custom dashboard panels or understand what powers each panel in the imported dashboard.
Replace `$namespace` with your Kubernetes namespace or use `.*` to match all namespaces.
```promql theme={"dark"}
sum(increase(gateway_request_time_secs_count{
routeId="mcp-server",
namespace=~"$namespace"
}[$__range]))
```
```promql theme={"dark"}
sum(increase(gateway_request_time_secs_count{
routeId="mcp-server",
namespace=~"$namespace"
}[$__interval]))
```
```promql theme={"dark"}
sum(rate(gateway_request_time_secs_count{
routeId="mcp-server",
namespace=~"$namespace"
}[$__rate_interval]))
```
```promql theme={"dark"}
100 * (
sum(rate(gateway_request_time_secs_count{
routeId="mcp-server",
namespace=~"$namespace",
outcome=~"CLIENT_ERROR|SERVER_ERROR"
}[$__rate_interval])) or vector(0)
) / sum(rate(gateway_request_time_secs_count{
routeId="mcp-server",
namespace=~"$namespace"
}[$__rate_interval]))
```
```promql theme={"dark"}
1000 * sum(rate(gateway_request_time_secs_sum{
routeId="mcp-server",
namespace=~"$namespace"
}[$__rate_interval])) / sum(rate(gateway_request_time_secs_count{
routeId="mcp-server",
namespace=~"$namespace"
}[$__rate_interval]))
```
```promql theme={"dark"}
1000 * histogram_quantile(0.50, sum by (le) (
rate(gateway_request_time_secs_bucket{
routeId="mcp-server",
namespace=~"$namespace"
}[$__rate_interval])
))
```
```promql theme={"dark"}
1000 * histogram_quantile(0.95, sum by (le) (
rate(gateway_request_time_secs_bucket{
routeId="mcp-server",
namespace=~"$namespace"
}[$__rate_interval])
))
```
```promql theme={"dark"}
1000 * histogram_quantile(0.99, sum by (le) (
rate(gateway_request_time_secs_bucket{
routeId="mcp-server",
namespace=~"$namespace"
}[$__rate_interval])
))
```
```promql theme={"dark"}
sum by (httpStatusCode) (
rate(gateway_request_time_secs_count{
routeId="mcp-server",
namespace=~"$namespace"
}[$__rate_interval])
)
```
## Troubleshooting
The scrape target may not be up, or no MCP traffic has occurred yet.
Verify with `up{job="fusion-api-gateway"}` and check that the `/mcp` endpoint is in use.
Confirm `gateway_request_time_secs_bucket` series exist.
Some scrape pipelines drop histogram buckets via a `keep`/`drop` relabel rule — make sure `_bucket` series are retained.
`gateway_request_time_secs` is measured in **seconds**; the dashboard multiplies by 1000 to display milliseconds.
If you have locked down the actuator endpoint, ensure your Prometheus scrape provides the necessary credentials.
## Notes
* This dashboard measures MCP usage at the **gateway**, which is the cleanest signal for request volume, status, and latency.
It does **not** break usage down per MCP **tool** (such as `search_documents`, `get_document`) — that would require additional instrumentation in the MCP server.
* The same metric (`spring_cloud_gateway_requests_seconds`) is also emitted by the gateway as a Micrometer summary; this guide uses the `gateway_request_time_secs` histogram because it supports latency percentiles.
## Related topics
* [Lucidworks MCP Server overview](/docs/5/fusion/getting-data-out/mcp)
# OpenTelemetry Distributed Tracing
Source: https://doc.lucidworks.com/docs/5/fusion/operations/monitoring-and-reporting/opentelemetry-implementation-fusion
How OpenTelemetry distributed tracing improves performance visibility and incident response for Fusion deployments
OpenTelemetry (OTel) distributed tracing gives request-level visibility across the Fusion microservice stack. Where Prometheus provides aggregate metrics, OTel traces individual requests from the moment they enter the API Gateway through the Query Service and into Solr, making it possible to pinpoint exactly which service is responsible for a slowdown.
**Version and deployment applicability:**
* **Self-hosted Fusion 5.17 and later**: You can configure the OTLP exporter to send traces to a collector you provide. See [Configure a customer-provided collector](#configure-a-customer-provided-collector-self-hosted-fusion-5-17-and-later).
* **Self-hosted Fusion 5.9.x**: OTLP export is not available. Traces are exported using Zipkin instead. See [Fusion 5.9.x: sending observability data to your own platform](#fusion-5-9-x-sending-observability-data-to-your-own-platform).
The services that support OpenTelemetry are API Gateway, Query, Indexing, Admin, and Apps Manager.
## How it works
The following flow shows the default pipeline used in Lucidworks-managed environments, where spans are sent to Grafana Tempo. Self-hosted Fusion 5.17 and later uses the same span instrumentation, but you can redirect the OTel Collector output to a collector you provide instead of Grafana Tempo. See [Configure a customer-provided collector](#configure-a-customer-provided-collector-self-hosted-fusion-5-17-and-later).
```mermaid theme={"dark"}
flowchart LR
accTitle: OpenTelemetry distributed tracing flow
accDescr: A flowchart showing how OpenTelemetry traces flow from incoming requests through the API Gateway and Query Service to the OTel Collector and Grafana Dashboard
A["Incoming Request"] --> B["API Gateway"]
B --"traceparent header propagated"--> C["Query Service"]
C --> OTel["OTel Collector"]
OTel --> Tempo["Grafana Tempo"]
Tempo --> Dashboard["Grafana Dashboard"]
C --> Kafka
C --> Feign
subgraph spans["Query Service child spans"]
Kafka["Kafka span (if enabled)"] --> Solr["Solr span"]
Feign["Feign/downstream span"]
end
```
The API Gateway receives the incoming request and creates the root trace span. It propagates trace context downstream using a `traceparent` header. Only query-path traffic is sampled, which keeps overhead minimal.
The Query Service picks up the trace context and creates child spans for each integration it calls. The following spans are created for every traced request:
* **Solr span**: captures query execution time within Solr.
* **Feign/downstream span**: captures HTTP calls made to downstream services.
The following span is optional and enabled per environment:
* **Kafka span**: captures Kafka producer and consumer calls made during query processing.
The OTel Collector receives spans from the Query Service and batches them for export.
The OTel Collector forwards the batched spans to Grafana Tempo, which stores them for search and analysis.
Traces stored in Grafana Tempo are available in the Grafana Dashboard, where Lucidworks engineers can search, filter, and drill into individual traces to identify performance bottlenecks.
## Viewing traces in Grafana
You can use the Grafana Tempo dashboard to search and visualize traces.
A trace can be filtered by service, time window, latency threshold, or error status.
Drilling into a trace shows the full span tree: API Gateway root span, Query Service child spans, and Solr spans nested within them.
This makes the following workflow possible:
1. Open Grafana and navigate to the Tempo traces dashboard.
2. Select the time window that corresponds to the reported latency spike or other incident.
3. Identify the slowest trace for that window.
4. Drill into the span tree to see which service or integration was the bottleneck.
This full workflow is achievable in under five minutes, without code changes or log searches.
## Instrumented integrations
| Integration | What it covers | How it's enabled |
| - | - | - |
| Solr | Query execution time within Solr for each request | Configurable |
| Kafka | Kafka producer/consumer calls made during query processing | Configurable |
| Feign HTTP client (`http`) | HTTP calls between Fusion microservices | Configurable |
| JDBC | Database calls made by Fusion services | Configurable on the Indexing and Apps Manager services only |
Solr, Kafka, Feign HTTP client, and JDBC tracing are each controlled by a boolean Helm value, `opentelemetry.integrations.` (for example, `opentelemetry.integrations.solr`), set in each service's Helm chart.
Each defaults to `false`, so set the ones you want to `true` to enable that integration's spans.
Not every integration is available on every service — see [Configure a customer-provided collector](#configure-a-customer-provided-collector-self-hosted-fusion-5-17-and-later) for which services support which integrations.
Trace volume and overhead can be tuned per integration without touching application code or requiring a Fusion release.
```yaml theme={"dark"}
opentelemetry:
integrations:
solr: false
kafka: false
http: false
jdbc: false
```
## Configure a customer-provided collector (self-hosted Fusion 5.17 and later)
Self-hosted Fusion 5.17 and later can export OTLP traces directly to a collector you provide, instead of Grafana Tempo. Five services support this:
* API Gateway
* Query
* Indexing
* Admin
* Apps Manager
The Query, Indexing, Admin, and Apps Manager services all share the same configuration structure; only the API Gateway differs, since it's the entry point that also controls trace propagation to each downstream service.
Both sides of a trace must be configured consistently. If a downstream service is listed as `true` in the API Gateway's `opentelemetry.services` map, that service must also have `opentelemetry.enabled: true` in its own Helm values — otherwise the trace chain breaks or spans are orphaned.
| Gateway `services.` | Service `opentelemetry.enabled` | Result |
| - | - | - |
| `true` | `true` | Full end-to-end trace: gateway and service spans linked |
| `true` | `false` | Gateway exports spans; service does not continue the trace — broken chain |
| `false` | `true` | Service exports spans; gateway does not propagate context — orphaned spans |
| `false` | `false` | No tracing for that service |
The only valid states are both enabled or both disabled for a given service.
The API Gateway is the single entry point for external traffic. It creates the root span and, for each service enabled in `opentelemetry.services`, propagates trace context downstream using a `traceparent` header. Enabling OTel on the gateway automatically disables Spring Cloud Sleuth.
| Value | Description |
| - | - |
| `opentelemetry.enabled` | Master toggle. Enables OTLP export and disables Spring Cloud Sleuth. |
| `opentelemetry.endpoint` | The OTLP endpoint of your collector. Optional — defaults to a collector on the same node, using the pod's `NODE_IP`. |
| `opentelemetry.protocol` | The OTLP protocol to use: `grpc` or `http`. |
| `opentelemetry.services.query` | Propagates trace context to the Query service and exports gateway spans for requests routed to it. |
| `opentelemetry.services.indexing` | Propagates trace context to the Indexing service. |
| `opentelemetry.services.admin` | Propagates trace context to the Admin service. |
| `opentelemetry.services.appsmanager` | Propagates trace context to the Apps Manager service. |
```yaml theme={"dark"}
opentelemetry:
enabled: false
endpoint: "http://$(NODE_IP):4317"
protocol: grpc
services:
query: false
indexing: false
admin: false
appsmanager: false
```
These services share the same configuration structure under `opentelemetry` in their Helm values.
| Value | Description |
| - | - |
| `opentelemetry.enabled` | Master toggle. Must be `true` if this service is also enabled in the API Gateway's `services` map. |
| `opentelemetry.endpoint` | The OTLP endpoint of your collector. Optional — defaults to a collector on the same node, using the pod's `NODE_IP`. |
| `opentelemetry.protocol` | The OTLP protocol to use: `grpc` or `http`. |
| `opentelemetry.serviceVersion` | Version tag attached to all spans and logs for this service. |
| `opentelemetry.logging.enabled` | Exports application logs to the OTLP collector, independent of tracing. |
| `opentelemetry.tracing.enabled` | Enables span export. Can be disabled independently of the master toggle, for example to export logs only. |
| `opentelemetry.tracing.samplingRatio` | Fraction of traces to sample, from `0.0` to `1.0`. |
| `opentelemetry.integrations.*` | Enables span coverage per integration. See [Instrumented integrations](#instrumented-integrations) — not every integration is available on every service. |
```yaml theme={"dark"}
opentelemetry:
enabled: false
endpoint: "http://$(NODE_IP):4317"
protocol: grpc
serviceVersion: "1.0.0"
logging:
enabled: false
tracing:
enabled: true
samplingRatio: 1.0
integrations:
solr: false
kafka: false
http: false
jdbc: false
```
Point the endpoint at any OTLP-compatible collector. Traces continue to include the same span coverage described in [Instrumented integrations](#instrumented-integrations); only the export destination changes.
## Fusion 5.9.x: sending observability data to your own platform
Self-hosted Fusion 5.9.x does not support OTLP export. Use the following supported paths to send observability data to your own platform instead.
Fusion 5.9.x exports traces using Spring Cloud Sleuth with Zipkin/Brave, in Zipkin v2 format. Configure the export endpoint with:
| Value | Description |
| - | - |
| `zipkin.enabled` | Enables trace export. |
| `zipkin.baseUrl` | The base URL of your Zipkin-compatible collector. |
Most modern observability platforms and OTel Collector builds accept Zipkin v2 format through a Zipkin receiver.
Fusion exposes metrics in Prometheus format. A customer-provided OTel Collector can scrape these endpoints using a `prometheus` receiver. See [Prometheus, Grafana, and Loki in Fusion](/docs/5/fusion/operations/monitoring-and-reporting/prometheus-grafana-loki) for the available endpoints.
Fusion writes structured logs to stdout. A customer-provided OTel Collector can ingest these using a `filelog` receiver. See [Grafana Loki for log exploration](/docs/5/fusion/operations/monitoring-and-reporting/grafana-loki) for the logging format Fusion produces.
# Monitoring
Source: https://doc.lucidworks.com/docs/5/fusion/operations/monitoring-and-reporting/overview
[localhost link]: http://localhost:3000/docs/5/fusion/operations/monitoring-and-reporting/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/monitoring-and-reporting/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/132
Fusion has functions built in for observability and monitoring with Fusion’s tools or third-party monitoring tools.
Fusion system metrics are logged and available in the UI, in log files, and through the REST API.
The [Mutual Transport Layer Security](/docs/5/fusion/operations/security/mtls-fusion) service provides network-layer authentication and requires both the client and server to present valid certificates during the TLS connection process. mTLS authentication events are logged in Google Cloud Logging.
You can use the following third-party tools to process and display metrics:
* Prometheus is a solution that is installed during the Fusion installation. Prometheus records metrics, stores them in a time series datastore, and then uses queries to return data for visualizations and alerts. For more information, see the [Prometheus documentation](https://prometheus.io/docs).
* Grafana is a tool installed during the Fusion installation. Grafana provides visualizations of multiple datasources and connects to Prometheus, which lets you access your Fusion data when you create or edit the Grafana dashboards. Fusion provides pre-configured Grafana dashboards, but you can also create custom dashboards to analyze metrics further. For more information, see the [Grafana documentation](https://grafana.com/docs).
* Loki is an optional, additional tool that stores, aggregates, and queries logs. You can filter logs based on keywords from specific Fusion namespaces, services, and other elements. For more information, see the [Grafana’s Loki documentation](https://grafana.com/docs/loki).
For more information about configuring these tools, see **Configure Grafana, Prometheus, Promtail, and Loki in Fusion**.
Before you perform these installation instructions, you must delete any existing persistent volume claims (PVCs) related to Prometheus, Grafana, Promtail, and Loki on your namespace.
## Clone the `fusion-cloud-native` repository
Open a terminal window and run following command:
```
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
## Install Grafana
1. In your local `fusion-cloud-native` repository, run the following command for your `` and ``:
```
./install-prom.sh -c -n
```
The following is a sample output. The errors are related to resource limits on the sample cluster, and can be ignored. Similar errors may display for your cluster, and do not impact Grafana logging.
```
Adding the stable chart repo to helm repo list
"prometheus-community" already exists with the same configuration, skipping
"grafana" already exists with the same configuration, skipping
Installing Prometheus and Grafana for monitoring Fusion metrics ... this can take a few minutes.
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "ckotzbauer" chart repository
...Successfully got an update from the "lucidworks" chart repository
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "prometheus-community" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 2 charts
Downloading prometheus from repo https://prometheus-community.github.io/helm-charts
Downloading grafana from repo https://grafana.github.io/helm-charts
Deleting outdated charts
Release "fe-foundry-monitoring" does not exist. Installing it now.
Error: context deadline exceeded
Successfully installed Prometheus and Grafana into the fe-foundry namespace.
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
fe-foundry fe-foundry 11 2023-08-07 15:10:55.373825 -0700 PDT deployed fusion-5.8.0 5.8.0
fe-foundry-jupyter fe-foundry 2 2023-07-20 11:29:38.481329 -0700 PDT deployed fusion-jupyter-0.2.5 1.0
fe-foundry-monitoring fe-foundry 1 2023-08-10 11:41:06.113257 -0700 PDT failed fusion-monitoring-1.0.1 1.0.1
```
2. Using the Grafana service endpoint in the newly-installed Grafana helm release, run the following command:
```
kubectl get services
```
The following is a sample output.
```
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
grafana LoadBalancer 3000:32589/TCP 87m
```
If the output does not display, run the following command to expose Grafana, including an `EXTERNAL_IP` for your Grafana LoadBalancer service:
```
kubectl expose deployment --type=LoadBalancer --name=grafana --port=3000 --target-port=3000
```
## Install Loki
To obtain Loki from the helm chart repository, run the following command for the unique `` for your cluster:
```
helm upgrade --install --namespace= grafana/loki-stack
```
If you do no enter the `` correctly, an error similar to the following displays:
```
Error: rendered manifests contain a resource that already exists. Unable to continue with install: PodSecurityPolicy "loki" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-namespace" must equal "fe-foundry": current value is "ps-intl".
```
If the helm upgrade is successful, the following is a sample output.
```
Release "fe-foundry-loki" does not exist. Installing it now.
W0810 11:47:07.890370 39624 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0810 11:47:09.396246 39624 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME: fe-foundry-loki
LAST DEPLOYED: Thu Aug 10 11:47:07 2023
NAMESPACE: fe-foundry
STATUS: deployed
REVISION: 1
NOTES:
The Loki stack has been deployed to your cluster. Loki can now be added as a datasource in Grafana.
See http://docs.grafana.org/features/datasources/loki/ for more detail.
```
## Obtain Admin credentials for Grafana
1. After you validate Grafana is running by accessing `:3000`, run the following command to obtain an `` for your Grafana instance:
```
kubectl get secret --namespace -monitoring-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```
2. Sign in to Grafana and change the password for security purposes.
3. Run the following command to display the promtail pods that are running:
```
kubectl get pods | grep -i promtail | nl
```
Promtail pods must match the number of Kubernetes nodes since an instance of Promtail runs on each node.
## Add the Loki datsource
1. Sign in to Grafana and in the toolbar, click the arrow below Home to display all of the options.
2. In the Configuration section, click **Data sources**.
3. Click **Add new data source**.
4. In the search bar for the data source, enter **Loki**.
5. In the URL field on the Settings screen, enter your unique ``. The default port for Loki is `3100`.
If you encounter issues with the `` information, open a terminal and run `kubectl get services | grep loki` to display a list of every service with a name that contains `loki` along with its associated IP address and port.
6. Complete the other fields and click **Save & test**.
## Splunk logging
If your policy requires Splunk instead of Grafana, use the instructions in **Logging to Splunk**.
Fusion services write log messages to `stdout` by default.
You can use OpenShift as your cluster’s logging pipeline to send container `stdout` and `stderr` to Splunk.
### Forward `stdout` to Splunk
1. Set Fusion logging values in your custom values YAML to enable JSON logs. This is optional, but recommended to make parsing easier.
```yaml theme={"dark"}
# values.yaml
global:
logging:
jsonOutput: true
```
2. Configure your logging stack to ship pod `stdout` to Splunk.
### OpenShift
If Fusion runs on OpenShift and already writes to `stdout`, use OpenShift’s log forwarding to Splunk HTTP Event Collector (HEC).
1. Create a secret with your Splunk HEC token:
```bash theme={"dark"}
oc -n openshift-logging create secret generic splunk-hec-secret \
--from-literal=hecToken=YOUR_HEC_TOKEN
```
2. Create or update a ClusterLogForwarder to send application logs to Splunk:
```yaml theme={"dark"}
apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
name: instance
namespace: openshift-logging
spec:
outputs:
- name: splunk
type: splunk
url: https://splunk.example.com:8088
secret:
name: splunk-hec-secret
pipelines:
- name: app-to-splunk
inputRefs: [ application ]
outputRefs: [ splunk ]
```
3. Verify ingestion in Splunk. Search for Kubernetes labels such as `kubernetes.pod_name` or `app.kubernetes.io/name`.
OpenShift provides its own collection and forwarding path so you do not need to install extra agents inside Fusion pods.
# Prometheus, Grafana, and Loki in Fusion
Source: https://doc.lucidworks.com/docs/5/fusion/operations/monitoring-and-reporting/prometheus-grafana-loki
[localhost link]: http://localhost:3000/docs/5/fusion/operations/monitoring-and-reporting/prometheus-grafana-loki
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/monitoring-and-reporting/prometheus-grafana-loki
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/131
Fusion 5.6 and later utilizes Prometheus, Grafana, and Loki for metrics and log analytics.
## Prometheus
Prometheus provides a powerful solution for recording metrics, persisting them to a pluggable time series datastore, and querying them for both visualizations and alerting purposes. Metrics data is pulled from Fusion and stored in a time series datastore.
For more information, see the [Prometheus documentation](https://prometheus.io/docs).
## Grafana
Grafana provides an industry-standard metric visualization tool that supports complex visualizations over any number of different pluggable data sources, organized into panels and managed dashboards. Grafana connects Prometheus, allowing you to access your Fusion data when creating or editing panels in your dashboards.
Pre-configured Grafana dashboards are provided with Fusion deployments. However, custom dashboards extend your metrics analysis capabilities.
Lucidworks Search offers read-only access for Grafana. If you are a Lucidworks Search customer, you cannot create custom dashboards.
For more information, see the [Grafana documentation](https://grafana.com/docs).
## Loki
Loki is a log aggregation system designed to store and query logs. With Loki, you can define a log stream from Fusion’s namespaces, services, pods, and more. Then, you can filter the log stream for key data, such as keyword matches for "error."
For more information, see the [Grafana’s Loki documentation](https://grafana.com/docs/loki).
## Setup and configuration
Prometheus and Grafana are enabled during Fusion installation, and Loki is an optional addition to the Prometheus and Grafana integration. For more information, see **Configure Grafana, Prometheus, Promtail, and Loki in Fusion**.
Before you perform these installation instructions, you must delete any existing persistent volume claims (PVCs) related to Prometheus, Grafana, Promtail, and Loki on your namespace.
## Clone the `fusion-cloud-native` repository
Open a terminal window and run following command:
```
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
## Install Grafana
1. In your local `fusion-cloud-native` repository, run the following command for your `` and ``:
```
./install-prom.sh -c -n
```
The following is a sample output. The errors are related to resource limits on the sample cluster, and can be ignored. Similar errors may display for your cluster, and do not impact Grafana logging.
```
Adding the stable chart repo to helm repo list
"prometheus-community" already exists with the same configuration, skipping
"grafana" already exists with the same configuration, skipping
Installing Prometheus and Grafana for monitoring Fusion metrics ... this can take a few minutes.
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "ckotzbauer" chart repository
...Successfully got an update from the "lucidworks" chart repository
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "prometheus-community" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 2 charts
Downloading prometheus from repo https://prometheus-community.github.io/helm-charts
Downloading grafana from repo https://grafana.github.io/helm-charts
Deleting outdated charts
Release "fe-foundry-monitoring" does not exist. Installing it now.
Error: context deadline exceeded
Successfully installed Prometheus and Grafana into the fe-foundry namespace.
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
fe-foundry fe-foundry 11 2023-08-07 15:10:55.373825 -0700 PDT deployed fusion-5.8.0 5.8.0
fe-foundry-jupyter fe-foundry 2 2023-07-20 11:29:38.481329 -0700 PDT deployed fusion-jupyter-0.2.5 1.0
fe-foundry-monitoring fe-foundry 1 2023-08-10 11:41:06.113257 -0700 PDT failed fusion-monitoring-1.0.1 1.0.1
```
2. Using the Grafana service endpoint in the newly-installed Grafana helm release, run the following command:
```
kubectl get services
```
The following is a sample output.
```
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
grafana LoadBalancer 3000:32589/TCP 87m
```
If the output does not display, run the following command to expose Grafana, including an `EXTERNAL_IP` for your Grafana LoadBalancer service:
```
kubectl expose deployment --type=LoadBalancer --name=grafana --port=3000 --target-port=3000
```
## Install Loki
To obtain Loki from the helm chart repository, run the following command for the unique `` for your cluster:
```
helm upgrade --install --namespace= grafana/loki-stack
```
If you do no enter the `` correctly, an error similar to the following displays:
```
Error: rendered manifests contain a resource that already exists. Unable to continue with install: PodSecurityPolicy "loki" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-namespace" must equal "fe-foundry": current value is "ps-intl".
```
If the helm upgrade is successful, the following is a sample output.
```
Release "fe-foundry-loki" does not exist. Installing it now.
W0810 11:47:07.890370 39624 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0810 11:47:09.396246 39624 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME: fe-foundry-loki
LAST DEPLOYED: Thu Aug 10 11:47:07 2023
NAMESPACE: fe-foundry
STATUS: deployed
REVISION: 1
NOTES:
The Loki stack has been deployed to your cluster. Loki can now be added as a datasource in Grafana.
See http://docs.grafana.org/features/datasources/loki/ for more detail.
```
## Obtain Admin credentials for Grafana
1. After you validate Grafana is running by accessing `:3000`, run the following command to obtain an `` for your Grafana instance:
```
kubectl get secret --namespace -monitoring-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```
2. Sign in to Grafana and change the password for security purposes.
3. Run the following command to display the promtail pods that are running:
```
kubectl get pods | grep -i promtail | nl
```
Promtail pods must match the number of Kubernetes nodes since an instance of Promtail runs on each node.
## Add the Loki datsource
1. Sign in to Grafana and in the toolbar, click the arrow below Home to display all of the options.
2. In the Configuration section, click **Data sources**.
3. Click **Add new data source**.
4. In the search bar for the data source, enter **Loki**.
5. In the URL field on the Settings screen, enter your unique ``. The default port for Loki is `3100`.
If you encounter issues with the `` information, open a terminal and run `kubectl get services | grep loki` to display a list of every service with a name that contains `loki` along with its associated IP address and port.
6. Complete the other fields and click **Save & test**.
* [Grafana Loki for log exploration](/docs/5/fusion/operations/monitoring-and-reporting/grafana-loki)
# Operations
Source: https://doc.lucidworks.com/docs/5/fusion/operations/overview
[localhost link]: http://localhost:3000/docs/5/fusion/operations/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/4181
To ensure Fusion operates efficiently and successfully, use the information in these topics to install and configure Fusion correctly and monitor system processes.
## Fusion 5 Survival Guide
The [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview) details how to install, configure, and run Fusion 5 in production in high availability on Kubernetes. This comprehensive cloud operations guide is curated by the Fusion Engineering Team.
## Security options available
Security is essential to ensure successful operations.
The [Security concept page](/docs/5/fusion/operations/security/overview) provides general information about authentication and authorization of users, as well as links to more detailed instructions. In addition, the [Mutual Transport Layer Security](/docs/5/fusion/operations/security/mtls-fusion) topic details information about network-layer authentication for Fusion that requires both the client and server to present valid certificates during the TLS connection process.
## Monitoring functions
After Fusion is configured and operational, monitoring is required to maintain successful platform operations. The [Monitoring](/docs/5/fusion/operations/monitoring-and-reporting/overview) section provides information and links to observe, investigate, and troubleshoot issues.
## Fusion jobs and job scheduling
Jobs are system tasks that load and aggregate data, train models, and perform other activities to help keep your search platform running efficiently. For more information, see [Jobs and scheduling](/docs/5/fusion/operations/jobs-and-scheduling/overview).
# API Keys
Source: https://doc.lucidworks.com/docs/5/fusion/operations/security/access-control/api-keys
[localhost link]: http://localhost:3000/docs/5/fusion/operations/security/access-control/api-keys
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/security/access-control/api-keys
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/7zriph
API Keys are an access control feature that allows users to create, store, and manage their API credentials in a secure manner. Only Fusion users with admin permissions can manage API Keys.
You can use an API key instead of a username and password combination to power your search application or operations.
Pass your API key to Fusion using either `apiKey` or `x-api-key`, as in these examples:
```bash wrap theme={"dark"}
curl -H "x-api-key: API_KEY" "https://FUSION_HOST/api/query/status"
curl -H "apiKey: API_KEY" "https://FUSION_HOST/api/query/status"
curl https://FUSION_HOST/api/query/status?apiKey=API_KEY
```
There isn’t a hard limit on the amount of API Keys you can create.
**Important**
API keys require an **apikey** realm in Fusion to work. If your Fusion cluster was created prior to Fusion 5.6.0, it lacks the required realm for API keys. Upgrading to Fusion 5.6.0 through 5.10.0 does not create the realm automatically.
For more information, see [Create an apikey realm](#create-an-apikey-realm).
## Manage API keys in the Fusion UI
You can create, delete, and view the names of API keys in the UI:
1. Navigate to **System > Access Control > API Keys**.
2. Select **Add API Key**.
3. Give the API key a name.
4. Assign the appropriate roles to give the API key the correct permissions for your search application.
5. Select **Generate Key**.
6. Copy the API key that appears. Save this key in a secure location.
API keys are treated like an automatically generated password. They are only visible during the key creation and do not display again. You must copy and securely store the API key to see it again after creation.
## Manage API keys with the API
You can use `GET`, `POST`, and `DELETE` HTTP requests to manage your API keys. The following examples use an API key to authenticate. To use an API key in an API request, use the `x-api-key` HTTP request header.
### Create a new API key
Use a `POST` HTTP request to create a new API key. Assign the appropriate roles to give the API key the correct permissions for your search application.
API keys are treated like an automatically generated password. They are only visible during the key creation and do not display again. You must copy and securely store the API key to see it again after creation.
Request
```bash wrap theme={"dark"}
curl -H "x-api-key: API_KEY" -X POST "https://FUSION_HOST:FUSION_PORT/api/api-key" -d '
{
"name": "searchaccess",
"id": "api-key-searchaccess-16744867518441",
"roles": ["rules", "search-template", "search"]
}
'
```
Response
```json wrap theme={"dark"}
{
"name": "searchaccess",
"id": "api-key-searchaccess-16744867518441",
"key": "API_KEY",
"roles": [
"rules", "search-template", "search"
],
"createdAt": "2023-01-23T15:12:32Z"
}
```
The value of the `key` field is the API key.
### List all available API keys
Use a `GET` HTTP request to view all API keys.
The API key values are not included in the response. You can only access API keys at the time of creation.
Request
```bash wrap theme={"dark"}
curl -H "x-api-key: API_KEY" -X GET "https://FUSION_HOST:FUSION_PORT/api/api-key"
```
Response
```json wrap theme={"dark"}
[{
"name": "searchaccess",
"id": "api-key-searchaccess-16744867518441",
"createdAt": "2023-01-23T15:12:32Z",
"roles": ["rules", "search-template", "search"]
}, {
"name": "adminaccess",
"id": "api-key-test-5-16630249022972",
"createdAt": "2022-09-12T23:21:42Z",
"roles": ["admin"]
}]
```
### Get details about an API key
Use a `GET` HTTP request to get the details of a specific API key. You must supply the ID of the API key as part of your request URL.
Request
```bash wrap theme={"dark"}
curl -H "x-api-key: API_KEY" -X GET "https://FUSION_HOST:FUSION_PORT/api/api-key/API_KEY_ID"
```
Response
```json wrap theme={"dark"}
{
"name": "searchaccess",
"id": "api-key-searchaccess-16744867518441",
"createdAt": "2023-01-23T15:12:32Z",
"roles": ["rules", "search-template", "search"]
}
```
### Delete an API key
Use a `DELETE` HTTP request to delete a specific API key. You must supply the ID of the API key as part of your request.
The request returns an HTTP `200 OK` status code when the API key is successfully deleted.
Request
```bash wrap theme={"dark"}
curl -H "x-api-key: API_KEY" -X DELETE "https://FUSION_HOST:FUSION_PORT/api/api-key/API_KEY_ID"
```
## Create an apikey realm
This section only applies to Fusion clusters created prior to Fusion 5.6.0, but have since been updated to 5.6.0 through 5.10.0.
API keys require an **apikey** realm in Fusion to work. If your Fusion cluster was created prior to Fusion 5.6.0, it lacks the required realm for API keys. Upgrading to Fusion 5.6.0 through 5.10.0 does not create the realm automatically.
As a result, when you use an API key, you’ll receive an HTTP `401 Unauthorized` status code:
```json wrap theme={"dark"}
{
"service": "proxy",
"error": "unauthorized",
"path": "/api/query/status",
"timestamp": "2023-01-25T18:01:44.059096"
}
```
In order to use API keys with an upgraded Fusion cluster, you must create the **apikey** realm using a `POST` HTTP request.
This cannot be done in the Fusion UI.
Request
```bash wrap theme={"dark"}
curl -u USERNAME:PASSWORD -X POST "https://FUSION_HOST:FUSION_PORT/api/realm-configs" -H 'Content-type: application/json' -d '
{
"realmType": "apikey",
"name": "apikey",
"enabled": true
}
'
```
After creating the **apikey** realm, you can use API keys to perform API requests as expected.
### Examples
Using an API key for a GET request:
```bash wrap theme={"dark"}
curl --request GET \
--url https://fusion_host.com/api/query-stage-plugins \
--header 'Accept: application/json' \
--header 'x-api-key: 123'
```
Using an API key for a POST request:
```bash wrap theme={"dark"}
curl --request POST \
--url https://fusion_host.com/api/query/id/ \
--header 'Accept: application/*, text/*' \
--header 'Content-Type: application/json' \
--header 'x-api-key: 123'
```
# Access Control
Source: https://doc.lucidworks.com/docs/5/fusion/operations/security/access-control/overview
[localhost link]: http://localhost:3000/docs/5/fusion/operations/security/access-control/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/security/access-control/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/124
## User Authentication and Authorization
Fusion provides application security by restricting access to known [users](/docs/5/fusion/operations/security/access-control/users)
via a two-stage process consisting of:
* **Authentication.** users must sign on using a username and password.
* **Authorization.** each username is associated with one or more permissions which specify the Fusion UI components and REST API requests that user has access to.
[Permissions](/docs/5/fusion/operations/security/access-control/permissions) can be restricted to specific endpoints and path parameters.
[Roles](/docs/5/fusion/operations/security/access-control/roles) are named sets of permissions which provide access to a specific function.
The access control component runs in the same process as the Fusion UI.
It referred to as the "auth proxy" because it handles authentication and authorization for all requests to the Fusion REST API services.
All requests to Fusion must be authenticated, as described in section **User Access Request Parameters**.
The [Mutual Transport Layer Security](/docs/5/fusion/operations/security/mtls-fusion) service provides network-layer authentication and complements application-layer authentication. mTLS requires both the client and server to present valid certificates during the TLS connection process.
Fusion requests must come from a known user, i.e., a user with a unique user id (UUID).
Fusion’s ZooKeeper registry tracks all users across all realms.
Usernames must be unique within a realm.
Fusion creates a globally unique user ID for all users based on the combination of username and realm.
All requests to the Fusion REST API require either a username, password, and security realm name,
or the session cookie which contains the unique user ID.
## Per-Request Authentication
To pass authentication information with each request, the realmName is specified as a query parameter on the request itself:
```bash wrap theme={"dark"}
curl -u joe.smith:password123 "http://www.acme.com:8764/api/collections?realmName=acmeLDAP"
```
The default realmName parameter is "native", so for native authentication, this parameter can be omitted.
## Session Cookies
The Fusion UI service endpoint "api/session" can be used to generate a session cookie which contains the unique user id
via a POST request whose body consists of a JSON object which contains the username, password information.
For users belonging to a realm other than the native realm, the request parameter "realmName" must be specified.
The command to generate a session cookie for the admin user with password "password123" is:
```bash theme={"dark"}
curl \
-c cookie -i -X POST -H "Content-type:application/json" -d @- \
http://localhost:8764/api/session?realmName=native \
<`. The argument `-d @-` reads the data from stdin.
The header output shows the cookie information:
```bash wrap theme={"dark"}
HTTP/1.1 201 Created
Set-Cookie: id=996e4adf-bd04-4058-a926-8ea8ca08c05a;Secure;HttpOnly;Path=/api
Content-Length: 0
Server: Jetty(9.2.11.v20150529)
```
Once the session cookie file has been created, it can be sent along in all subsequent requests to the REST API.
For the curl command-line client, the `-b` flag is used to send the contents of the cookie file to the server along with the request.
The following command sends a GET request to the Fusion REST API Collections service to check the status of the `system_logs` collection.
The `-b` flag sends in a freshly generated session cookie.
```bash theme={"dark"}
> curl -b cookie -i http://localhost:8764/api/collections/system_logs
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Content-Encoding: gzip
Vary: Accept-Encoding, User-Agent
Content-Length: 278
Server: Jetty(9.2.11.v20150529)
{
"id" : "system_logs",
"createdAt" : "2016-03-04T23:29:47.779Z",
"searchClusterId" : "default",
"commitWithin" : 10000,
"solrParams" : {
"name" : "system_logs",
"numShards" : 1,
"replicationFactor" : 1
},
"type" : "METRICS",
"metadata" : { }
}
```
If the session cookie has expired, the system returns a 401 Unauthorized code:
```bash wrap theme={"dark"}
> curl -b cookie -i http://localhost:8764/api/collections/system_logs
HTTP/1.1 401 Unauthorized
Content-Type: application/json; charset=utf-8
Content-Length: 31
Server: Jetty(9.2.11.v20150529)
{"code":"session-idle-timeout"}
```
## User Account Administration
A Fusion [Security Realm](/docs/5/fusion/operations/security/access-control/security-realms) encapsulates a user database together with specific authentication and authorization mechanisms.
This information is stored in [ZooKeeper](http://zookeeper.apache.org/) so that is it always available to all Fusion components across the deployment.
Fusion’s native security realm manages both authentication and authorization directly.
All user information is stored in ZooKeeper: usernames, hashes of passwords, roles, and permissions.
Passwords are hashed using [bcrypt](https://en.wikipedia.org/wiki/Bcrypt).
Authentication compares a hash of the entered login password with the stored password hash.
The native realm is the home of the Fusion admin user and is the default realm type.
Fusion can be configured to use the host domain’s security mechanism for user administration.
The following configurations are possible:
* **LDAP.** Fusion stores a local user record in ZooKeeper.\
Authentication is performed by the LDAP server.\
LDAP group membership can be used to assign Fusion permissions.
* **Kerberos.** Fusion stores a local user record in ZooKeeper.
[SPNEGO](https://en.wikipedia.org/wiki/SPNEGO) is used for authentication via Kerberos.
* **Kerberos authentication and LDAP authorization.** Fusion stores a local user record in ZooKeeper.
[SPNEGO](https://en.wikipedia.org/wiki/SPNEGO) is used for authentication via Kerberos\
LDAP group membership can be used to assign Fusion permissions.
* **SAML.** Fusion stores a local user record in ZooKeeper. The SAML 2.0 protocol is used to provide web-browser single sign-on.
* **JWT.** JSON Web token.
## Video tutorial
## More information
These additional topics explain how to configure the supported authentication methods:
* [Configure Fusion for JWT](/docs/5/fusion/reference/config-ref/configure-fusion-for-jwt-5x)
* Configure Fusion for Kerberos in **Configure Fusion for Kerberos in Unix** and **Configure Fusion for Kerberos in Windows**
* **Configure Fusion 5.x for LDAP**
* **Configure Fusion for SAML**
* **Configure Fusion for SSO**
To configure the Fusion UI service to use Kerberos for user authentication, you must create a Kerberos [security realm](/docs/5/fusion/operations/security/access-control/security-realms).
Kerberos support requires Fusion 5.9.5.
Kerberos is a system that provides authenticated access for users and services on a network.
Instead of sending passwords in plaintext over the network, encrypted passwords are used to generate time-sensitive tickets that are used for authentication. SPNEGO provides a mechanism for extending Kerberos to Web applications through the standard HTTP protocol.
Kerberos uses symmetric-key cryptography and a trusted third party called a Key Distribution Center (KDC) to authenticate users to a suite of network services. (By users we mean both end users and client programs).
The computers managed by that KDC and any secondary KDCs constitute a realm. When a user authenticates to the KDC, the KDC sends a set of credentials (a ticket) specific to that session back to the user’s machine.
Kerberos-aware services use the ticket on the user’s machine for authentication instead of requiring sign-on with a password. Because tickets are used rather than passwords, this provides the convenience of Single Sign-On (SSO) in addition to security.
A Kerberized process is one that has been configured so it can get tickets from a KDC and negotiate with Kerberos-aware services. When a user sends an HTTP request, Fusion tries to authenticate using the Kerberos/SPNEGO protocol.
If the request was sent from a browser, Fusion does not display the initial sign-on panel; instead on login, the user sees the main Fusion collections panel.
This article focuses on configuring Fusion for Kerberos *in Unix*.
To Kerberize Fusion, you must:
* Configure the Kerberos client on the server that the Fusion UI service will be running on so that it can talk to the KDC (section [Configuring the Kerberos client](#configuring-the-kerberos-client) below).
* Configure the security realm of the Fusion UI (section [Configuring Fusion Authentication for Kerberos Realm](#configuring-fusion-authentication-for-kerberos-realm) below).
* Depending on the encryption used by the KDC, you may also need to install additional Java security libraries into the Fusion distribution.
These are freely available from Oracle, see download and installation instructions below.
To do this, you need following information, which you can get from your sysadmin:
* **Kerberos realm name.** In most cases, this is your domain name in upper case.
* **KDC name.** This is usually a combination of `kerberos.` and your domain name. For example, `kerberos.example.com`
* **Kerberos principal name and password.** A principal is a unique identity to which Kerberos can assign tickets. When the entity is a client program, this is called the Service Principal name.
* **A keytab file.** This holds the encrypted credentials.
The usual scenario in an enterprise organization is to have a Kerberos admin create a service principal with a random key password.
Then, the admin generates a keytab, which is then used for Fusion service principal authentication.
The Kerberos commands needed for configuration and testing are:
* `kinit`. Obtain and cache a ticket from the KDC (i.e., domain login)
* `kdestroy`. Destroys credentials (i.e., domain logout)
* `klist`. Lists cached credentials
* `ktutil`. Create or add credentials to a keytab file
If your browser is not already configured to use the Kerberos/SPNEGO, you need to do so in order to test the Fusion configuration.
## Configuring the Kerberos client
### Step 1: Edit the Kerberos configuration file.
To configure your local Kerberos client so that it can talk to the Kerberized server,
you must edit Kerberos configuration file named `krb5.conf`.
On most Unix systems, this file is located at `/etc/krb5.conf`.
This file contains Kerberos configuration information, including the locations of KDCs and admin servers for the Kerberos realms of interest, defaults for the current realm and for Kerberos applications, and mappings of hostnames onto Kerberos realms.
If your organization realm name is "MYORG.ORG", and your KDC server is named "kerberos.myorg.org", then you edit two entries.
The first entry is `libdefaults`. Set MYORG.ORG as the default realm:
```
[libdefaults]
default_realm = MYORG.ORG
```
The second entry is `realms`. Add MYORG.ORG as a realm:
```
[realms]
MYORG.ORG = {
kdc = kerberos.myrealm.com
}
```
For example, for realm LUCIDWORKS.IO, the krb5.conf file is just like the above example, except that instead of "myorg.org" we specify "lucidworks.io".
### Step 2: Authenticate to Kerberos
The command `kinit` is the Kerberos authentication command.
To get started, you authenticate to Kerberos using the Kerberos principal name and password (which you may need to obtain from your sysadmin).
For this example, the principal name is "prince".
```
> kinit prince
```
The `kinit` command prompts for a password. Successful authentication is silent. Unsuccessful authentication results in an error message.
The command `klist` shows all cached Kerberos credentials. To check that you have successfully authenticated, run this command:
```
> klist
```
Output should be in this form, but with your data:
```
Credentials cache: API:C980F9F5-415C-4A3E-9C67-883C7D5FFFBE
Principal: prince@MYORG.ORG
Issued Expires Principal
May 6 15:14:55 2015 May 7 01:12:47 2015 krbtgt/MYORG.ORG@MYORG.ORG
```
## The Service Principal Keytab file
The usual scenario in an enterprise organization is to have a Kerberos admin create a service principal with a random key password.
Then, the admin generates a keytab, which is then used for Fusion service principal authentication.
If you are your own Kerberos admin, then you will need to create this file for yourself.
### Step 3. Create a Keytab file
The command `ktutil` creates the service principal keytab file
which holds the encrypted credentials that the Fusion UI Proxy will use for Kerberos authentication.
In order to generate the keytab file, you must have a set of cached credentials, therefore, first run the `kinit` command (step 2).
From the command line, run the command `ktutil`.
You must enter your password twice.
```
> ktutil -k http-myrealm.org.keytab add -p HTTP/myrealm.org@MYORG.ORG -e aes256-cts-hmac-sha1-96 -V 0
```
The -k argument specifies the name of the keytab file which will be created or updated. The command "add" takes the following argument flags:
* "-p" : service principal, format \/\@REALM
* "-e" : [encryption type](http://web.mit.edu/kerberos/www/krb5-latest/doc/admin/conf_files/kdc_conf.html#encryption-types). Depending on the encryption type, you may need to download additional Java security libraries for strong encryption.
The following encryption types are not supported by Fusion: \* DES-CBC-CRC \* DES\_CBC\_MD5 \* Microsoft Windows 2000 RC4-HMAC
* "-V" : key version number (kvno). Key version numbers are used in the Kerberos V5 protocol to distinguish between different keys in the same domain.
If successful, this command creates a keytab file called "http-myrealm.org.keytab".
Note the directory you are in - you will need this full path when creating the Proxy realm-config later.
### Step 4. Test the Keytab file
The location of this keytab file will be used to configure UI Proxy configuration.
Before configuring the Fusion UI Proxy, you should check that the keytab file is valid.
Testing the keytab requires the following sequence of steps:
* Clear any existing credentials via command `kdestroy`.
* Log in using the keytab as an argument to the command `kinit -kt `, where `` is the name of a principal within the keytab file.
* Examine your credentials via command `klist`.
* Clear credentials via command `kdestroy`, which removes any existing credentials, effectively logging you out of Kerberos.
To remove cached credentials, use the `kdestroy` command. This command succeeds silently.
To check that credentials have been removed, re-run the `klist` command:
```
> kdestroy
> klist
klist: krb5_cc_get_principal: No credentials cache file found
```
Use the keytab to login as the service principal, without being prompted for a password:
```
> kinit -t http-myrealm.org.keytab HTTP/myrealm.org
```
Examine your credentials via the command `klist`. The output should be similar to this:
```
Credentials cache: API:51D488FF-5CD9-4E16-98FA-B47743F5B4ED
Principal: HTTP/myrealm.org@MYORG.ORG
Issued Expires Principal
Apr 1 09:15:02 2015 Apr 1 19:13:42 2015 krbtgt/MYORG.ORG@MYORG.ORG
```
Logout again with `kdestroy`.
## Configuring Fusion Authentication for Kerberos Realm
Once you have tested both the user and service principal logins, you must create the service principal realm-config in the Fusion Authentication Proxy. This allows the Proxy to authenticate to Kerberos as the service principal, without a password.
### Step 5. Configure the Fusion Realm
Fusion security realms can be configured either via the Fusion UI Admin tool or the Fusion REST API. The advantage of using the Fusion UI Admin Tool is that a single panel’s worth of configuration requires a series of calls to the REST API. It is important to understand the set of configuration properties collected by the Fusion UI and how they are used by the REST API.
A security realm for Kerberos has the following properties:
* name : unique string identifier
* realmType : "kerberos"
* enabled : whether or not the realm is available for users to use with system authentication
* config : this property is required for the realm type "kerberos". It takes two key-value pairs:
* principal : the principal service name
* keytab : this must be the full path to the keytab file.
To configure Fusion via the Fusion UI:
1. Log in to Fusion as "admin" or as a user who has super-admin privileges.
2. In the left-hand side navigation bar, click **System > Access control**.
3. Click the **Security Realms** tab, then **Add Security Realm**.
4. On the New Security Realm form, choose type "kerberos" from the pulldown menu so that there are input boxes for the Kerberos realm properties "Service Principal" and "Keytab path".
5. Choose a realm name, and make sure the "enabled" box is checked.
6. If the **auto-created** box is unchecked, a user with admin privileges will need to create Fusion user accounts.
7. Select default roles. The default roles are assigned to a user the first time they access Fusion via this realm, using the Kerberos/SPNEGO protocol. For example, once you have defined a Kerberos realm "my-kerberos-realm" for domain "MYORG.ORG", when user "any.user" in domain "MYORG.ORG" authenticates to Fusion for the first time via this realm, they are added to the set of Fusion users as username "[any.user@MYORG.ORG](mailto:any.user@MYORG.ORG)" and they have all default roles.
It is prudent to allow the minimum set of default roles, as all users will have these permissions. Some users will require admin privileges and a few users will require super-admin privileges. There should always be a user with super-admin privileges that can authenticate to Fusion using the native security realm and can then grant permissions to individual users as needed.
The Fusion proxy Kerberos realm works only with one Fusion host. For a multi-node Fusion cluster where each node resides on a different host, configure a separate realm for each host with a dedicated service/principal.
8. Enter the service principal and the full path to the keytab file.
9. Click **Save**. Fusion displays a confirmation message.
## Kerberos/SPNEGO HTTP Authentication
[SPNEGO](http://en.wikipedia.org/wiki/SPNEGO) provides a mechanism for extending Kerberos to applications that use the HTTP protocol including web browsers and the `curl` command-line utility.
### Step 6. Configure the HTTP client
When a user sends a request to the Kerberized Fusion UI, a SPNEGO request (http\[s]) is made.
If the user is not already authenticated, the Fusion authentication proxy will yield a 401 status code and a Negotiate header.
This status/header response triggers compatible clients to fetch a local ticket from their Kerberos "ticket tray". This ticket is then encoded and sent back to the Fusion. The Fusion authentication proxy will then decode the ticket, and perform a SPN.doAs(user) authentication request to the KDC/Authentication Service. Depending on the results, the proxy then successfully executes the original request (along with a session cookie) or a 401 (without the Negotiate).
Clients can either choose to use the session cookie or continue authenticating on every request.
### Configuring Web Browsers and `curl` for SPNEGO
The `--negotiate` option enables SPNEGO in `curl`.
IE and Safari require no additional configuration to use SPNEGO.
To configure Firefox, access the low level configuration page by loading the about:config page. Then go to the network.negotiate-auth.trusted-uris preference and add the hostname or the domain of the web server that is HTTP Kerberos SPNEGO protected (if using multiple domains and hostname use comma to separate them).
The Chrome browser must be launched from the command line with several added parameters.
To run Chrome on linux:
```
> google-chrome --enable-plugins --args\
--auth-server-whitelist="*KERBEROS_DOMAIN"\
--auth-negotiate-delegate-whitelist="*KERBEROS_DOMAIN"\
--auth-schemes="basic,digest,ntlm,negotiate"
```
To run Chrome on a Mac:
```
> open 'Google Chrome.app' --args\
--auth-server-whitelist="*ROGUECLOUD.COM"\
--auth-negotiate-delegate-whitelist="*KERBEROS_DOMAIN"\
--auth-schemes="basic,digest,ntlm,negotiate"
```
To run Chrome on Windows:
```
chrome.exe --auth-server-whitelist="*KERBEROS_DOMAIN"\
--auth-negotiate-delegate-whitelist="*ROGUECLOUD.COM"\
--auth-schemes="basic,digest,ntlm,negotiate"
```
For more information, see [Using a Web Browser to Access an URL Protected by Kerberos HTTP SPNEGO](https://docs-archive.cloudera.com/documentation/enterprise/6/6.1/topics/cdh_sg_browser_access_kerberos_protected_url.html) and [Authentication + Authorization](http://www.roguelynn.com/words/apache-kerberos-for-django/).
### Session cookies
A successful Kerberos/SPNEGO login will yield a session cookie, this cookie is identical to the cookie yielded by the Fusion authentication proxy’s current POST-login-mechanism.
The expiration policy on the cookie is currently fixed at 8 hours. But has a 1 hour "idle" max, which means if you do not make a request for 1 hour, the cookie is invalidated. Otherwise the lifetime is pushed ahead until the 8 hour max is met.
The name of the cookie is "id" and the value is a UUID. This UUID is a key that maps to an in-memory value containing the real user ID.
## Testing and Troubleshooting
Once you have configured the Kerberos security realm, you can test it as follows:
1. Log out of Fusion and shut down the browser.
2. Check that you have a valid Kerberos authentication ticket via the `klist` command.
3. Open a new browser session and access the Fusion installation.
This should take you directly to the main Fusion panel, bypassing the Fusion "Welcome" login panel. To view your login profile, click on the profile icon at the top right. Your user name should be your login name + "@" + your domain name.
* If you see the Fusion login panel instead of the main Fusion panel, your browser is not configured for SPNEGO.
* If the Fusion display consists only of an empty top nav bar, this indicates an authentication failure. Check that the path to your keytab file is correct. Then check the Fusion logs.
* If your KDC uses a weak encryption method, Fusion rejects your login credentials. The following weak encryption types are disabled in Fusion:
* DES-CBC-CRC
* DES\_CBC\_MD5
* Microsoft Windows 2000 RC4-HMAC
## References and Tutorials
* [https://en.wikipedia.org/wiki/Kerberos\_%28protocol%29](https://en.wikipedia.org/wiki/Kerberos_%28protocol%29)
* [http://www.roguelynn.com/words/explain-like-im-5-kerberos/](http://www.roguelynn.com/words/explain-like-im-5-kerberos/)
* [https://en.wikipedia.org/wiki/SPNEGO](https://en.wikipedia.org/wiki/SPNEGO)
* [https://access.redhat.com/documentation/en-US/Red\_Hat\_Enterprise\_Linux/6/html/Managing\_Smart\_Cards/Using\_Kerberos.html#about-kerberos](https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/Using_Kerberos.html#about-kerberos)
* [http://www.oracle.com/technetwork/articles/idm/weblogic-sso-kerberos-1619890.html](http://www.oracle.com/technetwork/articles/idm/weblogic-sso-kerberos-1619890.html) - section "Install Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files"
* [http://www.cisco.com/c/en/us/support/docs/security-vpn/kerberos/16087-1.html](http://www.cisco.com/c/en/us/support/docs/security-vpn/kerberos/16087-1.html)
To configure the Fusion UI service to use Kerberos for user authentication, you must create a Kerberos [security realm](/docs/5/fusion/operations/security/access-control/security-realms).
Kerberos support requires Fusion 5.9.5.
Kerberos is a system that provides authenticated access for users and services on a network.
Instead of sending passwords in plaintext over the network, encrypted passwords are used to generate time-sensitive tickets that are used for authentication. SPNEGO provides a mechanism for extending Kerberos to Web applications through the standard HTTP protocol.
Kerberos uses symmetric-key cryptography and a trusted third party called a Key Distribution Center (KDC) to authenticate users to a suite of network services. (By users we mean both end users and client programs).
The computers managed by that KDC and any secondary KDCs constitute a realm. When a user authenticates to the KDC, the KDC sends a set of credentials (a ticket) specific to that session back to the user’s machine.
Kerberos-aware services use the ticket on the user’s machine for authentication instead of requiring sign-on with a password. Because tickets are used rather than passwords, this provides the convenience of Single Sign-On (SSO) in addition to security.
A Kerberized process is one that has been configured so it can get tickets from a KDC and negotiate with Kerberos-aware services. When a user sends an HTTP request, Fusion tries to authenticate using the Kerberos/SPNEGO protocol.
If the request was sent from a browser, Fusion does not display the initial sign-on panel; instead on login, the user sees the main Fusion collections panel.
This article focuses on configuring Fusion for Kerberos *in Windows*.Kerberos support requires Fusion 5.9.5.
## Set up and Configure the Active Directory Domain Service
### Install the Active Directory Domain Service
To begin, follow [Microsoft’s instructions](https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/deploy/install-active-directory-domain-services--level-100-) on how to install Active Directory Domain Services.
This topic uses the following domain throughout this example:
| | |
| - | - |
| **NETBIOS Domain** | FUSIONIS |
| **Fully qualified domain name** | fusionis.life |
### Create a New User
Launch the Server Manager. Navigate to **Tools > Active Directory Users and Computers**. Within the `fusionis.life` server, locate the "Users" folder, right-click on the folder and select **New > User**.
When creating a user for this purpose, ensure the `First name` and `Display name` fields both hold the exact same value.
These instructions use the following user name throughout this example:
| User name | Notes |
| - | - |
| **Distinguished name (DN)** |
CN = kerberos500
CN = Users
DC = fusionis
DC = life
|
| **User Principal Name (UPN)** | `kerberos500@fusionis.life` |
| **NETBIOS Domain\samAccountName** | FUSIONIS\kerberos500 |
Establish the desired delegation rules for the new user in the "Delegation" tab.
### Create a Keytab for the New User
Enter the following command into the command prompt to create a keytab for the user you just created:
```
ktpass -out c:\Users\Administrator\kerberos500.keytab -princ HTTP/fusionis.life@FUSIONIS.LIFE -mapUser FUSIONIS\kerberos500 -mapOp set -pass YOUR_PASSWORD -crypto AES256-SHA1 -pType KRB5_NT_PRINCIPAL -kvno 0
```
The following weak encryption types are not supported by Fusion:
* DES-CBC-CRC \* DES\_CBC\_MD5
* Microsoft Windows 2000 RC4-HMAC
The output will be similar to:
```
C:\Users\Administrator>ktpass -out c:\Users\Administrator\kerberos500.keytab -princ HTTP/fusionis.life@FUSIONIS.LIFE -mapUser kerberos500 -mapOp set -pass FroFro123# -crypto AES256-SHA1 -pType KRB5_NT_PRINCIPAL
Targeting domain controller: WIN-OVV6VHBGIB8.fusionis.life
Using legacy password setting method
Successfully mapped HTTP/fusionis.life to kerberos500.
Key created.
Key created.
Key created.
Key created.
Key created.
Output keytab to c:\Users\Administrator\kerberos500.keytab:
Keytab version: 0x502
keysize 71 HTTP/fusionis.life@FUSIONIS.LIFE ptype 1 (KRB5_NT_PRINCIPAL) vno 3 etype 0x1 (DES-CBC-CRC) keylength 8 (0xcd07200bea625d20)
keysize 71 HTTP/fusionis.life@FUSIONIS.LIFE ptype 1 (KRB5_NT_PRINCIPAL) vno 3 etype 0x3 (DES-CBC-MD5) keylength 8 (0xcd07200bea625d20)
keysize 79 HTTP/fusionis.life@FUSIONIS.LIFE ptype 1 (KRB5_NT_PRINCIPAL) vno 3 etype 0x17 (RC4-HMAC) keylength 16 (0xe39a141de38abd8750bf9c0bf49fd1c5)
keysize 95 HTTP/fusionis.life@FUSIONIS.LIFE ptype 1 (KRB5_NT_PRINCIPAL) vno 3 etype 0x12 (AES256-SHA1) keylength 32 (0xdd6018ab718a4c312038189a20498f70c07d09a19d77ceea31cdd4cd1b08800a)
keysize 79 HTTP/fusionis.life@FUSIONIS.LIFE ptype 1 (KRB5_NT_PRINCIPAL) vno 3 etype 0x11 (AES128-SHA1) keylength 16 (0x9b610485eae2d97dcf0ecaf4ab38a32b)
```
This process will create the `kerberos500.keytab` file required to set up the Kerberos realm in Fusion. The command will also set the service principal to `HTTP/fusionis.life@FUSIONIS.LIFE`.
If you see the following, the user was not created correctly: `DsCrackNames returned 0x2 in the name entry for kerberos500. Ktpass:failed getting target domain for specified user.` Delete the user, and repeat [the steps above](#create-a-new-user).
## Set up Kerberos on Fusion Server
### Install Fusion and set up the Kerberos Realm
If you have not done so already, install Fusion now.
Copy the keytab file to the server.
The keytab file must be installed on each Fusion node.
Log into Fusion as a native admin and navigate to **DevOps > Access Control > Realms > New Realm**. Use the following parameters:
| | |
| - | - |
| **Type** | Kerberos |
| **Keytab file** | The path to the keytab file you saved on the server |
| **Service principal** | The service principal you generated in the previous step |
| **Kerberos name mapping** | The field that specifies the mapping from a Kerberos Principal name to a short name |
You can specify the default rule, which should work for most situations:
```
DEFAULT
RULE:[1:$1@$0](.*@.*)s/@.*//
```
You can also add custom rules. See [Mapping Kerberos Principals to Short Names](https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cdh_sg_kerbprin_to_sn.html) for more information on creating custom rules.
Save your new realm before continuing. You should now be able test with Negotiate authentication.
## Verify Kerberos is Working
You will test to verify that Kerberos is working in the following sections. You must log into Windows as a user in the protected realm, meaning that the service principal name you are using has access to generate a ticket for your user.
### Verify with cURL
Windows users can log in as a user within the domain, launch the command prompt, and run the cURL command below:
```
curl --negotiate -u : http://fusionis.life:8764/api/apollo/collections
```
Linux users may also need to run a `kinit` to login as a user prior to running.
Navigate your web browser to `http://fusionis.life:8764`. If you are automatically logged in, Kerberos is working.
### Verify with Google Chrome
To verify Kerberos is working with Google Chrome, run Chrome with the following command parameters:
```
"c:\Program Files (x86)\Google\Chrome\Application\chrome.exe" --auth-server-whitelist="*fusionis.life" --auth-negotiate-delegate-whitelist="*fusionis.life" --auth-schemes="basic,digest,ntlm,negotiate"
```
Navigate your web browser to `http://fusionis.life:8764`. If you are automatically logged in, Kerberos is working.
### Verify with Firefox
To verify Kerberos is working with Firefox, navigate to `about:config` in your browser’s URL bar. If a message appears stating "This might void your warranty!", click "I accept the risk!".
Search for the string `network.negotiate-auth.trusted-uris`, double-click the field, and enter `fusionis.life` into the field that appears. Click "Ok".
Navigate your web browser to `http://fusionis.life:8764`. If you are automatically logged in, Kerberos is working.
### Troubleshooting
#### Defective token detected
If you encounter this issue, retrieve logs from the `api-gateway` pod after attempting to make a request: `kubectl logs -n NAMESPACE -l app=api-gateway`
Some deployments use `proxy` instead; if `api-gateway` returns no results, try `-l app=proxy`.
See [Stack Overflow](https://stackoverflow.com/questions/47227276/gssexception-defective-token-detected-when-trying-to-authenticate-to-tomcat) for additional steps to resolve this issue.
#### Kerberos Name mapping rules
You may see the following error if you did not specify a mapping rule and the Proxy could not figure out how to map the Kerberos Principal name to a short name:
```
2021-07-30T15:15:03,584 - ERROR [qtp353842779-52:clojure.tools.logging$eval11$fn__16@0] - {} - Uncaught application error!
org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to YOURUSERNAME@YOURDOMAIN.SUBDOMAIN.EXAMPLE.COM
```
Adjust your Kerberos name mapping rules to resolve this issue.
You can create security realms that use external LDAP servers for authentication. Optionally, Fusion can search in the LDAP for groups to which a user belongs, and then map those groups to Fusion roles. Fusion performs authorization using permissions stored in Fusion users and Fusion roles.
Fusion *does not* use permissions from the LDAP for authorization of UI access or API requests. It only obtains group names (optionally), which are mapped to role names. If an [Active Directory Security Query Trimming Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/active-directory-security-trimming-query-stage) is used, then directory-service permissions *are* used for trimming. If a connector supports security trimming, then connector permissions *are* used for trimming.
To configure Fusion to use an external LDAP as an authentication provider, you’ll need to get information about the LDAP server(s) running on your system, either from your system or your sysadmin.
## Add an LDAP Security Realm
1. Log in to the Fusion UI as the user admin, or as a different user with corresponding permissions.
2. Navigate to **System** > **Access Control**.
3. Click **Security Realms**.
4. Click **Add Security Realm**.
5. Specify info for the new realm:
* `name` – Name of the security realm. It must be unique. It should be descriptive but short.
* `type` – Select **ldap** from the pulldown menu.
When you select **ldap**, Fusion displays additional, LDAP-specific configuration options.
* `enabled` checkbox – Whether Fusion allows user logins for this security realm. The default is yes (checked).
* `auto-create users` checkbox – Whether a user account is created automatically upon initial authentication. The default is yes (checked). If the checkbox is unchecked, then a Fusion user with admin permissions must create Fusion users.
6. Scroll down and specify additional options as explained in detail below.
## Specify Static Roles (Optional)
Specify one or more Fusion roles for the security realm. These roles are always considered. They do not depend on searching for LDAP groups and mapping group names to Fusion role names.
In a security realm, you can specify these static roles, add to the list of roles dynamically through an LDAP search, or both. If you do neither, Fusion uses only the role(s) and permissions defined for the user.
## Specify LDAP Connection Details
Specify the hostname and port of the LDAP server. Check the checkbox if the server is running over SSL.
## Specify the Authentication Method
Specify the authentication method:
* **Bind.** LDAP authentication is carried out via a single "Bind" operation. See [Bind](#bind) below.
* **Search.** LDAP authentication is carried out indirectly via a Search operation followed by a Bind operation. See [Search](#search) below.
* **Kerberos.** Kerberos authenticates Fusion and an LDAP Search operation is carried out to find group-level authorizations. See [Kerberos](#kerberos) below.
### Bind
Use the Bind authentication method when the Fusion login username matches a part of the LDAP distinguished name (DN). Specify the remainder of the LDAP DN in the "DN Template" configuration entry, which uses a single pair of curly braces (`{}`) as a placeholder for the value of the Fusion username.
### Search
Use the Search authentication method when the username used for Fusion login *doesn’t* match a part of the LDAP DN. The search request returns a valid user DN, which is then used together with the user password for authentication via a Bind request.
1. Construct a search request.
The Search authentication method is generally required when working with Microsoft Active Directory servers. In this case, you need to know the username and password of *some* user who has sufficient privileges to query the LDAP server for user and group memberships; this user doesn’t have to be the superuser.
In addition to a privileged user DN and password, the Search authentication method requires constructing a search request. There are two parts to the request. The first part is the base DN of the LDAP directory tree that contains user account objects. The second part of the request is a Search Filter object that restricts the results to a matching subset of the information.
2. Provide the administrator bind DN:
### Kerberos
Use the Kerberos authentication method when Kerberos is the authentication provider.
## Search for LDAP Groups (Optional)
A Fusion role is a bundle of permissions tailored to the access needs of different kinds of users. Access to services and data for LDAP-managed users is controlled by mappings from LDAP users and groups to Fusion roles.
Roles can be assigned globally or restricted to specific LDAP groups. The security realm configuration panel contains a list of all Fusion roles with a checkbox for each, used to assign that role to all users in that realm. LDAP group names can be mapped directly to specific Fusion roles and LDAP group search and filter queries can also be used to map kinds of LDAP users to specific Fusion roles.
## Map LDAP Groups to Fusion Roles (Optional)
If LDAP group names returned by the search for groups match Fusion role names, you do not need to map the group names to role names. You must map any LDAP group names that do not match to Fusion role names (if you do not, they will not be used).
## Save the Security Realm Configuration
Click **Save**.
Fusion reports whether or not authentication was successful:
## Basic LDAP Concepts and Terminology
The [LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) protocol is used to share information about users, systems, networks, and services between servers on the internet. LDAP servers are used as a central store for usernames, passwords, and user and group permissions. Applications and services use the LDAP protocol to send user login and password information to the LDAP server. The server performs name lookup and password validation. LDAP servers also store Access Control Lists (ACLs) for file and directory objects which specify the users and groups and kinds of access allowed for those objects.
LDAP is an open standard protocol and there are many commercial and open source LDAP servers available. Microsoft environments generally use Active Directory. Unix servers use AD or other LDAP systems such as OpenLDAP, although many Unix systems do not use LDAP at all. To configure Fusion for LDAP, you’ll need to get information about the LDAP server(s) running on your system either from your sysadmin or via system utilities.
### Directories and Distinguished Names
An LDAP information store is a Directory Information Tree (DIT). The tree is composed of entry nodes; each node has a single parent and zero or more child nodes. Every node must have at least one attribute which uniquely distinguishes it from its siblings which is used as the node’s Relative Distinguished Name (RDN). A node’s Distinguished Name (DN) is a globally unique identifier.
The string representation of a DN is specified in [RFC 4514](https://tools.ietf.org/html/rfc4514). It consists of the node’s RDN followed by a comma, followed by the parent node’s DN. The string representation of the RDN is the attribute-value pair name, connected by an equals ("=") sign. This recursive definition means that the DN of a node is composed by working from the node back through its parent and ancestor nodes up to the root node.
Here is a small example of a DIT:
The person entry in this tree has the DN: "uid=babs, ou=people, dc=example, dc=com".
Attribute names include many short strings based on English words and abbreviations, e.g.:
| Name | Description |
| - | - |
| cn | commonName |
| dc | domainComponent |
| mail | email address |
| ou | organizationalUnitName |
| sn | surname |
| uid | userId |
LDAP entry attributes can refer to other LDAP entries by using the DN of the entry as value of that attribute. The following example of a directory which contains user and groups information shows how this works:
This tree contains two organizational units: "ou=people" and "ou=groups". The children of the "group" organizational unit are specific named groups, just as the child nodes of organization unit "people" are specific users. There are three user entries with RDNs "uid=bob", "uid=alice", "uid=bill" and two groups with RDNs "cn=user" and "cn=admin". The dotted lines and group labels around the person nodes indicate group membership. This relationship is declared on the groups nodes by adding an attribute named "member" whose value is a users DN. In the [LDAP data interchange format (LDIF)](https://en.wikipedia.org/wiki/LDAP_Data_Interchange_Format), this is written:
```
cn=user,ou=groups,dc=acme,dc=org
member: uid=bob,ou=people,dc=acme,dc=org
member: uid=alice,ou=people,dc=acme,dc=org
cn=admin,ou=groups,dc=acme,dc=org
member: uid=bill,ou=people,dc=acme,dc=org
```
See the [Wikipedia’s LDAP entry](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol#Directory_structure) for details.
### LDAP Protocol Operations
For authentication purposes, Fusion sends Bind operation requests to the LDAP server. The Bind operation authenticates clients (and the users or applications behind them) to the directory server, establishes authorization identity used for subsequent operations on that connection, and specifies the LDAP protocol version that the client will use.
Depending on the way that the host system uses LDAP to store login information about users and groups, it may be necessary to send Search operation requests to the LDAP server as well. The Search operation retrieves partial or complete copies of entries matching a given set of criteria.
[LDAP filters](https://www.ldap.com/ldap-filters) specify which entries should be returned. These are specified using prefix notation. Boolean operators are "&" for logical AND, "|" for logical OR, e.g., "A AND B" is written "(&(A)(B))". To tune and test search filters for a Unix-based LDAP system, see the [ldapsearch command line utility](http://docs.oracle.com/cd/E19450-01/820-6169/ldapsearch-examples.html) documentation. For Active Directory systems, see [AD Syntax Filters](http://social.technet.microsoft.com/wiki/contents/articles/5392.active-directory-ldap-syntax-filters.aspx).
[SAML 2.0](https://en.wikipedia.org/wiki/SAML_2.0) is a standard for exchanging authentication and authorization data between security domains. The SAML protocol allows web-browser single sign-on (SSO) through a sequence of messages sent to and from the browser, which is the intermediary between Fusion and the SAML authority acting as the Identity Provider (IdP).
To configure Fusion to use SAML 2.0 for user authentication and authorization you must create a SAML [security realm](/docs/5/fusion/operations/security/access-control/security-realms). In addition to configuring the Fusion security realm, you must configure the SAML identity provider to recognize the Fusion application.
Once Fusion is configured for a SAML realm, this realm is added to the list of available realms on the initial Fusion sign-on panel. When the SAML realm is chosen from the list of available realms, the browser then redirects to the IdP which handles user authentication. Upon successful authentication, the IdP sends a response back to the browser which contains authentication and authorization information as well as the URL of the Fusion application. The browser redirects back to the Fusion URL, passing along the SAML message with the user authentication and authorization information. Fusion then issues as session cookie which is used for subsequent user access.
## Add a Security Realm
1. In the Fusion workspace, navigate to **System** > **Access Control**.
2. Click **Security Realms**.
3. Click **Add Security Realm**.
4. Enter the following information for the new realm:
* Enter a **Name**. The name must be unique and should be descriptive yet short.
* Select **saml** from the **Type** pulldown menu.
When you select a type, Fusion displays additional configuration options.
* The default value for **Enabled** is true. This setting controls whether or not Fusion allows user logins for this security realm.
- The default value for **Ephemeral Users** is false. When disabled, this setting prevents ephemeral users from being created in ZooKeeper during login. If enabled, this property negates **Auto Create Users**.
- The default value for **Auto Create Users** is true. If enabled, a user account is created automatically upon initial authentication. If disabled, then a Fusion user with admin permissions must create Fusion users.
5) Under **SAML Realm**, configure the following information:
* Enter the **Identity Provider URL**. This URL is used by the SAML authority for single sign-on. For example: `https://www.my-idp.com//sso/saml`
The URL format may differ depending on the SAML Identity Provider.
* Enter the URL of the **IdP Issuer**. For example: `http://www.my-idp.com/exk686w2xi5KTuSXz0h7`.
* **IdP Issuer** must match `` in the SAML payload.
* Optionally, provide the **App Issuer**. This field is required if there is an `audienceRestriction` in the SAML assertion and must match `` in the SAML payload.
* In the **Certificate Fingerprint**, paste the contents of the SAML authority certificate, *without the certificate header and footer*.
* Optional: Enter the **User ID Attribute**. By default, the Fusion username is the same as the login name known to the Identity Provider. When another field or attribute in the user record stored by the IdP should be used as the Fusion username, that attribute name is the value of the User ID Attribute.
* Optional: Provide a **Post Login Redirect URL**. If not set, the Fusion URL is used.
* Optional: Provide a **Logout URL**.
6) Optional: Under **Groups Mapping**, specify the Group Name Attribute and add group mappings.
7) Click **Save**.
## Configure SAML identity provider
To finish setup, register Fusion with your SAML identity provider. The amount of information required to register varies depending on the SAML authority.
All systems require the Fusion URL to redirect to upon successful login. This includes the protocol, server, and port for the Fusion application, and path `api/saml`, such as `https://www.my-fusion-app.com:8764/api/saml`. If the Fusion application is running behind a load-balancer, then this URL is the load-balancer URL plus path `api/saml`.
Note that the load-balancer should be session-sticky in order for the sequence of messages that comprise the SAML protocol to run to completion successfully.
Some authorities may require additional information. In particular, the SAML 2.0 "AudienceRestriction" tag may be part of the SAML message. This tag specifies the domain for which the SAML trust conditions are valid, which is usually the domain in which the Fusion app is running, such as `https://www.my-fusion-app`.
## Example SAML Realm configuration
The Fusion endpoint `api/realm-configs` returns a JSON list of all the configuration objects for all realms. After configuring a SAML realm named `"saml-test"` using the okta.com developer preview tool, the configuration object for this realm is:
```json theme={"dark"}
{
"name":"saml-test",
"realmType":"saml",
"enabled":true,
"config":{
"autoCreateUsers":true,
"idpUrl":"https://dev-417804.oktapreview.com/app/dev417804_1/exk686w2xi5KTuSXz0h7/sso/saml",
"issuer":"http://www.okta.com/exk686w2xi5KTuSXz0h7",
"certificateFingerprint":"MIIDpDCCAoygAwIBAgIGAVQr4A4NMA0GCSqGSIb3DQEBBQUAMIGSMQswCQYDVQQGEwJVUzETMBEG\nA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzENMAsGA1UECgwET2t0YTEU\nMBIGA1UECwwLU1NPUHJvdmlkZXIxEzARBgNVBAMMCmRldi00MTc4MDQxHDAaBgkqhkiG9w0BCQEW\nDWluZm9Ab2t0YS5jb20wHhcNMTYwNDE5MDAxNTI0WhcNMjYwNDE5MDAxNjI0WjCBkjELMAkGA1UE\nBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExFjAUBgNVBAcMDVNhbiBGcmFuY2lzY28xDTALBgNV\nBAoMBE9rdGExFDASBgNVBAsMC1NTT1Byb3ZpZGVyMRMwEQYDVQQDDApkZXYtNDE3ODA0MRwwGgYJ\nKoZIhvcNAQkBFg1pbmZvQG9rdGEuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA\nuNz0trRFPw2d4xXoUvX2oWeZolTVeFtaTnB9SWUyjK4og0WdT7rNdBg10eTvB2ezBwXCf24UGGui\nr1kjkZjiHDqDxKtzQYWpGuzLCjh/4PxKFGDaiUNKcE1Ig5myiEBTvMvv99XtrcI75QdUGDhbMiBr\n2PR5FukWOYepzlBzqY0JSDzX9NYJBKPkz+syK4mj0I6dqtYOU+bcTvjF9sR7jiHtQ+d0Zl8rz1Ca\nyuE3mNUtFJ1IJrY/RArhH1AB6mXbV/de1CXmGhKQbqQAbx9SiKtki9n84gKEwuWdV0jIqcBLGxUQ\ngjbsaIVqed2oX+7F2fh6t0g/I8NPnOWXOTvA+wIDAQABMA0GCSqGSIb3DQEBBQUAA4IBAQCPt+DR\nliIsHO/iVnmLFPGfqrCO/gMv++xnq2wOB19YX7HhT1GIY2YZoVphrQpXH3OO0/8AZJ8ApCmq0E9x\nxUTQwQPBanVqlyLtu1Hr0c6dbAqcd5PtaEe6Ci33nayWPydhOmitvIyb/WtWtbel9HcdUkoGvkAl\ng305jnxkhwGLJm4jkzYe+eaYhd6oG3/JcHqKDGYGf7i2Z+ny0D7vxTeBQ+8PbZfsUKg0PlKyTocb\nydSmDPISsA1xOH5zlw+hzFIdKgD5vW7QZLvpIclNc2hqki/nWl/CHut0TnUuP/V3boREmhDu395n\n/u72pgNANfP7+2DTBb+CBTjGUsAxpKRF",
"userIdAttribute":""
},
"roleNames":["search"],
"id":"52e1c0d2-5e00-4c76-a3d4-57f1381bdb4f",
"createdAt":"2016-04-19T19:49:04Z",
"updatedAt":"2016-04-19T20:06:56Z"
}
```
## References
* [http://docs.oasis-open.org/security/saml/v2.0/saml-core-2.0-os.pdf](http://docs.oasis-open.org/security/saml/v2.0/saml-core-2.0-os.pdf)
* [http://docs.oasis-open.org/security/saml/v2.0/saml-glossary-2.0-os.pdf](http://docs.oasis-open.org/security/saml/v2.0/saml-glossary-2.0-os.pdf)
The "SSO Trusted HTTP" realm type (`trusted-http` in the REST API) is useful in single sign-on (SSO) environments.
If SSO is already set up in your environment, user identities and group information can be sent to Fusion through HTTP headers (REMOTE\_USER, for example). The SSO Trusted HTTP realm type provides the configuration options for integrating this into Fusion’s authentication systems. It also supports allowing access to only a set of known client IPs, and mapping groups to Fusion roles.
Use the [Realms API](/api-reference/realm-management/create-a-realm) to configure this realm type:
```
curl -u USERNAME:PASSWORD -H 'content-type:application/json' -X POST :3000/api/realm-configs -d @./realm-config.json
```
Below is a sample configuration:
```json theme={"dark"}
{"id":"test-id",
"enabled":true,
"name":"sso-test",
"realmType":"trusted-http",
"config":{"identityKey":"REMOTE_USER",
"groups": {"key":"GROUPS",
"delimiter":"|",
"roleMapping": [["a","admin"], ["b","foo"]]},
"allowedIps":["127.0.0.1", "0:0:0:0:0:0:0:1", "localhost"]}}
```
| | |
| - | - |
| `identityKey` | The name of an HTTP header. If this key is found in the request headers, its value is used as the identity of the client (username, for example). |
| `groups` | Configuration keys for auth groups:
`key` The name of an HTTP header, used as the source of group names.
`delimiter` The character used to split the value (defaults to comma).
`roleMapping` A set of 2-tuples, used for mapping the external group values to Fusion Roles.
|
| `allowedIps` | Allow access to only a set of known client IPs. When this property is defined and the client IP is not included in it, the realm logic return a 401. In Fusion 5.0.0+, leaving this field empty makes the realm nonoperational. To accept traffic from all destinations for development, you can use an IP such as 0.0.0.0/0. In production the concrete IP should be specified. Prior to Fusion 5.0.0, the `X-FORWARDED-FOR` header is inspected for this client IP first; the value is split on comma, and the first entry is taken. This would normally be used in cases where the client was forwarded to Fusion through one or more external proxy servers. If the `X-FORWARDED-FOR` header is not present in the request, the `REMOTE-ADDR` header value is used instead. |
## Learn more
After creating a new instance of Fusion, you will need to create a password for the admin user. This is typically done in the Fusion UI. In some cases, such as during the development stage when the Admin UI microservice is not running, you may want to create this password using the API.
Use the following example commands to create the initial admin password with the API, in this case, `password123`:
```json wrap theme={"dark"}
curl -X POST -H 'content-type: application/json' 'http://localhost:6764/api' --data-binary '{"password": "password123"}'
```
```json wrap theme={"dark"}
curl -X POST -H 'content-type: application/json' 'http://localhost:8764/api' --data-binary '{"password": "password123"}'
```
# Permissions
Source: https://doc.lucidworks.com/docs/5/fusion/operations/security/access-control/permissions
[localhost link]: http://localhost:3000/docs/5/fusion/operations/security/access-control/permissions
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/security/access-control/permissions
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/123
Permissions determine what a user can do in Fusion. There are two kinds of permissions:
* **UI permissions** – Control which parts of the Fusion UI a user can access. These parts show up in menus and the user can view them. But the ability to *use* the functionality depends on API permissions.
* **API permissions** – Control which requests a user can submit to which REST API endpoints.
Permissions can be defined by either a role or a user, or both. Fusion combines permissions for authorization as follows:
* UI permissions are positive (permission needs to be given) and additive (the user has the sum of all specified permissions). This is true of roles specified in a user definition, roles specified in a security realm, and roles determined dynamically based on groups in an LDAP authentication provider.
* API permissions specified in roles are positive (permission needs to be given) and additive (the user has the sum of all specified permissions; that is, for a specific endpoint, the most permissive permissions are used). This is true of roles specified in a user definition, roles specified in a security realm, and roles determined dynamically based on groups in an LDAP authentication provider.
* API permissions specified in the role(s) but not in the user definition are used.
* If an API permission for a specific endpoint is specified in both a role *and* in the user definition, then the permissions in the user definition are used, *overriding* the permissions in the role(s). Use permissions in user definitions to give specific users permissions that are less permissive than the permissions for their role(s).\
For example, say role A allows GET and POST access to a specific endpoint. User X is a member of role A and also has a user definition that allows only GET access to that endpoint. In this case, user X has only GET access to that specific endpoint.\
Alternatively, you could define less permissive roles.
## Specify UI Permissions
Specify UI permissions in [roles](/docs/5/fusion/operations/security/access-control/roles).
## Specify API Permissions
A Fusion API permission denotes an allowed request to a Fusion REST API endpoint or endpoints.
A permissions specification consists of:
* HTTP request method or methods allowed. Multiple HTTP methods are separated by commas.
* REST API services endpoint, which can contain wildcards or named variables. All calls to the REST API start with "api/", followed by the service name and any methods and parameters.\
The permissions specification includes everything following `api/`. The endpoint can include wildcards.\
Wildcards make it easy to grant broad access to Fusion services. The wildcard symbol `*` matches all possible values for a single path segment.\
Two wildcards match all possible values for any number of path segments. Granting access to a subset of Fusion functionality can require a list of narrowly defined permissions.\
Carefully defining each Fusion app to address a specific use case can simplify permission sets, by letting you grant access liberally within a single app.\
A path segment can be a named variable enclosed in curly braces: `{variable-name}`.\
Variables are used when a wildcard would be too permissive and a single path segment too restrictive.
* Optionally, the allowed values for any named variables in the endpoint. The variable specification component specifies the restricted value or values for all named variables in the path.\
Each specification consists of the variable name, followed by "=" (the equals sign), followed by one or more values separated by commas.\
If the endpoint specification has multiple variable, the semi-colon character ";" is used as the separator between parameter specifications.
A permission specification has three elements: methods, path, and parameters. A permissions specification is written as a string using the colon character ":" as the separator between the elements.
The lists of permission specifications are stored in the ZooKeeper User and Roles entries in JSON format.
## Example Permission Sets
### Admin permissions
The permissions for the admin user can be written in a single line:
```bash theme={"dark"}
GET,POST,PUT,DELETE,PATCH,HEAD:/**
```
This permission specification uses two wildcards. One wildcard specifies any path segment, and the other specifies any value.
### Best practice: Delegate access to a Fusion app
When possible, create a Fusion app for a specific use case and assign permissions on a per-app basis.
For example, to configure a Fusion account to be able to run queries against only a certain collection:
1. [Create a Fusion app](/docs/5/fusion/intro/ui-tour/fusion-apps).
2. [Create a query profile](/docs/5/fusion/getting-data-out/query-basics/query-pipelines/query-profiles) by defining a collection.\
This lets you change the underlying collection without having to redo permissions.
3. Assign permissions as follows:
```bash theme={"dark"}
GET:/apps/APP_NAME/query/QUERY_PROFILE/*
```
The above permission set allows a user to run queries against the collection specified by `QUERY_PROFILE` under the Fusion app `APP_NAME`.
# Roles
Source: https://doc.lucidworks.com/docs/5/fusion/operations/security/access-control/roles
[localhost link]: http://localhost:3000/docs/5/fusion/operations/security/access-control/roles
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/security/access-control/roles
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/126
Roles are named sets of [permissions](/docs/5/fusion/operations/security/access-control/permissions) that encapsulate the permissions needed for different kinds of users. Permissions grant users access to subsets of Fusion functionality. A role can specify UI permissions, API permissions, or both:
* UI permissions grant users access to parts of the Fusion UI
* API permissions grant users access to specific API commands for specific REST API endpoints.
See [Permissions](/docs/5/fusion/operations/security/access-control/permissions) for information about how permissions supplied by multiple roles and by user definitions combine.
Users who can create or modify code obtain access to the broader Fusion environment. This access can be used to create intentional or unintentional damage to Fusion.
## Where You Specify Roles
You can specify which roles to apply for a user in one or more of these places:
* **Security realm (directly)** – Under the heading **Roles**, specify the roles to always apply to all users in the security realm.
* **Security realm (from a group/role mapping)** – Security realms of types `ldap` and `trusted-http` can provide a list of groups to which the user belongs. The security realm can map the group names to role names.
* **User definition** – A user definition can specify roles for the user. These roles do not override the other roles. They are added to the other roles.
## Default Roles
At initial startup, Fusion creates a set of default roles for common types of users.
### admin
The admin role is the equivalent to the Unix `root` or superuser.
It allows full access to all Fusion services:
```bash theme={"dark"}
GET,POST,PUT,DELETE,PATCH,HEAD:/**
```
### developer
The developer role has all the read/write permissions required for building and running applications.
The developer role is updated for Fusion 5.10 to exclude permissions for creating, editing, and deleting Spark jobs, JavaScript stages in querying and indexing pipelines, and custom index and query stages. The new roles including these permissions are created when upgrading to Fusion 5.10, but the existing developer role is not updated in order to prevent unexpected changes. To gain access to the developer role’s new permissions, delete the developer role before upgrading to Fusion 5.10 or restart the api-gateway Fusion microservice after upgrading.
```bash theme={"dark"}
GET,POST,PUT:/system/**
GET,POST,PUT,DELETE,HEAD:/stopwords/**
GET,POST,PUT:/usage/**
GET:/features/**
GET,POST,PUT,DELETE,HEAD:/blobs/**
GET,POST,PUT,DELETE,HEAD:/scheduler/**
GET,POST,PUT,DELETE,HEAD:/experiments
GET:/introspect/**
PUT:/usage/**
GET,POST,PUT,DELETE,HEAD:/index-stages/**
GET,POST,PUT,DELETE,HEAD:/messaging/**
GET,POST,PUT,DELETE,HEAD:/catalog
GET,POST,PUT,DELETE,HEAD:/parsers/**
GET,POST,PUT:/appkit/**
GET,POST,PUT,DELETE,HEAD:/index-profiles/**
GET,POST,PUT:/recommend/**
GET,POST,PUT,DELETE,HEAD:/history/**
GET,POST,PUT,DELETE,HEAD:/apps/**
GET,POST,PUT,DELETE,HEAD:/solr/**
GET,POST:/query/**
GET,POST,PUT:/signals/**
GET,POST,PUT:/searchLogs/**
GET,POST,PUT,DELETE,HEAD:/query-pipelines/**
GET,POST,PUT:/configurations/**
GET:/suggestions/**
GET,POST,PUT,DELETE,HEAD:/searchCluster/**
GET,POST,PUT,DELETE,HEAD:/index-pipelines/**
GET:/license
GET,POST,PUT,DELETE,HEAD:/spark/**
GET,POST,PUT,DELETE,HEAD:/query-stages/**
GET,POST,PUT,DELETE,HEAD:/prefs/apps/search/*
GET:/nodes/**
GET,POST,PUT,DELETE,HEAD:/solrAdmin/**
GET,POST,PUT:/synonyms/**
GET,POST,PUT,DELETE,HEAD:/jobs/**
GET,POST,PUT,DELETE,HEAD,OPTIONS:/collections/**
GET,POST,PUT,DELETE,HEAD:/connectors/**
GET,POST,PUT,DELETE,HEAD:/groups/**
GET,POST,PUT,DELETE,HEAD:/query-profiles/**
GET,POST,PUT:/templates/**
GET,POST,PUT,DELETE,HEAD:/tasks/**
GET,POST,PUT,DELETE,HEAD:/links/**
PATCH:/users/{id}:id=#ID
GET,POST,PUT:/registration/**
POST:/index/**
GET,POST,PUT:/objects/**
GET,POST,PUT:/templates/**
```
The permission `PATCH:/users/{id}:id=#ID` uses the variable value `#ID` as a placeholder for the currently logged-in user ID. It is included so the Fusion UI "change password" feature is available to native realm users.
### rules
The rules role provides [query rewriting](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting) API access to all Fusion apps.
```bash theme={"dark"}
GET:/apps/*/query-profiles/**
GET,POST,PUT,PATCH,DELETE,HEAD:/apps/*/query-rewrite/**
GET:/solr/**
GET:/query/**
GET:/collections/**
GET:/apps/**
```
### search
The search role has read-only query and write-only signal API access to the Fusion "default" collection. These permissions are required for search applications, for example, for App Studio.
```bash theme={"dark"}
POST:/apps/*/signals/**
GET,POST:/query/**
POST:/signals/**
PATCH:/users/{id}:id=#ID
GET,POST:/apps/*/query/**
```
The permission `PATCH:/users/{id}:id=#ID` uses the variable value `#ID` as a placeholder for the currently logged-in user ID. It is included so the Fusion UI "change password" feature is available to native realm users.
If you are using Neural Hybrid Search (NHS), you must also add `GET,POST,PUT:/{LWAI_SERVICE_ACCOUNT_NAME}/**` to the `search` role, where `{LWAI_SERVICE_ACCOUNT_NAME}` is the name of your Lucidworks AI service account.
### webapps-role
The webapps role can list and download Fusion apps.
```bash theme={"dark"}
GET,HEAD:/webapps/**
GET,HEAD:/license
```
## Role Information
Fusion stores role information in [Apache ZooKeeper](http://zookeeper.apache.org/).
Each role in a ZooKeeper entry contains the following:
* `id`– ID string, created by Fusion
* `name`– Role name string
* `desc`– Text description; optional
* `permissions`– A list of Fusion permission specifications
* `ui-permisions`– A list of names of Fusion UI components
* `created-at`– Timestamp; created by Fusion
* `updated-at`– Timestamp for last edit; created by Fusion
## Manage Roles
Only Fusion users with admin privileges can manage roles.
Restricting access to a subset of Fusion’s functionality requires several narrowly defined permissions.
Path variables can be used to designate specific collections.
As an example, it is possible to define a role which allows read-only access to Fusion dashboards for a specific collection:
* `GET:/solr/{id}/*:id=test` – Read-only access to the collection "test"
* `GET:/solr/{id}/admin/luke:id=test` – Also read-only access
* `GET:/solr/system_banana/*` – Read-only access to dashboards
* `GET:/collections/system_banana` – Read-only access to the collection where dashboard definitions are stored
### Manage Roles in the Fusion UI
Manage roles in the Fusion UI. Click **System > Access Control > Roles**.
To create a new role from the Fusion UI, first choose a unique role name, then edit the set of permissions.
Specify API permissions one per line in the **Permissions** input box.
There is a separate list of checkboxes that allow access to the Fusion UI components.
If users who are assigned this role require access to the Fusion UI, then you must specify UI permissions in addition to REST API permissions.
### Manage Roles via HTTP Requests to the Roles API
See page [Roles API](/api-reference/role-management/list-roles).
## Assign job permissions
**Important**
The colon character `:` is a special character used in the permissions engine, so you must use an asterisk `*` in the command.
To assign permissions for a specific job, enter the following command, where you must enter the specific job in the `` field:
```bash theme={"dark"}
POST:/apps/*/jobs/task*/actions
```
### Assign job permissions in the UI
The command to assign permissions for a specific job in the UI is an exact call, so you must specify the ``.
For example, if the job name is `testing-call`, two example commands are:
```bash theme={"dark"}
POST:/apps//jobs/task:testing-call/actions
```
or
```bash theme={"dark"}
POST:/apps/*/jobs/task:testing-call/actions
```
### Assign permissions in the API
The command to assign permissions to a job using an API does not require you to specify the ``.
Two example commands are:
```bash theme={"dark"}
POST:/jobs/**
```
or
```bash theme={"dark"}
POST:/jobs/task*/actions
```
# Security Realms
Source: https://doc.lucidworks.com/docs/5/fusion/operations/security/access-control/security-realms
[localhost link]: http://localhost:3000/docs/5/fusion/operations/security/access-control/security-realms
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/security/access-control/security-realms
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/127
Fusion uses *security realms* to authenticate users of the Fusion UI. Each user has an assigned security realm, which the user must choose when logging in. Choosing a different realm results in an authentication failure.
The [Mutual Transport Layer Security](/docs/5/fusion/operations/security/mtls-fusion) service provides network-layer authentication and requires both the client and server to present valid certificates during the TLS connection process. mTLS operates before security realms authenticate the user at the application layer.
A security realm also provides a list of roles:
* The list always includes the role(s) that are specified in the security realm.
* (Optional) The security realm can reference one or more Fusion roles and/or get groups to which the user belongs from an external directory service that is the authentication provider. Fusion maps the group names to role names and adds these roles to the user’s list of roles.
Fusion does not use permissions from the LDAP to authorize UI access or API requests. It only obtains group names (optionally), which are used as role names or are mapped to role names.
If an [Active Directory Security Query Trimming Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/active-directory-security-trimming-query-stage) is used,
then directory-service permissions *are* used for trimming. If a connector supports security trimming, then connector permissions *are* used for trimming.
Requests to the Fusion REST API must specify a security realm for per-request authentication, unless a session cookie is used (which contains information about the security realm).
Fusion authorizes requested operations based on API permissions specified for the user and for the user’s role(s). Fusion considers the role(s) specified in the user definition and in the security realm. Fusion creates a list of roles when a session is created, that is, when a user logs in or when the [Sessions](/api-reference/system-status/get-the-current-user-session) REST API creates a session. Authorization based on permissions is at request time.
You can define multiple security realms for a Fusion instance. This lets you give different sets of users different levels of access to specific Fusion collections.
The directory service that you use to return group information must be the same one that is used for authentication.
## Security Realm Types
When you create a security realm, you can choose among the following security realm types:
### OpenID Connect
OpenID Connect is an identity authorization layer which supplements the OAuth 2.0 protocol.
**Important**
Fusion’s OpenID Connect security realm has been tested with [Google](https://developers.google.com/identity/protocols/OpenIDConnect) and [Okta](https://developer.okta.com/docs/reference/api/oidc).
See **Configure OpenID Connect Authentication**.
## Configuration
Use the [Realms API](/api-reference/realm-management/create-a-realm) to configure this realm type:
```
curl -u USERNAME:PASSWORD -H 'content-type:application/json' -X POST http://:6764/api/realm-configs -d @./realm-config.json
```
Below is a sample configuration:
```json theme={"dark"}
{
"realmType": "oidc",
"name": "{your_oidcName}",
"enabled": true,
"roleNames": [
"admin"
],
"config": {
"autoCreateUsers": true,
"groups": {
"roleMapping": [
[
"role_user",
"admin"
]
]
},
"code": {
"clientSecret": "{your_clientSecret}",
"redirectUri": "{your_redirectUri}",
"authorizationUri": "{your_authorizationUri}",
"tokenUri": "{your_tokenUri}"
},
"clientId": "{your_clientId}",
"jwkSetUri": "{your_jwkSetUri}",
"userIdAttribute": "email",
"scope": [
"openid",
"email",
"profile"
]
}
},
```
### Required fields
| Field | Description | Example |
| - | - | - |
| `name` | Name of the OIDC realm. | `oidc`. |
| `clientSecret` | A secret value shared between the application and the authentication server. | N/A |
| `redirectUri` | The URI to which the user will be redirected to after logging in. | `http://{fusion-url}:6764/admin` |
| `authorizationUri` | The authorization server URI. | `https://$\{yourOktaDomain}/oauth2/default/v1/authorize` |
| `tokenUri` | The URI to get access token from. | `https://$\{yourOktaDomain}/oauth2/default/v1/token` |
| `clientId` | A unique value which identifies the client. | N/A |
| `jwkSetUri` | The URL of the authorization server’s JSON Web Key Set (JWKS). | `https://$\{yourOktaDomain}/oauth2/default/v1/keys` |
## Google authentication
For authenticating with Google, use [Google’s OpenID Configuration](https://accounts.google.com/.well-known/openid-configuration) to retrieve the required values for `authorizationUri`, `tokenUri`, `jwkSetUri`, and `issuer`.
## Okta authentication
OpenID Connect authentication with Okta involves mapping Okta groups to Fusion roles. The Okta group information can be retrieved from Okta’s admin view:
1. Navigate to **API** > **Authorization Server**
2. Select the server you will configure for mapping
3. In the **Scope** menu, add the authentication groups
4. In the **Claims** menu, add new claim groups with ID token and set regexp to `.*`, which will expose all groups
### Native
Fusion has a single preconfigured security realm named *native*. The admin user is in the native realm.
The native realm also provides a fallback mechanism in case of LDAP server or communication failure.
This realm is required to bootstrap Fusion. Because all requests to Fusion require authentication and authorization,
on initial startup you must access the Fusion UI to set the admin password.
After Fusion has a valid admin password, it creates the admin account in the Fusion native realm.
For the native realm, Fusion manages all authentication and permissions information directly.
You can create Fusion user accounts and manage them using either the Fusion UI or the [User API](/api-reference/user-management/list-users).
Passwords are not stored, but are hashed using [bcrypt](https://en.wikipedia.org/wiki/Bcrypt).
### SSO Trusted HTTP
The "SSO Trusted HTTP" realm type (`trusted-http` in the REST API) is useful in single sign-on (SSO) environments.
If the SSO environment contains groups that make sense regarding partitioning Fusion functionality for Fusion users (that is, giving Fusion users different UI and API permissions based on the SSO groups), then you can configure an SSO Trusted HTTP security realm to return a list of group names, and then map the groups to Fusion roles in the security realm definition.
See **Configure Fusion for SSO**.
The "SSO Trusted HTTP" realm type (`trusted-http` in the REST API) is useful in single sign-on (SSO) environments.
If SSO is already set up in your environment, user identities and group information can be sent to Fusion through HTTP headers (REMOTE\_USER, for example). The SSO Trusted HTTP realm type provides the configuration options for integrating this into Fusion’s authentication systems. It also supports allowing access to only a set of known client IPs, and mapping groups to Fusion roles.
Use the [Realms API](/api-reference/realm-management/create-a-realm) to configure this realm type:
```
curl -u USERNAME:PASSWORD -H 'content-type:application/json' -X POST :3000/api/realm-configs -d @./realm-config.json
```
Below is a sample configuration:
```json theme={"dark"}
{"id":"test-id",
"enabled":true,
"name":"sso-test",
"realmType":"trusted-http",
"config":{"identityKey":"REMOTE_USER",
"groups": {"key":"GROUPS",
"delimiter":"|",
"roleMapping": [["a","admin"], ["b","foo"]]},
"allowedIps":["127.0.0.1", "0:0:0:0:0:0:0:1", "localhost"]}}
```
| | |
| - | - |
| `identityKey` | The name of an HTTP header. If this key is found in the request headers, its value is used as the identity of the client (username, for example). |
| `groups` | Configuration keys for auth groups:
`key` The name of an HTTP header, used as the source of group names.
`delimiter` The character used to split the value (defaults to comma).
`roleMapping` A set of 2-tuples, used for mapping the external group values to Fusion Roles.
|
| `allowedIps` | Allow access to only a set of known client IPs. When this property is defined and the client IP is not included in it, the realm logic return a 401. In Fusion 5.0.0+, leaving this field empty makes the realm nonoperational. To accept traffic from all destinations for development, you can use an IP such as 0.0.0.0/0. In production the concrete IP should be specified. Prior to Fusion 5.0.0, the `X-FORWARDED-FOR` header is inspected for this client IP first; the value is split on comma, and the first entry is taken. This would normally be used in cases where the client was forwarded to Fusion through one or more external proxy servers. If the `X-FORWARDED-FOR` header is not present in the request, the `REMOTE-ADDR` header value is used instead. |
### JWT
The JSON Web Token (JWT) realm uses an “Authorization” header in the request to authenticate the user and the data inside the JWT token for the authorization.
You can configure Fusion to use a shared secret key to encrypt the JWT payload.
For Fusion’s JWT realm, first create a JWT token, with the tool you use to validate users. Then create a Fusion realm.
See [Configure Fusion for JWT](/docs/5/fusion/reference/config-ref/configure-fusion-for-jwt-5x).
### Kerberos
In the case where a host domain uses Kerberos for authentication and LDAP for authorization,
Fusion can be configured to do the same, by configuring a realm of type "LDAP" and then
specifying Kerberos as the authentication mechanism.
Kerberos support requires Fusion 5.9.5.
Fusion stores a local user record in ZooKeeper and a mapping to the Kerberos [principal](https://docs.oracle.com/cd/E21455_01/common/tutorials/kerberos_principal.html).
[SPNEGO](https://en.wikipedia.org/wiki/SPNEGO) is used for authentication via Kerberos.
See how to configure Fusion for Kerberos in **Configure Fusion for Kerberos in Unix** and **Configure Fusion for Kerberos in Windows**.
To configure the Fusion UI service to use Kerberos for user authentication, you must create a Kerberos [security realm](/docs/5/fusion/operations/security/access-control/security-realms).
Kerberos support requires Fusion 5.9.5.
Kerberos is a system that provides authenticated access for users and services on a network.
Instead of sending passwords in plaintext over the network, encrypted passwords are used to generate time-sensitive tickets that are used for authentication. SPNEGO provides a mechanism for extending Kerberos to Web applications through the standard HTTP protocol.
Kerberos uses symmetric-key cryptography and a trusted third party called a Key Distribution Center (KDC) to authenticate users to a suite of network services. (By users we mean both end users and client programs).
The computers managed by that KDC and any secondary KDCs constitute a realm. When a user authenticates to the KDC, the KDC sends a set of credentials (a ticket) specific to that session back to the user’s machine.
Kerberos-aware services use the ticket on the user’s machine for authentication instead of requiring sign-on with a password. Because tickets are used rather than passwords, this provides the convenience of Single Sign-On (SSO) in addition to security.
A Kerberized process is one that has been configured so it can get tickets from a KDC and negotiate with Kerberos-aware services. When a user sends an HTTP request, Fusion tries to authenticate using the Kerberos/SPNEGO protocol.
If the request was sent from a browser, Fusion does not display the initial sign-on panel; instead on login, the user sees the main Fusion collections panel.
This article focuses on configuring Fusion for Kerberos *in Unix*.
To Kerberize Fusion, you must:
* Configure the Kerberos client on the server that the Fusion UI service will be running on so that it can talk to the KDC (section [Configuring the Kerberos client](#configuring-the-kerberos-client) below).
* Configure the security realm of the Fusion UI (section [Configuring Fusion Authentication for Kerberos Realm](#configuring-fusion-authentication-for-kerberos-realm) below).
* Depending on the encryption used by the KDC, you may also need to install additional Java security libraries into the Fusion distribution.
These are freely available from Oracle, see download and installation instructions below.
To do this, you need following information, which you can get from your sysadmin:
* **Kerberos realm name.** In most cases, this is your domain name in upper case.
* **KDC name.** This is usually a combination of `kerberos.` and your domain name. For example, `kerberos.example.com`
* **Kerberos principal name and password.** A principal is a unique identity to which Kerberos can assign tickets. When the entity is a client program, this is called the Service Principal name.
* **A keytab file.** This holds the encrypted credentials.
The usual scenario in an enterprise organization is to have a Kerberos admin create a service principal with a random key password.
Then, the admin generates a keytab, which is then used for Fusion service principal authentication.
The Kerberos commands needed for configuration and testing are:
* `kinit`. Obtain and cache a ticket from the KDC (i.e., domain login)
* `kdestroy`. Destroys credentials (i.e., domain logout)
* `klist`. Lists cached credentials
* `ktutil`. Create or add credentials to a keytab file
If your browser is not already configured to use the Kerberos/SPNEGO, you need to do so in order to test the Fusion configuration.
## Configuring the Kerberos client
### Step 1: Edit the Kerberos configuration file.
To configure your local Kerberos client so that it can talk to the Kerberized server,
you must edit Kerberos configuration file named `krb5.conf`.
On most Unix systems, this file is located at `/etc/krb5.conf`.
This file contains Kerberos configuration information, including the locations of KDCs and admin servers for the Kerberos realms of interest, defaults for the current realm and for Kerberos applications, and mappings of hostnames onto Kerberos realms.
If your organization realm name is "MYORG.ORG", and your KDC server is named "kerberos.myorg.org", then you edit two entries.
The first entry is `libdefaults`. Set MYORG.ORG as the default realm:
```
[libdefaults]
default_realm = MYORG.ORG
```
The second entry is `realms`. Add MYORG.ORG as a realm:
```
[realms]
MYORG.ORG = {
kdc = kerberos.myrealm.com
}
```
For example, for realm LUCIDWORKS.IO, the krb5.conf file is just like the above example, except that instead of "myorg.org" we specify "lucidworks.io".
### Step 2: Authenticate to Kerberos
The command `kinit` is the Kerberos authentication command.
To get started, you authenticate to Kerberos using the Kerberos principal name and password (which you may need to obtain from your sysadmin).
For this example, the principal name is "prince".
```
> kinit prince
```
The `kinit` command prompts for a password. Successful authentication is silent. Unsuccessful authentication results in an error message.
The command `klist` shows all cached Kerberos credentials. To check that you have successfully authenticated, run this command:
```
> klist
```
Output should be in this form, but with your data:
```
Credentials cache: API:C980F9F5-415C-4A3E-9C67-883C7D5FFFBE
Principal: prince@MYORG.ORG
Issued Expires Principal
May 6 15:14:55 2015 May 7 01:12:47 2015 krbtgt/MYORG.ORG@MYORG.ORG
```
## The Service Principal Keytab file
The usual scenario in an enterprise organization is to have a Kerberos admin create a service principal with a random key password.
Then, the admin generates a keytab, which is then used for Fusion service principal authentication.
If you are your own Kerberos admin, then you will need to create this file for yourself.
### Step 3. Create a Keytab file
The command `ktutil` creates the service principal keytab file
which holds the encrypted credentials that the Fusion UI Proxy will use for Kerberos authentication.
In order to generate the keytab file, you must have a set of cached credentials, therefore, first run the `kinit` command (step 2).
From the command line, run the command `ktutil`.
You must enter your password twice.
```
> ktutil -k http-myrealm.org.keytab add -p HTTP/myrealm.org@MYORG.ORG -e aes256-cts-hmac-sha1-96 -V 0
```
The -k argument specifies the name of the keytab file which will be created or updated. The command "add" takes the following argument flags:
* "-p" : service principal, format \/\@REALM
* "-e" : [encryption type](http://web.mit.edu/kerberos/www/krb5-latest/doc/admin/conf_files/kdc_conf.html#encryption-types). Depending on the encryption type, you may need to download additional Java security libraries for strong encryption.
The following encryption types are not supported by Fusion: \* DES-CBC-CRC \* DES\_CBC\_MD5 \* Microsoft Windows 2000 RC4-HMAC
* "-V" : key version number (kvno). Key version numbers are used in the Kerberos V5 protocol to distinguish between different keys in the same domain.
If successful, this command creates a keytab file called "http-myrealm.org.keytab".
Note the directory you are in - you will need this full path when creating the Proxy realm-config later.
### Step 4. Test the Keytab file
The location of this keytab file will be used to configure UI Proxy configuration.
Before configuring the Fusion UI Proxy, you should check that the keytab file is valid.
Testing the keytab requires the following sequence of steps:
* Clear any existing credentials via command `kdestroy`.
* Log in using the keytab as an argument to the command `kinit -kt `, where `` is the name of a principal within the keytab file.
* Examine your credentials via command `klist`.
* Clear credentials via command `kdestroy`, which removes any existing credentials, effectively logging you out of Kerberos.
To remove cached credentials, use the `kdestroy` command. This command succeeds silently.
To check that credentials have been removed, re-run the `klist` command:
```
> kdestroy
> klist
klist: krb5_cc_get_principal: No credentials cache file found
```
Use the keytab to login as the service principal, without being prompted for a password:
```
> kinit -t http-myrealm.org.keytab HTTP/myrealm.org
```
Examine your credentials via the command `klist`. The output should be similar to this:
```
Credentials cache: API:51D488FF-5CD9-4E16-98FA-B47743F5B4ED
Principal: HTTP/myrealm.org@MYORG.ORG
Issued Expires Principal
Apr 1 09:15:02 2015 Apr 1 19:13:42 2015 krbtgt/MYORG.ORG@MYORG.ORG
```
Logout again with `kdestroy`.
## Configuring Fusion Authentication for Kerberos Realm
Once you have tested both the user and service principal logins, you must create the service principal realm-config in the Fusion Authentication Proxy. This allows the Proxy to authenticate to Kerberos as the service principal, without a password.
### Step 5. Configure the Fusion Realm
Fusion security realms can be configured either via the Fusion UI Admin tool or the Fusion REST API. The advantage of using the Fusion UI Admin Tool is that a single panel’s worth of configuration requires a series of calls to the REST API. It is important to understand the set of configuration properties collected by the Fusion UI and how they are used by the REST API.
A security realm for Kerberos has the following properties:
* name : unique string identifier
* realmType : "kerberos"
* enabled : whether or not the realm is available for users to use with system authentication
* config : this property is required for the realm type "kerberos". It takes two key-value pairs:
* principal : the principal service name
* keytab : this must be the full path to the keytab file.
To configure Fusion via the Fusion UI:
1. Log in to Fusion as "admin" or as a user who has super-admin privileges.
2. In the left-hand side navigation bar, click **System > Access control**.
3. Click the **Security Realms** tab, then **Add Security Realm**.
4. On the New Security Realm form, choose type "kerberos" from the pulldown menu so that there are input boxes for the Kerberos realm properties "Service Principal" and "Keytab path".
5. Choose a realm name, and make sure the "enabled" box is checked.
6. If the **auto-created** box is unchecked, a user with admin privileges will need to create Fusion user accounts.
7. Select default roles. The default roles are assigned to a user the first time they access Fusion via this realm, using the Kerberos/SPNEGO protocol. For example, once you have defined a Kerberos realm "my-kerberos-realm" for domain "MYORG.ORG", when user "any.user" in domain "MYORG.ORG" authenticates to Fusion for the first time via this realm, they are added to the set of Fusion users as username "[any.user@MYORG.ORG](mailto:any.user@MYORG.ORG)" and they have all default roles.
It is prudent to allow the minimum set of default roles, as all users will have these permissions. Some users will require admin privileges and a few users will require super-admin privileges. There should always be a user with super-admin privileges that can authenticate to Fusion using the native security realm and can then grant permissions to individual users as needed.
The Fusion proxy Kerberos realm works only with one Fusion host. For a multi-node Fusion cluster where each node resides on a different host, configure a separate realm for each host with a dedicated service/principal.
8. Enter the service principal and the full path to the keytab file.
9. Click **Save**. Fusion displays a confirmation message.
## Kerberos/SPNEGO HTTP Authentication
[SPNEGO](http://en.wikipedia.org/wiki/SPNEGO) provides a mechanism for extending Kerberos to applications that use the HTTP protocol including web browsers and the `curl` command-line utility.
### Step 6. Configure the HTTP client
When a user sends a request to the Kerberized Fusion UI, a SPNEGO request (http\[s]) is made.
If the user is not already authenticated, the Fusion authentication proxy will yield a 401 status code and a Negotiate header.
This status/header response triggers compatible clients to fetch a local ticket from their Kerberos "ticket tray". This ticket is then encoded and sent back to the Fusion. The Fusion authentication proxy will then decode the ticket, and perform a SPN.doAs(user) authentication request to the KDC/Authentication Service. Depending on the results, the proxy then successfully executes the original request (along with a session cookie) or a 401 (without the Negotiate).
Clients can either choose to use the session cookie or continue authenticating on every request.
### Configuring Web Browsers and `curl` for SPNEGO
The `--negotiate` option enables SPNEGO in `curl`.
IE and Safari require no additional configuration to use SPNEGO.
To configure Firefox, access the low level configuration page by loading the about:config page. Then go to the network.negotiate-auth.trusted-uris preference and add the hostname or the domain of the web server that is HTTP Kerberos SPNEGO protected (if using multiple domains and hostname use comma to separate them).
The Chrome browser must be launched from the command line with several added parameters.
To run Chrome on linux:
```
> google-chrome --enable-plugins --args\
--auth-server-whitelist="*KERBEROS_DOMAIN"\
--auth-negotiate-delegate-whitelist="*KERBEROS_DOMAIN"\
--auth-schemes="basic,digest,ntlm,negotiate"
```
To run Chrome on a Mac:
```
> open 'Google Chrome.app' --args\
--auth-server-whitelist="*ROGUECLOUD.COM"\
--auth-negotiate-delegate-whitelist="*KERBEROS_DOMAIN"\
--auth-schemes="basic,digest,ntlm,negotiate"
```
To run Chrome on Windows:
```
chrome.exe --auth-server-whitelist="*KERBEROS_DOMAIN"\
--auth-negotiate-delegate-whitelist="*ROGUECLOUD.COM"\
--auth-schemes="basic,digest,ntlm,negotiate"
```
For more information, see [Using a Web Browser to Access an URL Protected by Kerberos HTTP SPNEGO](https://docs-archive.cloudera.com/documentation/enterprise/6/6.1/topics/cdh_sg_browser_access_kerberos_protected_url.html) and [Authentication + Authorization](http://www.roguelynn.com/words/apache-kerberos-for-django/).
### Session cookies
A successful Kerberos/SPNEGO login will yield a session cookie, this cookie is identical to the cookie yielded by the Fusion authentication proxy’s current POST-login-mechanism.
The expiration policy on the cookie is currently fixed at 8 hours. But has a 1 hour "idle" max, which means if you do not make a request for 1 hour, the cookie is invalidated. Otherwise the lifetime is pushed ahead until the 8 hour max is met.
The name of the cookie is "id" and the value is a UUID. This UUID is a key that maps to an in-memory value containing the real user ID.
## Testing and Troubleshooting
Once you have configured the Kerberos security realm, you can test it as follows:
1. Log out of Fusion and shut down the browser.
2. Check that you have a valid Kerberos authentication ticket via the `klist` command.
3. Open a new browser session and access the Fusion installation.
This should take you directly to the main Fusion panel, bypassing the Fusion "Welcome" login panel. To view your login profile, click on the profile icon at the top right. Your user name should be your login name + "@" + your domain name.
* If you see the Fusion login panel instead of the main Fusion panel, your browser is not configured for SPNEGO.
* If the Fusion display consists only of an empty top nav bar, this indicates an authentication failure. Check that the path to your keytab file is correct. Then check the Fusion logs.
* If your KDC uses a weak encryption method, Fusion rejects your login credentials. The following weak encryption types are disabled in Fusion:
* DES-CBC-CRC
* DES\_CBC\_MD5
* Microsoft Windows 2000 RC4-HMAC
## References and Tutorials
* [https://en.wikipedia.org/wiki/Kerberos\_%28protocol%29](https://en.wikipedia.org/wiki/Kerberos_%28protocol%29)
* [http://www.roguelynn.com/words/explain-like-im-5-kerberos/](http://www.roguelynn.com/words/explain-like-im-5-kerberos/)
* [https://en.wikipedia.org/wiki/SPNEGO](https://en.wikipedia.org/wiki/SPNEGO)
* [https://access.redhat.com/documentation/en-US/Red\_Hat\_Enterprise\_Linux/6/html/Managing\_Smart\_Cards/Using\_Kerberos.html#about-kerberos](https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Managing_Smart_Cards/Using_Kerberos.html#about-kerberos)
* [http://www.oracle.com/technetwork/articles/idm/weblogic-sso-kerberos-1619890.html](http://www.oracle.com/technetwork/articles/idm/weblogic-sso-kerberos-1619890.html) - section "Install Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files"
* [http://www.cisco.com/c/en/us/support/docs/security-vpn/kerberos/16087-1.html](http://www.cisco.com/c/en/us/support/docs/security-vpn/kerberos/16087-1.html)
To configure the Fusion UI service to use Kerberos for user authentication, you must create a Kerberos [security realm](/docs/5/fusion/operations/security/access-control/security-realms).
Kerberos support requires Fusion 5.9.5.
Kerberos is a system that provides authenticated access for users and services on a network.
Instead of sending passwords in plaintext over the network, encrypted passwords are used to generate time-sensitive tickets that are used for authentication. SPNEGO provides a mechanism for extending Kerberos to Web applications through the standard HTTP protocol.
Kerberos uses symmetric-key cryptography and a trusted third party called a Key Distribution Center (KDC) to authenticate users to a suite of network services. (By users we mean both end users and client programs).
The computers managed by that KDC and any secondary KDCs constitute a realm. When a user authenticates to the KDC, the KDC sends a set of credentials (a ticket) specific to that session back to the user’s machine.
Kerberos-aware services use the ticket on the user’s machine for authentication instead of requiring sign-on with a password. Because tickets are used rather than passwords, this provides the convenience of Single Sign-On (SSO) in addition to security.
A Kerberized process is one that has been configured so it can get tickets from a KDC and negotiate with Kerberos-aware services. When a user sends an HTTP request, Fusion tries to authenticate using the Kerberos/SPNEGO protocol.
If the request was sent from a browser, Fusion does not display the initial sign-on panel; instead on login, the user sees the main Fusion collections panel.
This article focuses on configuring Fusion for Kerberos *in Windows*.Kerberos support requires Fusion 5.9.5.
## Set up and Configure the Active Directory Domain Service
### Install the Active Directory Domain Service
To begin, follow [Microsoft’s instructions](https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/deploy/install-active-directory-domain-services--level-100-) on how to install Active Directory Domain Services.
This topic uses the following domain throughout this example:
| | |
| - | - |
| **NETBIOS Domain** | FUSIONIS |
| **Fully qualified domain name** | fusionis.life |
### Create a New User
Launch the Server Manager. Navigate to **Tools > Active Directory Users and Computers**. Within the `fusionis.life` server, locate the "Users" folder, right-click on the folder and select **New > User**.
When creating a user for this purpose, ensure the `First name` and `Display name` fields both hold the exact same value.
These instructions use the following user name throughout this example:
| User name | Notes |
| - | - |
| **Distinguished name (DN)** |
CN = kerberos500
CN = Users
DC = fusionis
DC = life
|
| **User Principal Name (UPN)** | `kerberos500@fusionis.life` |
| **NETBIOS Domain\samAccountName** | FUSIONIS\kerberos500 |
Establish the desired delegation rules for the new user in the "Delegation" tab.
### Create a Keytab for the New User
Enter the following command into the command prompt to create a keytab for the user you just created:
```
ktpass -out c:\Users\Administrator\kerberos500.keytab -princ HTTP/fusionis.life@FUSIONIS.LIFE -mapUser FUSIONIS\kerberos500 -mapOp set -pass YOUR_PASSWORD -crypto AES256-SHA1 -pType KRB5_NT_PRINCIPAL -kvno 0
```
The following weak encryption types are not supported by Fusion:
* DES-CBC-CRC \* DES\_CBC\_MD5
* Microsoft Windows 2000 RC4-HMAC
The output will be similar to:
```
C:\Users\Administrator>ktpass -out c:\Users\Administrator\kerberos500.keytab -princ HTTP/fusionis.life@FUSIONIS.LIFE -mapUser kerberos500 -mapOp set -pass FroFro123# -crypto AES256-SHA1 -pType KRB5_NT_PRINCIPAL
Targeting domain controller: WIN-OVV6VHBGIB8.fusionis.life
Using legacy password setting method
Successfully mapped HTTP/fusionis.life to kerberos500.
Key created.
Key created.
Key created.
Key created.
Key created.
Output keytab to c:\Users\Administrator\kerberos500.keytab:
Keytab version: 0x502
keysize 71 HTTP/fusionis.life@FUSIONIS.LIFE ptype 1 (KRB5_NT_PRINCIPAL) vno 3 etype 0x1 (DES-CBC-CRC) keylength 8 (0xcd07200bea625d20)
keysize 71 HTTP/fusionis.life@FUSIONIS.LIFE ptype 1 (KRB5_NT_PRINCIPAL) vno 3 etype 0x3 (DES-CBC-MD5) keylength 8 (0xcd07200bea625d20)
keysize 79 HTTP/fusionis.life@FUSIONIS.LIFE ptype 1 (KRB5_NT_PRINCIPAL) vno 3 etype 0x17 (RC4-HMAC) keylength 16 (0xe39a141de38abd8750bf9c0bf49fd1c5)
keysize 95 HTTP/fusionis.life@FUSIONIS.LIFE ptype 1 (KRB5_NT_PRINCIPAL) vno 3 etype 0x12 (AES256-SHA1) keylength 32 (0xdd6018ab718a4c312038189a20498f70c07d09a19d77ceea31cdd4cd1b08800a)
keysize 79 HTTP/fusionis.life@FUSIONIS.LIFE ptype 1 (KRB5_NT_PRINCIPAL) vno 3 etype 0x11 (AES128-SHA1) keylength 16 (0x9b610485eae2d97dcf0ecaf4ab38a32b)
```
This process will create the `kerberos500.keytab` file required to set up the Kerberos realm in Fusion. The command will also set the service principal to `HTTP/fusionis.life@FUSIONIS.LIFE`.
If you see the following, the user was not created correctly: `DsCrackNames returned 0x2 in the name entry for kerberos500. Ktpass:failed getting target domain for specified user.` Delete the user, and repeat [the steps above](#create-a-new-user).
## Set up Kerberos on Fusion Server
### Install Fusion and set up the Kerberos Realm
If you have not done so already, install Fusion now.
Copy the keytab file to the server.
The keytab file must be installed on each Fusion node.
Log into Fusion as a native admin and navigate to **DevOps > Access Control > Realms > New Realm**. Use the following parameters:
| | |
| - | - |
| **Type** | Kerberos |
| **Keytab file** | The path to the keytab file you saved on the server |
| **Service principal** | The service principal you generated in the previous step |
| **Kerberos name mapping** | The field that specifies the mapping from a Kerberos Principal name to a short name |
You can specify the default rule, which should work for most situations:
```
DEFAULT
RULE:[1:$1@$0](.*@.*)s/@.*//
```
You can also add custom rules. See [Mapping Kerberos Principals to Short Names](https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cdh_sg_kerbprin_to_sn.html) for more information on creating custom rules.
Save your new realm before continuing. You should now be able test with Negotiate authentication.
## Verify Kerberos is Working
You will test to verify that Kerberos is working in the following sections. You must log into Windows as a user in the protected realm, meaning that the service principal name you are using has access to generate a ticket for your user.
### Verify with cURL
Windows users can log in as a user within the domain, launch the command prompt, and run the cURL command below:
```
curl --negotiate -u : http://fusionis.life:8764/api/apollo/collections
```
Linux users may also need to run a `kinit` to login as a user prior to running.
Navigate your web browser to `http://fusionis.life:8764`. If you are automatically logged in, Kerberos is working.
### Verify with Google Chrome
To verify Kerberos is working with Google Chrome, run Chrome with the following command parameters:
```
"c:\Program Files (x86)\Google\Chrome\Application\chrome.exe" --auth-server-whitelist="*fusionis.life" --auth-negotiate-delegate-whitelist="*fusionis.life" --auth-schemes="basic,digest,ntlm,negotiate"
```
Navigate your web browser to `http://fusionis.life:8764`. If you are automatically logged in, Kerberos is working.
### Verify with Firefox
To verify Kerberos is working with Firefox, navigate to `about:config` in your browser’s URL bar. If a message appears stating "This might void your warranty!", click "I accept the risk!".
Search for the string `network.negotiate-auth.trusted-uris`, double-click the field, and enter `fusionis.life` into the field that appears. Click "Ok".
Navigate your web browser to `http://fusionis.life:8764`. If you are automatically logged in, Kerberos is working.
### Troubleshooting
#### Defective token detected
If you encounter this issue, retrieve logs from the `api-gateway` pod after attempting to make a request: `kubectl logs -n NAMESPACE -l app=api-gateway`
Some deployments use `proxy` instead; if `api-gateway` returns no results, try `-l app=proxy`.
See [Stack Overflow](https://stackoverflow.com/questions/47227276/gssexception-defective-token-detected-when-trying-to-authenticate-to-tomcat) for additional steps to resolve this issue.
#### Kerberos Name mapping rules
You may see the following error if you did not specify a mapping rule and the Proxy could not figure out how to map the Kerberos Principal name to a short name:
```
2021-07-30T15:15:03,584 - ERROR [qtp353842779-52:clojure.tools.logging$eval11$fn__16@0] - {} - Uncaught application error!
org.apache.hadoop.security.authentication.util.KerberosName$NoMatchingRule: No rules applied to YOURUSERNAME@YOURDOMAIN.SUBDOMAIN.EXAMPLE.COM
```
Adjust your Kerberos name mapping rules to resolve this issue.
### LDAP
You can use LDAP as an authentication provider for Fusion. If the LDAP contains groups that make sense regarding partitioning Fusion functionality for Fusion users (that is, giving Fusion users different UI and API permissions based on the LDAP-group memberships of LDAP users), then you can configure an LDAP security realm to search for LDAP groups and to map the LDAP groups to Fusion roles.
Fusion stores a local user record in ZooKeeper, and authentication is performed by the LDAP server.
User accounts can be managed by Fusion, or created automatically, in which case the
Fusion user ID maps directly to the LDAP [Distinguished Name](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol#Directory_structure) (DN).
Fusion permissions can be assigned automatically based on LDAP group membership.
See **Configure Fusion 5.x for LDAP**.
You can create security realms that use external LDAP servers for authentication. Optionally, Fusion can search in the LDAP for groups to which a user belongs, and then map those groups to Fusion roles. Fusion performs authorization using permissions stored in Fusion users and Fusion roles.
Fusion *does not* use permissions from the LDAP for authorization of UI access or API requests. It only obtains group names (optionally), which are mapped to role names. If an [Active Directory Security Query Trimming Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/active-directory-security-trimming-query-stage) is used, then directory-service permissions *are* used for trimming. If a connector supports security trimming, then connector permissions *are* used for trimming.
To configure Fusion to use an external LDAP as an authentication provider, you’ll need to get information about the LDAP server(s) running on your system, either from your system or your sysadmin.
## Add an LDAP Security Realm
1. Log in to the Fusion UI as the user admin, or as a different user with corresponding permissions.
2. Navigate to **System** > **Access Control**.
3. Click **Security Realms**.
4. Click **Add Security Realm**.
5. Specify info for the new realm:
* `name` – Name of the security realm. It must be unique. It should be descriptive but short.
* `type` – Select **ldap** from the pulldown menu.
When you select **ldap**, Fusion displays additional, LDAP-specific configuration options.
* `enabled` checkbox – Whether Fusion allows user logins for this security realm. The default is yes (checked).
* `auto-create users` checkbox – Whether a user account is created automatically upon initial authentication. The default is yes (checked). If the checkbox is unchecked, then a Fusion user with admin permissions must create Fusion users.
6. Scroll down and specify additional options as explained in detail below.
## Specify Static Roles (Optional)
Specify one or more Fusion roles for the security realm. These roles are always considered. They do not depend on searching for LDAP groups and mapping group names to Fusion role names.
In a security realm, you can specify these static roles, add to the list of roles dynamically through an LDAP search, or both. If you do neither, Fusion uses only the role(s) and permissions defined for the user.
## Specify LDAP Connection Details
Specify the hostname and port of the LDAP server. Check the checkbox if the server is running over SSL.
## Specify the Authentication Method
Specify the authentication method:
* **Bind.** LDAP authentication is carried out via a single "Bind" operation. See [Bind](#bind) below.
* **Search.** LDAP authentication is carried out indirectly via a Search operation followed by a Bind operation. See [Search](#search) below.
* **Kerberos.** Kerberos authenticates Fusion and an LDAP Search operation is carried out to find group-level authorizations. See [Kerberos](#kerberos) below.
### Bind
Use the Bind authentication method when the Fusion login username matches a part of the LDAP distinguished name (DN). Specify the remainder of the LDAP DN in the "DN Template" configuration entry, which uses a single pair of curly braces (`{}`) as a placeholder for the value of the Fusion username.
### Search
Use the Search authentication method when the username used for Fusion login *doesn’t* match a part of the LDAP DN. The search request returns a valid user DN, which is then used together with the user password for authentication via a Bind request.
1. Construct a search request.
The Search authentication method is generally required when working with Microsoft Active Directory servers. In this case, you need to know the username and password of *some* user who has sufficient privileges to query the LDAP server for user and group memberships; this user doesn’t have to be the superuser.
In addition to a privileged user DN and password, the Search authentication method requires constructing a search request. There are two parts to the request. The first part is the base DN of the LDAP directory tree that contains user account objects. The second part of the request is a Search Filter object that restricts the results to a matching subset of the information.
2. Provide the administrator bind DN:
### Kerberos
Use the Kerberos authentication method when Kerberos is the authentication provider.
## Search for LDAP Groups (Optional)
A Fusion role is a bundle of permissions tailored to the access needs of different kinds of users. Access to services and data for LDAP-managed users is controlled by mappings from LDAP users and groups to Fusion roles.
Roles can be assigned globally or restricted to specific LDAP groups. The security realm configuration panel contains a list of all Fusion roles with a checkbox for each, used to assign that role to all users in that realm. LDAP group names can be mapped directly to specific Fusion roles and LDAP group search and filter queries can also be used to map kinds of LDAP users to specific Fusion roles.
## Map LDAP Groups to Fusion Roles (Optional)
If LDAP group names returned by the search for groups match Fusion role names, you do not need to map the group names to role names. You must map any LDAP group names that do not match to Fusion role names (if you do not, they will not be used).
## Save the Security Realm Configuration
Click **Save**.
Fusion reports whether or not authentication was successful:
## Basic LDAP Concepts and Terminology
The [LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) protocol is used to share information about users, systems, networks, and services between servers on the internet. LDAP servers are used as a central store for usernames, passwords, and user and group permissions. Applications and services use the LDAP protocol to send user login and password information to the LDAP server. The server performs name lookup and password validation. LDAP servers also store Access Control Lists (ACLs) for file and directory objects which specify the users and groups and kinds of access allowed for those objects.
LDAP is an open standard protocol and there are many commercial and open source LDAP servers available. Microsoft environments generally use Active Directory. Unix servers use AD or other LDAP systems such as OpenLDAP, although many Unix systems do not use LDAP at all. To configure Fusion for LDAP, you’ll need to get information about the LDAP server(s) running on your system either from your sysadmin or via system utilities.
### Directories and Distinguished Names
An LDAP information store is a Directory Information Tree (DIT). The tree is composed of entry nodes; each node has a single parent and zero or more child nodes. Every node must have at least one attribute which uniquely distinguishes it from its siblings which is used as the node’s Relative Distinguished Name (RDN). A node’s Distinguished Name (DN) is a globally unique identifier.
The string representation of a DN is specified in [RFC 4514](https://tools.ietf.org/html/rfc4514). It consists of the node’s RDN followed by a comma, followed by the parent node’s DN. The string representation of the RDN is the attribute-value pair name, connected by an equals ("=") sign. This recursive definition means that the DN of a node is composed by working from the node back through its parent and ancestor nodes up to the root node.
Here is a small example of a DIT:
The person entry in this tree has the DN: "uid=babs, ou=people, dc=example, dc=com".
Attribute names include many short strings based on English words and abbreviations, e.g.:
| Name | Description |
| - | - |
| cn | commonName |
| dc | domainComponent |
| mail | email address |
| ou | organizationalUnitName |
| sn | surname |
| uid | userId |
LDAP entry attributes can refer to other LDAP entries by using the DN of the entry as value of that attribute. The following example of a directory which contains user and groups information shows how this works:
This tree contains two organizational units: "ou=people" and "ou=groups". The children of the "group" organizational unit are specific named groups, just as the child nodes of organization unit "people" are specific users. There are three user entries with RDNs "uid=bob", "uid=alice", "uid=bill" and two groups with RDNs "cn=user" and "cn=admin". The dotted lines and group labels around the person nodes indicate group membership. This relationship is declared on the groups nodes by adding an attribute named "member" whose value is a users DN. In the [LDAP data interchange format (LDIF)](https://en.wikipedia.org/wiki/LDAP_Data_Interchange_Format), this is written:
```
cn=user,ou=groups,dc=acme,dc=org
member: uid=bob,ou=people,dc=acme,dc=org
member: uid=alice,ou=people,dc=acme,dc=org
cn=admin,ou=groups,dc=acme,dc=org
member: uid=bill,ou=people,dc=acme,dc=org
```
See the [Wikipedia’s LDAP entry](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol#Directory_structure) for details.
### LDAP Protocol Operations
For authentication purposes, Fusion sends Bind operation requests to the LDAP server. The Bind operation authenticates clients (and the users or applications behind them) to the directory server, establishes authorization identity used for subsequent operations on that connection, and specifies the LDAP protocol version that the client will use.
Depending on the way that the host system uses LDAP to store login information about users and groups, it may be necessary to send Search operation requests to the LDAP server as well. The Search operation retrieves partial or complete copies of entries matching a given set of criteria.
[LDAP filters](https://www.ldap.com/ldap-filters) specify which entries should be returned. These are specified using prefix notation. Boolean operators are "&" for logical AND, "|" for logical OR, e.g., "A AND B" is written "(&(A)(B))". To tune and test search filters for a Unix-based LDAP system, see the [ldapsearch command line utility](http://docs.oracle.com/cd/E19450-01/820-6169/ldapsearch-examples.html) documentation. For Active Directory systems, see [AD Syntax Filters](http://social.technet.microsoft.com/wiki/contents/articles/5392.active-directory-ldap-syntax-filters.aspx).
### SAML
Fusion stores a local user record in ZooKeeper and the URL and information about the SAML Identity Provider. The SAML 2.0 protocol is used to provide web browser single sign-on.
See **Configure Fusion for SAML**.
[SAML 2.0](https://en.wikipedia.org/wiki/SAML_2.0) is a standard for exchanging authentication and authorization data between security domains. The SAML protocol allows web-browser single sign-on (SSO) through a sequence of messages sent to and from the browser, which is the intermediary between Fusion and the SAML authority acting as the Identity Provider (IdP).
To configure Fusion to use SAML 2.0 for user authentication and authorization you must create a SAML [security realm](/docs/5/fusion/operations/security/access-control/security-realms). In addition to configuring the Fusion security realm, you must configure the SAML identity provider to recognize the Fusion application.
Once Fusion is configured for a SAML realm, this realm is added to the list of available realms on the initial Fusion sign-on panel. When the SAML realm is chosen from the list of available realms, the browser then redirects to the IdP which handles user authentication. Upon successful authentication, the IdP sends a response back to the browser which contains authentication and authorization information as well as the URL of the Fusion application. The browser redirects back to the Fusion URL, passing along the SAML message with the user authentication and authorization information. Fusion then issues as session cookie which is used for subsequent user access.
## Add a Security Realm
1. In the Fusion workspace, navigate to **System** > **Access Control**.
2. Click **Security Realms**.
3. Click **Add Security Realm**.
4. Enter the following information for the new realm:
* Enter a **Name**. The name must be unique and should be descriptive yet short.
* Select **saml** from the **Type** pulldown menu.
When you select a type, Fusion displays additional configuration options.
* The default value for **Enabled** is true. This setting controls whether or not Fusion allows user logins for this security realm.
- The default value for **Ephemeral Users** is false. When disabled, this setting prevents ephemeral users from being created in ZooKeeper during login. If enabled, this property negates **Auto Create Users**.
- The default value for **Auto Create Users** is true. If enabled, a user account is created automatically upon initial authentication. If disabled, then a Fusion user with admin permissions must create Fusion users.
5) Under **SAML Realm**, configure the following information:
* Enter the **Identity Provider URL**. This URL is used by the SAML authority for single sign-on. For example: `https://www.my-idp.com//sso/saml`
The URL format may differ depending on the SAML Identity Provider.
* Enter the URL of the **IdP Issuer**. For example: `http://www.my-idp.com/exk686w2xi5KTuSXz0h7`.
* **IdP Issuer** must match `` in the SAML payload.
* Optionally, provide the **App Issuer**. This field is required if there is an `audienceRestriction` in the SAML assertion and must match `` in the SAML payload.
* In the **Certificate Fingerprint**, paste the contents of the SAML authority certificate, *without the certificate header and footer*.
* Optional: Enter the **User ID Attribute**. By default, the Fusion username is the same as the login name known to the Identity Provider. When another field or attribute in the user record stored by the IdP should be used as the Fusion username, that attribute name is the value of the User ID Attribute.
* Optional: Provide a **Post Login Redirect URL**. If not set, the Fusion URL is used.
* Optional: Provide a **Logout URL**.
6) Optional: Under **Groups Mapping**, specify the Group Name Attribute and add group mappings.
7) Click **Save**.
## Configure SAML identity provider
To finish setup, register Fusion with your SAML identity provider. The amount of information required to register varies depending on the SAML authority.
All systems require the Fusion URL to redirect to upon successful login. This includes the protocol, server, and port for the Fusion application, and path `api/saml`, such as `https://www.my-fusion-app.com:8764/api/saml`. If the Fusion application is running behind a load-balancer, then this URL is the load-balancer URL plus path `api/saml`.
Note that the load-balancer should be session-sticky in order for the sequence of messages that comprise the SAML protocol to run to completion successfully.
Some authorities may require additional information. In particular, the SAML 2.0 "AudienceRestriction" tag may be part of the SAML message. This tag specifies the domain for which the SAML trust conditions are valid, which is usually the domain in which the Fusion app is running, such as `https://www.my-fusion-app`.
## Example SAML Realm configuration
The Fusion endpoint `api/realm-configs` returns a JSON list of all the configuration objects for all realms. After configuring a SAML realm named `"saml-test"` using the okta.com developer preview tool, the configuration object for this realm is:
```json theme={"dark"}
{
"name":"saml-test",
"realmType":"saml",
"enabled":true,
"config":{
"autoCreateUsers":true,
"idpUrl":"https://dev-417804.oktapreview.com/app/dev417804_1/exk686w2xi5KTuSXz0h7/sso/saml",
"issuer":"http://www.okta.com/exk686w2xi5KTuSXz0h7",
"certificateFingerprint":"MIIDpDCCAoygAwIBAgIGAVQr4A4NMA0GCSqGSIb3DQEBBQUAMIGSMQswCQYDVQQGEwJVUzETMBEG\nA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzENMAsGA1UECgwET2t0YTEU\nMBIGA1UECwwLU1NPUHJvdmlkZXIxEzARBgNVBAMMCmRldi00MTc4MDQxHDAaBgkqhkiG9w0BCQEW\nDWluZm9Ab2t0YS5jb20wHhcNMTYwNDE5MDAxNTI0WhcNMjYwNDE5MDAxNjI0WjCBkjELMAkGA1UE\nBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExFjAUBgNVBAcMDVNhbiBGcmFuY2lzY28xDTALBgNV\nBAoMBE9rdGExFDASBgNVBAsMC1NTT1Byb3ZpZGVyMRMwEQYDVQQDDApkZXYtNDE3ODA0MRwwGgYJ\nKoZIhvcNAQkBFg1pbmZvQG9rdGEuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA\nuNz0trRFPw2d4xXoUvX2oWeZolTVeFtaTnB9SWUyjK4og0WdT7rNdBg10eTvB2ezBwXCf24UGGui\nr1kjkZjiHDqDxKtzQYWpGuzLCjh/4PxKFGDaiUNKcE1Ig5myiEBTvMvv99XtrcI75QdUGDhbMiBr\n2PR5FukWOYepzlBzqY0JSDzX9NYJBKPkz+syK4mj0I6dqtYOU+bcTvjF9sR7jiHtQ+d0Zl8rz1Ca\nyuE3mNUtFJ1IJrY/RArhH1AB6mXbV/de1CXmGhKQbqQAbx9SiKtki9n84gKEwuWdV0jIqcBLGxUQ\ngjbsaIVqed2oX+7F2fh6t0g/I8NPnOWXOTvA+wIDAQABMA0GCSqGSIb3DQEBBQUAA4IBAQCPt+DR\nliIsHO/iVnmLFPGfqrCO/gMv++xnq2wOB19YX7HhT1GIY2YZoVphrQpXH3OO0/8AZJ8ApCmq0E9x\nxUTQwQPBanVqlyLtu1Hr0c6dbAqcd5PtaEe6Ci33nayWPydhOmitvIyb/WtWtbel9HcdUkoGvkAl\ng305jnxkhwGLJm4jkzYe+eaYhd6oG3/JcHqKDGYGf7i2Z+ny0D7vxTeBQ+8PbZfsUKg0PlKyTocb\nydSmDPISsA1xOH5zlw+hzFIdKgD5vW7QZLvpIclNc2hqki/nWl/CHut0TnUuP/V3boREmhDu395n\n/u72pgNANfP7+2DTBb+CBTjGUsAxpKRF",
"userIdAttribute":""
},
"roleNames":["search"],
"id":"52e1c0d2-5e00-4c76-a3d4-57f1381bdb4f",
"createdAt":"2016-04-19T19:49:04Z",
"updatedAt":"2016-04-19T20:06:56Z"
}
```
## References
* [http://docs.oasis-open.org/security/saml/v2.0/saml-core-2.0-os.pdf](http://docs.oasis-open.org/security/saml/v2.0/saml-core-2.0-os.pdf)
* [http://docs.oasis-open.org/security/saml/v2.0/saml-glossary-2.0-os.pdf](http://docs.oasis-open.org/security/saml/v2.0/saml-glossary-2.0-os.pdf)
## Manage Security Realms
Only Fusion users with [admin privileges](/docs/5/fusion/operations/security/access-control/roles) can manage security realms. There are two ways to manage security realms:
In the Fusion UI
Navigate to **System** > **Access Control** > **Security Realms**.
**Using the Realms API**
Use the `http://FUSION_HOST:FUSION_PORT/api/realm-configs/` endpoint to manage security realms. See the
[Realms API](/api-reference/realm-management/get-a-realm)
reference for details.
## Learn more
When a user is assigned to more than 200 groups, Microsoft Azure sends the link to the Microsoft Graph API in its token response instead of listing the user’s groups. Fusion must make a custom call to Azure services in order to download the user’s groups. See [OpenID Connect Core](https://openid.net/specs/openid-connect-core-1_0.html#AggregatedDistributedClaims) for more information.
You can enable this feature in the OpenID Connect realm by adding an additional scope to the configuration in Fusion and adding app permissions in Azure.
## Add the scope in Fusion
1. Navigate to **System** > **Access Control** > **Security Realms**, then click the existing OpenID Connect realm.\
You can also create a new OpenID Connect Realm with **Add Security Realm**.
2. In the **Scope** section, add `https://graph.windows.net/Directory.Read.All`.
3. Click **Save**.
## Add app permissions in Azure
1. Navigate to **App registrations** > **Your App Name** >, then scroll down to **View API permissions**
2. Click **+ Add a permission**, then choose **Microsoft Graph** from the list of Microsoft API options.
3. Click **Application permissions**.
4. Select all permissions in the following drop-downs:
1. **AccessReview**
2. **AgreementAcceptance**
3. **Agreement**
4. **AppCatalog**
### Example configuration
All configuration settings can be found in the **OpenID Connect metadata document**, which is located in **App registrations** > **Your App Name** > **Endpoints** in the Azure overview. See [Fetch the OpenID Connect metadata document](https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-protocols-oidc#fetch-the-openid-connect-metadata-document) for more information.
```json wrap theme={"dark"}
{
"realmType": "oidc",
"name": "azure-realm",
"enabled": true,
"config": {
"autoCreateUsers": true,
"code": {
"clientSecret": "XXXXXXX",
"redirectUri": "http://url-to-fusion-gateway:6764/",
"authorizationUri": "https://login.microsoftonline.com/XXXXXX-XXXXXX-XXXXX-XXXXXX/oauth2/v2.0/authorize",
"tokenUri": "https://login.microsoftonline.com/XXXXXX-XXXXXX-XXXXX-XXXXXX/oauth2/v2.0/token"
},
"clientId": "XXXXXX-XXXXXX-XXXXX-XXXXXX",
"jwkSetUri": "https://login.microsoftonline.com/XXXXXX-XXXXXX-XXXXX-XXXXXX/discovery/v2.0/keys",
"userIdAttribute": "preferred_username",
"scope": [
"openid",
"email",
"profile",
"https://graph.windows.net/Directory.Read.All"
]
}
}
```
# Users
Source: https://doc.lucidworks.com/docs/5/fusion/operations/security/access-control/users
[localhost link]: http://localhost:3000/docs/5/fusion/operations/security/access-control/users
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/security/access-control/users
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/125
All Fusion requests must come from a registered user.
## Add Users
The first user who logs in becomes the user `admin`.
There are two approaches for adding users:
* **Manual** – Add users manually to a security realm that does not auto-create users.
* **Automatic** – For a security realm that uses an external authentication provider, Fusion can add users automatically. When creating the security realm, check **auto-create users**. Fusion creates a user the first time someone logs into Fusion.
When you add a new user manually, you must provide a unique username and valid password.
All other information is optional. However, unless either roles or permissions are specified (or both),
this user will not be able to do anything in Fusion.
If you specify API permissions in a user definition, those permissions *override* corresponding permissions defined in the user’s roles. See [Permissions](/docs/5/fusion/operations/security/access-control/permissions) for more information about how permissions supplied by multiple roles and by user definitions combine.
There is a soft limit of 100 users in Fusion. Above that number, performance degradation may occur and API calls may take longer to complete. Using the **Ephemeral Users** parameter prevents the list of users from being saved and can be used to bypass this soft limit.
## Manage Users in the Fusion UI
Only Fusion users with administrative privileges (for example, those who are assigned the built-in role `admin`) can manage users.
Manage users in the Fusion UI. Click **System > Access Control > Users**.
## Manage Users via HTTP Requests to the Users API
See page [Users API](/api-reference/user-management/list-users).
## User Information
Fusion stores user information in [Apache ZooKeeper](http://zookeeper.apache.org/).
Each User entry in ZooKeeper contains the following:
* `id`– A globally unique user ID (UUID), created by Fusion based on username, realm-name
* `realm-name`– The Fusion [security realm](/docs/5/fusion/operations/security/access-control/security-realms) name; the default is "native".
* `username`– The username string, which is unique within the specified security realm
* `permissions`– List of permissions that have been explicitly assigned to the user in the Fusion UI (in **System > Access Control**)
* `role-names`– List of roles assigned to the user in the Fusion UI (in **System > Access Control**)
* `created-at`– Timestamp; created by Fusion
* `updated-at`– Timestamp for the last edit; created by Fusion
The following JSON shows the ZooKeeper record for the Fusion admin user:
```json wrap theme={"dark"}
{
"id":"57f539d2-3f53-4011-ad6f-257a3f00fc6b",
"username":"admin",
"realm-name":"native",
"password-hash":"$2a$08$3I82umlXLPSshQIW6ngj.Or06DOVgDLGohGmCB9GC0yRtvy5Nfkn6",
"permissions":[],
"role-names":["admin"],
"created-at":"2016-01-28T00:00:18Z"
}
```
The following JSON shows the ZooKeeper record for a user entry managed by Fusion:
```json wrap theme={"dark"}
{
"id":"ae9b345a-79e2-4e6d-8620-e6ed4ed2cc16",
"username":"firstname.lastname",
"realm-name":"lwLDAP",
"permissions":[{"path":"collections/**", "methods":["GET"]}],
"role-names":[],
"created-at":"2016-04-01T21:17:36Z",
"updated-at":"2016-04-01T21:42:15Z"
}
```
## Learn more
## Change your Fusion password
While you are logged in, from anywhere in the Fusion UI:
1. Click **Account Settings**.
2. Click **My Profile**.
3. Follow the UI instructions to update your password.
## Change another user’s Fusion password
If you are an admin, you can change the password for any of your users. While you are logged in, from any app in the Fusion UI:
1. Click **System > Access Control**.
2. Make sure the **Users** tab is selected.
3. Click the user’s name.
4. Check the box under **Change password**.
5. Enter and confirm a new password, and click **Update**.
## Reset a lost admin password
If you have lost the password for the admin account,
[contact Lucidworks Support](https://support.lucidworks.com/hc/en-us/requests/new).
# Secrets management using Fusion Vault
Source: https://doc.lucidworks.com/docs/5/fusion/operations/security/external-vault
This topic describes Fusion Vault, which is a Fusion service that watches labeled Kubernetes secrets and makes them available to Fusion pipelines, jobs, and connectors by logical reference.
Secret references are logical names stored and transferred as plain text in pipeline and connector configurations. The same reference name can point to different underlying Kubernetes secrets in the development, staging, and production environments. This enables portable pipeline and connector configurations that do not require modification when moving between environments.
Credential values never display in Fusion configurations.
These examples illustrate the datasources, index and query pipeline stages, and job settings in Fusion.
Your managed secret is found in the datasource configuration in **Crawl Authentication Properties > Basic Authentication > Host**.
If selected, **Use Managed Secret** applies to the name, password, and secret reference.
Your managed secret is found in **Advanced > Scripts Secrets**.
Kubernetes secrets can be created manually or synchronized from external vault providers using the [Kubernetes External Secrets Operator (ESO)](https://external-secrets.io/latest/).
Fusion Vault supports query and indexing pipelines and [V2 connectors](/docs/fusion-connectors/connectors/overview).
The [ZooKeeper-based secrets management option](/docs/5/fusion/operations/security/zookeeper) is the legacy approach, where credentials are encrypted and embedded directly in pipeline and connector configurations. It is maintained for backward compatibility, and uses V1 connectors.
## Benefits of Fusion Vault
Using Fusion Vault for secrets management, your organization can perform the following functions:
* Create and store credentials in Kubernetes secrets instead of embedding them directly in a pipeline. Kubernetes secrets can be populated from external providers such as HashiCorp Vault, Amazon Web Services (AWS) Secrets Manager, Google Cloud Platform (GCP) Secret Manager, and Azure Key Vault using ESO. Fusion configurations only store references, not actual values.
* Reference credentials by logical name rather than by value. References are stored and exported as plain text, so the same pipeline or connector configuration can be imported into any environment. Each environment can define different underlying Kubernetes secrets behind the same reference name, with no changes to the configuration itself.
* When you update a credential in your vault, all related pipelines change automatically without manual pipeline updates or service restarts.
* Maintain compliance with strict Kubernetes security policies.
## Use cases
This section provides examples of use cases that illustrate how your organization can use Fusion Vault for secrets management.
* A data scientist creates a query pipeline that needs to access multiple external services securely. The fields of the pipeline are configured to reference the API and database credentials entered in your organization's Kubernetes vault. The data scientist does not enter the values of the credentials in the Fusion pipeline. When the pipeline runs, authorization is completed using the credentials in the external vault. In addition, the same secret can be reused in multiple pipelines and stages, and are logged for audit purposes.
* A data engineer needs to configure an AWS S3 connector using credentials stored in AWS Secrets Manager. Other backend procedures such as configuring an ESO in the cluster need to be created as well. The data engineer creates an AWS S3 connector in the Fusion Admin UI by selecting the reference names of the credentials, but does not enter the values of the credentials in the Fusion UI. When the connector starts, credentials are pulled from the external vault. Credentials can be updated without reconfiguring the connector, and ESO automatically synchronizes any changes to all instances.
## Operational flow
Fusion Vault watches for Kubernetes secrets labeled `fusion/vault-managed: "true"` and caches their values. The following describes how the components interact:
* **Fusion Admin UI** queries Fusion Vault for the list of available secret labels and presents them in dropdowns when configuring pipelines, jobs, and connectors. Users select a secret reference (for example, `my-creds.password`), but users never see or enter the actual secret value.
* **Fusion services** (jobs, connectors, pipelines, and stages) store only the logical secret reference in their configuration. At runtime, they resolve the actual value through Fusion Vault using a service account. End users cannot access secret values directly.
* **Kubernetes secrets** are the source of truth. Fusion Vault watches them continuously and updates its cache when they change. Secrets can be created directly using `kubectl` or Helm, or are synchronized from an external vault provider (AWS Secrets Manager, Azure Key Vault, HashiCorp Vault, GCP Secret Manager) using the optional [Kubernetes External Secrets Operator (ESO)](https://external-secrets.io/latest/).
When a Kubernetes secret changes, whether updated manually or synchronized by ESO, Fusion Vault detects the update and automatically provides the new value to Fusion services without manual updates or restarts.
Multi-tier caching ensures Fusion pipelines continue to run even if the Kubernetes API is temporarily unavailable, with automatic resynchronization when connectivity is restored.
## Fusion Vault functions
Fusion Vault manages credentials such as database credentials, API keys, cloud storage access keys, and authentication tokens used by pipelines, jobs, and connectors to connect to external services.
Fusion Vault does not manage secrets required for Fusion itself to operate, such as service account passwords, TLS certificates, and encryption keys. Those are defined during installation through Helm chart values or Kubernetes secrets and are not affected by this feature.
## Configuration
Fusion Vault reads Kubernetes secrets that match its required format. You can create conforming secrets using a method that suits your infrastructure. Create secrets using Helm, a CI/CD pipeline, or any tool capable of producing Kubernetes secrets with the required label and annotations.
Ensure you have the following prerequisites in place:
* A Kubernetes cluster with Fusion installed
* Access to the Kubernetes cluster using `kubectl` commands
* Administrative privileges in Fusion for testing and validation
* Administrative privileges in your external vault provider (only required if using a synchronizing tool such as ESO)
For Fusion Vault to pick up a Kubernetes secret, the secret must have the label `fusion/vault-managed: "true"`. Secrets without this label are ignored.
The following annotations are optional but recommended:
* **`fusion/label`**: A human-readable name used when referencing the secret in Fusion. If omitted, the Kubernetes secret name is used in references instead
* **`fusion/description`**: A description of the secret's purpose
**Example:**
```yaml theme={"dark"}
apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: database-creds
namespace: fusion
labels:
fusion/vault-managed: "true"
annotations:
fusion/label: "database-creds"
fusion/description: "PostgreSQL production credentials"
data:
username:
password:
```
Any method to create Kubernetes secrets is valid: `kubectl`, Helm, CI/CD pipelines, or automated synchronizing tools as long as the resulting secret carries the `fusion/vault-managed: "true"` label.
Choose one of the following approaches to create conforming secrets.
Create Kubernetes secrets yourself using `kubectl`, Helm, or a CI/CD pipeline.
Here is an example of using `kubectl` to create a Kubernetes secret and apply the required label and optional annotations:
```bash theme={"dark"}
kubectl create secret generic database-creds \
--from-literal=username=myuser \
--from-literal=password=mypassword \
-n fusion
kubectl label secret database-creds fusion/vault-managed=true -n fusion
kubectl annotate secret database-creds \
fusion/label=database-creds \
fusion/description="PostgreSQL production credentials" \
-n fusion
```
Fusion Vault detects the labeled secret and makes it available for reference in pipelines, jobs, and connectors as `database-creds.username` and `database-creds.password`.
Use any synchronizing tool that creates Kubernetes secrets from an external source and can apply the required label and annotations.
ESO is one example because it automatically synchronizes secrets into Kubernetes from external vault providers such as AWS Secrets Manager, Azure Key Vault, HashiCorp Vault, GCP Secret Manager.
The following diagram illustrates how ESO syncs secrets from external vault providers into Kubernetes secrets that Fusion Vault then reads.
Refer to the [External Secrets Operator installation guide](https://external-secrets.io/latest/introduction/getting-started/) for detailed installation instructions using Helm or `kubectl`.
**Best practices**
Follow these guidelines to optimize the use of the ESO with Fusion:
* Ensure all ExternalSecret resources include the required Fusion label in the template:
```yaml theme={"dark"}
spec:
target:
template:
metadata:
labels:
fusion/vault-managed: "true"
```
* Add annotations to help organize and identify secrets:
```yaml theme={"dark"}
annotations:
fusion/description: "PostgreSQL production credentials"
fusion/label: "postgres-prod"
```
* Use the following recommended refresh intervals:
* Production database credentials: `1h`
* API keys with regular rotation: `30m`
* Development and testing credentials: `4h`
* Use the `creationPolicy: Owner` to ensure Kubernetes secrets are removed cleanly when ExternalSecret resources are deleted.
* Verify Fusion can see external secrets by using the following command:
```bash theme={"dark"}
kubectl get secrets -l fusion/vault-managed=true -n fusion
curl -H "Authorization: Bearer $TOKEN" \
https://fusion.example.com/api/vault/labels?type=secret
```
For detailed information about ESO configuration, security best practices, and operational guidance, see the [External Secrets Operator documentation](https://external-secrets.io/latest/).
A SecretStore is a Kubernetes Custom Resource Definition (CRD) provided by the External Secrets Operator.
It is not part of Fusion's Helm chart. You manage it as a separate infrastructure resource, deployed in the same namespace as Fusion.
The SecretStore acts as a configuration bridge that tells ESO which external vault to connect to and how to authenticate to it. Define it as a YAML manifest and apply it to your cluster:
```bash theme={"dark"}
kubectl apply -f secretstore.yaml -n fusion
```
You can configure multiple SecretStores simultaneously, one per environment (development, staging, production) or one per vault provider.
Each ExternalSecret resource references the appropriate store by name.
```yaml theme={"dark"}
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: gcpsm-secret-store
namespace: default
spec:
provider:
gcpsm:
projectID: "your-gcp-project"
auth:
workloadIdentity:
clusterLocation: us-central1
clusterName: your-cluster
serviceAccountRef:
name: fusion-secrets-sa
```
```yaml theme={"dark"}
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: fusion-aws-secrets
namespace: fusion
spec:
provider:
aws:
service: SecretsManager
region: us-west-2
auth:
jwt:
serviceAccountRef:
name: fusion-external-secrets
```
```yaml theme={"dark"}
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: fusion-azure-secrets
namespace: fusion
spec:
provider:
azurekv:
vaultUrl: "https://your-vault.vault.azure.net"
authType: WorkloadIdentity
serviceAccountRef:
name: fusion-external-secrets
```
```yaml theme={"dark"}
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: fusion-vault-secrets
namespace: fusion
spec:
provider:
vault:
server: "https://vault.example.com"
path: "secret"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "fusion-role"
serviceAccountRef:
name: fusion-external-secrets
```
```yaml theme={"dark"}
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: kubernetes-secret-store
namespace: default
spec:
provider:
kubernetes:
server:
url: "https://kubernetes.default.svc"
auth:
serviceAccount:
name: fusion-secrets-sa
```
Use your vault provider's console or command line interface (CLI) to create the actual secret values. Store the vault paths and keys securely for reference when creating ExternalSecret resources.
Use the Google Cloud Console or enter the `gcloud secrets create` command.
Use the AWS Console or enter the `aws secretsmanager create-secret` command.
Use the Azure Portal or or enter the `az keyvault secret set` command.
Enter the `vault kv put` command.
Create ExternalSecret resources that define how secrets synchronize from your vault to Kubernetes.
ExternalSecret resources must include the `fusion/vault-managed: "true"` label in `spec.target.template.metadata.labels` for Fusion to discover them. This feature requires Kubernetes External Secrets Operator version 0.10 or later.
As you create ExternalSecret resources, enter the following labels and annotations to identify the secrets.
* The `fusion/vault-managed: "true"` label is required and enables Fusion discovery of secrets.
* The following annotations are recommended, but not required:
* **`fusion/label`**: A human-readable identifier used in secret references
* **`fusion/description`**: A description of the secret's purpose
The following tabs are example for credential information:
These are example database credentials from GCP.
```yaml theme={"dark"}
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: database-credentials-sync
namespace: fusion
spec:
refreshInterval: 1h
secretStoreRef:
name: gcpsm-secret-store
kind: SecretStore
target:
name: database-creds
creationPolicy: Owner
template:
metadata:
labels:
fusion/vault-managed: "true"
annotations:
fusion/description: "PostgreSQL production database credentials"
fusion/label: "database-creds"
data:
username: "{{ .db_username }}"
password: "{{ .db_password }}"
host: "{{ .db_host }}"
port: "{{ .db_port }}"
data:
- secretKey: db_username
remoteRef:
key: prod/fusion/database
property: username
- secretKey: db_password
remoteRef:
key: prod/fusion/database
property: password
- secretKey: db_host
remoteRef:
key: prod/fusion/database
property: host
- secretKey: db_port
remoteRef:
key: prod/fusion/database
property: port
```
These are example AWS S3 credentials from AWS Secrets Manager.
```yaml theme={"dark"}
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: data-lake-sync
namespace: fusion
spec:
refreshInterval: 1h
secretStoreRef:
name: fusion-aws-secrets
kind: SecretStore
target:
name: data-lake-creds
creationPolicy: Owner
template:
metadata:
labels:
fusion/vault-managed: "true"
annotations:
fusion/description: "AWS S3 credentials for data lake"
fusion/label: "data-lake"
data:
accessKeyId: "{{ .access_key_id }}"
secretAccessKey: "{{ .secret_access_key }}"
data:
- secretKey: access_key_id
remoteRef:
key: prod/fusion/s3-credentials
property: access_key_id
- secretKey: secret_access_key
remoteRef:
key: prod/fusion/s3-credentials
property: secret_access_key
```
This is an example that illustrates how you can source different secrets from different vault providers.
```yaml theme={"dark"}
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: database-creds
namespace: fusion
spec:
secretStoreRef:
name: fusion-gcp-secrets
kind: SecretStore
target:
name: database-credentials
template:
metadata:
labels:
fusion/vault-managed: "true"
annotations:
fusion/label: "database-creds"
data:
- secretKey: username
remoteRef:
key: fusion-db-credentials
property: username
- secretKey: password
remoteRef:
key: fusion-db-credentials
property: password
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: api-creds
namespace: fusion
spec:
secretStoreRef:
name: fusion-aws-secrets
kind: SecretStore
target:
name: api-credentials
template:
metadata:
labels:
fusion/vault-managed: "true"
annotations:
fusion/label: "api-creds"
data:
- secretKey: apiKey
remoteRef:
key: /fusion/api-credentials
property: apiKey
```
This multi-provider approach allows you to perform the following actions:
* Source secrets from the most appropriate vault for each use case.
* Gradually migrate between vault providers.
* Use different providers for different environments (for example, AWS for production, GCP for development).
* Maintain existing vault infrastructure while adding Fusion.
* Use `kubectl` to verify that secrets exist in Kubernetes with the required label:
1. List all Fusion-managed secrets:
```bash theme={"dark"}
kubectl get secrets -l fusion/vault-managed=true -n fusion
```
2. View secret details:
```bash theme={"dark"}
kubectl describe secret database-creds -n fusion
```
3. Check ExternalSecret status
```bash theme={"dark"}
kubectl describe externalsecret database-credentials-sync -n fusion
```
4. Verify labels and annotations
```bash theme={"dark"}
kubectl get secrets -l fusion/vault-managed=true -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.fusion/label}{"\n"}{end}' -n fusion
```
* Verify that Fusion can access the secrets through the Fusion Vault API by querying the Fusion Vault labels endpoint:
```bash theme={"dark"}
curl -H "Authorization: Bearer $TOKEN" \
https://fusion.example.com/api/vault/labels?type=secret
```
Configure Fusion datasources (connectors), pipelines, and jobs to reference the synchronized secrets.
In the Fusion UI, preview the fields used for secrets:
Your managed secret is found in the datasource configuration in **Crawl Authentication Properties > Basic Authentication > Host**.
If selected, **Use Managed Secret** applies to the name, password, and secret reference.
Your managed secret is found in **Advanced > Scripts Secrets**.
Test each configured pipeline, job, and connector using these steps:
1. Run a test indexing job using the configured datasource.
2. Verify the connector, pipeline, or job can authenticate using the secret reference.
3. Check Fusion logs for any secret resolution errors.
4. Monitor Fusion processes to confirm secrets are accessible.
When secret rotation is necessary, update the secret values in your vault provider. ESO automatically synchronizes changes to Kubernetes based on the `refreshInterval`, and Fusion picks up the new values without manual updates or restarts.
## Migrate secrets management from ZooKeeper to Fusion Vault
Migration from ZooKeeper-based secrets management to Fusion Vault secrets management is manual and optional. Each pipeline, job, connector, and datasource must be **manually updated** to configure where secrets are stored.
To migrate, follow the steps described in the [Configuration](#configuration) section.
Implement migrations during maintenance windows to avoid service disruption.
If your organization does not migrate, your current secrets management method remains in place. However, [secrets management using ZooKeeper](/docs/5/fusion/operations/security/zookeeper) will be deprecated at some point in the future.
# High Availability and Disaster Recovery
Source: https://doc.lucidworks.com/docs/5/fusion/operations/security/ha-dr
[localhost link]: http://localhost:3000/docs/5/fusion/operations/security/ha-dr
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/security/ha-dr
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/tdkc10
## High availability
High system availability is sustained by sufficient levels of redundancy in the processing stack. Factors to consider when determining system availability include:
* Redundant resource costs
* System performance
* System synchronization methods and the level of synchronization each method provides
* Computational costs
* Network latency
Redundancy levels available are:
* **Data Redundancy.** Create multiple copies of the data on a single storage device to increase the probability of at least one copy being available at all times.
* **Storage Redundancy.** Distribute the data across multiple physical storage devices to ensure data is available if one of the physical storage devices is not accessible.
* **Process/Pod Redundancy.** Create multiple copies of each service/pod to reduce the risk of a service becoming unavailable if an individual copy fails for some reason.
* **Process/Pod Distribution.** Distribute duplicate pods across different nodes in a cluster to protect against the loss of service due to a failed node in the cluster.
* **Node Redundancy.** Create multiple nodes in the cluster within a single Availability Zone to protect against the loss of any single node. For this option:
* The level of node over-provisioning is a function of both the probability of a node failing, and the average time required to provision and return a new node to service after a failure.
* Sufficient redundancy means that if one node fails, the remaining nodes have sufficient computing capacity to manage the load while a new node is provisioned.
* **Availability Zone Redundancy.** Distribute resources across multiple zones to allow the system to tolerate the loss of an entire zone without compromising the availability of the system as a whole. While this is an option, Lucidworks does not use this because it does not consider this best practice.
* **Cluster Redundancy.** - Create multiple copies at the cluster level within a single data center to ensure that there is no loss of service even if an entire cluster is lost.
* **Data Center Redundancy.** Distribute clusters across multiple data centers in a region. If availability to a specific data center is lost, the system is still available in other data centers.
* **Region Redundancy.** Distribute clusters across multiple regions, for example, east and west. If a major outage occurs and availability to a specific region is lost, the system is still available in the other regions.
## Query runtime options
The following options describe Fusion configurations.
**Important**
The options are listed in order, from the least highly-available and lower disaster-prepared state, to the optimal highly-available and highest disaster-prepared state.
* **Single cluster located in on-premise client hardware.** This provides the lowest level of availability and disaster recovery if an outage or data corruption occurs.
* **Single cluster deployed in the cloud.** This provides the lowest cloud-based level of availability and disaster recovery if an outage or data corruption occurs.
* **Active-Passive disaster recovery.** Redundant clusters exist, but only one is running at a given time.
* **Green-Blue-Active-Passive disaster recovery.** Redundant clusters exist, but only one is available to service user requests.
* **Active-Active disaster recovery.** Redundant clusters exist that contain identical live infrastructure and all clusters are actively servicing user requests. For more information about the configuration to synchronize indexes between data clusters, see [Active - Active](#active-active).
* **Green-Blue Active-Active disaster recovery.** Redundant clusters exist that contain duplicate Kubernetes-deployed environments and all clusters are actively servicing user requests.
* **Fully Active-Active Traffic Shaped disaster recovery.** - Redundant clusters exist and all clusters are actively servicing user requests, with traffic directed to a specific cluster according to geography or user affinity.
## Fusion High Availability architecture types
Architecture types depend on the number of nodes in a cluster.
| Nodes in cluster | Services on node |
| - | - |
| All services on 3+ nodes | All services run on each node. |
| All services on 2+1 nodes | ● All services run on each node. ● Requires an additional node to run only Zookeeper. |
| Deploy services separately on 10 nodes | ● 3 nodes for only search services (admin-ui, api, Zookeeper) ● 2 nodes for only Solr service (search engine) ● 2 nodes for signals services (Spark master and worker) ● 2 nodes for connectors services |
## Fusion Disaster Recovery deployment architecture types
Disaster Recovery uses independent Fusion clusters on both data centers.
The key tasks are to synchronize:
* data center configurations
* the actual data (indexes)
### Strategy to sync data center configuration
Extract configurations from lower environment and move to higher environment. During this process we ensure configurations apply to high environments for both data centers.
### Setups to sync indexes between data centers
#### Active - Active
1. **Hot - Hot**: The data centers connect to a common node for good network bandwidth.
1. Fusion clusters are on both data centers without Solr.
2. Deploy Solr service on the common node.
3. Both Fusion clusters connect to Solr node to GET/PUT catalog or signals records.
4. Solr index has its own backup and restore procedure.
2. **Hot - Warm:** There is no common data node between the data centers.
1. Deploy Fusion full stack clusters on both data centers.
2. Use Fusion connectors or PBL job to transfer catalog and signals from one datacenter to another.
3. Solr index is already replicated on both data centers.
4. During data transfer, it is possible that data is not fully synchronized.
#### Active - Passive
1. **Hot - Cold**
1. Fusion has 1-2 nodes on the secondary datacenter.
2. Configurations are periodically copied over to the secondary data center.
3. Connectors job is set up to copy indexes.
4. Could also take periodic Solr Active cluster backups and cache them temporarily so the most recent backup is always available to restore into the Cold cluster when it starts.
## Data transfer options
### Fusion Parallel Bulk Loader (PBL) job
For the [Fusion Parallel Bulk Loader (PBL) job](/docs/5/fusion/getting-data-in/parallel-bulk-loader):
* Pros include:
* Easy to configure Fusion Spark job.
* Blazing fast batch transfer with scheduling and monitoring options available.
* Configurable to filter out documents.
* Cons include:
* None, because this option does not need to run Spark services.
### Solr streaming expressions
Here is an example of Solr streaming expressions:
```js theme={"dark"}
daemon(id="uniqueId",
runInterval="5000",
terminate="false",
commit(Target,
batchSize=5000,
update(Target,
batchSize=500,
topic(
Source-Checkpoint-Coll,
Source,
zkHost="localhost:9983/solr",
q="*:*",
fq="mimeType_s:application/pdf",
fl="id,*",
sort="id asc",
qt="/export"
)
)
)
)
```
* Pros include:
* Keeps track of what has been read to date and only sends what is new.
* Open source and easy to configure.
* Does not depend on shard or replica count, and will tolerate differences on either end of the transaction.
* Tolerates source-side outages and picks up where it left off.
* Cons include:
* Can only send stored fields.
* Not parallelizable like Spark jobs.
* Does not tolerate destination-side outages. When the target is not available, data transfers can drop because the source-side high-water-mark may move before the target-side failure is detected.
## Automated Solr backups
You can schedule backups of Solr collections, store the backups for a configurable period of time, and restore these backups into a specified Fusion cluster when needed.
The following guide uses Google Kubernetes Engine (GKE) for examples and assumes you used the setup scripts in the [fusion-cloud-native](https://github.com/lucidworks/fusion-cloud-native) repository to install Fusion.
The standard approach of using a provider-specific Persistent Volume Claim (PVC) for storing collection backups ensures consistency in configuration. However, this method does not leverage the unique storage features offered by each cloud provider. For instance, Google Cloud provides Google Cloud Storage, which includes additional features such as access control management, various storage tiers, and other capabilities that are not available when using a PVC. To take advantage of these features, Solr instances running within Fusion require additional provider-specific information.
Refer to the Solr documentation for detailed information on the [repositories available for configuring collection backups](https://solr.apache.org/guide/solr/latest/deployment-guide/backup-restore.html#backuprestore-storage-repositories). Each repository type comes with specific configuration options and features. Generally, you will need to integrate the provider-specific configuration into the `solr.xml` configuration file and ensure that the appropriate library or module for the provider is included in the Solr classpath. This step is necessary for the repository implementation to resolve correctly at runtime.
For example, when configuring `GCSBackupRepository` to store backups in Google Cloud Storage (GCS), it is essential to include the corresponding library for the provider in the Solr classpath. Additionally, you will need to add a section to the `solr.xml` file similar to the XML example below to specify the target bucket where backups will be stored:
```xml wrap theme={"dark"}
solrBackups/local/path/to/credential/file/default/gcs/backup/location515001.510000
```
After configuring the backup provider, you can utilize the standard [Solr backup and restore APIs](https://solr.apache.org/guide/solr/latest/deployment-guide/backup-restore.html#user-managed-clusters-and-single-node-installations) to create new backups or restore from existing ones. Instead of writing to a PVC, backups will be stored in the storage solution specific to the provider.
Backups are taken using the Solr collection [BACKUP command](https://solr.apache.org/guide/solr/latest/deployment-guide/collection-management.html#backup). This requires that each Solr node has access to a shared volume or a `ReadWriteMany` volume in Kubernetes. Most cloud providers offer a simple way of creating a shared filestore and exposing it as a `PersistentVolumeClaim` within Kubernetes to mount into the Solr pods. An option is added to the **setup\_f5\_PROVIDER.sh** scripts in the **fusion-cloud-native** repository to provision these.
The backup action of the script is invoked by a Kubernetes CronJob to run the backup schedule. The backups are saved to a configurable directory with an automatically generated name: `-`.
A separate CronJob is responsible for cleanup and retention of backups. Cleanup can be disabled if not needed. Setting a series of retention periods can automatically remove backups as they become outdated.
For example, a cluster that backs up a collection every 3 hours could specify a retention policy that:
* Keeps all backups for a single day.
* Keeps a single daily backup for a week.
* Keeps a single weekly backup for a month.
* Keeps a single monthly backup for 6 months.
* Deletes all backups that are older than 6 months.
All times are configurable as part of the `configmap` for this service.
The process for restoring a collection is a manual step involving `kubectl run` to invoke the Solr `RESTORE` action pointing to the collection and the name of the backup being restored. Use the following command, replacing the placeholder values with your environment-specific values:
```bash wrap theme={"dark"}
kubectl run --namespace "${NAMESPACE}" --rm=true -it \
--image=curlimages/curl:latest solr-restore -- \
curl "http://${RELEASE}-solr-headless:8983/solr/admin/collections?action=RESTORE&name=&location=/var/solr/data/solr-backups&collection="
```
Replace the following placeholder values:
* `${NAMESPACE}` with your Kubernetes namespace (for example, `default`)
* `${RELEASE}` with your Helm release name (for example, `f5`)
* `` with the name of the backup directory (for example, `mycollection-2024-01-15T00:00:00`)
* `` with the name of the collection to restore into
The backup directory name is generated automatically at backup time in the format `-`. You can list available backups by running:
```bash wrap theme={"dark"}
kubectl exec --namespace "${NAMESPACE}" "${RELEASE}-solr-0" -- \
ls /var/solr/data/solr-backups
```
These instructions are for GKE only. For other platforms, backup and restoration involves copying the collection to the cloud and using Parallel Bulk Loader.
### Install using a PVC with GKE
The `solr-backup-runner` requires that a `ReadWriteMany` volume is mounted onto all `solr` and `backup-runner` pods so they all back up to the same filesystem.
The easiest way to install on GKE is by using a GCP Filestore as the `ReadWriteMany` volume.
1. Create the Filestore.
```bash wrap theme={"dark"}
gcloud --project "${GCLOUD_PROJECT}" filestore instances create "${NFS_NAME}" --tier=STANDARD --file-share=name="solrbackups,capacity=${SOLR_BACKUP_NFS_GB}GB" --zone="${GCLOUD_ZONE}" --network=name="${network_name}"
```
2. Fetch the IP of the Filestore.
```bash wrap theme={"dark"}
NFS_IP="$(gcloud filestore instances describe "${NFS_NAME}" --project="${GCLOUD_PROJECT}" --zone="${GCLOUD_ZONE}" --format="value(networks.ipAddresses[0])")"
```
3. Create a Persistent Volume in Kubernetes that is backed by this volume.
```yaml theme={"dark"}
cat <
# Ingress and Security
Source: https://doc.lucidworks.com/docs/5/fusion/operations/security/ingress
[localhost link]: http://localhost:3000/docs/5/fusion/operations/security/ingress
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/security/ingress
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/3226
The Fusion proxy service serves as an API gateway and provides authentication and authorization. Route all external access to Fusion services through the Fusion proxy service. The most common approach is to set up a [Kubernetes Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) that routes Fusion service requests to the proxy service as shown in the example ingress definition below. It’s also common to do [TLS termination](https://cloud.google.com/load-balancing/docs/https/#tls_support) at the Ingress, so that all traffic to and from the Kubernetes cluster is encrypted.
Internal requests happen over unencrypted HTTP.
```yaml wrap expandable theme={"dark"}
apiVersion: v1
items:
- apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
...
labels:
...
name: -api-gateway
namespace:
spec:
rules:
- host:
http:
paths:
- backend:
serviceName: proxy
servicePort: 6764
path: "/*"
tls:
- hosts:
-
secretName: -api-gateway-tls-secret
status:
loadBalancer:
ingress:
- ip:
```
## EKS
For deployments that run on EKS, the [setup\_f5\_eks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_eks.sh) setup script creates a classic elastic load balancer (ELB) to expose the Fusion proxy service for demonstration purposes. However, you’re strongly encouraged to configure a Kubernetes Ingress with TLS termination in front of the proxy service. See [How do I terminate HTTPS traffic on Amazon EKS workloads with ACM?](https://aws.amazon.com/premiumsupport/knowledge-center/terminate-https-traffic-eks-acm/) for more information.
You can also opt to use an [Application Load balancer](https://github.com/lucidworks/fusion-cloud-native#eks-ingress) instead of the proxy service.
## AKS
AKS deployments can use the [setup\_f5\_aks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_aks.sh) setup script, which exposes the Fusion proxy service on an external IP using HTTP for demonstration purposes. We recommend that you configure a [Kubernetes Ingress with TLS termination](https://github.com/lucidworks/fusion-cloud-native#aks-ingress) in front of the proxy service.
## Other Kubernetes Platforms
If your deployment runs on OpenShift using a cluster created with OpenShift Infrastructure Provider, the [setup\_f5\_k8s.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_k8s.sh) setup script will install Fusion 5 on your existing cluster. OpenShift Online services are not supported.
See the [RedHat OpenShift](https://github.com/lucidworks/fusion-cloud-native#redhat-openshift) section of the readme for more tips toward a successful installation.
If your deployment runs on GKE, the [setup\_f5\_gke.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_gke.sh) setup script in the `fusion-cloud-native` repo provides the option to create the [Ingress and TLS cert](https://github.com/lucidworks/fusion-cloud-native#gke-ingress-and-tls).
This process uses [Let’s Encrypt](https://letsencrypt.org/).
For other Kubernetes providers, refer to provider’s documentation on creating an Ingress and TLS certificates.
# Mutual Transport Layer Security
Source: https://doc.lucidworks.com/docs/5/fusion/operations/security/mtls-fusion
Mutual Transport Layer Security (mTLS) provides network-layer authentication for Fusion by requiring both the client and server to present valid certificates during the TLS connection process.
This feature ensures that only trusted, certificate-authenticated clients can access Fusion APIs and UI endpoints.
## Overview
Standard TLS encrypts traffic and allows clients to verify the server's identity.
mTLS extends this by adding client authentication, which means the server also verifies the client's identity before allowing the connection.
mTLS complements but does not replace application-layer authentication methods like SSO and OIDC. It provides an additional security layer at the network level that satisfies zero-trust architecture requirements.
The mTLS authentication process is essential if your organization needs to follow any of the following guidelines:
* Operate under zero-trust security frameworks
* Have strict compliance requirements such as SOC2, ISO 27001, or HIPAA
* Restrict API access to specific authenticated services or devices
* Require network-layer security controls in addition to application authentication
* Enforce different trust policies for various environments (development, staging, production)
### Common use cases
Use mTLS to perform the following functions:
* **Enterprise API security**: Restrict Fusion API access to only authenticated services within your infrastructure. Services must present a valid client certificate signed by your organization's certificate authority (CA).
* **Multi-environment access control**: Enforce different certificate requirements for development, staging, and production environments. Development might use self-signed certificates while production requires certificates from your enterprise public key infrastructure (PKI).
* **Compliance and audit**: Meet regulatory requirements for network-layer authentication.
All client certificate usage is logged in Google Cloud Logging for audit purposes.
* **Zero-trust architecture**: Implement defense-in-depth security where every connection must be authenticated at both network and application layers.
### Authentication flow
Fusion implements mTLS at the Google Cloud External Application Load Balancer level. The client certificate validation occurs before traffic reaches the Fusion cluster.
This diagram illustrates the authentication process.
```mermaid theme={"dark"}
sequenceDiagram
accTitle: mTLS authentication flow
accDescr: A sequence diagram showing the mutual TLS handshake between a client, global application load balancer, GKE cluster, and Lucidworks search services
participant C as Client
participant LB as Global Application Load Balancer
participant GKE as GKE Cluster
participant LWS as Lucidworks Search Services
C->>LB: ClientHello
LB-->>C: ServerHello
LB-->>C: Server Certificate
Note over C: Verify Server Certificate
C->>LB: Client Key Exchange
LB-->>C: Finished
C->>LB: Finished
Note over C,LWS: Encrypted Data Exchange
C->>LB: TLS Encrypted Request
LB->>GKE: Authenticated HTTPS
GKE->>LWS: Route to Lucidworks Search services
LWS-->>C: Response via TLS Channel
```
## Prerequisites
This section describes the prerequisites to enable mTLS.
### Configure certificate infrastructure
Configure the following certificate aspects:
* **Certificate Authority (CA)**: Establish access to a trusted CA for signing client certificates
* Option 1: Use your organization's existing PKI infrastructure
* Option 2: Use Google Cloud Certificate Authority Service (platform-managed)
* **CA Certificate**: Configure a privacy enhanced mail (PEM) encoded root or intermediate CA certificate to use as a trust anchor
* **Client Certificates**: Set up the ability to generate and distribute `PKCS#12` (`.p12`) or PEM certificates to clients
### Configure access and permissions
Grant the following access and permissions abilities:
* Administrative access to the applicable Fusion environment
* Ability to upload CA certificates to Google Cloud
* Permissions to configure load balancer settings
### Client capabilities
Ensure the following capabilities are in place:
* Client configuration must support TLS 1.2 or later
* Client systems must be able to load and present client certificates
* For browser access, users need certificates imported to their browser's certificate store
* For API access, applications need certificate configuration in their HTTP client
## Configuration
This section contains the information to configure mTLS.
### Configuration steps
These are the high-level steps to configure mTLS.
1. **Provide CA Certificate**: Supply your organization's CA certificate (PEM format) as a trust anchor.
2. **Configure TrustConfig**: Load balancer TrustConfig is updated with your CA certificate.
3. **Enable mTLS Policy**: Server TLS policy is set to `MTLS` mode with `REJECT_INVALID` validation.
4. **Testing**: Verify mTLS enforcement in a non-production environment first.
5. **Certificate Distribution**: Distribute client certificates to your users and services.
6. **Production Rollout**: Enable mTLS for production after successful testing.
### Trust configuration options
**Customer-managed certificate authority**
Use your organization's existing certificate authority.
You provide the CA certificate and manage client certificate issuance.
This is the most common option for enterprises with established PKI.
Plan certificate distribution carefully before enabling mTLS. All clients must have valid certificates before enforcement begins, or they will be unable to connect.
## Using mTLS
### Browser access
For users accessing the Fusion UI through a web browser:
1. **Import certificate**: Import the `PKCS#12` (`.p12`) certificate into your browser's certificate store. These are the access paths for the specified browser:
* Chrome/Edge: **Settings > Privacy and security > Security > Manage certificates**.
* Firefox: **Settings > Privacy & Security > Certificates > View Certificates > Your Certificates > Import**.
* Safari: Use the **macOS Keychain Access** to import the certificate.
2. **Access Fusion**: Navigate to your Fusion URL.
3. **Certificate selection**: Select the certificate provided by your administrator. If prompted, confirm the certificate usage.
4. **Authenticated access**: When the certificate is validated, access Fusion.
### API access
Use one of the following commands to configure applications and services making API requests to Fusion:
```bash theme={"dark"}
# Use PEM certificate and key
curl -X GET \
--cert CLIENT_CERT.pem \
--key CLIENT_KEY.pem \
https://YOUR_INSTANCE.lucidworks.cloud/api/apps
# Use PKCS#12 certificate
curl -X GET \
--cert-type P12 \
--cert CLIENT_CERT.p12:PASSWORD \
https://YOUR_INSTANCE.lucidworks.cloud/api/apps
```
```python theme={"dark"}
import requests
# Use PEM certificate and key
response = requests.get(
'https://YOUR_INSTANCE.lucidworks.cloud/api/apps',
cert=('CLIENT_CERT.pem', 'CLIENT_KEY.pem')
)
# Use PKCS#12 certificate (requires requests_pkcs12)
from requests_pkcs12 import get
response = get(
'https://YOUR_INSTANCE.lucidworks.cloud/api/apps',
pkcs12_filename='CLIENT_CERT.p12',
pkcs12_password='PASSWORD'
)
```
```java theme={"dark"}
import javax.net.ssl.*;
import java.security.KeyStore;
import java.io.FileInputStream;
// Load PKCS#12 keystore
KeyStore keyStore = KeyStore.getInstance("PKCS12");
keyStore.load(new FileInputStream("CLIENT_CERT.p12"),
"password".toCharArray());
// Create KeyManagerFactory
KeyManagerFactory kmf = KeyManagerFactory.getInstance(
KeyManagerFactory.getDefaultAlgorithm());
kmf.init(keyStore, "password".toCharArray());
// Create SSLContext
SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(kmf.getKeyManagers(), null, null);
// Use with HttpsURLConnection or HTTP client
HttpsURLConnection.setDefaultSSLSocketFactory(
sslContext.getSocketFactory());
```
```javascript theme={"dark"}
const https = require('https');
const fs = require('fs');
const options = {
hostname: 'YOUR_INSTANCE.lucidworks.cloud',
path: '/api/apps',
method: 'GET',
cert: fs.readFileSync('CLIENT_CERT.pem'),
key: fs.readFileSync('CLIENT_KEY.pem'),
ca: fs.readFileSync('CA_CERT.pem') // optional
};
const req = https.request(options, (res) => {
console.log('statusCode:', res.statusCode);
res.on('data', (d) => {
process.stdout.write(d);
});
});
req.end();
```
### Certificate management
Perform the following actions to manage certificates:
**Certificate expiration**: Monitor certificate expiration dates and plan for renewal well in advance. Expired certificates will cause connection failures.
**Certificate rotation**: Perform the following actions to rotate certificates without downtime:
1. Generate new certificates from the same CA.
2. Distribute new certificates to clients.
3. Clients update to use new certificates.
Old certificates remain valid during transition period.
4. Remove old certificates after all clients have migrated.
**Revocation**: Perform the following actions if a certificate is compromised:
1. Revoke the certificate by updating the `TrustConfig`.
2. Issue replacement certificates from the CA.
3. Update all affected clients.
## Security considerations
mTLS is one layer of security. Perform the following actions to maintain multiple security controls:
* **Application authentication**: Continue using SSO, OIDC, or JWT tokens.
* **Authorization**: Enforce role-based access control (RBAC).
* **Network security**: Use IP `allowlisting` where appropriate.
* **Data encryption**: Confirm all traffic is encrypted via TLS 1.2+.
### Certificate storage
Follow these guidelines for certificate storage:
* **Protect private keys**
* Never commit certificates or private keys to source control.
* Store certificates in secure key management systems such as HashiCorp Vault, AWS Secrets Manager, and others.
* Use environment variables or secure configuration files for deployment.
* Restrict file system permissions on certificate files such as `chmod 600`.
* **Environment recommendations**
Follow these guidelines to secure different environments:
* Use separate certificate authorities or intermediate CAs for different environments.
* Production certificates should have stricter issuance policies.
* Development/testing can use self-signed certificates if approved by security team.
### Monitoring and logging
This section provides information and guidelines about mTLS event monitoring and logs:
All mTLS authentication events are logged in Google Cloud Logging. The events include the following instances:
* Successful certificate validations
* Failed authentication attempts
* Certificate validation errors
Configure alerts for the following events:
* Unusual patterns in failed authentication attempts
* Upcoming certificate expirations
* Certificate validation errors
## Troubleshooting
This section provides potential issues and their corresponding resolutions.
### Connection refused or certificate errors
**Symptom**: Client receives connection refused or certificate validation error
**Common causes**:
* Client certificate not signed by trusted CA
* Certificate expired
* Certificate not properly configured in client
* Wrong certificate selected (browser)
* Certificate format mismatch (PEM vs PKCS#12)
**Resolution**:
1. Verify the certificate is signed by the CA configured in `TrustConfig`.
2. Check the certificate expiration date: `openssl x509 -in client-cert.pem -noout -dates`.
3. Verify the certificate chain is complete: `openssl verify -CAfile ca-cert.pem client-cert.pem`.
4. Test the certificate with curl: `curl -v --cert CLIENT_CERT.pem --key CLIENT_KEY.pem https://YOUR_INSTANCE.lucidworks.cloud`.
### Browser not prompting for certificate
**Symptom**: Browser doesn't show certificate selection dialog
**Common causes**:
* Certificate not imported to browser certificate store
* Certificate imported to wrong store, for example, to the CA certificates instead of the client certificates
* Browser cached a previous certificate selection
**Resolution**:
1. Verify the certificate is in browser's personal/client certificate store.
2. Clear the browser cache and cookies.
3. Try a private or incognito window.
4. Re-import the certificate, if necessary.
### Certificate works in testing but not production
**Symptom**: Certificate validates in non-production but fails in production
**Common causes**:
* Different CA certificates configured per environment
* Certificate signed by wrong CA
* `TrustConfig` not updated in production
**Resolution**:
1. Verify the same CA certificate is configured in both environments.
2. Check the certificate's issuer: `openssl x509 -in CLIENT_CERT.pem -noout -issuer`.
3. Verify the `TrustConfig` settings.
### Application cannot load certificate
**Symptom**: Application error loading certificate file
**Common causes**:
* Incorrect file path
* Insufficient file permissions
* Certificate format not supported by library
* Missing intermediate certificates
**Resolution**:
1. Verify the file exists and is readable: `ls -la CLIENT_CERT.pem`.
2. Check the certificate's format: `openssl x509 -in CLIENT_CERT.pem -noout -text`.
3. For PKCS#12, verify the password is correct.
4. Ensure the certificate chain includes all intermediate certificates.
5. Convert between formats if needed:
* PEM to PKCS#12: `openssl pkcs12 -export -out cert.p12 -inkey key.pem -in cert.pem`
* PKCS#12 to PEM: `openssl pkcs12 -in cert.p12 -out cert.pem -nodes`
## Frequently asked questions
This section contains commonly-asked questions and their corresponding answers.
**Does mTLS replace application authentication?**
No. mTLS provides network-layer authentication but does not replace application-layer authentication methods like SSO, OIDC, or API tokens. Both layers should be used for defense in depth.
**Can I use mTLS with IP `allowlisting`?**
Yes. mTLS and IP `allowlisting` are complementary security controls. You can use both simultaneously for enhanced security.
**What happens if a certificate expires during business hours?**
Clients with expired certificates will be unable to connect. Monitor certificate expiration dates and renew well in advance. Most organizations set up automated alerts 30-60 days before expiration.
**Can I use different CAs for different groups of clients?**
Yes. The `TrustConfig` can include multiple CA certificates as trust anchors. This allows different departments or environments to use separate CAs while accessing the same Fusion instance.
**How do I handle certificate distribution to multiple users?**
Organizations typically use the following devices, applications, and features:
* Mobile Device Management (MDM) systems for employee devices
* Configuration management tools for servers such as Ansible, Puppet, and Chef
* Secure file sharing with password-protected archives
* Hardware security modules (HSMs) or smart cards for high-security environments
**Does mTLS impact performance?**
The mTLS connection process adds minimal latency, which is typically measured in sub-milliseconds. After the initial connection, there is no performance impact on data transfer. The mTLS session resumption can further reduce connection overhead for repeat connections.
**Can I test mTLS before enabling in production?**
Yes. Lucidworks strongly recommends testing mTLS in a non-production environment first. Configure mTLS in a staging environment before production rollout.
## Related topics
For more information, see the following topics:
* [Security Overview](/docs/5/fusion/operations/security/overview) for comprehensive security features
* [Access Control](/docs/5/fusion/operations/security/access-control/overview) for user authentication and authorization
* [Security Realms](/docs/5/fusion/operations/security/access-control/security-realms) for authentication providers
* [Monitoring](/docs/5/fusion/operations/monitoring-and-reporting/overview) for monitoring and alerting
# Security
Source: https://doc.lucidworks.com/docs/5/fusion/operations/security/overview
[localhost link]: http://localhost:3000/docs/5/fusion/operations/security/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/security/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/122
**Important**
Lucidworks recommends *not* virus scanning the `fusion/data` folder. Virus scanning can cause slow performance, and it can cause downtime if it quarantines an index file identified as a possible virus.
Fusion uses a number of security measures:
* **Authenticating UI users** – Fusion authenticates users when they log in. Logging in creates a new Fusion session. Fusion also authenticates users when the [Sessions](/api-reference/system-status/get-the-current-user-session) REST API creates a session.
* **Authorizing UI users** – Fusion authorizes users to use specific parts of the Fusion UI.
UI users must also be authorized to make API requests, because the UI makes API requests.
* **Authenticating and authorizing users who make API requests**
* **Password Encryption**- Fusion uses 128-bit AES keys to encrypt passwords and "AES/CBC/PKCS7Padding" for the cipher. The ciphertext is also signed.
* **Using session cookies**
* **Using an external authentication provider** (optional) - A security realm can specify use of an external authentication provider, such as LDAP, JWT, or SAML.
* **Using SSL/TLS to ensure that data in flight between your application and Fusion is not observable** (optional) - See **SSL Security (Unix)** or **SSL Security (Windows)**. For more information, see [Transport Layer Security](/docs/5/fusion/reference/microservices#transport-layer-security-tls).
* **Authenticating using Mutual Transport Layer Security (mTLS)** - In Fusion 5.9.15 and later, mTLS provides network-layer authentication using the third-party provider that handles the endpoint. mTLS requires both the client and server to present valid certificates during the TLS connection process. mTLS ensures that only trusted, certificate-authenticated clients can access Fusion APIs and UI endpoints. For more information, see [Mutual Transport Layer Security](/docs/5/fusion/operations/security/mtls-fusion).
* **Constraining the documents that are indexed** (optional)
* **Trimming the documents that are returned by queries based on authorization** (optional)
Fusion’s UI (which is accessed through the Fusion Proxy service) can run over SSL for secure communication with any HTTP client, using the
[Java Secure Socket Extension (JSSE)](http://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html) framework. You configure Fusion for SSL by configuring Fusion’s Proxy and UI services.
Fusion 4.0.2 or later is required to enable SSL security. Version 4.0.2 fixed a bug in the authentication proxy that incorrectly redirected some requests when SSL is enabled.
## Required software
Configuring Fusion for SSL requires the following software:
* **Java Development Kit.** To store certificates, you can use the Java [keytool Key and Certificate Management utility](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html) which is a part of the JDK.
The JDK is also [a requirement for Fusion Server](/docs/4/fusion-server/reference/system-requirements).
* [**OpenSSL**](https://www.openssl.org/). You might need the `openssl` command line tool:
* If you have the certificate chain and private key as separate files, then you can use the `openssl` command line tool to create a PKCS #12 file.
* If you have an intermediate CA certificate, then you can use it and `openssl` to generate the certificate chain and private key files.
## Load an SSL certificate into a Fusion keystore
The SSL protocol is based on public-key cryptography where encryption keys come in public key/private key pairs. An SSL certificate is used to verify the authenticity of a particular server. It contains the web site name, contact email address, company information and the public key used to encrypt the communication which is shared with the entities that communicate with the owner of the public/private key pair.
The server has a locally-protected private key that is accessible using a
[JSEE keystore](http://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#SSLOverview).
The keystore maintains both the server certificate and the private key, so that when a server authenticates itself to the client, it uses the private key from its keystore for the initial SSL handshake.
Load the certificate into a Fusion keystore. Perform the tasks in the appropriate section:
* [**Self-signed certificate.**](#alternative-1-self-signed-certificate) If Fusion is behind a firewall, you can use a self-signed certificate for SSL communication with other hosts in your internal network. Create a keystore for the Fusion Proxy service and load the keystore with the self-signed PKCS #12 certificate.
* [**Certificate signed by a certificate authority.**](#alternative-2-ca-signed-certificate) In a production environment, SSL certificates typically originate with certificate signing requests (CSRs) and are signed by a trusted third-party [Certificate Authority (CA)](http://en.wikipedia.org/wiki/Certificate_authority). Create a keystore for the Fusion Proxy service and load the keystore with the PKCS #12 certificate from a CA.
### Alternative 1: Self-signed certificate
If you are using a CSR-originated certificate from a trusted certificate authority, proceed to [Alternative 2: CA-signed certificate](#alternative-2-ca-signed-certificate).
If Fusion is behind a firewall, you can use a self-signed certificate for SSL communication with other hosts in your internal network. Create a keystore for the Fusion Proxy service and load the keystore with the self-signed PKCS #12 certificate.
To store certificates, you can use the Java [keytool](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html)
Key and Certificate Management utility which is a part of the JDK.
**How to create a keystore and load a self-signed certificate**
1. Set environment variables:
```bash theme={"dark"}
export JAVA_HOME=JavaHomeDirectory
export FUSION_HOME=FusionHomeDirectory
```
For example, in Fusion 4.1:
```bash theme={"dark"}
export JAVA_HOME=/usr/java/jdk1.8.0_171
export FUSION_HOME=/opt/lucidworks/fusion/4.1.2
```
In Fusion 4.0.x:
```bash theme={"dark"}
export JAVA_HOME=/usr/java/jdk1.8.0_171
export FUSION_HOME=/opt/lucidworks/fusion/4.0.2
```
2. Create the Fusion Proxy service keystore, generate the key pair and self-signed certificate, and load them into the keystore:
```bash theme={"dark"}
"$JAVA_HOME/bin/keytool" -genkeypair -keystore "$FUSION_HOME/apps/jetty/proxy/etc/keystore" -dname "CN=CommonName, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown" -keypass KeyPassword -storepass KeystorePassword -keyalg RSA -alias selfsigned -deststoretype pkcs12 -ext SAN=dns:ServerFqdn,ip:ServerIpAddress
```
You must include the qualified domain name and/or the IP address of the Fusion server in the `-ext SAN` part of the command. Failure to do so results in SSL validation errors.
Example command:
```bash theme={"dark"}
"$JAVA_HOME/bin/keytool" -genkeypair -keystore "$FUSION_HOME/apps/jetty/proxy/etc/keystore" -dname "CN=search.mycorp, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown" -keypass 59Winter.Is.Long45 -storepass 46I.Prefer.Vanilla24 -keyalg RSA -alias selfsigned -deststoretype pkcs12 -ext SAN=dns:search.mycorp,ip:192.168.1.40,dns:localhost
```
The resulting certificate enables validated SSL transport to these hosts:
```bash theme={"dark"}
https://search.mycorp:ProxyPort
https://192.168.1.40:ProxyPort
https://localhost:ProxyPort
```
Where `ProxyPort` is the Fusion Proxy port.
### Alternative 2: CA-signed certificate
If Fusion is behind a firewall and you are using a self-signed certificate, skip this section. Perform the tasks in [Alternative 1: Self-signed certificate](#alternative-1-self-signed-certificate).
In a production environment, SSL certificates typically originate with certificate signing requests (CSRs) and are signed by a trusted third-party [Certificate Authority (CA)](http://en.wikipedia.org/wiki/Certificate_authority).
The steps here assume that you are the person who will be obtaining the SSL certificate chain and private key files. If you are not that person, contact your system administrator.
#### Preliminary steps
1. Obtain a domain from a domain registrar.
2. Change the A record of your domain to the public IP address of your web server instance.
#### Generate SSL certificate files
Use an SSL certificate provider to generate the certificate chain and private key files, or a PKCS #12 certificate, from a trusted CA:
* **Certificate chain and private key files.** In this case, you will need to convert these files into a single certificate file in PKCS #12 format.
* **A PKCS #12 certificate.** This must contain both the certificate chain and private key. In this case, no conversion is necessary.
**How to obtain certificate files**
1. In most cases, you will need to temporarily open ports 80 and 443 in your firewall configuration. The SSL certificate provider must be able to make successful HTTP and HTTPS requests to your server through the Domain Name System (DNS).
2. Use an SSL certificate provider to generate the certificate chain (`fullchain.pem`) and private key (`privkey.pem`) files, or the PKCS #12 certificate, from a trusted CA. Steps will vary based on the certificate provider. Contact your certificate provider for details.
3. Close ports 80 and 443 in your firewall configuration.
4. Change the A record of your domain to the public domain-name address of your web server instance.
Proceed to the next sections as follows:
* If you have certificate chain and private key files, perform the steps in [Convert the certificate chain and private key files to a PKCS #12 certificate](#convert-the-certificate-chain-and-private-key-files-to-a-pkcsand-160-12-certificate) and [Import the PKCS #12 certificate into the Fusion Proxy service keystore](#create-the-fusion-proxy-service-keystore-and-import-the-pkcsand-160-12-certificate).
* If you have a PKCS #12 certificate, perform the steps in [Import the PKCS #12 certificate into the Fusion Proxy service keystore](#create-the-fusion-proxy-service-keystore-and-import-the-pkcsand-160-12-certificate).
#### Convert the certificate chain and private key files to a PKCS #12 certificate
If you have a PKCS #12 certificate, skip this section and proceed to the section [Import the PKCS #12 certificate into the Fusion Proxy service keystore](#create-the-fusion-proxy-service-keystore-and-import-the-pkcsand-160-12-certificate).
With the certificate chain and private key as separate files, use the `openssl` command line tool in [OpenSSL](https://www.openssl.org/) to create a PKCS #12 certificate.
```bash theme={"dark"}
openssl pkcs12 -export -out /path/to/keystore.p12 -in /path/to/fullchain.pem -inkey /path/to/privkey.pem
```
Do not enter a blank password.
#### Create the Fusion Proxy service keystore and import the PKCS #12 certificate
Use the Java [keytool](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html)
Key and Certificate Management utility to create a keystore for the Fusion Proxy service (`$FUSION_HOME/apps/jetty/proxy/etc/keystore`) and import the PKCS #12 certificate file. Fusion uses this certificate to perform SSL.
1. To create the keystore and import the PKCS #12 certificate:
2. Use the keytool `import` command to create a JSSE keystore.
```bash theme={"dark"}
keytool -importkeystore -srckeystore /path/to/keystore.p12 -srcstoretype PKCS12 -destkeystore "$FUSION_HOME/apps/jetty/proxy/etc/keystore" -deststoretype PKCS12
```
3. (Optional) If desired, delete the PKCS #12 certificate file that resides outside of the Fusion Proxy service keystore (the one you created from the certificate chain and private key files, or obtained from a trusted CA.
```bash theme={"dark"}
rm /path/to/keystore.p12
```
## Enable HTTPS in the Fusion Proxy service
Before beginning these steps, [load an SSL certificate into a Fusion keystore](#alternative-1-self-signed-certificate).
**How to enable HTTPS in the Fusion Proxy service:**
1. (*Only for Fusion Server 4.0.x and 4.1.0*) Prevent the `start.jar` program from downloading a default keystore file, which is not needed. Edit `$FUSION_HOME/apps/jetty/home/modules/ssl.mod`. Comment out the indicated line using `#`. Change:
```bash theme={"dark"}
[files]
https://raw.githubusercontent.com/eclipse/jetty.project/master/jetty-server/src/test/config/etc/keystore?id=${jetty.tag.version}|etc/keystore
```
To:
```bash theme={"dark"}
[files]
# https://raw.githubusercontent.com/eclipse/jetty.project/master/jetty-server/src/test/config/etc/keystore?id=${jetty.tag.version}|etc/keystore
```
2. Set environment variables:
```bash theme={"dark"}
export JAVA_HOME=JavaHomeDirectory
export FUSION_HOME=FusionHomeDirectory
```
For example, in Fusion 4.1:
```bash theme={"dark"}
export JAVA_HOME=/usr/java/jdk1.8.0_171
export FUSION_HOME=/opt/lucidworks/fusion/4.1.2
```
In Fusion 4.0:
```bash theme={"dark"}
export JAVA_HOME=/usr/java/jdk1.8.0_171
export FUSION_HOME=/opt/lucidworks/fusion/4.0.2
```
3. Add HTTPS protocol support to the Jetty TLS (SSL) connector:
```bash theme={"dark"}
cd "$FUSION_HOME/apps/jetty/proxy/"
java -jar "$FUSION_HOME/apps/jetty/home/start.jar" --add-to-start=https
```
Example output:
```bash theme={"dark"}
INFO: ssl initialized (transitively) in ${jetty.base}/start.ini
INFO: https initialized in ${jetty.base}/start.ini
INFO: Base directory was modified
```
4. Get the obfuscated version of your keystore password:
```bash theme={"dark"}
java -cp "$FUSION_HOME/apps/jetty/home/lib/*" org.eclipse.jetty.util.security.Password PASSWORD
```
Replace `PASSWORD` with the password you used for the keystore. If the password contains special characters, [URL encode them](https://www.w3schools.com/tags/ref_urlencode.asp).
Example output:
```bash theme={"dark"}
2018-05-15 12:32:48.988:INFO::main: Logging initialized @133ms
password345XYZ
OBF:1vn21ugu1saj1v9i1v941sar1ugw1vo0
MD5:b91cd1a54781790beaa2baf741fa6789
```
5. Edit the file `$FUSION_HOME/apps/jetty/proxy/start.ini`:
1. Include obfuscated passwords by adding these properties to the end of the file:
```bash theme={"dark"}
* `jetty.sslContext.keyStorePassword`
* `jetty.sslContext.keyManagerPassword`
* `jetty.sslContext.trustStorePassword`
```
2. Use the OBF-encrypted password from step 4 (including the `OBF:` string) as the value for all three of the properties.\
For example:
```bash theme={"dark"}
## Keystore password
jetty.sslContext.keyStorePassword=OBF:2uha1vgt1jg01a4b1a4j1jda1vg11ugg
...
## KeyManager password
jetty.sslContext.keyManagerPassword=OBF:2uha1vgt1jg01a4b1a4j1jda1vg11ugg
## Truststore password
jetty.sslContext.trustStorePassword=OBF:2uha1vgt1jg01a4b1a4j1jda1vg11ugg
```
3. Set the local SSL port by adding the `jetty.ssl.port` property to the end of the file, and providing the port number. For example:
```bash theme={"dark"}
## Connector port to listen on
jetty.ssl.port=8443
```
4. Save the file `$FUSION_HOME/apps/jetty/proxy/start.ini`.
## Restart Fusion and test access through HTTPS
1. Restart all Fusion services:
```bash theme={"dark"}
./bin/fusion restart
```
HTTPS should now be enabled in the Fusion Proxy service.
2. Sign in to the Fusion UI. Specify the HTTPS URL scheme and SSL port, for example, `https://search.mycorp:8443`.
## Disable HTTP access to the Fusion Proxy service
Disable HTTP access. You have a choice. Perform the tasks in the appropriate section:
* [**Disable HTTP access on the firewall or load balancer**](#alternative-1-disable-http-access-on-the-firewall-or-load-balancer) - *This is the preferred approach.*
* [**Disable listening for HTTP requests in the Fusion Proxy service**](#disable-http-access-to-the-fusion-proxy-service)
### Alternative 1: Disable HTTP access on the firewall or load balancer
Disable HTTP access to the Fusion Proxy service on the firewall or load balancer:
1. Disallow all requests for port 8764 from the outside world. Only `localhost` should be able to communicate with Fusion on the non-SSL port 8764. Block all other requestors.
2. If you are using a firewall or load balancer in front of Fusion, use it to redirect all HTTP requests to use HTTPS instead. For example, Apache would [redirect all incoming HTTP traffic to HTTPS](https://httpd.apache.org/docs/2.4/rewrite/avoid.html).
### Alternative 2: Disable listening for HTTP requests in the Fusion Proxy service
Ideally, you should [disable HTTP access using the firewall or load balancer](#alternative-1-disable-http-access-on-the-firewall-or-load-balancer). Follow the steps in this section *only* if disabling HTTP access on the firewall or load balancer is not feasible.
You can only use this alternative if your SSL certificate covers a hostname that can be accessed from the local host. For example, if your certificate only covers `https://fusion.com`, then your local machine must be able to access Fusion from that exact host. If necessary, change the `hosts` file so that this can work.
**How to disable HTTP**
1. Edit `/opt/lucidworks/fusion/latest.x/apps/jetty/proxy/start.d/http.ini`.
1. Change this line:
```bash theme={"dark"}
--module=http
```
To:
```bash theme={"dark"}
#--module=http
```
2. Save the file.
2. Edit the Fusion configuration file, `/opt/lucidworks/fusion/latest.x/conf/fusion.cors` (`fusion.properties` in Fusion 4.x).
1. Ensure that the Agent JVM uses the Fusion Proxy service’s keystore by adding this to the end of the file:
```bash theme={"dark"}
agent.jvmOptions=-Djavax.net.ssl.trustStore="${FUSION_HOME}/apps/jetty/proxy/etc/keystore" -Djavax.net.ssl.trustStorePassword=PASSWORD -Djavax.net.ssl.keyStore="${FUSION_HOME}/apps/jetty/proxy/etc/keystore" -Djavax.net.ssl.keyStorePassword=PASSWORD
```
Replace `PASSWORD` with your Fusion keystore password.
2. Uncomment the `default.address` and change it to the hostname of the server that is validated by your SSL certificate.\
If the hostname saved in `default.address` is not validated by your SSL certificate, then the Fusion Proxy service will not start, because the agent’s liveness detector will not be able to access the HTTPS port to determine whether Fusion is running.
If you self-signed the certificate, then the `default.address` must match the hostname you specified while signing the certificate. Failure to do this will result in the Fusion Proxy service not starting after you have disabled HTTP.
For example, if your SSL certificate’s validated hostname is `search.mycorp`, then change:
```bash theme={"dark"}
#default.address = 127.0.0.1
```
To:
```bash theme={"dark"}
default.address = search.mycorp
```
3. Change the `proxy.port` to the SSL port you chose.
4. Uncomment `proxy.ssl` and change its value to `true`. Change:
```bash theme={"dark"}
# proxy.ssl=false
```
To:
```bash theme={"dark"}
proxy.ssl=true
```
Stopping and starting Fusion is needed for these changes to take effect. But there is a bit more to do concerning the Fusion Agent before restarting Fusion.
## References and tutorials
* [Transport Layer Security](http://en.wikipedia.org/wiki/Transport_Layer_Security) (Wikipedia)
* [Public Key Certificate](http://en.wikipedia.org/wiki/Public_key_certificate) (Wikipedia)
* [OpenSSL Cookbook](https://www.feistyduck.com/library/openssl-cookbook/online/) (free ebook)
* [OpenSSL Command Line Utilities](https://wiki.openssl.org/index.php/Command_Line_Utilities) (OpenSSL wiki)
* [Java Tutorials: Generating and Verifying Certificates](https://docs.oracle.com/javase/tutorial/security/apisign/index.html)
* [IBM developerWorks: What is the JSSE all about?](http://www.ibm.com/developerworks/websphere/techjournal/0502_benantar/0502_benantar.html)
Fusion’s UI (which is accessed through the Fusion Proxy service) can run over SSL for secure communication with any HTTP client, using the
[Java Secure Socket Extension (JSSE)](http://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html) framework. You configure Fusion for SSL by configuring Fusion’s Proxy and UI services.
Fusion 4.0.2 or later is required to enable SSL security. Version 4.0.2 fixed a bug in the authentication proxy that incorrectly redirected some requests when SSL is enabled.
## Required software
Configuring Fusion for SSL requires the following software:
* **Java Development Kit.** To store certificates, you can use the Java [keytool Key and Certificate Management utility](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/keytool.html) which is a part of the JDK.
The JDK is also [a requirement for Fusion Server](/docs/4/fusion-server/reference/system-requirements).
* [**OpenSSL**](https://www.openssl.org/). You might need the `openssl` command line tool:
* If you have the certificate chain and private key as separate files, then you can use the `openssl` command line tool to create a PKCS #12 file.
* If you have an intermediate CA certificate, then you can use it and `openssl` to generate the certificate chain and private key files.
## Overview of procedure
**How to configure Fusion for SSL:**
1. [Load an SSL certificate into a Fusion keystore](#alternative-1-self-signed-certificate).
2. [Enable SSL in the Fusion Proxy service](#generate-ssl-certificate-files).
3. [Restart Fusion and test access through HTTPS](#restart-fusion-and-test-access-through-https).
4. [Disable HTTP access to the Fusion Proxy service](#disable-http-access-to-the-fusion-proxy-service).
## Load an SSL certificate into a Fusion keystore
The SSL protocol is based on public-key cryptography where encryption keys come in public key/private key pairs. An SSL certificate is used to verify the authenticity of a particular server. It contains the web site name, contact email address, company information and the public key used to encrypt the communication which is shared with the entities that communicate with the owner of the public/private key pair.
The server has a locally-protected private key that is accessible using a
[JSEE keystore](http://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html#SSLOverview).
The keystore maintains both the server certificate and the private key, so that when a server authenticates itself to the client, it uses the private key from its keystore for the initial SSL handshake.
Load the certificate into a Fusion keystore. Perform the tasks in the appropriate section:
* [**Self-signed certificate.**](#alternative-1-self-signed-certificate) If Fusion is behind a firewall, you can use a self-signed certificate for SSL communication with other hosts in your internal network. Create a keystore for the Fusion Proxy service and load the keystore with the self-signed PKCS #12 certificate.
* [**Certificate signed by a certificate authority.**](#alternative-2-ca-signed-certificate) In a production environment, SSL certificates typically originate with certificate signing requests (CSRs) and are signed by a trusted third-party [Certificate Authority (CA)](http://en.wikipedia.org/wiki/Certificate_authority). Create a keystore for the Fusion Proxy service and load the keystore with the PKCS #12 certificate from a CA.
### Alternative 1: Self-signed certificate
If you are using a CSR-originated certificate from a trusted certificate authority, proceed to [Alternative 2: CA-signed certificate](#alternative-2-ca-signed-certificate).
If Fusion is behind a firewall, you can use a self-signed certificate for SSL communication with other hosts in your internal network. Create a keystore for the Fusion Proxy service and load the keystore with the self-signed PKCS #12 certificate.
To store certificates, you can use the Java [keytool](https://docs.oracle.com/javase/8/docs/technotes/tools/windows/keytool.html)
Key and Certificate Management utility which is a part of the JDK.
**How to create a keystore and load a self-signed certificate**
1. Set environment variables:
```bash theme={"dark"}
set JAVA_HOME=JavaHomeDirectory
set FUSION_HOME=FusionHomeDirectory
```
2. Create the Fusion Proxy service keystore, generate the key pair and self-signed certificate, and load them into the keystore:
```bash theme={"dark"}
"%JAVA_HOME%\bin\keytool.exe" -genkeypair -keystore "%FUSION_HOME%\apps\jetty\proxy\etc\keystore" -dname "CN=CommonName, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown" -keypass KeyPassword -storepass KeystorePassword -keyalg RSA -alias selfsigned -deststoretype pkcs12 -ext SAN=dns:ServerFqdn,ip:ServerIpAddress
```
You must include the qualified domain name and/or the IP address of the Fusion server in the `-ext SAN` part of the command. Failure to do so results in SSL validation errors.
Example command:
```bash theme={"dark"}
"%JAVA_HOME%\bin\keytool.exe" -genkeypair -keystore "%FUSION_HOME%\apps\jetty\proxy\etc\keystore" -dname "CN=CommonName, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown" -keypass 59Winter.Is.Long45 -storepass 46I.Prefer.Vanilla24 -keyalg RSA -alias selfsigned -deststoretype pkcs12 -ext SAN=dns:search.mycorp,ip:192.168.1.40,dns:localhost
```
The resulting certificate enables validated SSL transport to these hosts:
```bash theme={"dark"}
https://search.mycorp:ProxyPort
https://192.168.1.40:ProxyPort
https://localhost:ProxyPort
```
Where `ProxyPort` is the Fusion Proxy port.
### Alternative 2: CA-signed certificate
If Fusion is behind a firewall and you are using a self-signed certificate, skip this section. Perform the tasks in [Alternative 1: Self-signed certificate](#alternative-1-self-signed-certificate).
In a production environment, SSL certificates typically originate with certificate signing requests (CSRs) and are signed by a trusted third-party [Certificate Authority (CA)](http://en.wikipedia.org/wiki/Certificate_authority).
The steps here assume that you are the person who will be obtaining the SSL certificate chain and private key files. If you are not that person, contact your system administrator.
#### Preliminary steps
1. Obtain a domain from a domain registrar.
2. Change the A record of your domain to the public IP address of your web server instance.
#### Generate SSL certificate files
Use an SSL certificate provider to generate the certificate chain and private key files, or a PKCS #12 certificate, from a trusted CA:
* **Certificate chain and private key files.** In this case, you will need to convert these files into a single certificate file in PKCS #12 format.
* **A PKCS #12 certificate that contains both the certificate chain and private key.** In this case, no conversion is necessary.
**How to obtain certificate files**
1. In most cases, you will need to temporarily open ports 80 and 443 in your firewall configuration. The SSL certificate provider must be able to make successful HTTP and HTTPS requests to your server through the Domain Name System (DNS).
2. Use an SSL certificate provider to generate the certificate chain (`fullchain.pem`) and private key (`privkey.pem`) files, or the PKCS #12 certificate, from a trusted CA. Steps will vary based on the certificate provider. Contact your certificate provider for details.
3. Close ports 80 and 443 in your firewall configuration.
4. Change the A record of your domain to the public domain-name address of your web server instance.
Proceed to the next sections as follows:
* If you have certificate chain and private key files, perform the steps in [Convert the certificate chain and private key files to a PKCS #12 certificate](#convert-the-certificate-chain-and-private-key-files-to-a-pkcsand-160-12-certificate) and [Import the PKCS #12 certificate into the Fusion Proxy service keystore](#create-the-fusion-proxy-service-keystore-and-import-the-pkcsand-160-12-certificate).
* If you have a PKCS #12 certificate, perform the steps in [Import the PKCS #12 certificate into the Fusion Proxy service keystore](#create-the-fusion-proxy-service-keystore-and-import-the-pkcsand-160-12-certificate).
#### Convert the certificate chain and private key files to a PKCS #12 certificate
If you have a PKCS #12 certificate, skip this section and proceed to the section [Import the PKCS #12 certificate into the Fusion Proxy service keystore](#create-the-fusion-proxy-service-keystore-and-import-the-pkcsand-160-12-certificate).
**How to create a PKCS #12 certificate**
With the certificate chain and private key as separate files, use the `openssl` command line tool in [OpenSSL](https://www.openssl.org/) to create a PKCS #12 certificate.
```bash theme={"dark"}
openssl pkcs12 -export -out \path\to\keystore.p12 -in \path\to\fullchain.pem -inkey \path\to\privkey.pem
```
Do not enter a blank password.
#### Create the Fusion Proxy service keystore and import the PKCS #12 certificate
Use the Java [keytool](https://docs.oracle.com/javase/8/docs/technotes/tools/windows/keytool.html)
Key and Certificate Management utility to create a keystore for the Fusion Proxy service (`%FUSION_HOME%\apps\jetty\proxy\etc\keystore`) and import the PKCS #12 certificate file. Fusion uses this certificate to perform SSL.
If you are starting with a certificate file in PFX format (file extension `.pfx`), that format is now identical to PKCS #12 format.
1. To create the keystore and import the PKCS #12 certificate:
```bash theme={"dark"}
"%JAVA_HOME%\bin\keytool.exe" -genkeypair -alias mykeystore -keyalg RSA -keysize 2048 -keystore keystore.jks
"%JAVA_HOME%\bin\keytool.exe" -importkeystore -srckeystore mycertificate.p12 -srcstoretype PKCS12 -destkeystore keystore.jks
```
2. Use the keytool `import` command to create a JSSE keystore.
```bash theme={"dark"}
keytool -importkeystore -srckeystore \path\to\keystore.p12 -srcstoretype PKCS12 -destkeystore "%FUSION_HOME%\apps\jetty\proxy\etc\keystore" -deststoretype PKCS12
```
3. (Optional) If desired, delete the PKCS #12 certificate file that resides outside of the Fusion Proxy service keystore (the one you created from the certificate chain and private key files, or obtained from a trusted CA.
```bash theme={"dark"}
del \path\to\keystore.p12
```
## Enable HTTPS in the Fusion Proxy service
Before beginning these steps, [load an SSL certificate into a Fusion keystore](#alternative-1-self-signed-certificate).
**How to enable HTTPS in the Fusion Proxy service:**
1. Set environment variables:
```bash theme={"dark"}
set JAVA_HOME=JavaHomeDirectory
set FUSION_HOME=FusionHomeDirectory
```
2. Add HTTPS protocol support to the Jetty TLS (SSL) connector:
```bash theme={"dark"}
cd "%FUSION_HOME%\apps\jetty\proxy\"
java -jar "%FUSION_HOME%\apps\jetty\home\start.jar" --add-to-start=https
```
Example output:
```log theme={"dark"}
INFO: ssl initialized (transitively) in ${jetty.base}/start.ini
INFO: https initialized in ${jetty.base}/start.ini
INFO: Base directory was modified
```
3. Get the obfuscated version of your keystore password:
```bash theme={"dark"}
java -cp "$FUSION_HOME/apps/jetty/home/lib/*" org.eclipse.jetty.util.security.Password PASSWORD
```
Replace `PASSWORD` with the password you used for the keystore. If the password contains special characters, [URL encode them](https://www.w3schools.com/tags/ref_urlencode.asp).
Example output:
```log theme={"dark"}
2018-05-15 12:32:48.988:INFO::main: Logging initialized @133ms
password345XYZ
OBF:1vn21ugu1saj1v9i1v941sar1ugw1vo0
MD5:b91cd1a54781790beaa2baf741fa6789
```
4. Edit the file `%FUSION_HOME%\apps\jetty\proxy\start.ini`:
1. Include obfuscated passwords by adding these properties to the end of the file:
* `jetty.sslContext.keyStorePassword`
* `jetty.sslContext.keyManagerPassword`
* `jetty.sslContext.trustStorePassword`
2. Use the OBF-encrypted password from step 4 (including the `OBF:` string) as the value for all three of the properties.\
For example:
```ini theme={"dark"}
## Keystore password
jetty.sslContext.keyStorePassword=OBF:2uha1vgt1jg01a4b1a4j1jda1vg11ugg
...
## KeyManager password
jetty.sslContext.keyManagerPassword=OBF:2uha1vgt1jg01a4b1a4j1jda1vg11ugg
## Truststore password
jetty.sslContext.trustStorePassword=OBF:2uha1vgt1jg01a4b1a4j1jda1vg11ugg
```
3. Set the local SSL port by adding the `jetty.ssl.port` property to the end of the file, and providing the port number. For example:
```ini theme={"dark"}
## Connector port to listen on
jetty.ssl.port=8443
```
4. Save the file `%FUSION_HOME%\apps\jetty\proxy\start.ini`.
## Restart Fusion and test access through HTTPS
1. Restart all Fusion services:
```bash theme={"dark"}
bin\fusion.cmd restart
```
HTTPS should now be enabled in the Fusion Proxy service.
2. Sign in to the Fusion UI. Specify the HTTPS URL scheme and SSL port, for example, `https://search.mycorp:8443`.
## Disable HTTP access to the Fusion Proxy service
Disable HTTP access. You have a choice. Perform the tasks in the appropriate section:
* [**Disable HTTP access on the firewall or load balancer**](#alternative-1-disable-http-access-on-the-firewall-or-load-balancer) - *This is the preferred approach.*
* [**Disable listening for HTTP requests in the Fusion Proxy service**](#disable-http-access-to-the-fusion-proxy-service)
### Alternative 1: Disable HTTP access on the firewall or load balancer
Disable HTTP access to the Fusion Proxy service on the firewall or load balancer:
1. Disallow all requests for port 8764 from the outside world. Only `localhost` should be able to communicate with Fusion on the non-SSL port 8764. Block all other requestors.
2. If you are using a firewall or load balancer in front of Fusion, use it to redirect all HTTP requests to use HTTPS instead. For example, Apache would [redirect all incoming HTTP traffic to HTTPS](https://httpd.apache.org/docs/2.4/rewrite/avoid.html).
### Alternative 2: Disable listening for HTTP requests in the Fusion Proxy service
Ideally, you should [disable HTTP access using the firewall or load balancer](#alternative-1-disable-http-access-on-the-firewall-or-load-balancer). Follow the steps in this section *only* if disabling HTTP access on the firewall or load balancer is not feasible.
You can only use this alternative if your SSL certificate covers a hostname that can be accessed from the local host. For example, if your certificate only covers `https://fusion.com`, then your local machine must be able to access Fusion from that exact host. If necessary, change the `hosts` file so that this can work.
**How to disable HTTP**
1. Edit `\lucidworks\fusion{backslash}latest.x\apps\jetty\proxy\start.d\http.ini`.
1. Change this line:
```
--module=http
```
To:
```
#--module=http
```
2. Save the file.
2. Edit the Fusion configuration file, `\lucidworks\fusion{backslash}latest.x\conf\fusion.cors` (`fusion.properties` in Fusion 4.x).
1. Ensure that the Agent JVM uses the Fusion Proxy service’s keystore by adding this to the end of the file:
```properties theme={"dark"}
agent.jvmOptions=-Djavax.net.ssl.trustStore="%{FUSION_HOME}%\\apps\\jetty\\proxy\\etc\\keystore" -Djavax.net.ssl.trustStorePassword=PASSWORD -Djavax.net.ssl.keyStore="%{FUSION_HOME}%\\apps\\jetty\\proxy\\etc\\keystore" -Djavax.net.ssl.keyStorePassword=PASSWORD
```
Replace `PASSWORD` with your Fusion keystore password.
2. Uncomment the `default.address` and change it to the hostname of the server that is validated by your SSL certificate.\
If the hostname saved in `default.address` is not validated by your SSL certificate, then the Fusion Proxy service will not start, because the agent’s liveness detector will not be able to access the HTTPS port to determine whether Fusion is running.
If you self-signed the certificate, then the `default.address` must match the hostname you specified while signing the certificate. Failure to do this will result in the Fusion Proxy service not starting after you have disabled HTTP.
For example, if your SSL certificate’s validated hostname is `search.mycorp`, then change:
```
#default.address = 127.0.0.1
```
To:
```
default.address = search.mycorp
```
3. Change the `proxy.port` to the SSL port you chose. For example, change:
```
proxy.port = 8764
```
To:
```
proxy.port = 8443
```
4. Uncomment `proxy.ssl` and change its value to `true`. Change:
```
# proxy.ssl=false
```
To:
```
proxy.ssl=true
```
Stop and start Fusion for these changes to take effect.
## References and tutorials
* [Transport Layer Security](http://en.wikipedia.org/wiki/Transport_Layer_Security) (Wikipedia)
* [Public Key Certificate](http://en.wikipedia.org/wiki/Public_key_certificate) (Wikipedia)
* [OpenSSL Cookbook](https://www.feistyduck.com/library/openssl-cookbook/online/) (free ebook)
* [OpenSSL Command Line Utilities](https://wiki.openssl.org/index.php/Command_Line_Utilities) (OpenSSL wiki)
* [Java Tutorials: Generating and Verifying Certificates](https://docs.oracle.com/javase/tutorial/security/apisign/index.html)
* [IBM developerWorks: What is the JSSE all about?](http://www.ibm.com/developerworks/websphere/techjournal/0502_benantar/0502_benantar.html)
## Fusion user login
When logging into the Fusion UI, a user provides a username and password, as well as their assigned security realm. An administrator must specify these in Fusion (using the native security realm) or configure Fusion to use an external authentication provider (for example, LDAP or SAML).
See [Access control](/docs/5/fusion/operations/security/access-control/overview).
Fusion uses [roles](/docs/5/fusion/operations/security/access-control/roles) defined by [permissions](/docs/5/fusion/operations/security/access-control/permissions)
to authorize Fusion UI access and perform tasks in Fusion, including searching. The recommended method to delegate permissions is as follows:
* Assign each user to a role and create custom roles as needed.
* Assign permissions on a per-app basis.
## Manage users with security realms
Fusion uses [security realms](/docs/5/fusion/operations/security/access-control/security-realms) to authenticate users of the Fusion UI.
Each user has an assigned security realm, which the user must select when logging in. If the user selects a different realm, authentication fails.
A security realm also provides a list of roles as follows:
* The list always includes the role(s) that are specified in the security realm.
* (Optional) If an external directory service (such as LDAP) is used for authentication, the list can also contain roles that are mapped from the names of the directory-service groups. That is, you can configure a security realm to return group information for users from *that same* directory service.
* (Optional) The security realm can reference one or more Fusion roles or, when using an external directory service provider, use group membership information from the provider to determine roles for users. Fusion maps the group names to role names and adds these roles to the user’s list of roles.
Fusion does not use permissions from LDAP to authorize UI access or API requests. It only obtains group names (optionally), which are used as role names or are mapped to role names. If an [Active Directory Security Query Trimming Stage](/docs/5/fusion/reference/config-ref/pipeline-stages/query-stages/active-directory-security-trimming-query-stage) is used, then directory-service permissions *are* used for trimming. If a connector supports security trimming, then connector permissions *are* used for trimming.
## Per-request authentication
Requests to the Fusion REST API must specify a security realm for per-request authentication, unless a session cookie is used (which contains information about the security realm).
Fusion authorizes requested operations based on API permissions specified for the user and for the user’s role(s). Fusion considers the role(s) specified in the user definition and in the security realm.
Fusion creates a list of roles when a session is created, that is, when a user logs in or when the [Sessions](/api-reference/system-status/get-the-current-user-session) REST API creates a session.
Authorization based on [permissions](/docs/5/fusion/operations/security/access-control/permissions) and its layering is at request time.
You can define multiple [security realms](/docs/5/fusion/operations/security/access-control/security-realms) for a Fusion instance. A Fusion instance can manage multiple security realms, which allows users from different domains
to have (different levels of) access to specific Fusion collections.
## Read-only root file system
Fusion 5.9.9 and later supports a read-only root file system to safeguard against unauthorized modifications, protecting your deployment against malicious software and other attacks.
Most services operate efficiently in this mode, and for components that need write access, a separate writable mount is available using specific helm configurations.
Read-only mode should only be enabled for services that do not require CRUD access.
See **Enable Read-Only Root File System** for examples of how to configure this feature.
This topic describes how to configure a read-only root file system for a Fusion deployment.
Read-only mode safeguards your file system against unauthorized changes, such as by malicious software or other attacks.
Enabling this feature entails configuring the `readOnlyRootFilesystem` attribute in the Fusion Helm chart.
This feature is available starting in Fusion 5.9.9 and in all subsequent Fusion 5.9 releases.
Beginning with Fusion 5.9.10, all of [Fusion’s services](/docs/5/fusion/reference/microservices) are designed to work with a read-only root file system, but some external services may require write access to the file system.
In that case, you can configure a read-only root file system and mount a separate writable file system for these services.
The examples below show how to configure a read-only root file system for various Fusion services.
## Solr
```yaml theme={"dark"}
fusion:
solr:
containerSecurityContext:
readOnlyRootFilesystem: true
```
**With optional services:**
```yaml theme={"dark"}
fusion:
solr:
exporter:
enabled: true
securityContext:
readOnlyRootFilesystem: true
containerSecurityContext:
readOnlyRootFilesystem: true
```
## Zookeeper
```yaml theme={"dark"}
fusion:
zookeeper:
containerSecurityContext:
readOnlyRootFilesystem: true
```
**With optional services:**
```yaml theme={"dark"}
fusion:
zookeeper:
containerSecurityContext:
readOnlyRootFilesystem: true
exporters:
jmx:
enabled: true
zookeeper:
enabled: true
jobs:
chroots:
enabled: true
config:
create:
- /root-read-only
```
## Kafka
```yaml theme={"dark"}
fusion:
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
```
**With optional services:**
```yaml expandable theme={"dark"}
fusion:
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
volumePermissions:
enabled: true
containerSecurityContext:
readOnlyRootFilesystem: true
metrics:
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
enabled: true
jmx:
enabled: true
containerSecurityContext:
enabled: true
readOnlyRootFilesystem: true
externalAccess:
enabled: true
autoDiscovery:
enabled: true
containerSecurityContext:
enabled: true
readOnlyRootFilesystem: true
provisioning:
enabled: true
topics:
- name: test
partitions: 1
replicationFactor: 1
## https://kafka.apache.org/documentation/#topicconfigs
config:
max.message.bytes: 64000
flush.messages: 1
containerSecurityContext:
readOnlyRootFilesystem: true
enabled: true
```
## ML Model Service
```yaml expandable theme={"dark"}
fusion:
ml-model-service:
ambassador:
containerSecurityContext:
readOnlyRootFilesystem: true
milvus:
mysql:
containerSecurityContext:
readOnlyRootFilesystem: true
securityContext:
readOnlyRootFilesystem: true
With optional services:
fusion:
ml-model-service:
ambassador:
containerSecurityContext:
readOnlyRootFilesystem: true
prometheusExporter:
enabled: true
securityContext:
readOnlyRootFilesystem: true
milvus:
mysql:
metrics:
enabled: true
securityContext:
readOnlyRootFilesystem: true
containerSecurityContext:
readOnlyRootFilesystem: true
cluster:
enabled: false
mishards:
securityContext:
readOnlyRootFilesystem: true
admin:
enabled: true
securityContext:
readOnlyRootFilesystem: true
securityContext:
readOnlyRootFilesystem: true
```
## Argo
```yaml theme={"dark"}
fusion:
argo:
minio:
containerSecurityContext:
readOnlyRootFilesystem: true
extraVolumes:
- name: config-dir
emptyDir: {}
- name: certs-dir
emptyDir: {}
extraVolumeMounts:
- name: config-dir
mountPath: /.minio
- name: certs-dir
mountPath: /etc/minio/certs
server:
securityContext:
readOnlyRootFilesystem: true
controller:
securityContext:
readOnlyRootFilesystem: true
mainContainer:
securityContext:
readOnlyRootFilesystem: true
executor:
securityContext:
readOnlyRootFilesystem: true
```
## Seldon Core Operator
```yaml theme={"dark"}
fusion:
seldon-core-operator:
containersSecurityContext:
readOnlyRootFilesystem: true
```
## Argo Common Workflows
```yaml theme={"dark"}
fusion:
argo-common-workflows:
containerSecurityContext:
readOnlyRootFilesystem: true
```
## Question Answering
```yaml theme={"dark"}
fusion:
question-answering:
containersSecurityContext:
readOnlyRootFilesystem: true
```
## Classification
```yaml theme={"dark"}
fusion:
classification:
containersSecurityContext:
readOnlyRootFilesystem: true
```
## All-in-one values.yaml example
```yaml expandable theme={"dark"}
fusion:
solr:
containerSecurityContext:
readOnlyRootFilesystem: true
solr-managed:
containerSecurityContext:
readOnlyRootFilesystem: true
zookeeper:
containerSecurityContext:
readOnlyRootFilesystem: true
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
ml-model-service:
ambassador:
containerSecurityContext:
readOnlyRootFilesystem: true
milvus:
mysql:
containerSecurityContext:
readOnlyRootFilesystem: true
securityContext:
readOnlyRootFilesystem: true
argo:
minio:
containerSecurityContext:
readOnlyRootFilesystem: true
extraVolumes:
- name: config-dir
emptyDir: {}
- name: certs-dir
emptyDir: {}
extraVolumeMounts:
- name: config-dir
mountPath: /.minio
- name: certs-dir
mountPath: /etc/minio/certs
server:
securityContext:
readOnlyRootFilesystem: true
controller:
securityContext:
readOnlyRootFilesystem: true
mainContainer:
securityContext:
readOnlyRootFilesystem: true
executor:
securityContext:
readOnlyRootFilesystem: true
seldon-core-operator:
containersSecurityContext:
readOnlyRootFilesystem: true
argo-common-workflows:
containerSecurityContext:
readOnlyRootFilesystem: true
question-answering:
containersSecurityContext:
readOnlyRootFilesystem: true
classification:
containersSecurityContext:
readOnlyRootFilesystem: true
```
## All-in-one values.yaml with optional services example
```yaml expandable theme={"dark"}
fusion:
argo:
minio:
persistence:
enabled: false
containerSecurityContext:
readOnlyRootFilesystem: true
extraVolumes:
- name: config-dir
emptyDir: {}
- name: certs-dir
emptyDir: {}
extraVolumeMounts:
- name: config-dir
mountPath: /.minio
- name: certs-dir
mountPath: /etc/minio/certs
server:
securityContext:
readOnlyRootFilesystem: true
controller:
securityContext:
readOnlyRootFilesystem: true
mainContainer:
securityContext:
readOnlyRootFilesystem: true
executor:
securityContext:
readOnlyRootFilesystem: true
solr:
exporter:
enabled: true
securityContext:
readOnlyRootFilesystem: true
containerSecurityContext:
readOnlyRootFilesystem: true
# tls:
# enabled: true
solr-managed:
exporter:
enabled: true
securityContext:
readOnlyRootFilesystem: true
enableExternalFiles: true
enabledStorage:
- gcs
processRaw:
image:
repository: fusion-dev-docker.ci-artifactory.lucidworks.com
cloudRoot: gs://lw-managed-fusion-data/tmp
gcs:
secret: gcs-key
secretFieldName: key.json
securityContext:
readOnlyRootFilesystem: true
containerSecurityContext:
readOnlyRootFilesystem: true
# tls:
# enabled: true
zookeeper:
containerSecurityContext:
readOnlyRootFilesystem: true
exporters:
jmx:
enabled: true
zookeeper:
enabled: true
jobs:
chroots:
# enabled: true
config:
create:
- /root-read-only
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
volumePermissions:
enabled: true
image:
repository: "fusion-dev-docker.ci-artifactory.lucidworks.com/os-shell"
containerSecurityContext:
readOnlyRootFilesystem: true
metrics:
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
enabled: true
jmx:
enabled: true
containerSecurityContext:
enabled: true
readOnlyRootFilesystem: true
externalAccess:
enabled: true
autoDiscovery:
enabled: true
containerSecurityContext:
enabled: true
readOnlyRootFilesystem: true
provisioning:
enabled: true
topics:
- name: test
partitions: 1
replicationFactor: 1
## https://kafka.apache.org/documentation/#topicconfigs
config:
max.message.bytes: 64000
flush.messages: 1
containerSecurityContext:
readOnlyRootFilesystem: true
enabled: true
ml-model-service:
enabled: true
ambassador:
containerSecurityContext:
readOnlyRootFilesystem: true
prometheusExporter:
enabled: true
securityContext:
readOnlyRootFilesystem: true
milvus:
mysql:
metrics:
enabled: true
securityContext:
readOnlyRootFilesystem: true
containerSecurityContext:
readOnlyRootFilesystem: true
cluster:
enabled: false
mishards:
securityContext:
readOnlyRootFilesystem: true
admin:
enabled: true
securityContext:
readOnlyRootFilesystem: true
securityContext:
readOnlyRootFilesystem: true
seldon-core-operator:
containersSecurityContext:
readOnlyRootFilesystem: true
argo-common-workflows:
containerSecurityContext:
readOnlyRootFilesystem: true
question-answering:
containersSecurityContext:
readOnlyRootFilesystem: true
classification:
containersSecurityContext:
readOnlyRootFilesystem: true
```
## Supported services
The table below lists the services that support a read-only root file system, the ones that have it enabled by default, and the Fusion release in which support was added:
| Chart Name | Pod Name | Container Name | Supported | Default enabled | Supported version |
| - | - | - | - | - | - |
| admin-ui | admin-ui | admin-ui | ✅ | ✅ | 5.9.9+ |
| api-gateway | api-gateway | init/api-gateway | ✅ | ✅ | 5.9.9+ |
| api-gateway | api-gateway | api-gateway | ✅ | ✅ | 5.9.9+ |
| api-gateway | api-gateway | generate-jks | ✅ | ✅ | 5.9.9+ |
| apps-manager | apps-manager | apps-manager | ✅ | ✅ | 5.9.9+ |
| argo | argo-server | argo-server | ✅ | ❌ | 5.9.10+ |
| argo | argo-executor | executor | ✅ | ❌ | 5.9.10+ |
| argo | argo-mainContainer | mainContainer | ✅ | ❌ | 5.9.10+ |
| argo | argo-controller | controller | ✅ | ✅ | 5.9.10+ |
| argo/minio | minio | minio | ✅ | ❌ | 5.9.10+ |
| argo/minio | minio | minio | ✅ | ❌ | 5.9.10+ |
| argo/minio | make-bucket-job | minio-mc | ✅ | ❌ | 5.9.10+ |
| argo-common-workflows | delete-model | init/main/wait | ✅ | ❌ | 5.9.10+ |
| argo-common-workflows | deploy-model | init/main/wait | ✅ | ❌ | 5.9.10+ |
| argo-common-workflows | milvus-maintenance | init/main/wait | ✅ | ❌ | 5.9.10+ |
| argo-common-workflows | upload-model-to-cloud | init/main/wait | ✅ | ❌ | 5.9.10+ |
| async-parsing | async-parsing | tika-server | ✅ | ✅ | 5.9.9+ |
| async-parsing | async-parsing | async-parsing | ✅ | ✅ | 5.9.9+ |
| auth-ui | auth-ui | auth-ui | ✅ | ✅ | 5.9.9+ |
| classic-rest-service | classic-rest-service | init/import-certs | ✅ | ✅ | 5.9.10+ |
| classic-rest-service | classic-rest-service | classic-rest-service | ✅ | ✅ | 5.9.9+ |
| classification | argo/classification | init/wait/main | ✅ | ❌ | 5.9.10+ |
| connector-plugin | connector-plugin | init/import-certs | ✅ | ❌ | 5.9.10+ |
| connector-plugin | connector-plugin | connector-plugin | ✅ | ✅ | 5.9.9+ |
| connectors | connectors | connectors | ✅ | ✅ | 5.9.9+ |
| connectors-backend | CRD | | ✅ | ✅ | 5.9.9+ |
| connectors-backend | connectors-backend | connectors-backend | ✅ | ✅ | 5.9.9+ |
| fusion-admin | fusion-admin | admin | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-admin | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-api-gateway | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-indexing | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-kafka | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-logstash | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-pulsar | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | setup-keystore-and-properties | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-zk | ✅ | ✅ | 5.9.9+ |
| fusion-config-sync | fusion-config-sync | fusion-config-sync | ✅ | ✅ | 5.9.9+ |
| fusion-data-augmentation | argo/data-augmentation/volume-fix | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/init-workspace | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/write-job-configs | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/write-io-configs | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/add-zkhost | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/pull-data-training-and-metadata | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/pull-data-training-and-metadata-cloud | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/volume-fix2 | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/synonym-list | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/download-synonym-dictionary | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/keystroke-list | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/download-keystroke-blob | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/augment | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/push-augmented-data | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/push-augmented-data-cloud | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-indexing | fusion-indexing | fusion-indexing | ✅ | ✅ | 5.9.9+ |
| fusion-resources | fusion-resources-secret-hook | setup-keystore-and-properties | ✅ | ✅ | 5.9.10+ |
| insights | insights | insights | ✅ | ✅ | 5.9.9+ |
| job-launcher | job-launcher | job-launcher | ✅ | ✅ | 5.9.9+ |
| job-launcher | job-launcher | kubectl-runner | ✅ | ✅ | 5.9.9+ |
| job-launcher | job-launcher-spark-cleanup | kubectl-runner | ✅ | ✅ | 5.9.9+ |
| job-launcher | spark-kubernetes-driver | spark-kubernetes-driver | ✅ | ❌ | 5.9.10+ |
| job-launcher | spark-kubernetes-executor | spark-kubernetes-executor | ✅ | ❌ | 5.9.10+ |
| job-rest-server | job-rest-server | job-rest-server | ✅ | ✅ | 5.9.9+ |
| kafka | kafka-metrics | kafka-exporter-archived | ✅ | ❌ | 5.9.10+ |
| kafka | kafka-provisioning | init/wait-for-available-kafka | ✅ | ❌ | 5.9.10+ |
| kafka | kafka-provisioning | kafka-provisioning | ✅ | ❌ | 5.9.10+ |
| kafka | kafka | kafka | ✅ | ❌ | 5.9.10+ |
| kafka | kafka | jmx-exporter | ✅ | ❌ | 5.9.10+ |
| kafka | kafka | init/check-zk | ✅ | ✅ | 5.9.9+ |
| kafka | kafka | init/auto-discovery | ✅ | ❌ | 5.9.10+ |
| kafka | kafka | init/volume-permissions | ✅ | ❌ | 5.9.10+ |
| lwai-gateway | lwai-gateway | lwai-gateway | ✅ | ✅ | 5.9.9+ |
| ml-model-service | ml-model-service | java-service | ✅ | ✅ | 5.9.9+ |
| ml-model-service | ml-model-service-namespace-hook | kubectl-runner | ✅ | ✅ | 5.9.9+ |
| ml-model-service/ambassador | ambassador | ambassador | ✅ | ❌ | 5.9.10+ |
| ml-model-service/ambassador | ambassador | prometheus-exporter | ✅ | ❌ | 5.9.10+ |
| ml-model-service/milvus | milvus-writable | milvus | ✅ | ❌ | 5.9.10+ |
| ml-model-service/milvus | milvus-writable | init/wait-for-mysql | ✅ | ✅ | 5.9.10+ |
| ml-model-service/milvus | milvus-writable | init/create-for-share-storage | ✅ | ❌ | 5.9.10+ |
| ml-model-service/milvus | milvus-admin | admin | ✅ | ❌ | 5.9.10+ |
| ml-model-service/milvus | milvus-admin | init/wait-for-milvus | ✅ | ✅ | 5.9.10+ |
| ml-model-service/milvus | milvus-mishards | init/wait-for-mysql | ✅ | ✅ | 5.9.10+ |
| ml-model-service/milvus | milvus-mishards | init/wait-for-mysql | ✅ | ✅ | 5.9.10+ |
| ml-model-service/milvus | milvus-mishards | mishards | ✅ | ❌ | 5.9.10+ |
| ml-model-service/milvus/mysql | mysql | init/remove-lost-found | ✅ | ✅ | 5.9.10+ |
| ml-model-service/milvus/mysql | mysql | mysql | ✅ | ❌ | 5.9.10+ |
| pm-ui | pm-ui | pm-ui | ✅ | ✅ | 5.9.9+ |
| query-pipeline | query-pipeline | query-pipeline | ✅ | ✅ | 5.9.9+ |
| question-answering | argo/qna-coldstart/init-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/write-job-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/add-zkHost | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/pull-data | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/pull-data-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/train | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/list-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/post-model | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/apply-seldon-deployment | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/init-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/write-job-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/add-zkhost | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/pull-qa-data-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/pull-qa-data | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/train | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/train-with-texts | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/list-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/post-model | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/apply-seldon-deployment | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/init-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/write-job-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/add-zkhost | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/list-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/pull-eval-data | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/pull-eval-data-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/evaluate | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/push-eval-results-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/push-eval-results | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/init-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/write-job-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/add-zkhost | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/pull-data-training-and-metadata | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/pull-data-training-and-metadata-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/train-with-metadata | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/train-without-metadata | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/push-recommendations | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/push-recommendations-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/add-default-exclude-query | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/init-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/copy-model | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/write-job-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/add-zkhost | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/pull-data | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/pull-data-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/train | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/push-content | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/push-content-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/add-default-exclude-query | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/delete-old-content-recommendations | init/wait/main | ✅ | ❌ | 5.9.10+ |
| reverse-search | reverse-search | init/set-reverse-search-zone | ✅ | ✅ | 5.9.10+ |
| reverse-search | reverse-search | init/check-zk | ✅ | ✅ | 5.9.9+ |
| reverse-search | reverse-search | init/enable-tls-in-reverse-search | ✅ | ✅ | 5.9.10+ |
| reverse-search | reverse-search | reverse-search | ✅ | ❌ | 5.9.10+ |
| rules-ui | rules-ui | rules-ui | ✅ | ✅ | 5.9.9+ |
| seldon-core-operator | seldon-controller-manager | manager | ✅ | ❌ | 5.9.10+ |
| seldon-core-operator | seldon-spartakus-volunteer | seldon-spartakus-volunteer | ✅ | ❌ | 5.9.9+ |
| seldon-core-operator | crd/SeldonDeployment | | ✅/❌ | ❌ | 5.9.9+ |
| solr | solr | init/set-solr-zone | ✅ | ✅ | 5.9.10+ |
| solr | solr | init/enable-tls-in-solr | ✅ | ✅ | 5.9.10+ |
| solr | configset-bootstrap | configset-bootstrap | ✅ | ✅ | 5.9.10+ |
| solr | solr | solr | ✅ | ❌ | 5.9.10+ |
| solr | solr-exporter | exporter | ✅ | ❌ | 5.9.10+ |
| solr | solr-exporter | init/solr-init | ✅ | ✅ | 5.9.10+ |
| solr-managed | configset-bootstrap | configset-bootstrap | ✅ | ✅ | 5.9.10+ |
| solr-managed | exporter | exporter | ✅ | ❌ | 5.9.10+ |
| solr-managed | exporter | init/solr-init | ✅ | ✅ | 5.9.10+ |
| solr-managed | solr | init/set-solr-zone | ✅ | ✅ | 5.9.10+ |
| solr-managed | solr | init/enable-tls-in-solr | ✅ | ✅ | 5.9.10+ |
| solr-managed | solr | solr | ✅ | ❌ | 5.9.10+ |
| solr-managed | solr | process-raw | ✅ | ❌ | 5.9.10+ |
| solr-backup-runner | solr-backup-runner-backup | solr-backups | ✅ | ❌ | 5.9.10+ |
| solr-backup-runner | solr-backup-runner-prune | solr-prune | ✅ | ❌ | 5.9.10+ |
| templating | templating | templating | ✅ | ✅ | 5.9.9+ |
| webapps | webapps | webapps | ✅ | ✅ | 5.9.9+ |
| zookeeper | zookeeper | zookeeper | ✅ | ❌ | 5.9.10+ |
| zookeeper | zookeeper | jmx-exporter | ✅ | ❌ | 5.9.10+ |
| zookeeper | zookeeper | zookeeper-exporter | ✅ | ❌ | 5.9.10+ |
| zookeeper | zookeeper-chroots | main | ✅ | ❌ | 5.9.10+ |
The table below lists the services that support a read-only root file system, the ones that have it enabled by default, and the Fusion release in which support was added.
| Chart Name | Pod Name | Container Name | Supported | Default enabled | Supported version |
| - | - | - | - | - | - |
| `admin-ui` | `admin-ui` | `admin-ui` | ✅ | ✅ | 5.9.9+ |
| `api-gateway` | `api-gateway` | `init/api-gateway` | ✅ | ✅ | 5.9.9+ |
| `api-gateway` | `api-gateway` | `api-gateway` | ✅ | ✅ | 5.9.9+ |
| `api-gateway` | `api-gateway` | `generate-jks` | ✅ | ✅ | 5.9.9+ |
| `apps-manager` | `apps-manager` | `apps-manager` | ✅ | ✅ | 5.9.9+ |
| `argo` | `argo-server` | `argo-server` | ✅ | ❌ | 5.9.10+ |
| `argo` | `argo-executor` | `executor` | ✅ | ❌ | 5.9.10+ |
| `argo` | `argo-mainContainer` | `mainContainer` | ✅ | ❌ | 5.9.10+ |
| `argo` | `argo-controller` | `controller` | ✅ | ✅ | 5.9.10+ |
| `argo/minio` | `minio` | `minio` | ✅ | ❌ | 5.9.10+ |
| `argo/minio` | `minio` | `minio` | ✅ | ❌ | 5.9.10+ |
| `argo/minio` | `make-bucket-job` | `minio-mc` | ✅ | ❌ | 5.9.10+ |
| `argo-common-workflows` | `delete-model` | `init/main/wait` | ✅ | ❌ | 5.9.10+ |
| `argo-common-workflows` | `deploy-model` | `init/main/wait` | ✅ | ❌ | 5.9.10+ |
| `argo-common-workflows` | `milvus-maintenance` | `init/main/wait` | ✅ | ❌ | 5.9.10+ |
| `argo-common-workflows` | `upload-model-to-cloud` | `init/main/wait` | ✅ | ❌ | 5.9.10+ |
| `async-parsing` | `async-parsing` | `tika-server` | ✅ | ✅ | 5.9.9+ |
| `async-parsing` | `async-parsing` | `async-parsing` | ✅ | ✅ | 5.9.9+ |
| `auth-ui` | `auth-ui` | `auth-ui` | ✅ | ✅ | 5.9.9+ |
| `classic-rest-service` | `classic-rest-service` | `init/import-certs` | ✅ | ✅ | 5.9.10+ |
| `classic-rest-service` | `classic-rest-service` | `classic-rest-service` | ✅ | ✅ | 5.9.9+ |
| `classification` | `argo/classification` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `connector-plugin` | `connector-plugin` | `init/import-certs` | ✅ | ❌ | 5.9.10+ |
| `connector-plugin` | `connector-plugin` | `connector-plugin` | ✅ | ✅ | 5.9.9+ |
| `connectors` | `connectors` | `connectors` | ✅ | ✅ | 5.9.9+ |
| `connectors-backend` | `CRD` | | ✅ | ✅ | 5.9.9+ |
| `connectors-backend` | `connectors-backend` | `connectors-backend` | ✅ | ✅ | 5.9.9+ |
| `fusion-admin` | `fusion-admin` | `admin` | ✅ | ✅ | 5.9.9+ |
| `fusion-commons` | | `check-admin` | ✅ | ✅ | 5.9.9+ |
| `fusion-commons` | | `check-api-gateway` | ✅ | ✅ | 5.9.9+ |
| `fusion-commons` | | `check-indexing` | ✅ | ✅ | 5.9.9+ |
| `fusion-commons` | | `check-kafka` | ✅ | ✅ | 5.9.9+ |
| `fusion-commons` | | `check-logstash` | ✅ | ✅ | 5.9.9+ |
| `fusion-commons` | | `check-pulsar` | ✅ | ✅ | 5.9.9+ |
| `fusion-commons` | | `setup-keystore-and-properties` | ✅ | ✅ | 5.9.9+ |
| `fusion-commons` | | `check-zk` | ✅ | ✅ | 5.9.9+ |
| `fusion-config-sync` | `fusion-config-sync` | `fusion-config-sync` | ✅ | ✅ | 5.9.9+ |
| `fusion-data-augmentation` | `argo/data-augmentation/volume-fix` | `init/main/wait` | ✅ | ❌ | 5.9.10+ |
| `fusion-data-augmentation` | `argo/data-augmentation/init-workspace` | `init/main/wait` | ✅ | ❌ | 5.9.10+ |
| `fusion-data-augmentation` | `argo/data-augmentation/write-job-configs` | `init/main/wait` | ✅ | ❌ | 5.9.10+ |
| `fusion-data-augmentation` | `argo/data-augmentation/write-io-configs` | `init/main/wait` | ✅ | ❌ | 5.9.10+ |
| `fusion-data-augmentation` | `argo/data-augmentation/add-zkhost` | `init/main/wait` | ✅ | ❌ | 5.9.10+ |
| `fusion-data-augmentation` | `argo/data-augmentation/pull-data-training-and-metadata` | `init/main/wait` | ✅ | ❌ | 5.9.10+ |
| `fusion-data-augmentation` | `argo/data-augmentation/pull-data-training-and-metadata-cloud` | `init/main/wait` | ✅ | ❌ | 5.9.10+ |
| `fusion-data-augmentation` | `argo/data-augmentation/volume-fix2` | `init/main/wait` | ✅ | ❌ | 5.9.10+ |
| `fusion-data-augmentation` | `argo/data-augmentation/synonym-list` | `init/main/wait` | ✅ | ❌ | 5.9.10+ |
| `fusion-data-augmentation` | `argo/data-augmentation/download-synonym-dictionary` | `init/main/wait` | ✅ | ❌ | 5.9.10+ |
| `fusion-data-augmentation` | `argo/data-augmentation/keystroke-list` | `init/main/wait` | ✅ | ❌ | 5.9.10+ |
| `fusion-data-augmentation` | `argo/data-augmentation/download-keystroke-blob` | `init/main/wait` | ✅ | ❌ | 5.9.10+ |
| `fusion-data-augmentation` | `argo/data-augmentation/augment` | `init/main/wait` | ✅ | ❌ | 5.9.10+ |
| `fusion-data-augmentation` | `argo/data-augmentation/push-augmented-data` | `init/main/wait` | ✅ | ❌ | 5.9.10+ |
| `fusion-data-augmentation` | `argo/data-augmentation/push-augmented-data-cloud` | `init/main/wait` | ✅ | ❌ | 5.9.10+ |
| `fusion-indexing` | `fusion-indexing` | `fusion-indexing` | ✅ | ✅ | 5.9.9+ |
| `fusion-resources` | `fusion-resources-secret-hook` | `setup-keystore-and-properties` | ✅ | ✅ | 5.9.10+ |
| `insights` | `insights` | `insights` | ✅ | ✅ | 5.9.9+ |
| `job-launcher` | `job-launcher` | `job-launcher` | ✅ | ✅ | 5.9.9+ |
| `job-launcher` | `job-launcher` | `kubectl-runner` | ✅ | ✅ | 5.9.9+ |
| `job-launcher` | `job-launcher-spark-cleanup` | `kubectl-runner` | ✅ | ✅ | 5.9.9+ |
| `job-launcher` | `spark-kubernetes-driver` | `spark-kubernetes-driver` | ✅ | ❌ | 5.9.10+ |
| `job-launcher` | `spark-kubernetes-executor` | `spark-kubernetes-executor` | ✅ | ❌ | 5.9.10+ |
| `job-rest-server` | `job-rest-server` | `job-rest-server` | ✅ | ✅ | 5.9.9+ |
| `kafka` | `kafka-metrics` | `kafka-exporter-archived` | ✅ | ❌ | 5.9.10+ |
| `kafka` | `kafka-provisioning` | `init/wait-for-available-kafka` | ✅ | ❌ | 5.9.10+ |
| `kafka` | `kafka-provisioning` | `kafka-provisioning` | ✅ | ❌ | 5.9.10+ |
| `kafka` | `kafka` | `kafka` | ✅ | ❌ | 5.9.10+ |
| `kafka` | `kafka` | `jmx-exporter` | ✅ | ❌ | 5.9.10+ |
| `kafka` | `kafka` | `init/check-zk` | ✅ | ✅ | 5.9.9+ |
| `kafka` | `kafka` | `init/auto-discovery` | ✅ | ❌ | 5.9.10+ |
| `kafka` | `kafka` | `init/volume-permissions` | ✅ | ❌ | 5.9.10+ |
| `lwai-gateway` | `lwai-gateway` | `lwai-gateway` | ✅ | ✅ | 5.9.9+ |
| `ml-model-service` | `ml-model-service` | `java-service` | ✅ | ✅ | 5.9.9+ |
| `ml-model-service` | `ml-model-service-namespace-hook` | `kubectl-runner` | ✅ | ✅ | 5.9.9+ |
| `ml-model-service/ambassador` | `ambassador` | `ambassador` | ✅ | ❌ | 5.9.10+ |
| `ml-model-service/ambassador` | `ambassador` | `prometheus-exporter` | ✅ | ❌ | 5.9.10+ |
| `ml-model-service/milvus` | `milvus-writable` | `milvus` | ✅ | ❌ | 5.9.10+ |
| `ml-model-service/milvus` | `milvus-writable` | `init/wait-for-mysql` | ✅ | ✅ | 5.9.10+ |
| `ml-model-service/milvus` | `milvus-writable` | `init/create-for-share-storage` | ✅ | ❌ | 5.9.10+ |
| `ml-model-service/milvus` | `milvus-admin` | `admin` | ✅ | ❌ | 5.9.10+ |
| `ml-model-service/milvus` | `milvus-admin` | `init/wait-for-milvus` | ✅ | ✅ | 5.9.10+ |
| `ml-model-service/milvus` | `milvus-mishards` | `init/wait-for-mysql` | ✅ | ✅ | 5.9.10+ |
| `ml-model-service/milvus` | `milvus-mishards` | `init/wait-for-mysql` | ✅ | ✅ | 5.9.10+ |
| `ml-model-service/milvus` | `milvus-mishards` | `mishards` | ✅ | ❌ | 5.9.10+ |
| `ml-model-service/milvus/mysql` | `mysql` | `init/remove-lost-found` | ✅ | ✅ | 5.9.10+ |
| `ml-model-service/milvus/mysql` | `mysql` | `mysql` | ✅ | ❌ | 5.9.10+ |
| `pm-ui` | `pm-ui` | `pm-ui` | ✅ | ✅ | 5.9.9+ |
| `query-pipeline` | `query-pipeline` | `query-pipeline` | ✅ | ✅ | 5.9.9+ |
| `question-answering` | `argo/qna-coldstart/init-workspace` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-coldstart/write-job-configs` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-coldstart/write-io-configs` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-coldstart/write-io-configs` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-coldstart/add-zkHost` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-coldstart/pull-data` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-coldstart/pull-data-cloud` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-coldstart/train` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-coldstart/list-workspace` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-coldstart/post-model` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-coldstart/apply-seldon-deployment` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-supervised/init-workspace` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-supervised/write-job-configs` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-supervised/write-io-configs` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-supervised/add-zkhost` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-supervised/pull-qa-data-cloud` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-supervised/pull-qa-data` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-supervised/train` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-supervised/train-with-texts` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-supervised/list-workspace` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-supervised/post-model` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-supervised/apply-seldon-deployment` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-evaluation/init-workspace` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-evaluation/write-job-configs` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-evaluation/write-io-configs` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-evaluation/add-zkhost` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-evaluation/list-workspace` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-evaluation/pull-eval-data` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-evaluation/pull-eval-data-cloud` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-evaluation/evaluate` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-evaluation/push-eval-results-cloud` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `question-answering` | `argo/qna-evaluation/push-eval-results` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-user` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-user/init-workspace` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-user/write-job-configs` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-user/write-io-configs` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-user/add-zkhost` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-user/pull-data-training-and-metadata` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-user/pull-data-training-and-metadata-cloud` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-user/train-with-metadata` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-user/train-without-metadata` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-user/push-recommendations` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-user/push-recommendations-cloud` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-user/add-default-exclude-query` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-user/init-workspace` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-content/copy-model` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-content/write-job-configs` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-content/write-io-configs` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-content/add-zkhost` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-content/pull-data` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-content/pull-data-cloud` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-content/train` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-content/push-content` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-content/push-content-cloud` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-content/add-default-exclude-query` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `recommender` | `argo/item-recommender-content/delete-old-content-recommendations` | `init/wait/main` | ✅ | ❌ | 5.9.10+ |
| `reverse-search` | `reverse-search` | `init/set-reverse-search-zone` | ✅ | ✅ | 5.9.10+ |
| `reverse-search` | `reverse-search` | `init/check-zk` | ✅ | ✅ | 5.9.9+ |
| `reverse-search` | `reverse-search` | `init/enable-tls-in-reverse-search` | ✅ | ✅ | 5.9.10+ |
| `reverse-search` | `reverse-search` | `reverse-search` | ✅ | ❌ | 5.9.10+ |
| `rules-ui` | `rules-ui` | `rules-ui` | ✅ | ✅ | 5.9.9+ |
| `seldon-core-operator` | `seldon-controller-manager` | `manager` | ✅ | ❌ | 5.9.10+ |
| `seldon-core-operator` | `seldon-spartakus-volunteer` | `seldon-spartakus-volunteer` | ✅ | ❌ | 5.9.9+ |
| `seldon-core-operator` | `crd/SeldonDeployment` | | ✅/❌ | ❌ | 5.9.9+ |
| `solr` | `solr` | `init/set-solr-zone` | ✅ | ✅ | 5.9.10+ |
| `solr` | `solr` | `init/enable-tls-in-solr` | ✅ | ✅ | 5.9.10+ |
| `solr` | `configset-bootstrap` | `configset-bootstrap` | ✅ | ✅ | 5.9.10+ |
| `solr` | `solr` | `solr` | ✅ | ❌ | 5.9.10+ |
| `solr` | `solr-exporter` | `exporter` | ✅ | ❌ | 5.9.10+ |
| `solr` | `solr-exporter` | `init/solr-init` | ✅ | ✅ | 5.9.10+ |
| `solr-managed` | `configset-bootstrap` | `configset-bootstrap` | ✅ | ✅ | 5.9.10+ |
| `solr-managed` | `exporter` | `exporter` | ✅ | ❌ | 5.9.10+ |
| `solr-managed` | `exporter` | `init/solr-init` | ✅ | ✅ | 5.9.10+ |
| `solr-managed` | `solr` | `init/set-solr-zone` | ✅ | ✅ | 5.9.10+ |
| `solr-managed` | `solr` | `init/enable-tls-in-solr` | ✅ | ✅ | 5.9.10+ |
| `solr-managed` | `solr` | `solr` | ✅ | ❌ | 5.9.10+ |
| `solr-managed` | `solr` | `process-raw` | ✅ | ❌ | 5.9.10+ |
| `solr-backup-runner` | `solr-backup-runner-backup` | `solr-backups` | ✅ | ❌ | 5.9.10+ |
| `solr-backup-runner` | `solr-backup-runner-prune` | `solr-prune` | ✅ | ❌ | 5.9.10+ |
| `templating` | `templating` | `templating` | ✅ | ✅ | 5.9.9+ |
| `webapps` | `webapps` | `webapps` | ✅ | ✅ | 5.9.9+ |
| `zookeeper` | `zookeeper` | `zookeeper` | ✅ | ❌ | 5.9.10+ |
| `zookeeper` | `zookeeper` | `jmx-exporter` | ✅ | ❌ | 5.9.10+ |
| `zookeeper` | `zookeeper` | `zookeeper-exporter` | ✅ | ❌ | 5.9.10+ |
| `zookeeper` | `zookeeper-chroots` | `main` | ✅ | ❌ | 5.9.10+ |
## Restrictive mode for query pipelines
In Fusion 5.9.15 and later, you can choose restrictive mode for any query pipeline.
Restrictive mode safeguards the pipeline against unintended or unsafe changes.
The default mode is permissive mode.
See [Restrictive Mode](/docs/5/fusion/getting-data-out/query-basics/query-pipelines/restrictive-mode) for complete details about how restrictive mode works and how to enable it.
## Lucidworks AI Gateway
[Lucidworks AI Gateway](/docs/lw-platform/lw-ai/lw-ai-gateway) acts as a bridge between Fusion and Lucidworks AI, handling the complexities of authentication and communication.
By managing credentials and other sensitive information behind the scenes, it ensures that Fusion can interact with Lucidworks AI without exposing the details of the underlying accounts or requiring manual authentication.
In Lucidworks AI, you have one or more integrations that each represent a set of credentials and a security scope.
When you add an integration to Lucidworks AI Gateway, it allows Fusion to seamlessly connect to Lucidworks AI at any time without additional authentication.
The diagram below displays the interaction between Lucidworks AI and Fusion.
Note that Lucidworks AI Gateway only posts to Lucidworks AI; Lucidworks Platform and Lucidworks AI cannot reach out to Lucidworks AI Gateway or to Fusion.
### Fusion firewall and port considerations
Lucidworks AI Gateway is a critical component of Lucidworks Fusion, enabling secure and authenticated communication between Fusion and Lucidworks AI. It facilitates seamless integration with AI-powered search enhancements by managing credentials and access permissions.
To function correctly, the `lwai-gateway` pod must be able to communicate externally on port 443 for:
* Authentication service: Okta authentication hosted at identity.lucidworks.com for secure OAuth-based authentication.
* LWAI services: Reaching the private LWAI endpoint hosted on the Lucidworks Platform.
If a firewall restricts outbound connections, administrators must allow traffic over port 443 to these destinations.
### Security and configuration
Lucidworks AI Gateway securely handles API credentials and account scopes, allowing Fusion to interact with AI models without exposing sensitive data. Configuration is managed using Helm values files, ensuring flexible multi-integration support for Fusion instances.
By ensuring proper firewall rules and secure credentials management, the Lucidworks AI Gateway enables a robust AI-powered search experience within Fusion.
## Learn more
You can configure Active Directory Federation Services (AD FS) for SAML single sign-on (SSO) authentication in Fusion. This guide explains how to set up Spring Boot OAuth2 with AD FS.
OAuth-based authentication against AD FS requires AD FS 3.0 or later which is available from Windows Server 2012 R2 onwards.
## Configure a Security Realm in Fusion
In Fusion, create a Security Realm for AD FS.
1. In the Fusion workspace, navigate to **System** > **Access Control**.
2. Click **Security Realms**.
3. Click **Add Security Realm**.
4. Enter the following information for the new realm:
* Enter a **Name**. The name must be unique and should be descriptive yet short.
* Select **saml** from the **Type** pulldown menu.
When you select a type, Fusion displays additional configuration options.
* The default value for **Enabled** is true. This setting controls whether or not Fusion allows user logins for this security realm.
* The default value for **Ephemeral Users** is false. When disabled, this setting prevents ephemeral users from being created in ZooKeeper during login. If enabled, this property negates **Auto Create Users**.
* The default value for **Auto Create Users** is true. If enabled, a user account is created automatically upon initial authentication. If disabled, then a Fusion user with admin permissions must create Fusion users.
Under **SAML Realm**, configure the following information:
* Enter the **Identity Provider URL**. This URL is used by the SAML authority for single sign-on. For example: `https://www.example.com/APP-PATH/adfs/ls`
* Enter the URL of the **IdP Issuer**. For example: `http://www.example.com/adfs/services/trust`.
* **IdP Issuer** must match `` in the SAML payload.
* Optional: Provide the **App Issuer**. This field is required if there is an `audienceRestriction` in the SAML assertion and must match `` in the SAML payload.
* In the **Certificate Fingerprint**, paste the contents of the SAML authority certificate, *without the certificate header and footer*.
* Optional: Enter the **User ID Attribute**. By default, the Fusion username is the same as the login name known to the Identity Provider. When another field or attribute in the user record stored by the IdP should be used as the Fusion username, that attribute name is the value of the User ID Attribute.
* Optional: Provide a **Post Login Redirect URL**. If not set, the Fusion URL is used.
* Optional: Provide a **Logout URL**.
1. Optional: Under **Groups Mapping**, specify the Group Name Attribute and add group mappings.
2. Click **Save**.
## Set up the application (relying party trust) in AD FS
Create a relying party trust in the AD FS Management console. For more details, refer to the [Microsoft AD FS documentation](https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-r2-and-2012/dn486828\(v=ws.11\)#to-create-a-relying-party-trust-manually).
The following screenshots may differ from your setup depending on the version of AD FS you’re using.
1. Launch the Add Relying Party Trust Wizard, then select **Claims aware**.
2. Choose the option to enter the data manually.
3. Set a Display name.
4. Configure URL. Enable the option for SAML 2.0 WebSSO protocol.
5. Set the **Relying party trust identifier** to the URL of the Fusion application.
6. Edit the Claim Rules to pass through the User Principal Name as the `NameID` claim.
7. In a terminal, create the OAuth AD FS `client` for the Fusion application:
```bash wrap theme={"dark"}
Add-ADFSClient -Name "Fusion OAuth" -ClientId "1234567890-ABCDEF" -RedirectUri="http://localhost:8080/oauthLogin"
```
The `ClientId` should be a GUID and the `RedirectUri` must point to the Fusion application URL, with `/oauthLogin` appended. Here, `localhost` is used for testing a Fusion application running on the local development machine.
## Configure the OAuth module for the application setup in AD FS
Create a new configuration file in `conf/security/oauth.conf`.
```yaml theme={"dark"}
client-id: 168f3ee4-63fc-4723-a61a-6473f6cb515d
adfs-url: https://your-adfs-server/adfs
resource: http://localhost:8080
high-trust: false
```
Replace the `client-id`, `adfs-url` and `resource` parameters for your environment.
* The `client-id` is the ID that was set up against AD FS using the `Add-ADFSClient` PowerShell command.
* The `adfs-url` is the URL of the AD FS server with the `/adfs` context appended.
* The `resource` is the relying party trust identifier set up in AD FS management.
* The `high-trust` parameter is only required when integrating the AD FS and SharePoint modules. This article is only concerned with authentication against AD FS.
## Test the authentication
When a user goes to the Fusion login page, they will be redirected to the AD FS OAuth login page.
After the user logs in, the user will be returned to Fusion as an authenticated user.
The Fusion user’s details will also be populated with any basic information available from the decoded OAuth token such as roles and the user principal name.
Enterprise customers may want to use a signed certificate from an Enterprise Certificate Authority (CA), such as GoDaddy or Let’s Encrypt for their Fusion 5 deployment.
The Fusion 5 default setup will create a self-signed Java Keystore (JKS) file that is used as the JSON Web Token (JWT) signer.
The following instructions detail how to create the `api-gateway.jks` file using a CA signed certificate.
## Prerequisites
* [Oracle Java JDK 1.8](https://www.oracle.com/java/technologies/downloads/)
## New installations
### 1. Create a JKS
Run the following command, replacing the `dname`(Distinguished Name) field with the name that matches your customer environment, and the `keypass` and `storepass` fields with a secure password.
This will create the `api-gateway.jks`:
`keytool -genkeypair -keyalg rsa -keysize 2048 -validity 3650 -dname "CN=sample, OU=sample, O=sample, L=sample, S=sample, C=sample" -storetype jks -keystore api-gateway.jks -keypass password -storepass password -alias jwt-signer`
If you change the `alias` or keystore/truststore passwords, you will need to update the api-gateway deployment YAML with the new values.
### 2. Create a Certificate Signing Request (CSR)
Run the following command to create a new `fusion.csr`, which will be sent to the CA:
`keytool -keystore api-gateway.jks -certreq -alias jwt-signer -file fusion.csr`
### 3. Send the CSR to your CA to get a signed certificate
The CA will return one of two possible responses:
* A **PKCS12** certificate file, such as `CAreply.pkcs`.
This is a full keystore containing all the keys you need, including the root certificate and the signed chain certificate.
* Several **PEM** files, such as `0000_cert.pem`, `0000_chain.pem`, and `0001_chain.pem`.
These files contain the root certificate, the chain certificate(s), and the signed chain certificate.
### 4. Import the signed certificate into the `api-gateway.jks`
* If you received a **PKCS** file, import it using the following keytool command:
`keytool -importcert -alias jwt-signer -file -keystore api-gateway.jks -trustcacerts`
* If you receive **PEM** files, run the following commands to import the keys:
* `keytool -importcert -alias root -file 0000_cert.pem -keystore api-gateway.jks -trustcacerts`
* `keytool -importcert -alias intermediate -file 0000_chain.pem -keystore api-gateway.jks -trustcacerts`
* `keytool -importcert -alias jwt-signer -file 0001_chain.pem -keystore api-gateway.jks -trustcacerts`
### 5. Upload the `api-gateway.jks` as a Kubernetes generic secret
The api-gateway service will pull down the `api-gateway.jks` file during initialization as a Kubernetes secret:
1. Delete the `-api-gateway.jks` secret if it already exists.
2. Upload the new secret with the following command:
`k create secret generic -api-gateway-jks --from-file api-gateway.jks`
### 6. Install Fusion 5 using Helm
See Deploy Fusion 5 on Other Kubernetes Platforms for more information.
## Existing installation
### 1. Create a JKS with your CA certificate or the java keytool
If you are using the Java keytool, you can run the same command used in the [Create a JKS](#1-create-a-jks) section:
`keytool -genkeypair -keyalg rsa -keysize 2048 -validity 3650 -dname "CN=sample, OU=sample, O=sample, L=sample, S=sample, C=sample" -storetype jks -keystore api-gateway.jks -keypass myPassword -storepass myPassword -alias jwt-signer`
### 2. Scale down Fusion deployment to 0
This step will allow Fusion services to find the latest JKS once it is scaled up at the final step.
See Deploy Fusion at Scale for more information on the process.
### 3. Delete the existing Kubernetes JKS secret for Fusion
As in the [Upload the `api-gateway.jks` as a Kubernetes generic secret](#5-upload-the-api-gateway-jks-as-a-kubernetes-generic-secret) section, delete the existing `-api-gateway.jks` secret.
### 4. Create a Kubernetes secret with the JKS using the name of the deployment
Upload a new secret using the api-gateway.jks file with the following command:
`k create secret generic -api-gateway-jks --from-file api-gateway.jks`
### 5. Scale up Fusion deployment
Fusion should accept the JKS once this step is complete.
Sometimes when you call out to HTTPS resources in connectors, index pipelines, query pipelines, or spark jobs, the HTTPS web resource is signed using an Enterprise or Self-Signed CA. This results in `Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target`.
You can ignore all SSL validation, but that’s not secure.
Whether it’s Spark, Connectors, Indexing, Query, or something else, adding trusted SSL certificates lets you can access these HTTPS web resources.
## Obtain each public certificate in .pem format.
The first thing you need is the public certificates in X.509 PEM format. Refer to [How to save a remote server SSL certificate locally as a file](https://superuser.com/questions/97201/how-to-save-a-remote-server-ssl-certificate-locally-as-a-file) for information on how to do this.
One example is to run:
```bash wrap theme={"dark"}
openssl s_client -connect self-signed.badssl.com:443 2>/dev/null /home/ndipiazza/Downloads/spark-ssl-test/badssl-com-chain.pem
```
This creates `/home/ndipiazza/Downloads/spark-ssl-test/badssl-com-chain.pem`, which is a public certificate from a public self-signed HTTPS site.
## Trust store certificate option
1. [Embed at runtime](#embed-each-enterprise-self-signed-certificate-to-a-local-trust-store-used-by-the-apache-httpclient-at-runtime) each enterprise/self-signed certificate to a local trust store used by the Apache HttpClient.
### Add each public certificate to the trust store
This option is only available in Fusion releases 5.9.2 and later.
Only use this option if you have installed Fusion using the `$tlsEnabled=true`, which follows this guide: Enable Transport Layer Security (TLS) for Fusion Microservices.
The steps are to:
1. Download the truststore from the `fusion-truststore` Kubernetes secret to a local folder.
2. Add any additional certificates to the truststore that you need using Java’s `keytool`.
3. Upload the updated truststore back to the secrets store.
At this point, you will no longer have the SSL verification errors when communicating with your HTTPS resource.
Here is a bash script for this first option:
```bash theme={"dark"}
# add a cert to the truststore
cert_to_add=/home/ndipiazza/Downloads/spark-ssl-test/badssl-com-chain.pem
kubectl get secret fusion-truststore \
--namespace=$NAMESPACE \
-o jsonpath="{['data']['truststore\\.jks']}" | \
base64 -d > truststore.jks
keytool -import -alias self-signed.badssl.com -keystore truststore.jks -file "${cert_to_add}" -storepass changeit
echo "\"$(cat truststore.jks | base64 -w 0)\"" > truststore.jks.base64
kubectl get secret fusion-truststore \
--namespace=$NAMESPACE \
-o json | \
jq --slurpfile enc truststore.jks.base64 '.data["truststore.jks"]=$enc[0]' | \
kubectl apply -f -
```
#### Verifying it works
Test access to this HTTPS resource with a Scala example. Below is an example Scala script that uses an Apache HttpClient that fetches a URL, but that URL is self-signed. If you you do not have this public certificate in your trust store, SSL validation will fail.
In this example, we attempt to talk to `self-signed.badssl.com`, which is a public site with a self-signed SSL certificate. Prior to adding the SSL certificate into the trust store, it fails. When you add the cert, it works. Save this script as **Collections → Job → Add → Script**, then save this job and run it.
```scala theme={"dark"}
import org.apache.commons.io.IOUtils
import org.apache.http.client.methods.HttpGet
import org.apache.http.impl.client.HttpClients
import java.io.{IOException, InputStream}
import java.security.cert.{CertificateFactory, X509Certificate}
import java.security.{GeneralSecurityException, KeyStore}
def url = "https://self-signed.badssl.com"
def createEmptyKeyStore(): KeyStore = {
val keyStore = KeyStore.getInstance("JKS")
keyStore.load(null, null);
keyStore;
}
@throws[IOException]
@throws[GeneralSecurityException]
def loadCertificate(publicCertIn: InputStream): X509Certificate = {
val factory = CertificateFactory.getInstance("X.509")
val cert = factory.generateCertificate(publicCertIn).asInstanceOf[X509Certificate]
cert
}
/**
* Returns the text content from a REST URL. Returns a blank String if there
* is a problem.
*/
def getRestContent(url:String): String = {
val httpClient = HttpClients.custom()
.build();
val httpResponse = httpClient.execute(new HttpGet(url))
val entity = httpResponse.getEntity
var content = ""
if (entity != null) {
val inputStream = entity.getContent
content = IOUtils.toString(inputStream)
inputStream.close()
}
httpClient.close()
content
}
val content = getRestContent(url)
println(content);
```
### Embed each enterprise/self-signed certificate to a local trust store used by the Apache HttpClient at runtime
This option embeds the public certificates for the trusted domains in the code, then inserts them into a local keystore used by Apache HttpClient.
Use this approach:
* If you do not have Kubernetes admin access to edit the keystore secret.
* If the HTTPS URL is isolated to a single job or pipeline stage, as it is simpler to do it this way versus adding it to the keystore.
#### How to use it
1. Use the X.509 certificate file created earlier and copy it as text. In your program, create a map of `{alias, pem file string}` called `trustedPublicCertificates` that contains your public certificates.
2. Replace `trustedPublicCertificates` with an alias name as the key of the map, then set the PEM file string contents as the value.
3. Use the Apache HttpClient to access the resource.
#### Examples
Here are examples for Apache HttpClient using a custom set of public keys in the SSL trust store to communicate with HTTPS sites with enterprise/self-signed certificates .
Examples:
* [Scala](#scala-example)
* [Nashorn JavaScript](#nashorn-javascript-example)
* [Java ](#java-example)
##### Scala Example:
```scala theme={"dark"}
import org.apache.commons.io.IOUtils
import org.apache.http.client.methods.HttpGet
import org.apache.http.conn.ssl.SSLConnectionSocketFactory
import org.apache.http.impl.client.HttpClients
import org.apache.http.ssl.SSLContexts
import java.io.{ByteArrayInputStream, IOException, InputStream}
import java.security.{GeneralSecurityException, KeyStore}
import java.security.cert.{CertificateFactory, X509Certificate}
import javax.net.ssl.HostnameVerifier
def url = "https://self-signed.badssl.com"
def trustedPublicCertificates = Map("sharepoint-local" -> "-----BEGIN CERTIFICATE-----\nMIIDeTCCAmGgAwIBAgIJAMnA8BB8xT6wMA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV\nBAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRYwFAYDVQQHDA1TYW4gRnJhbmNp\nc2NvMQ8wDQYDVQQKDAZCYWRTU0wxFTATBgNVBAMMDCouYmFkc3NsLmNvbTAeFw0y\nMTEwMTEyMDAzNTRaFw0yMzEwMTEyMDAzNTRaMGIxCzAJBgNVBAYTAlVTMRMwEQYD\nVQQIDApDYWxpZm9ybmlhMRYwFAYDVQQHDA1TYW4gRnJhbmNpc2NvMQ8wDQYDVQQK\nDAZCYWRTU0wxFTATBgNVBAMMDCouYmFkc3NsLmNvbTCCASIwDQYJKoZIhvcNAQEB\nBQADggEPADCCAQoCggEBAMIE7PiM7gTCs9hQ1XBYzJMY61yoaEmwIrX5lZ6xKyx2\nPmzAS2BMTOqytMAPgLaw+XLJhgL5XEFdEyt/ccRLvOmULlA3pmccYYz2QULFRtMW\nhyefdOsKnRFSJiFzbIRMeVXk0WvoBj1IFVKtsyjbqv9u/2CVSndrOfEk0TG23U3A\nxPxTuW1CrbV8/q71FdIzSOciccfCFHpsKOo3St/qbLVytH5aohbcabFXRNsKEqve\nww9HdFxBIuGa+RuT5q0iBikusbpJHAwnnqP7i/dAcgCskgjZjFeEU4EFy+b+a1SY\nQCeFxxC7c3DvaRhBB0VVfPlkPz0sw6l865MaTIbRyoUCAwEAAaMyMDAwCQYDVR0T\nBAIwADAjBgNVHREEHDAaggwqLmJhZHNzbC5jb22CCmJhZHNzbC5jb20wDQYJKoZI\nhvcNAQELBQADggEBAC4DensZ5tCTeCNJbHABYPwwqLUFOMITKOOgF3t8EqOan0CH\nST1NNi4jPslWrVhQ4Y3UbAhRBdqXl5N/NFfMzDosPpOjFgtifh8Z2s3w8vdlEZzf\nA4mYTC8APgdpWyNgMsp8cdXQF7QOfdnqOfdnY+pfc8a8joObR7HEaeVxhJs+XL4E\nCLByw5FR+svkYgCbQGWIgrM1cRpmXemt6Gf/XgFNP2PdubxqDEcnWlTMk8FCBVb1\nnVDSiPjYShwnWsOOshshCRCAiIBPCKPX0QwKDComQlRrgMIvddaSzFFTKPoNZjC+\nCUspSNnL7V9IIHvqKlRSmu+zIpm2VJCp1xLulk8=\n-----END CERTIFICATE-----")
def createEmptyKeyStore(): KeyStore = {
val keyStore = KeyStore.getInstance("JKS")
keyStore.load(null, null);
keyStore;
}
@throws[IOException]
@throws[GeneralSecurityException]
def loadCertificate(publicCertIn: InputStream): X509Certificate = {
val factory = CertificateFactory.getInstance("X.509")
val cert = factory.generateCertificate(publicCertIn).asInstanceOf[X509Certificate]
cert
}
/**
* Returns the text content from a REST URL. Returns a blank String if there
* is a problem.
*/
def getRestContent(url:String): String = {
val sslBuilder = SSLContexts.custom()
val keyStore = createEmptyKeyStore()
for ((alias,certificate) <- trustedPublicCertificates) {
keyStore.setCertificateEntry(alias, loadCertificate(new ByteArrayInputStream(certificate.getBytes)))
}
val sslContextBuilder = sslBuilder.loadTrustMaterial(keyStore, null)
val sslContext = sslContextBuilder.build()
val hostnameVerifier = null : HostnameVerifier
val sslConSocFactory = new SSLConnectionSocketFactory(sslContext, hostnameVerifier)
val httpClient = HttpClients.custom()
.setSSLSocketFactory(sslConSocFactory)
.build();
val httpResponse = httpClient.execute(new HttpGet(url))
val entity = httpResponse.getEntity
var content = ""
if (entity != null) {
val inputStream = entity.getContent
content = IOUtils.toString(inputStream)
inputStream.close()
}
httpClient.close()
content
}
val content = getRestContent(url)
println(content);
```
##### Nashorn JavaScript Example
```javascript expandable theme={"dark"}
var IOUtils = org.apache.commons.io.IOUtils;
var CloseableHttpResponse = org.apache.http.client.methods.CloseableHttpResponse;
var HttpGet = org.apache.http.client.methods.HttpGet;
var SSLConnectionSocketFactory = org.apache.http.conn.ssl.SSLConnectionSocketFactory;
var CloseableHttpClient = org.apache.http.impl.client.CloseableHttpClient;
var HttpClients = org.apache.http.impl.client.HttpClients;
var SSLContextBuilder = org.apache.http.ssl.SSLContextBuilder;
var SSLContexts = org.apache.http.ssl.SSLContexts;
var HostnameVerifier = javax.net.ssl.HostnameVerifier;
var SSLContext = javax.net.ssl.SSLContext;
var ByteArrayInputStream = java.io.ByteArrayInputStream;
var IOException = java.io.IOException;
var InputStream = java.io.InputStream;
var StandardCharsets = java.nio.charset.StandardCharsets;
var KeyStore = java.security.KeyStore;
var KeyStoreException = java.security.KeyStoreException;
var NoSuchAlgorithmException = java.security.NoSuchAlgorithmException;
var CertificateException = java.security.cert.CertificateException;
var CertificateFactory = java.security.cert.CertificateFactory;
var X509Certificate = java.security.cert.X509Certificate;
var HashMap = java.util.HashMap;
var Map = java.util.Map;
var TRUSTED_PUBLIC_CERTS = new HashMap();
TRUSTED_PUBLIC_CERTS.put("your-alias-here", "-----BEGIN CERTIFICATE-----\nMIIDPTCCAiW...GOwbrrZXosMs=\n-----END CERTIFICATE-----");
var url = "https://your-self-signed-site-or-whatever.local";
var sslBuilder = SSLContexts.custom();
var keyStore = createEmptyKeyStore();
for each (var e in TRUSTED_PUBLIC_CERTS.entrySet()) {
try {
keyStore.setCertificateEntry(e.getKey(), loadCertificate(new ByteArrayInputStream(e.getValue().getBytes())));
} catch (e) {
throw new RuntimeException(e);
}
}
var sslContextBuilder = sslBuilder.loadTrustMaterial(keyStore, null);
var sslContext = sslContextBuilder.build();
var sslConSocFactory = new SSLConnectionSocketFactory(sslContext, null);
var httpClient = HttpClients.custom()
.setSSLSocketFactory(sslConSocFactory)
.build();
var response = httpClient.execute(new HttpGet(url));
try {
if (response.getStatusLine().getStatusCode() != 200) {
throw new Exception("Bad status code " + response.getStatusLine());
}
print(IOUtils.toString(response.getEntity().getContent(), StandardCharsets.UTF_8));
} finally {
response.close();
httpClient.close();
}
function createEmptyKeyStore() {
var keyStore = KeyStore.getInstance("JKS");
keyStore.load(null, null);
return keyStore;
}
function loadCertificate(publicCertIn) {
var factory = CertificateFactory.getInstance("X.509");
return factory.generateCertificate(publicCertIn);
}
```
##### Java Example
```java expandable theme={"dark"}
import org.apache.commons.io.IOUtils;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.ssl.SSLContextBuilder;
import org.apache.http.ssl.SSLContexts;
import javax.net.ssl.HostnameVerifier;
import javax.net.ssl.SSLContext;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.security.KeyStore;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.cert.CertificateException;
import java.security.cert.CertificateFactory;
import java.security.cert.X509Certificate;
import java.util.HashMap;
import java.util.Map;
public class HttpClientWithTruststoreLoadedAtRuntime {
static final String url = "https://self-signed.badssl.com";
static final Map TRUSTED_PUBLIC_CERTS = new HashMap<>();
static {
// Put your certificate here
TRUSTED_PUBLIC_CERTS.put("your-alias-here" -> "-----BEGIN CERTIFICATE-----\nMIIDeTCCAmGgAwIBAgIJAMnA8BB8xT6wMA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV\nBAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRYwFAYDVQQHDA1TYW4gRnJhbmNp\nc2NvMQ8wDQYDVQQKDAZCYWRTU0wxFTATBgNVBAMMDCouYmFkc3NsLmNvbTAeFw0y\nMTEwMTEyMDAzNTRaFw0yMzEwMTEyMDAzNTRaMGIxCzAJBgNVBAYTAlVTMRMwEQYD\nVQQIDApDYWxpZm9ybmlhMRYwFAYDVQQHDA1TYW4gRnJhbmNpc2NvMQ8wDQYDVQQK\nDAZCYWRTU0wxFTATBgNVBAMMDCouYmFkc3NsLmNvbTCCASIwDQYJKoZIhvcNAQEB\nBQADggEPADCCAQoCggEBAMIE7PiM7gTCs9hQ1XBYzJMY61yoaEmwIrX5lZ6xKyx2\nPmzAS2BMTOqytMAPgLaw+XLJhgL5XEFdEyt/ccRLvOmULlA3pmccYYz2QULFRtMW\nhyefdOsKnRFSJiFzbIRMeVXk0WvoBj1IFVKtsyjbqv9u/2CVSndrOfEk0TG23U3A\nxPxTuW1CrbV8/q71FdIzSOciccfCFHpsKOo3St/qbLVytH5aohbcabFXRNsKEqve\nww9HdFxBIuGa+RuT5q0iBikusbpJHAwnnqP7i/dAcgCskgjZjFeEU4EFy+b+a1SY\nQCeFxxC7c3DvaRhBB0VVfPlkPz0sw6l865MaTIbRyoUCAwEAAaMyMDAwCQYDVR0T\nBAIwADAjBgNVHREEHDAaggwqLmJhZHNzbC5jb22CCmJhZHNzbC5jb20wDQYJKoZI\nhvcNAQELBQADggEBAC4DensZ5tCTeCNJbHABYPwwqLUFOMITKOOgF3t8EqOan0CH\nST1NNi4jPslWrVhQ4Y3UbAhRBdqXl5N/NFfMzDosPpOjFgtifh8Z2s3w8vdlEZzf\nA4mYTC8APgdpWyNgMsp8cdXQF7QOfdnqOfdnY+pfc8a8joObR7HEaeVxhJs+XL4E\nCLByw5FR+svkYgCbQGWIgrM1cRpmXemt6Gf/XgFNP2PdubxqDEcnWlTMk8FCBVb1\nnVDSiPjYShwnWsOOshshCRCAiIBPCKPX0QwKDComQlRrgMIvddaSzFFTKPoNZjC+\nCUspSNnL7V9IIHvqKlRSmu+zIpm2VJCp1xLulk8=\n-----END CERTIFICATE-----")
}
public static void main(String[] args) throws Exception {
SSLContextBuilder sslBuilder = SSLContexts.custom();
KeyStore keyStore = createEmptyKeyStore();
TRUSTED_PUBLIC_CERTS.forEach((alias, certificate) -> {
try {
keyStore.setCertificateEntry(alias, loadCertificate(new ByteArrayInputStream(certificate.getBytes())));
} catch (KeyStoreException | CertificateException e) {
throw new RuntimeException(e);
}
});
SSLContextBuilder sslContextBuilder = sslBuilder.loadTrustMaterial(keyStore, null);
SSLContext sslContext = sslContextBuilder.build();
SSLConnectionSocketFactory sslConSocFactory = new SSLConnectionSocketFactory(sslContext, (HostnameVerifier) null);
try (CloseableHttpClient httpClient = HttpClients.custom()
.setSSLSocketFactory(sslConSocFactory)
.build()) {
try (CloseableHttpResponse response = httpClient.execute(new HttpGet(url))) {
if (response.getStatusLine().getStatusCode() != 200) {
throw new Exception("Bad status code " + response.getStatusLine());
}
System.out.println(IOUtils.toString(response.getEntity().getContent(), StandardCharsets.UTF_8));
}
}
}
private static KeyStore createEmptyKeyStore()
throws KeyStoreException, CertificateException, IOException, NoSuchAlgorithmException {
KeyStore keyStore = KeyStore.getInstance("JKS");
keyStore.load(null, null);
return keyStore;
}
private static X509Certificate loadCertificate(InputStream publicCertIn) throws CertificateException {
CertificateFactory factory = CertificateFactory.getInstance("X.509");
return (X509Certificate) factory.generateCertificate(publicCertIn);
}
}
```
# Secrets management using ZooKeeper
Source: https://doc.lucidworks.com/docs/5/fusion/operations/security/zookeeper
This topic describes the legacy ZooKeeper-based secrets management approach in Fusion, where credentials are encrypted and stored directly in ZooKeeper. This method is maintained for backward compatibility and is suitable for development, test environments, and deployments that do not use Kubernetes secret management infrastructure.
For new deployments, use the [Secrets management using Fusion Vault](/docs/5/fusion/operations/security/external-vault) method.
V1 connectors, which use only this ZooKeeper approach, are [deprecated in Fusion 5.18.0 and scheduled to be removed in Fusion 6.0.0](/docs/5/fusion/release-notes/5.18/5.18.0-release-notes#deprecations).
## Benefits of ZooKeeper-based secrets management
Using ZooKeeper for secrets management, your organization can perform the following functions:
* Store credentials directly within Fusion without external dependencies.
* Automatic encryption at rest using ZooKeeper's built-in encryption.
* Self-contained configuration suitable for development and test environments.
* Backward compatibility with existing Fusion deployments.
* Simpler setup with no external vault provider required.
## Use cases
This section provides examples of use cases that illustrate how your organization can use ZooKeeper-based secrets management.
* A development team needs to quickly set up a test environment without configuring external vault providers. They can use ZooKeeper-based secrets to store database credentials and API keys directly in Fusion, with automatic encryption and redaction.
* An organization with existing Fusion deployments wants to maintain their current secrets management approach while evaluating migration to Fusion Vault. ZooKeeper-based secrets continue to work without changes.
* A small deployment without Kubernetes infrastructure needs to manage credentials securely. The ZooKeeper approach provides encrypted storage without requiring external secret management services.
## Config Sync
Config Sync is a Fusion service that synchronizes configuration files between Git repositories and Fusion. For information about how ZooKeeper-encrypted secrets are handled in version-controlled configuration files, see [Config Sync secrets management](/docs/5/fusion/operations/config-sync/secrets-management-used-in-config-sync).
# High-Performance Query Processing with Auto-Scaling
Source: https://doc.lucidworks.com/docs/5/fusion/operations/survival-guide/auto-scaling
[localhost link]: http://localhost:3000/docs/5/fusion/operations/survival-guide/auto-scaling
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/survival-guide/auto-scaling
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/146
To further illustrate key concepts about the Fusion 5 architecture, this section describes how query execution works and the microservices involved. Note some important details:
import from '/snippets/LwTemplate.jsx';
* There are many microservices involved in query execution. This illustrates the value and importance of having a robust orchestration layer like Kubernetes.
* Fusion comes pre-configured. You don’t need to configure all details depicted below.
* **Point A** - Background Spark jobs aggregate signals—which powers the signal boosting stage—and analyzes signals for [query rewriting operations](/docs/5/fusion/getting-data-out/query-enhancement/query-rewriting). This includes head/tail, synonym detection, and misspelling detection.
* **Point B** - Fusion uses a [Solr auto-scaling policy](https://solr.apache.org/guide/solrcloud-autoscaling-overview.html) in conjunction with Kubernetes node pools to govern replica placement for various Fusion collections. For example, to support high performance query traffic, place the primary collection with sidecar collections for query rewriting, signal boosting, and rules matching. Solr pods supporting high volume, low-latency reads are backed by an HPA linked to CPU or custom metrics in Prometheus. Fusion services store configurations, such as query pipeline definitions, in ZooKeeper (**Point C**).
* **Point 1** - A query request comes into the cluster from a [Kubernetes Ingress](https://cloud.google.com/kubernetes-engine/docs/concepts/ingress). The Ingress routes requests to the Fusion API Gateway service. The gateway performs authentication and authorization to ensure the user has the correct permissions to execute the query. The Fusion API Gateway load-balances requests across multiple query pipeline services using native Kubernetes service discovery (**Point 2**).
* **Point 3** - The gateway issues a JWT, which is sent to downstream services. An internal JWT has identifying information about a user, including their roles and permissions, to allow Fusion services to perform fine-grained authorization. The JWT is returned as a Set-Cookie header to improve performance of later requests. Alternatively, API requests can use the `/oauth2/token` endpoint in the gateway to retrieve the JWT using OAuth2 semantics.
* **Point 4** - The query service executes the pipeline stages to enrich the query, before sending it to the primary collection. Typically, this involves multiple lookups to sidecar collections—such as the `APP_NAME_query_rewrite` collection—to perform spell correction, synonym expansion, and rules matching. The query pipeline may also call out to the Fusion ML Model service to generate predictions, such as deciding query intent. The ML Model service may also use an HPA tied to CPU, to scale out and support the desired queries per second (**Point 5**).
* **Point 6** - After executing the query in the primary collection, Fusion generates a **response** signal to track query request parameters and Solr response metrics, such as `numFound` and `qTime`. Fusion stores raw signals in the **signals** collection, which typically runs in the analytics partition to support high-volume writes.
* **Point 7** - Every Fusion microservice exposes detailed metrics. Prometheus scrapes the metrics using pod annotations. The query microservice exposes stage-specific metrics to help you understand query performance.
* **Point 8** - Every Fusion microservice sends logs to Logstash. You can configure Logstash to index log messages into the `system_logs` collection in Solr or to an external service like Elastic.
The course for **Autoscaling in Fusion** focuses on understanding how autoscaling works in Fusion.
# GPU Support for Deep Learning Training
Source: https://doc.lucidworks.com/docs/5/fusion/operations/survival-guide/gpu-support-deep-learning
[localhost link]: http://localhost:3000/docs/5/fusion/operations/survival-guide/gpu-support-deep-learning
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/survival-guide/gpu-support-deep-learning
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/dmyrqm
In Fusion 5.3 and later, jobs that involve training deep learning-based models automatically use GPU resources for training if deployed on a GPU-enabled node. Your cloud provider likely has a custom set of nodeSelectors and tolerations required to map these jobs to their GPU compute. The following section provides an example using Smart Answers and GKE.
### Training Smart Answers on GPU with GKE
To use GPU resources within GKE for Smart Answers training, create a GPU resource within your cluster.
Create a new nodepool with a pre-emptible GPU node that will spin down when not in use. Give the nodepool a label of `node_pool:gpu`. By default, GKE will also add a taint of `nvidia.com/gpu:present=NoSchedule`. Consider that fact when updating your Helm chart values.
Additionally, add a specific resource limit of `nvidia.com/gpu: 1 `. (This value is specific to GKE.) Create another standard nodepool without GPU resources with a label of `node_pool:deploy` for the eventual Seldon Core deployment.
In your custom values YAML file, add:
```yaml wrap expandable theme={"dark"}
question-answering:
nodeSelector:
default:
cloud.google.com/gke-nodepool: gpu-nodepool
supervised:
seldon:
cloud.google.com/gke-nodepool: cpu-nodepool
coldstart:
seldon:
cloud.google.com/gke-nodepool: cpu-nodepool
tolerations:
default:
- key: "nvidia.com/gpu"
operator: "Equal"
value: "present"
effect: "NoSchedule"
supervised:
seldon: []
coldstart:
seldon: []
resources:
supervised:
train:
requests:
nvidia.com/gpu: 1
limits:
nvidia.com/gpu: 1
coldstart:
train:
requests:
nvidia.com/gpu: 1
limits:
nvidia.com/gpu: 1
```
This setup deploys all workflow steps onto the GPU node *except* for the Seldon Core deployment. As the deployment will live on after the workflow has completed, assigning the Seldon Core deployment to the GPU node it would prevent GKE from spinning the GPU node down. This increases operating expense.
### Setting up Milvus on GPU with GKE
Setting up Milvus on GPU first requires the creation of a GPU resource.
At the end of the `ml-model-service` section of your custom values YAML file, add a section for Milvus as shown below:
```yaml wrap expandable theme={"dark"}
ml-model-service:
# ml-model-service yaml settings:
# ...
# followed by the Milvus settings:
milvus:
gpu:
enabled: true
image:
repository: milvusdb/milvus
tag: 0.10.2-gpu-d081520-8a2393
pullPolicy: "IfNotPresent"
resources:
requests:
nvidia.com/gpu: 1
limits:
nvidia.com/gpu: 1
nodeSelector:
cloud.google.com/gke-nodepool: gpu-nodepool
tolerations:
- key: "nvidia.com/gpu"
operator: "Equal"
value: "present"
effect: "NoSchedule"
```
The example above assumes the name of the GPU nodepool is `gpu-nodepool`.
The taints/tolerations and resource keys shown below are for a GKE setup. These values may vary depending on your cloud provider.
# Multiple Zones for High Availability
Source: https://doc.lucidworks.com/docs/5/fusion/operations/survival-guide/high-availability
[localhost link]: http://localhost:3000/docs/5/fusion/operations/survival-guide/high-availability
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/survival-guide/high-availability
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/151
Fusion relies on ZooKeeper to keep quorum. As a result, you need at least three ZooKeeper pods in an ensemble. ZooKeeper deploys as a StatefulSet in the Fusion Helm chart. With GKE, you can launch a [regional cluster](https://cloud.google.com/kubernetes-engine/docs/concepts/regional-clusters) that distributes nodes across three availability zones. With a multi-zone setup, your cluster can withstand the loss of one zone without experiencing downtime. However, you may experience degraded performance from losing one third of your total compute capacity, assuming your pods are evenly distributed. The multi-zone cluster also ensures higher availability for the Kubernetes control plane services.
Lucidworks provides affinity rules to ensure multiple pods per service get distributed across multiple zones. See **Configure Pod Affinity** for details on configuring.
Affinity rules govern how Kubernetes schedules pods for Fusion components across the cluster. All components have the same affinity setup, which follows this logic:
* When scheduling, prefer to put a pod on a node in an availability zone that doesn’t have a running instance of this component.
* Require that pods are deployed on a host that doesn’t have a running instance of the component that is being scheduled.
With this logic, the loss of a host can only bring down one component at most. However, the cluster must be at least as large as the number of replicas in the largest deployment.
To run a large number of a component, consider relaxing the "required" policy by changing it to a "preferred" policy on the hostname for the `kubernetes.io/hostname` policies:
| | |
| - | - |
| Before | requiredDuringSchedulingIgnoredDuringExecution: |
| After | preferredDuringSchedulingIgnoredDuringExecution: |
If you used the `--with-affinity-rules` option when running the `./customize_fusion_values.sh` script, the pod affinity rules are configured for your cluster. Alternatively, copy [`affinity.yaml`](https://github.com/lucidworks/fusion-cloud-native/blob/master/example-values/affinity.yaml), and rename it using the following naming convention: `___fusion_affinity.yaml`.
To implement the file, append the following to your upgrade script:
```
MY_VALUES="${MY_VALUES} --values gke_search_f5_fusion_affinity.yaml"
```
When running in a multi-zone cluster, each Solr node has a `solr_zone` system property set to the zone it is running in, such as `-Dsolr_zone=us-west1-a`. This guide covers how to use the `solr_zone` property to distribute replicas across zones in the **Deploy Fusion at Scale** section. Setting the `solr_zone` property for Solr pods requires the Solr service account to have a ClusterRoleBinding that allows it to get node metadata from the Kubernetes API service.
Before you begin, see [Fusion Server Deployment](/docs/5/fusion/operations/deployment) to understand the architecture and requirements.
This article explains how to plan and execute a Fusion deployment at the scale required for staging or production.
While the `setup_f5_*.sh` scripts are handy for getting started and proof-of-concept purposes, this article covers the planning process for building a production-ready environment.
The course for **Preparing for Fusion Implementation** focuses on the key elements for a successful implementation, defining your business requirements, preparing clean data, and involving the right personnel.
## Prerequisites
You must meet the following prerequisites before you can customize your Fusion cluster:
* A local copy of the [fusion-cloud-native repository](https://github.com/lucidworks/fusion-cloud-native). This must be up-to-date with the latest master branch.
* Any cloud provider-specific **command line tools**, such as `gcloud` or `aws`, and `kubectl`.\
See the platform-specific instructions linked above, or check with your cloud provider.
* Helm v3
* To install on a Mac:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
* For other operating systems, download from [Helm Releases](https://github.com/helm/helm/releases).
* Verify your installation:
```bash theme={"dark"}
helm version --short
v3.0.0+ge29ce2a
```
* Kubernetes namespace
* Collect the following information about your Kubernetes environment:
* **CLUSTER**: Cluster name (passed to our setup scripts using the `-c` arg)
* **NAMESPACE**: Kubernetes namespace where to install Fusion; a namespace should only contain lowercase letters (a-z), digits (0-9), or dash. No periods or underscores allowed.
* *(optional)* Clarify your organization’s DockerHub policy. The Fusion Helm chart points to public Docker images on DockerHub. Your organization may not allow Kubernetes to pull images directly from DockerHub or may require extra security scanning before loading images into production clusters.\
Consult your Kubernetes and Docker admin team to find how to get the Fusion images loaded into a registry that’s accessible to your cluster. You can update the image for each service using the [custom values YAML file](#custom-values-yaml-file).
**Kubernetes namespace tips**
* Fusion 5 service discovery requires all services for the same release be deployed in the same namespace. Moreover, you should only run one instance of Fusion in a namespace. If you need multiple instances of Fusion running in the same Kubernetes cluster, then you need to deploy them in separate namespaces.
* If your organization requires CPU / Memory quotas for namespaces, you can start with a minimum of 12 CPU and 45GB of RAM (such as 3 x n1-standard-4 on GKE), but you will need to increase the quotas once you start load testing Fusion with production workloads and real datasets.
* Fusion requires at least 3 ZooKeeper nodes and 2 Solr nodes to achieve high availability.
## Custom values YAML file
1. Clone the `fusion-cloud-native` repository: `git clone https://github.com/lucidworks/fusion-cloud-native`
2. Run the `customize_fusion_values.sh` script.
```bash theme={"dark"}
./customize_fusion_values.sh --provider -c -n \
--num-solr 3 \
--solr-disk-gb 100 \
--node-pool \
--prometheus true \
--with-resource-limits \
--with-affinity-rules
```
Pass the `--help` parameter to see script usage details.
The script creates the following files:
| File | Description |
| - | - |
| `___fusion_values.yaml` | Main custom values YAML used to override Helm chart defaults for Fusion microservices. |
| `___monitoring_values.yaml` | Custom values yaml used to configure Prometheus and Grafana. |
| `___fusion_resources.yaml` | Resource requests and limits for all Microservices. |
| `___fusion_affinity.yaml` | Pod affinity rules to ensure multiple replicas for a single service are evenly distributed across zones and nodes. |
| `___upgrade_fusion.sh` | Script used to install and/or upgrade Fusion using the aforementioned custom values YAML files. |
For an explanation of these placeholder values, see [Configuration Values](#custom-values-yaml-file) below.
3. Add the new files to version control. You will make changes to it over time as you fine-tune your Fusion installation. You will also need it to perform upgrades. If you try to upgrade your Fusion installation and don’t provide the custom values YAML, your deployment will revert to chart defaults.\
Review the `___fusion_values.yaml` file to familiarize yourself with its structure and contents. Notice it contains a separate section for each of the Fusion microservices. The example configuration of the `query-pipeline` service below illustrates some important concepts about the custom values YAML file.
```yaml highlight={1,2,3,5,6} theme={"dark"}
query-pipeline:
enabled: true
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
javaToolOptions: "..."
pod:
annotations:
prometheus.io/port: "8787"
prometheus.io/scrape: "true"
prometheus.io/path: "/actuator/prometheus"
```
4. Service-specific setting overrides under the top-level heading
5. Every Fusion service has an implicit enabled flag that defaults to true, set to false to remove this service from your cluster
6. Node selector identifies the label find nodes to schedule pods on
7. Used to pass JVM options to the service
8. Pod annotations to allow Prometheus to scrape metrics from the service
Once we go through all of the configuration topics in this topic, you’ll have a well-configured custom values YAML file for your Fusion 5 installation. You’ll then use this file during the Helm v3 installation at the end of this topic.
### Deployment-specific values
The script creates a custom values YAML file using the naming convention: `___fusion_values.yaml`. For example, `gke_search_f5_fusion_values.yaml`.
| Parameter | Description |
| - | - |
| `` | The K8s platform you’re running on, such as `gke`. |
| `` | The name of your cluster. |
| `` | The K8s namespace where you want to install Fusion. |
| `` | Specifies a `nodeSelector` label to find nodes to schedule Fusion pods on. |
Providing the correct `--node-pool ` label is very important. Using the wrong value will cause your pods to be stuck in the `pending` state. If you’re not sure about the correct value for your cluster, pass ’'\` to let Kubernetes decide which nodes to schedule Fusion pods on.
Default `nodeSelector` labels are provider-specific. The `fusion-cloud-native` scripts use the following defaults for GKE and EKS:
| Provider | Default node selector |
| - | - |
| GKE | cloud.google.com/gke-nodepool: default-pool |
| EKS | alpha.eksctl.io/nodegroup-name: standard-workers |
If you are deploying Fusion 5.9.12, add the following to your `values.yaml` file to avoid a known issue that prevents the `kuberay-operator` pod from launching successfully: `yaml kuberay-operator: crd: create: true `
### Flags
The script provides flags for additional configuration:
| Flag | Description |
| - | - |
| `--node-pool` | Add a Fusion specific label to your nodes. |
| `--with-resource-limits` | Configure resource requests/limits. |
| `--with-replicas` | Configure replica counts. |
| `--with-affinity-rules` | Configure pod affinity rules for Fusion services. |
Use `--node-pool` to add a Fusion specific label to your nodes by doing:
```bash theme={"dark"}
kubectl label fusion_node_type=
```
Then, pass `--node-pool 'fusion_node_type: '`.
## Configure Solr sizing
When you’re ready to build a production-ready setup for Fusion 5, you need to customize the Fusion Helm chart to ensure Fusion is well-configured for production workloads.
You’ll be able to scale the number of nodes for Solr up and down after building the cluster, but you need to establish the initial size of the nodes (memory and CPU) and the size and type of disks you need.
See the example config below to learn which parameters to change in the custom values YAML file.
```yaml expandable theme={"dark"}
solr:
resources: # Set resource limits for Solr to help K8s pod scheduling;
limits: # these limits are not just for the Solr process in the pod,
cpu: "7700m" # so allow ample memory for loading index files into the OS cache (mmap)
memory: "26Gi"
requests:
cpu: "7000m"
memory: "25Gi"
logLevel: WARN
nodeSelector:
fusion_node_type: search # Run this Solr StatefulSet in the "search" node pool
exporter:
enabled: true # Enable the Solr metrics exporter (for Prometheus) and
# schedule on the default node pool (system partition)
podAnnotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9983"
prometheus.io/path: "/metrics"
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
image:
tag: 8.4.1
updateStrategy:
type: "RollingUpdate"
javaMem: "-Xmx3g -Dfusion_node_type=system" # Configure memory settings for Solr
solrGcTune: "-XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UseStringDeduplication -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=150 -XX:+UseLargePages -XX:+AlwaysPreTouch"
volumeClaimTemplates:
storageSize: "100Gi" # Size of the Solr disk
replicaCount: 6 # Number of Solr pods to run in this StatefulSet
zookeeper:
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
replicaCount: 3 # Number of Zookeepers
persistence:
size: 20Gi
resources: {}
env:
ZK_HEAP_SIZE: 1G
ZOO_AUTOPURGE_PURGEINTERVAL: 1
```
To be clear, you can tune GC settings and number of replicas after the cluster is built. But changing the size of the persistent volumes is more complicated so you should try to pick a good size initially.
### Configure storage class for Solr pods (optional)
If you wish to run with a storage class other than the default you can create a storage class for your Solr pods before you install. For example, to create regional disks in GCP you can create a file called `storageClass.yaml` with the following contents:
```yaml theme={"dark"}
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: solr-gke-storage-regional
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-standard
replication-type: regional-pd
zones: us-west1-b, us-west1-c
```
and then provision into your cluster by calling:
```bash theme={"dark"}
kubectl apply -f storageClass.yaml
```
to then have Solr use the storage class by adding the following to the custom values YAML:
```yaml theme={"dark"}
solr:
volumeClaimTemplates:
storageClassName: solr-gke-storage-regional
storageSize: 250Gi
```
We’re not advocating that you must use regional disks for Solr storage, as that would be redundant with Solr replication. We’re just using this as an example of how to configure a custom storage class for Solr disks if you see the need. For instance, you could use regional disks without Solr replication for write-heavy type collections.
## Configure multiple node pools
Lucidworks recommends isolating search workloads from analytics workloads using multiple node pools. The included scripts do not do this for you; this is a manual process.
See the example script for GKE, see [create\_gke\_cluster\_node\_pools.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/additional_environments/create_gke_cluster_node_pools.sh).
In the custom values YAML file, you can add additional Solr StatefulSets by adding their names to the list under the `nodePools` property. If any property for that statefulset needs to be changed from the default set of values, then it can be set directly on the object representing the node pool, any properties that are omitted are defaulted to the base value. See the following example (additional whitespace added for display purposes only):
```yaml expandable highlight={3,4,9,17,22,31} theme={"dark"}
solr:
nodePools:
- name: ""
- name: "analytics"
javaMem: "-Xmx6g"
replicaCount: 6
storageSize: "100Gi"
nodeSelector:
fusion_node_type: analytics
resources:
requests:
cpu: 2
memory: 12Gi
limits:
cpu: 3
memory: 12Gi
- name: "search"
javaMem: "-Xms11g -Xmx11g"
replicaCount: 12
storageSize: "50Gi"
nodeSelector:
fusion_node_type: search
resources:
limits:
cpu: "7700m"
memory: "26Gi"
requests:
cpu: "7000m"
memory: "25Gi"
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
...
```
1. The empty string `""` is the suffix for the default partition.
2. Overrides the settings for the **analytics** Solr pods.
3. Assigns the **analytics** Solr pods to the node pool and attaches the label `fusion_node_type=analytics`. You can use the `fusion_node_type` property in Solr auto-scaling policies to govern replica placement during collection creation.
4. Overrides the settings for the **search** Solr pods.
5. Assigns the **search** Solr pods to the node pool and attaches the label `fusion_node_type=search`.
6. Sets the default settings for all Solr pods, if not specifically overridden in the `nodePools` section above.
Do not edit the `nodePools` value `""`.
In the example above, the **analytics** partition `replicaCount`, or number of Solr pods, is six. The **search** partition `replicaCount` is twelve.
Each nodePool is automatically be assigned the -Dfusion\_node\_type property of ``, ``, or ``. This value matches the name of the nodePool. For example, `-Dfusion_node_type=`.
The Solr pods have a `fusion_node_type` system property, as shown below:
## Solr auto-scaling policy
Use [replica placement plugins](https://solr.apache.org/guide/solr/latest/configuration-guide/replica-placement-plugins.html) to control how replicas are placed in Solr.
## Pod network policy
A Kubernetes network policy governs how groups of pods communicate with each other and other network endpoints. With Fusion, all incoming traffic flows through the API Gateway service. All Fusion services in the same namespace expect an internal JWT, which is supplied by the Gateway, as part of the request. As a result, Fusion services enforce a basic level of API security and don’t need an additional network policy to protect them from other pods in the cluster.
To install the network policy for Fusion services, pass `--set global.networkPolicyEnabled=true` when installing the Fusion Helm chart.
## On-premises private Docker registries
For on-premises Kubernetes deployments, your organization may not allow Kubernetes to pull [Fusion’s Docker images from DockerHub](https://hub.docker.com/u/lucidworks/). See the instructions below for details on using a private Docker registry with Fusion. These are general instructions that may need to be adapted to work within your organization’s security policies:
1. Transfer the public images from DockerHub to your private Docker registry.
2. Establish a workstation that has access to [DockerHub](https://hub.docker.com). This workstation must connect to your internal Docker registry, most likely via VPN connection. In this example, the workstation is referred to as `envoy`.
3. Install Docker on `envoy`. You need at least 100GB of free disk for Docker.
4. Pull all of the images from DockerHub to `envoy`’s local registry. For example, to pull the query pipeline image, run `docker pull lucidworks/query-pipeline:5.9.0`. See `docker pull --help` for more information about pulling Docker images.
5. Establish a connection from `envoy` to the private Docker registry, most likely via a VPN connection. In this example, the private Docker registry is referred to as ``.
6. Push the images from `envoy`’s Docker registry to the private registry. This will take a long time.
1. You’ll need to re-tag all images for the internal registry. For example, to tag the query-pipeline image, run:
```bash theme={"dark"}
docker tag lucidworks/query-pipeline:5.9.0 /query-pipeline:5.9.0
```
2. Push each image to the internal repo:
```bash theme={"dark"}
docker push /query-pipeline:5.9.0
```
7. Install the Docker registry secret in Kubernetes. Create the Docker registry secret in the Kubernetes namespace where you want to install Fusion:
```bash theme={"dark"}
SECRET_NAME=
REPO=
kubectl create secret docker-registry "${SECRET_NAME}" \
--namespace "${NAMESPACE}" \
--docker-server="${REPO}" \
--docker-username=${REPO_USER} \
--docker-password=${REPO_PASS} \
--docker-email=${REPO_USER}
```
For details, see the Kubernetes article [Pull an Image from a Private Registry](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/).
8. Update the custom values YAML for your cluster to point to your private registry and secret to allow Kubernetes to pull images. For example:
```yaml theme={"dark"}
query-pipeline:
image:
imagePullSecrets:
- name:
repository:
```
Repeat the process for all Fusion services.
### Customize Helm Chart
Every Fusion service allows you to override the `imagePullSecrets` setting using custom values YAML. However, other 3rd party services--including Zookeeper, Pulsar, Prometheus, and Grafana--don’t allow you to supply the pull secret using the custom values YAML.
To patch the default service account for your namespace and add the pull secret, run the following:
```bash theme={"dark"}
kubectl patch sa default -n $NAMESPACE \
-p '"imagePullSecrets": [{"name": "" }]'
```
In Windows using PowerShell or another CLI, you might have to escape the double quotes with a backslash (`\`) or reverse the order of single and double quotes:
```bash theme={"dark"}
kubectl patch sa default -n $NAMESPACE \
-p "'imagePullSecrets': [{'name': ''}]"
```
Replace `` with the name of the secret you created in the steps above.
This allows the default service account to pull images from the private registry without specifying the pull secret on the resources directly.
## Add additional trusted certificate(s) to Fusion’s indexing and querying services *(optional)*
You can add custom trusted certificates to support Fusion’s indexing and querying services. You may want to use custom trusted certificates if, for example, you have specific security requirements for data handling or need to support an existing infrastructure and its security needs. This method involves updating your Helm chart.
If you want to add custom trusted certificates for both the indexing and querying services, follow these instructions twice: once for the indexing service, and once for the querying service. To add different certificates to the indexing and querying services, create one YAML file with the indexing service certificates and one YAML file for the querying service certificates before following these instructions.
You may use the same YAML file if you want to use the same certificates for both services.
To add custom trusted certificates:
1. Create a new YAML file for your custom trusted certificates. The `customcerts.yaml` file is the example file in these instructions.
2. Add the custom certificate(s) in the YAML file created in the previous step. For example:
```yaml theme={"dark"}
trustedCertificates:
enabled: true
files:
some.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE-----
other.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE---------
```
3. Update the indexing or querying service by running the following Helm command. Replace `EXAMPLE-VALUES-FILE.yaml` with your previous values file.
```bash theme={"dark"}
helm upgrade --install --namespace ${EXAMPLE-NAMESPACE} ${HELM-RELEASE} ${HELM-CHART-PATH} --values EXAMPLE-VALUES-FILE.yaml --values customcerts.yaml
```
4. Verify the indexing or querying pod has a new `init-container` with the name `import-certs`.
## Add additional trusted certificate(s) for connectors to allow crawling of web resources with SSL/TLS enabled *(optional)*
To crawl a datasource which for some reason is using a self-signed certificate, add arbitrary certificates to connectors. For example:
```yaml wrap expandable theme={"dark"}
classic-rest-service:
trustedCertificates:
enabled: true
files:
some.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE-----
other.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE---------
connector-plugin:
trustedCertificates:
enabled: true
files:
some.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE-----
other.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE---------
```
### Generating the certificate on linux command line
Use the following command to generate a `.crt` file in `$fusion_home/apps/jetty/connectors/etc/yourcertname.crt`:
```bash theme={"dark"}
openssl s_client -servername remote.server.net -connect remote.server.net:443 $fusion_home/apps/jetty/connectors/etc/yourcertname.crt
```
### Generating the certificate using Firefox web browser
1. Navigate to the SharePoint host.
2. Click the in the address bar, then click the icon.
3. Next, navigate to **More Information** > **View Certificate** > **Export**.\
Save the file to the following folder:
`$fusion_home/apps/jetty/connectors/etc/yourcertname.crt`
### Generating the certificate using Chrome web browser
1. Navigate to **Chrome menu** > **More Tools** > **Developer Tools** > **Security Tab**.
This will display the **Security overview**.
2. Click the **View certificate** button.
3. Save the file to the following folder:
`$fusion_home/apps/jetty/connectors/etc/yourcertname.crt`
### Generating the certificate using powershell
Use the following script to generate a `.crt`` file in `\$fusion\_home\apps\jetty\connectors\etc\yourcertname.crt\`\`:
```bash theme={"dark"}
$fusion_home = c:\your\fusion\install\directory
$webRequest = [Net.WebRequest]::Create("https://your-hostname")
try { $webRequest.GetResponse() } catch {}
$cert = $webRequest.ServicePoint.Certificate
$bytes = $cert.Export([Security.Cryptography.X509Certificates.X509ContentType]::Cert)
set-content -value $bytes -encoding byte -path "$fusion_home\apps\jetty\connectors\etc\yourcertname.binary.crt"
certutil -encode "$fusion_home\apps\jetty\connectors\etc\yourcertname.binary.crt" "$fusion_home\apps\jetty\connectors\etc\yourcertname.crt"
rm "$fusion_home\apps\jetty\connectors\etc\yourcertname.binary.crt" -f
```
## Install Fusion 5 on Kubernetes
At this point, you’re ready to install Fusion 5 using the custom values YAML files and upgrade script. If you used the `customize_fusion_values.sh` script, run it using BASH:
```bash theme={"dark"}
./gke_search_f5_upgrade_fusion.sh
```
Once the installation is complete, verify your Fusion installation is running correctly.
## Monitoring Fusion with Prometheus and Grafana
Lucidworks recommends using Prometheus and Grafana for monitoring the performance and health of your Fusion cluster. Your operations team may already have these services installed. If not, install them into the Fusion namespace.
The [Custom values YAML file shown above](#custom-values-yaml-file) activates the Solr metrics exporter service and adds pod annotations so Prometheus can scrape metrics from Fusion services.
1. Run the `customize_fusion_values.sh` script with the `--prometheus true` option. This creates an extra custom values YAML file for installing Prometheus and Grafana, `___monitoring_values.yaml`. For example: `gke_search_f5_monitoring_values.yaml`.
2. Commit the YAML file to version control.
3. Review its contents to ensure that the settings suit your needs. For example, decide how long you want to keep metrics. The default is 36 hours.\
See the [Prometheus documentation](https://github.com/helm/charts/tree/master/stable/prometheus) and [Grafana documentation](https://github.com/helm/charts/tree/master/stable/grafana) for details.
4. Run the `install_prom.sh` script to install Prometheus & Grafana in your namespace. Include the provider, cluster name, namespace, and helm release as in the example below:
```bash theme={"dark"}
./install_prom.sh --provider gke -c search -n f5 -r 5-5-1
```
Pass the `--help` parameter to see script usage details.
The Grafana dashboards from [monitoring/grafana](https://github.com/lucidworks/fusion-cloud-native/tree/master/monitoring/grafana) are installed automatically by the `install_prom.sh` script.
# Ingress and Security
Source: https://doc.lucidworks.com/docs/5/fusion/operations/survival-guide/ingress
[localhost link]: http://localhost:3000/docs/5/fusion/operations/survival-guide/ingress
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/survival-guide/ingress
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/152
The Fusion proxy service serves as an API gateway and provides authentication and authorization. Route all external access to Fusion services through the Fusion proxy service. The most common approach is to set up a [Kubernetes Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) that routes Fusion service requests to the proxy service as shown in the example ingress definition below. It’s also common to do [TLS termination](https://cloud.google.com/load-balancing/docs/https/#tls_support) at the Ingress, so that all traffic to and from the Kubernetes cluster is encrypted.
Internal requests happen over unencrypted HTTP.
```yaml wrap expandable theme={"dark"}
apiVersion: v1
items:
- apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
...
labels:
...
name: -api-gateway
namespace:
spec:
rules:
- host:
http:
paths:
- backend:
serviceName: proxy
servicePort: 6764
path: "/*"
tls:
- hosts:
-
secretName: -api-gateway-tls-secret
status:
loadBalancer:
ingress:
- ip:
```
## EKS
For deployments that run on EKS, the [setup\_f5\_eks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_eks.sh) setup script creates a classic elastic load balancer (ELB) to expose the Fusion proxy service for demonstration purposes. However, you’re strongly encouraged to configure a Kubernetes Ingress with TLS termination in front of the proxy service. See [How do I terminate HTTPS traffic on Amazon EKS workloads with ACM?](https://aws.amazon.com/premiumsupport/knowledge-center/terminate-https-traffic-eks-acm/) for more information.
You can also opt to use an [Application Load balancer](https://github.com/lucidworks/fusion-cloud-native#eks-ingress) instead of the proxy service.
## AKS
AKS deployments can use the [setup\_f5\_aks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_aks.sh) setup script, which exposes the Fusion proxy service on an external IP using HTTP for demonstration purposes. We recommend that you configure a [Kubernetes Ingress with TLS termination](https://github.com/lucidworks/fusion-cloud-native#aks-ingress) in front of the proxy service.
## Other Kubernetes Platforms
If your deployment runs on OpenShift using a cluster created with OpenShift Infrastructure Provider, the [setup\_f5\_k8s.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_k8s.sh) setup script will install Fusion 5 on your existing cluster. OpenShift Online services are not supported.
See the [RedHat OpenShift](https://github.com/lucidworks/fusion-cloud-native#redhat-openshift) section of the readme for more tips toward a successful installation.
If your deployment runs on GKE, the [setup\_f5\_gke.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_gke.sh) setup script in the `fusion-cloud-native` repo provides the option to create the [Ingress and TLS cert](https://github.com/lucidworks/fusion-cloud-native#gke-ingress-and-tls).
This process uses [Let’s Encrypt](https://letsencrypt.org/).
For other Kubernetes providers, refer to provider’s documentation on creating an Ingress and TLS certificates.
# Stateless Sessions with JWT
Source: https://doc.lucidworks.com/docs/5/fusion/operations/survival-guide/jwt
[localhost link]: http://localhost:3000/docs/5/fusion/operations/survival-guide/jwt
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/survival-guide/jwt
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/144
The Fusion API gateway requires incoming requests to be authenticated. The gateway supports a variety of authentication mechanisms, including Security Assertion Markup Language (SAML), OpenID Connect (OIDC), Kerberos, and Basic authentication. Once authenticated, the gateway issues a JWT and returns it in the `id` cookie.
Client applications get the best performance by using the `id` cookie (or JWT Authorization header) instead of using Basic authentication for every query request. Verifying a JWT is fast and safe to cache. Hashing a password is CPU intensive and slow. (Fusion uses [bcrypt](https://en.wikipedia.org/wiki/Bcrypt).)
All Fusion services require requests to include a JWT to identify the caller.
# Fusion 5 On-prem Kubernetes Deployment
Source: https://doc.lucidworks.com/docs/5/fusion/operations/survival-guide/k8s-on-prem
[localhost link]: http://localhost:3000/docs/5/fusion/operations/survival-guide/k8s-on-prem
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/survival-guide/k8s-on-prem
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/145
The [`setup_f5_k8s.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_k8s.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository provides deployment support for any Kubernetes platform, including on-premise, private cloud, public cloud, and hybrid platforms.
This script is used by the [`setup_f5_gke.sh`](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_gke.sh), [`setup_f5_eks.sh`](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_eks.sh), and [`setup_f5_aks.sh`](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_aks.sh) scripts, which provide additional platform-specific support for Google Kubernetes Engine (GKE), Amazon Elastic Kubernetes Service (EKS), and Azure Kubernetes Service (AKS).
See **Fusion 5 Upgrades** for more information about deployment instructions for:
This guide describes how to perform Fusion 5 upgrades.
Before upgrading, be aware of changes by checking for [Deprecations and Removals](/docs/5/fusion/deprecations-and-removals) between versions.
Lucidworks recommends upgrading to the next minor version only. For example, you should upgrade from Fusion 5.6.1 to Fusion 5.7.1 before upgrading to Fusion 5.8.0.
The [general upgrade process](#general-upgrade-process) is described in this article. However, the specific upgrade procedures may vary depending on your upgrade path. For the most accurate instructions, please refer to the upgrade article specific to your upgrade.
## General upgrade process
Fusion natively supports deployments on supported Kubernetes platforms, including AKS, EKS, and GKE.
Fusion includes an upgrade script for AKS, EKS, and GKE. This script is not generated for other Kubernetes deployments.
Upgrades differ from platform to platform. See below for more information about upgrading on your platform of choice.
Whenever you upgrade Fusion, you must also update your [remote connectors](/docs/fusion-connectors/developers/remote-v2-connectors), if you are running any.
You can download the latest files at [V2 Connectors Downloads](/docs/fusion-connectors/downloads/v2-connectors-downloads).
### Natively supported deployment upgrades
| Deployment type | Platform |
| - | - |
| **Azure Kubernetes Service (AKS)** | `aks` |
| **Amazon Elastic Kubernetes Service (EKS)** | `eks` |
| **Google Kubernetes Engine (GKE)** | `gke` |
Fusion includes upgrade scripts for natively supported deployment types. To upgrade:
1. Open the `___upgrade_fusion.sh` upgrade script file for editing.
2. Update the `CHART_VERSION` to your target Fusion version, and save your changes.
3. Run the `___upgrade_fusion.sh` script. The `` value is the same as your namespace, unless you overrode the default value using the `-r` option.
After running the upgrade, use `kubectl get pods` to see the changes applied to your cluster. It may take several minutes to perform the upgrade, as new Docker images are pulled from DockerHub. To see the versions of running pods, do:
```bash wrap theme={"dark"}
kubectl get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq
```
### Other Kubernetes deployment upgrades
To update an existing installation, do:
```bash wrap theme={"dark"}
RELEASE=f5
NAMESPACE=default
helm repo update
helm upgrade ${RELEASE} "lucidworks/fusion" --namespace "${NAMESPACE}" --values "${MY_VALUES}"
```
Except for ZooKeeper, all K8s deployments and statefulsets use a `RollingUpdate` update policy:
```yaml theme={"dark"}
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
```
ZooKeeper instances use `OnDelete` to avoid changing critical stateful pods in the Fusion deployment. To apply changes to Zookeeper after performing the upgrade (uncommon), you need to manually delete the pods. For example:
```bash theme={"dark"}
kubectl delete pod f5-zookeeper-0
```
Delete one pod at a time. Verify the new pod is healthy and serving traffic, before deleting the next healthy pod.
You can also set the `updateStrategy` under the `zookeeper` section in your `"${MY_VALUES}"` file:
```yaml theme={"dark"}
solr:
...
zookeeper:
updateStrategy:
type: "RollingUpdate"
```
#### Upgrades with Helm v3
One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. For example, Fusion 5 allows customers to upgrade from Fusion 5.1.0 to a later 5.x.y version on a live cluster with zero downtime or disruption of service.
When Kubernetes performs a rolling update to an individual microservice, there is a mix of old and new services in the cluster. Requests from other services route to both versions.
Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same minor release version (5.x). We also ensure that the stored configuration remains compatible in the same minor release version.
Lucidworks releases minor updates to individual services frequently. Pull in those upgrades using Helm at your discretion.
**How to upgrade Fusion**
1. Clone the [**fusion-cloud-native** repo](https://github.com/lucidworks/fusion-cloud-native), if you haven’t already.
2. Locate the `setup_f5_.sh` script that matches your Kubernetes platform.
3. Run the script with the `--upgrade` option.
To see what would be upgraded, pass the `--dry-run` option to the script.
The scripts in the **fusion-cloud-native** repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.
#### Helm upgrade script
Once you deploy a working cluster, use the upgrade script created by the `customize_fusion_values.sh` script. The upgrade script hard-codes the parameters and eases the need to remember which parameters to pass to the script. This is helpful when working with multiple K8s clusters. Make sure you check the script into version control alongside your custom values YAML files.
Whenever you change the custom values YAML files for your cluster, you need to run the upgrade script to apply the changes. The script calls `helm upgrade` with the correct parameters and `--values` options.
If you run `helm upgrade` without passing the custom values YAML files, the deployment will revert to using chart defaults, which you never want to do.The script assumes your `kubeconfig` is pointing to the correct cluster and you’re using Heml v3. If not, the upgrade fails. Select the correct `kubeconfig` before running the script.
* Natively supported deployments
* Other Kubernetes deployments
## Considerations for on-prem deployments
### Solr
Making changes to Solr’s memory settings after the initial installation, including changing the number of pods, is simple. Changing the size and type of disk, however, can be difficult. Before you begin, identify some basic sizing information for Solr:
* Memory settings
* Disk size
* Disk type (e.g. SSD)
### Docker images
Fusion’s public Docker images are hosted on DockerHub. If you do not allow your Kubernetes environment to pull images from DockerHub, you will need to copy the images to your internal registry. This can be a time-consuming process.
This teaches you how to use a private repository to manage your Docker images. When you install or upgrade Fusion, you can specify the private repository instead of the public repository. If you’re deploying on-premises, you’ll need to set up on-premises private Docker registries.
1. Add a `values.sh` file with the following. Some third-party dependencies might need service account level secrets if the chart doesn’t expose that field. You can use the Fusion install scripts to skip creating this file manually.
```yaml expandable theme={"dark"}
cat > values.yaml << EOF
---
admin-ui:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
api-gateway:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
argo:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
minio:
image:
repository: ${REPOSITORY}/minio
argo-common-workflows:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
auth-ui:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
classic-rest-service:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
classification:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
connector-plugin:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
connectors:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
connectors-backend:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
devops-ui:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
fusion-admin:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
fusion-data-augmentation:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
fusion-indexing:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
fusion-jupyter:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
fusion-log-forwarder:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
fusion-resources:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
fusion-superset:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
insights:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
job-launcher:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
job-rest-server:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
kafka:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
ml-model-service:
ambassador:
image:
repository: ${REPOSITORY}/datawire/ambassador
imagePullSecrets:
- name: ${REPOSITORY}
pro:
image:
repository: ${REPOSITORY}/datawire/ambassador_pro
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
milvus:
admin:
image:
repository: ${REPOSITORY}/milvus-em
image:
repository: ${REPOSITORY}/milvus
initContainerImage: ${REPOSITORY}/alpine:3.8
mishards:
image:
repository: ${REPOSITORY}/mishards
mysql:
busybox:
image: ${REPOSITORY}/busybox
image: ${REPOSITORY}/mysql
preinstall:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}/bitnami
pm-ui:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
pulsar:
dashboard:
image:
repository: ${REPOSITORY}/pulsar-dashboard
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
images:
imagePullSecrets:
- name: ${REPOSITORY}
initContainer:
image:
repository: ${REPOSITORY}
keytoolUtils:
image:
repository: ${REPOSITORY}
pulsar_metadata:
image:
repository: ${REPOSITORY}/pulsar-all
query-pipeline:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
question-answering:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
recommender:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
rules-ui:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
seldon-core-operator:
engine:
image:
repository: ${REPOSITORY}/engine
executor:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}/seldon-core-executor
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}/seldon-core-operator
solr:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}/solr
initContainer:
image:
repository: ${REPOSITORY}
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
zookeeper:
exporters:
jmx:
image:
repository: ${REPOSITORY}/jmx-prometheus-exporter
zookeeper:
image:
repository: ${REPOSITORY}/zookeeper-exporter
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}/zookeeper
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
solr-backup-runner:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
sql-service:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
templating:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
tikaserver:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}/apache
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
webapps:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
zookeeper:
exporters:
jmx:
image:
repository: ${REPOSITORY}/jmx-prometheus-exporter
zookeeper:
image:
repository: ${REPOSITORY}/zookeeper-exporter
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}/zookeeper
keytoolUtils:
image:
imagePullSecrets:
- name: ${REPOSITORY}
repository: ${REPOSITORY}
EOF
```
2. Run the script: `./values.sh`.\
If you receive an error stating `bash: ./values.sh: Permission denied`, make the script executable: `chmod +x values.sh` and run the script again: `./values.sh`.\
The script creates a new file, `___fusion_repository.yaml`. The contents of the file resembles the following:
```yaml theme={"dark"}
---
admin-ui:
initContainer:
image:
repository: altostrat.com/repo
image:
repository: altostrat.com/repo
api-gateway:
keytoolUtils:
image:
repository: "altostrat.com/repo"
initContainer:
image:
repository: altostrat.com/repo
image:
repository: altostrat.com/repo
argo:
images:
...
```
3. Open the `___upgrade_fusion.sh` script for editing.
4. Update the `MY_VALUES` values to refer to the `___fusion_repository.yaml` file.
```bash theme={"dark"}
MY_VALUES=""
MY_VALUES="$MY_VALUES --values ___fusion__repository.yaml"
MY_VALUES="$MY_VALUES --values ___fusion_values.yaml"
MY_VALUES="$MY_VALUES --values ___fusion_resources.yaml"
MY_VALUES="$MY_VALUES --values ___fusion_affinity.yaml"
MY_VALUES="$MY_VALUES --values ___fusion_replicas.yaml"
```
5. Run the `___upgrade_fusion.sh` script.
### Helm charts
Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster. We supply a Helm chart for the Fusion installation.
If you are using Helm V2, the `setup_f5_k8s.sh` script installs Helm’s Tiller component into your Kubernetes cluster with the cluster admin role. If you want to prevent the installation of Tiller, [upgrade to Helm V3](https://v3.helm.sh/docs/topics/v2_v3_migration).
### Nodes
We recommend installing Fusion on at least 3 nodes, each starting in its own namespace. Only 1 Fusion instance per namespace is supported.
Development and test environments only require 1 node with at least 12 GB of RAM.
It is possible to install other apps alongside Fusion in the same namespace, if desired.
### Pod resources
We recommend not placing resource limits on pods until you have run production-like workloads with your data and fully understand your resource requirements.
### Fusion gateway service
If you will be exposing the Fusion gateway service externally, you will need to consider your Kubernetes Ingress strategy:
* How will you route to internal services, such as the gateway?
* Will you do TLS termination at the Ingress?
### Deploying with a read-only root file system
Fusion 5.9.9 and later supports a read-only root file system to safeguard against unauthorized modifications. Most services operate efficiently in this mode, and for components that need write access, a separate writable mount is available using specific helm configurations.
See **Enable Read-Only Root File System** for examples of how to configure this feature.
This topic describes how to configure a read-only root file system for a Fusion deployment.
Read-only mode safeguards your file system against unauthorized changes, such as by malicious software or other attacks.
Enabling this feature entails configuring the `readOnlyRootFilesystem` attribute in the Fusion Helm chart.
This feature is available starting in Fusion 5.9.9 and in all subsequent Fusion 5.9 releases.
Beginning with Fusion 5.9.10, all of [Fusion’s services](/docs/5/fusion/reference/microservices) are designed to work with a read-only root file system, but some external services may require write access to the file system.
In that case, you can configure a read-only root file system and mount a separate writable file system for these services.
The examples below show how to configure a read-only root file system for various Fusion services.
## Solr
```yaml theme={"dark"}
fusion:
solr:
containerSecurityContext:
readOnlyRootFilesystem: true
```
**With optional services:**
```yaml theme={"dark"}
fusion:
solr:
exporter:
enabled: true
securityContext:
readOnlyRootFilesystem: true
containerSecurityContext:
readOnlyRootFilesystem: true
```
## Zookeeper
```yaml theme={"dark"}
fusion:
zookeeper:
containerSecurityContext:
readOnlyRootFilesystem: true
```
**With optional services:**
```yaml theme={"dark"}
fusion:
zookeeper:
containerSecurityContext:
readOnlyRootFilesystem: true
exporters:
jmx:
enabled: true
zookeeper:
enabled: true
jobs:
chroots:
enabled: true
config:
create:
- /root-read-only
```
## Kafka
```yaml theme={"dark"}
fusion:
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
```
**With optional services:**
```yaml expandable theme={"dark"}
fusion:
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
volumePermissions:
enabled: true
containerSecurityContext:
readOnlyRootFilesystem: true
metrics:
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
enabled: true
jmx:
enabled: true
containerSecurityContext:
enabled: true
readOnlyRootFilesystem: true
externalAccess:
enabled: true
autoDiscovery:
enabled: true
containerSecurityContext:
enabled: true
readOnlyRootFilesystem: true
provisioning:
enabled: true
topics:
- name: test
partitions: 1
replicationFactor: 1
## https://kafka.apache.org/documentation/#topicconfigs
config:
max.message.bytes: 64000
flush.messages: 1
containerSecurityContext:
readOnlyRootFilesystem: true
enabled: true
```
## ML Model Service
```yaml expandable theme={"dark"}
fusion:
ml-model-service:
ambassador:
containerSecurityContext:
readOnlyRootFilesystem: true
milvus:
mysql:
containerSecurityContext:
readOnlyRootFilesystem: true
securityContext:
readOnlyRootFilesystem: true
With optional services:
fusion:
ml-model-service:
ambassador:
containerSecurityContext:
readOnlyRootFilesystem: true
prometheusExporter:
enabled: true
securityContext:
readOnlyRootFilesystem: true
milvus:
mysql:
metrics:
enabled: true
securityContext:
readOnlyRootFilesystem: true
containerSecurityContext:
readOnlyRootFilesystem: true
cluster:
enabled: false
mishards:
securityContext:
readOnlyRootFilesystem: true
admin:
enabled: true
securityContext:
readOnlyRootFilesystem: true
securityContext:
readOnlyRootFilesystem: true
```
## Argo
```yaml theme={"dark"}
fusion:
argo:
minio:
containerSecurityContext:
readOnlyRootFilesystem: true
extraVolumes:
- name: config-dir
emptyDir: {}
- name: certs-dir
emptyDir: {}
extraVolumeMounts:
- name: config-dir
mountPath: /.minio
- name: certs-dir
mountPath: /etc/minio/certs
server:
securityContext:
readOnlyRootFilesystem: true
controller:
securityContext:
readOnlyRootFilesystem: true
mainContainer:
securityContext:
readOnlyRootFilesystem: true
executor:
securityContext:
readOnlyRootFilesystem: true
```
## Seldon Core Operator
```yaml theme={"dark"}
fusion:
seldon-core-operator:
containersSecurityContext:
readOnlyRootFilesystem: true
```
## Argo Common Workflows
```yaml theme={"dark"}
fusion:
argo-common-workflows:
containerSecurityContext:
readOnlyRootFilesystem: true
```
## Question Answering
```yaml theme={"dark"}
fusion:
question-answering:
containersSecurityContext:
readOnlyRootFilesystem: true
```
## Classification
```yaml theme={"dark"}
fusion:
classification:
containersSecurityContext:
readOnlyRootFilesystem: true
```
## All-in-one values.yaml example
```yaml expandable theme={"dark"}
fusion:
solr:
containerSecurityContext:
readOnlyRootFilesystem: true
solr-managed:
containerSecurityContext:
readOnlyRootFilesystem: true
zookeeper:
containerSecurityContext:
readOnlyRootFilesystem: true
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
ml-model-service:
ambassador:
containerSecurityContext:
readOnlyRootFilesystem: true
milvus:
mysql:
containerSecurityContext:
readOnlyRootFilesystem: true
securityContext:
readOnlyRootFilesystem: true
argo:
minio:
containerSecurityContext:
readOnlyRootFilesystem: true
extraVolumes:
- name: config-dir
emptyDir: {}
- name: certs-dir
emptyDir: {}
extraVolumeMounts:
- name: config-dir
mountPath: /.minio
- name: certs-dir
mountPath: /etc/minio/certs
server:
securityContext:
readOnlyRootFilesystem: true
controller:
securityContext:
readOnlyRootFilesystem: true
mainContainer:
securityContext:
readOnlyRootFilesystem: true
executor:
securityContext:
readOnlyRootFilesystem: true
seldon-core-operator:
containersSecurityContext:
readOnlyRootFilesystem: true
argo-common-workflows:
containerSecurityContext:
readOnlyRootFilesystem: true
question-answering:
containersSecurityContext:
readOnlyRootFilesystem: true
classification:
containersSecurityContext:
readOnlyRootFilesystem: true
```
## All-in-one values.yaml with optional services example
```yaml expandable theme={"dark"}
fusion:
argo:
minio:
persistence:
enabled: false
containerSecurityContext:
readOnlyRootFilesystem: true
extraVolumes:
- name: config-dir
emptyDir: {}
- name: certs-dir
emptyDir: {}
extraVolumeMounts:
- name: config-dir
mountPath: /.minio
- name: certs-dir
mountPath: /etc/minio/certs
server:
securityContext:
readOnlyRootFilesystem: true
controller:
securityContext:
readOnlyRootFilesystem: true
mainContainer:
securityContext:
readOnlyRootFilesystem: true
executor:
securityContext:
readOnlyRootFilesystem: true
solr:
exporter:
enabled: true
securityContext:
readOnlyRootFilesystem: true
containerSecurityContext:
readOnlyRootFilesystem: true
# tls:
# enabled: true
solr-managed:
exporter:
enabled: true
securityContext:
readOnlyRootFilesystem: true
enableExternalFiles: true
enabledStorage:
- gcs
processRaw:
image:
repository: fusion-dev-docker.ci-artifactory.lucidworks.com
cloudRoot: gs://lw-managed-fusion-data/tmp
gcs:
secret: gcs-key
secretFieldName: key.json
securityContext:
readOnlyRootFilesystem: true
containerSecurityContext:
readOnlyRootFilesystem: true
# tls:
# enabled: true
zookeeper:
containerSecurityContext:
readOnlyRootFilesystem: true
exporters:
jmx:
enabled: true
zookeeper:
enabled: true
jobs:
chroots:
# enabled: true
config:
create:
- /root-read-only
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
volumePermissions:
enabled: true
image:
repository: "fusion-dev-docker.ci-artifactory.lucidworks.com/os-shell"
containerSecurityContext:
readOnlyRootFilesystem: true
metrics:
kafka:
containerSecurityContext:
readOnlyRootFilesystem: true
enabled: true
jmx:
enabled: true
containerSecurityContext:
enabled: true
readOnlyRootFilesystem: true
externalAccess:
enabled: true
autoDiscovery:
enabled: true
containerSecurityContext:
enabled: true
readOnlyRootFilesystem: true
provisioning:
enabled: true
topics:
- name: test
partitions: 1
replicationFactor: 1
## https://kafka.apache.org/documentation/#topicconfigs
config:
max.message.bytes: 64000
flush.messages: 1
containerSecurityContext:
readOnlyRootFilesystem: true
enabled: true
ml-model-service:
enabled: true
ambassador:
containerSecurityContext:
readOnlyRootFilesystem: true
prometheusExporter:
enabled: true
securityContext:
readOnlyRootFilesystem: true
milvus:
mysql:
metrics:
enabled: true
securityContext:
readOnlyRootFilesystem: true
containerSecurityContext:
readOnlyRootFilesystem: true
cluster:
enabled: false
mishards:
securityContext:
readOnlyRootFilesystem: true
admin:
enabled: true
securityContext:
readOnlyRootFilesystem: true
securityContext:
readOnlyRootFilesystem: true
seldon-core-operator:
containersSecurityContext:
readOnlyRootFilesystem: true
argo-common-workflows:
containerSecurityContext:
readOnlyRootFilesystem: true
question-answering:
containersSecurityContext:
readOnlyRootFilesystem: true
classification:
containersSecurityContext:
readOnlyRootFilesystem: true
```
## Supported services
The table below lists the services that support a read-only root file system, the ones that have it enabled by default, and the Fusion release in which support was added:
| Chart Name | Pod Name | Container Name | Supported | Default enabled | Supported version |
| - | - | - | - | - | - |
| admin-ui | admin-ui | admin-ui | ✅ | ✅ | 5.9.9+ |
| api-gateway | api-gateway | init/api-gateway | ✅ | ✅ | 5.9.9+ |
| api-gateway | api-gateway | api-gateway | ✅ | ✅ | 5.9.9+ |
| api-gateway | api-gateway | generate-jks | ✅ | ✅ | 5.9.9+ |
| apps-manager | apps-manager | apps-manager | ✅ | ✅ | 5.9.9+ |
| argo | argo-server | argo-server | ✅ | ❌ | 5.9.10+ |
| argo | argo-executor | executor | ✅ | ❌ | 5.9.10+ |
| argo | argo-mainContainer | mainContainer | ✅ | ❌ | 5.9.10+ |
| argo | argo-controller | controller | ✅ | ✅ | 5.9.10+ |
| argo/minio | minio | minio | ✅ | ❌ | 5.9.10+ |
| argo/minio | minio | minio | ✅ | ❌ | 5.9.10+ |
| argo/minio | make-bucket-job | minio-mc | ✅ | ❌ | 5.9.10+ |
| argo-common-workflows | delete-model | init/main/wait | ✅ | ❌ | 5.9.10+ |
| argo-common-workflows | deploy-model | init/main/wait | ✅ | ❌ | 5.9.10+ |
| argo-common-workflows | milvus-maintenance | init/main/wait | ✅ | ❌ | 5.9.10+ |
| argo-common-workflows | upload-model-to-cloud | init/main/wait | ✅ | ❌ | 5.9.10+ |
| async-parsing | async-parsing | tika-server | ✅ | ✅ | 5.9.9+ |
| async-parsing | async-parsing | async-parsing | ✅ | ✅ | 5.9.9+ |
| auth-ui | auth-ui | auth-ui | ✅ | ✅ | 5.9.9+ |
| classic-rest-service | classic-rest-service | init/import-certs | ✅ | ✅ | 5.9.10+ |
| classic-rest-service | classic-rest-service | classic-rest-service | ✅ | ✅ | 5.9.9+ |
| classification | argo/classification | init/wait/main | ✅ | ❌ | 5.9.10+ |
| connector-plugin | connector-plugin | init/import-certs | ✅ | ❌ | 5.9.10+ |
| connector-plugin | connector-plugin | connector-plugin | ✅ | ✅ | 5.9.9+ |
| connectors | connectors | connectors | ✅ | ✅ | 5.9.9+ |
| connectors-backend | CRD | | ✅ | ✅ | 5.9.9+ |
| connectors-backend | connectors-backend | connectors-backend | ✅ | ✅ | 5.9.9+ |
| fusion-admin | fusion-admin | admin | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-admin | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-api-gateway | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-indexing | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-kafka | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-logstash | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-pulsar | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | setup-keystore-and-properties | ✅ | ✅ | 5.9.9+ |
| fusion-commons | | check-zk | ✅ | ✅ | 5.9.9+ |
| fusion-config-sync | fusion-config-sync | fusion-config-sync | ✅ | ✅ | 5.9.9+ |
| fusion-data-augmentation | argo/data-augmentation/volume-fix | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/init-workspace | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/write-job-configs | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/write-io-configs | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/add-zkhost | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/pull-data-training-and-metadata | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/pull-data-training-and-metadata-cloud | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/volume-fix2 | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/synonym-list | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/download-synonym-dictionary | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/keystroke-list | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/download-keystroke-blob | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/augment | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/push-augmented-data | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-data-augmentation | argo/data-augmentation/push-augmented-data-cloud | init/main/wait | ✅ | ❌ | 5.9.10+ |
| fusion-indexing | fusion-indexing | fusion-indexing | ✅ | ✅ | 5.9.9+ |
| fusion-resources | fusion-resources-secret-hook | setup-keystore-and-properties | ✅ | ✅ | 5.9.10+ |
| insights | insights | insights | ✅ | ✅ | 5.9.9+ |
| job-launcher | job-launcher | job-launcher | ✅ | ✅ | 5.9.9+ |
| job-launcher | job-launcher | kubectl-runner | ✅ | ✅ | 5.9.9+ |
| job-launcher | job-launcher-spark-cleanup | kubectl-runner | ✅ | ✅ | 5.9.9+ |
| job-launcher | spark-kubernetes-driver | spark-kubernetes-driver | ✅ | ❌ | 5.9.10+ |
| job-launcher | spark-kubernetes-executor | spark-kubernetes-executor | ✅ | ❌ | 5.9.10+ |
| job-rest-server | job-rest-server | job-rest-server | ✅ | ✅ | 5.9.9+ |
| kafka | kafka-metrics | kafka-exporter-archived | ✅ | ❌ | 5.9.10+ |
| kafka | kafka-provisioning | init/wait-for-available-kafka | ✅ | ❌ | 5.9.10+ |
| kafka | kafka-provisioning | kafka-provisioning | ✅ | ❌ | 5.9.10+ |
| kafka | kafka | kafka | ✅ | ❌ | 5.9.10+ |
| kafka | kafka | jmx-exporter | ✅ | ❌ | 5.9.10+ |
| kafka | kafka | init/check-zk | ✅ | ✅ | 5.9.9+ |
| kafka | kafka | init/auto-discovery | ✅ | ❌ | 5.9.10+ |
| kafka | kafka | init/volume-permissions | ✅ | ❌ | 5.9.10+ |
| lwai-gateway | lwai-gateway | lwai-gateway | ✅ | ✅ | 5.9.9+ |
| ml-model-service | ml-model-service | java-service | ✅ | ✅ | 5.9.9+ |
| ml-model-service | ml-model-service-namespace-hook | kubectl-runner | ✅ | ✅ | 5.9.9+ |
| ml-model-service/ambassador | ambassador | ambassador | ✅ | ❌ | 5.9.10+ |
| ml-model-service/ambassador | ambassador | prometheus-exporter | ✅ | ❌ | 5.9.10+ |
| ml-model-service/milvus | milvus-writable | milvus | ✅ | ❌ | 5.9.10+ |
| ml-model-service/milvus | milvus-writable | init/wait-for-mysql | ✅ | ✅ | 5.9.10+ |
| ml-model-service/milvus | milvus-writable | init/create-for-share-storage | ✅ | ❌ | 5.9.10+ |
| ml-model-service/milvus | milvus-admin | admin | ✅ | ❌ | 5.9.10+ |
| ml-model-service/milvus | milvus-admin | init/wait-for-milvus | ✅ | ✅ | 5.9.10+ |
| ml-model-service/milvus | milvus-mishards | init/wait-for-mysql | ✅ | ✅ | 5.9.10+ |
| ml-model-service/milvus | milvus-mishards | init/wait-for-mysql | ✅ | ✅ | 5.9.10+ |
| ml-model-service/milvus | milvus-mishards | mishards | ✅ | ❌ | 5.9.10+ |
| ml-model-service/milvus/mysql | mysql | init/remove-lost-found | ✅ | ✅ | 5.9.10+ |
| ml-model-service/milvus/mysql | mysql | mysql | ✅ | ❌ | 5.9.10+ |
| pm-ui | pm-ui | pm-ui | ✅ | ✅ | 5.9.9+ |
| query-pipeline | query-pipeline | query-pipeline | ✅ | ✅ | 5.9.9+ |
| question-answering | argo/qna-coldstart/init-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/write-job-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/add-zkHost | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/pull-data | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/pull-data-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/train | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/list-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/post-model | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-coldstart/apply-seldon-deployment | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/init-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/write-job-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/add-zkhost | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/pull-qa-data-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/pull-qa-data | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/train | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/train-with-texts | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/list-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/post-model | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-supervised/apply-seldon-deployment | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/init-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/write-job-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/add-zkhost | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/list-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/pull-eval-data | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/pull-eval-data-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/evaluate | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/push-eval-results-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| question-answering | argo/qna-evaluation/push-eval-results | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/init-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/write-job-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/add-zkhost | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/pull-data-training-and-metadata | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/pull-data-training-and-metadata-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/train-with-metadata | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/train-without-metadata | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/push-recommendations | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/push-recommendations-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/add-default-exclude-query | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-user/init-workspace | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/copy-model | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/write-job-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/write-io-configs | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/add-zkhost | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/pull-data | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/pull-data-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/train | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/push-content | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/push-content-cloud | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/add-default-exclude-query | init/wait/main | ✅ | ❌ | 5.9.10+ |
| recommender | argo/item-recommender-content/delete-old-content-recommendations | init/wait/main | ✅ | ❌ | 5.9.10+ |
| reverse-search | reverse-search | init/set-reverse-search-zone | ✅ | ✅ | 5.9.10+ |
| reverse-search | reverse-search | init/check-zk | ✅ | ✅ | 5.9.9+ |
| reverse-search | reverse-search | init/enable-tls-in-reverse-search | ✅ | ✅ | 5.9.10+ |
| reverse-search | reverse-search | reverse-search | ✅ | ❌ | 5.9.10+ |
| rules-ui | rules-ui | rules-ui | ✅ | ✅ | 5.9.9+ |
| seldon-core-operator | seldon-controller-manager | manager | ✅ | ❌ | 5.9.10+ |
| seldon-core-operator | seldon-spartakus-volunteer | seldon-spartakus-volunteer | ✅ | ❌ | 5.9.9+ |
| seldon-core-operator | crd/SeldonDeployment | | ✅/❌ | ❌ | 5.9.9+ |
| solr | solr | init/set-solr-zone | ✅ | ✅ | 5.9.10+ |
| solr | solr | init/enable-tls-in-solr | ✅ | ✅ | 5.9.10+ |
| solr | configset-bootstrap | configset-bootstrap | ✅ | ✅ | 5.9.10+ |
| solr | solr | solr | ✅ | ❌ | 5.9.10+ |
| solr | solr-exporter | exporter | ✅ | ❌ | 5.9.10+ |
| solr | solr-exporter | init/solr-init | ✅ | ✅ | 5.9.10+ |
| solr-managed | configset-bootstrap | configset-bootstrap | ✅ | ✅ | 5.9.10+ |
| solr-managed | exporter | exporter | ✅ | ❌ | 5.9.10+ |
| solr-managed | exporter | init/solr-init | ✅ | ✅ | 5.9.10+ |
| solr-managed | solr | init/set-solr-zone | ✅ | ✅ | 5.9.10+ |
| solr-managed | solr | init/enable-tls-in-solr | ✅ | ✅ | 5.9.10+ |
| solr-managed | solr | solr | ✅ | ❌ | 5.9.10+ |
| solr-managed | solr | process-raw | ✅ | ❌ | 5.9.10+ |
| solr-backup-runner | solr-backup-runner-backup | solr-backups | ✅ | ❌ | 5.9.10+ |
| solr-backup-runner | solr-backup-runner-prune | solr-prune | ✅ | ❌ | 5.9.10+ |
| templating | templating | templating | ✅ | ✅ | 5.9.9+ |
| webapps | webapps | webapps | ✅ | ✅ | 5.9.9+ |
| zookeeper | zookeeper | zookeeper | ✅ | ❌ | 5.9.10+ |
| zookeeper | zookeeper | jmx-exporter | ✅ | ❌ | 5.9.10+ |
| zookeeper | zookeeper | zookeeper-exporter | ✅ | ❌ | 5.9.10+ |
| zookeeper | zookeeper-chroots | main | ✅ | ❌ | 5.9.10+ |
## Learn more
This article teaches you how to use a private repository to manage your Docker images. When you install or upgrade Fusion, you can specify the private repository instead of the public repository. If you’re deploying on-premises, you’ll need to \[set up on-premises private Docker registries.
1. [Download](/assets/attachments/values.sh) the `values.sh` script.
2. Run the script: `./values.sh`.\
If you receive an error stating `bash: ./values.sh: Permission denied`, make the script executable: `chmod +x values.sh` and run the script again: `./values.sh`.\
The script creates a new file, `___fusion_repository.yaml`. The contents of the file resembles the following:
```yaml theme={"dark"}
---
admin-ui:
initContainer:
image:
repository: altostrat.com/repo
image:
repository: altostrat.com/repo
api-gateway:
keytoolUtils:
image:
repository: "altostrat.com/repo"
initContainer:
image:
repository: altostrat.com/repo
image:
repository: altostrat.com/repo
argo:
images:
...
```
3. Open the `___upgrade_fusion.sh` script for editing.
4. Update the `MY_VALUES` values to refer to the `___fusion_repository.yaml` file:
```bash theme={"dark"}
MY_VALUES=""
MY_VALUES="$MY_VALUES --values ___fusion__repository.yaml"
MY_VALUES="$MY_VALUES --values ___fusion_values.yaml"
MY_VALUES="$MY_VALUES --values ___fusion_resources.yaml"
MY_VALUES="$MY_VALUES --values ___fusion_affinity.yaml"
MY_VALUES="$MY_VALUES --values ___fusion_replicas.yaml"
```
5. Run the `___upgrade_fusion.sh` script.
# Fusion 5 Survival Guide
Source: https://doc.lucidworks.com/docs/5/fusion/operations/survival-guide/overview
[localhost link]: http://localhost:3000/docs/5/fusion/operations/survival-guide/overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/survival-guide/overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/119
The purpose of this guide is to help you install, configure, and run Fusion 5 in production with high availability on Kubernetes. You can also find this guide in the [fusion-cloud-native repo](https://github.com/lucidworks/fusion-cloud-native/).
The Fusion 5 Survival Guide prefers Google Kubernetes Engine (GKE) terminology. Although your Kubernetes provider’s terminology may differ, the general concepts are consistent between providers.
## Foundational concepts
* [Which Kubernetes?](/docs/5/fusion/operations/survival-guide/platforms)
* [Multiple Zones for High Availability](/docs/5/fusion/operations/survival-guide/high-availability)
* [Fusion Microservices](/docs/5/fusion/reference/microservices)
* [Ingress and Security](/docs/5/fusion/operations/survival-guide/ingress)
* [Stateless Sessions with JWT](/docs/5/fusion/operations/survival-guide/jwt)
* [Workload Isolation with Multiple Node Pools](/docs/5/fusion/operations/survival-guide/workload-isolation)
* [High-Performance Query Processing with Auto-Scaling](/docs/5/fusion/operations/survival-guide/auto-scaling)
* [Fusion 5 Frequently Asked Questions](/docs/5/fusion/operations/faq)
## Planning your deployment
* **Deploy Fusion at Scale**
Before you begin, see [Fusion Server Deployment](/docs/5/fusion/operations/deployment) to understand the architecture and requirements.
This article explains how to plan and execute a Fusion deployment at the scale required for staging or production.
While the `setup_f5_*.sh` scripts are handy for getting started and proof-of-concept purposes, this article covers the planning process for building a production-ready environment.
The course for **Preparing for Fusion Implementation** focuses on the key elements for a successful implementation, defining your business requirements, preparing clean data, and involving the right personnel.
## Prerequisites
You must meet the following prerequisites before you can customize your Fusion cluster:
* A local copy of the [fusion-cloud-native repository](https://github.com/lucidworks/fusion-cloud-native). This must be up-to-date with the latest master branch.
* Any cloud provider-specific **command line tools**, such as `gcloud` or `aws`, and `kubectl`.\
See the platform-specific instructions linked above, or check with your cloud provider.
* Helm v3
* To install on a Mac:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
* For other operating systems, download from [Helm Releases](https://github.com/helm/helm/releases).
* Verify your installation:
```bash theme={"dark"}
helm version --short
v3.0.0+ge29ce2a
```
* Kubernetes namespace
* Collect the following information about your Kubernetes environment:
* **CLUSTER**: Cluster name (passed to our setup scripts using the `-c` arg)
* **NAMESPACE**: Kubernetes namespace where to install Fusion; a namespace should only contain lowercase letters (a-z), digits (0-9), or dash. No periods or underscores allowed.
* *(optional)* Clarify your organization’s DockerHub policy. The Fusion Helm chart points to public Docker images on DockerHub. Your organization may not allow Kubernetes to pull images directly from DockerHub or may require extra security scanning before loading images into production clusters.\
Consult your Kubernetes and Docker admin team to find how to get the Fusion images loaded into a registry that’s accessible to your cluster. You can update the image for each service using the [custom values YAML file](#custom-values-yaml-file).
**Kubernetes namespace tips**
* Fusion 5 service discovery requires all services for the same release be deployed in the same namespace. Moreover, you should only run one instance of Fusion in a namespace. If you need multiple instances of Fusion running in the same Kubernetes cluster, then you need to deploy them in separate namespaces.
* If your organization requires CPU / Memory quotas for namespaces, you can start with a minimum of 12 CPU and 45GB of RAM (such as 3 x n1-standard-4 on GKE), but you will need to increase the quotas once you start load testing Fusion with production workloads and real datasets.
* Fusion requires at least 3 ZooKeeper nodes and 2 Solr nodes to achieve high availability.
## Custom values YAML file
1. Clone the `fusion-cloud-native` repository: `git clone https://github.com/lucidworks/fusion-cloud-native`
2. Run the `customize_fusion_values.sh` script.
```bash theme={"dark"}
./customize_fusion_values.sh --provider -c -n \
--num-solr 3 \
--solr-disk-gb 100 \
--node-pool \
--prometheus true \
--with-resource-limits \
--with-affinity-rules
```
Pass the `--help` parameter to see script usage details.
The script creates the following files:
| File | Description |
| - | - |
| `___fusion_values.yaml` | Main custom values YAML used to override Helm chart defaults for Fusion microservices. |
| `___monitoring_values.yaml` | Custom values yaml used to configure Prometheus and Grafana. |
| `___fusion_resources.yaml` | Resource requests and limits for all Microservices. |
| `___fusion_affinity.yaml` | Pod affinity rules to ensure multiple replicas for a single service are evenly distributed across zones and nodes. |
| `___upgrade_fusion.sh` | Script used to install and/or upgrade Fusion using the aforementioned custom values YAML files. |
For an explanation of these placeholder values, see [Configuration Values](#custom-values-yaml-file) below.
3. Add the new files to version control. You will make changes to it over time as you fine-tune your Fusion installation. You will also need it to perform upgrades. If you try to upgrade your Fusion installation and don’t provide the custom values YAML, your deployment will revert to chart defaults.\
Review the `___fusion_values.yaml` file to familiarize yourself with its structure and contents. Notice it contains a separate section for each of the Fusion microservices. The example configuration of the `query-pipeline` service below illustrates some important concepts about the custom values YAML file.
```yaml highlight={1,2,3,5,6} theme={"dark"}
query-pipeline:
enabled: true
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
javaToolOptions: "..."
pod:
annotations:
prometheus.io/port: "8787"
prometheus.io/scrape: "true"
prometheus.io/path: "/actuator/prometheus"
```
4. Service-specific setting overrides under the top-level heading
5. Every Fusion service has an implicit enabled flag that defaults to true, set to false to remove this service from your cluster
6. Node selector identifies the label find nodes to schedule pods on
7. Used to pass JVM options to the service
8. Pod annotations to allow Prometheus to scrape metrics from the service
Once we go through all of the configuration topics in this topic, you'll have a well-configured custom values YAML file for your Fusion 5 installation. You'll then use this file during the Helm v3 installation at the end of this topic.
#### Custom labels and annotations
Starting with Fusion 5.9.16 and later, you can configure custom labels and annotations for all Kubernetes resources through values files without modifying chart templates. This eliminates upgrade complexity and maintenance overhead.
Configure global values that apply to all Fusion services:
```yaml theme={"dark"}
global:
labels:
owner: search-platform
annotations:
cost-center: fin-1234
```
Configure service-specific values that apply only to individual charts:
```yaml theme={"dark"}
fusion-indexing:
labels:
team: platform-search
annotations:
monitoring: enabled
```
The chart automatically merges and renders these values into Kubernetes resource metadata. Common use cases include cost allocation labels, ownership labels, monitoring annotations, and service mesh annotations.
### Deployment-specific values
The script creates a custom values YAML file using the naming convention: `___fusion_values.yaml`. For example, `gke_search_f5_fusion_values.yaml`.
| Parameter | Description |
| - | - |
| `` | The K8s platform you’re running on, such as `gke`. |
| `` | The name of your cluster. |
| `` | The K8s namespace where you want to install Fusion. |
| `` | Specifies a `nodeSelector` label to find nodes to schedule Fusion pods on. |
Providing the correct `--node-pool ` label is very important. Using the wrong value will cause your pods to be stuck in the `pending` state. If you’re not sure about the correct value for your cluster, pass ’'\` to let Kubernetes decide which nodes to schedule Fusion pods on.
Default `nodeSelector` labels are provider-specific. The `fusion-cloud-native` scripts use the following defaults for GKE and EKS:
| Provider | Default node selector |
| - | - |
| GKE | cloud.google.com/gke-nodepool: default-pool |
| EKS | alpha.eksctl.io/nodegroup-name: standard-workers |
If you are deploying Fusion 5.9.12, add the following to your `values.yaml` file to avoid a known issue that prevents the `kuberay-operator` pod from launching successfully: `yaml kuberay-operator: crd: create: true `
### Flags
The script provides flags for additional configuration:
| Flag | Description |
| - | - |
| `--node-pool` | Add a Fusion specific label to your nodes. |
| `--with-resource-limits` | Configure resource requests/limits. |
| `--with-replicas` | Configure replica counts. |
| `--with-affinity-rules` | Configure pod affinity rules for Fusion services. |
Use `--node-pool` to add a Fusion specific label to your nodes by doing:
```bash theme={"dark"}
kubectl label fusion_node_type=
```
Then, pass `--node-pool 'fusion_node_type: '`.
## Configure Solr sizing
When you’re ready to build a production-ready setup for Fusion 5, you need to customize the Fusion Helm chart to ensure Fusion is well-configured for production workloads.
You’ll be able to scale the number of nodes for Solr up and down after building the cluster, but you need to establish the initial size of the nodes (memory and CPU) and the size and type of disks you need.
See the example config below to learn which parameters to change in the custom values YAML file.
```yaml expandable theme={"dark"}
solr:
resources: # Set resource limits for Solr to help K8s pod scheduling;
limits: # these limits are not just for the Solr process in the pod,
cpu: "7700m" # so allow ample memory for loading index files into the OS cache (mmap)
memory: "26Gi"
requests:
cpu: "7000m"
memory: "25Gi"
logLevel: WARN
nodeSelector:
fusion_node_type: search # Run this Solr StatefulSet in the "search" node pool
exporter:
enabled: true # Enable the Solr metrics exporter (for Prometheus) and
# schedule on the default node pool (system partition)
podAnnotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9983"
prometheus.io/path: "/metrics"
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
image:
tag: 8.4.1
updateStrategy:
type: "RollingUpdate"
javaMem: "-Xmx3g -Dfusion_node_type=system" # Configure memory settings for Solr
solrGcTune: "-XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+UseStringDeduplication -XX:+PerfDisableSharedMem -XX:+ParallelRefProcEnabled -XX:MaxGCPauseMillis=150 -XX:+UseLargePages -XX:+AlwaysPreTouch"
volumeClaimTemplates:
storageSize: "100Gi" # Size of the Solr disk
replicaCount: 6 # Number of Solr pods to run in this StatefulSet
zookeeper:
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
replicaCount: 3 # Number of Zookeepers
persistence:
size: 20Gi
resources: {}
env:
ZK_HEAP_SIZE: 1G
ZOO_AUTOPURGE_PURGEINTERVAL: 1
```
To be clear, you can tune GC settings and number of replicas after the cluster is built. But changing the size of the persistent volumes is more complicated so you should try to pick a good size initially.
### Configure storage class for Solr pods (optional)
If you wish to run with a storage class other than the default you can create a storage class for your Solr pods before you install. For example, to create regional disks in GCP you can create a file called `storageClass.yaml` with the following contents:
```yaml theme={"dark"}
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: solr-gke-storage-regional
provisioner: kubernetes.io/gce-pd
parameters:
type: pd-standard
replication-type: regional-pd
zones: us-west1-b, us-west1-c
```
and then provision into your cluster by calling:
```bash theme={"dark"}
kubectl apply -f storageClass.yaml
```
to then have Solr use the storage class by adding the following to the custom values YAML:
```yaml theme={"dark"}
solr:
volumeClaimTemplates:
storageClassName: solr-gke-storage-regional
storageSize: 250Gi
```
We’re not advocating that you must use regional disks for Solr storage, as that would be redundant with Solr replication. We’re just using this as an example of how to configure a custom storage class for Solr disks if you see the need. For instance, you could use regional disks without Solr replication for write-heavy type collections.
## Configure multiple node pools
Lucidworks recommends isolating search workloads from analytics workloads using multiple node pools. The included scripts do not do this for you; this is a manual process.
See the example script for GKE, see [create\_gke\_cluster\_node\_pools.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/additional_environments/create_gke_cluster_node_pools.sh).
In the custom values YAML file, you can add additional Solr StatefulSets by adding their names to the list under the `nodePools` property. If any property for that statefulset needs to be changed from the default set of values, then it can be set directly on the object representing the node pool, any properties that are omitted are defaulted to the base value. See the following example (additional whitespace added for display purposes only):
```yaml expandable highlight={3,4,9,17,22,31} theme={"dark"}
solr:
nodePools:
- name: ""
- name: "analytics"
javaMem: "-Xmx6g"
replicaCount: 6
storageSize: "100Gi"
nodeSelector:
fusion_node_type: analytics
resources:
requests:
cpu: 2
memory: 12Gi
limits:
cpu: 3
memory: 12Gi
- name: "search"
javaMem: "-Xms11g -Xmx11g"
replicaCount: 12
storageSize: "50Gi"
nodeSelector:
fusion_node_type: search
resources:
limits:
cpu: "7700m"
memory: "26Gi"
requests:
cpu: "7000m"
memory: "25Gi"
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
...
```
1. The empty string `""` is the suffix for the default partition.
2. Overrides the settings for the **analytics** Solr pods.
3. Assigns the **analytics** Solr pods to the node pool and attaches the label `fusion_node_type=analytics`. You can use the `fusion_node_type` property in Solr auto-scaling policies to govern replica placement during collection creation.
4. Overrides the settings for the **search** Solr pods.
5. Assigns the **search** Solr pods to the node pool and attaches the label `fusion_node_type=search`.
6. Sets the default settings for all Solr pods, if not specifically overridden in the `nodePools` section above.
Do not edit the `nodePools` value `""`.
In the example above, the **analytics** partition `replicaCount`, or number of Solr pods, is six. The **search** partition `replicaCount` is twelve.
Each nodePool is automatically be assigned the -Dfusion\_node\_type property of ``, ``, or ``. This value matches the name of the nodePool. For example, `-Dfusion_node_type=`.
The Solr pods have a `fusion_node_type` system property, as shown below:
## Solr auto-scaling policy
Use [replica placement plugins](https://solr.apache.org/guide/solr/latest/configuration-guide/replica-placement-plugins.html) to control how replicas are placed in Solr.
## Pod network policy
A Kubernetes network policy governs how groups of pods communicate with each other and other network endpoints. With Fusion, all incoming traffic flows through the API Gateway service. All Fusion services in the same namespace expect an internal JWT, which is supplied by the Gateway, as part of the request. As a result, Fusion services enforce a basic level of API security and don’t need an additional network policy to protect them from other pods in the cluster.
To install the network policy for Fusion services, pass `--set global.networkPolicyEnabled=true` when installing the Fusion Helm chart.
## On-premises private Docker registries
For on-premises Kubernetes deployments, your organization may not allow Kubernetes to pull [Fusion’s Docker images from DockerHub](https://hub.docker.com/u/lucidworks/). See the instructions below for details on using a private Docker registry with Fusion. These are general instructions that may need to be adapted to work within your organization’s security policies:
1. Transfer the public images from DockerHub to your private Docker registry.
2. Establish a workstation that has access to [DockerHub](https://hub.docker.com). This workstation must connect to your internal Docker registry, most likely via VPN connection. In this example, the workstation is referred to as `envoy`.
3. Install Docker on `envoy`. You need at least 100GB of free disk for Docker.
4. Pull all of the images from DockerHub to `envoy`’s local registry. For example, to pull the query pipeline image, run `docker pull lucidworks/query-pipeline:5.9.0`. See `docker pull --help` for more information about pulling Docker images.
5. Establish a connection from `envoy` to the private Docker registry, most likely via a VPN connection. In this example, the private Docker registry is referred to as ``.
6. Push the images from `envoy`’s Docker registry to the private registry. This will take a long time.
1. You’ll need to re-tag all images for the internal registry. For example, to tag the query-pipeline image, run:
```bash theme={"dark"}
docker tag lucidworks/query-pipeline:5.9.0 /query-pipeline:5.9.0
```
2. Push each image to the internal repo:
```bash theme={"dark"}
docker push /query-pipeline:5.9.0
```
7. Install the Docker registry secret in Kubernetes. Create the Docker registry secret in the Kubernetes namespace where you want to install Fusion:
```bash theme={"dark"}
SECRET_NAME=
REPO=
kubectl create secret docker-registry "${SECRET_NAME}" \
--namespace "${NAMESPACE}" \
--docker-server="${REPO}" \
--docker-username=${REPO_USER} \
--docker-password=${REPO_PASS} \
--docker-email=${REPO_USER}
```
For details, see the Kubernetes article [Pull an Image from a Private Registry](https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/).
8. Update the custom values YAML for your cluster to point to your private registry and secret to allow Kubernetes to pull images. For example:
```yaml theme={"dark"}
query-pipeline:
image:
imagePullSecrets:
- name:
repository:
```
Repeat the process for all Fusion services.
### Customize Helm Chart
Every Fusion service allows you to override the `imagePullSecrets` setting using custom values YAML. However, other 3rd party services--including Zookeeper, Pulsar, Prometheus, and Grafana--don’t allow you to supply the pull secret using the custom values YAML.
To patch the default service account for your namespace and add the pull secret, run the following:
```bash theme={"dark"}
kubectl patch sa default -n $NAMESPACE \
-p '"imagePullSecrets": [{"name": "" }]'
```
In Windows using PowerShell or another CLI, you might have to escape the double quotes with a backslash (`\`) or reverse the order of single and double quotes:
```bash theme={"dark"}
kubectl patch sa default -n $NAMESPACE \
-p "'imagePullSecrets': [{'name': ''}]"
```
Replace `` with the name of the secret you created in the steps above.
This allows the default service account to pull images from the private registry without specifying the pull secret on the resources directly.
## Add additional trusted certificate(s) to Fusion’s indexing and querying services *(optional)*
You can add custom trusted certificates to support Fusion’s indexing and querying services. You may want to use custom trusted certificates if, for example, you have specific security requirements for data handling or need to support an existing infrastructure and its security needs. This method involves updating your Helm chart.
If you want to add custom trusted certificates for both the indexing and querying services, follow these instructions twice: once for the indexing service, and once for the querying service. To add different certificates to the indexing and querying services, create one YAML file with the indexing service certificates and one YAML file for the querying service certificates before following these instructions.
You may use the same YAML file if you want to use the same certificates for both services.
To add custom trusted certificates:
1. Create a new YAML file for your custom trusted certificates. The `customcerts.yaml` file is the example file in these instructions.
2. Add the custom certificate(s) in the YAML file created in the previous step. For example:
```yaml theme={"dark"}
trustedCertificates:
enabled: true
files:
some.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE-----
other.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE---------
```
3. Update the indexing or querying service by running the following Helm command. Replace `EXAMPLE-VALUES-FILE.yaml` with your previous values file.
```bash theme={"dark"}
helm upgrade --install --namespace ${EXAMPLE-NAMESPACE} ${HELM-RELEASE} ${HELM-CHART-PATH} --values EXAMPLE-VALUES-FILE.yaml --values customcerts.yaml
```
4. Verify the indexing or querying pod has a new `init-container` with the name `import-certs`.
## Add additional trusted certificate(s) for connectors to allow crawling of web resources with SSL/TLS enabled *(optional)*
To crawl a datasource which for some reason is using a self-signed certificate, add arbitrary certificates to connectors. For example:
```yaml wrap expandable theme={"dark"}
classic-rest-service:
trustedCertificates:
enabled: true
files:
some.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE-----
other.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE---------
connector-plugin:
trustedCertificates:
enabled: true
files:
some.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE-----
other.cert: |-
-----BEGIN CERTIFICATE-----
MIIDeTCCAmGgAwIBAgIJAPziuikCTox4MA0GCSqGSIb3DQEBCwUAMGIxCzAJBgNV
(...)
EVA0pmzIzgBg+JIe3PdRy27T0asgQW/F4TY61Yk=
-----END CERTIFICATE---------
```
### Generating the certificate on linux command line
Use the following command to generate a `.crt` file in `$fusion_home/apps/jetty/connectors/etc/yourcertname.crt`:
```bash theme={"dark"}
openssl s_client -servername remote.server.net -connect remote.server.net:443 $fusion_home/apps/jetty/connectors/etc/yourcertname.crt
```
### Generating the certificate using Firefox web browser
1. Navigate to the SharePoint host.
2. Click the in the address bar, then click the icon.
3. Next, navigate to **More Information** > **View Certificate** > **Export**.\
Save the file to the following folder:
`$fusion_home/apps/jetty/connectors/etc/yourcertname.crt`
### Generating the certificate using Chrome web browser
1. Navigate to **Chrome menu** > **More Tools** > **Developer Tools** > **Security Tab**.
This will display the **Security overview**.
2. Click the **View certificate** button.
3. Save the file to the following folder:
`$fusion_home/apps/jetty/connectors/etc/yourcertname.crt`
### Generating the certificate using powershell
Use the following script to generate a `.crt`` file in `\$fusion\_home\apps\jetty\connectors\etc\yourcertname.crt\`\`:
```bash theme={"dark"}
$fusion_home = c:\your\fusion\install\directory
$webRequest = [Net.WebRequest]::Create("https://your-hostname")
try { $webRequest.GetResponse() } catch {}
$cert = $webRequest.ServicePoint.Certificate
$bytes = $cert.Export([Security.Cryptography.X509Certificates.X509ContentType]::Cert)
set-content -value $bytes -encoding byte -path "$fusion_home\apps\jetty\connectors\etc\yourcertname.binary.crt"
certutil -encode "$fusion_home\apps\jetty\connectors\etc\yourcertname.binary.crt" "$fusion_home\apps\jetty\connectors\etc\yourcertname.crt"
rm "$fusion_home\apps\jetty\connectors\etc\yourcertname.binary.crt" -f
```
## Install Fusion 5 on Kubernetes
At this point, you’re ready to install Fusion 5 using the custom values YAML files and upgrade script. If you used the `customize_fusion_values.sh` script, run it using BASH:
```bash theme={"dark"}
./gke_search_f5_upgrade_fusion.sh
```
Once the installation is complete, verify your Fusion installation is running correctly.
## Monitoring Fusion with Prometheus and Grafana
Lucidworks recommends using Prometheus and Grafana for monitoring the performance and health of your Fusion cluster. Your operations team may already have these services installed. If not, install them into the Fusion namespace.
The [Custom values YAML file shown above](#custom-values-yaml-file) activates the Solr metrics exporter service and adds pod annotations so Prometheus can scrape metrics from Fusion services.
1. Run the `customize_fusion_values.sh` script with the `--prometheus true` option. This creates an extra custom values YAML file for installing Prometheus and Grafana, `___monitoring_values.yaml`. For example: `gke_search_f5_monitoring_values.yaml`.
2. Commit the YAML file to version control.
3. Review its contents to ensure that the settings suit your needs. For example, decide how long you want to keep metrics. The default is 36 hours.\
See the [Prometheus documentation](https://github.com/helm/charts/tree/master/stable/prometheus) and [Grafana documentation](https://github.com/helm/charts/tree/master/stable/grafana) for details.
4. Run the `install_prom.sh` script to install Prometheus & Grafana in your namespace. Include the provider, cluster name, namespace, and helm release as in the example below:
```bash theme={"dark"}
./install_prom.sh --provider gke -c search -n f5 -r 5-5-1
```
Pass the `--help` parameter to see script usage details.
The Grafana dashboards from [monitoring/grafana](https://github.com/lucidworks/fusion-cloud-native/tree/master/monitoring/grafana) are installed automatically by the `install_prom.sh` script.
## "Day two" operations
* **Configure Grafana, Prometheus, Promtail, and Loki in Fusion**
* **Configure Pod Affinity**
* **Configure Resource Limits**
* **Spark Operations**
* **Fusion 5 Upgrades**
Before you perform these installation instructions, you must delete any existing persistent volume claims (PVCs) related to Prometheus, Grafana, Promtail, and Loki on your namespace.
## Clone the `fusion-cloud-native` repository
Open a terminal window and run following command:
```
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
## Install Grafana
1. In your local `fusion-cloud-native` repository, run the following command for your `` and ``:
```
./install-prom.sh -c -n
```
The following is a sample output. The errors are related to resource limits on the sample cluster, and can be ignored. Similar errors may display for your cluster, and do not impact Grafana logging.
```
Adding the stable chart repo to helm repo list
"prometheus-community" already exists with the same configuration, skipping
"grafana" already exists with the same configuration, skipping
Installing Prometheus and Grafana for monitoring Fusion metrics ... this can take a few minutes.
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "ckotzbauer" chart repository
...Successfully got an update from the "lucidworks" chart repository
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "prometheus-community" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 2 charts
Downloading prometheus from repo https://prometheus-community.github.io/helm-charts
Downloading grafana from repo https://grafana.github.io/helm-charts
Deleting outdated charts
Release "fe-foundry-monitoring" does not exist. Installing it now.
Error: context deadline exceeded
Successfully installed Prometheus and Grafana into the fe-foundry namespace.
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
fe-foundry fe-foundry 11 2023-08-07 15:10:55.373825 -0700 PDT deployed fusion-5.8.0 5.8.0
fe-foundry-jupyter fe-foundry 2 2023-07-20 11:29:38.481329 -0700 PDT deployed fusion-jupyter-0.2.5 1.0
fe-foundry-monitoring fe-foundry 1 2023-08-10 11:41:06.113257 -0700 PDT failed fusion-monitoring-1.0.1 1.0.1
```
2. Using the Grafana service endpoint in the newly-installed Grafana helm release, run the following command:
```
kubectl get services
```
The following is a sample output.
```
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
grafana LoadBalancer 3000:32589/TCP 87m
```
If the output does not display, run the following command to expose Grafana, including an `EXTERNAL_IP` for your Grafana LoadBalancer service:
```
kubectl expose deployment --type=LoadBalancer --name=grafana --port=3000 --target-port=3000
```
## Install Loki
To obtain Loki from the helm chart repository, run the following command for the unique `` for your cluster:
```
helm upgrade --install --namespace= grafana/loki-stack
```
If you do no enter the `` correctly, an error similar to the following displays:
```
Error: rendered manifests contain a resource that already exists. Unable to continue with install: PodSecurityPolicy "loki" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-namespace" must equal "fe-foundry": current value is "ps-intl".
```
If the helm upgrade is successful, the following is a sample output.
```
Release "fe-foundry-loki" does not exist. Installing it now.
W0810 11:47:07.890370 39624 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0810 11:47:09.396246 39624 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME: fe-foundry-loki
LAST DEPLOYED: Thu Aug 10 11:47:07 2023
NAMESPACE: fe-foundry
STATUS: deployed
REVISION: 1
NOTES:
The Loki stack has been deployed to your cluster. Loki can now be added as a datasource in Grafana.
See http://docs.grafana.org/features/datasources/loki/ for more detail.
```
## Obtain Admin credentials for Grafana
1. After you validate Grafana is running by accessing `:3000`, run the following command to obtain an `` for your Grafana instance:
```
kubectl get secret --namespace -monitoring-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```
2. Sign in to Grafana and change the password for security purposes.
3. Run the following command to display the promtail pods that are running:
```
kubectl get pods | grep -i promtail | nl
```
Promtail pods must match the number of Kubernetes nodes since an instance of Promtail runs on each node.
## Add the Loki datsource
1. Sign in to Grafana and in the toolbar, click the arrow below Home to display all of the options.
2. In the Configuration section, click **Data sources**.
3. Click **Add new data source**.
4. In the search bar for the data source, enter **Loki**.
5. In the URL field on the Settings screen, enter your unique ``. The default port for Loki is `3100`.
If you encounter issues with the `` information, open a terminal and run `kubectl get services | grep loki` to display a list of every service with a name that contains `loki` along with its associated IP address and port.
6. Complete the other fields and click **Save & test**.
Affinity rules govern how Kubernetes schedules pods for Fusion components across the cluster. All components have the same affinity setup, which follows this logic:
* When scheduling, prefer to put a pod on a node in an availability zone that doesn’t have a running instance of this component.
* Require that pods are deployed on a host that doesn’t have a running instance of the component that is being scheduled.
With this logic, the loss of a host can only bring down one component at most. However, the cluster must be at least as large as the number of replicas in the largest deployment.
To run a large number of a component, consider relaxing the "required" policy by changing it to a "preferred" policy on the hostname for the `kubernetes.io/hostname` policies:
| | |
| - | - |
| Before | requiredDuringSchedulingIgnoredDuringExecution: |
| After | preferredDuringSchedulingIgnoredDuringExecution: |
If you used the `--with-affinity-rules` option when running the `./customize_fusion_values.sh` script, the pod affinity rules are configured for your cluster. Alternatively, copy [`affinity.yaml`](https://github.com/lucidworks/fusion-cloud-native/blob/master/example-values/affinity.yaml), and rename it using the following naming convention: `___fusion_affinity.yaml`.
To implement the file, append the following to your upgrade script:
```
MY_VALUES="${MY_VALUES} --values gke_search_f5_fusion_affinity.yaml"
```
Determining resource limits is a process that should take place after the initial setup of your cluster. This is especially true for proof-of-concept clusters. Once you have configured your cluster to accommodate a critical mass of data, tune the resource limits for your Fusion services.
In the case of production or production-like environments, define resource limits to help Kubernetes schedule pods correctly across the nodes in your cluster. This is important for Kubernetes clusters that host namespaces other than Fusion.
If you used the `--with-resource-limits` option when running the `./customize_fusion_values.sh` script, resource limits are already configured for your cluster. The script creates a YAML file for this purpose named `___fusion_resources.yaml`. Alternatively, you can copy [`resources.yaml`](https://github.com/lucidworks/fusion-cloud-native/blob/master/example-values/resources.yaml), and rename it using the following naming convention: `___fusion_resources.yaml`.
You can refine the resource requests and limits as you test your cluster’s behavior, while preparing for a production environment with Fusion.
### Node Selectors
You can control which nodes Spark executors are scheduled on using a Spark configuration property for a job:
```
spark.kubernetes.node.selector.
This guide describes how to perform Fusion 5 upgrades.
Before upgrading, be aware of changes by checking for [Deprecations and Removals](/docs/5/fusion/deprecations-and-removals) between versions.
Lucidworks recommends upgrading to the next minor version only. For example, you should upgrade from Fusion 5.6.1 to Fusion 5.7.1 before upgrading to Fusion 5.8.0.
The [general upgrade process](#general-upgrade-process) is described in this article. However, the specific upgrade procedures may vary depending on your upgrade path. For the most accurate instructions, please refer to the upgrade article specific to your upgrade.
## General upgrade process
Fusion natively supports deployments on supported Kubernetes platforms, including AKS, EKS, and GKE.
Fusion includes an upgrade script for AKS, EKS, and GKE. This script is not generated for other Kubernetes deployments.
Upgrades differ from platform to platform. See below for more information about upgrading on your platform of choice.
Whenever you upgrade Fusion, you must also update your [remote connectors](/docs/fusion-connectors/developers/remote-v2-connectors), if you are running any.
You can download the latest files at [V2 Connectors Downloads](/docs/fusion-connectors/downloads/v2-connectors-downloads).
### Natively supported deployment upgrades
| Deployment type | Platform |
| - | - |
| **Azure Kubernetes Service (AKS)** | `aks` |
| **Amazon Elastic Kubernetes Service (EKS)** | `eks` |
| **Google Kubernetes Engine (GKE)** | `gke` |
Fusion includes upgrade scripts for natively supported deployment types. To upgrade:
1. Open the `___upgrade_fusion.sh` upgrade script file for editing.
2. Update the `CHART_VERSION` to your target Fusion version, and save your changes.
3. Run the `___upgrade_fusion.sh` script. The `` value is the same as your namespace, unless you overrode the default value using the `-r` option.
After running the upgrade, use `kubectl get pods` to see the changes applied to your cluster. It may take several minutes to perform the upgrade, as new Docker images are pulled from DockerHub. To see the versions of running pods, do:
```
kubectl get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq
```
### Other Kubernetes deployment upgrades
To update an existing installation, do:
```
RELEASE=f5
NAMESPACE=default
helm repo update
helm upgrade ${RELEASE} "lucidworks/fusion" --namespace "${NAMESPACE}" --values "${MY_VALUES}"
```
Except for ZooKeeper, all K8s deployments and statefulsets use a `RollingUpdate` update policy:
```yaml theme={"dark"}
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
```
ZooKeeper instances use `OnDelete` to avoid changing critical stateful pods in the Fusion deployment. To apply changes to Zookeeper after performing the upgrade (uncommon), you need to manually delete the pods. For example:
```
kubectl delete pod f5-zookeeper-0
```
Delete one pod at a time. Verify the new pod is healthy and serving traffic, before deleting the next healthy pod.
You can also set the `updateStrategy` under the `zookeeper` section in your `"${MY_VALUES}"` file:
```yaml theme={"dark"}
solr:
...
zookeeper:
updateStrategy:
type: "RollingUpdate"
```
#### Upgrades with Helm v3
One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. For example, Fusion 5 allows customers to upgrade from Fusion 5.1.0 to a later 5.x.y version on a live cluster with zero downtime or disruption of service.
When Kubernetes performs a rolling update to an individual microservice, there is a mix of old and new services in the cluster. Requests from other services route to both versions.
Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same minor release version (5.x). We also ensure that the stored configuration remains compatible in the same minor release version.
Lucidworks releases minor updates to individual services frequently. Pull in those upgrades using Helm at your discretion.
**How to upgrade Fusion**
1. Clone the [**fusion-cloud-native** repo](https://github.com/lucidworks/fusion-cloud-native), if you haven’t already.
2. Locate the `setup_f5_.sh` script that matches your Kubernetes platform.
3. Run the script with the `--upgrade` option.
To see what would be upgraded, pass the `--dry-run` option to the script.
The scripts in the **fusion-cloud-native** repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.
#### Helm upgrade script
Once you deploy a working cluster, use the upgrade script created by the `customize_fusion_values.sh` script. The upgrade script hard-codes the parameters and eases the need to remember which parameters to pass to the script. This is helpful when working with multiple K8s clusters. Make sure you check the script into version control alongside your custom values YAML files.
Whenever you change the custom values YAML files for your cluster, you need to run the upgrade script to apply the changes. The script calls `helm upgrade` with the correct parameters and `--values` options.
If you run `helm upgrade` without passing the custom values YAML files, the deployment will revert to using chart defaults, which you never want to do.The script assumes your `kubeconfig` is pointing to the correct cluster and you’re using Heml v3. If not, the upgrade fails. Select the correct `kubeconfig` before running the script.
# Which Kubernetes?
Source: https://doc.lucidworks.com/docs/5/fusion/operations/survival-guide/platforms
[localhost link]: http://localhost:3000/docs/5/fusion/operations/survival-guide/platforms
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/survival-guide/platforms
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/148
The image above is for illustrative purposes only. It depicts services available in Fusion 5.9, but your actual implementation may differ.
To help users get started with Fusion 5, Lucidworks provides setup scripts for Google Kubernetes Engine (GKE), Amazon Elastic Kubernetes Service (EKS), and Azure Kubernetes Service (AKS). It’s possible to use other Kubernetes providers, but the instructions and details provided in this article target the Kubernetes providers listed within release notes. To run Fusion 5, use your cloud provider’s stable release channel.
Although GKE refers to the latest stable version as the "stable release channel," your cloud provider may use different terminology.
For more information, see:
* [GKE Release Channels](https://cloud.google.com/kubernetes-engine/docs/concepts/release-channels)
* [EKS Kubernetes Release Calendar](https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-release-calendar)
* [AKS Kubernetes Release Calendar](https://docs.microsoft.com/en-us/azure/aks/supported-kubernetes-versions#aks-kubernetes-release-calendar)
Before installing Fusion, review the release notes for the specific version you are installing to confirm the minimum Helm version required. Helm version requirements may vary across Fusion releases.
## More information
Here are instructions for deploying on some popular Kubernetes platforms:
Fusion supports deployment on Google Kubernetes Engine (GKE). This topic explains how to deploy a Fusion cluster on GKE using the [`setup_f5_gke.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_gke.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository.
## Prerequisites
This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes.
### Release Name and Namespace
Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) Kubernetes namespace to install Fusion into.
Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*.
However, please *do not* install more than one Fusion release in the same namespace.
All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_
Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default.
### Install Helm
Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster.
Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform.
On MacOS, you can do:
```bash theme={"dark"}
brew install kubernetes-helm
```
If you already have helm installed, make sure you're using the latest version:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/)
The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`.
#### Helm User Permissions
If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory.
When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.:
```bash theme={"dark"}
alias k=kubectl
```
To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into:
```bash theme={"dark"}
k create namespace fusion-namespace
```
Apply the `role.yaml` and `cluster-role.yaml` files to that namespace
```bash theme={"dark"}
k apply -f cluster-role.yaml
k config set-context --current --namespace=$NAMESPACE
k apply -f role.yaml
```
Then bind the rolebinding and clusterolebinding to the install user:
```bash wrap theme={"dark"}
k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user
k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user
```
You will then be able to run the `helm install` command as the ``
### Clone fusion-cloud-native from GitHub
You should clone this repo from github as you'll need to run the scripts on your local workstation:
```bash theme={"dark"}
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts.
```bash theme={"dark"}
cd fusion-cloud-native
git pull
```
Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip).
Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory.
## Prerequisites
This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes.
### Release Name and Namespace
Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) Kubernetes namespace to install Fusion into.
Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*.
However, please *do not* install more than one Fusion release in the same namespace.
All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_
Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default.
### Install Helm
Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster.
Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform.
On MacOS, you can do:
```bash theme={"dark"}
brew install kubernetes-helm
```
If you already have helm installed, make sure you're using the latest version:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/)
The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`.
#### Helm User Permissions
If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory.
When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.:
```bash theme={"dark"}
alias k=kubectl
```
To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into:
```bash theme={"dark"}
k create namespace fusion-namespace
```
Apply the `role.yaml` and `cluster-role.yaml` files to that namespace
```bash theme={"dark"}
k apply -f cluster-role.yaml
k config set-context --current --namespace=$NAMESPACE
k apply -f role.yaml
```
Then bind the rolebinding and clusterolebinding to the install user:
```bash wrap theme={"dark"}
k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user
k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user
```
You will then be able to run the `helm install` command as the ``
### Clone fusion-cloud-native from GitHub
You should clone this repo from github as you'll need to run the scripts on your local workstation:
```bash theme={"dark"}
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts.
```bash theme={"dark"}
cd fusion-cloud-native
git pull
```
Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip).
Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory.
The [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_gke.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_gke.sh) `setup_f5_gke.sh` script provided in this repo is strictly optional.
The script is mainly to help those new to Kubernetes and/or Fusion get started quickly.
If you're already familiar with K8s, Helm, and GKE, then you can skip the script and just use Helm directly to install Fusion into an existing cluster or one you create yourself using the process described `<>`.
### Set up the Google Cloud SDK (one time only)
If you've already installed the `gcloud` command-line tools, you can skip to `<>`.
These steps set up your local Google Cloud SDK environment so that you're ready to use the command-line tools to manage your Fusion deployment.
Usually, you only need to perform these setup steps once. After that, you're ready to [create a cluster](#create-a-single-node-demo-cluster).
For a nice getting started tutorial for GKE, see: [https://cloud.google.com/kubernetes-engine/docs/deploy-app-cluster](https://cloud.google.com/kubernetes-engine/docs/deploy-app-cluster)
How to set up the Google Cloud SDK:
1. [https://console.cloud.google.com/apis/library/container.googleapis.com?q=kubernetes%20engine](https://console.cloud.google.com/apis/library/container.googleapis.com?q=kubernetes%20engine)
2. Log in to Google Cloud: `gcloud auth login`
3. Set up the Google Cloud SDK:
1. `gcloud config set compute/zone `
If you are working with regional clusters instead of zone clusters, use `gcloud config set compute/region ` instead.
2. `gcloud config set core/account `
3. *New GKE projects only:* `gcloud projects create `
If you have already created a project, for example in [https://console.cloud.google.com/](https://console.cloud.google.com/), then skip to the next step.
4. `gcloud config set project `
Make sure you install the Kubernetes command-line tool `kubectl` using:
```bash theme={"dark"}
gcloud components install kubectl
gcloud components update
```
### Create a single-node demo cluster
Run the [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_gke.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_gke.sh) `setup_f5_gke.sh` script to install Fusion 5.x in a GKE cluster. To create a new, single-node *demo* cluster and install Fusion, simply do:
```bash theme={"dark"}
./setup_f5_gke.sh -c -p --create demo
```
Use the `--help` option to see script usage. If you want the script to create a cluster for you, then you need to pass the `--create` option with either `demo` or `multi_az`. If you don't want the script to create a cluster, then you need to create a cluster before running the script and simply pass the name of the existing cluster using the `-c` parameter.
If you pass `--create demo` to the script, then we create a single node GKE cluster (defaults to using `n1-standard-8` node type). The minimum node type you'll need for a 1 node cluster is an `n1-standard-8` (on GKE) which has 8 CPU and 30 GB of memory. This is cutting it very close in terms of resources as you also need to host all of the Kubernetes system pods on this same node. Obviously, this works for kicking the tires on Fusion 5.1 but is not sufficient for production workloads.
You can change the instance type using the `-i` parameter; see: [https://cloud.google.com/compute/docs/regions-zones/#available](https://cloud.google.com/compute/docs/regions-zones/#available) for a list of which machine types are available in your desired region.
If not provided the script generates a custom values file named `gke___fusion_values.yaml` which you can use to customize the Fusion chart.\_\_
\#WARNING# If using Helm V2, the `setup_f5_gke.sh` script installs Helm's `tiller` component into your GKE cluster with the cluster admin role. If you don't want this, then please upgrade to Helm v3.
If you see an error similar to the following, then wait a few seconds and try running the `setup_f5_gke.sh` script again with the same arguments as this is usually a transient issue:
```bash wrap theme={"dark"}
Error: could not get apiVersions from Kubernetes: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
```
After running the `setup_f5_gke.sh` script, proceed to the `<>` section below.
When you're ready to deploy Fusion to a production-like environment, see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
### Create a three-node regional cluster to withstand a zone outage
With a three-node regional cluster, nodes are deployed across three separate availability zones.
```bash wrap theme={"dark"}
./setup_f5_gke.sh -c -p -n --region --create multi_az
```
* `` value should be the name of a non-existent cluster; the script will create the new cluster.
* `` must match the name of an existing project in GKE. Run `gcloud config get-value project` to get this value, or see the [GKE setup instructions](#set-up-the-google-cloud-sdk-one-time-only).
* `` Kubernetes namespace to install Fusion into, defaults to `default` with release `f5`
* `` value should be the name of a GKE region, defaults to `us-west1`. Run `gcloud config get-value compute/zone` to get this value, or see the [GKE setup instructions](#set-up-the-google-cloud-sdk-one-time-only) to set the value.
In this configuration, Kubernetes deploys a ZooKeeper and Solr pod on each of the three nodes, which allows the cluster to retain ZK quorum and remain operational after losing one node, such as during an outage in one availability zone.
When running in a multi-zone cluster, each Solr node has the `solr_zone` system property set to the zone it is running in, such as `-Dsolr_zone=us-west1-a`.
After running the `setup_f5_gke.sh` script, proceed to the `<>` section below.
When you're ready to deploy Fusion to a production-like environment, see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
#### GKE Ingress and TLS
The Fusion proxy service provides authentication and serves as an API gateway for accessing all other Fusion services.
It's typical to use an Ingress for TLS termination in front of the proxy service.
The `setup_f5_gke.sh` supports creating an Ingress with a TLS cert for a domain you own by passing: `-t -h `
After the script runs, you need to create an A record in GCP's DNS service to map your domain name to the Ingress IP. Once this occurs, our script setup uses [https://letsencrypt.org/](https://letsencrypt.org/) to issue a TLS cert for your Ingress.
To see the status of the Let's Encrypt issued certificate, do:
```bash theme={"dark"}
kubectl get managedcertificates -n -o yaml
```
Please refer to the Kubernetes documentation on configuring an Ingress for GKE: [https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer](https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer)
The GCP Ingress defaults to a 30 second timeout, which can lead to false negatives for long running requests such as importing apps. To configure the timeout for the backend in kubernetes:
Create a BackendConfig object in your namespace:
```yaml theme={"dark"}
---
apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
name: backend_config_name
spec:
timeoutSec: 120
connectionDraining:
drainingTimeoutSec: 60
```
Then make sure that the following entries are in the right place in your values.yaml file:
```yaml theme={"dark"}
api-gateway:
service:
annotations:
beta.cloud.google.com/backend-config: '{"ports": {"6764":"backend_config_name"}}'
```
and upgrade your release to apply the configuration changes
#### Ingresses and externalTrafficPolicy
When running a fusion cluster behind an externally controlled LoadBalancer it can be advantageous
to configure the `externalTrafficPolicy` of the `proxy` service to `Local`. This preserves the client
source IP and avoids a second hop for LoadBalancer and NodePort type services, but risks potentially
imbalanced traffic spreading. Although when running in a cluster with a dedicated pool for spark jobs
that can scale up and down freely it can prevent unwanted request failures. This behaviour can be
altered with the `api-gateway.service.externalTrafficPolicy` value, which is set to `Local` if the example values
file is used.
**You must use `externalTrafficPolicy`=`Local` for the Trusted HTTP Realm to work correctly.**
If you are already using a custom 'values.yaml' file, create an entry for `externalTrafficPolicy` under `api-gateway` service.
```yaml theme={"dark"}
api-gateway:
service:
externalTrafficPolicy: Local
```
#### Considerations when using the nginx ingress controller
If you are using the `nginx` ingress controller to fulfil your ingress definitions there are a couple
of options that are recommended to be set in the configmap:
```bash wrap theme={"dark"}
enable-underscores-in-headers: "true" # Fusion can return some headers that have underscores, these have to be explicitly enabled in nginx
proxy-body-size: "0" # By default nginx places a maximum size on request bodies, either increase as needed or disable by setting to 0
proxy-read-timeout: "300" # Increases the timeout for potential slow queries.
```
#### Custom values
There are some example values files that can be used as a starting point for
resources, affinity and replica count configuration in the `example-values` folder.
These can be passed to the install script using the `--values` option, for example:
```bash wrap theme={"dark"}
./setup_f5_gke.sh -c -p -r -n \
--values example-values/affinity.yaml --values example-values/resources.yaml --values example-values/replicas.yaml
```
The `--values` option can be passed multiple times, if the same configuration property is contained within multiple `values` files then the values from the latest file passed as a `--values` option are used.
##### Connectors custom values
If you are using Fusion 5.9 or later, you can specify resources and replica count per connector. This allows you to set different resource limits for each connector. If you do not set custom values for a connector, that connector uses the default values.
Set each connector's resource values in the `connector-plugin` section under `pluginValues`. The `pluginValues` section is a list of plugins and its resources. The following sample shows an example.
```yaml theme={"dark"}
pluginValues:
- id: "plugin-id" <1>
resources: <2>
limits:
cpu: "2"
memory: "3Gi"
requests:
cpu: "250m"
memory: "2Gi"
replicaCount: 1 <3>
```
* `<1>` The plugin ID. The plugin ID must match the plugin ID on the plugin ZIP file. without the `lucidworks.` prefix. For example, if the plugin ID on the plugin ZIP file is `lucidworks.sharepoint-optimized`, the plugin ID is `sharepoint-optimized`.
* `<2>` The resources settings. You may specify the limits, the requests, and the CPU and memory for each.
* `<3>` The number of replicas per connector. This value is 1 by default.
After editing the `connector-plugin` section, you must reinstall the affected connector.
#### Upgrades and Ingress
If you used the `-t -h ` options when installing your cluster, our script created an additional values yaml file named `tls-values.yaml`.
To make things easier for you when upgrading, you should add the settings from this file into your main custom values yaml file, e.g.:
```yaml theme={"dark"}
api-gateway:
service:
type: "NodePort"
ingress:
enabled: true
host: ""
tls:
enabled: true
annotations:
"networking.gke.io/managed-certificates": "-managed-certificate"
"kubernetes.io/ingress.class": "gce"
```
This way you don't have to remember to pass the additional `tls-values.yaml` file when upgrading.
## Verifying the Fusion Installation
In this section, we provide some tips on how to verify the Fusion installation.
Check if the Fusion Admin UI is available at `\https://:6764/admin/`.
Let's review some useful kubectl commands.
### Enhance the K8s Command-line Experience
Here is a list of tools we found useful for improving your command-line experience with Kubernetes:
* krew (kubectl plugin mgr): [https://github.com/kubernetes-sigs/krew/](https://github.com/kubernetes-sigs/krew/)
* kube-ps1 (show current context on command line prompt): [https://github.com/jonmosco/kube-ps1](https://github.com/jonmosco/kube-ps1)
* kubectx / kubens (switch between clusters / namespaces): [https://github.com/ahmetb/kubectx](https://github.com/ahmetb/kubectx)
### Useful kubectl commands
kubectl reference: [https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands)
Set the namespace for `kubectl` if not using the default:
```bash theme={"dark"}
kubectl config set-context --current --namespace=
```
**This saves you from having to pass `-n` with every command.**
Get a list of running pods: `k get pods`
Get logs for a pod using a label: `k logs –l app.kubernetes.io/component=query-pipeline`
Get pod deployment spec and details: `k get pods -o yaml`
Get details about a pod events: `k describe po `
Port forward to a specific pod: `k port-forward 8983:8983`
SSH into a pod: `k exec -it -- /bin/bash`
CPU/Memory usage report for pods: `k top pods`
Forcefully kill a pod: `k delete po --force --grace-period 0`
Scale up (or down) a deployment: `k scale deployment.v1.apps/ --replicas=N`
Get a list of pod versions: `k get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq`
### Check Fusion Pods and Services
Once the install script completes, you can check that all pods and services are available using:
```bash theme={"dark"}
kubectl get pods
```
If all goes well, you should see a list of pods similar to:
```bash wrap theme={"dark"}
NAME READY STATUS RESTARTS AGE
seldon-controller-manager-6675874894-qxwrv 1/1 Running 0 8m45s
f5-admin-ui-74d794f4f8-m5jms 1/1 Running 0 8m45s
f5-ambassador-fd6b9b5dc-7ghf6 1/1 Running 0 8m43s
f5-api-gateway-6b9998b9c-tmchk 1/1 Running 0 8m45s
f5-auth-ui-7565564b4c-rdc74 1/1 Running 0 8m42s
f5-classic-rest-service-0 1/1 Running 3 8m44s
f5-devops-ui-77bb867ffb-fbzxd 1/1 Running 0 8m42s
f5-fusion-admin-78b8f8fc7f-4d7l8 1/1 Running 0 8m42s
f5-fusion-indexing-599c8d448-xzsvm 1/1 Running 0 8m44s
f5-insights-665fd9f6fc-g5psw 1/1 Running 0 8m43s
f5-job-launcher-84dd4c5c96-p8528 1/1 Running 0 8m44s
f5-job-rest-server-6d44d964b8-xtnxw 1/1 Running 0 8m45s
f5-logstash-0 1/1 Running 0 8m45s
f5-ml-model-service-6987dc94c9-9ppp8 2/2 Running 1 8m45s
f5-monitoring-grafana-5d499dbb58-pzw72 1/1 Running 0 10m
f5-monitoring-prometheus-kube-state-metrics-54d6678dv9h7h 1/1 Running 0 10m
f5-monitoring-prometheus-pushgateway-7d65c65b85-vwrwf 1/1 Running 0 10m
f5-monitoring-prometheus-server-0 2/2 Running 0 10m
f5-pm-ui-86cbc5bb65-nd2n8 1/1 Running 0 8m44s
f5-pulsar-bookkeeper-0 1/1 Running 0 8m45s
f5-pulsar-broker-b56cc776f-56msx 1/1 Running 0 8m45s
f5-query-pipeline-5d75d7d5f4-l2mdf 1/1 Running 0 8m43s
f5-connectors-7bb6cfc65f-7wfs2 1/1 Running 0 8m42s
f5-connectors-backend-987fdc648-dldwv 1/1 Running 0 8m45s
f5-rules-ui-6b9d55b78f-9hzzj 1/1 Running 0 8m43s
f5-solr-0 1/1 Running 0 8m44s
f5-solr-exporter-c4687c785-jsm7x 1/1 Running 0 8m45s
f5-ui-6cdbcc68c6-rj9cq 1/1 Running 0 8m45s
f5-webapps-6d6bb9bfd-hm4qx 1/1 Running 0 8m45s
f5-workflow-controller-7b66679fb7-sjbvp 1/1 Running 0 8m44s
f5-zookeeper-0 1/1 Running 0 8m45s
```
The number of pods per deployment / statefulset will vary based on your cluster size and replicaCount settings in your custom values YAML file.
Also, don't worry if you see some pods having been restarted as that just means they were too slow to come up and Kubernetes killed and restarted them.
You do want to see at least one pod running for every service. If a pod is not running after waiting a sufficient amount of time,
use `kubectl logs ` to see the logs for that pod; to see the logs for previous versions of a pod, use: `kubectl logs -p`.
You can also look at the actions Kubernetes performed on the pod using `kubectl describe po `.
To see a list of Fusion services, do:
```bash theme={"dark"}
kubectl get svc
```
For an overview of the various Fusion 5 microservices, see: [Fusion microservices](/docs/5/fusion/reference/microservices).
Once you're ready to build a Fusion cluster for production, please see see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
### Upgrading with Zero Downtime
One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. Fusion 5 allows customers to upgrade from Fusion 5.x.y to a later 5.x.z version on a live cluster with zero downtime or disruption of service.
When Kubernetes performs a rolling update to an individual microservice, there will be a mix of old and new services in the cluster concurrently (only briefly in most cases) and requests from other services will be routed to both versions. Consequently, Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same 5.x line of releases. We also ensure stored configuration remains compatible in the same 5.x release line.
Lucidworks releases minor updates to individual services frequently, so our customers can pull in those upgrades using Helm at their discretion.
To upgrade your cluster at any time, use the `--upgrade` option with our setup scripts in this repo.
The scripts in this repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.
To see what would be upgraded, you can pass the `--dry-run` option to the script.
### Grafana Dashboards
Get the initial Grafana password from a K8s secret by doing:
```bash wrap theme={"dark"}
kubectl get secret --namespace "${NAMESPACE}" ${RELEASE}-monitoring-grafana \
-o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```
With Grafana, you can either setup a temporary port-forward to a Grafana pod or expose Grafana on an external IP using a K8s LoadBalancer.
To define a LoadBalancer, do (replace \$ with your Helm release label):
```bash wrap theme={"dark"}
kubectl expose deployment ${RELEASE}-monitoring-grafana --type=LoadBalancer --name=grafana --port=3000 --target-port=3000
```
You can use `kubectl get services --namespace ` to determine when the load balancer is setup and its IP address. Direct your browser to `http://:3000` and enter the username `admin@localhost` and the password that was returned in the previous step.
This will log you into the application. It is recommended that you create another administrative user with a more desirable password.
The dashboards and datasoure will be setup for you in grafana, simply navigate to `Dashboards` -> `Manage` to view the available dashboards
## Learn more
The course for **Deploying Fusion 5** focuses on the prerequisite software needed to deploy Fusion, the necessary setup steps, and the physical act of deployment.
Fusion supports deployment on Amazon Elastic Kubernetes Service (EKS). This topic explains how to deploy a Fusion cluster on EKS using the [`setup_f5_eks.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_eks.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository.
In addition, this topic provides information about how to [configure IAM roles for the service account](#configure-iam-roles-for-the-service-account).
## Prerequisites
This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes.
### Release Name and Namespace
Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) Kubernetes namespace to install Fusion into.
Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*.
However, please *do not* install more than one Fusion release in the same namespace.
All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_
Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default.
### Install Helm
Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster.
Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform.
On MacOS, you can do:
```bash theme={"dark"}
brew install kubernetes-helm
```
If you already have helm installed, make sure you're using the latest version:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/)
The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`.
#### Helm User Permissions
If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory.
When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.:
```bash theme={"dark"}
alias k=kubectl
```
To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into:
```bash theme={"dark"}
k create namespace fusion-namespace
```
Apply the `role.yaml` and `cluster-role.yaml` files to that namespace
```bash theme={"dark"}
k apply -f cluster-role.yaml
k config set-context --current --namespace=$NAMESPACE
k apply -f role.yaml
```
Then bind the rolebinding and clusterolebinding to the install user:
```bash wrap theme={"dark"}
k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user
k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user
```
You will then be able to run the `helm install` command as the ``
### Clone fusion-cloud-native from GitHub
You should clone this repo from github as you'll need to run the scripts on your local workstation:
```bash theme={"dark"}
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts.
```bash theme={"dark"}
cd fusion-cloud-native
git pull
```
Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip).
Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory.
The [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_eks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_eks.sh) `setup_f5_eks.sh` script provided in this repo is strictly optional.
The script is mainly to help those new to Kubernetes and/or Fusion get started quickly.
If you're already familiar with K8s, Helm, and EKS, then you use Helm directly to install Fusion into an existing cluster or one you create yourself using the process described `<>`.
If you're new to Amazon Web Services (AWS), then please visit the Amazon Web Services [https://aws.amazon.com/getting-started/](https://aws.amazon.com/getting-started/) to set up an account.
If you're new to Kubernetes and EKS, then we recommend going through Amazon's [https://eksworkshop.com/introduction/](https://eksworkshop.com/introduction/) before proceeding with Fusion.
### Set up the AWS CLI tools
Before launching an EKS cluster, you need to install and configure `kubectl`, `aws`, `eksctl`, `aws-iam-authenticator` using the links provided below:
Required AWS Command-line Tools:
* kubectl: [https://kubernetes.io/docs/tasks/tools/install-kubectl/](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
* aws: [https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html)
* eksctl: [https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html)
* aws-iam-authenticator: [https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html](https://docs.aws.amazon.com/eks/latest/userguide/install-aws-iam-authenticator.html)
Run `aws configure` to configure a profile for authenticating to AWS. You'll use the profile name you configure in this step, which defaults to `default`, as the `-p` argument to the `setup_f5_eks.sh` script in the next section.
When working in Ubuntu, avoid using the eksctl snap version. Alternative sources can have different versions that could cause command failures. Also, always make sure you are using the latest version for each one of the required tools.
### Set up Fusion on EKS
To create a cluster in EKS the following IAM policies are required:
* AmazonEC2FullAccess
* AWSCloudFormationFullAccess
EKS Permissions:
```bash theme={"dark"}
eks:CreateCluster
eks:DeleteCluster
eks:DescribeCluster
eks:DescribeUpdate
eks:ListClusters
eks:ListUpdates
eks:UpdateClusterVersion
```
VPC Permissions:
```bash theme={"dark"}
ec2:AssociateSubnetCidrBlock
ec2:AssociateVpcCidrBlock
ec2:AttachInternetGateway
ec2:CreateInternetGateway
ec2:CreateSubnet
ec2:CreateVpc
ec2:CreateVpcEndpoint
ec2:DeleteInternetGateway
ec2:DeleteSubnet
ec2:DeleteVpc
ec2:DeleteVpcEndpoints
ec2:DescribeSubnets
ec2:DescribeVpcAttribute
ec2:DescribeVpcs
ec2:DetachInternetGateway
ec2:DisassociateSubnetCidrBlock
ec2:DisassociateVpcCidrBlock
ec2:ModifySubnetAttribute
ec2:ModifyVpcAttribute
ec2:ModifyVpcEndpoint
```
.IAM Permissions
```bash theme={"dark"}
iam:AddRoleToInstanceProfile
iam:AttachRolePolicy
iam:CreateInstanceProfile
iam:CreatePolicy
iam:CreatePolicyVersion
iam:CreateRole
iam:DeleteInstanceProfile
iam:DeletePolicy
iam:DeletePolicyVersion
iam:DeleteRole
iam:DeleteRolePolicy
iam:DetachRolePolicy
iam:GetInstanceProfile
iam:GetPolicy
iam:GetPolicyVersion
iam:GetRole
iam:GetRolePolicy
iam:ListInstanceProfiles
iam:ListInstanceProfilesForRole
iam:PassRole
iam:PutRolePolicy
iam:RemoveRoleFromInstanceProfile
iam:TagRole
iam:UntagRole
```
Download and run the [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_eks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_eks.sh) `setup_f5_eks.sh` script to install Fusion 5.x in an EKS cluster.
This script does not support multiple node pools and should not be used for production clusters.
* To create a new cluster and install Fusion, run the following command:
```
./setup_f5_eks.sh -c my-eks-cluster -p profile-name -n fusion-namespace --create demo
```
* Replace `my-eks-cluster`, `profile-name`, and `fusion-namespace` with your cluster, profile, and namespace values.
* Pass the `--create` option with either `demo` or `multi_az`.
* To use an existing cluster and install Fusion, run the following command:
```
./setup_f5_eks.sh -c cluster-name -p profile-name
```
* Replace `cluster-name` with the name of the cluster you already created.
* Replace `profile-name` with the name of your profile.
The profile is automatically set to `default` if you ran the AWS configure command without giving the profile a name.
Use the `--help` option to see full script usage.
If using Helm V2, the `setup_f5_eks.sh` script installs Helm's `tiller` component into your EKS cluster with the cluster admin role. If you don't want this, then please upgrade to Helm v3.
The `setup_f5_eks.sh` script creates a service account that provides S3 read-only permissions to the created pods.
After running the `setup_f5_eks.sh` script, proceed to the `<>` section below.
#### EKS cluster overview
The EKS cluster is created using `eksctl` ([https://eksctl.io/](https://eksctl.io/)). By default it will setup the following resources in your AWS account:
* A dedicated VPC for the EKS cluster in the specified region with CIDR: `192.168.0.0/16`
* 3 Public and 3 Private subnets within the created VPC, each with a `/19` CIDR range, along with the corresponding route tables.
* A NAT gateway in each Public subnet
* An Auto Scaling Group of the instance type specified by the script, which defaults to `m5.2xlarge`, with 3 instances spanning the public subnets.
See [https://eksctl.io/usage/vpc-networking/](https://eksctl.io/usage/vpc-networking/) for more information on the networking setup.
#### EKS Ingress
The `setup_f5_eks.sh` script exposes the Fusion proxy service on an external DNS name provided by an ELB over HTTP. This is done for demo or getting started purposes. However, you're strongly encouraged to configure a K8s Ingress with TLS termination in front of the proxy service.
See: [https://aws.amazon.com/premiumsupport/knowledge-center/terminate-https-traffic-eks-acm/](https://aws.amazon.com/premiumsupport/knowledge-center/terminate-https-traffic-eks-acm/)
Our EKS script creates a classic ELB for exposing fusion proxy service. In case you need to change this behavior and use [https://github.com/kubernetes-sigs/aws-load-balancer-controller](https://github.com/kubernetes-sigs/aws-load-balancer-controller) instead you can use the following parameters when running the `setup_f5_eks.sh` script:
```bash theme={"dark"}
--deploy-alb # Tells the script to deploy an ALB
```
By default the `kube-system` namespace is being used for installing the `aws-load-balancer-controller` because pods `priorityClassName` is set to `system-cluster-critical`.
In case you need to deploy an internal ALB you can use the `--internal-alb` option. This will create the nodes in the internal subnets. Fusion will be reachable from an AWS instance located in any of the external subnets on the same VPC. To use an ALB also an ingress with a DNS name is required, you can use the `-h` option to create an ingress with the required DNS name.
Finally, use Route 53 or your DNS provider for creating an A ALIAS DNS record for your DNS name pointing to the ingress ADRESS. You can get the address listing the ingress using the command `kubectl get ing`.
## Verifying the Fusion Installation
In this section, we provide some tips on how to verify the Fusion installation.
Check if the Fusion Admin UI is available at `\https://:6764/admin/`.
Let's review some useful kubectl commands.
### Enhance the K8s Command-line Experience
Here is a list of tools we found useful for improving your command-line experience with Kubernetes:
* krew (kubectl plugin mgr): [https://github.com/kubernetes-sigs/krew/](https://github.com/kubernetes-sigs/krew/)
* kube-ps1 (show current context on command line prompt): [https://github.com/jonmosco/kube-ps1](https://github.com/jonmosco/kube-ps1)
* kubectx / kubens (switch between clusters / namespaces): [https://github.com/ahmetb/kubectx](https://github.com/ahmetb/kubectx)
### Useful kubectl commands
kubectl reference: [https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands)
Set the namespace for `kubectl` if not using the default:
```bash theme={"dark"}
kubectl config set-context --current --namespace=
```
**This saves you from having to pass `-n` with every command.**
Get a list of running pods: `k get pods`
Get logs for a pod using a label: `k logs –l app.kubernetes.io/component=query-pipeline`
Get pod deployment spec and details: `k get pods -o yaml`
Get details about a pod events: `k describe po `
Port forward to a specific pod: `k port-forward 8983:8983`
SSH into a pod: `k exec -it -- /bin/bash`
CPU/Memory usage report for pods: `k top pods`
Forcefully kill a pod: `k delete po --force --grace-period 0`
Scale up (or down) a deployment: `k scale deployment.v1.apps/ --replicas=N`
Get a list of pod versions: `k get po -o jsonpath='{..image}'
tr -s '[[:space:]]' '\n'
sort
uniq`
### Check Fusion Pods and Services
Once the install script completes, you can check that all pods and services are available using:
```bash theme={"dark"}
kubectl get pods
```
If all goes well, you should see a list of pods similar to:
```bash wrap theme={"dark"}
NAME READY STATUS RESTARTS AGE
seldon-controller-manager-6675874894-qxwrv 1/1 Running 0 8m45s
f5-admin-ui-74d794f4f8-m5jms 1/1 Running 0 8m45s
f5-ambassador-fd6b9b5dc-7ghf6 1/1 Running 0 8m43s
f5-api-gateway-6b9998b9c-tmchk 1/1 Running 0 8m45s
f5-auth-ui-7565564b4c-rdc74 1/1 Running 0 8m42s
f5-classic-rest-service-0 1/1 Running 3 8m44s
f5-devops-ui-77bb867ffb-fbzxd 1/1 Running 0 8m42s
f5-fusion-admin-78b8f8fc7f-4d7l8 1/1 Running 0 8m42s
f5-fusion-indexing-599c8d448-xzsvm 1/1 Running 0 8m44s
f5-insights-665fd9f6fc-g5psw 1/1 Running 0 8m43s
f5-job-launcher-84dd4c5c96-p8528 1/1 Running 0 8m44s
f5-job-rest-server-6d44d964b8-xtnxw 1/1 Running 0 8m45s
f5-logstash-0 1/1 Running 0 8m45s
f5-ml-model-service-6987dc94c9-9ppp8 2/2 Running 1 8m45s
f5-monitoring-grafana-5d499dbb58-pzw72 1/1 Running 0 10m
f5-monitoring-prometheus-kube-state-metrics-54d6678dv9h7h 1/1 Running 0 10m
f5-monitoring-prometheus-pushgateway-7d65c65b85-vwrwf 1/1 Running 0 10m
f5-monitoring-prometheus-server-0 2/2 Running 0 10m
f5-pm-ui-86cbc5bb65-nd2n8 1/1 Running 0 8m44s
f5-pulsar-bookkeeper-0 1/1 Running 0 8m45s
f5-pulsar-broker-b56cc776f-56msx 1/1 Running 0 8m45s
f5-query-pipeline-5d75d7d5f4-l2mdf 1/1 Running 0 8m43s
f5-connectors-7bb6cfc65f-7wfs2 1/1 Running 0 8m42s
f5-connectors-backend-987fdc648-dldwv 1/1 Running 0 8m45s
f5-rules-ui-6b9d55b78f-9hzzj 1/1 Running 0 8m43s
f5-solr-0 1/1 Running 0 8m44s
f5-solr-exporter-c4687c785-jsm7x 1/1 Running 0 8m45s
f5-ui-6cdbcc68c6-rj9cq 1/1 Running 0 8m45s
f5-webapps-6d6bb9bfd-hm4qx 1/1 Running 0 8m45s
f5-workflow-controller-7b66679fb7-sjbvp 1/1 Running 0 8m44s
f5-zookeeper-0 1/1 Running 0 8m45s
```
The number of pods per deployment / statefulset will vary based on your cluster size and replicaCount settings in your custom values YAML file.
Also, don't worry if you see some pods having been restarted as that just means they were too slow to come up and Kubernetes killed and restarted them.
You do want to see at least one pod running for every service. If a pod is not running after waiting a sufficient amount of time,
use `kubectl logs ` to see the logs for that pod; to see the logs for previous versions of a pod, use: `kubectl logs -p`.
You can also look at the actions Kubernetes performed on the pod using `kubectl describe po `.
To see a list of Fusion services, do:
```bash theme={"dark"}
kubectl get svc
```
For an overview of the various Fusion 5 microservices, see: [Fusion microservices](/docs/5/fusion/reference/microservices).
Once you're ready to build a Fusion cluster for production, please see see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
### Upgrading with Zero Downtime
One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. Fusion 5 allows customers to upgrade from Fusion 5.x.y to a later 5.x.z version on a live cluster with zero downtime or disruption of service.
When Kubernetes performs a rolling update to an individual microservice, there will be a mix of old and new services in the cluster concurrently (only briefly in most cases) and requests from other services will be routed to both versions. Consequently, Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same 5.x line of releases. We also ensure stored configuration remains compatible in the same 5.x release line.
Lucidworks releases minor updates to individual services frequently, so our customers can pull in those upgrades using Helm at their discretion.
To upgrade your cluster at any time, use the `--upgrade` option with our setup scripts in this repo.
The scripts in this repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.
To see what would be upgraded, you can pass the `--dry-run` option to the script.
### Grafana Dashboards
Get the initial Grafana password from a K8s secret by doing:
```bash wrap theme={"dark"}
kubectl get secret --namespace "${NAMESPACE}" ${RELEASE}-monitoring-grafana \
-o jsonpath="{.data.admin-password}"
base64 --decode ; echo
```
With Grafana, you can either setup a temporary port-forward to a Grafana pod or expose Grafana on an external IP using a K8s LoadBalancer.
To define a LoadBalancer, do (replace \$ with your Helm release label):
```bash wrap theme={"dark"}
kubectl expose deployment ${RELEASE}-monitoring-grafana --type=LoadBalancer --name=grafana --port=3000 --target-port=3000
```
You can use `kubectl get services --namespace ` to determine when the load balancer is setup and its IP address. Direct your browser to `http://:3000` and enter the username `admin@localhost` and the password that was returned in the previous step.
This will log you into the application. It is recommended that you create another administrative user with a more desirable password.
The dashboards and datasoure will be setup for you in grafana, simply navigate to `Dashboards` -> `Manage` to view the available dashboards
## Configure IAM roles for the service account
Configuring IAM roles lets you utilize the Amazon Web Services Security Token Service (AWS STS) for short-term authentication credentials to access services like the Amazon S3 simple storage service.
To configure IAM roles, your user account must be granted `admin` permissions or `IAM:FullAccess`. Complete the following steps:
1. To create the OpenID Connect (OIDC) provider, run the following command:
```bash wrap theme={"dark"}
eksctl utils associate-iam-oidc-provider --cluster cluster_name --approve
```
2. To create an IAM role for the service account associated with the plugin-pod, run the following command:
```bash wrap theme={"dark"}
eksctl create iamserviceaccount --name f5-connector-plugin --namespace default --cluster cluster_name --attach-policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess --approve --override-existing-serviceaccounts
```
This command:
* Creates an IAM role and attaches the target policy.
* Updates the existing Kubernetes `f5-connector-plugin` service account and annotates it with the IAM role.
* Uses the existing `policy/AmazonS3ReadOnlyAccess` policy.
If the IAM role was already created without the command, and you want to associate the service account, run the following command:
```bash wrap theme={"dark"}
kubectl annotate serviceaccount -n default f5-connector-plugin eks.amazonaws.com/role-arn=arn:aws:iam::411271863668:role/FUS_ROLE --overwrite=true
```
To utilize this feature, create a data source with the settings in **S3 Authentication Settings > AWS Instance Credentials Authentication Settings**. For detailed installation information, see [AWS S3 Pro connector](/docs/fusion-connectors/connectors/amazon-aws-s3-pro).
For more information, see:
* [Welcome to the AWS Security Token Service API Reference](https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html)
* [Temporary security credentials in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html)
* [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html)
* [What is Amazon EKS?](https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html)
* [Fine-grained IAM roles for service accounts](https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts)
## Learn more
The course for **Deploying Fusion 5** focuses on the prerequisite software needed to deploy Fusion, the necessary setup steps, and the physical act of deployment.
Fusion supports deployment on Azure Kubernetes Service (AKS). This topic explains how to deploy a Fusion cluster on AKS using the [`setup_f5_aks.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_aks.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository.
The `setup_f5_aks.sh` script is the basic foundation for getting started and proof-of-concept purposes. For information about custom values in a production-ready environment, see Custom values YAML file.
## Prerequisites
This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes.
### Release Name and Namespace
Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) namespace to install Fusion into.
Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*.
However, please *do not* install more than one Fusion release in the same namespace.
All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_
Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default.
### Install Helm
Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster.
Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform.
On MacOS, you can do:
```bash theme={"dark"}
brew install kubernetes-helm
```
If you already have helm installed, make sure you're using the latest version:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/)
The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`.
#### Helm User Permissions
If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory.
When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.:
```bash theme={"dark"}
alias k=kubectl
```
To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into:
```bash theme={"dark"}
k create namespace fusion-namespace
```
Apply the `role.yaml` and `cluster-role.yaml` files to that namespace
```bash theme={"dark"}
k apply -f cluster-role.yaml
k config set-context --current --namespace=$NAMESPACE
k apply -f role.yaml
```
Then bind the rolebinding and clusterolebinding to the install user:
```bash wrap theme={"dark"}
k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user
k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user
```
You will then be able to run the `helm install` command as the ``
### Clone fusion-cloud-native from GitHub
You should clone this repo from github as you'll need to run the scripts on your local workstation:
```bash theme={"dark"}
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts.
```bash theme={"dark"}
cd fusion-cloud-native
git pull
```
Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip).
Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory.
The [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_aks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_aks.sh) script provided in this repo is strictly optional.
The script is mainly to help those new to Kubernetes and/or Fusion get started quickly.
If you're already familiar with K8s, Helm, and AKS, then you use Helm directly to install Fusion into an existing cluster or one you create yourself using the process described `<>`.
If you're new to Azure, then please visit [https://azure.microsoft.com/en-us/free/search/](https://azure.microsoft.com/en-us/free/search/) to set up an account.
### Set up the AKS CLI tools
Before launching an AKS cluster, you need to install and configure `kubectl` and `az` using the links provided below:
Required AKS Command-line Tools:
* `kubectl`: [https://kubernetes.io/docs/tasks/tools/install-kubectl/](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
* `az`: [https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest)
To confirm your account access and command-line tools are set up correctly, run the `az login` command (`az login –help` to see available options).
### Azure Prerequisites
To launch a cluster in AKS (or pretty much do anything with Azure) you need to setup a Resource Group. Resource Groups are a way of organizing and managing related resources in Azure.
For more information about resource groups, see [https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview#resource-groups](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-overview#resource-groups).
You also need to choose a location where you want to spin up your AKS cluster, such as `westus2`. For a list of locations you can choose, see [https://azure.microsoft.com/en-us/global-infrastructure/locations/](https://azure.microsoft.com/en-us/global-infrastructure/locations/).
Use the Azure console in your browser to create a resource group, or simply do:
```bash theme={"dark"}
az group create -g $AZURE_RESOURCE_GROUP -l $AZURE_LOCATION
```
To recap, you should have the following requirements in place:
* Azure Account set up.
* `azure-cli` (`az`) command-line tools installed.
* `az` login working.
* Created an Azure Resource Group and selected a location to launch the cluster.
### Set up Fusion on AKS
Download and run the [https://github.com/lucidworks/fusion-cloud-native/blob/master/setup\_f5\_aks.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_aks.sh) to install Fusion 5.x in a AKS cluster. To create a new cluster and install Fusion, simply do:
```bash theme={"dark"}
./setup_f5_aks.sh -c -p
```
If you don't want the script to create a cluster, then you need to create a cluster before running the script and simply pass the name of the existing cluster using the `-c` parameter.
Use the `--help` option to see full script usage.
By default, our script installs Fusion into the default namespace; think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster in separate namespaces. However, please do not install more than one Fusion release in the same namespace.
You can override the namespace using the `-n` option. In addition, our script uses f5 for the Helm release name; you can customize this using the `-r` option. Helm uses the release name you provide to track a specific instance of an installation, allowing you to perform updates and rollback changes for that specific release only.
You can also pass the `--preview` option to the script, which enables soon-to-be-released features for AKS, such as deploying a multi-zone cluster across 3 availability zones for higher availability guarantees. For more information about the Availability Zone feature, see [https://docs.microsoft.com/en-us/azure/aks/availability-zones](https://docs.microsoft.com/en-us/azure/aks/availability-zones).
It takes a while for AKS to spin up the new cluster. The cluster will have three Standard\_D4\_v3 nodes which have 4 CPU cores and 16 GB of memory. Behind the scenes, our script calls the `az aks create` command.
If using Helm V2, the `setup_f5_aks.sh` script installs Helm's `tiller` component into your AKS cluster with the cluster admin role. If you don't want this, then please upgrade to Helm v3.
After running the `setup_f5_aks.sh` script, proceed to `<>`.
#### AKS Ingress
The `setup_f5_aks.sh` script exposes the Fusion proxy service on an external IP over HTTP. This is done for demo or getting started purposes. However, you're strongly encouraged to configure a K8s Ingress with TLS termination in front of the proxy service.
Use the `-t` and `-h ` options to have our script create an Ingress with a TLS certificate issued by Let's Encrypt.
#### Upgrades and Ingress
IMPORTANT: If you used the `-t -h ` options when installing your cluster, our script created an additional values yaml file named `tls-values.yaml`.
To make things easier for you when upgrading, you should add the settings from this file into your main custom values yaml file. For example:
```yaml theme={"dark"}
api-gateway:
service:
type: "NodePort"
ingress:
enabled: true
host: ""
tls:
enabled: true
annotations:
"networking.gke.io/managed-certificates": "-managed-certificate"
"kubernetes.io/ingress.class": "gce"
```
This way, you don't have to remember to pass the additional `tls-values.yaml` file when upgrading.
## Verifying the Fusion Installation
In this section, we provide some tips on how to verify the Fusion installation.
Check if the Fusion Admin UI is available at `\https://:6764/admin/`.
Let's review some useful kubectl commands.
### Enhance the K8s Command-line Experience
Here is a list of tools we found useful for improving your command-line experience with Kubernetes:
* krew (kubectl plugin mgr): [https://github.com/kubernetes-sigs/krew/](https://github.com/kubernetes-sigs/krew/)
* kube-ps1 (show current context on command line prompt): [https://github.com/jonmosco/kube-ps1](https://github.com/jonmosco/kube-ps1)
* kubectx / kubens (switch between clusters / namespaces): [https://github.com/ahmetb/kubectx](https://github.com/ahmetb/kubectx)
### Useful kubectl commands
kubectl reference: [https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands)
Set the namespace for `kubectl` if not using the default:
```bash theme={"dark"}
kubectl config set-context --current --namespace=
```
**This saves you from having to pass `-n` with every command.**
Get a list of running pods: `k get pods`
Get logs for a pod using a label: `k logs –l app.kubernetes.io/component=query-pipeline`
Get pod deployment spec and details: `k get pods -o yaml`
Get details about a pod events: `k describe po `
Port forward to a specific pod: `k port-forward 8983:8983`
SSH into a pod: `k exec -it -- /bin/bash`
CPU/Memory usage report for pods: `k top pods`
Forcefully kill a pod: `k delete po --force --grace-period 0`
Scale up (or down) a deployment: `k scale deployment.v1.apps/ --replicas=N`
Get a list of pod versions: `k get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq`
### Check Fusion Pods and Services
Once the install script completes, you can check that all pods and services are available using:
```bash theme={"dark"}
kubectl get pods
```
If all goes well, you should see a list of pods similar to:
```bash wrap theme={"dark"}
NAME READY STATUS RESTARTS AGE
seldon-controller-manager-6675874894-qxwrv 1/1 Running 0 8m45s
f5-admin-ui-74d794f4f8-m5jms 1/1 Running 0 8m45s
f5-ambassador-fd6b9b5dc-7ghf6 1/1 Running 0 8m43s
f5-api-gateway-6b9998b9c-tmchk 1/1 Running 0 8m45s
f5-auth-ui-7565564b4c-rdc74 1/1 Running 0 8m42s
f5-classic-rest-service-0 1/1 Running 3 8m44s
f5-devops-ui-77bb867ffb-fbzxd 1/1 Running 0 8m42s
f5-fusion-admin-78b8f8fc7f-4d7l8 1/1 Running 0 8m42s
f5-fusion-indexing-599c8d448-xzsvm 1/1 Running 0 8m44s
f5-insights-665fd9f6fc-g5psw 1/1 Running 0 8m43s
f5-job-launcher-84dd4c5c96-p8528 1/1 Running 0 8m44s
f5-job-rest-server-6d44d964b8-xtnxw 1/1 Running 0 8m45s
f5-logstash-0 1/1 Running 0 8m45s
f5-ml-model-service-6987dc94c9-9ppp8 2/2 Running 1 8m45s
f5-monitoring-grafana-5d499dbb58-pzw72 1/1 Running 0 10m
f5-monitoring-prometheus-kube-state-metrics-54d6678dv9h7h 1/1 Running 0 10m
f5-monitoring-prometheus-pushgateway-7d65c65b85-vwrwf 1/1 Running 0 10m
f5-monitoring-prometheus-server-0 2/2 Running 0 10m
f5-pm-ui-86cbc5bb65-nd2n8 1/1 Running 0 8m44s
f5-pulsar-bookkeeper-0 1/1 Running 0 8m45s
f5-pulsar-broker-b56cc776f-56msx 1/1 Running 0 8m45s
f5-query-pipeline-5d75d7d5f4-l2mdf 1/1 Running 0 8m43s
f5-connectors-7bb6cfc65f-7wfs2 1/1 Running 0 8m42s
f5-connectors-backend-987fdc648-dldwv 1/1 Running 0 8m45s
f5-rules-ui-6b9d55b78f-9hzzj 1/1 Running 0 8m43s
f5-solr-0 1/1 Running 0 8m44s
f5-solr-exporter-c4687c785-jsm7x 1/1 Running 0 8m45s
f5-ui-6cdbcc68c6-rj9cq 1/1 Running 0 8m45s
f5-webapps-6d6bb9bfd-hm4qx 1/1 Running 0 8m45s
f5-workflow-controller-7b66679fb7-sjbvp 1/1 Running 0 8m44s
f5-zookeeper-0 1/1 Running 0 8m45s
```
The number of pods per deployment / statefulset will vary based on your cluster size and replicaCount settings in your custom values YAML file.
Also, don't worry if you see some pods having been restarted as that just means they were too slow to come up and Kubernetes killed and restarted them.
You do want to see at least one pod running for every service. If a pod is not running after waiting a sufficient amount of time,
use `kubectl logs ` to see the logs for that pod; to see the logs for previous versions of a pod, use: `kubectl logs -p`.
You can also look at the actions Kubernetes performed on the pod using `kubectl describe po `.
To see a list of Fusion services, do:
```bash theme={"dark"}
kubectl get svc
```
For an overview of the various Fusion 5 microservices, see: [Fusion microservices](/docs/5/fusion/reference/microservices).
Once you're ready to build a Fusion cluster for production, please see see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
### Upgrading with Zero Downtime
One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. Fusion 5 allows customers to upgrade from Fusion 5.x.y to a later 5.x.z version on a live cluster with zero downtime or disruption of service.
When Kubernetes performs a rolling update to an individual microservice, there will be a mix of old and new services in the cluster concurrently (only briefly in most cases) and requests from other services will be routed to both versions. Consequently, Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same 5.x line of releases. We also ensure stored configuration remains compatible in the same 5.x release line.
Lucidworks releases minor updates to individual services frequently, so our customers can pull in those upgrades using Helm at their discretion.
To upgrade your cluster at any time, use the `--upgrade` option with our setup scripts in this repo.
The scripts in this repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.
To see what would be upgraded, you can pass the `--dry-run` option to the script.
### Grafana Dashboards
Get the initial Grafana password from a K8s secret by doing:
```bash wrap theme={"dark"}
kubectl get secret --namespace "${NAMESPACE}" ${RELEASE}-monitoring-grafana \
-o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```
With Grafana, you can either setup a temporary port-forward to a Grafana pod or expose Grafana on an external IP using a K8s LoadBalancer.
To define a LoadBalancer, do (replace \$ with your Helm release label):
```bash wrap theme={"dark"}
kubectl expose deployment ${RELEASE}-monitoring-grafana --type=LoadBalancer --name=grafana --port=3000 --target-port=3000
```
You can use `kubectl get services --namespace ` to determine when the load balancer is setup and its IP address. Direct your browser to `http://:3000` and enter the username `admin@localhost` and the password that was returned in the previous step.
This will log you into the application. It is recommended that you create another administrative user with a more desirable password.
The dashboards and datasoure will be setup for you in grafana, simply navigate to `Dashboards` -> `Manage` to view the available dashboards
## More deployment options
* How to deploy Fusion 5 in Google Kubernetes Engine
* How to deploy Fusion 5 in Amazon Elastic Kubernetes Service
* How to deploy Fusion on 5 other Kubernetes platforms
## Frequently Asked Questions
**Can the stateful database services, for example, MySQL, be supported by an Azure PaaS service?**
This option is not supported. In theory, it may be possible to implement this function.
**Is it possible to use cross-zone storage solutions rather than volumes, such as Microsoft Azure file storage, for stateful services?**
While in theory it may be possible to implement this functionality, the configuration has not been tested by Lucidworks and is not supported.
## Learn more
The course for **Deploying Fusion 5** focuses on the prerequisite software needed to deploy Fusion, the necessary setup steps, and the physical act of deployment.
The [`setup_f5_k8s.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_k8s.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository provides deployment support for any Kubernetes platform, including on-premise, private cloud, public cloud, and hybrid platforms.
This script is used by the [`setup_f5_gke.sh`](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_gke.sh), [`setup_f5_eks.sh`](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_eks.sh), and [`setup_f5_aks.sh`](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_aks.sh) scripts, which provide additional platform-specific support for Google Kubernetes Engine (GKE), Amazon Elastic Kubernetes Service (EKS), and Azure Kubernetes Service (AKS).
This topic explains how to deploy a Fusion cluster in Kubernetes using the [`setup_f5_k8s.sh` script](https://github.com/lucidworks/fusion-cloud-native/blob/master/setup_f5_k8s.sh) in the [`fusion-cloud-native`](https://github.com/lucidworks/fusion-cloud-native) repository.
If you’re deploying on-premises or using a localized repository, you’ll need to use a private repository for Docker images.
## Prerequisites
This section covers prerequisites and background knowledge needed to help you understand the structure of this document and how the Fusion installation process works with Kubernetes.
### Release Name and Namespace
Before installing Fusion, you need to choose a [https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/) Kubernetes namespace to install Fusion into.
Think of a K8s namespace as a virtual cluster within a physical cluster. You can install multiple instances of Fusion in the same cluster *in separate namespaces*.
However, please *do not* install more than one Fusion release in the same namespace.
All Fusion services must run in the same namespace, i.e. you should not try to split a Fusion cluster across multiple namespaces.\_\_
Use a short name for the namespace, containing only letters, digits, or dashes (no dots or underscores). The setup scripts in this repo use the namespace for the Helm release name by default.
### Install Helm
Helm is a package manager for Kubernetes that helps you install and manage applications on your Kubernetes cluster.
Regardless of which Kubernetes platform you're using, you need to install *`helm`* as it is required to install Fusion for any K8s platform.
On MacOS, you can do:
```bash theme={"dark"}
brew install kubernetes-helm
```
If you already have helm installed, make sure you're using the latest version:
```bash theme={"dark"}
brew upgrade kubernetes-helm
```
For other OS, please refer to the Helm installation docs: [https://helm.sh/docs/using\_helm/](https://helm.sh/docs/using_helm/)
The Fusion helm chart requires that helm is greater than version `3.0.0`; check your Helm version by running `helm version --short`.
#### Helm User Permissions
If you require that fusion is installed by a user with minimal permissions, instead of an admin user, then the role and cluster role that will have to be assigned to the user within the namespace that you wish to install fusion in are documented in the `install-roles` directory.
When working with Kubernetes on the command-line, it's useful to create a shell alias for `kubectl`, e.g.:
```bash theme={"dark"}
alias k=kubectl
```
To use these role in a cluster, as an admin user first create the namespace that you wish to install fusion into:
```bash theme={"dark"}
k create namespace fusion-namespace
```
Apply the `role.yaml` and `cluster-role.yaml` files to that namespace
```bash theme={"dark"}
k apply -f cluster-role.yaml
k config set-context --current --namespace=$NAMESPACE
k apply -f role.yaml
```
Then bind the rolebinding and clusterolebinding to the install user:
```bash wrap theme={"dark"}
k create --namespace fusion-namespace rolebinding fusion-install-rolebinding --role fusion-installer --user
k create clusterrolebinding fusion-install-rolebinding --clusterrole fusion-installer --user
```
You will then be able to run the `helm install` command as the ``
### Clone fusion-cloud-native from GitHub
You should clone this repo from github as you'll need to run the scripts on your local workstation:
```bash theme={"dark"}
git clone https://github.com/lucidworks/fusion-cloud-native.git
```
You should get into the habit of pulling this repo for the latest changes before performing any maintenance operations on your Fusion cluster to ensure you have the latest updates to the scripts.
```bash theme={"dark"}
cd fusion-cloud-native
git pull
```
Cloning the github repo is preferred so that you can pull in updates to the scripts, but if you are not a git user, then you can download the project: [https://github.com/lucidworks/fusion-cloud-native/archive/master.zip](https://github.com/lucidworks/fusion-cloud-native/archive/master.zip).
Once downloaded, extract the zip and cd into the `fusion-cloud-native-master` directory.
## Deployment
If you're not running on a managed K8s platform like GKE, AKS, or EKS, you can use Helm to install the Fusion chart to an existing Kubernetes cluster.
Fusion version 5.5 now includes support for the Rancher Kubernetes Engine (RKE) platform. Before deploying Fusion to RKE, you must download and install the [latest RKE software](https://rancher.com/docs/rke/latest/en/). After configuring your cluster, you can proceed with the Helm v3 installation.
You must have a working cluster configured before performing the Helm v3 installation.
### Use Helm v3 to Install Fusion
You should upgrade to the latest version of Helm v3 for working with Fusion. If you need to keep Helm V2 for other clusters,
ensure Helm V3 is ahead of Helm V2 in your working shell's PATH before proceeding.
#### Customize Fusion Chart Settings
Fusion aims to be well-configured out-of-the-box, but you can customize any of the built-in settings using a custom values `YAML` file. If you use one of our setup scripts, such as `setup_f5_gke.sh`, then it will create a custom values YAML file for you the first time you run it using [https://github.com/lucidworks/fusion-cloud-native/blob/master/customize\_fusion\_values.yaml.example](https://github.com/lucidworks/fusion-cloud-native/blob/master/customize_fusion_values.yaml.example) as a template.
If you're working with Helm directly and not using one of our setup scripts, then run the [https://github.com/lucidworks/fusion-cloud-native/blob/master/customize\_fusion\_values.sh](https://github.com/lucidworks/fusion-cloud-native/blob/master/customize_fusion_values.sh) script to create a custom values YAML file from our [https://github.com/lucidworks/fusion-cloud-native/blob/master/customize\_fusion\_values.yaml.example](https://github.com/lucidworks/fusion-cloud-native/blob/master/customize_fusion_values.yaml.example) template as a starting point:
```bash theme={"dark"}
./customize_fusion_values.sh -c -n \
--provider --num-solr 1 --node-pool ""
```
Pass `--help` for usage details.
In this example:
* `` is the K8s platform you're running on, such as `gke`
* `` is the name of your cluster
* `` is the K8s namespace where you plan to install Fusion
The `--node-pool` option specifies the node selector label for determining which nodes to run Fusion pods. You can pass `"{}"` to let Kubernetes decide which nodes to schedule pods on.
This file is referred to as `${MY_VALUES}` in the commands belo. Replace the filename with the correct filename for your environment. Keep this file handy, as you'll need it to customize Fusion settings and upgrade to a newer version.
Review the settings in the custom values YAML file to ensure the defaults are appropriate for your environment, including the number of Solr and Zookeeper replicas.
Add the Lucidworks Helm repo:
```bash theme={"dark"}
helm repo add lucidworks https://charts.lucidworks.com
```
The `customize_fusion_values.sh` script creates an upgrade script to install/upgrade Fusion into Kubernetes using Helm. Look in the directory where you ran `customize_fusion_values.sh` for a script named like:
`___upgrade_fusion.sh`. Run this script to install Fusion.
## Verifying the Fusion Installation
In this section, we provide some tips on how to verify the Fusion installation.
Check if the Fusion Admin UI is available at `\https://:6764/admin/`.
Let's review some useful kubectl commands.
### Enhance the K8s Command-line Experience
Here is a list of tools we found useful for improving your command-line experience with Kubernetes:
* krew (kubectl plugin mgr): [https://github.com/kubernetes-sigs/krew/](https://github.com/kubernetes-sigs/krew/)
* kube-ps1 (show current context on command line prompt): [https://github.com/jonmosco/kube-ps1](https://github.com/jonmosco/kube-ps1)
* kubectx / kubens (switch between clusters / namespaces): [https://github.com/ahmetb/kubectx](https://github.com/ahmetb/kubectx)
### Useful kubectl commands
kubectl reference: [https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands)
Set the namespace for `kubectl` if not using the default:
```bash theme={"dark"}
kubectl config set-context --current --namespace=
```
**This saves you from having to pass `-n` with every command.**
Get a list of running pods: `k get pods`
Get logs for a pod using a label: `k logs –l app.kubernetes.io/component=query-pipeline`
Get pod deployment spec and details: `k get pods -o yaml`
Get details about a pod events: `k describe po `
Port forward to a specific pod: `k port-forward 8983:8983`
SSH into a pod: `k exec -it -- /bin/bash`
CPU/Memory usage report for pods: `k top pods`
Forcefully kill a pod: `k delete po --force --grace-period 0`
Scale up (or down) a deployment: `k scale deployment.v1.apps/ --replicas=N`
Get a list of pod versions: `k get po -o jsonpath='{..image}' | tr -s '[[:space:]]' '\n' | sort | uniq`
### Check Fusion Pods and Services
Once the install script completes, you can check that all pods and services are available using:
```bash theme={"dark"}
kubectl get pods
```
If all goes well, you should see a list of pods similar to:
```bash wrap theme={"dark"}
NAME READY STATUS RESTARTS AGE
seldon-controller-manager-6675874894-qxwrv 1/1 Running 0 8m45s
f5-admin-ui-74d794f4f8-m5jms 1/1 Running 0 8m45s
f5-ambassador-fd6b9b5dc-7ghf6 1/1 Running 0 8m43s
f5-api-gateway-6b9998b9c-tmchk 1/1 Running 0 8m45s
f5-auth-ui-7565564b4c-rdc74 1/1 Running 0 8m42s
f5-classic-rest-service-0 1/1 Running 3 8m44s
f5-devops-ui-77bb867ffb-fbzxd 1/1 Running 0 8m42s
f5-fusion-admin-78b8f8fc7f-4d7l8 1/1 Running 0 8m42s
f5-fusion-indexing-599c8d448-xzsvm 1/1 Running 0 8m44s
f5-insights-665fd9f6fc-g5psw 1/1 Running 0 8m43s
f5-job-launcher-84dd4c5c96-p8528 1/1 Running 0 8m44s
f5-job-rest-server-6d44d964b8-xtnxw 1/1 Running 0 8m45s
f5-logstash-0 1/1 Running 0 8m45s
f5-ml-model-service-6987dc94c9-9ppp8 2/2 Running 1 8m45s
f5-monitoring-grafana-5d499dbb58-pzw72 1/1 Running 0 10m
f5-monitoring-prometheus-kube-state-metrics-54d6678dv9h7h 1/1 Running 0 10m
f5-monitoring-prometheus-pushgateway-7d65c65b85-vwrwf 1/1 Running 0 10m
f5-monitoring-prometheus-server-0 2/2 Running 0 10m
f5-pm-ui-86cbc5bb65-nd2n8 1/1 Running 0 8m44s
f5-pulsar-bookkeeper-0 1/1 Running 0 8m45s
f5-pulsar-broker-b56cc776f-56msx 1/1 Running 0 8m45s
f5-query-pipeline-5d75d7d5f4-l2mdf 1/1 Running 0 8m43s
f5-connectors-7bb6cfc65f-7wfs2 1/1 Running 0 8m42s
f5-connectors-backend-987fdc648-dldwv 1/1 Running 0 8m45s
f5-rules-ui-6b9d55b78f-9hzzj 1/1 Running 0 8m43s
f5-solr-0 1/1 Running 0 8m44s
f5-solr-exporter-c4687c785-jsm7x 1/1 Running 0 8m45s
f5-ui-6cdbcc68c6-rj9cq 1/1 Running 0 8m45s
f5-webapps-6d6bb9bfd-hm4qx 1/1 Running 0 8m45s
f5-workflow-controller-7b66679fb7-sjbvp 1/1 Running 0 8m44s
f5-zookeeper-0 1/1 Running 0 8m45s
```
The number of pods per deployment / statefulset will vary based on your cluster size and replicaCount settings in your custom values YAML file.
Also, don't worry if you see some pods having been restarted as that just means they were too slow to come up and Kubernetes killed and restarted them.
You do want to see at least one pod running for every service. If a pod is not running after waiting a sufficient amount of time,
use `kubectl logs ` to see the logs for that pod; to see the logs for previous versions of a pod, use: `kubectl logs -p`.
You can also look at the actions Kubernetes performed on the pod using `kubectl describe po `.
To see a list of Fusion services, do:
```bash theme={"dark"}
kubectl get svc
```
For an overview of the various Fusion 5 microservices, see: [Fusion microservices](/docs/5/fusion/reference/microservices).
Once you're ready to build a Fusion cluster for production, please see see more information at [Fusion 5 Survival Guide](/docs/5/fusion/operations/survival-guide/overview).
### Upgrading with Zero Downtime
One of the most powerful features provided by Kubernetes and a cloud-native microservices architecture is the ability to do a rolling update on a live cluster. Fusion 5 allows customers to upgrade from Fusion 5.x.y to a later 5.x.z version on a live cluster with zero downtime or disruption of service.
When Kubernetes performs a rolling update to an individual microservice, there will be a mix of old and new services in the cluster concurrently (only briefly in most cases) and requests from other services will be routed to both versions. Consequently, Lucidworks ensures all changes we make to our service do not break the API interface exposed to other services in the same 5.x line of releases. We also ensure stored configuration remains compatible in the same 5.x release line.
Lucidworks releases minor updates to individual services frequently, so our customers can pull in those upgrades using Helm at their discretion.
To upgrade your cluster at any time, use the `--upgrade` option with our setup scripts in this repo.
The scripts in this repo automatically pull in the latest chart updates from our Helm repository and deploy any updates needed by doing a diff of your current installation and the latest release from Lucidworks.
To see what would be upgraded, you can pass the `--dry-run` option to the script.
### Grafana Dashboards
Get the initial Grafana password from a K8s secret by doing:
```bash wrap theme={"dark"}
kubectl get secret --namespace "${NAMESPACE}" ${RELEASE}-monitoring-grafana \
-o jsonpath="{.data.admin-password}" | base64 --decode ; echo
```
With Grafana, you can either setup a temporary port-forward to a Grafana pod or expose Grafana on an external IP using a K8s LoadBalancer.
To define a LoadBalancer, do (replace \$ with your Helm release label):
```bash wrap theme={"dark"}
kubectl expose deployment ${RELEASE}-monitoring-grafana --type=LoadBalancer --name=grafana --port=3000 --target-port=3000
```
You can use `kubectl get services --namespace ` to determine when the load balancer is setup and its IP address. Direct your browser to `http://:3000` and enter the username `admin@localhost` and the password that was returned in the previous step.
This will log you into the application. It is recommended that you create another administrative user with a more desirable password.
The dashboards and datasoure will be setup for you in grafana, simply navigate to `Dashboards` -> `Manage` to view the available dashboards
## Learn more
The course for **Deploying Fusion 5** focuses on the prerequisite software needed to deploy Fusion, the necessary setup steps, and the physical act of deployment.
# Spark Administration in Kubernetes
Source: https://doc.lucidworks.com/docs/5/fusion/operations/survival-guide/spark-kubernetes-overview
[localhost link]: http://localhost:3000/docs/5/fusion/operations/survival-guide/spark-kubernetes-overview
[mintlify link]: https://doc.lucidworks.com/docs/5/fusion/operations/survival-guide/spark-kubernetes-overview
[old doc.lw link]: https://doc.lucidworks.com/fusion/5.9/190
In Fusion 5.x, Spark operates in native [Kubernetes mode](https://spark.apache.org/docs/latest/running-on-kubernetes.html), rather than the standalone mode used in Fusion 4.x. This topic describes Spark operations in Fusion 5.x.
## Node Selectors
You can control which nodes Spark executors are scheduled on using a Spark configuration property for a job:
```bash theme={"dark"}
spark.kubernetes.node.selector.