Skip to main content
When you have rapidly-changing or personalized data you need to deliver instantly, Dynamic Index is leaner, faster, and more scalable than an indexed collection. When data in Dynamic Index is updated, it’s reflected in all applicable SKUs within seconds instead of minutes or hours—with no indexing. Because your index is not affected, response times are fast, even with millions of dynamic permutations of attributes and entitlements. Consider these examples:

B2C

Imagine your business has 1 million products distributed through 5,000 stores, and each store’s pricing varies. That’s 5 billion pricing permutations you need to access on demand.By storing those permutations in Dynamic Index, you keep your main index lean and fast, so customers get accurate pricing instantly on every product in every store.

B2B

Imagine your company sells industrial components to 50,000 accounts, each with custom pricing and contract terms. That’s 50,000 distinct permutations of product data.By storing those variations in Dynamic Index, you keep your product index streamlined while ensuring that every customer sees accurate pricing based on their unique account.
Generally, you’ll use Dynamic Index in addition to an indexed collection, as in this example:
Dynamic Index provides the latest dynamic data about an item

Dynamic index with location-based pricing

In the example above:
  • Data that changes rapidly comes from Dynamic Index, including the number of items in stock and the latest prices for members and non-members at each store location.
  • Data that seldom changes comes from an indexed collection, including the product name, photo, description, and so on.
  • Both types of data are united to compose an individual product listing that includes up-to-the-moment pricing and inventory data, without impacting response times.

​
Benefits

For end users

  • Users see up-to-the-moment pricing and inventory every time.
  • Bespoke experiences that reflect each user’s specific entitlements, location, customer segment, and more.
  • Even complex permutations are delivered with no impact on query response times.

For your team

  • Bespoke experiences drive higher conversions.
  • More control over data visibility.
  • Faster performance due to less index bloat.
  • Dynamic data is easier and less expensive to scale.

​
How it works

Dynamic Index supports millions of dynamic permutations of attributes and entitlements, all scaled to suit your evolving data and traffic. With Dynamic Index enabled, the query flow works like this:
1

A user submits a query

  • Fusion looks up the user’s profile, contract, store, segment, or other context.
  • Dynamic Index identifies the correct prices, inventory, or entitlements for the user’s attributes.
2

Lucidworks composes the response

  • Fusion fetches relevant search results from an indexed collection.
  • Dynamic Index attaches the correct prices, inventory, or entitlements to the results.
3

Accurate, personalized results

  • The user gets accurate pricing, inventory, and more—every time.
  • Updated data is delivered instantly on subsequent searches.

​
When to use Dynamic Index

Dynamic Index works well for rapidly updating these types of data:
  • Different prices for different accounts, locations, or contract terms
  • Different inventory for each store, warehouse, or distribution center
  • Different entitlements for users, groups, or segments
  • Frequent updates across thousands or millions of SKUs

Dynamic Index is not recommended for these scenarios:
  • Multi-list joins at query time
  • Multi-currency in a single price list
  • Self-hosted environments
  • Non-GCS implementations
  • Infrequently-updated data

​
Requirements

  • Managed Fusion
  • Google Cloud Storage (GCS)
  • A prebuilt JAR, provided by Lucidworks, that includes the required utility that syncs your local external files to cloud storage

​
Enabling Dynamic Index

To enable Dynamic Index, you need to prepare your data, create the directory structure, then upload the data to GCS. The Lucidworks team then initiates synchronization and your Dynamic Index goes live.
1

Prepare your data

Your data must be formatted as .txt files using one of these formats:
  • With ID and float values (one per line): 
    id1 float1
id2 float2
  • For sharded collections, use ID, route key, and float: 
    id1 routeKey1 float1
id2 routeKey1 float2
  • Each id value must match a document in your indexed collection.
  • Each float must parse as a valid Java float.
  • IDs and route keys must not contain whitespace.
  • Blank lines and # comments are allowed.
2

Create the directory structure

Your raw files must be contained in a directory structure like this:
EXTERNAL_FILES_ROOT /
  COLLECTION /
    BUCKET /            ← For best performance, use 8 or more of these top-level subdirectories
      BASENAME /        ← Matches the filename, without extension
        SEQUENCE_ID /   ← A positive number representing the version of this file
          RAW_DATA.txt  ← A single file per sequence directory
An example path might look like this:
$root/commerce/user_profiles/customer_data/1661540544007/customer_data.txt
3

Upload to GCS

Use the provided RawPush utility to upload your files like this:
java -cp dynamic-index.jar org.apache.solr.util.external.RawPush file:///SOURCE_PATH gs://DESTINATION_BUCKET
This tool automatically generates the required manifest files in each directory, enabling fast, reliable replication to Fusion.
4

Lucidworks enables Dynamic Index

When your data is ready, open a ticket to request Dynamic Index enablement.Your Lucidworks representative will work with you to determine the correct replica multiplier to achieve high availability and performance.

​
Updating your dynamic data

Dynamic Index supports two methods for updating your dynamic data. Updates take effect instantly so users see the latest values in real time.
You can use both methods on the same file as long as you use a different SEQUENCE_ID for each method. Updates are applied sequentially from lowest to highest SEQUENCE_ID.
Use this method when you’re doing a complete replacement of all of the values for a dynamic field.
1
Create the new version of the full file you want to replace.
2
Create a new SEQUENCE_ID folder with a higher ID than the previous one and place the new file in this folder.
3
Upload the new folder:
java -cp dynamic-index.jar org.apache.solr.util.external.RawPush file:///SOURCE_PATH gs://DESTINATION_BUCKET
Managed Fusion uses the latest available version based on the highest numeric SEQUENCE_ID.
You can use an incremental update to modify the data in an existing SEQUENCE_ID.
1
Compile your incremental updates into an include file (.inc), using the same format as your full files. You only need to include the lines you want to change.
2
Name your incremental files like this:
BASENAME-SEQUENCE_ID.inc
3
Place the .inc file inside the BASENAME directory, like this:
EXTERNAL_FILES_ROOT /
  COLLECTION /
    BUCKET / 
      BASENAME /                    ← Place `.inc` files here
        BASENAME-SEQUENCE_ID.inc ← This file will be merged
4
Upload the new folder:
java -cp dynamic-index.jar org.apache.solr.util.external.RawPush file:///SOURCE_PATH gs://DESTINATION_BUCKET
Managed Fusion automatically merges the .inc file with the latest full file version, writes it to a new SEQUENCE_ID directory, then deletes the .inc file.