Set up Dynamic Pricing

Table of Contents

Format raw data
- Text file format
- Incremental updates

Dynamic Pricing lets you sort, facet, boost, and query multiple pricing lists and product inventories efficiently.

Extensive product catalogs and pricing lists result in large datasets that can overwhelm Solr. Dynamic Pricing solves this big data issue by storing versioned price lists outside of Solr. These files are then processed into a binary format easily ingested by Solr.

Contact your Lucidworks representative to enable this feature.

Lucidworks offers free training to help you get started with Fusion. Check out the Dynamic Pricing course, which focuses on how Dynamic Pricing maximizes custom pricing strategies:

Visit the LucidAcademy to see the full training catalog.

Format raw data

To use Dynamic Pricing, you need to prepare and format your files. Store the raw external files in the following directory structure:

EXTERNAL_FILES_ROOT/COLLECTION/N_TOP_LEVEL_SUBDIRS/BASENAME/SEQUENCE_ID/FILE

Format Description Example

Format	Description	Example
EXTERNAL_FILES_ROOT	The root directory for the external files.	`root`
COLLECTION	The Solr collection the external files use.	`bucket1`
N_TOP_LEVEL_SUBDIRS	Subdirectories to hold files. How many subdirectories, what they are named and which files they hold does not affect the output of the file processing, but it does impact replication performance. Replication of raw files is applied at the top level subdirectories. Lucidworks recommends that you have at least eight top level directories to maximize the benefits of parallelization.	`SubDir`
BASENAME	A directory for each file. The directory name must be the same as the filename without the extension (basename). For example if the file is named `example.txt` the directory must be named `example`.	`example`
SEQUENCE_ID	The sequenceId directories hold different versions of the external files. SequenceIds must be positive whole numbers, parsable to a Java Long class. New versions of the file must be placed in a higher sequenceId directory.	`1661540544007`
FILE	Each sequencedD directory holds one external file. Files that are being written must have a temporary extension other than `.txt` and be renamed to a `.txt` extension after being written. If Solr dynamic fields are used, the file name must end with a "_" postfix that maps directly to a Solr dynamic field.	`example.txt`

EXTERNAL_FILES_ROOT

The root directory for the external files.

root

COLLECTION

The Solr collection the external files use.

bucket1

N_TOP_LEVEL_SUBDIRS

Subdirectories to hold files. How many subdirectories, what they are named and which files they hold does not affect the output of the file processing, but it does impact replication performance. Replication of raw files is applied at the top level subdirectories. Lucidworks recommends that you have at least eight top level directories to maximize the benefits of parallelization.

SubDir

BASENAME

A directory for each file. The directory name must be the same as the filename without the extension (basename). For example if the file is named example.txt the directory must be named example.

example

SEQUENCE_ID

The sequenceId directories hold different versions of the external files. SequenceIds must be positive whole numbers, parsable to a Java Long class. New versions of the file must be placed in a higher sequenceId directory.

1661540544007

FILE

Each sequencedD directory holds one external file. Files that are being written must have a temporary extension other than .txt and be renamed to a .txt extension after being written. If Solr dynamic fields are used, the file name must end with a "_" postfix that maps directly to a Solr dynamic field.

example.txt

Sample full file path: $root/bucket1/example/1661540544007/example_ef.txt

Text file format

The format of the text file is a whitespace-separated list of fields that appears as:

id1 float1
id2 float2

id1 routeKey1 float1
id2 routeKey1 float2

The id must map to a unique documentId in a Solr collection. The float value must parse to a Java float. The routeKey must be provided if documents are routed to shards with a specific route key. Each file is split into separate partitions for each shard. Do not include whitespaces in ids and route keys because this causes file format errors.

Optional blank lines and comments starting with # are allowed.

Incremental updates

Dynamic Pricing supports incremental updates of external files. Store the incremental update files in the root of the BASENAME directory:

EXTERNAL_FILES_ROOT/COLLECTION/N_TOP_LEVEL_SUBDIRS/BASENAME

Structure the file name of the incremental files as <BASENAME>-<SEQUENCE_ID>.inc.

The extension of the incremental files is .inc. When incremental files are being written, use a temporary extension other than .inc and rename them to a .inc extension after being written.

Sample path of an incremental file: $root/bucket1/foo_ef/foo_ef-1666023034.inc

Incremental updates apply to the full external file in the latest sequenceId directory. The existing external file does not change. Instead, Dynamic Pricing writes a new full external file to a new sequenceId directory with the merged incremental updates. The new sequenceId directory is taken from the highest sequenceId of the incremental files. Once the new external file is written, the incremental files are deleted.

The format of the records in the incremental files must be the same as the records in the external files.