Cross Data Center Replication

​

CrossDC and ConfigSync

• Solr collections, which can optionally include creating and deleting collections
• Rules stored in *_query_rewriter and *_query_rewrite_staging collections
• Any other new Solr collection data that you configure to be synchronized, as explained below

​

Before you begin

​

Configure CrossDC

1. In the source data center, configure Solr and Kafka.
   Configure the source Solr instance

   To enable CrossDC for Solr in the source data center, you need to configure the following components:

   Solr class	Where configured	Role in CrossDC
   fusion-crossdc-producer	solr.xml	This module contains the necessary classes. It is included in the fusion-solr-managed Docker image.
   MirroringUpdateRequestProcessorFactory	solrconfig.xml	This processor mirrors Solr indexing updates (such as document additions, updates, or deletions) to the source Kafka instance.
   MirroringConfigSetsHandler	solr.xml	This handler mirrors configset changes to the source Kafka instance.
   MirroringCollectionsHandler	solr.xml	This handler mirrors Solr collection admin commands (such as collection creation or deletion) to the source Kafka instance.
   FusionCollectionsHandler	solr.xml	This extended version of the MirroringCollectionsHandler mirrors Solr collection admin commands to the source Kafka instance and adds the ability to filter (whitelist) the commands you want to mirror.

   The steps and examples below show you how to configure your source Solr instance.

   1. Pull and deploy the fusion-solr-managed Docker image.
   2. In solrconfig.xml for the configset used by each collection, configure the MirroringUpdateRequestProcessorFactory handler:
      Copy

      <updateRequestProcessorChain  name="mirrorUpdateChain" default="true">
      
          <processor class="org.apache.solr.update.processor.MirroringUpdateRequestProcessorFactory">
          <str name="bootstrapServers">${bootstrapServers:}</str>
          <str name="topicName">${topicName:}</str>
          </processor>
      
          <processor class="solr.LogUpdateProcessorFactory" />
          <processor class="solr.RunUpdateProcessorFactory" />
      </updateRequestProcessorChain>
      

      Only collections with MirroringUpdateRequestProcessorFactory configured in the updateRequestProcessorChain are mirrored; other collections are ignored.
      Both bootstrapServers and topicName are required:

      Parameter	Type	Description
      bootstrapServers	string	A comma-separated list of servers used to connect to the source Kafka cluster
      topicName	string	The name of the Kafka topic to which Solr updates will be pushed. This topic must already exist.

      These parameters are optional for the MirroringUpdateRequestProcessorFactory:

      Parameter	Type	Description
      batchSizeBytes	integer	Maximum batch size in bytes for the Kafka queue.
      bufferMemoryBytes	integer	Memory allocated by the Producer in total for buffering.
      lingerMs	integer	Amount of time that the Producer will wait to add to a batch.
      requestTimeout	integer	Request timeout for the Producer.
      enableDataCompression	boolean	Whether to use compression for data sent over the Kafka queue, one of the following: • none (default) • gzip • snappy • lz4 • zstd
      numRetries	integer	Setting a value greater than zero will cause the Producer to resend any record whose send fails with a potentially transient error.
      retryBackoffMs	integer	The amount of time to wait before attempting to retry a failed request to a given topic partition.
      deliveryTimeoutMS	integer	Updates sent to the Kafka queue will be failed before the number of retries has been exhausted if the timeout configured by delivery.timeout.ms expires first.
      maxRequestSizeBytes	integer	The maximum size of a Kafka queue request in bytes – limits the number of requests that will be sent over the queue in a single batch.
      dlqTopicName	string	If not empty, then requests that failed processing maxAttempts times will be sent to a “dead letter queue” topic in Kafka (must exist if configured).
      indexUnmirrorableDocs	boolean	If set to true, updates that are too large for the Kafka queue will still be indexed locally into the source collection.
      mirrorCommits	boolean	If true, then standalone commit requests will be mirrored as separate requests; otherwise they will be processed only locally.
      expandDbq	enum	If set to expand (default), then Delete-By-Query is expanded before mirroring into a series of Delete-By-Id, which may help with correct processing of out-of-order requests on the consumer side. If set to none, then Delete-By-Query requests are mirrored as-is.

   3. If you are not using ConfigSync: Configure configset mirroring in solr.xml:
      Copy

      <solr>
      <str name="configSetsHandler">org.apache.solr.handler.admin.MirroringConfigSetsHandler</str>
      ...
      </solr>
      

   4. If you are not using ConfigSync: Configure collection admin request mirroring in solr.xml. You can choose one of these handlers to use for this:
      • MirroringCollectionsHandler is the native Solr handler. It mirrors all admin actions for all collections, or you can select specific collections to mirror.
      • FusionCollectionsHandler has all the same capabilities and configuration options, plus action whitelisting so you can mirror only selected actions and ignore others.
      Copy

      <solr>
      <str name="collectionsHandler">org.apache.solr.handler.admin.FusionCollectionsHandler</str>
      ...
      </solr>
      

      By default, admin commands are mirrored for all collections. To mirror admin commands for specific collections only, you can set this system property:
      ​

      mirror.collections

      string

      A comma-separated list of collections for which the admin commands will be mirrored. If this list is empty or the property is not set, then admin commands for all collections mirrored. This property is supported by both MirroringCollectionsHandler and FusionCollectionsHandler.
      If you are using FusionCollectionsHandler, you can also configure action whitelisting by configuring the following system property:
      ​

      collectionActionsWhitelist

      string

      A comma-separated list of actions to mirror. If it is not empty, then only the listed actions are mirrored; all others are ignored. See the Solr documentation for the list of actions.
   Configure the source Kafka

   In the source data center’s Kafka instance, the Kafka topic must already exist and be configured to accept messages from the Solr instance. Make sure that the bootstrapServers you configured in Solr are reachable by Solr and that the configured topicName exists.If you configured Solr to use a Dead-Letter Queue (DLQ) topic (dlqTopicName), you must also create that topic in the source Kafka instance.See the Kafka documentation for configuration details.
2. In the target data center, configure Solr, Kafka, and the Consumer.
   Configure the target Solr

   Because CrossDC only mirrors new commands, the existing collections, documents, and configsets from your source Solr must already exist on the target Solr before mirroring begins. Create them if needed. The target collections must have the same names as the source collections, and they must use the same configsets as the source collections.

   New collections created on the source Solr are not automatically created on the target Solr unless ConfigSync is enabled or you have enabled either MirroringCollectionsHandler or FusionCollectionsHandler on the source Solr.
   Configure the target Kafka

   In the target data center’s Kafka instance, you must create the same topic that you configured in MirrorMaker and the Consumer. If you configured a Dead-Letter Queue (DLQ) topic in the source Solr instance, you must also create that topic in the target Kafka instance.See the Kafka documentation for configuration details.
   Configure the Consumer

   1. Pull and deploy the fusion-crossdc-consumer Docker image for your Fusion release, such as 5.9.14.
   2. Configure the required system properties listed below, and any optional ones that apply to your use case.

      Parameter	Required?	Description
      bootstrapServers	required	A list of Kafka bootstrap servers.
      topicName	required	Kafka topicName used to indicate which Kafka topic the Solr updates will be read from. This can be a comma-separated list to consume multiple topics.
      zkConnectString	required	The ZooKeeper connection string used for connecting to the target Solr instance.
      consumerProcessingThreads	optional	The number of threads used by the consumer to concurrently process updates from the Kafka queue.
      port	optional	The local port for the API endpoints. Default is 8090.
      collapseUpdates	optional (enum)	When set to all, all incoming update requests will be collapsed into a single UpdateRequest, as long as their parameters are identical. When set to partial (default), only requests without deletions are collapsed; requests with any delete ops are sent individually in order to preserve ordering of updates. When set to none, the incoming update requests are sent individually without any collapsing. Requests of other types than UPDATE are never collapsed.

      These additional optional configuration properties are used when the Consumer must retry by putting updates back in the Kafka queue:

      Parameter	Description
      batchSizeBytes	The maximum batch size in bytes for the Kafka queue.
      bufferMemoryBytes	The memory allocated by the Producer in total for buffering.
      lingerMs	The amount of time that the Producer will wait to add to a batch.
      requestTimeout	The request timeout for the Producer.
      maxPollIntervalMs	The maximum delay between invocations of poll() when using Consumer group management.

3. Configure Kafka MirrorMaker to connect to Kafka in both data centers.
   Configure MirrorMaker

   You can deploy MirrorMaker in either of your data centers, or somewhere else. Ensure that it can access both the source Kafka and target Kafka instances. Configure the source and target topic names to correspond with the names configured in MirroringUpdateRequestProcessorFactory and the Consumer application.See the MirrorMaker documentation for configuration details.

​

Metrics and monitoring

​

Producer metrics

​

Consumer metrics

​

Counters

​

Timers

​

Limitations

• Only updates are mirrored, not existing indexes. You should create a copy of each existing collection, with its documents and configset, on the target Solr before you turn on CrossDC.
• Data loss can lead to divergence. If any of the components in your CrossDC configuration experience an event that causes data loss, the source collections and target collections can potentially diverge. Diverged indexes are not automatically detected or re-synchronized.
• Document size is limited. The CrossDC Producer module tries to estimate the size of each message and avoid sending messages that are too large to Kafka. It does not split messages that are too large; instead, it rejects them. If this happens after the update has already been processed locally, then the contents of the mirrored collections can diverge. Kafka’s maximum message size is 1MB by default, configured with message.max.bytes.
• Retries can lead to divergence The CrossDC Producer module first applies updates locally, and attempts mirroring only if they succeed. If sending a mirrored request fails, the request is retried, and if it’s still failing then it’s logged and the message is discarded (or sent to a dead-letter queue). Since the update was already applied locally, this can cause divergence of the local and mirrored collections.
• Commands can be re-ordered when collapsed. The CrossDC Producer module can optionally preserve exact ordering of updates and deletes sent in a single request, but this negatively affects performance. If you do not need strict ordering of multiple commands in a single request, then you should use collapseUpdates=partial or collapseUpdates=all.
• Delete-by-Query expansion can lead to divergence. The CrossDC Producer can either mirror Delete-By-Query requests as-is or expand them into individual Delete-By-Id requests (except for *:* which is always sent as-is). In extreme cases this expansion may produce a request that is too large to be mirrored. Delete-By-Query expansion helps to ensure the strict ordering of deletes and updates in the target Solr collection but it may also lead to divergence of the local and mirrored collections if the expansion fails or the resulting request is too large to mirror.
• Collection creation and deletion requires an existing configset. The CrossDC Producer module can optionally mirror collection creation and deletion requests. However, the target Solr instance must already have the corresponding configset available in ZooKeeper. If it doesn’t, this causes an error when the target collection is created or deleted. The Consumer application may also experience significant slow-downs when it receives update requests to non-existent target collections. These slow-downs affect processing requests for other collections, too.
• ConfigSet creation and deletion behavior depends on the handler. If you are using MirroringConfigSetsHandler, then new configsets created on the source Solr are mirrored automatically to the target Solr. If you are not using MirroringConfigSetsHandler, then new configsets are not mirrored; you must use ConfigSync or create them manually on the target Solr to avoid an error when the target collection is created or deleted. These errors also impact the performance of the Consumer application.
• In some cases, admin request whitelisting is needed. If you are using MirroringCollectionsHandler, then all collection admin requests are mirrored. This may not always be desirable if the target Solr cluster is expected to differ or is managed externally (such as by an autoscaling operator). In this case, you should use FusionCollectionsHandler instead, and configure the collectionActionsWhitelist property to restrict the mirrored collection admin requests to only those that are needed.