> ## Documentation Index > Fetch the complete documentation index at: https://doc.lucidworks.com/llms.txt > Use this file to discover all available pages before exploring further. # Chunk text using the specified chunker > This is an all-purpose chunking endpoint you can use by specifying the `chunker` and `modelId` to use. The `POST` request submits text to be split into chunks based on the `CHUNKER` (chunking strategy) and `MODEL_ID` sent in the request. The API responds with the following information: * `chunkingId` that is a unique UUID for the submitted chunking task, and can be used later to retrieve the results. * `status` that indicates the current state of the chunking task. IMPORTANT: The available use cases by `chunker` are detailed in their own section in this specification. ## OpenAPI ````yaml /api-reference/saas/machine-learning-platform-async-chunking.json post /ai/async-chunking/{CHUNKER}/{MODEL_ID} openapi: 3.0.1 info: title: Lucidworks AI Async Chunking API version: v0 description: >- The Lucidworks AI Async Chunking API asynchronously separates large pieces of text into smaller pieces, called chunks. The API then returns the chunks and their associated vectors. Currently, the maximum text size allowed for input is approximately 1 MB. Breaking text into chunks can produce a significant number of chunks, especially if there is overlap between chunks or small chunk sizes, so there are limits on how many chunks and vectors can be generated. These limits are based on factors such as the dimension size of the embedding model and whether vector quantization is used. To process large chunks of text efficiently, Lucidworks recommends you enter the appropriate value in the `"modelConfig": "vectorQuantizationMethod"` field to ensure that as much of the text as possible is chunked, even for large inputs. Quantized vectors are less resource intensive to store and compute, which decreases index and query processing time. In addition, due to their size, more quantized vectors can be used to reach the same amount of memory as typical vectors. For example, if the quantized vector sizes are [1,0,2], [2,3,1], [6,0,0], [0,0,2] and typical vectors are [0.012341,0.23434,0.01334], [0.5434,0.02134,0.05434], [0.76534,0.0953,0.1334], [0.398,0.38574,0.01384], and the amount of memory is 5MB: The quantized vector can have 5000 vectors that reach 5MB because they are smaller. But typical vectors can only have 500 because they have more numerical values to save in memory. In this example, the numbers 5MB, 5000, and 500 are random numbers. The API contains two requests: * POST Request: This request is used to submit text for a chunking strategy and model. Upon submission, the API responds with the following information: * `chunkingId` that is a unique UUID for the submitted chunking task, and can be used later to retrieve the results. * `status` that indicates the current state of the chunking task. * GET Request: This request is used to retrieve the results of a previously-submitted chunking request. You must provide the unique `chunkingId` received from the POST request. The API then returns the results of the chunking request associated with that `chunkingId`. The `async-chunking` endpoints require an authentication token with scope `machinelearning.predict`. contact: name: Lucidworks url: https://lucidworks.com/ email: support@lucidworks.com termsOfService: https://lucidworks.com/legal/developer-license-agreement/ license: name: Lucidworks url: https://lucidworks.com/legal/developer-license-agreement/ servers: - url: https://APPLICATION_ID.applications.lucidworks.com description: Production security: [] tags: - name: Split content into chunks description: Split documents into chunks. - name: Request chunked content description: Get the results of a chunking operation. paths: /ai/async-chunking/{CHUNKER}/{MODEL_ID}: parameters: - schema: type: string example: gte-small name: MODEL_ID in: path required: true description: Unique identifier for the model. - schema: type: string enum: - dynamic-newline - dynamic-sentence - regex-splitter - semantic - sentence name: CHUNKER in: path required: true description: Chunking strategy for the way text is split. example: semantic post: tags: - Split content into chunks summary: Chunk text using the specified chunker description: >- This is an all-purpose chunking endpoint you can use by specifying the `chunker` and `modelId` to use. The `POST` request submits text to be split into chunks based on the `CHUNKER` (chunking strategy) and `MODEL_ID` sent in the request. The API responds with the following information: * `chunkingId` that is a unique UUID for the submitted chunking task, and can be used later to retrieve the results. * `status` that indicates the current state of the chunking task. IMPORTANT: The available use cases by `chunker` are detailed in their own section in this specification. operationId: post-ai-async-chunking-chunker-model-id parameters: - in: header name: Authorization schema: type: string required: true description: >- Bearer token used for authentication. Format: `Authorization: Bearer ACCESS_TOKEN`. example: Bearer abc123def456 - in: header name: Content-Type schema: type: string example: application/json description: The content type of the request body. requestBody: content: application/json: schema: $ref: '#/components/schemas/BasicGenericRequest' example: batch: - text: |- The itsy bitsy spider climbed up the waterspout. Down came the rain. And washed the spider out. Out came the sun. And dried up all the rain. And the itsy bitsy spider climbed up the spout again. useCaseConfig: dataType: passage modelConfig: vectorQuantizationMethod: max-scale responses: '200': description: OK content: application/json: schema: $ref: '#/components/schemas/POSTresponse' example: chunkingId: 09c5f8f1-e790-4a06-9e4e-60f577792043 status: SUBMITTED 4XX: $ref: '#/components/responses/Error' components: schemas: BasicGenericRequest: title: BasicGenericRequest type: object x-examples: {} properties: batch: type: array description: The batch of key:value pairs used in the chunking request. items: type: object properties: text: type: string description: >- The content to be split into chunks based on the chunking strategy (`chunker`) sent in the request. The maximum text size allowed for input is approximately 1 MB. example: 'The content to be split into chunks. ' modelConfig: $ref: '#/components/schemas/ModelConfig' useCaseConfig: $ref: '#/components/schemas/UseCaseConfigChunking' chunkerConfig: description: >- This parameter contains fields that determine how the text is split and the size of text chunks created. IMPORTANT: Specific information is detailed in each use case. type: object POSTresponse: title: POST response type: object description: >- This is the response to the POST chunking request submitted for a specific `chunker` and `modelId`. properties: chunkingId: type: string format: uuid description: >- The universal unique identifier (UUID) returned in the POST request. This UUID is required in the GET request to retrieve results. example: 441eb3be-7de6-470a-8141-e416a15c7db1 status: type: string example: SUBMITTED description: >- The current status of the request. Allowed values are: * SUBMITTED - The POST request was successful and the response has returned the `chunkingId` and `status` that is used by the GET request. * ERROR - An error was generated when the GET request was sent. * READY - The results associated with the `chunkingId` are available and ready to be retrieved. * RETRIEVED - The results associated with the `chunkingId` are returned successfully when the GET request was sent. ModelConfig: title: ModelConfig type: object description: Provides fields and values that specify ranges for tokens. properties: vectorQuantizationMethod: type: string description: >- Vector quantization compresses data size, as well as reducing memory usage. The methods are: * `min-max` - Creates tensors of the text and converts it to uint8 by normalizing it to the range [0, 255]. * `max-scale` - Finds the maximum absolute value for the encoded text, normalizes it by scaling the text to a range of -127 to 127, and then returns the quantized text as an 8-bit integer tensor. example: min-max dimReductionSize: type: integer description: >- Used to reduce vector size while maintaining good quality. This field allows any integer above 0, but less than or equal to the vector dimension of the model. If you send a vector dimension larger than the model, a 400 Bad Request error is returned. Not every model is designed to support this parameter. In this scenario, a warning message is generated that indicates quality can decrease. minimum: 1 example: 256 UseCaseConfigChunking: title: UseCaseConfigChunking type: object properties: dataType: type: string description: > This optional parameter enables model-specific handling in the Async Chunking API to help improve model accuracy. Use the most applicable fields based on available dataTypes and the dataType value that best aligns with the text sent to the Async Chunking API. The string values to use are: "dataType": "query" for the query. "dataType": "passage" for fields searched at query time. example: query chunkingId: title: chunkingId type: string description: >- The universal unique identifier (UUID) returned in the POST request. This UUID is required in the GET request to retrieve results. format: uuid example: 441eb3be-7de6-470a-8141-e416a15c7db1 status: title: status type: string description: >- The current status of the request. Allowed values are: * SUBMITTED - The POST request was successful and the response has returned the `chunkingId` and `status` that is used by the GET request. * READY - The results associated with the `chunkingId` are available and ready to be retrieved. * ERROR - An error was generated when the GET request was sent. * RETRIEVED - The results associated with the `chunkingId` are returned successfully when the GET request was sent. example: READY responses: Error: description: >- The error varies based on the issue encountered regarding the `chunkingId` or related information. content: application/json: schema: type: object properties: chunkingId: $ref: '#/components/schemas/chunkingId' status: $ref: '#/components/schemas/status' message: type: string description: >- The error generated if the `chunkingId` cannot be located or the related information cannot be retrieved. For example, "System prompt exceeded the maximum number of allowed input tokens." ````