Models
Authentication
Use Case
Predict
- POSTModel predictions by use case
- POSTEnglish language model text encoder
- POSTMultilingual language model text encoder
- POSTCustom model prediction
- POSTClassification use case
- POSTPassthrough use case
- POSTRAG use case
- POSTStandalone query rewriter
- POSTSummarization use case
- POSTKeyword extraction use case
- POSTNamed Entity Recognition (NER) use case
- POST
Async Chunking
Async Predict
Prompt Preview
Signals
Tokenization
RAG use case
The rag use case uses candidate documents that are inserted into a LLM’s context to ground the generated response to those documents instead of generating an answer from details stored in the LLM’s trained weights. This type of search adds guardrails so the LLM can search private data collections.
The RAG search can perform queries against external documents passed in as part of the request.
Headers
The authentication and authorization access token.
application/json
"application/json"
Path Parameters
Unique identifier for the model.
"6a092bd4-5098-466c-94aa-40bf6829430\""
Body
Show child attributes
Show child attributes
Content for the model to analyze. Multiple instances of text can be sent in the request.
"What is RAG?"
Show child attributes
Show child attributes
The contents of the document.
"Retrieval Augmented Generation, known as RAG, a framework promising to optimize generative AI."
The URL that identifies the source of the document.
"http://rag.com/22"
The title of the document.
"What are the benefits of RAG?"
The date and time the document was created, displayed in the required ISO-8601 format of yyyy-mm-ddThh:mm:ssZ.
"2022-01-31T19:31:34Z"
Show child attributes
Show child attributes
The universal unique identifier (UUID) stored in the trained set of data in the model that is used in the model request.
This parameter is optional, and is used when previous chat history reference information is available.
"27a887fe-3d7c-4ef0-9597-e2dfc054c20e"
This determines if relevant content can be extracted in the request. Set to true to help preserve key information in longer documents that might be lost during the default truncation, but at the cost of more LLM calls and a slower response time.
This parameter is optional, and can be passed to change the response if the LLM cannot answer the request. The default is "Not possible to answer given this content."
Provides fields and values that specify ranges for tokens. Fields used for specific use cases and models are specified. The default values are used if other values are not specified.
Show child attributes
Show child attributes
A sampling temperature between 0 and 2. A higher sampling temperature such as 0.8, results in more random (creative) output. A lower value such as 0.2 results in more focused (conservative) output. A lower value does not guarantee the model returns the same response for the same input. We recommend staying at or below a temperature of 1.0. Values above 1.0 might return nonsense unless the topP value is lowered to be more deterministic.
0 <= x <= 20.8
A floating-point number between 0 and 1 that controls the cumulative probability of the top tokens to consider, known as the randomness of the LLM's response. This parameter is also referred to as top probability. Set topP to 1 to consider all tokens. A higher value specifies a higher probability threshold and selects tokens whose cumulative probability is greater than the threshold. The higher the value, the more diverse the output.
0 <= x <= 11
An integer that controls the number of top tokens to consider. Set top_k to -1 to consider all tokens.
-1
A floating-point number between -2.0 and 2.0 that penalizes new tokens based on whether they have already appeared in the text. This increases the model's use of diverse tokens. A value greater than zero (0) encourages the model to use new tokens. A value less than zero (0) encourages the model to repeat existing tokens. This is applicable for all OpenAI, Mistral, and Llama models.
-2 <= x <= 22
A floating-point number between -2.0 and 2.0 that penalizes new tokens based on their frequency in the generated text. A value greater than zero (0) encourages the model to use new tokens. A value less than zero (0) encourages the model to repeat existing tokens. This is applicable for all OpenAI, Mistral, and Llama models.
-2 <= x <= 21
The maximum number of tokens to generate per output sequence. The value is different for each model. Review individual model specifications when the value exceeds 2048.
1
This optional parameter is only required when using the model for prediction. You can find this value in your model's settings:
-
OpenAI: Copy and paste the API key found in your organization's settings. For more information, see OpenAI Authentication API keys.
-
Azure OpenAI: Copy and paste the API key found in your Azure portal. See Authenticate with API key.
-
Anthropic: Copy and paste the API key found in your Anthrophic console or by using the Anthropic API.
-
Google Vertex AI: Copy and paste the base64-encoded service account key JSON found in your Google Cloud console. This service account key must have the Vertex AI user role enabled. For more information, see generate service account key.
"API key specific to use case and model"
This optional parameter is the name of the deployed Azure OpenAI model and is only required when a deployed Azure OpenAI model is used for prediction.
"DEPLOYMENT_NAME"
This optional parameter is the URL endpoint of the deployed Azure OpenAI model and is only required when a deployed Azure OpenAI model is used for prediction.
"https://azure.endpoint.com"
This parameter is optional, and is only required when a Google Vertex AI model is used for prediction.
"[GOOGLE_PROJECT_ID]"
This parameter is optional, and is only required when a Google Vertex AI model is used for prediction. The possible region values are:
- us-central1
- us-west4
- northamerica-northeast1
- us-east4
- us-west1
- asia-northeast3
- asia-southeast1
- asia-northeast
"[GOOGLE_PROJECT_REGION_OF_MODEL_ACCESS]"
Response
OK
Show child attributes
Show child attributes
The unparsed response returned from the request.
"ANSWER: \\\"Retrieval Augmented Generation, known as RAG, a framework promising to optimize generative AI.\"\\nSOURCES: [\\\"http://example.com/112\\\"]"
Show child attributes
Show child attributes
The number of tokens generated to prompt the model to continue generating results.
148
The number of tokens used until the model completes.
27
The sum of the prompt and completion tokens used in the model.
175
The parsed response text from the document.
"Retrieval Augmented Generation, known as RAG, a framework promising to optimize generative AI."
The URL that identifies the source of the document returned in the response. Multiple results may be returned.
"http://example.com/112"
The universal unique identifier (UUID) stored in the trained set of data in the model that is used in the model request.
This parameter is optional, and is used when previous chat history reference information is available.
"27a887fe-3d7c-4ef0-9597-e2dfc054c20e"
This parameter is optional, and can be passed to change the response if the LLM cannot answer the request. The default is "Not possible to answer given this content."
This parameter is false if the value in the answerNotFoundMessage field is used in the response. If an answer is returned for the request, this parameter is true.
Was this page helpful?