Input validation error: `inputs` tokens + `max_new_tokens` must be <= 2048.

I have deployed Mistral on Sagemaker using the Huggignface image. I am getting good response for small input prompts. when I send little big size promt I am gettitng errror:

"error": "Input validation error: `inputs` tokens + `max_new_tokens` must be <= 2048. Given: 13179 `inputs` tokens and 1024 `max_new_tokens`",
"error_type": "validation"

To solve this error, we need to set environment variables MAX_INPUT_LENGTH and MAX_TOTAL_TOKEN.

While deploying llm with sagemaker add this environment variables

hub = {
‘SM_NUM_GPUS’: json.dumps(8),
“MAX_INPUT_LENGTH”: ‘30000’, => put here any value upto 32768 as per your requirement.
“MAX_TOTAL_TOKENS”: ‘32768’,


Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button