I have deployed Mistral on Sagemaker using the Huggignface image. I am getting good response for small input prompts. when I send little big size promt I am gettitng errror:
"error": "Input validation error: `inputs` tokens + `max_new_tokens` must be <= 2048. Given: 13179 `inputs` tokens and 1024 `max_new_tokens`",
"error_type": "validation"To solve this error, we need to set environment variables MAX_INPUT_LENGTH and MAX_TOTAL_TOKEN.
While deploying llm with sagemaker add this environment variables
hub = {
‘HF_MODEL_ID’:’mistralai/Mixtral-8x7B-Instruct-v0.1′,
‘SM_NUM_GPUS’: json.dumps(8),
“MAX_INPUT_LENGTH”: ‘30000’, => put here any value upto 32768 as per your requirement.
“MAX_TOTAL_TOKENS”: ‘32768’,
“MAX_BATCH_PREFILL_TOKENS”: ‘32768’,
“MAX_BATCH_TOTAL_TOKENS”: ‘32768’,
}
