botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: Input is too long for requested model.

Question

im using llamaindex as a framework and making a call to bedrock i have given max token window as 100k but with 60k characters  its throwing an exception

code:
    llm = Bedrock(model="anthropic.claude-v2", max_tokens=30000, temperature=temperature,
                  aws_access_key_id=Config.AWS_ACCESS_KEY_ID, aws_secret_access_key=Config.AWS_SECRET_ACCESS_KEY,
                  region_name=Config.AWS_REGION)
    program = LLMTextCompletionProgram.from_defaults(
        output_cls=model,
        llm=llm,
        prompt=prompt,
        verbose=True,
    )
    output = program(text=text)
    details = output.dict()

Answer

Hi THere

I think you are seeing this error (albeit misleading) due to `max_tokens=30000`

The maximum value for output is 4096.  try setting `max_tokens=4096`

See

https://docs.anthropic.com/claude/docs/models-overview#model-comparison

https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-anthropic-claude-text-completion.html#model-parameters-anthropic-claude-text-completion-request-response

Also keep in mind, # of token does not equal # of characters.   Every model caluclates tokens a bit differently.  For Anthropic Claude v2, 100k tokens is equal to apx 75k WORDS.  See https://www.anthropic.com/news/100k-context-windows

botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the InvokeModel operation: Input is too long for requested model.

Relevanter Inhalt