Using Bedrock with Mistral 2 Large, converse API with tools would not let me use streaming feature.

Question

I am using Bedrock to access Mistral 2 Large, with converse_stream API. I am using tools too. 
I get this message back:

BedrockManager - ERROR - get_model_response_stream failed: An error occurred (ValidationException) when calling the ConverseStream operation: This model doesn't support tool use in streaming mode.

The call looks like this:

final_response = self.bedrock.converse_stream(
                modelId=self.config.model_id,
                system=system,
                messages=messages,
                inferenceConfig=inferenceConfig,
                toolConfig=tool_config_tool
            )

tool_config is

tool_config_tool = {
    "tools": [
        {
            "toolSpec": {
                "name": "search_document",
                "description": "Search through the uploaded document for relevant passages",
                "inputSchema": {
                    "json": {
                        "type": "object",
                        "properties": {"query": {"type": "string"}},
                        "required": ["query"],
                    }
                },
            }
        }
    ],
    "toolChoice": {"auto": {}},
}

My question is two-fold:

1. When will AWS support this for Mistral 2? 
 2. How do I change the tool_config_tool to make sure the API is not trying to use tools in this instance to allow for streaming.

re:Post Agent · Answer

Based on the error message you're receiving, it appears that Mistral 2 Large currently doesn't support tool use in streaming mode when using the ConverseStream API.

To address your questions:

1. Regarding when AWS will support tool use with streaming for Mistral 2 Large: There is no specific timeline mentioned in the available information. AWS regularly updates its services, but there's no announced roadmap for when this specific capability will be added for Mistral 2 Large.

2. To modify your approach to allow streaming without tools:

You have two options:

- Remove the toolConfig parameter entirely from your ConverseStream call. This would allow you to use streaming but without tool functionality:
```python
final_response = self.bedrock.converse_stream(
modelId=self.config.model_id,
system=system,
messages=messages,
inferenceConfig=inferenceConfig
# toolConfig parameter removed
)
```

- Alternatively, if you need both streaming and tool functionality, you might need to implement a hybrid approach where you use non-streaming calls (the regular `converse` API) when tool use is required, and switch to `converse_stream` when streaming without tools is acceptable.

It's worth noting that there are some limitations in the current Bedrock APIs. For example, another known limitation is that guardrails cannot be enabled when using tools in converse operations. These are current implementation constraints that may be addressed in future updates.

If tool functionality is critical for your application, you might need to temporarily use the non-streaming version of the API until AWS adds support for tool use in streaming mode for Mistral 2 Large.  
**Sources**  
[able to use streamingConfigurations in AgentsforBedrockRuntime invoke_agent boto3 sdk | AWS re:Post](https://repost.aws/questions/QUuzL1XVQ0TVyp5dEPZn5BZw/able-to-use-streamingconfigurations-in-agentsforbedrockruntime-invoke-agent-boto3-sdk)  
[Why Guardrails cannot be enabled when using tools in converse operations | AWS re:Post](https://repost.aws/questions/QUAYu-Ap_1RtWDxQBTbvqRAA/why-guardrails-cannot-be-enabled-when-using-tools-in-converse-operations)  
[AWS Bedrock Agents Support Streaming Responses? | AWS re:Post](https://repost.aws/questions/QUsqK6QQwoQ-a-crTHpkjfEA/aws-bedrock-agents-support-streaming-responses)  
[Converse API examples - Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/conversation-inference-examples.html)

Using Bedrock with Mistral 2 Large, converse API with tools would not let me use streaming feature.

Add your answer

Relevant content