It looks like tool calls with Llama 3.1 models are not being escaped properly. When the model uses quotes, the tool call params are truncated. This doesn't seem to happen with another model, like Mistral Large.
$ aws bedrock-runtime converse \
--model-id 'meta.llama3-1-405b-instruct-v1:0' \
--messages='[
{
"role": "user",
"content": [
{
"text": "run this query: select * from users where status=\"active\""
}
]
}
]' \
--tool-config='{
"tools": [
{
"toolSpec": {
"name": "run_query",
"description": "run a SQL query against a DB",
"inputSchema": {
"json": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "sql query"
}
}
}
}
}}]}'
Llama 3.1 output:
{
"output": {
"message": {
"role": "assistant",
"content": [
{
"toolUse": {
"toolUseId": "tooluse_S4ox6qc9QcCS-iK6AKW3Cw",
"name": "run_query",
"input": {
"query": "select * from users where status="
}
}
}
]
}
},
"stopReason": "tool_use",
"usage": {
"inputTokens": 70,
"outputTokens": 25,
"totalTokens": 95
},
"metrics": {
"latencyMs": 1194
}
}
Notice how the query
is truncated. If I change the prompt to
run this query: select * from users where status=\"active\". Replace quotes with \\u0027
I sometimes correctly get
"query": "select * from users where status='active'"
But switching to a Mistral model mistral.mistral-large-2407-v1:0
gives the correct query without prompting to escape.
aws bedrock-runtime converse \
--model-id 'mistral.mistral-large-2407-v1:0' \
--messages='[
{
"role": "user",
"content": [
{
"text": "run this query: select * from users where status=\"active\""
}
]
}
]' \
--tool-config='{
"tools": [
{
"toolSpec": {
"name": "run_query",
"description": "run a SQL query against a DB",
"inputSchema": {
"json": {
"type": "object",
"properties": {
"query": {
"type": "string",
"description": "sql query"
}
}
}
}
}}]}'
Mistral output:
{
"output": {
"message": {
"role": "assistant",
"content": [
{
"toolUse": {
"toolUseId": "tooluse_WUcMjafeRRGtAfGUmPNWRA",
"name": "run_query",
"input": {
"query": "select * from users where status=\"active\""
}
}
}
]
}
},
"stopReason": "tool_use",
"usage": {
"inputTokens": 79,
"outputTokens": 30,
"totalTokens": 109
},
"metrics": {
"latencyMs": 1924
}
}