How do I perform a date filter in my bedrock knowledgebase using boto? My guess is that there's a type mismatch between what boto3 is providing and what OpenSearch is expecting, but would like to confirm before trying database rebuild options. The examples that I've found in AWS documentation filter by year, which might not be a date type.
I have a knowledge base configured with AWS OpenSearch. In OpenSearch, I can run a date range query like the following with no issues:
GET bedrock-knowledge-base-default-index/_search
{
"query": {
"bool": {
"must": [
{
"range": {
"publication_date": {
"gte": "2024-01-30",
"lte": "2025-01-30",
"format": "yyyy-MM-dd"
}}}]}}}
I have tried passing dates as strings of yyyy-MM-dd and other formats (to include string of epoch millis) but this seems to present type mismatches. Converting my dates so epoch millis seems to get a step closer
However, when calling with the retrieve_and_generate function, I get errors like the following:
File "/var/lang/lib/python3.12/site-packages/botocore/client.py", line 1061, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the RetrieveAndGenerate operation: failed to parse date field [1.6015104E12] with format [strict_date_optional_time||epoch_millis]: [failed to parse date field [1.6015104E12] with format [strict_date_optional_time||epoch_millis]]
Some additional background:
My filter is set up as follows (which works well for booleans and string types):
import boto3
def retrieve_and_generate(input_text, chat_filter):
knowledge_base_config = {
"generationConfiguration": {
"promptTemplate": {
"textPromptTemplate": PROMPT #your prompt text here
}
},
"knowledgeBaseId": KB_ID, #your knowledge base ID here
"modelArn": GENERATIVE_MODEL_ARN,#your model ARN here
"retrievalConfiguration": {
"vectorSearchConfiguration": {
"filter": chat_filter,
"numberOfResults": 20
}
}
}
retrieve_and_generate_config = {
"knowledgeBaseConfiguration": knowledge_base_config,
"type": "KNOWLEDGE_BASE"
}
if session_id:
response = bar_client.retrieve_and_generate(
input={"text": input_text},
retrieveAndGenerateConfiguration=retrieve_and_generate_config,
sessionId=session_id
)
else:
response = bar_client.retrieve_and_generate(
input={"text": input_text},
retrieveAndGenerateConfiguration=retrieve_and_generate_config
)
return response
How do I configure the boto3 retrieve_and_generate function to filter knowledge base retrieval for a date range?
I have a knowledge base configured with AWS OpenSearch. In OpenSearch, I can run a date range query like the following with no issues:
GET bedrock-knowledge-base-default-index/_search
{
"query": {
"bool": {
"must": [
{
"range": {
"publication_date": {
"gte": "2024-01-30",
"lte": "2025-01-30",
"format": "yyyy-MM-dd"
}}}]}}}
I have tried passing dates as strings of yyyy-MM-dd and other formats (to include string of epoch millis) but this seems to present type mismatches. Converting my dates so epoch millis seems to get a step closer
However, when calling with the retrieve_and_generate function, I get errors like the following:
File "/var/lang/lib/python3.12/site-packages/botocore/client.py", line 1061, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the RetrieveAndGenerate operation: failed to parse date field [1.6015104E12] with format [strict_date_optional_time||epoch_millis]: [failed to parse date field [1.6015104E12] with format [strict_date_optional_time||epoch_millis]]
My filter is set up as follows (which works well for booleans and string types):
import boto3
def retrieve_and_generate(input_text, chat_filter):
knowledge_base_config = {
"generationConfiguration": {
"promptTemplate": {
"textPromptTemplate": PROMPT #your prompt text here
}
},
"knowledgeBaseId": KB_ID, #your knowledge base ID here
"modelArn": GENERATIVE_MODEL_ARN,#your model ARN here
"retrievalConfiguration": {
"vectorSearchConfiguration": {
"filter": chat_filter,
"numberOfResults": 20
}
}
}
retrieve_and_generate_config = {
"knowledgeBaseConfiguration": knowledge_base_config,
"type": "KNOWLEDGE_BASE"
}
if session_id:
response = bar_client.retrieve_and_generate(
input={"text": input_text},
retrieveAndGenerateConfiguration=retrieve_and_generate_config,
sessionId=session_id
)
else:
response = bar_client.retrieve_and_generate(
input={"text": input_text},
retrieveAndGenerateConfiguration=retrieve_and_generate_config
)
return response
I'm defining chat_filter in the main lambda call as follows:
def lambda_handler(event, context):
# input type might vary if using AWS test event
try:
body = json.loads(event['body']) # Parse if it's a string
except Exception as e:
body = event.get('body') # Directly assign if it's a dict
user_query = body.get('msg')
# Construct the filter dynamically based on the input
filters = []
# Date Range Filters (just a start date for minimum example)
pub_date_gte = body.get('publication_date_gte')
pub_date_gte = datetime.fromisoformat(pub_date_gte.replace('Z', '+00:00'))
pub_date_gte = pub_date_gte.astimezone(timezone.utc)
pub_date_gte = int(pub_date_gte.timestamp()*1000)
pub_date_gte_type = type(pub_date_gte)
if pub_date_gte:
filters.append({
"greaterThanOrEquals": { # Fixed operator name
"key": "effective_date",
"value": pub_date_gte
}
})
# Combine filters with AND logic
# If no filters are added then filter needs to be {} instead of {"andAll": []}.
# Bedrock will throw a fit if you send an empty list.
if filters:
chat_filter = {"andAll": filters}
else:
chat_filter = {} #likely need an if/else for the whole statement
# Retrieve and generate api for chat
result = retrieve_and_generate(user_query, session_id, chat_filter)
return result
