Hello,

When trying to extract values from JSON keys that contain special characters like colons (:), specifically when working with AWS SES event tags such as "ses:caller-identity".

To begin with, EventBridge input transformers have limitations with JSONPath syntax. EventBridge does not support all JSONPath syntax and evaluates it at runtime with restricted capabilities. The supported syntax includes:

• Dot notation (for example, $.detail) • Dashes • Underscores
• Alphanumeric characters • Array indices • Wildcards (*)

Unfortunately, special characters like colons (:) are not supported in the input path, which is why we get the "InputPath for target is invalid" error when trying to use "$.detail.mail.tags["ses:caller-identity"][0]".

Workaround Solution:

Try modifying your input path as follows:

{
  "configuration_set": "$.resources[0]",
  "email_destination": "$.detail.mail.destination[0]",
  "email_message_id": "$.detail.mail.messageId",
  "email_source": "$.detail.mail.sourceArn",
  "email_subject": "$.detail.mail.commonHeaders.subject",
  "event_type": "$.detail.eventType",
  "caller_identity": "$.detail.mail.tags[*][0]",
  "timestamp": "$.time"
}

The key change is using "$.detail.mail.tags[*][0]" instead of trying to reference the specific key with colons.

Keep your Input Template as is:

{
  "timestamp": "<timestamp>",
  "message": "Caller :<caller_identity>,SES Event Type: <event_type>, configuration_set:<configuration_set>, email_source:<email_source>, email_destination:<email_destination> , email_message_id:<email_message_id>, email_subject:<email_subject>"
}

Important Consideration:

For this wildcard approach to work reliably, you need to ensure the order of tags in your SES events is consistent. In your sample event, you'll need to modify the tags section order so that "ses:caller-identity" appears first:

"tags": {
  "ses:caller-identity": ["ses_user"],
  "ses:configuration-set": ["ConfigSet"],
  "ses:source-ip": ["192.0.2.0"],
  "ses:from-domain": ["example.com"]
}

With this configuration, the wildcard [*][0] will capture the first tag's first value, which would be "ses_user" from the "ses:caller-identity" array.

Expected Output:

{
  "timestamp": "2023-07-17T16:48:05Z",
  "message": "Caller :ses_user,SES Event Type: Bounce, configuration_set:arn:aws:ses:us-east-1:123456789012:identity/example.com, email_source:arn:aws:ses:us-east-1:123456789012:identity/, email_destination: , email_message_id:EXAMPLE7c191be45-e9aedb9a-02f9-4d12-a87d-dd0099a07f8a-000000, email_subject:Message sent from Amazon SES"
}

Alternative Approach:

If the wildcard solution doesn't meet your requirements due to the dependency on tag ordering, I recommend considering below alternative:

1. Lambda Function Intermediary: Use a Lambda function as an intermediary to process the SES events before sending them to CloudWatch Logs. This gives you full control over JSON parsing and transformation, including handling keys with special characters.

The Lambda approach would be more robust and flexible, especially if you need to handle multiple SES event types or if the tag ordering varies in your environment.

Reference:

[1]. EventBridge JSONPath syntax limitations - https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-transform-target-input.html#eb-transform-input-transformer

When working with EventBridge input transformers and JSON paths that contain special characters like colons (:) in the key names, you need to use a specific syntax to reference these keys.

The issue you're encountering is related to how EventBridge processes JSON paths with special characters. For keys that contain colons like ses:caller-identity, you need to use bracket notation rather than dot notation.

Try modifying your input path to use this format:

{
"configuration_set": "$.resources[0]",
"email_destination": "$.detail.mail.destination[0]",
"email_message_id": "$.detail.mail.messageId",
"email_source": "$.detail.mail.sourceArn",
"email_subject": "$.detail.mail.commonHeaders.subject",
"event_type": "$.detail.eventType",
"caller_identity": "$.detail.mail.tags.['ses:caller-identity'][0]",
"timestamp": "$.time"
}

Note the change in the caller_identity path: $.detail.mail.tags.['ses:caller-identity'][0] instead of using escaped quotes.

However, it's important to note that EventBridge documentation specifically mentions that you must use JSON dot notation, not bracket notation, for InputPathsMap. This creates a challenge when dealing with keys containing special characters.

As an alternative approach, you might need to:

1. Use a Lambda function as an intermediary to process the event and transform it before sending it to your target
2. Or restructure your rule to avoid needing to directly reference the keys with colons

If you continue to experience issues, consider reaching out to AWS Support as this appears to be a limitation in how EventBridge handles special characters in JSON paths.
Sources
InputTransformer - Amazon EventBridge
Amazon EventBridge Pipes input transformation - Amazon EventBridge