How do I troubleshoot message filtering issues in Amazon SNS?

Short description

By default, an Amazon SNS topic subscriber receives every message that's published to the topic. To receive only a subset of the messages, a subscriber must assign a filter policy to the topic subscription. For more information, see Amazon SNS message filtering.

The FilterPolicyScope subscription attribute lets you choose the filtering scope by setting one of the following values:

• MessageAttributes: The filter policy is applied to the message attributes. This is the default.
• MessageBody: The filter policy is applied to the message body.

Note: While applying the subscription filter policy, if the filter policy scope isn't defined, the scope defaults to MessageAttributes.

Resolution

Here are some common issues related to message filtering and the ways to troubleshoot them:

Amazon SNS subscriptions have a filter policy with a 15-minute wait

AWS Identity and Access Management (IAM) and Amazon SNS use a distributed computing model called eventual consistency. Additions or changes to a subscription filter policy require up to 15 minutes to fully take effect. This duration can't be reduced.

Error: The total combination of values (120) must not exceed 100

For Amazon SNS subscription filter policies, the default per topic limit, per AWS account, is 200. The filter policies' limit per AWS account is 10,000.

The Amazon SNS subscription filter policy limit increase requires an internal service team's approval. The approval is based on factors that include use case, Region, number of messages published, and the number of subscriptions details.

If you expect an increase in your requirement for the Amazon SNS subscription filter policy, consider an alternative option.

Possible solutions

• Have multiple conditions (or filters) in the same filter policy associated with the topic subscription.
• Subscribe an AWS Lambda function to an Amazon SNS topic and handle the filter logic in the Lambda function code. The Lambda function handles the filter logic and publishes based on the message attributes to a different SNS topic with subscriptions.

Error: AuthorizationErrorException

This error appears when the user or role making the SetSubscriptionAttributes request doesn't have enough permissions.

Add the SetSubscriptionAttributes permissions to the user and then edit the subscription filter policy. For more information on adding permissions, see Using identity-based policies with Amazon SNS.

This error also occurs when there's a permission issue, usually in a cross-account environment. For example, you set up a cross-account with Amazon SNS. Users adding a filter policy to the subscription using the SetSubscriptionAttributes API call get the "Not authorized to access this subscription" error.

The SetSubscriptionAttributes allows a subscription owner to set an attribute-like filter policy of the subscription. In a cross-account setup with SNS (example: Amazon SNS-Amazon Simple Queue Service cross account), the subscription owner is the AWS account used to make the Subscribe API call to create the subscription. A user from another account (which doesn't own the subscription) might try to add a filter policy to the subscription using the SetSubscriptionAttributes API call. This user now sees the "Not authorized to access this subscription" error.

Notifications aren't being sent to SNS topic even after applying the correct filter policy

A subscription accepts a message under the following conditions:

• Filter policy scope is set to MessageAttributes
• Filter policy scope is set to MessageBody

Note: For more information on filter policy, see Amazon SNS subscription filter policies.

To troubleshoot and investigate the issue, check the following:

• Check that you are passing the correct message attributes.
• For message body filtering, check that the message payload is a well-formed JSON object. For more information, see this blog for payload-based message filtering.
• Check the SNS Amazon CloudWatch metrics shown here to verify if the data point is populated for them:

NumberOfNotificationsFilteredOut: The number of messages that were rejected by subscription filter policies. A filter policy rejects a message when the message attributes don't match the policy attributes.

NumberOfNotificationsFilteredOut-MessageAttributes: The number of messages that were rejected by subscription filter policies for attribute-based filtering.

NumberOfNotificationsFilteredOut-MessageBody: The number of messages that were rejected by subscription filter policies for payload-based filtering.

NumberOfNotificationsFilteredOut-InvalidAttributes: The number of messages that were rejected by subscription filter policies because the messages' attributes are not valid. For example, the message was rejected because the attribute JSON is incorrectly formatted.

NumberOfNotificationsFilteredOut-NoMessageAttributes: The number of messages that were rejected by subscription filter policies because the messages have no attributes.

NumberOfNotificationsFilteredOut-InvalidMessageBody: The number of messages that were rejected by subscription filter policies because the message body is not valid for filtering. For example, the message has a non-valid JSON message body.

For more information, see Amazon SNS metrics.

Related information

Amazon SNS resource