How do I transition AWS WAF managed rules from Count mode to Block mode without a disruption to traffic?

Short description

Before you transition to Block mode, you must analyze the Count mode matches to identify false positives and create appropriate exceptions. Also, turn on AWS WAF logging to send logs to Amazon CloudWatch Logs, Amazon Simple Storage Service (Amazon S3), or Amazon Data Firehose. Then, transition one rule at a time from Count to Block.

Resolution

Analyze Count mode matches

To identify the managed rules that match traffic, run the following query in Amazon CloudWatch Logs Insights:

SOURCE "arn:aws:logs:us-east-1:012345678123:log-group:aws-waf-logs-CloudFront-Distribution" START=-604800s END=0s | 
fields @timestamp, @message 
  | filter @message like "COUNT" 
  | parse @message '"clientIp":"*"' as clientIp 
  | parse @message '"country":"*"' as country 
  | parse @message '"ruleId":"*"' as ruleId 
  | parse @message '"uri":"*"' as uri 
  | parse @message '"httpMethod":"*"' as httpMethod 
  | parse @message '"args":"*"' as queryString 
  | parse @message '"name":"Host","value":"*"' as hostHeader 
  | parse @message '"name":"User-Agent","value":"*"' as userAgent 
  | parse @message '"action":"*"' as action 
  | stats count(*) as requestCount by clientIp, country, ruleId, uri, httpMethod, userAgent, action 
  | sort requestCount desc  
  | limit 100 

To query AWS WAF logs that you store in Amazon S3, use a partition to create a table, and then run the following query in Amazon Athena:

WITH test_dataset AS   
      (SELECT * FROM waf_logs   
        CROSS JOIN UNNEST(rulegrouplist) AS t(rulegroup)  
        CROSS JOIN UNNEST(t.rulegroup.nonterminatingmatchingrules) AS r(nontermrule))  
    SELECT   
      COUNT(*) AS count,   
      httprequest.clientip, 
      webaclid, 
      rulegroup.rulegroupid,  
      nontermrule.ruleid,  
      nontermrule.action  
    FROM test_dataset   
    WHERE from_unixtime(timestamp/1000) > now() - interval '7' day  
    GROUP BY httprequest.clientip, webaclid, rulegroup.rulegroupid, nontermrule.ruleid, nontermrule.action  
    ORDER BY count DESC 

Review the results to determine the rules that most frequently match, what types of requests match, and whether the matches represent legitimate traffic or threats.

Identify false positives

False positives are legitimate requests that a rule incorrectly flags. True positives are threats that a rule correctly identifies.

The following patterns can represent false positives, but require verification:

• API endpoints that accept large data can initiate SizeRestrictions_BODY rules.
• Content management systems (CMSs) that accept HTML input can initiate XSS rules.
• Database administration tools can initiate SQLi rules.

To determine whether matched requests are legitimate traffic, check for the following in your AWS WAF logs:

• Check whether the source IP addresses belong to trusted internal systems or known partners.
• Confirm that the request URIs correspond to legitimate application endpoints.
• Confirm that the request headers contain expected values, such as content types or authorization tokens.
• In the Matched Data field, check whether legitimate application data or traffic initiated the rule match.

For more information about AWS WAF log fields and request components, see Log fields for protection pack (web ACL) traffic.

Create exceptions for false positives

If you identify false positives, then create exceptions before you transition to Block mode. To create an exception, you can either use labels with custom rules, scope-down statements, or exclude specific request components from inspection.

Use labels with custom rules

A managed rule group that has rule actions that are overridden to Count still applies labels to matching requests.

To create an exception rule that allows specific traffic patterns based on the labels, complete the following steps:

1. Open the AWS WAF console.
2. In the navigation pane, choose Protection packs (web ACLs).
3. Select your web ACL, and then choose the Manage rules tab.
4. Choose Add rules, then choose Custom rule, and then choose Custom rule.
5. Choose Visual.
6. Enter a Rule name for the rule, and then configure the following rule settings:
   For Action, choose Block.
   For If a request, choose matches the statement (AND).
   For Inspect, choose Has a label.
   For Statement, choose Label.
   For Match key, enter the label applied by the managed rule group, such as awswaf:managed:aws:core-rule-set:NoUserAgent_Header.
7. For the second Statement, for Inspect select an attribute of known good traffic such as an IPSet or Header and a suitable Statement.
8. Select Negate Statement (NOT).
9. Choose Add rule.
10. Set the rule priority to a later numeric value than the managed rule group so that the label is applied appropriately.
11. Choose Save.

For more examples of Exception rules, see How do I detect false positives caused by AWS Managed Rules?

Use scope-down statements

Add a scope-down statement to the managed rule group to exclude specific traffic from evaluation. The scope-down statements exempt traffic from inspection by all rules in the rule group, rather than individual rules.

For more information, see How to customize behavior of AWS Managed Rules for AWS WAF.

Exclude specific request components from inspection

For rules that inspect request body content, use field-level exclusions to exclude specific fields from inspection. To configure field-level exclusions, modify the rule action to Count mode. Then, create a subsequent rule that checks for the label of the Count mode rule and a NOT statement for the field that you want to exclude.

Incrementally transition rules

Don't transition all rules from Count to Block at the same time. Instead, start with the rule that has the fewest Count matches and the lowest false positive risk. Then, continue to transition one rule at a time.

To transition a rule from Count to Block, remove the Count override for each rule.

After you transition the rules, monitor the BlockedRequests CloudWatch metric of one rule for 24-72 hours. If no legitimate traffic is blocked, then monitor the next rule. If traffic is blocked, then reapply the Count override for the rule and create additional exceptions.

Monitor metrics after you transition the modes

To view your web access control list (web ACL) and rule group metrics, use CloudWatch. To review your All rules graph and Sample requests table, see Viewing a sample of web requests.

After you transition rules to Block mode, take the following actions:

• Create a CloudWatch alarm for the BlockedRequests metric to alert you when the number of blocked requests unexpectedly increases.
• Create a rollback procedure to quickly change rules to Count mode again, if needed.
• Continue to monitor your AWS WAF rules for at least 2 weeks after the final transition. Modify exceptions as your application evolves and traffic patterns change.
• Periodically review AWS WAF logs to identify new false positives that might occur.

Related information

Testing and tuning your AWS WAF protections