What to do when you receive the AWS_KAFKA_HIGH_RISK_CONFIG_RF_EQUALS_MINISR Health event for Amazon MSK

This article refers to the replication factor as RF and to min.insync.replicas as minISR.

Symptoms

• You receive the AWS_KAFKA_HIGH_RISK_CONFIG_RF_EQUALS_MINISR event in AWS Health.
• The event body contains only the cluster ARN and the detection time. It does not name the affected topics.
• When a broker goes offline during a rolling security patch, broker replacement, or an Availability Zone impairment, acks=all producers can fail to write with a NotEnoughReplicas or NotEnoughReplicasAfterAppend exception.

Diagnosis

The event does not tell you which topics are affected, and the MSK API cannot retrieve the historical RF and minISR settings for a topic, so you inspect the current settings directly. The examples below use $BS for the bootstrap broker string.

1. Check the ReplicationFactor of a topic.

   kafka-topics.sh --bootstrap-server $BS --describe --topic <topic-name>
   

2. Check the topic-level min.insync.replicas override.

   kafka-configs.sh --bootstrap-server $BS --describe \
     --entity-type topics --entity-name <topic-name>
   

3. If there is no override, the cluster default applies. Check the min.insync.replicas and default.replication.factor in your MSK configuration, and confirm whether minISR was raised to 2 on a 2-AZ cluster.
4. Check the CloudWatch UnderMinIsrPartitionCount metric. It should always be 0 under normal conditions; a value greater than 0 requires immediate action.

Note: If every topic has a minISR lower than RF and UnderMinIsrPartitionCount is 0, no further action may be required. The event reflects the configuration observed at detection time, so RF and minISR might have been temporarily equal during topic creation or a configuration change and then returned to normal. Confirm that the current settings are actually RF == minISR before you make any change.

Root cause

By default, Amazon MSK uses RF 3 and minISR 2 in 3-AZ clusters, and RF 2 and minISR 1 in 2-AZ clusters, so RF is always greater than minISR. RF equal to minISR therefore almost always comes from a user setting or a custom configuration.

1. You set minISR equal to RF in a custom MSK configuration. For example, you raised minISR to 3 on a 3-AZ cluster.
2. You explicitly created a topic with RF equal to minISR. For example, you set min.insync.replicas to 2 on a topic with RF 2, or set both to 1.
3. The most common mistake: a minISR of 2 assumes a 3-AZ cluster with RF 3, but you applied it to a 2-AZ cluster. Because a 2-AZ cluster has a default RF of 2, every topic created with the default RF becomes RF 2 and minISR 2.

Resolution

Based on the current RF, make minISR one less than RF.

• If RF is sufficient, lower minISR. This is a dynamic setting, so it applies immediately and moves no data.

  kafka-configs.sh --bootstrap-server $BS --alter \
    --entity-type topics --entity-name <topic-name> \
    --add-config min.insync.replicas=2
  

• If RF is low, increase RF to 3 and then set minISR to 2. Increasing RF requires a partition reassignment with kafka-reassign-partitions.sh; follow the procedure in the related information below.

To prevent the issue on new topics, set your custom MSK configuration defaults to default.replication.factor 3 and min.insync.replicas 2. For the reasoning behind the recommended values and the RF increase procedure, see Build highly available clusters and How do I make sure that security patches don't disrupt my client I/O?.