Why can't I increase the Amazon EBS disk volume in my Amazon MSK cluster?

Resolution

You don't have the required permissions to modify the EBS volume

Make sure that you have the required permissions to modify the Amazon EBS disk volume for your cluster. Attach the AmazonMSKFullAccess policy to your AWS Identity and Access Management (IAM) user or role.

Your cluster is in the optimizing state from a prior EBS volume upgrade

When your cluster is in the optimizing state from a previous EBS volume upgrade, you receive the following error:

"An EBS volume in this cluster is under modification from a previous operation"

When you increase the size of an EBS volume that's attached to your Amazon MSK broker node, the newly added storage is available to you immediately. The cluster takes 6 to 24 hours or more to optimize. The storage size determines how long the cluster optimization takes.

When you increase the size of the EBS volume, make sure that the added volume is sufficient for the next 48 hours or more. During this period, you can't add more volumes to your cluster. You can't update the volume again until the cluster completes optimization.

Important: If your cluster is in the optimizing state and you start to run out of disk space, then check if you have automatic scaling turned on. If automatic scaling is on, then turn it off. If you keep automatic scaling turned on, then the volume increases only by 10% or 10GB when the optimization is complete. This increase might be too low, and it can cause issues with volume modification. After you turn off automatic scaling, review your traffic patterns and increase your storage space accordingly. If your cluster is in the process of modification and the KafkaDataLogsDiskUsed metric is less than 94%, then delete unused or old topics to free up space. Run the following command to determine the amount of space occupied by each topic:

kafka-topics.sh --command-config /tmp/client.properties --bootstrap-server $bservers --list | xargs -I{} sh -c "echo -n '{} - ' && kafka-log-dirs.sh --command-config /tmp/client.properties --bootstrap-server $bservers --topic-list {} --describe | grep '^{'  | jq '[ ..|.size? | numbers ] | add' | numfmt --to iec " | tee /tmp/topics-by-size.list

Note: If KafkaDataLogsDiskUsed is higher than 94%, then the preceding command might be unresponsive and you might be unable to delete the topics.

For more information, see Modifying topics on the Apache Kafka website.