Storage Capacity on MSK maxed out
A new cluster with 3 brokers having EBS storage of 2 GB each was created which worked fine for a day . on the second day, the producer stopped sending messages and while checking the logs, found this error in the logs - 'Give up sending metadata request since no node is available'. We even tried to connect to the cluster but was receiving this exception. kafka.errors.NoBrokersAvailable: NoBrokersAvailable. Checked the Cloudwatch metrics and found that the storage capacity has reached a maximum limit. changed the values of configuration to retention.ms to 86400000 ms (24 hours) and retention.bytes to 500000000 bytes (500 MB) in order to release memory and also increased the EBS storage to 15 GB but the brokers were still not available.
- 최신
- 최다 투표
- 가장 많은 댓글
Hello there, when storage capacity on MSK is already maxed out, deleting topics, adjusting retention etc wouldn't take any effect as the brokers will be in unresponsive state. In this scenario only option would be to increase the storage and once the brokers are healthy retention kicks in. I see you mentioned storage is increased to 15gb. Are the brokers still not available? Can you please make sure if storage update went through successfully by describing the cluster or checking whether 'EBS storage volume per broker' shows 15gb in MSK console.
Hi, when you re-size the EBS, what time take this proccess? because I re-size 500 GB to 1TB (13 hours have passed since the modification) and I can't send data to MSK. . I need reboot brocker? or It's automatically
