Severe IO performance drop after modifying gp3 volume (state OPTIMIZING)

Question

We have an ec2 machine with an attached EBS volume of type gp3 (size 750GB, 3000 IOPS, 125 MB/s throughput). After modifying that volume (750 GB -> 760 GB) the read performance of already written data to that volume maxes out at 80 IOPS and 15 MB/s.  
  
Newly written files have full IOPS when writing and reading, only data already written to the volume before the modify started is severely affected by degraded read performance.  
  
eu-central-1, volume-id: vol-0962cfcd0b28acb95  
  
ec2 type makes no difference (testet t3.2xlarge and c5.4xlarge). After 4 hours the state of the volume is still in state OPTIMIZING (35%).  
  
We never noticed performance degradation below the source or the target provisioned setting when resizing a volume before.  
  
AWS status dash board shows no issues in eu-central-1, right now.  
  
Any ideas what might have caused this?  
  
Thanks a lot.  
  
Edit: I opened a ticket with AWS support. Will report back with findings.  
  
Edited by: MarcelSJ on Nov 2, 2021 1:04 PM

Answer

AWS did reply to my support ticket after reaching out to the internal EBS team:  
  
> Your volume vol-0962cfcd0b28acb95 experienced higher than normal latency while modifying. This issue only impacts volumes while modifying where the source volume type is gp3. Once the modification is complete, the impact to latency subsides. Your volume and any snapshots of your volume taken during this period are in a healthy state, and no action is required on your part. **We are actively working to deploy a fix and expect completion in Q4 2021.**  
  
Thanks for coming to my TED talk.

Severe IO performance drop after modifying gp3 volume (state OPTIMIZING)

Relevant content