Why is the RefreshCache operation taking a long time on my file gateway?

3 minute read

I initiated the RefreshCache operation on my file gateway in AWS Storage Gateway. However, the operation is taking a long time to complete. What's the reason for this delay?


The RefreshCache operation identifies the changes (updates, uploads, or deletes) of Amazon Simple Storage Service (Amazon S3) objects since the last time the gateway identified and cached the objects. To do this operation, the file gateway runs a recursive LIST operation on the S3 bucket, and then runs a HEAD object operation on every object that comes back from the LIST operation. The HEAD operation grabs the metadata, which is then stored in the file gateway cache.

The following factors can impact how long it takes for a RefreshCache operation to complete:

  • If there's a large number of objects in the S3 bucket, then the run time of RefreshCache increases. This is because the file gateway runs a HEAD object against all objects in the bucket.
  • RefreshCache operations are specific to individual file shares within a file gateway. One file share supports two RefreshCache API operations at any given time. If you send more requests to initiate a cache refresh, then more operations are triggered even before the completion of operations that are in progress. This can result in an InvalidGatewayRequestException error.
  • S3 buckets can support 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per prefix. These supported requests rates also apply to the requests made by the file gateway to your S3 buckets, which impacts how quickly a RefreshCache operation can be completed. The run time of RefreshCache can increase if the S3 bucket is also being used by services other than the file gateway.

Consider the following ways you can decrease the run time of a RefreshCache operation:

Related information

Automating cache refresh process for File Gateway on AWS Storage Gateway

AWS OFFICIALUpdated 2 years ago