How do I troubleshoot Amazon S3 Batch Operations issues?

8 minute read
0

When I create an Amazon Simple Storage Service (Amazon S3) Batch Operations job for objects in my bucket, Amazon S3 returns an error. Or, the batch job fails.

Short description

If an Amazon S3 Batch Operations job encounters an issue that doesn't allow it to run successfully, then the job fails. A job that fails generates one or more failure codes and reasons. To view the failure codes and reasons for an Amazon S3 Batch Operation, request the job's details. You can also review your failure codes and reasons in the completion report for the job.

To prevent jobs from running a large number of unsuccessful operations, Amazon S3 imposes a task-failure threshold on every Batch Operations job. Amazon S3 monitors the task failure rate after at least 1,000 tasks are run. If a job exceeds the failure rate of 50%, then the job fails. To resolve this failure, review the causes of the failures and correct them. Then, resubmit the job.

Resolution

The manifest file format is incorrect (.csv or JSON)

Amazon S3 Batch Operations supports .csv and JSON (Amazon S3 Inventory report) manifest files. If the manifest file isn't correct, you must create a new batch job in Amazon S3 and specify the correct format.

  • For the Amazon S3 Inventory report, use a CSV-formatted report and specify the manifest.json file that's associated with the inventory report.

  • For .csv files, include the bucket name and object key in each row in the manifest file. Optionally, include the object version. If you include version IDs in the manifest, then you must specify IDs for all objects. Otherwise, don't include any version IDs. Object keys must be URL-encoded.

    Note: If the objects in your manifest are in a versioned bucket, then you must specify the version IDs for the objects. Otherwise, the batch job fails. Or, Amazon S3 might apply the batch job to the incorrect version of the object.

For more information, see Specifying a manifest.

The manifest file specifies multiple bucket names or contains multiple header rows

With S3 Batch Operations, all objects that the manifest file lists must exist in the same bucket. Otherwise, you receive this error:

"Reasons for failure: Cannot have more than 1 bucket per Job. JOB_ID"

For S3 Batch Operation jobs, make sure that your manifest file specifies only one bucket name and doesn't contain any header rows. In this example, Amazon S3 returns an error because the manifest file contains multiple header rows.

bucket,key
my-batch-bucket,object001.txt
my-batch-bucket,object002.txt
my-batch-bucket,object003.txt
my-batch-bucket,object004.txt

The IAM role is missing permissions to read the manifest file

The AWS Identity and Access Management (IAM) role that creates the S3 Batch Operations job must have GetObject read permissions for the manifest file. Check the object's metadata for any access mismatches with S3 Object ownership. Also, look for any unsupported AWS Key Management Service (AWS KMS) keys that encrypt the manifest file.

If the IAM role doesn't have the correct permissions, then you receive these errors when you create an S3 Batch Operations job:

AWS CLI error example

"Reason for failure Reading the manifest is forbidden: AccessDenied"

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

Amazon S3 console error example

"Warning: Unable to get the manifest object's ETag. Specify a different object to continue"

Note: S3 Batch Operations supports CSV inventory reports that are AWS KMS encrypted. S3 Batch Operations doesn't support .csv manifest files that are AWS KMS encrypted. For more information, see Configuring inventory by using the S3 console.

The batch job is in a different Region

S3 Batch Operations copy jobs must be in the same AWS Region as the destination bucket where copy the objects to. When you create a batch job, select the same Region as your destination bucket. For example, if the destination bucket is in the us-west-2 Region, then select us-west-2 as the Region for the batch job.

The target bucket for your S3 Inventory report is missing

There must be a target bucket for the manifest that S3 Batch Operations generates. The Amazon S3 bucket policy must also allow the s3:PutObject action. If the report is sent to another AWS account, then confirm that the target bucket allows the IAM role to perform the s3:PutObject action.

The IAM role's trust policy is missing

Note: Make sure that you specify an IAM role and not an IAM user.

The trust policy for an IAM role defines the conditions that must be met for other principals to assume it. To allow the S3 Batch Operations service principal to assume the IAM role, attach a trust policy to the role.

This trust policy example grants access to Amazon S3. It reduces any risks that are associated with privilege escalation:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "batchoperations.s3.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

The IAM permissions to create a batch job are missing

Before you create and run S3 Batch Operations jobs, grant the IAM role the required permissions. If your IAM role is missing the required permissions to perform the S3 Batch Operations job, then the batch job fails.

To create an S3 Batch Operations job, grant the IAM role the s3:CreateJob permissions. The same entity that creates the job must also have iam:PassRole permissions. This allows the entity to pass the IAM role that you specify for the batch job. For more information, see IAM JSON policy elements: Resource.

Access to the source bucket, S3 Inventory report, or destination bucket is missing

Make sure that the IAM role that you use for S3 Batch Operations has the required permissions to perform the batch job.

Here's an example IAM policy for the copy operation:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:PutObjectTagging"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::{{DestinationBucket}}/*"
    },
    {
      "Action": [
        "s3:GetObject",
        "s3:GetObjectAcl",
        "s3:GetObjectTagging",
        "s3:ListBucket"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::{{SourceBucket}}",
        "arn:aws:s3:::{{SourceBucket}}/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:GetObjectVersion"
      ],
      "Resource": [
        "arn:aws:s3:::{{ManifestBucket}}/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:s3:::{{ReportBucket}}/*"
      ]
    }
  ]
}

For more information, see Granting permissions for Amazon S3 Batch Operations.

The Organizations SCP is restrictive

If you use AWS Organizations, then confirm that there aren't any Deny statements that deny access to Amazon S3. For example, your service control policy (SCP) explicitly denies all S3 actions. In this case, you might get an Access Denied error when you create a batch job.

This example policy explicitly denies all S3 actions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Principal": "*",
      "Effect": "Deny",
      "Action": "s3:*",
      "Resource": "*"
    }
  ]
}

To apply a restrictive policy, add the IAM role that S3 Batch Operations uses to perform the operation to the allow list. This example adds an exception to the policy:

{
  "Version": "2012-10-17",
  "Statement": \[
    {
      "Principal": "\*",
      "Effect": "Deny",
      "Action": "s3:\*",
      "Resource": "\*",
      "Condition": {
        "StringNotLike": {
          "aws:userId": \[
            "AROAEXAMPLEID:\*",
            "AIDAEXAMPLEID",
            "111111111111"
          \]
        }
      }
    }
  \]
}

The version ID for an object is missing in the manifest

If a Batch Operations job encounters an object in the manifest with an empty version ID field, then you receive this error:

"Error: BUCKET_NAME,prefix/file_name,failed,400,InvalidRequest,Task failed due to missing VersionId"

If the manifest format uses version IDs during the operation, then the version ID field can't be an empty string. Instead, the version ID field must be a "null" string. Non-versioned jobs don't encounter this error. They operate on the most recent version of each object and not the version IDs that are found in the manifest. To fix this error, convert the empty version IDs to null strings.

Note: The Batch Operations fails for that particular object, but not the entire job.

The job report isn't delivered when Amazon S3 Object Lock retention mode is turned on

When you configure Object Lock retention mode on a destination bucket in either governance mode or compliance mode, you might receive this error:

"Error: Reasons for failure. The job report could not be written to your bucket. Please check your permissions."

Amazon S3 doesn't support Object Lock for destination buckets with retention mode configurations. When retention mode is configured, the bucket is write-once-read-many (WORM) protected. To fix this error, choose a destination bucket for your job completion reports that doesn't have Object Lock retention mode configured.

Note: It's the completion report that fails, not the job. The job completes successfully, and all objects process.

The ETag versions don't match

When you specify the manifest in the batch operation job, you can specify the manifest object key, ETag, and optional version ID. When you specify the manifest file, check that the ETag's value matches the ETag of the manifest object's latest version in the S3 bucket. In the Amazon S3 console's Batch operations tab, check the Manifest object ETag in the manifest file properties. In the AWS CLI, check the value of the Etag that the manifest specification passes.

If the populated ETag that's in the console or AWS CLI doesn't match the Etag in the S3 bucket, then this error occurs:

"Error reading the manifest. Caused by: ETag mismatch. Expected ETag: 69f52a4e9f797e987155d9c8f5880897"

Note the Expected ETag in this error, and make sure that both versions of your ETag match. For more information, see Specifying a manifest.

AWS OFFICIAL
AWS OFFICIALUpdated 6 months ago
4 Comments

There's an apparent factual inaccuracy in this article: The "Cannot have more than 1 bucket per job" issue does not apply only to the three operations specified, it also appears to apply to the InvokeLambda Job Operation, and may in fact apply to all job operations.

Also, the line that states that CSV manifests must have "object key, ETag, and optional version ID" fields is inaccurate. The fields are bucket name, object key, and optional version ID; ETag is not a required or even allowed field.

replied a year ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied a year ago

I am facing this error, not one of the above listed:

Error reading the manifest. Caused by: ETag mismatch. Expected ETag: 69f52a4e9f797e987155d9c8f5880897

replied 10 months ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 10 months ago