How do I resolve Amazon EMR cluster logs that I can't view in an Amazon S3 bucket with an SSE-KMS encryption policy?

Resolution

To write logs to an Amazon S3 bucket that has an SSE-KMS encryption policy, use the sync command to manually upload the files.

Note: 

• If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.
• With Amazon EMR versions 5.30.0 and later (except Amazon EMR 6.0.0), you can encrypt log files that are stored in an Amazon S3 bucket with an AWS KMS customer managed key.

To use the sync command to manually upload your log files, complete the following steps:

1. Use SSH to connect to the primary node.
2. Locate the log files that you want to copy. For example, step logs are stored in /mnt/var/log/hadoop/steps on the primary node.
3. To copy the log files to the bucket, run the sync command:

   aws s3 sync /mnt/var/log/hadoop/steps/ s3://example-bucket/elasticmapreduce/$example-cluster-id/steps/ --sse aws:kms --sse-kms-key-id example-kms-key-id

   Note: Replace example-bucket with your bucket name, example-cluster-id with the cluster ID, and example-kms-key-id with the AWS KMS key ID.

Note: To automate the sync command, use a cron job. To configure the cron job, run a custom bootstrap action on all nodes when you launch an Amazon EMR cluster.

Related information

Using server-side encryption with AWS KMS keys (SSE-KMS)

What happens to new or existing objects when I turn on default encryption with AWS KMS on my Amazon S3 bucket?