By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Getting 500 error when try to create a manual snapshot OpenSearch domain

0

I want to create a manual snapshot of our OpenSearch domain. I used this article for that.

  1. Created bucket: elk-snapshots-mycompany-1 (arn:aws:s3:::elk-snapshots-mycompany-1)
  2. Created access role for this bucket: s3-elk-snapshots-access (arn:aws:iam::112233445566:role/s3-elk-snapshots-access) with es.amazonaws.com entity
  3. Created access role for es:ESHttpPut to my OpenSearch domain and described above PassRole with ec2.amazonaws.com entity elk-snapshots (arn:aws:iam::112233445566:role/elk-snapshots)
  4. Attached elk-snapshots to the EC2 instances that running at the same region as OpenSearch domain
  5. Our domain uses fine-grained access control, so I added to arn:aws:iam::112233445566:role/elk-snapshots to manage_snapshots roles in Kibana
  6. Install pyhton3, requests and requests-aws4auth libs and start the script mentioned in the article above.

I got the following error:

500
{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[s3-manual] path  is not accessible on master node"}],"type":"repository_verification_exception","reason":"[s3-manual] path  is not accessible on master node","caused_by":{"type":"i_o_exception","reason":"Unable to upload object [tests-oENHme5iSL6nLJr-aUNZ4A/master.dat] using a single upload","caused_by":{"type":"amazon_s3_exception","reason":"Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: X7X48KRM4YNRBNM1; S3 Extended Request ID: umFBa+PTbhEbRZlG7Y8jvU+8wmUNOL9GbZQzYyIljmmpkCtXfOrDChVAoHmWDKFLxVvbOm+9mTs=)"}}},"status":500}

After that I was setup the cloudtrail service and found the sample of error access:

{"eventVersion":"1.08","userIdentity":{"type":"AssumedRole","principalId":"AROA2UHKJHDPMDZ4ZPZTG:cloudsearch-snapshot","arn":"arn:aws:sts::112233445566:assumed-role/s3-elk-snapshots-access/cloudsearch-snapshot","accountId":"112233445566","accessKeyId":"ASIA2UHKJHDPBWZRKLQO","sessionContext":{"sessionIssuer":{"type":"Role","principalId":"AROA2UHKJHDPMDZ4ZPZTG","arn":"arn:aws:iam::112233445566:role/s3-elk-snapshots-access","accountId":"112233445566","userName":"s3-elk-snapshots-access"},"attributes":{"creationDate":"2023-07-04T19:17:45Z","mfaAuthenticated":"false"}},"invokedBy":"es.amazonaws.com"},"eventTime":"2023-07-04T19:28:16Z","eventSource":"s3.amazonaws.com","eventName":"PutObject","awsRegion":"eu-central-1","sourceIPAddress":"es.amazonaws.com","userAgent":"es.amazonaws.com","errorCode":"AccessDenied","errorMessage":"Access Denied","requestParameters":{"bucketName":"elk-snapshots-mycompany-1","Host":"elk-snapshots-mycompany-1.s3.eu-central-1.amazonaws.com","x-amz-acl":"private","key":"tests-ZQ3DPKFLQGOgBfNNvmQplQ/master.dat","x-amz-storage-class":"STANDARD"},"responseElements":null,"additionalEventData":{"SignatureVersion":"SigV4","aclRequired":"Yes","CipherSuite":"ECDHE-RSA-AES128-GCM-SHA256","bytesTransferredIn":0,"AuthenticationMethod":"AuthHeader","x-amz-id-2":"e3+3jACTWfehOwcpvO+KeNF1QJd0x5mE4fN9a3t6ADYFpusynRSAgoLtEp2HS1iBeyooVLy86ihnQ+VjRdinBw==","bytesTransferredOut":255},"requestID":"SPFKDSP7SVG1E4EW","eventID":"4d1bc948-d25c-4b71-966f-310fec953a3b","readOnly":false,"resources":[{"type":"AWS::S3::Object","ARN":"arn:aws:s3:::elk-snapshots-mycompany-1/tests-ZQ3DPKFLQGOgBfNNvmQplQ/master.dat"},{"accountId":"112233445566","type":"AWS::S3::Bucket","ARN":"arn:aws:s3:::elk-snapshots-mycompany-1"}],"eventType":"AwsApiCall","managementEvent":false,"recipientAccountId":"112233445566","eventCategory":"Data"}

I also read this arcticle And added "server_side_encryption": "true" setting, but it didn't help. If you give me any idea I will be very grateful

Topics
ServerlessAnalytics
Tags
Amazon OpenSearch Service
Language
English
Evgenii
asked 10 months ago667 views
3 Answers
0

PutObject operation is denied for arn:aws:sts::112233445566:assumed-role/s3-elk-snapshots-access/cloudsearch-snapshot. Does this role have access to S3? Does S3 have permission to be accessed?

profile picture
myronix88
answered 10 months ago
  • Evgenii
    10 months ago

    Does this role have access to S3?

    I believe yes. I also checked this role on AWS policy simulator. And granted access. I don't know that does it mean: /cloudsearch-snapshot As I understood arn:aws:sts::112233445566:assumed-role pointed that role was assumed successfully.

    The policy properties: arn:aws:iam::112233445566:role/s3-elk-snapshots-access Permissions:

    {
	"Version": "2012-10-17",
	"Statement": [
		{
			"Action": [
				"s3:ListBucket"
			],
			"Effect": "Allow",
			"Resource": [
				"arn:aws:s3:::arn:aws:s3:::elk-snapshots-mycompany"
			]
		},
		{
			"Action": [
				"s3:GetObject",
				"s3:PutObject",
				"s3:DeleteObject"
			],
			"Effect": "Allow",
			"Resource": [
				"arn:aws:s3:::arn:aws:s3:::elk-snapshots-mycompany/*"
			]
		}
	]
}

    Trust relationships:

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "es.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

    Does S3 have permission to be accessed?

    Where can I show it?

0

I checked and S3 bucket policy is not required. Add also permissions to this role: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-snapshots.html

profile picture
myronix88
answered 10 months ago
  • Evgenii
    10 months ago

    Add also permissions to this role

    What do you mean? I have another role: arn:aws:iam::112233445566:role/elk-snapshots Permissions:

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::112233445566:role/s3-elk-snapshots-access"
        },
        {
            "Effect": "Allow",
            "Action": "es:ESHttpPut",
            "Resource": "arn:aws:es:eu-central-1:112233445566:domain/domain_name/*"
        }
    ]
}

    Trust relationships

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

    And added this role to EC2 instance. I tried to start this script form the EC2 instance. I also checked SG permissions from the instance above. And I don't see any problems here as well.

0

Does anyone have any ideas? I will be very thankful

Evgenii
answered 10 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content