Durch die Nutzung von AWS re:Post stimmt du den AWS re:Post Nutzungsbedingungen
AWS re:Postre:Post

Getting 500 error when try to create a manual snapshot OpenSearch domain

0

I want to create a manual snapshot of our OpenSearch domain. I used this article for that.

  1. Created bucket: elk-snapshots-mycompany-1 (arn:aws:s3:::elk-snapshots-mycompany-1)
  2. Created access role for this bucket: s3-elk-snapshots-access (arn:aws:iam::112233445566:role/s3-elk-snapshots-access) with es.amazonaws.com entity
  3. Created access role for es:ESHttpPut to my OpenSearch domain and described above PassRole with ec2.amazonaws.com entity elk-snapshots (arn:aws:iam::112233445566:role/elk-snapshots)
  4. Attached elk-snapshots to the EC2 instances that running at the same region as OpenSearch domain
  5. Our domain uses fine-grained access control, so I added to arn:aws:iam::112233445566:role/elk-snapshots to manage_snapshots roles in Kibana
  6. Install pyhton3, requests and requests-aws4auth libs and start the script mentioned in the article above.

I got the following error:

500
{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[s3-manual] path  is not accessible on master node"}],"type":"repository_verification_exception","reason":"[s3-manual] path  is not accessible on master node","caused_by":{"type":"i_o_exception","reason":"Unable to upload object [tests-oENHme5iSL6nLJr-aUNZ4A/master.dat] using a single upload","caused_by":{"type":"amazon_s3_exception","reason":"Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: X7X48KRM4YNRBNM1; S3 Extended Request ID: umFBa+PTbhEbRZlG7Y8jvU+8wmUNOL9GbZQzYyIljmmpkCtXfOrDChVAoHmWDKFLxVvbOm+9mTs=)"}}},"status":500}

After that I was setup the cloudtrail service and found the sample of error access:

{"eventVersion":"1.08","userIdentity":{"type":"AssumedRole","principalId":"AROA2UHKJHDPMDZ4ZPZTG:cloudsearch-snapshot","arn":"arn:aws:sts::112233445566:assumed-role/s3-elk-snapshots-access/cloudsearch-snapshot","accountId":"112233445566","accessKeyId":"ASIA2UHKJHDPBWZRKLQO","sessionContext":{"sessionIssuer":{"type":"Role","principalId":"AROA2UHKJHDPMDZ4ZPZTG","arn":"arn:aws:iam::112233445566:role/s3-elk-snapshots-access","accountId":"112233445566","userName":"s3-elk-snapshots-access"},"attributes":{"creationDate":"2023-07-04T19:17:45Z","mfaAuthenticated":"false"}},"invokedBy":"es.amazonaws.com"},"eventTime":"2023-07-04T19:28:16Z","eventSource":"s3.amazonaws.com","eventName":"PutObject","awsRegion":"eu-central-1","sourceIPAddress":"es.amazonaws.com","userAgent":"es.amazonaws.com","errorCode":"AccessDenied","errorMessage":"Access Denied","requestParameters":{"bucketName":"elk-snapshots-mycompany-1","Host":"elk-snapshots-mycompany-1.s3.eu-central-1.amazonaws.com","x-amz-acl":"private","key":"tests-ZQ3DPKFLQGOgBfNNvmQplQ/master.dat","x-amz-storage-class":"STANDARD"},"responseElements":null,"additionalEventData":{"SignatureVersion":"SigV4","aclRequired":"Yes","CipherSuite":"ECDHE-RSA-AES128-GCM-SHA256","bytesTransferredIn":0,"AuthenticationMethod":"AuthHeader","x-amz-id-2":"e3+3jACTWfehOwcpvO+KeNF1QJd0x5mE4fN9a3t6ADYFpusynRSAgoLtEp2HS1iBeyooVLy86ihnQ+VjRdinBw==","bytesTransferredOut":255},"requestID":"SPFKDSP7SVG1E4EW","eventID":"4d1bc948-d25c-4b71-966f-310fec953a3b","readOnly":false,"resources":[{"type":"AWS::S3::Object","ARN":"arn:aws:s3:::elk-snapshots-mycompany-1/tests-ZQ3DPKFLQGOgBfNNvmQplQ/master.dat"},{"accountId":"112233445566","type":"AWS::S3::Bucket","ARN":"arn:aws:s3:::elk-snapshots-mycompany-1"}],"eventType":"AwsApiCall","managementEvent":false,"recipientAccountId":"112233445566","eventCategory":"Data"}

I also read this arcticle And added "server_side_encryption": "true" setting, but it didn't help. If you give me any idea I will be very grateful

Themen
ServerlosAnalysen
Tags
Amazon OpenSearch-Dienst
Sprache
English
Evgenii
gefragt vor 10 Monaten689 Aufrufe
3 Antworten
0

PutObject operation is denied for arn:aws:sts::112233445566:assumed-role/s3-elk-snapshots-access/cloudsearch-snapshot. Does this role have access to S3? Does S3 have permission to be accessed?

profile picture
myronix88
beantwortet vor 10 Monaten
  • Evgenii
    vor 10 Monaten

    Does this role have access to S3?

    I believe yes. I also checked this role on AWS policy simulator. And granted access. I don't know that does it mean: /cloudsearch-snapshot As I understood arn:aws:sts::112233445566:assumed-role pointed that role was assumed successfully.

    The policy properties: arn:aws:iam::112233445566:role/s3-elk-snapshots-access Permissions:

    {
	"Version": "2012-10-17",
	"Statement": [
		{
			"Action": [
				"s3:ListBucket"
			],
			"Effect": "Allow",
			"Resource": [
				"arn:aws:s3:::arn:aws:s3:::elk-snapshots-mycompany"
			]
		},
		{
			"Action": [
				"s3:GetObject",
				"s3:PutObject",
				"s3:DeleteObject"
			],
			"Effect": "Allow",
			"Resource": [
				"arn:aws:s3:::arn:aws:s3:::elk-snapshots-mycompany/*"
			]
		}
	]
}

    Trust relationships:

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "es.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

    Does S3 have permission to be accessed?

    Where can I show it?

0

I checked and S3 bucket policy is not required. Add also permissions to this role: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-snapshots.html

profile picture
myronix88
beantwortet vor 10 Monaten
  • Evgenii
    vor 10 Monaten

    Add also permissions to this role

    What do you mean? I have another role: arn:aws:iam::112233445566:role/elk-snapshots Permissions:

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::112233445566:role/s3-elk-snapshots-access"
        },
        {
            "Effect": "Allow",
            "Action": "es:ESHttpPut",
            "Resource": "arn:aws:es:eu-central-1:112233445566:domain/domain_name/*"
        }
    ]
}

    Trust relationships

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

    And added this role to EC2 instance. I tried to start this script form the EC2 instance. I also checked SG permissions from the instance above. And I don't see any problems here as well.

0

Does anyone have any ideas? I will be very thankful

Evgenii
beantwortet vor 10 Monaten

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen

Relevanter Inhalt