Durch die Nutzung von AWS re:Post stimmt du den AWS re:Post Nutzungsbedingungen

AWS Step Function 无法识别EMR工作流ID的问题

0

【以下的问题经过翻译处理】 我创建了EMR集群,然后通过Step Function向 EMR 集群添加一个步骤。

发现jobflowid 未被识别。

请问该如何解决这个问题?或者是我的操作过程有任何疏忽的吗? Step Function状态机如下:

{
  "StartAt": "add_emr_step",
  "States": {
    "add_emr_step": {
      "End": true,
      "InputPath": "$",
      "Parameters": {
        "ClusterId": "j-INTERESTINGID",
        "Step": {
          "Name": "$$.Execution.Id",
          "ActionOnFailure": "CONTINUE",
          "HadoopJarStep": {
            "Jar": "command-runner.jar",
            "Args": [
              "spark-submit --deploy-mode cluster s3://myfavouritebucketname86fda855-nhjwrxp55888/scripts/csv_to_parquet.py"
            ]
          }
        }
      },
      "Type": "Task",
      "Resource": "arn:aws:states:::elasticmapreduce:addStep.sync",
      "ResultPath": "$.guid"
    }
  },
  "TimeoutSeconds": 30
}

EMR 集群被设置为所有用户可见。 即使具有管理员权限,将 jobflowid 复制并粘贴到EMR Console上,也会返回以下错误:

Error

EMR.AmazonElasticMapReduceException

Cause

Specified job flow ID does not exist. (Service: AmazonElasticMapReduce; Status Code: 400; Error Code: ValidationException; Request ID: be70a470-ef6d-4356-98cd-0fc36d2cd132)

EMR IAM 角色定义如下:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "elasticmapreduce:AddJobFlowSteps",
                "elasticmapreduce:DescribeStep",
                "elasticmapreduce:CancelSteps"
            ],
            "Resource": "arn:aws:elasticmapreduce:us-east-1:accountid:cluster/*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "events:PutTargets",
                "events:PutRule",
                "events:DescribeRule"
            ],
            "Resource": "arn:aws:events:us-east-1:accountid:rule/StepFunctionsGetEventForEMRAddJobFlowStepsRule",
            "Effect": "Allow"
        }
    ]
}

通过CloudTrail Log查询API执行日志如下:

{
    "eventVersion": "1.05",
    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AROA36BAFPT62RGX2RMVQ:mYIsrbWoGnzXqWuTAXdcCriqSbTPXkdb",
        "arn": "arn:aws:sts::accountid:assumed-role/emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE/mYIsrbWoGnzXqWuTAXdcCriqSbTPXkdb",
        "accountId": "accountid",
        "accessKeyId": "ASIA36BAFPT6UG3XKKHI",
        "sessionContext": {
            "sessionIssuer": {
                "type": "Role",
                "principalId": "AROA36BAFPT62RGX2RMVQ",
                "arn": "arn:aws:iam::accountid:role/emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE",
                "accountId": "accountid",
                "userName": "emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE"
            },
            "webIdFederationData": {},
            "attributes": {
                "mfaAuthenticated": "false",
                "creationDate": "2020-06-08T23:29:29Z"
            }
        },
        "invokedBy": "states.amazonaws.com"
    },
    "eventTime": "2020-06-08T23:29:29Z",
    "eventSource": "elasticmapreduce.amazonaws.com",
    "eventName": "AddJobFlowSteps",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "states.amazonaws.com",
    "userAgent": "states.amazonaws.com",
    "errorCode": "ValidationException",
    "errorMessage": "Specified job flow ID does not exist.",
    "requestParameters": {
        "jobFlowId": "j-288AS5TZY5CY7",
        "steps": [
            {
                "name": "$$.Execution.Id",
                "actionOnFailure": "CONTINUE",
                "hadoopJarStep": {
                    "jar": "command-runner.jar",
                    "args": []
                }
            }
        ]
    },
    "responseElements": null,
    "requestID": "00c685e0-0d22-43ac-8fce-29bd808462bf",
    "eventID": "2a51a9ef-20a5-4fd7-bd33-be42cdf3088c",
    "eventType": "AwsApiCall",
    "recipientAccountId": "accountid"
}
profile picture
EXPERTE
gefragt vor einem Jahr37 Aufrufe
1 Antwort
0

【以下的回答经过翻译处理】 在Console上试试这个。

 "ClusterId.$": "j-INTERESTINGID",

另外,也请检查Step Function角色是否具有以下权限。

 {
            "Action": [
                "elasticmapreduce:List*",
                "elasticmapreduce:Describe*"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "elasticmapreduce:*"
            ],
            "Resource": [
                "arn:aws:elasticmapreduce:<region>:<account>:cluster/*"
            ],
            "Effect": "Allow"
        }
profile picture
EXPERTE
beantwortet vor einem Jahr

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen