【以下的问题经过翻译处理】 我创建了EMR集群,然后通过Step Function向 EMR 集群添加一个步骤。
发现jobflowid 未被识别。
请问该如何解决这个问题?或者是我的操作过程有任何疏忽的吗?
Step Function状态机如下:
{
"StartAt": "add_emr_step",
"States": {
"add_emr_step": {
"End": true,
"InputPath": "$",
"Parameters": {
"ClusterId": "j-INTERESTINGID",
"Step": {
"Name": "$$.Execution.Id",
"ActionOnFailure": "CONTINUE",
"HadoopJarStep": {
"Jar": "command-runner.jar",
"Args": [
"spark-submit --deploy-mode cluster s3://myfavouritebucketname86fda855-nhjwrxp55888/scripts/csv_to_parquet.py"
]
}
}
},
"Type": "Task",
"Resource": "arn:aws:states:::elasticmapreduce:addStep.sync",
"ResultPath": "$.guid"
}
},
"TimeoutSeconds": 30
}
EMR 集群被设置为所有用户可见。
即使具有管理员权限,将 jobflowid 复制并粘贴到EMR Console上,也会返回以下错误:
Error
EMR.AmazonElasticMapReduceException
Cause
Specified job flow ID does not exist. (Service: AmazonElasticMapReduce; Status Code: 400; Error Code: ValidationException; Request ID: be70a470-ef6d-4356-98cd-0fc36d2cd132)
EMR IAM 角色定义如下:
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"elasticmapreduce:AddJobFlowSteps",
"elasticmapreduce:DescribeStep",
"elasticmapreduce:CancelSteps"
],
"Resource": "arn:aws:elasticmapreduce:us-east-1:accountid:cluster/*",
"Effect": "Allow"
},
{
"Action": [
"events:PutTargets",
"events:PutRule",
"events:DescribeRule"
],
"Resource": "arn:aws:events:us-east-1:accountid:rule/StepFunctionsGetEventForEMRAddJobFlowStepsRule",
"Effect": "Allow"
}
]
}
通过CloudTrail Log查询API执行日志如下:
{
"eventVersion": "1.05",
"userIdentity": {
"type": "AssumedRole",
"principalId": "AROA36BAFPT62RGX2RMVQ:mYIsrbWoGnzXqWuTAXdcCriqSbTPXkdb",
"arn": "arn:aws:sts::accountid:assumed-role/emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE/mYIsrbWoGnzXqWuTAXdcCriqSbTPXkdb",
"accountId": "accountid",
"accessKeyId": "ASIA36BAFPT6UG3XKKHI",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "AROA36BAFPT62RGX2RMVQ",
"arn": "arn:aws:iam::accountid:role/emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE",
"accountId": "accountid",
"userName": "emr-step-statematchineRoleC75C4884-1FDZWJ1GBIVUE"
},
"webIdFederationData": {},
"attributes": {
"mfaAuthenticated": "false",
"creationDate": "2020-06-08T23:29:29Z"
}
},
"invokedBy": "states.amazonaws.com"
},
"eventTime": "2020-06-08T23:29:29Z",
"eventSource": "elasticmapreduce.amazonaws.com",
"eventName": "AddJobFlowSteps",
"awsRegion": "us-east-1",
"sourceIPAddress": "states.amazonaws.com",
"userAgent": "states.amazonaws.com",
"errorCode": "ValidationException",
"errorMessage": "Specified job flow ID does not exist.",
"requestParameters": {
"jobFlowId": "j-288AS5TZY5CY7",
"steps": [
{
"name": "$$.Execution.Id",
"actionOnFailure": "CONTINUE",
"hadoopJarStep": {
"jar": "command-runner.jar",
"args": []
}
}
]
},
"responseElements": null,
"requestID": "00c685e0-0d22-43ac-8fce-29bd808462bf",
"eventID": "2a51a9ef-20a5-4fd7-bd33-be42cdf3088c",
"eventType": "AwsApiCall",
"recipientAccountId": "accountid"
}