I am executing pyspark script on EMR serverless and i am using AWS step function for creating EMR application and submitting job in it. I want to know how to add logging option in AWS step function configuration while creating EMR application or submitting job in it so that i can see log files from spark job run in cloudwatch or in s3. this is my current block from StartJobRun state from AWS step function configuration.
"EMR Serverless StartJobRun":
{ "Type": "Task", "Resource": "arn:aws:states:::emr-serverless:startJobRun", "Parameters":
{ "ApplicationId.$": "$.ApplicationId",
"ExecutionRoleArn": "arn:aws:iam:/demo-emrserverless-iam-role",
"JobDriver":
{ "SparkSubmit":
{ "EntryPoint": "s3://demo/test.py",
"EntryPointArguments.$":[],
"SparkSubmitParameters": "--conf } },
i tried to add "JobConfiguration" but it is not accepting JobConfiguration