How to pass parameters from EventBridge Pipe to Batch Job?


I set up an eventbridge pipe that reads from an SQS queue and should trigger a batch job. The problem is that the pipe is not passing the parameters to the batch job. Here are the two logs: The first shows that the pipe got the parameters succesfully from the sqs queue. The second shows that the request to the batch job submit does NOT have those parameters. I've tried all possible permutations of the Target Input Transformer, but still had no success.

LOG 1 { "resourceArn": "arn:aws:pipes:eu-west-2:355280565032:pipe/video_2", "timestamp": 1708684686652, "executionId": "39f111cc-590d-4c14-be5f-4a4e916c072b", "messageType": "TargetTransformationSucceeded", "logLevel": "TRACE", "payload": "[{"Parameters":{"recording_id":"15","recording_url":"la luna nera 2"}}]" }

LOG 2 { "resourceArn": "arn:aws:pipes:eu-west-2:355280565032:pipe/video_2", "timestamp": 1708684686806, "executionId": "39f111cc-590d-4c14-be5f-4a4e916c072b", "messageType": "TargetInvocationFailed", "logLevel": "ERROR", "error": { "message": "Target invocation failed with error from Batch.", "httpStatusCode": 400, "awsService": "batch", "requestId": "6d2e015d-8675-4b0b-9ce6-05faf45a0665", "exceptionType": "BadRequest", "resourceArn": "arn:aws:batch:eu-west-2:355280565032:job-queue/video_transcription_job_queue" }, "awsRequest": "{"jobName":"video_job","jobQueue":"arn:aws:batch:eu-west-2:355280565032:job-queue/video_transcription_job_queue","shareIdentifier":null,"schedulingPriorityOverride":null,"arrayProperties":null,"dependsOn":null,"jobDefinition":"arn:aws:batch:eu-west-2:355280565032:job-definition/video_transcription_job:6","parameters":null,"containerOverrides":null,"nodeOverrides":null,"retryStrategy":null,"propagateTags":null,"timeout":null,"tags":null,"eksPropertiesOverride":null}", "awsResponse": "Unable to substitute value. No parameter found for reference recording_id (Service: Batch, Status Code: 400, Request ID: 6d2e015d-8675-4b0b-9ce6-05faf45a0665)" }

EDIT: Added a screenshot of the pipe I created Enter image description here Enter image description here

2 Answers
Accepted Answer

I'm updating this post for the community. The solution that I found was to use the instructions provided in this post :

In summary: it is not possible to set optional BatchParameters using the GUI when creating a Pipe in EventBridge. BUT you can create a pipe with those optional parameters using the CLI command or by updating an existing pipe (always with the CLI commands).

So my solution was:

  1. First I got a json of the most important params of my pipe.

aws pipes describe-pipe --name MY_PIPE_NAME --query '{Name: Name, RoleArn: RoleArn, TargetParameters: TargetParameters}'

  1. The above command prints the json information to your terminal, so you can copy it and save it to a file, let's say the file name is update.json
  2. Next you can open the json file and add the optional parameters you want to pass to your batch job, like this:
    "Name": "video_3",
    "RoleArn": "arn:aws:iam::*:role/service-role/Amazon_EventBridge_Pipe_video_3_a6617541",
    "TargetParameters": {
        "BatchJobParameters": {
            "JobDefinition": "arn:aws:batch:eu-west-2:*:job-definition/video_transcription_job:6",
            "JobName": "video_job",
            "Parameters": {"recording_id":"$.body.recording_id", "recording_url": "$.body.recording_url"}

Notice that I put the dynamic json params in here. I expect to find recording_id and recording_url in the body of the message of my sqs queue that triggers this pipe.

  1. Once you update the json like this, you can use the update command to update your pipe:

aws pipes update-pipe --cli-input-json file://update.json

Notice that you need this "file://" before the name of your json file to load it.

  1. DONE! Now if you go the GUI and check your pipe target definition, you will find the optional parameters field you specified in the json file.

Enter image description here

answered 2 months ago
profile picture
reviewed a month ago

Hello Jacek.

Try to use Input Transformation to ensure parameter delivery.

Amazon EventBridge Pipes input transformation


 aws events put-targets --rule RuleName \
    --targets "Id"="1",
                  'InputPathsMap': {
                  'Template': '{\"jobName\": \"NomeDoJob\", \"jobQueue\": \"NomeDaFila\", \"jobDefinition\": \"NomeDefiniçãoJob\", \"containerOverrides\": {\"command\": [\"<comando ou script a ser executado>\", \"--recordingId\", <recordingId>, \"--recordingUrl\", \"<recordingUrl>\"]}}'
profile picture
answered 2 months ago
  • Thank you for answering. I added a screenshot of the pipe I created and the batch job container details. I'm using the UI to create the pipe. I didn't create any Rule for this task, since I want to read a message from a sqs queue, extract the two parameters from the message and pass them to the batch Job. Any idea why it is not working?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions