How do I resolve the "failed to initialize logging driver: failed to create CloudWatch log stream status code: 400" error when I run an AWS Batch job?

Short description

This error means that you reached your Amazon CloudWatch service quota for API requests. When you reach this quota, your CreateLogStream API call is throttled.

You can receive this error when using any of the following AWS Cloud services to programmatically submit multiple jobs at the same time:

• AWS Command Line Interface (AWS CLI)
• AWS SDKs
• AWS Lambda
• AWS Step Functions

Resolution

Consider implementing one or more of the following solutions:

• Apply a retry strategy to your jobs using a job definition.
• Use retries and an exponential backoff algorithm with Jitter, and then resubmit your job.
• Avoid throttling by using AWS Batch array jobs to submit jobs with a single SubmitJob API call (up to 10,000 copies of a job).