Bitbucket webhooks for AWS CodeBuild throttled (rate limited)

0

Hello,

We have over 100 Bitbucket webhooks, each is linked to a separate CodeBuild project. Occasionally, we have to push code to a "global" directory, which should trigger the build for around 50 CodeBuild projects simultaneously. We noticed that out of the 50 that should be triggered, only 20-30 work (random number). We checked the Bitbucket event history for our webhooks and it looks like some of them are throttled.

Enter image description here

There are always 3 attempts, each returns status code 400, x-amzn-ErrorType header: ThrottlingException:http://internal.amazon.com/coral/com.amazon.coral.availability/ exception and a message body containing: {"message":"Rate exceeded"}

Each code build project is using Linux BUILD_GENERAL1_SMALL, the applied quota value for Concurrently running builds for Linux/Small environment is 300, so this should not have been exceeded.

Is there any quota that might have been exceeded by simultaneous webhook requests?

What can we do to solve that issue?

Thank you in advance for any hints.

radek
질문됨 일 년 전299회 조회
1개 답변
2
수락된 답변

The issue you're encountering is related to the rate at which AWS CodeBuild accepts incoming build requests. While you might not be exceeding the concurrent build limit, AWS CodeBuild imposes a limit on the number of build requests that can be made per second.

ThrottlingException errors typically occur when the number of requests exceeds the AWS CodeBuild API request rate limits.

AWS does not publicly disclose the exact rate limit, but it's important to know that it exists and is separate from the limit on concurrently running builds. This rate limit applies to the number of requests made to start builds, not the number of builds that are running.

To resolve this issue, you can implement a few strategies:

  1. Retry logic: Implement retry logic in your webhook handler to handle throttling exceptions. After receiving a ThrottlingException, wait for a few seconds and then retry the request.

  2. Queueing system: Use a queueing system like Amazon SQS to handle bursts of build requests. Your webhook handler can push messages to an SQS queue, and a separate worker can pull messages from the queue and start builds at a controlled rate.

  3. Request limit increase: You can also contact AWS Support to request an increase in your API rate limit.

Remember, these strategies can be used in combination. For example, you could use an SQS queue to handle bursts of requests, and implement retry logic in your worker to handle any throttling exceptions that still occur.

I hope this helps!

profile picture
답변함 일 년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠