Error Received response status [FAILED] from custom resource. Message returned: Configuration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping

0

my task use case is

  1. Create sqs queue to process excels uploads + dlq
  2. Create producer and consumer lambda functions
  3. Add s3 upload event trigger to producer lambda
  4. Add sqs worker lambda to consumer lambda

for that i have followed below link

s3event triiger tutorial

don’t create anything manually from the aws console, every feature must be go through serverless framework Have a look at the existing producer and lambda workflows in the serverless repository, it has all these features except s3 event triggers

for that i created aws c# template project and write cloudformation template(serverless.yml) as below when i deploy project using sls deploy --stage dev it throws

Received response status [FAILED] from custom resource. Message returned: Configuration is ambiguously defined. Cannot have overlapping suffixes in two rules if the prefixes are overlapping for the same event type

Error occur if i add dealPriceExcelUploadProducerTrigger function there is no any issues on creating resources

  1. please help me to sort out this issue
  2. how can i read queue URL and db context connection string values to DealPriceExcelUploadProducer.cs from appsetting or env using c#
service: deal-price-excel-upload
frameworkVersion: '3'

custom:
  region: eu-west-2
  bucketName: ag-staging-buckets
  # region: ${env:AWS_DEFAULT_REGION} TODO
  
provider:
  name: aws
  runtime: dotnet6
  stage: ${sls:stage}
  region: ${self:custom.region}
        
package:
  individually: true
  patterns:
    - '**/*'

functions:
  dealPriceExcelUploadProducerTrigger:
    handler: src/Handler/DealPrices/DealPriceExcelUploadProducer.Handler
    events:
      - s3:
          bucket: ${self:custom.bucketName}
          event: s3:ObjectCreated:*
          rules:
            - prefix: uploads/deals/deal-price-excel
          existing: true


resources:
  Resources:
    DealPriceExcelConsumerQueue:
      Type: AWS::SQS::Queue
      Properties:
        QueueName: DealPriceExcelConsumerQueue
        RedrivePolicy:
          deadLetterTargetArn: !GetAtt DealPriceExcelConsumerDLQ.Arn
          maxReceiveCount: 5
      UpdateReplacePolicy: Snapshot
    
    DealPriceExcelProducerDLQ:
      Type: AWS::SQS::Queue
      Properties:
        QueueName: DealPriceExcelProducerDLQ
      UpdateReplacePolicy: Snapshot

    DealPriceExcelConsumerDLQ:
      Type: AWS::SQS::Queue
      Properties:
        QueueName: DealPriceExcelConsumerDLQ
      UpdateReplacePolicy: Snapshot
    
    DealPriceExcelConsumerQueuePolicy:
      Type: AWS::SQS::QueuePolicy
      Properties:
        Queues:
          - !Ref DealPriceExcelConsumerQueue
          - !Ref DealPriceExcelConsumerDLQ
          - !Ref DealPriceExcelProducerDLQ
        PolicyDocument:
          Statement:
            - Effect: Allow
              Action: 
                # - 'sqs:*'
                - 'sqs:DeleteMessage'
                - 'sqs:GetQueueAttributes'
                - 'sqs:ReceiveMessage'
                - 'sqs:SendMessage'
                - 'logs:CreateLogGroup'
                - 'logs:CreateLogStream'
                - 'logs:PutLogEvents'
              Resource: 'arn:aws:sqs:${self:custom.region}:${AWS::AccountId}:DealPriceExcelConsumerQueue'
            - Effect: Allow
              Action: 
                # - 'sqs:*'
                - 'sqs:DeleteMessage'
                - 'sqs:GetQueueAttributes'
                - 'sqs:ReceiveMessage'
                - 'sqs:SendMessage'
                - 'logs:CreateLogGroup'
                - 'logs:CreateLogStream'
                - 'logs:PutLogEvents'
              Resource: 'arn:aws:sqs:${self:custom.region}:${AWS::AccountId}:DealPriceExcelUploadConsumersDlq'
            - Effect: Allow
              Action: 
                # - 'sqs:*'
                - 'sqs:DeleteMessage'
                - 'sqs:GetQueueAttributes'
                - 'sqs:ReceiveMessage'
                - 'sqs:SendMessage'
                - 'logs:CreateLogGroup'
                - 'logs:CreateLogStream'
                - 'logs:PutLogEvents'
              Resource: 'arn:aws:sqs:${self:custom.region}:${AWS::AccountId}:DealPriceExcelProducerDLQ'
    
    DealPriceExcelEventTriggerPolicy:
      Type: AWS::IAM::Policy
      Properties:
        PolicyName: DealPriceExcelEventTriggerPolicy
        PolicyDocument:
          Statement:
            - Effect: Allow
              Action:
                - logs:PutLogEvents
                - logs:CreateLogGroup
                - logs:CreateLogStream
              Resource: 'arn:aws:logs:*:*:*'
            - Effect: Allow
              Action:
                - s3:GetObject
              Resource: 'arn:aws:s3:::${self:custom.bucketName}/*'
        Roles: 
          - !Ref DealPriceExcelExecutionRole

    DealPriceExcelExecutionRole:
      Type: AWS::IAM::Role
      Properties:
        AssumeRolePolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Effect: Allow
              Principal:
                Service:
                  - lambda.amazonaws.com
                  - sqs.amazonaws.com
                  - s3.amazonaws.com
              Action:
                - sts:AssumeRole
        Policies:
          - PolicyName: DealPriceExcelEventTriggerPolicy
            PolicyDocument:
              Version: "2012-10-17"
              Statement:
                - Effect: Allow
                  Action:
                    - logs:PutLogEvents
                    - logs:CreateLogGroup
                    - logs:CreateLogStream
                  Resource: 'arn:aws:logs:*:*:*'
                - Effect: Allow
                  Action:
                    - s3:GetObject
                  Resource: 'arn:aws:s3:::${self:custom.bucketName}/*'
          - PolicyName: SQSPolicy
            PolicyDocument:
              Version: "2012-10-17"
              Statement:
                - Effect: Allow
                  Action:
                  - lambda:CreateEventSourceMapping
                  - lambda:ListEventSourceMappings
                  - lambda:ListFunctions
                  - sqs:DeleteMessage
                  - sqs:GetQueueAttributes
                  - sqs:ReceiveMessage
                  - sqs:SendMessage
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                  Resource: "*"

    DealPriceExcelUploadProducerFunction:
      Type: AWS::Lambda::Function
      Properties: 
        Role: !GetAtt DealPriceExcelExecutionRole.Arn
        Runtime:  nodejs18.x
        FunctionName: DealPriceExcelUploadProducerFunction
        Handler:  index.handler
        DeadLetterConfig:
          TargetArn: !GetAtt DealPriceExcelProducerDLQ.Arn
        Environment:
          Variables:
            producerQueueUrl: !Ref DealPriceExcelConsumerQueue
            producerDlQueueUrl: !Ref DealPriceExcelProducerDLQ
        Code:
          ZipFile: |
            import { S3Client } from "@aws-sdk/client-s3";
            import { SQS } from "@aws-sdk/client-sqs";
            const region = process.env.AWS_REGION ??"eu-west-2";
            const s3 = new S3Client({ region: region });
            const sqs = new SQS({ apiVersion: "2012-11-05",region:region });
            const producerQueueUrl = process.env.producerQueueUrl;
            const producerDlQueueUrl = process.env.producerDlQueueUrl;

            export const handler = async (event): Promise<string | undefined> => {
              for (const record of event.Records) {
                let retryCount = 0;
                const maxRetries = 5;
                while (retryCount <= maxRetries) {
                  try {
                  await  ProcessEvent(record,producerQueueUrl);
                    retryCount = 0;
                    return event;
                  } catch (err) {
                    if (retryCount == maxRetries) {

                        AddToDlq(record);
                      return event;
                    }
                    retryCount++;
                  }
                }
              };
              return  event;
            };

            const ProcessEvent = async (
              record,
              queueUrl:string
            ) => {
              const bucket = record.s3.bucket.name;
              const key = decodeURIComponent(record.s3.object.key.replace(/\+/g, " "));
              const payload = {
                Bucket: bucket,
                Key: key,
              };

              const messageBody = JSON.stringify(payload);
              await sqs.sendMessage({
                MessageBody: messageBody,
                QueueUrl:   queueUrl
              });
            };

            const AddToDlq= async( record)=>{
              await  ProcessEvent(record, producerDlQueueUrl);
            };
    
 

  Outputs:
    DealPriceExcelConsumerQueueURL:
      Description: URL of DealPriceExcelConsumer Queue
      Value: !GetAtt DealPriceExcelConsumerQueue.QueueUrl

    DealPriceExcelConsumerQueueARN:
      Description: ARN of DealPriceExcelConsumer Queue
      Value: !GetAtt DealPriceExcelConsumerQueue.Arn

    DealPriceExcelUploadConsumerDlqURL:
      Description: URL of dead-letter queue
      Value: !Ref DealPriceExcelConsumerDLQ

    DealPriceExcelUploadConsumerDlqARN:
      Description: ARN of DealPriceExcelUploadConsumer Dlq
      Value: !GetAtt DealPriceExcelConsumerDLQ.Arn

Handler function is

using Amazon.Lambda.S3Events;
using TechneTravel.src.Entity.Dto;
using Amazon.SQS;
using Amazon.SQS.Model;
using Newtonsoft.Json;
using Amazon.Lambda.SQSEvents;


namespace TechneTravel.src.Handler.DealPrices;

public class DealPriceExcelUploadProducer
{
    private readonly AmazonSQSClient _sqsClient;

    public DealPriceExcelUploadProducer(AmazonSQSClient sqsClient)
    {
        _sqsClient = sqsClient;

    }

    public async Task Handler(S3Event evt, ILambdaContext context)
    {
        await Task.WhenAll(
         evt.Records.Select(async record =>
         {
             await Task.Run(async () =>
             {
                 int retryCount = 0;
                 int maxRetries = 5;

                 while (retryCount <= maxRetries)
                 {
                     try
                     {
                         //  TODO add suitable env access
                         // DealPriceExcelProducerqueue
                         var queue = $"DEAL_PRICE_EXCEL_QUEUE_URL";
                         await ProcessEvent(record, queue, context);
                         retryCount = 0;
                         return;
                     }
                     catch (Exception exception)
                     {
                         context.Logger.LogLine($"Exception thrown in deal price ProcessExcelFile {exception.Message}");
                         if (retryCount == maxRetries)
                         {
                             var dlqueue = $"DEAL_PRICE_EXCEL_QUEUE_URL";

                             await AddToDlq(record, dlqueue, context);
                             return;
                         }
                         retryCount++;
                     }
                 }
             });
         })
     );
    }


    async Task ProcessEvent(S3Event.S3EventNotificationRecord record, string queueUrl, ILambdaContext context)
    {
        var bucketName = record.S3.Bucket.Name;
        var objectKey = record.S3.Object.Key;
        var payload = new DealPriceSqsQueuePayload
        {
            BucketName = bucketName,
            Key = objectKey
        };

        var messageBody = JsonConvert.SerializeObject(payload);
        LogInformation(context, $"{messageBody}");
        await SendMessage(queueUrl, messageBody, context);
    }

    private async Task SendMessage(
      string qUrl, string messageBody, ILambdaContext context)
    {
        SendMessageResponse responseSendMsg =
          await _sqsClient.SendMessageAsync(qUrl, messageBody);
        LogInformation(context, $"Message added to queue\n  {qUrl}");
        LogInformation(context, $"HttpStatusCode: {responseSendMsg.HttpStatusCode}");
    }

    private async Task AddToDlq(S3Event.S3EventNotificationRecord record, string queueUrl, ILambdaContext context)
    {
        await ProcessEvent(record, queueUrl, context);
    }

    public async Task HandleDealPriceExcelProducerDLQ(SQSEvent evt, ILambdaContext context)
    {
        await Task.WhenAll(
         evt.Records.Select(async record =>
         {

             int retryCount = 0;
             int maxRetries = 5;

             while (retryCount <= maxRetries)
             {
                 try
                 {
                     //  TODO add suitable env access
                     // DealPriceExcelProducerqueue
                     var queue = $"DEAL_PRICE_EXCEL_PRODUCERDlQ_URL";
                     await ProcessDlqEvent(record, queue, context);
                     await DeleteMessage(queue, record.ReceiptHandle, context);
                     retryCount = 0;
                     return;
                 }
                 catch (Exception exception)
                 {
                     context.Logger.LogLine($"Exception thrown in deal price ProcessExcelFile {exception.Message}");
                     if (retryCount == maxRetries)
                     {
                         await HandleDealPriceExcelProducerDLQ(evt, context);
                         return;
                     }
                     retryCount++;
                 }
             }


         })
     );

    }

    async Task ProcessDlqEvent(SQSEvent.SQSMessage message, string queueUrl, ILambdaContext context)
    {
        LogInformation(context, $"Processed dealPriceExcelUploadProduce dlq message {message.Body}");

        DealPriceSqsQueuePayload payload = JsonConvert.DeserializeObject<DealPriceSqsQueuePayload>(message.Body);

        var messageBody = JsonConvert.SerializeObject(payload);
        LogInformation(context, $"{messageBody}");
        await SendMessage(queueUrl, messageBody, context);
    }

    private async Task DeleteMessage(
      string qUrl, string receiptHandle, ILambdaContext context)
    {
        var responseSendMsg =
          await _sqsClient.DeleteMessageAsync(qUrl, receiptHandle, default);
        LogInformation(context, $"Message deleted to queue\n  {qUrl} {receiptHandle}");
        LogInformation(context, $"HttpStatusCode: {responseSendMsg.HttpStatusCode}");
    }

    private void LogInformation(ILambdaContext context, string message)
    {
        context.Logger.LogInformation($"{message}");
    }

}

1 Resposta
0

Hello,

I understand that you are observing "Error Received response status [FAILED] from custom resource. Message returned: Configuration is ambiguously defined." error whilde delpoying the cloudformation template.

The error can occur when you're doing one of the following:

  • Recreating an S3 event notification that you recently deleted.
  • Creating S3 event notifications for multiple overlapping events using overlapping prefixes or suffixes.

The detailed steps to resolve this issue are shared in the well-curated document here: [+] https://repost.aws/knowledge-center/lambda-s3-event-configuration-error

Additionally, I also understand that you have queries regarding reading queue url and db connection string using C#.

Please find some third party articles for your reference here: [+] https://docs.aws.amazon.com/sdkfornet/latest/apidocs/items/MSQSSQSGetQueueUrlGetQueueUrlRequestNET45.html [+] https://stackoverflow.com/questions/68760921/retrieve-the-sqs-queue-url-dynamically-from-the-lambda-event-or-context [+] https://stackoverflow.com/questions/15631602/how-can-i-set-an-sql-server-connection-string

If you require additional assistance , I would recommend you to reach out to AWS Premium Support by raising a Support Case using the same account with all the information.

AWS
Vastav
respondido há um mês

Você não está conectado. Fazer login para postar uma resposta.

Uma boa resposta responde claramente à pergunta, dá feedback construtivo e incentiva o crescimento profissional de quem perguntou.

Diretrizes para responder a perguntas