Cloudformation template to export the dataset into S3 Location

0

Hi All,

Created a Cloudformation template, which exports the subscribed dataset latest revision to S3 bucket. Lambda job looks for the latest revision and uses api 'ExportAssetsToS3' to write the data into s3 bucket. I am trying to achieve, lambda job to write the data to a location with prefix like - bucket/prefix/files .

Prefix is like current date and it should be changed for everyday.

In the ExportAssetsToS3, in AssetDestinations passed Key : '/2020-08-06/ and error came as below.

[ERROR] ValidationException: An error occurred (ValidationException) when calling the CreateJob operation: The request was rejected because it attempted to export distinct assets to the same target Amazon S3 object. When no key is specified in the request, Asset name is used as the default target key.
Traceback (most recent call last):
File "/var/task/index.py", line 69, in handler
{ 'AssetId': asset['Id'], 'Bucket': destination_bucket, 'Key': '/2020-08-06/' } for asset in assets_chunk
File "/var/runtime/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 626, in _make_api_call
raise error_class(parsed_response, operation_name)

Ex: Bucket - ABC
Prefix: /2020-08-06/
Final S3 location - ABC/2020-08-06/file.txt

Is there any approach to add the prefix to the S3 bucket.

Thanks,
Naveen J

Hi All,

Here is the Solution -

Change the code in export_job, AssetDestinations as below

'AssetDestinations': [
{ 'AssetId': asset['Id'], 'Bucket': destination_bucket, 'Key': '
abc/2020-08-06/' + asset['Name'] } for asset in assets_chunk ]

Thanks,
Naveen J

Edited by: naveen838 on Aug 7, 2020 9:33 AM

Edited by: naveen838 on Aug 7, 2020 9:33 AM

asked 4 years ago396 views
1 Answer
0

Hi naveen838, apologies for the late answer. Does the revision you are trying to export have multiple assets with the same name? If multiple assets have the same name, the export fails due to conflict of the target S3 object key.

Edited by: akram-AWS on Jan 7, 2021 3:09 PM

AWS
answered 4 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions