AWS DMS connected successfully to Redshift created table but not loading data. Connected to S3 but failing at Uploading files to redshift.

Question

Any help please? Logs is as below:

: Failed to upload file '/rdsdbdata/data/tasks/KXHGVEZM7WRYI636BOJOQYTCOIGFGS6AOGE3MNY/cloud/1/LOAD00000001.gzip' to bucket 'dms-in2tox7r7fgrwfjrshecwnp6s4y2uvsre63afzy' as 'redshift-FGKQA6KH2TIZ5J6G4T3NMVU7REZGRBNUYSGQI3Y/KXHGVEZM7WRYI636BOJOQYTCOIGFGS6AOGE3MNY/1/LOAD00000001.csv', status = 4 (FAILED) [1001705] (transfer_client.cpp:514)

2022-11-17T14:59:38.000+00:00 2022-11-17T14:59:38 [FILE_FACTORY ]E: Failed to upload  to  [1000722] (at_s3_ff.c:429)

2022-11-17T14:59:38.000+00:00 2022-11-17T14:59:38 [FILE_FACTORY ]E: Failed to write entire file (second trial) [1000722] (at_universal_fs_object.c:631)

2022-11-17T14:59:38.000+00:00 2022-11-17T14:59:38 [FILE_FACTORY ]E: Write entire file failed: source = '/rdsdbdata/data/tasks/KXHGVEZM7WRYI636BOJOQYTCOIGFGS6AOGE3MNY/cloud/1/LOAD00000001.gzip' target = 'dms-in2tox7r7fgrwfjrshecwnp6s4y2uvsre63afzy/redshift-FGKQA6KH2TIZ5J6G4T3NMVU7REZGRBNUYSGQI3Y/KXHGVEZM7WRYI636BOJOQYTCOIGFGS6AOGE3MNY/1/LOAD00000001.csv' open type = 3 [1000731] (at_universal_fs_object.c:316

Answer

Hi,

Have you checked that the used IAM Role has the S3 Get Object permission?

The last section found in this link could be helpful https://aws.amazon.com/premiumsupport/knowledge-center/dms-task-successful-no-data-s3/

Thanks,

Answer

Same issue here, and yes, IAM role has the S3 Get Object Permission.
Going from RDS to Redshift, tables are created, but no content + the error:

```
Failed to upload file '/rdsdbdata/data/tasks/XXXX/cloud/81/LOAD00000001.csv.gz' to bucket 'dms-XXXX' as 'redshift-XXXX/XXXX/81/LOAD00000001.csv', status = 4 (FAILED) [1001705]  (transfer_client.cpp:514)
```

Edit: After more trial and error than I would like to admit:
This pointed me into the correct direction -> https://docs.aws.amazon.com/dms/latest/userguide/CHAP_VPC_Endpoints.html

For me, the selected routing table for the VPC endpoint did not include any table with a connected internet gateway, which caused the "Failed to upload file" error because there was obviously no route to the S3 bucket…

Here are some general tips, if you end up in a similar situation:

- Make sure you check your VPC endpoint configuration!
 - Make sure the related roles have all the required permissions (dms-access-for-endpoint, dms-access-for-tasks and dms-vpc-role)
 - Check the following cases: https://repost.aws/knowledge-center/dms-task-successful-no-data-s3
 - You can turn on the S3 access logs.(If you turn them on and no logs appear, it is probably a network configuration problem)

Hope this helps, good luck!

AWS DMS connected successfully to Redshift created table but not loading data. Connected to S3 but failing at Uploading files to redshift.

관련 콘텐츠