Why has the DMS s3 Target Endpoint MaxFileSize been limited to 1GB

Question

My high-level Goal:
Download/Dump my Postgres database from RDS and restore it to a Postgres database on my local machine.

Current approach:
Get a dump of the schema, and restore the schema. Use DMS to copy db tables to S3, download the CSV files, and write a custom script to copy the CSV files to my local db tables.
If anyone has suggestions on a better approach please let me know!

My Issue:
When writing files to s3 using DMS, it seems once a table exceeds 1 GB it chunks it into 1GB CSV files, (in my case Gzips it to ~300MB files) and saves it. The issue I'm running into is when copying the data to my local db I have to load and combine all these files to achieve an efficient copy. Why was this value only set to 1GB surely in today's times this is too low?

**s3 Documentation:**
MaxFileSize
A value that specifies the maximum size (in KB) of any .csv file to be created while migrating to an S3 target during full load.

The default value is 1,048,576 KB (1 GB). Valid values include 1 to 1,048,576.

Type: Integer

Required: No

Answer

Hi,

You may follow a different path: load a local copy of the database directly via DMS Postgres to Postgres transfer. It doesn't have the instance you work on but a replica from RDS that you can then snapshot locally and restore as your work copy.

By doing so, you'll avoid the hop to S3, the download and reload steps (after concatenating the files as you mention). Is it applicable to your situation.

The 1GB limit is a hard one: you cannot change it.

Best,

Didier

Why has the DMS s3 Target Endpoint MaxFileSize been limited to 1GB

Relevant content