Redshift - During COPY space usage reached to 99%


I would like to UNLOAD 250 millions records of one redshift table (100 GB) and COPY that table to different account. UNLOAD has created 350 GB of CSV files. During execution of COPY command on destination cluster it was about to use 100% space of cluster so had to terminate COPY at 99% of space usage. I have 130 GB of free space in destination cluster. Any suggestion for this or any other alternative for single table ?

asked 2 years ago212 views
2 Answers

You may want to consider unloading file in a different format like Parquet which takes significantly less space on S3. If source table is taking only 100GB, you can try creating destination table first using same ddl as source table to take advantage of compression or make sure option COMPUDATE [ON] when you copy the data. when you have this option on best column compression is determined & set by applying different compression codecs on sample set of column data. Also Copy performance will be lot better when you have multiple files (based on #of slices).

answered 2 years ago

There are some factors to reduce used segments(blocks) of a cluster. First sort keys make additional temporary segments. So test it with tables which has no sort key. And check the encoding(compression) of target table columns. And load from split files not just one file with compressed format (such like .gz)

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions