Redshift - Zero-ETL integration RDS Aurora - Resync Required

0

Hi, we have a Zero-ETL integration setup running but without us changing anything it have now started a Resync of some tables. That is fine but the problem is that it stops in the middle of the resync with an error about not enough memory. It should be set that we got it fine to work with the initial setup of the Zero-ETL. It has also been running for about 14 days and then it starts the resync that cannot run because of lack of memory. We have also after having the error the first time upgraded to having two nodes which increased the cost 2x and now we have the same problem with two nodes.

Replicating intitial data for table XXXX failed with error Insufficient memory to run query: consider increasing compute size or re-write the query (Out of Memory).

Any idea of how to solve this without deleting the Zero-ETL integration? (last time we had to delete it to get it to work) I just feel that we must be missing something as it should be able to run as a managed service without us having to redo the integration over again. When it goes into Failed status "Table not synced" then it does not start the resync again.


UPDATE: Have found this page (https://docs.aws.amazon.com/redshift/latest/mgmt/zero-etl-using.describing.html) where it states about refreshing all tables. It actually seems to have restarted a resync of all tables: "To refresh your integration, you can use the ALTER DATABASE command. Doing so replicates all of the data from your integration source into your destination database. The following example refreshes all synced and failed tables within your zero-ETL integration."

"ALTER DATABASE sample_integration_db INTEGRATION REFRESH ALL tables;"

Let see if the Zero-etl will work again after this. But we still do not know why this happens at it comes into this state.

New Update: The sql above actually worked and all tabels are now in sync :-)


Overview of tables from the zero-etl integration, some failed, Synced and Resync Initiated

ThomasP
asked 4 months ago308 views
1 Answer
0
Accepted Answer

The sql below worked and now all tables are in sync again :-)

Have found this page (https://docs.aws.amazon.com/redshift/latest/mgmt/zero-etl-using.describing.html) where it states about refreshing all tables. It actually seems to have restarted a resync of all tables: "To refresh your integration, you can use the ALTER DATABASE command. Doing so replicates all of the data from your integration source into your destination database. The following example refreshes all synced and failed tables within your zero-ETL integration."

"ALTER DATABASE sample_integration_db INTEGRATION REFRESH ALL tables;"

ThomasP
answered 4 months ago
profile picture
EXPERT
reviewed 10 hours ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions