Is there a reliable way to know that an aws_s3.query_export completed successfully?

0

I'm working on a feature to archive old data from our Aurora Postgres database to S3 using the aws_s3 extension. This operation takes 20-30 minutes, and sometimes my client gets disconnected and retries. It appears that, even if my client gets disconnected, the aws_s3 extension continues in the background, so on a retry, I end up transferring the entire amount of data twice, and I would prefer if there is a reliable way to query the S3 objects and know they are complete copy of the data in the query. The table partitions I am archiving are 15-20Gb each, and I notice that the S3 objects appear to be chunked into 6Gb chunks, so it seems that the existence of an S3 object following the right naming convention that is significantly smaller than 6Gb would imply that an earlier operation completed, but it's hard to be certain.

gefragt vor 6 Monaten787 Aufrufe
1 Antwort
0
Akzeptierte Antwort

Realized that the S3 SelectObjectContent API allows us to count the rows in objects written as CSV. I think it would be awesome if the aws_s3 extension could write some object metadata so we could get this data without reading the full objects.

beantwortet vor 6 Monaten

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen