Data Pipeline stops processing files in S3 bucket

0

I have a Data Pipeline which reads CSV files from an S3 bucket and copies the data into an RDS database. I specify the bucket/folder name and it goes through each CSV file in the bucket/folder and processes it. When it is done, a ShellCommandActivity moves the files to another 'folder' in the S3 bucket. That's how it works in testing. With the real data it just stops after a few files.

The last line in the logs is

07 Dec 2021 09:57:55,755 [INFO] (TaskRunnerService-resource:df-1234xxx1_@Ec2Instance_2021-12-07T09:53:00-0) df-1234xxx1 amazonaws.datapipeline.connector.s3.RetryableS3Reader: Reopening connection and advancing 0

The logs show that it usually downloads the CSV file, then it writes the 'Reopening connection and advancing 0' line, then it deletes a temp file, then goes onto the the next file. But on the seventh file it just stops on 'Reopening connection and advancing 0'.

It isn't the next file that is the problem, as it will process fine on it's own. I've already tried making the files smaller - originally it was stopping on the second file but now the filesizes are about 1.7MB it's getting through six of them before it stops.

The status of each task (both DataLoadActivity and ShellCommandActivity) show 'CANCELLED' after one attempt (3 attempts are allowed) and there is no error message.

I'm guessing this is some sort of timeout. How can I make the pipeline reliable so that it will process all of the files?

2 Risposte
2
Risposta accettata

The place to start checking is the ShellCommmandActivity node - has this got an Attempt Timeout field? This would cause the node to fail after a given time. You can also look at other nodes in the pipeline (such as EC2Resource nodes) as these can have timeouts as well.

AWS
Nick
con risposta 3 anni fa
AWS
ESPERTO
verificato 3 anni fa
  • Aha! The Ec2 Resource is set to Terminate after 10 minutes! I will try changing that.

  • That was it. It's still running and it'll take a few more hours to complete. Thanks for pointing me in the right direction.

0

While I think you have found your answer , you could also try to look at AWS Glue to process your data from S3 into RDS, it is a serverless service and has a bookmark concept that will allow you to process each file only once without the need to relocate them (if you are not doing so for other reasons).

AWS
ESPERTO
con risposta 3 anni fa
  • Thanks for the tip! I will check out AWS Glue.

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande