1 Answer
- Newest
- Most votes
- Most comments
0
I think, the issue seems due to the way AWS Glue handles concurrent runs of the same job. When you run the same job multiple times with different input data, AWS Glue will reuse the same executors and resources from the previous run, which can lead to memory issues. You may consider 1) Increase the worker memory, 2) Try Glue's built-in data frame capabilities , 3) Try Glue's data catalog partitioning, 4) Try Glue's bounded execution. You may try troubleshooting using this reference - https://docs.aws.amazon.com/glue/latest/dg/monitor-profile-debug-oom-abnormalities.html
answered 4 months ago
Relevant content
- Accepted Answerasked 2 years ago
- Accepted Answerasked 6 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 3 years ago