What to do after my training job fails with "InternalServerError"?

I have a training job with resnet-50 on a 50GB/ml.p2.xlarge (1 instance), Pipe mode, object detection model with 1073 images in training and 268 images in validation. According to CloudWatch, it ran up to epoch 77 (for about 4 hours) but then failed with no specific message recorded in CloudWatch. I only get the dreaded "InternalServerError: We encountered an internal error. Please try again." which is not okay because it costs money and I need to know what is failing.

The CPU utilization is stable at around 270% (a number that needs to be divided by the number of vCPUs which is 4 so really this is about 68% per vCPU), GPU Utilization is constant at under 60%, GPU Memory Utilization is constant at around 18%, Memory Utilization is constant at 3.2%, Disk Utilization is stable at 0.22%.

Is there an obvious mistake I am doing? Thanks for the help!

Themen

Maschinelles Lernen & KI

Relevanter Inhalt

Wie behebe ich den Fehler „The specified queue does not exist or you do not have access to it.“, wenn ich einen AWS Glue Job ausführe, um Nachrichten an Amazon SQS in einer anderen Region zu senden?
AWS OFFICIALAktualisiert vor 3 Jahren
Wie behebe ich den Fehler „canceling statement due to conflict with recovery“ (Anweisung wird aufgrund eines Konflikts mit der Wiederherstellung abgebrochen) beim Abfragen des Lesereplikats für meine DB-Instance in RDS für PostgreSQL?
AWS OFFICIALAktualisiert vor 2 Jahren
Wie behebe ich den AWS-KMS-Schlüsselrichtlinienfehler „Policy contains a statement with one or more invalid principals“ (Richtlinie enthält Aussage mit einem oder mehreren ungültigen Pinzipalen)?
AWS OFFICIALAktualisiert vor 2 Jahren
Wie behebe ich den Fehler „Bucket cannot have ACLs set with ObjectOwnership's BucketOwnerEnforced setting“ in CloudFormation?
AWS OFFICIALAktualisiert vor einem Jahr