training nn is slower then usual almost 2.5 slower

0

i am using 4 gpu g5.12xlarge instance to train object detection models. each iteration takes 1 sec. this week when i started new training in 3 different instances. training is taking more time. 2.5 sec for iteration. i didnt do any change in the model. added some small amount of data. but nothing else. i checked my montior and i can see that some monitors are lower then last week.

cpu utilization. network in , network out , network packets.

thank u for helping

aviw
asked a month ago72 views
No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions