training nn is slower then usual almost 2.5 slower
0
i am using 4 gpu g5.12xlarge instance to train object detection models.
each iteration takes 1 sec.
this week when i started new training in 3 different instances. training is taking more time. 2.5 sec for iteration.
i didnt do any change in the model. added some small amount of data. but nothing else.
i checked my montior and i can see that some monitors are lower then last week.
cpu utilization. network in , network out , network packets.