SageMaker Experiment tracking duplication

0

The detailed StackOverflow question can be found in this link

I would like to initialize one training job that will be attached to existent Experiment. As mentioned in the best practice guide, I would like to initialize the Experiment and the Run in the notebook, and run the Training Job remotely, using a sagemaker Estimator in script mode.

The problem is, when doing that, Sagemaker creates 2 separate Runs - One Run is being initialized when calling an estimator.fit() from the notebook:

with Run(experiment_name, run_name, sagemaker_session=sess) as run:
    estimator = PyTorch(entry_point="my_script.py", ...)
    estimator.fit(inputs={"train": s3_input_train, "validation": s3_input_test})

And another Run is being created when loading the Run from my entry_point script "my_script.py":

sess = Session(boto3.session.Session(region_name=my_region))
with load_run(sagemaker_session=sess) as run:
    ...

I've tried adding the experiment_name and run_name to the with_load() function as well, but nothing seem to work.. I'm getting two separate runs, where some of the parameters being saved to one run, and some parameters to the other.

Also, it seems like the name of the Run being initialized through the estimator.fit() adds "-aws-training-job" to the job name and the Type "SageMakerTrainingJob".

Can someone help me with that?

rgabay
已提问 2 个月前123 查看次数
没有答案

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则

相关内容