Sagemaker training Logs in CloudWatch

Question

Hi everyone,

I've been using AWS for training models for a week now. I'm able to submit and run serverless training jobs but unfortunately I'm unable to see the logs of these jobs. From what I've read in https://docs.aws.amazon.com/sagemaker/latest/dg/logging-cloudwatch.html, everything logged in stderr and stdout should be automatically logged in CloudWhatch under the log group "/aws/sagemaker/TrainingJobs". I've made sure to create this log group beforehand but I don't see any log stream under this log group. Also, when I try to access logs from Sagemaker UI by directly clicking on "show logs" for a training job, I correctly get redirected to ClouldWatch under the log group "/aws/sagemaker/TrainingJobs" but there is no log. Am I missing something ?

Many thanks in advance for your help

Riku_Kobayashi · Answer

Hello.

What are the settings of the IAM role used by SageMaker?  
Try setting the following IAM policy for the IAM role used by the notebook instance.  
```
{
     "Version": "2012-10-17",
     "Statement": [
         {
             "Action": [   
                 "logs:CreateLogDelivery",      
                 "logs:CreateLogGroup",
                 "logs:CreateLogStream",
                 "logs:DeleteLogDelivery",
                 "logs:Describe*",
                 "logs:GetLogEvents",
                 "logs:GetLogDelivery",
                 "logs:ListLogDeliveries",
                 "logs:PutLogEvents",
                 "logs:PutResourcePolicy",
                 "logs:UpdateLogDelivery"
             ],
             "Resource": "*",
             "Effect": "Allow"
         } 
     ]
 }
```

Sagemaker training Logs in CloudWatch

Relevant content