Understanding the Usage of suggested_constraints() and suggest_baseline() in SageMaker Model Monitoring with Pipelines

0

I'm currently working with a SageMaker pipeline setup, that uses QualityCheckStep() to run ModelQualityCheck and DataQualityCheck jobs. These jobs accept parameters such as register_new_baseline and skip_check. After the pipeline runs, the model is registered and ready to be approved. Once it is approved a lambda function is triggered which deploys the model to an endpoint and creates the monitoring schedules.

I'm using the ModelQualityMonitor() and DefaultModelMonitor() in the lambda function to configure the monitoring schedules for data and model quality monitoring, but these both accept baseline statistics and constraints. In the documentation examples I have seen, the suggested_constraints() and suggest_baseline() methods are used to retrieve the latest baselining job. This doesn't seem to work in the scenario where the baselining jobs are run in the SageMaker pipeline, and then the monitoring schedules are set up afterwards.

To get around this, I've located the baseline jobs associated with the pipeline execution that built the model and can pass the s3 uris when creating the monitoring schedule from the lambda. But it seems the constraints are very tight and I'm constantly getting violations - making me wonder if the issue is that I'm not using the provided suggested_constraints() and suggest_baseline() methods.

Is there a different approach I should be taking? I also tried without providing baselines, and this worked for the ModelQualityMonitor, but then I'm constantly getting no violations regardless of what I send to the endpoint.

Any insights or guidance on these aspects would be greatly appreciated!

lhobbs
asked 9 months ago144 views
No Answers

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions