Modle Evaluation Step with Sagemake Pipelines and custom metrics

Question

Examples such as https://github.com/aws/amazon-sagemaker-examples/tree/main/sagemaker-pipeline-compare-model-versions have a model evaluation step that generates a json file contain eval metrics

This can then be attached to the model in the register model step, i.e.

```
model_metrics = ModelMetrics(
    model_statistics=MetricsSource(
        s3_uri=".../evaluation.json",
        content_type="application/json",
    )
)
```

But all I can find about the schema of evaluation.json is https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-model-quality-metrics.html which implies (to me at least) that only certain architectures and metrics are supported (i.e. there are 3 top level keys "regression_metrics", "binary_classification_metrics" and "multiclass_classification_metrics"). Within each there seems to be fairly standard metrics and it looks like most have the format

"accuracy" : {
      "value" : 0.6288167938931297,
      "standard_deviation" : 0.00375663881299405
    },

But my model is a sentence transformer and my evaluation metrics are the standard ST information retrieval metrics (precision@1, precision@3, recall@1, recall@3 ...). Does this still work or does studio only support the metrics from the model-monitor-model-quality-metrics.html page?

Answer

Hi there,

Thanks for using AWS Sagemaker.

Have you checked out the paramaterised pipeline notebook -> https://github.com/aws/amazon-sagemaker-examples/blob/main/sagemaker-pipeline-parameterization/evaluate.py

You will see that we can create virtually any metrics, using a processing job to evaluate the model. Once the evaluation report is created which is just a JSON file containing our metrics that was created by the previous evaluate model step (processing job) we can run this through our accuracy step.

You can basically make your own metrics.

Hope this information helps.

To further understand the issue more in depth as I have limited visibility on your setup, I'd recommend you to reach to AWS Support by creating a support case[+] so that the engineer can investigate further and help you overcome the issue.

[+] Open a support case with AWS using the link: https://console.aws.amazon.com/support/home?#/case/create

Answer

Thanks, but the example you linked is the reason I'm asking the question, the json has a specific format

```
    # Available metrics to add to model: https://docs.aws.amazon.com/sagemaker/latest/dg/model-monitor-model-quality-metrics.html
    report_dict = {
        "binary_classification_metrics": {
            "accuracy": {"value": accuracy, "standard_deviation": "NaN"},
            "precision": {"value": precision, "standard_deviation": "NaN"},

...
```

In particular there's a top level key ("binary_classification_metrics") and then specific metrics for that class of problem. And the comment specifically mentions "available metrics" which implies only specific metrics/models (i.e. regression, binary of multi-class classification) mentioned on the page you provided are supported?

Modle Evaluation Step with Sagemake Pipelines and custom metrics

Relevant content