How to use Cross-Validation with Bayesian Hyperparameter Tuning Job with DeepAR model?

Question

I'm currently exploring using Sagemaker DeepAR for a forecasting problem. I've set up a simple aproach with the Sagemaker SDK for model training, using the Sagemaker Tuner with the Bayesian Optimization approach, setting up 10 HP Optimization Jobs. How would I go about adding cross-validation to this simple approach? Is there a sagemaker framework / build-in solutions for this? I could not find any documentation for this, neither for Sagemaker in general nor specified for the DeepAR algorithm.

I appreciate any hints and recommendations.

Answer

Hello,

I understand that you are concerned about using Cross-Validation with a Bayesian hyperparameter tuning job with the DeepAR model and would like to gather more information on the same.

Firstly, Improving and evaluating the performance of a machine learning model often requires a variety of ingredients. Hyperparameter tuning and cross-validation are two such ingredients. The first finds the best version of a model, while the second estimates how a model will generalise to unseen data. These steps, combined, introduce computing challenges as they require training and validating a model multiple times, in parallel and/or in sequence. [2]

One approach could be to use Amazon Jumpstart, as JumpStart has the possibility to train models incrementally. This way, training models with both the old and new data will take much less time. Also, JumpStart received support for model tuning with SageMaker Automatic Model Tuning. This feature automates the process of searching for the best hyperparameter configuration for a model. [1] and [2]

I would request that you please refer to the aforementioned documentation once and please reach out to AWS [7] with the detailed use case so that we can assist you better.

If you have any difficulty verifying any of the above-mentioned points or if you still run into issues, please reach out to AWS Support [7] (Sagemaker) along with your issue or use case in detail, and we would be happy to assist you further.

References:
=========
[1] https://aws.amazon.com/blogs/machine-learning/deep-demand-forecasting-with-amazon-sagemaker/

[2] https://towardsdatascience.com/fast-and-scalable-hyperparameter-tuning-and-cross-validation-in-aws-sagemaker-d2b4095412eb

[3] https://machinelearningmastery.com/nested-cross-validation-for-machine-learning-with-python/

[4] https://www.mdpi.com/1999-4893/16/1/17

[5] https://www.mdpi.com/2227-7390/10/18/3276

[6] https://dl.acm.org/doi/abs/10.1145/2487575.2487629

[7] Creating support cases and case management - https://docs.aws.amazon.com/awssupport/latest/user/case-management.html#creating-a-support-casehttps://docs.aws.amazon.com/awssupport/latest/user/case-management.html#creating-a-support-case

How to use Cross-Validation with Bayesian Hyperparameter Tuning Job with DeepAR model?

References:

Relevant content