Hello,

I understand that you are concerned about using Cross-Validation with a Bayesian hyperparameter tuning job with the DeepAR model and would like to gather more information on the same.

Firstly, Improving and evaluating the performance of a machine learning model often requires a variety of ingredients. Hyperparameter tuning and cross-validation are two such ingredients. The first finds the best version of a model, while the second estimates how a model will generalise to unseen data. These steps, combined, introduce computing challenges as they require training and validating a model multiple times, in parallel and/or in sequence. [2]

One approach could be to use Amazon Jumpstart, as JumpStart has the possibility to train models incrementally. This way, training models with both the old and new data will take much less time. Also, JumpStart received support for model tuning with SageMaker Automatic Model Tuning. This feature automates the process of searching for the best hyperparameter configuration for a model. [1] and [2]

I would request that you please refer to the aforementioned documentation once and please reach out to AWS [7] with the detailed use case so that we can assist you better.

If you have any difficulty verifying any of the above-mentioned points or if you still run into issues, please reach out to AWS Support [7] (Sagemaker) along with your issue or use case in detail, and we would be happy to assist you further.

References:

[1] https://aws.amazon.com/blogs/machine-learning/deep-demand-forecasting-with-amazon-sagemaker/

[2] https://towardsdatascience.com/fast-and-scalable-hyperparameter-tuning-and-cross-validation-in-aws-sagemaker-d2b4095412eb

[3] https://machinelearningmastery.com/nested-cross-validation-for-machine-learning-with-python/

[4] https://www.mdpi.com/1999-4893/16/1/17

[5] https://www.mdpi.com/2227-7390/10/18/3276

[6] https://dl.acm.org/doi/abs/10.1145/2487575.2487629

[7] Creating support cases and case management - https://docs.aws.amazon.com/awssupport/latest/user/case-management.html#creating-a-support-casehttps://docs.aws.amazon.com/awssupport/latest/user/case-management.html#creating-a-support-case