How do I resolve errors when I fine-tune models on Amazon Bedrock?

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Access and availability issues

Prerequisite: Request access to the base model.

Amazon Bedrock supports fine-tuning in certain AWS Regions and foundational models. Make sure that you use a supported AWS Region for your model with fine-tuning capabilities. If your Region doesn't support your model, then the Amazon Bedrock model picker loads continuously. If you can't create a fine-tuning job, then you might receive the following error message:

"Unable to Create a fine-tuning job ("The provided model identifier is invalid" error)"

To resolve your access error, confirm that the model and fine-tuning capabilities are supported in your preferred AWS Region.

If you see a blank screen or can't select a model, then your model isn't available in that Region or your AWS account doesn't have access to the model. To resolve, select a model across different networks and browsers. When you select models, capture a HAR file. Then, check that you don't have restrictions for your VPN, firewall, or virtual desktop infrastructure (VDI). If you still aren't able to access the Amazon Bedrock console, then contact your network administrator.

IAM permissions issues

To access the Amazon Simple Storage Service (Amazon S3) bucket to customize a model, you must attach the correct permissions to your AWS Identity and Access Management (IAM) role. If you didn't correctly configure permissions for the IAM role, then you receive the following error message:

"Encountered an unexpected error when processing the request, please try again","AccessDenied"."

To resolve this error, grant IAM permissions for Amazon Bedrock for model customization.

Cross-Region access errors

Fine-tuning jobs can only access Amazon S3 data in the same Region that the job runs. If you try to access training data stored in an S3 bucket that's located in a different Region than your fine-tuning job, then you receive the following error message:

"Could not validate GetObject permissions to access S3 bucket"

To resolve this error, confirm that your S3 bucket and Amazon Bedrock fine-tuning job are both in the same Region. Then, move your training data to an S3 bucket in the same Region as Amazon Bedrock.

Data format and validation errors

Your datasets must meet the training and validation model requirements. If you submit a training dataset with an incorrect JSON structure or format, then you might receive one of the following errors:

"The provided S3 URI is invalid", ";validation error detected: Value 's3://..' failed to satisfy constraint:" Member must satisfy regular expression pattern.."

"Invalid training input data configuration. Check the input data S3 Uris(Uri should be a file, not a directory) and retry request"

To resolve, make sure that you use file uniform resource identifiers (URI) and confirm that you don't use directory URIs.

URL example: s3://bucket-name/datasets/train/train.jsonl

Directory URL example: s3://bucket-name/datasets/train/

Then, check that the path points to the exact file location. If the path doesn't point to the exact file location, then verify the complete path structure in your S3 bucket. Update your file URI to match the exact location and file name where you store your training data.

Dataset structure validation

Different text models have different data requirements and expected JSON schemas. If you submit a training dataset with an incorrect JSON structure or format, then you might receive the following error:

"JSON structure is not supported: JSON schema error: JSON does not match JSON schema at line."

To resolve this error, prepare data for fine-tuning text-to-text models.

Image fine-tuning issues

Amazon Bedrock must have access to the image file in the specified dimension. If you upload images that don't meet the specific dimensional requirements for the model that you're fine-tuning, then you might receive the following error:

"Image Dimensions are Invalid. Check that the images conform to accepted dimensions."

Check that image dimensions meet the requirements for the specific model. Then, validate that you prepared your image datasets for the specific model.

Note: Some models only support text fine-tuning and not image fine-tuning.

Validation errors

Your files must be compatible with your model. If you submit training data that doesn't meet the model's formatting requirements or contains structural issues, then you might receive the following error:

"Validation error: failed during preprocessing"

To resolve this error, verify your data format:

• Make sure that your JSONL file has one valid JSON object per line.
• Check that you included mandatory fields in each example.
• Verify that your examples don't exceed the model's maximum token quotas.
• Confirm that your file uses UTF-8 encoding without special character problems.

Monitor job status

You can use the Amazon Bedrock console or AWS CLI to monitor job status. Real-time progress monitoring isn't available in Amazon Bedrock.

To use the console, see Monitor your model customization job.

To use AWS CLI, run the following get-model-customization-job command:

aws bedrock get-model-customization-job -job-identifier "jobARN"

Note: Replace jobARN with your job"s Amazon Resource Name (ARN).

Then, you can estimate the time that it takes to complete each job as a benchmark against the model's actual completion time. When you estimate, consider the following factors:

• Fine-tuning duration depends on data size, number of epochs, and batch size.
• Early errors appear within the first few minutes.
• For larger datasets, fine-tuning can take several hours to complete.
• Training time lasts between 3-4 hours, up to 24 hours, depending on configuration and traffic.

If a job failed, then see Troubleshooting model customization issues.

If a job has been in Training state for more than 24 hours, then contact AWS Support.

Inference for fine-tuned models

For custom fine-tuned models, you must purchase Provisioned Throughput for models other than Amazon Nova to create a separate Provisioned Model ARN that's used instead of the original model ARN.

If you try to use the original model ARN directly in API calls, then you might receive the following error:

"An error occurred (ValidationException) when calling the xxxx operation: 1 validation error detected: Value 'arn:aws:bedrock:xxxxx' at 'modelId' failed to satisfy constraint: Member must satisfy regular expression pattern: ..."

To resolve this error, complete the following steps:

1. Create Provisioned Throughput for your fine-tuned model.
2. Note the Provisioned Model ARN from the response.
3. When you call fine-tuned models through the API, use the following model ARN format specific to custom models:
   arn:aws:bedrock:[region]:[account]:provisioned-model/[name]
   Note: Replace region with your Region, account with your account information and name with your model name.
4. Verify that the provisioned model is in the InService status before you use the GetProvisionedModelThroughput API call.

Related information

Set up inference for a custom model

Customize your model to improve its performance for your use case