How do I troubleshoot and resolve issues in Amazon CloudWatch Synthetics canary?

Resolution

Prerequisite: Confirm that your AWS Identity and Access Management (IAM) users and canaries have the required roles and permissions. For more information, see Required roles and permissions for users who manage CloudWatch canaries and Required roles and permissions for canaries.

Note: If you want to restrict an IAM user to view only specific canaries, then attach a custom policy to the IAM user or role.

Resolve Amazon S3 bucket issues

Resolve "the specified bucket does not exist" error

The Amazon Simple Storage Service (Amazon S3) bucket must exist before CloudWatch Synthetics can locate it. If the S3 bucket no longer exists, then you receive the “specified bucket does not exist” error message.

To resolve this error, create a lifecycle on the bucket to store your CloudWatch Synthetics canary artifacts. For more information, see Managing your storage lifecycle.

Resolve "access denied on upload" error

When the canary's IAM role doesn't have the required permissions to access the bucket, you receive the following error message:

"Unable to upload artifacts to S3, Exception: Unable to fetch S3 bucket location: Access Denied."

This error can also occur when encryption settings are misconfigured or when organizational policies restrict access.

To resolve this error, see Unable to upload artifacts to S3, Exception: Unable to fetch S3 bucket location: Access Denied.

For the bucket policy to allow the encryption mechanism that the canary uses, verify that the encryption mode for your canary and bucket policy is the same. If the encryption modes aren't the same, then choose a different encryption option.

Complete the following steps:

1. Open the Amazon CloudWatch console.
2. In the navigation pane, choose Synthetics Canaries, and then select your canary.
3. Choose Actions, and then choose Edit.
4. Under the Data storage section, choose Additional configuration.
5. Choose the type of key to use for encryption to choose a different encryption option.

Resolve bucket name validation error

To resolve the bucket name validation error, check your bucket and confirm whether it contains your canary script.

If the script is located in the bucket, then specify the bucket name in your script code. Don't include s3:// at the start of the bucket name. For example, use my-bucket-name instead of s3://my-bucket-name.

Resolve Lambda issues

Resolve the "The canary can't detect the latest Lambda layer" error

If you make manual changes or delete the AWS Lambda function, then the canary might not detect the latest AWS Lambda layer. To resolve this issue, create a new canary.

Resolve IAM role trust relationship errors for Lambda

The IAM role of your canary must have a trust relationship with the Lambda service.

Confirm that the IAM role contains the following statement:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Resolve a failed CloudWatch canary in your VPC

If your CloudWatch canary in a virtual private cloud (VPC) receives an error and fails, then follow the instructions in How do I resolve errors for the failed CloudWatch canary in my VPC?

Resolve Module Import and Dependency Errors

Resolve the "ModuleNotFoundError: No module named 'package_name'" error

When your Python canary can’t locate external dependencies, you receive the “ModuleNotFoundError” error message. CloudWatch Synthetics requires dependencies to be packaged in a specific directory structure and uploaded to Amazon S3.

To resolve this error, complete the following steps:

1. Create the correct directory structure for your Python packages:

   mkdir -p python/lib/python3.9/site-packages 

   Note: Replace python3.9 with your canary's Python runtime version. The directory path must match exactly.

2. To install the Python requests library to the custom target directory, run the following command:

   pip install requests -t python/lib/python3.9/site-packages/ 

3. To create a compressed ZIP archive file, run the following command:

   zip -r my-dependencies.zip python/ 

4. Run the following command to upload the dependency package to Amazon S3:

   aws s3 cp my-dependencies.zip s3://your-bucket-name/canary-dependencies/ --region us-east-1 

5. Run the following command to verify the upload succeeded:

   aws s3 ls s3://your-bucket-name/canary-dependencies/ --region us-east-1 

6. If you upload the dependency package to the same S3 bucket used for canary artifacts, the canary's IAM role already includes the required s3:GetObject permission. If you use a separate S3 bucket, add s3:GetObject permission for that bucket to the canary's IAM role. For more information, see Required roles and permissions for canaries.

Configure your canary to use the S3 dependency package

Complete the following steps:

1. Open the Amazon CloudWatch console.
2. In the navigation pane, choose Synthetics Canaries.
3. Select your canary, choose Actions, then choose Edit.
4. Under Additional configuration, add the S3 URI, s3://your-bucket-name/canary-dependencies/my-dependencies.zip
5. Choose Save.

After you take the preceding troubleshooting steps, you might receive one of the following errors:

• To resolve an "Access Denied" error, verify the IAM role has Amazon S3 read permissions on the bucket and key prefix.
• To resolve a "No such file or directory", verify your zip structure matches python/lib/python3.x/site-packages/> exactly.
• To resolve a "ModuleNotFoundError", verify the Python version in the directory path matches your canary's runtime version.
• If you receive intermittent failures with IaC deployments, you use Terraform or CloudFormation to deploy, and the canary fails only on initial deployment, then turn on the auto retry option. This resolves timing-related initialization issues.

Resolve Timeout Issues

Resolve the "TimeoutError: waiting for selector" or canary timeout errors

Timeout errors occur when page elements don’t load within the configured period. You might also receive timeout errors when per-step timeouts accumulate to exceed the total execution time. CloudWatch Synthetics has two types of timeouts. The per-step timeout is the maximum time to wait for individual actions. The default value is 30,000 ms. The Total canary timeout is the maximum time for entire canary execution. You can configure this setting in the canary settings.

Important: Per-step timeouts are cumulative. If your canary has 10 steps each with a 30-second timeout, it could take up to 5 minutes to complete, even if each step completes within its timeout.

To resolve timeout errors, complete the following options:

1. Increase per-step timeout in your canary script. It's a best practice to set per-step timeouts to twice your expected element load time. Use the following updates for Puppeteer (Node.js) canaries:

   const synthetics = require('Synthetics'); 
   const pageLoadBlueprint = async function () { 
    const page = await synthetics.getPage(); 
     page.setDefaultTimeout(60000); // 60 seconds 
   
    await page.goto('https://example.com', { 
     waitUntil: 'domcontentloaded', 
    timeout: 60000 
    }); 
   
    await page.waitForSelector('#site-selection-box', { timeout: 60000 }); 
    await page.click('#site-selection-box'); 
   }; 
   
   exports.handler = async () => { return await pageLoadBlueprint(); };

2. Adjust the total canary timeout in the canary settings. Under Additional configuration, set the timeout to a higher value. For more information, see Creating a canary. It's a best practice to set total canary timeout to accommodate all steps plus a 30-second buffer.

3. Use more reliable element selectors in your canary script. If your canary interacts with page elements such as, selecting buttons or filling in forms, then use stable selectors that are less likely to break when the page layout changes:
   For Use IDs, use #element-id
   For Use data attributes, use [data-testid="element"]
   For Avoid dynamic or positional selectors, use div > div > span:nth-child(3)

For more information about how to edit your canary script, see Edit or delete a canary.

Resolve Visual Monitoring and Screenshot Issues

Resolve screenshots don't appear in the console

If screenshots don't appear in the console, then verify that you included takeScreenshot() in your script with sufficient page load time before capture.

Example for Puppeteer (Node.js) canaries with takeScreenshot():

const synthetics = require('Synthetics'); 

const pageLoadBlueprint = async function () { 
 const page = await synthetics.getPage(); 
 await page.goto('https://example.com', { 
  waitUntil: 'networkidle0', 
 timeout: 60000 
 }); 

 // Wait for page to stabilize before taking screenshot 
 await page.waitForTimeout(2000); 

 // Take screenshot 
 await synthetics.takeScreenshot('homepage', 'loaded'); 
}; 

exports.handler = async () => { 
 return await pageLoadBlueprint(); 
}; 

In the preceding example, the timeout: 60000 value gives the page sufficient time to load before the screenshot is captured, and page.waitForTimeout(2000) adds a brief wait for the page to stabilize. To add page load time before capture, update the timeout parameter and the page_waitForTimeout parameter.

Also verify that the canary's IAM role has s3:PutObject permission on the canary's artifact S3 bucket where screenshots are stored. If you used the CloudWatch console to create the canary, then these permissions are included automatically.

Resolve visual monitoring configuration issues

Visual monitoring failures are usually caused by missing IAM permissions, incorrect threshold placement, or not following the visual monitoring blueprint.

To resolve visual monitoring configuration issues, see Visual monitoring of applications with Amazon CloudWatch Synthetics.

Note: Place threshold configuration before page operations in your script. If you place them after page operations, then Lambda resets to the default 0.02 threshold after the first run.

Important: There is a known limitation of 5-minute baseline inconsistency in visual monitoring. Visual monitoring might compare against the previous run's screenshot instead of the original baseline during a 5-minute window after baseline changes. This causes false negatives in defacement detection. The first comparison after a baseline change correctly fails. After 5 minutes, it correctly compares against the original baseline.

Resolve SSL/TLS Certificate Issues

CloudWatch Synthetics doesn't natively support self-signed certificates. When you monitor endpoints that use self-signed certificates, you might receive the following error:

"ERR_CERT_AUTHORITY_INVALID or SSL: CERTIFICATE_VERIFY_FAILED"

Confirm the error is certificate-related (Python runtime)

By default, Python runtime canaries have the --ignore-certificate-errors flag turned on. If you still receive certificate errors, then add the following to your script to confirm the issue is related to certificate validation:

from aws_synthetics.selenium import synthetics_webdriver as webdriver 
from selenium.webdriver.chrome.options import Options 
import urllib3 

def main(): 
 urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) 
  chrome_options = Options() 
  chrome_options.add_argument('--ignore-certificate-errors') 
  chrome_options.add_argument('--ignore-ssl-errors') 
 browser = webdriver.Chrome(chrome_options=chrome_options) 
  browser.get('https://your-endpoint-with-self-signed-cert.com') 
  browser.quit() 
def handler(event, context): 
 return main()

If the canary passes after you add these flags, the error is confirmed as certificate-related.

Important: When you bypass certificate verification, you turn off SSL/TLS validation. Don't use certificate bypass as a permanent solution in production environments.

Implement the recommended production solution

It's a best practice to store the self-signed certificate in AWS Secrets Manager and retrieve it within your canary script. For detailed instructions, see How to validate authentication with self-signed certificates in Amazon CloudWatch Synthetics.

Manage Runtime Versions and Upgrades

Monitor EOL timelines

Monitor the AWS What's New blog, AWS Health Dashboard, and the CloudWatch Synthetics runtime versions to receive runtime end of life (EOL) announcements.

Upgrade canary runtime version

To upgrade your canary to a newer runtime version, see Performing safe canary updates.

It's a best practice to perform a dry run before you apply runtime updates to verify compatibility with your canary script and dependencies.

Resolve the "ReferenceError: AWS is not defined" error after you clone a canary

If your cloned canary fails with "ReferenceError: AWS is not defined", then update the IAM policy to include the cloned canary's ARN or use a wildcard.

Example

// Before (resource-specific - will block cloned canary): 
"Resource": "arn:aws:synthetics:us-east-1:123456789012:canary:original-canary" 
// After (includes cloned canary or uses wildcard): 
"Resource": "arn:aws:synthetics:us-east-1:123456789012:canary:*"

Best practices for issues in CloudWatch Synthetics canary

Take the following actions:

• Check the Availability and Details tabs for failure details.
• Check the screenshots to verify the latest canary invocations.
• Check the log file to identify permission issues or throttling exceptions.
• Check the Monitoring tab for canary and Lambda metrics.
• Check the CloudWatch log groups with the name /aws/lambda/cwsyn-MyCanaryName-randomId for recent canary operation logs.
• Check the Amazon S3 bucket with the name cw-syn-results-accountID-region for artifact uploads such as screenshots or HAR files.
• Check the CloudWatch metrics published by canaries.
• Check CloudWatch Logs for ModuleNotFoundError messages and verify that your dependency package structure matches python/lib/python3.x/site-packages/.

Related information

Why did my CloudWatch canary stop running?

How do I use CloudWatch Synthetics to monitor the performance of my website?