Why is my Application Migration Service or Elastic Disaster Recovery replication process stuck at 100% with the "Finalizing Initial Sync" message?

Short description

When the replication process is stuck at 100% during syncs for Application Migration Service or Elastic Disaster Recovery, you see the following errors:

• "Finalizing Initial Sync - Flushing Backlog"
• "Finalizing Initial Sync - Creating First Launchable Snapshot"

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Troubleshoot the "Finalizing Initial Sync - Flushing Backlog" error

Wait for the backlog to complete flushing for the sync to initialize.

If the source machine is write intensive, then the backlog might increase in size. If the machine is stuck in the "Finalizing Initial Sync" state, then test the replication speed for Application Migration Service or Elastic Disaster Recovery. Also, calculate the required bandwidth for all replicating source machines. Make sure that the network throughput of the replication instance is sufficient.

To check your replication settings, complete the following steps:

1. Open the Application Migration Service or Elastic Disaster Recovery console.
2. In the navigation pane, choose Source servers.
3. Choose Edit replication settings.
4. Check whether Network bandwidth throttling is set to Throttle bandwidth. If you don't need to throttle bandwidth, then select Do not throttle bandwidth. If you require bandwidth throttling, then make sure to set the value to the minimum required bandwidth. For more information, see Data routing and throttling for Application Migration Service or Throttle network bandwidth for Elastic Disaster Recovery.

Use Amazon CloudWatch metrics to check the network and disk usage of the replication server. If a resource throttles the server, then use a dedicated replication server or a larger replication server type. Or, choose SSD-based storage. For more information, see AWS Application Migration Service best practices or Change staging disk type.

To check the replication server that a specific source machine uses, run the following command on the source machine based on your operating system (OS).

Linux:

netstat -anp | grep ":1500"

Windows:

netstat -ano | findstr ":1500"

Note the remote IP address that the machine connects to over port 1500.

Or, run the following command based on your OS to review the agent.log.0 file on the source machine and identify the exact replication server in use.

Linux:

sudo cat /var/lib/aws-replication-agent/agent.log.0 | grep :1500 | tail -n 1

Windows:

findstr /L ":1500" "C:\Program Files (x86)\AWS Replication Agent\agent.log.0"

Troubleshoot the "Finalizing Initial Sync - Creating First Launchable Snapshot" error

Check the user's IAM policy permissions

Make sure that the user's AWS Identity and Access Management (IAM) policy has permissions to run the Amazon Elastic Compute Cloud (Amazon EC2) APIs.

Check the user's credentials for Application Migration Service or Elastic Disaster Recovery. Or, check the AWS CloudTrail Event history to find API failures that occurred for the user.

Check your Amazon EC2 endpoints

Make sure that the replication server can communicate with the Amazon EC2 endpoints for Application Migration Service or Elastic Disaster Recovery in your AWS Region.

To test connectivity to your service endpoints, complete the following steps:

1. Launch a new Linux instance in the same subnet as your staging area.
2. Log in to the new instance , and then run the following commands:

   dig ec2.us-east-1.amazonaws.com  
   telnet ec2.us-east-1.amazonaws.com 443  
   wget https://ec2.us-east-1.amazonaws.com

   Note: Replace us-east-1 with your Region. If any of the preceding commands fail, then proceed to the following section to troubleshoot network connectivity issues. Confirm that the virtual private cloud (VPC), subnet, security group, network access control list (network ACL), and route table settings align with your replication settings. A misconfiguration might block communication to Amazon EC2 endpoints from the replication servers.

Identify network connectivity blockers for a replication server that you launched in a public subnet

Make sure that the security group, network ACLs, and the route table allow communication with Amazon EC2 endpoints on TCP port 443.

Verify that you set the enableDnsHostnames and enableDnsSupport attributes to true at the VPC level. Run the following describe-vpc-attribute AWS CLI command to check the enableDnsHostnames attribute:

aws ec2 describe-vpc-attribute --vpc-id vpc-abcdef --attribute enableDnsHostnames   

Note: Replace vpc-abcdef with your VPC ID.

Example output:

$ aws ec2 describe-vpc-attribute --vpc-id vpc-abcdef --attribute enableDnsHostnames  
{   
 "VpcId": "vpc-a01106c2",  
 "EnableDnsHostnames": {   
 "Value": true  
 }   
}

Run the following describe-vpc-attribute command to check the enableDnsSupport attribute:

aws ec2 describe-vpc-attribute --vpc-id vpc-abcdef --attribute enableDnsSupport 

Note: Replace vpc-abcdef with your VPC ID.

Example output:

$ aws ec2 describe-vpc-attribute --vpc-id vpc-abcdef --attribute enableDnsSupport     
{  
 "VpcId": "vpc-a01106c2",   
 "EnableDnsSupport": {  
 "Value": true   
 }  
}

If the attributes aren't set to true, then run the following modify-vpc-attribute command to activate them:

aws ec2 modify-vpc-attribute --vpc-id vpc-abdcef --enable-dns-hostnames "{\"Value\":true}"

Note: Replace vpc-abcdef with your VPC ID and enable-dns-hostnames with the attribute that's not set to true.

Identify network connectivity blockers for a replication server that you launched in a private subnet

Verify that the security group, network ACLs, and route table allow communication with Amazon EC2 private endpoints on TCP port 443. If you configured a NAT gateway or instance in the route table, then verify that outbound traffic to the Amazon EC2 endpoint on TCP port 443 works.

Check whether outbound traffic passes through a transit or virtual private gateway. If it does, then make sure that the route table allows traffic to Amazon EC2 endpoints on TCP port 443. Also, check whether the firewall blocks communication.

If the VPC has interface VPC endpoints, then make sure that communication occurs between Amazon EC2 endpoints on TCP port 443 through a private network. Verify that you set enableDnsHostnames and enableDnsSupport to true at the VPC level, and PrivateDnsEnabled to true on the VPC interface endpoints.

Run the following describe-vpc-attribute command to check the enableDnsHostnames attribute:

aws ec2 describe-vpc-attribute --vpc-id vpc-abcdef --attribute enableDnsHostnames --query 'EnableDnsHostnames'

Note: Replace vpc-abcdef with your VPC ID.

Example output:

$ aws ec2 describe-vpc-attribute --vpc-id vpc-a01106c2 --attribute enableDnsHostnames --query 'EnableDnsHostnames'  

{   
 "Value": true  
} 

Run the following describe-vpc-attribute command to check the enableDnsSupport attribute:

aws ec2 describe-vpc-attribute --vpc-id vpc-abcdef --attribute enableDnsSupport --query 'EnableDnsSupport'

Note: Replace vpc-abcdef with your VPC ID.

Example output:

$ aws ec2 describe-vpc-attribute --vpc-id vpc-a01106c2 --attribute enableDnsSupport --query 'EnableDnsSupport'  

{   
 "Value": true  
}

Run the following describe-vpc-attribute command to check the PrivateDnsEnabled attribute:

aws ec2 describe-vpc-endpoints --vpc-endpoint-ids vpce-12345abcde --query 'VpcEndpoints[0].PrivateDnsEnabled'  

Note: Replace vpce-12345abcde with your VPC endpoint.

Example output:

true

If the attributes aren't set to true, then run the following modify-vpc-attribute command to modify them:

aws ec2 modify-vpc-attribute --vpc-id vpc-abdcef --enable-dns-hostnames "{\"Value\":true}"

Note: Replace vpc-abcdef with your VPC ID and enable-dns-hostnames with the attribute that's not set to true.

Check for recent changes in Replication Settings

Check for recent changes in Replication settings for Application Migration Service or Elastic Disaster Recovery. Check your CloudTrail Event history for the UpdateReplicationConfiguration API call to identify changes. Use source server to filter the Resource name. For example, check whether there's a not valid tag inserted in the Replication resources tags field. For a list of allowed characters, see Tag restrictions.

Verify that you use the correct proxy settings

If your replication servers use a proxy server, then confirm that the proxy settings allow communication with Regional Amazon EC2 endpoints on TCP port 443.

Make sure that the SSL interception and authentication allowlist includes mgn.region.amazonaws.com for Application Service Migration and drs.region.amazonaws.com for Elastic Disaster Recovery.

Note: Replace region with your Region.

For more information, see Can a proxy server be used between the source server and the AWS Application Migration Service console? Also, see Can a proxy server be used between the source server and the Elastic Disaster Recovery Console?

Check the AWS Replication Agent

Confirm that the AWS Replication Agent works correctly on the source machine. To pinpoint issues, check the AWS Replication Agent logs for errors. You can find the AWS Replication Agent logs in the following file locations:

• For Linux Replication Agent logs, check /var/lib/aws-replication-agent/agent.log.0
• For Windows Replication Agent logs, check C:\Program Files (x86)\AWS Replication Agent\agent.log.0

Check for Amazon EC2 service quota issues

If you exceed a service quota for Application Migration Service or Elastic Disaster Recovery, then the services can't create a launchable recovery snapshot. You might also encounter API throttle or rate exceeded issues. Check the CloudTrail Event history to identify service quota or bandwidth throttling issues. If needed, then request a quota increase.