How to troubleshoot useless error logs from DMS?

Q) The error message seems to be empty. How do I know what went wrong?

**You may increase the log severity level for the components like SOURCE_UNLOAD and TARGET_LOAD to get more detailed information in logs. **

The levels of severity are in order from lowest to highest level of information. The higher levels always include information from the lower levels.

LOGGER_SEVERITY_ERROR – Error messages are written to the log.

LOGGER_SEVERITY_WARNING – Warnings and error messages are written to the log.

LOGGER_SEVERITY_INFO – Informational messages, warnings, and error messages are written to the log.

LOGGER_SEVERITY_DEFAULT – Informational messages, warnings, and error messages are written to the log.

LOGGER_SEVERITY_DEBUG – Debug messages, informational messages, warnings, and error messages are written to the log.

LOGGER_SEVERITY_DETAILED_DEBUG – All information is written to the log.

Please follow the link below for more details.

[+] https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Tasks.CustomizingTasks.TaskSettings.Logging.html [+] Why did my AWS DMS task fail with no errors?: https://repost.aws/knowledge-center/dms-task-failed-no-errors

To troubleshoot migration tasks in AWS Database Migration Service, Please follow the below mentioned link for more details. [+] https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Troubleshooting.html

Q) It's a full-load migration so my only option is to restart from scratch. How do I know it won't fail at the same place next time?

If you're migrating a single table, it's a best practice to restart the task rather than using the resume option.

Restarting the task will reload all table data from the beginning, using the relevant table preparation mode (e.g., Drop tables on target, Truncate, Do nothing). This is recommended if:

• If there is only one table
• The task was started manually during task creation.
• An error occurred that requires reloading all data, such as a fatal error or an error status where one or more tables couldn't be migrated.

If you use the resume option while migrating multiple tables, then tables that didn't fully complete starts over. Tables that completed migration aren't affected. Resuming the task will continue replication from the last point before the task failed or stopped. This can be considered if you are migrating multiple tables, and only some tables didn't fully complete.

In general, for a single table full-load migration, it's recommended to restart the task rather than resume to ensure a clean full reload of the data.

[+] https://repost.aws/knowledge-center/dms-restart-resume-failed-task

To know it won't fail at the same place next time, It is highly recommended to check the logs (increase severity of logging) and investigate/analyze the issue first and then proceed.

When troubleshooting seemingly useless error logs from AWS Database Migration Service (DMS), especially in a homogeneous full-load migration between two RDS MySQL instances, there are several steps you can take:

1. Check CloudWatch Logs: The error message you're seeing might be truncated. Check the full CloudWatch logs for the DMS task, as they may contain more detailed information about the error.

2. Examine Task Statistics: In the AWS DMS console, look at the task statistics and table statistics for more information on what might have caused the failure.

3. Review Target Database Logs: Check the logs of your target RDS MySQL instance. There might be more detailed error messages or warnings that can help identify the issue.

4. Monitor Resource Utilization: Ensure that both the source and target databases have sufficient resources (CPU, memory, IOPS) to handle the migration load. High resource utilization can cause failures.

5. Check for Data Inconsistencies: Verify if there are any data inconsistencies or constraints in the source data that might be causing issues during the migration.

6. Increase Log Verbosity: If possible, increase the logging verbosity for the DMS task to get more detailed information in subsequent runs.

7. Use Diagnostic Support Scripts: AWS DMS provides diagnostic support scripts that can help identify issues with your migration tasks.

To address your concern about the migration failing at the same place next time:

1. Consider breaking down the migration into smaller chunks or tables if possible.

2. If the issue persists, you might need to investigate the specific table or data chunk that's causing the problem. Look for any anomalies in the data or structure of the table where the failure occurred.

3. Before restarting, review and possibly adjust the task settings, such as LOB settings, parallel load threads, or other performance-related parameters.

4. If you suspect resource constraints, consider scaling up your DMS replication instance or adjusting the target database's resources.

5. Enable and review slow query logs on both source and target databases to identify any performance bottlenecks.

Remember, while full-load migrations typically require restarting from scratch, understanding and addressing the root cause of the failure is crucial to prevent recurring issues. If the problem persists, you may need to engage AWS support for more in-depth troubleshooting.
Sources
Troubleshoot Amazon RDS for MySQL and Amazon RDS for MariaDB Errors | AWS Database Blog
Troubleshooting migration tasks in AWS Database Migration Service - AWS Database Migration Service
Migrating data from MySQL databases with homogeneous data migrations in AWS DMS - AWS Database Migration Service