Is it possible with DMS to have more than one maxFullloadsubtasks and still respect load-order?

Question

It seems that allowing tables to be loaded in parallel (by setting maxFullloadsubtasks > 1) breaks load-order settings. I believe what is happening is that the tables are started in the order specified, but DMS is not waiting for them to finish before picking up a new table. This makes sense to some degree, because if every table needed to finish before the next one then no parallel loading would be possible.

However, I have a case where some tables can be loaded in parallel but some cannot be started until another table is finished. Is there any way to let the tables that are not dependent to be loaded in parallel?

However, I have a case where some tables can be loaded in parallel but some cannot be started until another table is finished. **Is there any way to let the tables that are not dependent to be loaded in parallel?**

Answer

Hello John,

Thank you for contacting us. As you have correctly stated that while using load-order settings, DMS will use the settings value to determine which tables can be started in parallel if the maxFullloadsubtasks parameter is set to a value greater than 1. However, DMS doesn't have any control as to when the full load for these tables will complete and hence load-order settings are just used to influence the commencing of table load operation.

Hence, we request Customers to control it by setting the maxFullloadsubtasks parameter to 1 but I do agree that it takes away the benefit of using parallelism even if two tables doesn't have any dependency and hence the task take longer to finish.

Considering your specific use case, I will recommend you to create logical grouping of tables across multiple dms tasks. You can load the independent ones using a higher value of maxFullloadsubtasks parameter via one task while the dependent ones can be loaded serially using load-order using another.

Do let me know if you have any followup questions.

Is it possible with DMS to have more than one maxFullloadsubtasks and still respect load-order?

Relevanter Inhalt