Skip to content

AWS DMS is ignoring LONGTEXT data type column in migration task

0

I have a migration task that migrates a table from MariaDB to S3, it migrated the data successfully, but it excluded a column with Longtext data type, based on my search the issue is that AWS DMB considers this column as LOB and I used limited LOB Mode in the migration task with LOB max file size way greater than the size of the longest value in this column, Limited LOB mode should truncate the column, if the it exceeds the limited size, but it doesn't include the column at all not even truncating the values even though the whole column is way less than the maximum file size.

Source info:

  • MariaDB engine hosted on RDS.

Destination info:

  • S3 Bucket
  • file format: CSV
  • Compression type: GZ

Column info:

  • Data type: longtext CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL CHECK (json_valid(column_name))
  • The longest value in the column is 108 characters in ASCI which is around 108 Byte.

Migration task info:

  • Limited LOB Mode (because S3 doesn't support Full LOB Mode as a destination).
  • LOB Max File Size: 32KB.

Solution I tried and failed:

  • Setting the Max file size to 200000 KB (which is the maximum file size Limited LOB mode supports).
  • Changing the destination file format to parquet.
1 Answer
0
Accepted Answer

The issue you're experiencing with AWS DMS ignoring the LONGTEXT column during migration is likely due to how DMS handles LOB (Large Object) data types.

DMS considers data types such as LONGTEXT as LOB fields during migration. When migrating to S3, DMS has some limitations in handling LOB data. Specifically, DMS doesn't support LOB data unless it is a CLOB (Character Large Object).

In your case, even though you've set the Limited LOB Mode with a max file size greater than your data, the column is still being excluded. This could be because DMS is treating the LONGTEXT column as a binary LOB rather than a character LOB.

To resolve this issue, you could try the following:

  1. In your task settings, ensure that "SupportLobs" is set to true in the "TargetMetadata" section.

  2. Try setting "InlineLobMaxSize" in addition to "LobMaxSize". This tells DMS to inline LOBs up to this size directly in the record, which might help with your LONGTEXT column.

  3. Consider changing the data type of the column in the source database from LONGTEXT to TEXT or MEDIUMTEXT if possible. DMS might handle these types differently.

  4. If the data in the LONGTEXT column is always valid JSON (as suggested by your CHECK constraint), you could try mapping it as a JSON type in your task settings. This might change how DMS handles the column.

  5. As a workaround, you could create a view in your source database that casts the LONGTEXT column to a VARCHAR type, and migrate from this view instead of the original table.

If these solutions don't work, you may need to contact AWS support for further assistance, as there might be an underlying issue with how DMS is interpreting your specific LONGTEXT column.
Sources
Step-by-step Amazon RDS for MySQL database to Amazon DynamoDB migration walkthrough - Database Migration Guide
AWS DMS Failing to Migrate Tables with LOBs and JSONB Columns | AWS re:Post

answered 7 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.