EMR Hive Meta Database on RDS Upgrade Issue

Question

My customer is using AWS EMR and is storing all the Hive meta data on an external RDS instance, using MySQL 5.7.*  And since MySQL 5.7 is running out of its lifecycle, we are pushing them to upgrade the RDS instance to MySQL 8.0.  However, my customer is very concerned in regarding the RDS upgrade.  So the basic question is, could its Hive (on EMR) still function the same way after the DB upgrade, as it did before?  And are there any foreseeable issues for such an upgrade like this?

Accepted Answer

Hello,

Yes, EMR on Hive functions as usual. Mostly there wont be any issue in the external RDS upgrade as far as I know.  As long as the external store that are compatible with MariaDB JDBC as your metastore, it works fine. However, I strongly recommend you to first test in your lower environment and have the necessary backup plans like have the backup of database and EMR side configurations in place before make any changes.

Post upgrade, Metastore might not be auto-initialized. If that is the case, use the hive schema tool to perform them. - https://cwiki.apache.org/confluence/display/Hive/Hive+Schema+Tool

This doc might help to refer the steps - https://aws.amazon.com/blogs/big-data/migrate-and-deploy-your-apache-hive-metastore-on-amazon-emr/

Prechecks before mysql upgrade - 
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.MySQL.html#USER_UpgradeDBInstance.MySQL.57to80Prechecks

EMR Hive Meta Database on RDS Upgrade Issue

Relevant content