Hello There,

Thank you for the query.

Please see my response below:

I would like to inform you that EMR service team is already working on upgrading the python version in EMR. Unfortunately, I cannot provide a specific date for its implementation as of now, as we don't have any visibility into the release team's product pipeline and enhancements.

In the meantime, if you would like to upgrade your Cluster to a higher version of Python, I can suggest the following two workarounds.

1. You can follow these instructions [1], which provides a step by step instruction on using Custom Python 3 version on EMR.
2. Alternately, if you would like to upgrade EMR to Python 3.9 for instance, here is a sample bootstrap script. You can verify if this has been successful, by SSH into Primary Node after EMR cluster is ready, by running commands "python --version" and "pyspark".

===============================================

#!/bin/bash
sudo yum install libffi-devel -y
sudo wget https://www.python.org/ftp/python/3.9.0/Python-3.9.0.tgz   
sudo tar -zxvf Python-3.9.0.tgz
cd Python-3.9.0
sudo ./configure --enable-optimizations
sudo make altinstall
python3.9 -m pip install --upgrade awscli --user
sudo ln -sf /usr/local/bin/python3.9 /usr/bin/python3

===============================================

As I understand you have already tried bootstrap solution, I would suggest you to try this bootstrap script, as it would be easier and quicker solution.

Please let me know if the above solutions work for you or if you have any further questions on this.

References:

[1] > https://github.com/aws-samples/aws-emr-utilities/blob/main/utilities/emr-ec2-custom-python3/README.md#reducing-cluster-start-time