Secret Rotation Lambda Unable to log in after rotating secret

0

Issue

We have an Aurora PostgreSQL version 14.5 RDS cluster. We have a secret in SecretsManager with credentials for a user we want to rotate the password for. When rotating the secret, the Lambda gets stuck at the setSecret step with the error Unable to log into database with previous, current, or pending secret. We have determined that this relates to the password_encryption option in the cluster parameter group. If we set it to md5 (whereas the default is, I believe, scram-sha-256) the rotation will work again after we update it manually. We can then rotate it as many times as we want.

Question

How can we get the secret rotation to work while using the default cluster parameter group for an Aurora PostgreSQL cluster?

To reproduce

  1. Have a secret formatted as expected.
  2. Have a Lambda running the python code provided by AWS.
  3. Have a version 14.5 Aurora PostgreSQL cluster using the default.aurora-postgresql14 cluster parameter group.
  4. Click the "Rotate secret immediately" button in the console
  5. In Lambda logs, see the error setSecret: Unable to log into database with previous, current, or pending secret of secret arn arn:aws:secretsmanager:....

How to Recover

  1. Create a new cluster parameter group that is a copy of default.aurora-postgresql14
  2. Change the password_encryption to be md5
  3. Apply this new parameter group to the cluster
  4. Cancel the secret rotation: aws secretsmanager cancel-rotate-secret --secret-id ....
  5. Manually change the password on the user to a new one
  6. Update the secret with the new password
  7. click the "Rotate secret immediately" button in the console
2개 답변
0
수락된 답변

I was able to figure it out! We're using the aws python lambda docker image. I had to compile the pg tools from source. The pg tools installable from yum on this container will only install major version 9 of the tools which is not compatible with the scram style of password encryption.

RUN yum install -y wget gcc tar make libpqxx-devel gzip && \
    yum install -y https://download.postgresql.org/pub/repos/yum/14/redhat/rhel-7-x86_64/postgresql14-libs-14.5-1PGDG.rhel7.x86_64.rpm && \
    wget https://ftp.postgresql.org/pub/source/v14.5/postgresql-14.5.tar.gz && \
	tar -xf  postgresql-14.5.tar.gz && \
	cd postgresql-14.5 && \
	./configure --with-python --without-readline --without-zlib && \
	make && \
	make install && \
	export PATH="/usr/local/pgsql/bin:$PATH" && \
    python3 -m pip install --upgrade pip setuptools wheel boto3 pygresql
답변함 일 년 전
profile picture
전문가
검토됨 16일 전
0

Hi,

may not answer directly your question, but have you considered ditching the lambda in favour of the native rds-secret manager integration released a few months ago?

https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/rds-secrets-manager.html

Let me know, regards

profile picture
전문가
답변함 일 년 전
  • This is a cool new feature I didn't know about!
    However it doesn't exactly fit our use-case. This will rotate the "master" user password but we have other users (ones we define) that we need to rotate. Thank you for the help though alatech!

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠