org.apache.kafka.common.errors.SaslAuthenticationException: Cannot change principals during re-authentication from IAM

The error you're encountering is related to IAM authentication with Amazon MSK. The specific error "Cannot change principals during re-authentication from IAM" suggests that the IAM principal (role) being used for authentication is changing during the connection's lifecycle.

Looking at your error message, there appears to be a mismatch between two different assumed roles:

• IAM.arn:aws:sts::######:assumed-role/NodeInstanceRole/dev-dr-settingsen-c220a744-8ea2-4d19-bda5-94f2b2e871d9
• IAM.arn:aws:sts::######:assumed-role/NodeInstanceRole/dev-dr-settingsen-3d131426-9e29-48d3-aabe-8f686c83a990

This issue typically occurs in containerized environments like EKS when:

1. The application is running on different nodes with different instance roles
2. The pod is being rescheduled to a different node during credential refresh
3. The temporary credentials are being rotated, but the new credentials are associated with a different session ID

When using AWS_MSK_IAM authentication, the Kafka client maintains a persistent connection to the MSK broker. During credential refresh (which happens automatically), the authentication protocol expects the same principal identity to be maintained. If the principal changes (as indicated by the different session IDs in your error), the broker rejects the authentication attempt.

To resolve this issue, you could:

1. Ensure consistent IAM roles across all nodes in your EKS cluster
2. Use a dedicated IAM role for your application (via service accounts and IRSA - IAM Roles for Service Accounts)
3. Implement proper handling of connection failures and reconnection logic in your application
4. Consider adjusting your consumer configuration to handle reconnection more gracefully

The configuration you've provided looks correct for AWS MSK IAM authentication, but the issue is related to the identity consistency during credential refresh rather than the configuration itself.
Sources
KafkaClusterClientAuthenticationDescription - Amazon MSK Connect
KafkaClusterClientAuthentication - Amazon MSK Connect
Authentication and authorization for Apache Kafka APIs - Amazon Managed Streaming for Apache Kafka