I have a spring boot app deployed on AWS EKS POD and have provisioned AWS MSK with IAM authentication they both are under the same VPC and roles has been configured as well as in MSK inbound rules the port 9098 has also being added.
To test connectivity between EKS and MSK i did telnet with broker name and port 9098 it was successfully connected as well when my run spring boot app in eks pod it gives the below error:
org.springframework.kafka.KafkaException: Send failed;nested exception in org.apache.kafka.common.errors. SaslAuthenticationException: [63a192cc-599-43e-bfe8-bc880e50c2e1]: Access Denied
org.apache. kafka.clients.Networkclient: [Producer clientId=producer-1] Connection to node -3 b-3.xxxx.xxxx.amazonaws.com/10.7.2.1:9098) failed authentication due to: [63a192cc-599-43e-bfe8-bc880e50
My spring boot kafka config:
ssl.truststore.location=path to trust file
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
Created a role in IAM and assigned the below policies to it:
{
"version": "2012-10-17",
"Statement": [
{
"Sid": "AllowMskAccessCluster",
"Effect": "Allow",
"Action": [
"kafka:ListScramSecrets",
"kafka:GetBootstrapBrokers",
"kafka:DescribeCluster",
"kafka-cluster:DescribeCluster",
"kafka-cluster:Connect",
"kafka-cluster:AlterCluster",
],
"Resource": "AWS_EKS_CLUSTER_ARN"
},
{
"Sid": "AllowMskAccessTopic",
"Effect": "Allow",
"Action": [
"kakfa-cluster:DescribeTopicDynamicConfiguration",
"kakfa-cluster:DescribeTopic",
"kakfa-cluster:DeleteTopic",
"kakfa-cluster:CreateTopic",
"kakfa-cluster:AlterTopicDynamicConfiguration",
"kakfa-cluster:AlterTopic",
],
"Resource": [
"arn:AWS_EKS_CLUSTER_ARN/*",
"*"
]
},
{
"Sid": "AllowMskAccessGroup",
"Effect": "Allow",
"Action": [
"kafka-cluster:DescribeCluster",
"kafka-cluster:DeleteGroup",
"kafka-cluster:AlterGroup",
],
"Resource": "AWS_EKS_CLUSTER_ARN/*"
}
]
}
{
"version": "2012-10-17",
"Statement": [
{
"Sid": "AllowMskAccessCluster",
"Effect": "Allow",
"Action": [
"kafka:ListScramSecrets",
"kafka:GetBootstrapBrokers",
"kafka:DescribeCluster",
"kafka-cluster:WriteDataIdempotently",
"kafka-cluster:Connect",
],
"Resource": "AWS_EKS_CLUSTER_ARN
},
{
"Sid": "AllowMskAccessTopic",
"Effect": "Allow",
"Action": [
"kakfa-cluster:WriteData",
"kakfa-cluster:DescribeTransactionalId",
"kakfa-cluster:DescribeTopic",
"kakfa-cluster:AlterTransactionalId",
],
"Resource":"*"
},
{
"Sid": "AllowMskAccessGroup",
"Effect": "Allow",
"Action": "kakfa-cluster":DescribeGroup,
"Resource": "AWS_EKS_CLUSTER_ARN/*"
}
]
}
{
"version": "2012-10-17",
"Statement": [
{
"Sid": "AllowMskAccessCluster",
"Effect": "Allow",
"Action": [
"kafka:ListScramSecrets",
"kafka:GetBootstrapBrokers",
"kafka:DescribeCluster",
"kafka-cluster:Connect",
],
"Resource": "AWS_EKS_CLUSTER_ARN"
},
{
"Sid": "AllowMskAccessTopic",
"Effect": "Allow",
"Action": [
"kakfa-cluster:ReadData",
"kakfa-cluster:DescribeTopic",
],
"Resource": "*"
},
{
"Sid": "AllowMskAccessGroup",
"Effect": "Allow",
"Action": [
"kafka-cluster:DescribeGroup",
"kafka-cluster:AlterGroup",
],
"Resource": "AWS_EKS_CLUSTER_ARN/*"
}
]
}
im using this dependencies in my spring app:
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>sts</artifactId>
<version>2.16.13</version>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>apache-client</artifactId>
<version>2.16.13</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.13</artifactId>
<version>3.0.1</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
<groupId>software.amazon.msk</groupId>
<artifactId>aws-msk-iam-auth</artifactId>
<version>1.0.0</version>
</dependency>