Establishing Multi-tenant, Custom Domain-based Secure Connectivity to Amazon MSK through a Kafka Proxy

Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service that makes it easy for you to build and run applications that use Apache Kafka to process streaming data. It runs the open-source versions of Apache Kafka. However, Apache Kafka does not natively support multiple custom domain names for the external world of consumers and producers through the Kafka protocol. To address this, you can use an open-source Kafka proxy to establish dedicated custom domain names for your Kafka clients. Kafka Proxy is a lightweight micro-service written in the Go programming language and licensed under the Apache 2 license. It acts as an intermediate layer between Kafka clients and Amazon MSK brokers, allowing tenants to connect to the MSK cluster through custom domains.

In this blog post, we will walk through the steps to set up Kafka Proxy and establish secure, multi-tenant, custom domain-based connectivity to your Amazon MSK clusters.

Solution overview

The proxy establishes a connection to the MSK cluster, while the client connects to the proxy using its DNS name. The communication pathway between the Kafka Proxy and Amazon MSK is secured through IAM authentication. Meanwhile, the connection between the Kafka client and the Kafka Proxy is encrypted using TLS. This setup ensures secure and authenticated communication throughout the entire chain, from the client through the proxy to the MSK cluster.

The following diagram shows all components used by the solution.

In this example, we will use an Amazon Route 53 private hosted zone to manage DNS records for the custom domain (example.com). This private hosted zone is linked to the Amazon MSK VPC, facilitating DNS resolution for clients within the same VPC. If your Kafka client resides in a different VPC than the MSK cluster, you will need to associate the private hosted zone with that client's VPC as well. It is important to note that the Route 53 private hosted zone is not a mandatory component of this solution. The critical requirement is that the client can successfully resolve DNS queries for the custom domain and receive the necessary responses. Alternative approaches include using your organization's existing DNS infrastructure.

In the following sections, we go through the steps to configure Kafka Proxy and establish a connection through Kafka-Client.

Kafka Proxy setup on Amazon EC2 Instance

Follow steps mentioned in this link to create an Amazon EC2 instance.

Connect to EC2 instance and Download Kafka Proxy

curl -Ls https://github.com/grepplabs/kafka-proxy/releases/download/v0.3.10/kafka-proxy-v0.3.10-darwin-amd64.tar.gz | tar xz

Move the binary in to your PATH

sudo mv ./kafka-proxy /usr/local/bin/kafka-proxy

Create Private Hosted Zone

aws route53 create-hosted-zone \
 --name example.com \
 —vpc VPCRegion=<enter-vpc-region>, VPCId=<enter-vpc-id> \
 —caller-reference <any unique string>

Within the private hosted zone, create A record example.com pointing to private IP address of Kafka Proxy EC2 instance.

aws route53 change-resource-record-sets \
  --hosted-zone-id YOUR_HOSTED_ZONE_ID \
  --change-batch '{
    "Changes": [
      {
        "Action": "CREATE",
        "ResourceRecordSet": {
          "Name": "example.com",
          "Type": "A",
          "TTL": 300,
          "ResourceRecords": [
            {
              "Value": "YOUR_EC2_PRIVATE_IP"
            }
          ]
        }
      }
    ]
  }

Create a certificate for Kafka Proxy. Kafka Client will use this certificate when communicating to the Kafka Proxy server.

SSLCONFIG="[req]
prompt = no
distinguished_name = req_distinguished_name
x509_extensions = v3_ca
[req_distinguished_name]
C = <<Country_Code>>
ST = <<State>>
L = <<City>>
O = <<Company>>
OU = <<Organizational Unit>>
emailAddress = admin@example.com
CN = example.com
[v3_ca]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment
subjectAltName = @alternate_names
[alternate_names]
DNS.1 = tenant1.example.com
"
openssl req -x509 -newkey rsa:2048 -days 365 -nodes \
    -config <(echo "$SSLCONFIG") \
    -keyout kafkaproxy-pvt-key.pem \
    -out kafkaproxy-certificate.pem

Run Kafka Proxy on Amazon EC2 Instance

./kafka-proxy server --bootstrap-server-mapping "<Amazon MSK Bootstrap server-AZ1>:9098,tenant1.example.com:9000" \
                   --bootstrap-server-mapping "<Amazon MSK Bootstrap server-AZ2>:9098,tenant1.example.com:9001" \
                   --bootstrap-server-mapping "<Amazon MSK Bootstrap server-AZ3>:9098,tenant1.example.com:9003" \
                   --tls-enable \
                   --proxy-listener-key-file kafkaproxy-pvt-key.pem \
                   --proxy-listener-cert-file kafkaproxy-certificate.pem \
                   --proxy-listener-tls-enable \
                   --sasl-enable \
                   --sasl-method "AWS_MSK_IAM" \
                   --sasl-aws-region "<AWS-Region>" \
                   --log-level debug

Kafka Client Setup

Follow steps mentioned in this link to setup Kafka Client.

Connect to Kafka Client and copy the kafkaproxy-pvt-key.pem & kafkaproxy-certificate.pem files from the Kafka Proxy EC2 instance and put it on Kafka Client.

Create keystore

keytool -genkey -keyalg RSA -keysize 2048 -keystore kafka.client.keystore.jks -validity 300 -storepass changeit -keypass changeit -dname "CN=req_distinguished_name" -alias kafka-proxy -storetype pkcs12

openssl pkcs12 -in kafka.client.keystore.jks -nodes -nocerts -out kafkaproxy-pvt-key.pem

Add the kafka proxy listener certificate to the truststore

cp /usr/lib/jvm/jre-11-openjdk/lib/security/cacerts ./kafka.client.truststore.jks

keytool -import -trustcacerts -noprompt -alias kafka-proxy -file kafkaproxy-certificate.pem -keystore kafka.client.truststore.jks

Create client_mtls.properties file

security.protocol=SSL
ssl.truststore.location=/home/ec2-user/kafka_2.13-3.5.1/bin/kafka.client.truststore.jks
ssl.keystore.location=/home/ec2-user/kafka_2.13-3.5.1/bin/kafka.client.keystore.jks
ssl.keystore.password=changeit
ssl.key.password=changeit

Use client_mtls.properties config file to create topic, produce and consume data

Create Topic

Produce and Consume Data

Conclusion

In this blog post, we have demonstrated how to set up multi-tenant, custom domain-based secure connectivity to Amazon MSK using a Kafka Proxy. This solution enables you to utilize multiple custom domain names for external consumers and producers communicating with an Amazon MSK Cluster via the Kafka protocol. We encourage you to implement this solution and experience its benefits firsthand. If you have any questions or feedback, please don't hesitate to share them in the comments section below.