How do I access Amazon MSK brokers from custom ports?

Short description

For custom port configuration, you can change the advertised.listerner settings from your default ports (9092-plaintext/9094-TLS/9096-SASL-SCRAM/9098-IAM) to any port, including 443. These settings persist in Apache ZooKeeper and remain unchanged after broker restarts.

You can implement a custom port configuration through either a single port configuration or a multiport configuration. A single port configuration, for example, port 443 requires separate Network Load Balancers for each broker. A multiple port configuration uses one Network Load Balancer with different ports for each broker.

The following resolution focuses on a single port configuration approach.

Important: Amazon MSK KRaft-based clusters currently don't support dynamic modifications to advertised listeners. This is a known limitation in Apache Kafka's KRaft implementation and applies to all KRaft-based clusters, including those that run on Amazon MSK. Amazon MSK supports KRaft on versions 3.7 and higher. For use-cases that require dynamic listener updates, use ZooKeeper based clusters.

Resolution

Update the broker port

To update the port for broker 1 from 9094 to 443, run a command similar to the following example from the client machine:

./kafka-configs.sh --bootstrap-server b-1.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com:9094 --entity-type 
brokers --entity-name 1 --alter --command-config client.properties 
--add-config   
advertised.listeners=[CLIENT_SECURE://b-1.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com:443,REPLICATION://b-1-internal.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com:9093,REPLICATION_SECURE://b-1-internal.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com:9095]

Example output:

Completed updating config for entity: broker 1

To update the port for broker 2 to 443, run a command similar to the following example from the client machine:

./kafka-configs.sh --bootstrap-server b-2.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com:9094 --entity-type 
brokers --entity-name 2 --alter --command-config client.properties 
--add-config   
advertised.listeners=[CLIENT_SECURE://b-2.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com:443,REPLICATION://b-2-internal.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com:9093,REPLICATION_SECURE://b-2-internal.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com:9095]

Example output:

Completed updating config for entity: broker 2

Best practices

Review the following best practices:

• You must run the preceding commands for each broker separately from the same client machine. This configuration update doesn't require a broker restart.
• When you modify the values for REPLICATION and REPLICATION_SECURE, make sure to add -internal to the name of the broker.
• Don't replace ports 9093 and 9095 in the preceding commands.
• Use the correct values for the client field when you update the broker port configuration.
  For example, for TLS - CLIENT_SECURE, use [CLIENT_SECURE://b-1.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com:443].
  For SASL - CLIENT_SASL_SCRAM, use [CLIENT_SASL_SCRAM://b-1.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com:443].
  For IAM - CLIENT_IAM, use [CLIENT_IAM://b-1.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com:443].

To check the broker's current listener configuration, complete the following steps:

1. To launch the Apache ZooKeeper shell, run the following command:

   ./zookeeper-shell.sh ZooKeeper_connection_string

2. To list the configuration information for broker 1, run the following command:

   get /brokers/ids/1

   Example output:

   {
       "listener_security_protocol_map": {
           "REPLICATION_SECURE": "SSL",
           "CLIENT_SASL_SCRAM": "SASL_SSL",
           "REPLICATION": "PLAINTEXT",
           "CLIENT": "PLAINTEXT",
           "CLIENT_SECURE": "SSL"
       },
       "endpoints": [
           "CLIENT://b-1.testdns.xx.c14.kafka.us-east-1.amazonaws.com:9092",
           "CLIENT_SASL_SCRAM://b-1.dns.691zg3.c14.kafka.us-east-1.amazonaws.com:9096",
           "CLIENT_SECURE://b-1.testdns.xx.c14.kafka.us-east-1.amazonaws.com:9094",
           "REPLICATION://b-1-internal.testdns.xx.c14.kafka.us-east-1.amazonaws.com:9093",
           "REPLICATION_SECURE://b-1-internal.testdns.xx.c14.kafka.us-east-1.amazonaws.com:9095"
       ],
       "rack": "use1-az2",
       "jmx_port": 9099,
       "port": 9092,
       "host": "b-1.testdns.xx.c14.kafka.us-east-1.amazonaws.com",
       "version": 4,
       "timestamp": "1645081828199"
   }

Create two target groups

Create two target groups in the same Amazon Virtual Private Cloud (Amazon VPC) as your Amazon MSK cluster with the following values:

• For target type, use ip.
• For Protocol, use TCP.
• For Port, use 9094.

Then, use the IP address of the broker to register the Amazon MSK broker to the respective target group.

To get the IP address of the broker, run a command similar to the following example:

Broker 1

dig +short b-1.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com

Example output:

10.0.3.##

Broker 2

dig +short b-2.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com

Example output:

10.0.4.##

Create two Network Load Balancers

Create two Network Load Balancers with the following settings:

• For Listeners, modify the port to point to the broker's advertised port 443.
• For Default action, select the respective target group to forward traffic.
• For Mappings, be sure to select the subnets where the Amazon MSK brokers exist.

Make sure to turn on cross-zone load balancing for the Network Load Balancers that you created.

Create an Amazon Route 53 private hosted zone

Create an Amazon Route 53 private hosted zone with the following values:

• For Domain Name, enter the domain name from your broker string. For example, kafka.us-east-1.amazonaws.com.
• For VPC ID, enter the VPC ID of the clients.

Create two A-type alias records

To prevent any TLS handshake failures, create two A-type alias records with record names that are identical to the respective DNS names of the brokers.

Make sure that the inbound rules for cluster's security group allow traffic from the IP address of the Network Load Balancers to port 9094.

After you perform all the preceding steps, connect to the broker advertised port 443.

Note: This solution works for all types of access control methods that you turned on for Amazon MSK clusters.

Related information

How Goldman Sachs builds cross-account connectivity to their Amazon MSK clusters with AWS PrivateLink

Custom Amazon MSK configurations

Configure a custom domain name for your Amazon MSK cluster

Establishing Multi-tenant, Custom Domain-based Secure Connectivity to Amazon MSK through a Kafka Proxy

Consuming private Amazon API Gateway APIs using mutual TLS