EventBridge Pipes + MSK Serverless source: "requires outbound internet access"

Question

I have an EventBridge Pipe (`my-pipe`) using an MSK Serverless cluster (`my-msk-cluster`, ACTIVE) as a source, targeting a CloudWatch Logs group. It's stuck in this state:

```
PROBLEM: Pipe VPC event source requires outbound internet access to send events to Pipes
```

Per the documented PrivateLink support for this exact scenario (https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes-msk.html#pipes-msk-vpc-config), I created a `com.amazonaws..pipes-data` interface endpoint in the same VPC/subnets as the MSK cluster's own VPC connection - not a NAT Gateway or Internet Gateway, since we can't use those in this VPC by policy.

**Everything I've been able to check comes back correct:**

- `pipes-data` endpoint: state `available`, `PrivateDnsEnabled: true`, correct subnets, default (full-access) endpoint policy, and its DNS entries + Route 53 private hosted zone are properly associated with the VPC (confirmed via `route53 list-hosted-zones-by-vpc`)
- Security groups: endpoint SG allows inbound 443 from the MSK cluster's SG, and vice versa for outbound
- NACLs on both subnets: allow all traffic both directions
- VPC `enableDnsSupport`/`enableDnsHostnames`: both `true`
- IAM execution role: `kafka:DescribeCluster(V2)`, `kafka:GetBootstrapBrokers`, and the `kafka-cluster:Connect/DescribeGroup/AlterGroup/DescribeTopic/ReadData/DescribeClusterDynamicConfiguration`data-plane permissions, all scoped correctly (verified with `iam simulate-principal-policy`, every action returns `allowed` except `kafka:GetBootstrapBrokers`, which I believe is an IAM simulator limitation for that specific action rather than a real gap, since it's in the same  statement/resource as `kafka:DescribeClusterV2` which does evaluate as allowed)
- No permissions boundary on the role
- MSK cluster itself: ACTIVE, bootstrap brokers resolve fine  (`boot-abc.c3.kafka-serverless..amazonaws.com:9098`), no failed cluster   operations in its history

**The clearest evidence:** VPC Flow Logs on the `pipes-data` endpoint's ENIs, over 15+ minute windows, show **zero traffic** - no ACCEPT, no REJECT, only periodic `NODATA` heartbeats. This suggests the Pipe isn't even attempting to route through the endpoint, rather than being blocked by something.

**Reproducibility:** this isn't a stale/old pipe issue - I reproduced it on a **freshly created** pipe with no prior state. Stopping/starting the pipe, and a full delete+recreate, did not change the error.

Has anyone gotten `pipes-data` PrivateLink actually working with an **MSK Serverless** source specifically (as opposed to provisioned MSK)? Wondering if there's a gap there that isn't called out in the docs. Also happy to hear if I'm missing a step - I've been through the troubleshooting guide (https://repost.aws/knowledge-center/eventbridge-pipes-troubleshoot) and the MSK-specific deployment guide (https://repost.aws/knowledge-center/eventbridge-pipes-deployment-msk) already.

Riku_Kobayashi · Answer

Hello.

Since MSK Serverless is explicitly listed as a source in the following document, it should be possible to configure it.  
https://docs.aws.amazon.com/ja_jp/eventbridge/latest/userguide/eb-pipes-msk.html  
> You can use EventBridge Pipes to receive records from an Amazon Managed Streaming for Apache Kafka (Amazon MSK) topic, including from the Amazon MSK Serverless cluster type. You can optionally filter or enhance these records before sending them to one of the available destinations for processing.

I understand you have already consulted the following document; have you created and tested the VPC endpoint described there?  
With IAM authentication, the Kafka consumer on the EventBridge Pipes side connects to MSK using AWS credentials.  
The AWS documentation for the MSK source also states that if IAM authentication is enabled and no secret is specified, IAM is automatically selected.  
Since MSK Serverless uses IAM authentication and port 9098 in this configuration, I would first try adding an Interface VPC Endpoint for:  
`com.amazonaws..sts`  
https://repost.aws/knowledge-center/eventbridge-pipes-troubleshoot  
> To troubleshoot this issue, create Amazon Virtual Private Cloud (Amazon VPC) endpoints for the following services:  
> * For EventBridge Pipes, create an endpoint for com.amazonaws.region.pipes-data.
> * For AWS Security Token Service (AWS STS), create an endpoint for com.amazonaws.region.sts.
> * (SASL/SCRAM authentication only) For AWS Secrets Manager, create an endpoint for com.amazonaws.region.secretsmanager.
> * For Lambda, create an endpoint for com.amazonaws.region.lambda.

Takahiro Nozaki · Answer

**Potential oversight: The security group's inbound rules are not open for the broker port.**

The rules required by the official documentation are as follows:

* Inbound: Allow all traffic on the Apache Kafka broker port from the security group specified as the source.
* Outbound: Allow all traffic on port 443 to all destinations, and allow all traffic on the broker port to the security group specified as the source.

You mentioned that the "endpoint SG allows inbound traffic on port 443 from the MSK cluster SG, and the same applies to outbound traffic," but this implies only port 443 is covered, omitting the MSK Serverless broker port (9098). The documentation requires both port 443 and the broker port. If the broker port is missing, it explains why Pipes determines the path is invalid during the configuration validation stage and rejects it before attempting an actual connection.

This hypothesis is also consistent with your observation that neither ACCEPT nor REJECT entries appear in the flow logs (indicating that routing was never even attempted).

**One more point: requirements differ between public and private subnets.**

The documentation explicitly distinguishes between using a NAT Gateway (or custom NAT) for private subnets and a VPC endpoint for public subnets. Please verify which category your subnet falls into (i.e., check if the route table includes a route to an Internet Gateway). If you are using a private subnet in an environment where NAT is unavailable, there may be a discrepancy between this description and your actual requirements—potentially representing a gap in the documentation.

EventBridge Pipes + MSK Serverless source: "requires outbound internet access"

Add your answer

Relevant content