Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Enabling Multicast on Amazon EKS with Isovalent Enterprise for Cilium

9 minute read
Content level: Advanced
0

A quick guide for deploying Isovalent Enterprise for Cilium on Amazon EKS to support IP multicast workloads

IP multicast is a bandwidth-efficient, one-to-many delivery model in which a single source stream is replicated by the network to many interested receivers, rather than being unicast separately to every one. It is a core part of system architectures for a number of industries, such as financial services, media or broadcasting, Web3/blockchain, and industrial, utility or energy systems.

However, since Amazon VPC does not natively forward multicast traffic, implementing multicast on Amazon EKS has traditionally required significant workarounds - provisioning and managing separate multicast ENIs, installing custom plugins, or relying on Transit Gateway multicast integrations, such as documented here.

Isovalent Enterprise for Cilium solves this challenge with Kubernetes-native multicast support by leveraging an IsovalentMulticastGroup custom resource (CRD). You declare the multicast groups once as a single Kubernetes object, and the Cilium operator watches that CRD and reconciles the group addresses into the eBPF datapath across every node. This keeps multicast configuration consistent as the cluster scales, and significantly simplifies ongoing operations.

This post walks you through deploying Isovalent Enterprise for Cilium on Amazon EKS as the primary CNI, configured specifically to support IP multicast between pods. Cilium implements multicast in its eBPF datapath and replicates traffic directly between nodes via VXLAN overlays, thus removing the dependency on native VPC multicast support or Transit Gateway requirements.

Disclaimer: Isovalent Enterprise for Cilium is a commercial offering from Isovalent/Cisco, and AWS does not provide support for it. You should consider obtaining support directly from Isovalent/Cisco before deploying it into production environments. See Alternate CNI plugins for Amazon EKS clusters for further details.

 

Architecture Overview

The diagram below provides a high-level overview of the demo environment. isovalent cilium multicast architecture

In this setup, Isovalent Enterprise for Cilium fully replaces the Amazon VPC CNI (no CNI chaining). Pods receive IP addresses from a Cilium-managed IPAM, and inter-node traffic is encapsulated in a VXLAN overlay tunnel.

Currently, Cilium multicast has the following requirements:

  • It works in tunnel routing mode only, and is currently restricted to the VXLAN tunnel protocol
  • Only IPv4 multicast is supported

For illustration purposes, I will be using the following for the demo:

  • EKS version - 1.35
  • Isovalent Enterprise for Cilium - 1.18.10
  • Overlay Pod CIDR (Cilium cluster-pool) - 10.10.0.0/16
  • Multicast group addresses - 225.0.0.10, 225.0.0.11, 225.0.0.12

 

Pre-requisites

  • An existing Amazon VPC with at least two private subnets that have outbound internet access (NAT Gateway)
  • The following tools on your workstation:
  • Access to the Isovalent Enterprise for Cilium Helm repository. (Contact your Isovalent/Cisco representative for access)

 

Walkthrough

I’ll walk you through the implementation covering the following aspects.

  1. Deploy a new EKS Cluster (Opt-Out VPC CNI)
  2. Install Isovalent Enterprise for Cilium as the CNI
  3. Add the managed node group
  4. Enable and validate multicast

 

Deploy a new EKS Cluster (Opt-Out VPC CNI)

We start with a cluster definition that contains no node group and disables the default add-ons, including disabling the VPC CNI and kube-proxy which will be replaced by the Cilium CNI. Only CoreDNS is retained.

$ cat cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: cilium-mcast-cluster
  region: <REGION>
  version: "1.35"

# Reuse an existing VPC and two private subnets
vpc:
  id: <your-vpc-id>
  subnets:
    private:
      <az_a>:
        id: <private_subnet_a_id>   # private-subnet-a 
      <az_b>:
        id: <private_subnet_b_id>    # private-subnet-b 

iam:
  withOIDC: true

# Disable the default VPC CNI and kube-proxy add-ons; keep only CoreDNS.
# Cilium will provide both the CNI and kube-proxy replacement.
addonsConfig:
  disableDefaultAddons: true
addons:
- name: coredns

Create a new EKS cluster, this provisions the EKS control plane with no worker nodes.

$ eksctl create cluster -f cluster.yaml

Once the cluster is provisioned, CoreDNS will remain Pending until nodes join - this is expected.

$ aws eks update-kubeconfig --name cilium-mcast-cluster --region <REGION>

$ kubectl get nodes
No resources found

$ kubectl get pods -A
NAMESPACE     NAME                       READY   STATUS    RESTARTS   AGE
kube-system   coredns-57765959f8-jzdkt   0/1     Pending   0          2m
kube-system   coredns-57765959f8-t5kqc   0/1     Pending   0          2m

 

Install Isovalent Cilium as the CNI

First, we need to capture the cluster API endpoint for the Cilium CNI installation.

$ ENDPOINT=$(aws eks describe-cluster --name cilium-mcast-cluster \
    --region <REGION> --query 'cluster.endpoint' --output text)
$ HOST=${ENDPOINT#https://}
$ echo $HOST
<CLUSTER_API_ENDPOINT>

The Helm values below provide minimum installation of Isovalent Enterprise for Cilium, enable the enterprise multicast feature, and configure the multicast-capable datapath: cluster-pool IPAM, VXLAN tunnelling, kube-proxy replacement. You can optionally enable additional Cilium features such as Hubble or Tetragon based on your own requirements. Set k8sServiceHost to the API endpoint captured above.

$ cat cilium-values-multicast.yaml
cluster:
  name: cilium-mcast-cluster
  id: 0

# Cluster-pool IPAM + VXLAN tunnel (required for multicast)
eni:
  enabled: false
enableIPv4Masquerade: true
ipam:
  mode: cluster-pool
  operator:
    clusterPoolIPv4PodCIDRList:
      - 10.10.0.0/16
    clusterPoolIPv4MaskSize: 24
routingMode: tunnel
tunnelProtocol: vxlan

# kube-proxy replacement (kube-proxy add-on is disabled)
kubeProxyReplacement: "true"
# provide your cluster API endpoint
k8sServiceHost: <CLUSTER_API_ENDPOINT>
k8sServicePort: 443

# Enterprise multicast feature gate
enterprise:
  featureGate:
    approved:
    - Multicast
  multicast:
    enabled: true

Add the Isovalent Helm repository and install Cilium.

$ helm repo add isovalent https://helm.isovalent.com
$ helm repo update isovalent

$ helm install cilium isovalent/cilium --version 1.18.10 \
    --namespace kube-system \
    -f cilium-values-multicast.yaml

 

Add the managed node group

Now that Cilium is installed, let's add the worker nodes. As each node joins, Cilium initializes it and the node transitions to Ready.

$ cat nodegroup.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: cilium-mcast-cluster
  region: <REGION>

managedNodeGroups:
- name: standard
  instanceType: m5.xlarge
  desiredCapacity: 4
  privateNetworking: true
  securityGroups:
    attachIDs:
      - sg-11111111aaaaaaaa   # Security Group for the node group

$ eksctl create nodegroup -f nodegroup.yaml

Verify that all nodes are showing Ready, Cilium deployment status is OK, and CoreDNS is now scheduled.

$ kubectl get nodes
NAME                                              STATUS   ROLES    AGE   VERSION
ip-10-250-4-251.ap-southeast-2.compute.internal   Ready    <none>   3m   v1.35.5-eks-3385e9b
ip-10-250-4-94.ap-southeast-2.compute.internal    Ready    <none>   3m   v1.35.5-eks-3385e9b
ip-10-250-5-166.ap-southeast-2.compute.internal   Ready    <none>   3m   v1.35.5-eks-3385e9b
ip-10-250-5-190.ap-southeast-2.compute.internal   Ready    <none>   3m   v1.35.5-eks-3385e9b

$ cilium status --wait
    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    OK
 \__/¯¯\__/    Hubble Relay:       OK
    \__/

$ cilium config view | grep -i multicast
enable-multicast   true

Confirm the Cilium operator is running.

$ kubectl get pods -n kube-system -l io.cilium/app=operator
NAME                               READY   STATUS    RESTARTS   AGE
cilium-operator-845dd79f75-j9c8x   1/1     Running   0          3m
cilium-operator-845dd79f75-qvz6c   1/1     Running   0          3m

 

Enable and validate multicast

Before pods can join multicast groups, an IsovalentMulticastGroup resource must define which group addresses are enabled in the cluster.

$ cat multicast-group.yaml
apiVersion: isovalent.com/v1alpha1
kind: IsovalentMulticastGroup
metadata:
  name: multicast-groups
  namespace: default
spec:
  groupAddrs:
    - "225.0.0.10"
    - "225.0.0.11"
    - "225.0.0.12"

$ kubectl apply -f multicast-group.yaml

Confirm the multicast groups are programmed into the eBPF datapath on a node.

$ kubectl -n kube-system exec -it ds/cilium -c cilium-agent -- \
    cilium-dbg bpf multicast group list
Group Address
225.0.0.10
225.0.0.11
225.0.0.12

Deploy a set of 4x Netshoot test pods, use a topologySpreadConstraints rule to ensure pods are spread across the 4x nodes.

$ cat netshoot.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: netshoot
spec:
  replicas: 4
  selector:
    matchLabels:
      app: netshoot
  template:
    metadata:
      labels:
        app: netshoot
    spec:
      terminationGracePeriodSeconds: 1
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: netshoot        
      containers:
      - name: netshoot
        image: nicolaka/netshoot:latest
        imagePullPolicy: Always
        command: ["sleep", "infinite"]

$ kubectl apply -f netshoot.yaml
$ kubectl get pods -o wide -l app=netshoot
NAME                        READY   STATUS    RESTARTS   AGE   IP            NODE                                              NOMINATED NODE   READINESS GATES
netshoot-58c8575577-4xwzg   1/1     Running   0          9s    10.10.2.28    ip-10-250-5-166.ap-southeast-2.compute.internal   <none>           <none>
netshoot-58c8575577-hv8fc   1/1     Running   0          9s    10.10.3.241   ip-10-250-5-190.ap-southeast-2.compute.internal   <none>           <none>
netshoot-58c8575577-n7ktn   1/1     Running   0          9s    10.10.0.71    ip-10-250-4-251.ap-southeast-2.compute.internal   <none>           <none>
netshoot-58c8575577-tfwc9   1/1     Running   0          9s    10.10.1.139   ip-10-250-4-94.ap-southeast-2.compute.internal    <none>           <none>

On two pods (via different terminal sessions), subscribe to a multicast group (225.0.0.11) using socat. Each of these acts as a receiver that joins group 225.0.0.11.

# Terminal session 1
kubectl exec -it netshoot-58c8575577-4xwzg -- \
    socat UDP4-RECVFROM:6666,reuseaddr,ip-add-membership=225.0.0.11:0.0.0.0,fork -

# Terminal session 2
kubectl exec -it netshoot-58c8575577-hv8fc -- \
    socat UDP4-RECVFROM:6666,reuseaddr,ip-add-membership=225.0.0.11:0.0.0.0,fork -

For comparison, start a plain UDP listener on a non-subscriber pod (the 3rd pod) on the same port without joining the multicast group (note the absence of ip-add-membership).

# Terminal 3 (non-subscriber pod)
kubectl exec -it netshoot-58c8575577-n7ktn -- \
    socat UDP4-RECVFROM:6666,reuseaddr,fork -

From a Cilium agent, confirm the 2x subscribers are tracked. You should see a local endpoint on the current node plus a remote-node entry for the other node that has the 2nd subscriber - confirming multicast membership is synced in the datapath. (Note: in the below output example, the ds/cilium resolves to one specific Cilium agent pod that lives on node ip-10-250-5-166.ap-southeast-2.compute.internal)

$ kubectl -n kube-system exec -it ds/cilium -c cilium-agent -- \
    cilium-dbg bpf multicast subscriber list all
Group           Subscriber      Type            
225.0.0.10      
225.0.0.11      10.10.2.28      Local Endpoint  
                10.250.5.190    Remote Node     
225.0.0.12

Now send a multicast packet from the 4th pod (on the 4th node).

# Terminal session 4
$ kubectl exec -it netshoot-58c8575577-tfwc9 -- bash -c \
    'echo "hello, this is a multicast message!" | socat -u - UDP4-DATAGRAM:225.0.0.11:6666'

You should see the first 2x subscribing pods print the received message simultaneously. The 3rd pod will not receive the message because it has not joined the multicast group.

hello, this is a multicast message!

 

Clean Up

To avoid incurring long-term charges, delete the AWS resources created as part of the demo walkthrough.

$ eksctl delete cluster --name cilium-mcast-cluster --region <REGION>

 

Conclusion

In this post, I walked you through deploying Isovalent Enterprise for Cilium as the primary CNI on Amazon EKS to support IP multicast workloads. By replacing the VPC CNI with Cilium's cluster-pool IPAM, VXLAN overlay and eBPF datapath, we enabled native in-cluster multicast - without depending on VPC multicast support or AWS Transit Gateway integrations. The IsovalentMulticastGroup CRD lets you centrally manage multicast groups and have the enterprise operator reconcile them across every node, keeping configuration consistent as the cluster scales and simplifying ongoing operations.

This pattern lets you bring multicast-dependent applications - across financial services, media, Web3, and industrial systems - onto Amazon EKS while maintaining consistent, Kubernetes-native operations.

For more information, see:

No comments

Relevant content