Bottlerocket causes significantly higher memory usage

The significant increase in memory usage you're observing when migrating from AL2 to Bottlerocket is likely related to differences in cgroup versions rather than Bottlerocket's OS design itself.

Amazon Linux 2 uses cgroup v1, while Bottlerocket (like Amazon Linux 2023) defaults to cgroup v2. This difference can cause Java applications and other workloads to behave differently in terms of memory allocation. Specifically, if your workloads include Java applications using older JDK versions (particularly JDK 8 versions prior to jdk8u372), the JVM may not be compatible with cgroup v2. When this incompatibility exists, the JVM cannot properly detect container memory limits because cgroup v2 stores memory limit data in different locations than cgroup v1. As a result, the JVM bases its heap allocation on the node's total memory rather than the pod's defined memory limit, which can lead to much higher memory consumption and even Out of Memory (OOM) exceptions.

To address this issue, you have several options:

1. Upgrade your Java version: If you're running Java workloads, upgrade to JDK 8u372 or later, or move to a newer JDK version with full cgroup v2 support.

2. Enable cgroup v1 on Bottlerocket: You can temporarily revert to cgroup v1 for compatibility with legacy workloads by setting the appropriate kernel parameters in your Bottlerocket configuration. This can help maintain compatibility while you work on longer-term solutions.

3. Review and adjust resource limits: Ensure your pod memory requests and limits are properly configured in your Kubernetes manifests, as the way containers consume resources may differ between cgroup versions.

The behavior you're seeing is not inherent to Bottlerocket's container-optimized design but rather stems from the cgroup version difference and how your applications interact with resource limits under the new system.
Sources
Guide to EKS AL2 & AL2-Accelerated AMIs transition features - Amazon EKS
Limiting process resource usage in AL2023 using cgroups - Amazon Linux 2023
How to Enable cgroup v1 for Kubernetes 1.34 Bottlerocket AMIs—and What’s Different from 1.33 | AWS re:Post