How to get node logs for a managed node on AWS CloudWatch or Dynatrace?

Question

Hello,
i was experimenting with AWS EKS auto mode and ApplicationNetworkPolicies and now is interested how to get consistently the events from network policy agent (var/log/aws-routed-eni/network-policy-agent.log) to easily troubleshoot what some domain(s) are blocked.
Is there a way how to say CNI to send logs to CloudWatch for managed AWS nodes?

There is a way to do it on demand https://docs.aws.amazon.com/eks/latest/userguide/auto-get-logs.html but as Karpenter can delete node its not what works well.

Thank you in advance.

Florian Turnwald · Accepted Answer

**Regarding you last comment:** You are richtig, 
enabling logMonitoring in your DynaKube configuration cannot bypass the OS-level restrictions of EKS Auto Mode. To get the network-policy-agent.log into Dynatrace, you need to decouple the host log shipping from the OneAgent. The recommended architecture is:

1. **Deploy Fluent Bit as a DaemonSet:** Set up a lightweight Fluent Bit instance across your cluster.

2. **Mount the HostPath:** Mount /var/log/aws-routed-eni/ into the Fluent Bit pods as a read-only hostPath volume.

3. **Tail the Logs:** Configure Fluent Bit's input plugin to read the network-policy-agent.log.

4. **Stream to Dynatrace API:** Use the Fluent Bit http output plugin to forward these logs directly to the

**Dynatrace Generic Log Ingest API**

(/api/v2/logs/ingest). You will need a Dynatrace API token with the logs.ingest scope for authentication.
This approach respects the immutable nature of EKS *Auto Mode:* your DynaKube handles the application monitoring side, while Fluent Bit acts as the bridge for the restricted node-level logs.

Florian Turnwald · Answer

As EKS Auto Mode uses managed instances that abstract away host-level access, traditional methods like installing the CloudWatch Agent directly on the host OS will not work. To reliably collect network-policy-agent.log events and persist them before nodes are recycled, you must use Kubernetes-native approaches.

### 1. Enable Network Policy Logs via NodeClass

In EKS Auto Mode, you shouldn't manually edit the VPC CNI configuration, as Auto Mode natively manages this. Instead, instruct Auto Mode to emit policy decision logs by updating your NodeClass custom resource:

```yaml
apiVersion: eks.amazonaws.com/v1
kind: NodeClass
metadata:
  name: my-node-class
spec:
  # Instructs the node agent to generate accept/deny logs
  networkPolicyEventLogs: Enabled

```

### 2. Stream the Logs Consistently (DaemonSet Approach)

Since the nodes are ephemeral, you must use a DaemonSet to scrape /var/log/ and continuously ship the logs off the node.

* **For AWS CloudWatch:** Install the official **Amazon CloudWatch Observability EKS Add-on**. It deploys Fluent Bit as a DaemonSet across your cluster. You can customize the fluent-bit-config ConfigMap to tail /var/log/aws-routed-eni/network-policy-agent.log and route it to your preferred CloudWatch Log Group. This seamlessly handles node churn.

* **For Dynatrace:** Deploy the **Dynatrace Operator** in your cluster. It automatically schedules the OneAgent log-ingest module as a DaemonSet across your Auto Mode nodes. It discovers and streams node-level log files to Dynatrace dynamically, entirely bypassing the need for host OS modifications.

This architecture fully respects the immutable nature of EKS Auto Mode while solving the ephemeral node lifecycle challenge.

re:Post Agent · Answer

For collecting logs from managed nodes in EKS Auto Mode, including network policy agent logs, you have several options:

**On-Demand Log Collection:**
The Node Monitoring Agent (starting with version v1.6.1-eksbuild.1) supports collecting logs directly to the node itself, rather than requiring S3. You can create a NodeDiagnostic resource with the log collection destination set to "node", which will temporarily store logs on the node for 10 minutes. There's also a kubectl plugin available for easier interaction and log collection. However, as you noted, this approach has limitations with Karpenter's node lifecycle management.

**Continuous Log Streaming to CloudWatch:**
For continuous log collection that persists beyond node lifecycle, you can use the Amazon CloudWatch agent on your managed nodes. The CloudWatch agent can be installed and configured to collect specific log files and send them to CloudWatch Logs. This provides:
- Centralized log file storage
- Quicker access to log files
- Indefinite log retention
- Real-time monitoring and searching capabilities

To set this up, you would install the CloudWatch agent on your nodes and configure it to monitor the specific log paths you're interested in (like /var/log/aws-routed-eni/network-policy-agent.log). The agent can be deployed using Systems Manager Run Command or as part of your node configuration.

This approach would work better with Karpenter's dynamic node management since logs are continuously streamed to CloudWatch before nodes are terminated, ensuring you don't lose troubleshooting data when nodes are recycled.  
**Sources**  
[Retrieve node logs for a managed node using kubectl and S3 - Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/auto-get-logs.html)  
[Usage of EKS Node Monitoring Agent for collecting node logs | AWS re:Post](https://repost.aws/articles/ARnsEZdON_TwmxWOu4JXlu9g/usage-of-eks-node-monitoring-agent-for-collecting-node-logs)  
[Sending node logs to unified CloudWatch Logs (CloudWatch agent) - AWS Systems Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/monitoring-cloudwatch-agent.html)

How to get node logs for a managed node on AWS CloudWatch or Dynatrace?

1. Enable Network Policy Logs via NodeClass

2. Stream the Logs Consistently (DaemonSet Approach)

Add your answer

Relevant content