Issues with Access Logs on Network type of ALB: Seeking Solutions

0

Dear Community,

I hope this message finds you well. I am currently experiencing an issue with enabling access logs for a Network Application Load Balancer (ALB) that was created via the kubectl API when setting up an Ingress. I have carefully followed the instructions provided in Enable access logs for your Network Load Balancer to grant access to the S3 bucket where the logs will be stored. However, upon attempting to access the logs, I encounter the following error message:

"Access Denied for bucket: alb-inter-devps. Please check S3 bucket permission."

Unfortunately, I am unable to determine which permission might be missing. Below are the relevant details regarding my configuration:

ALB Details:

  • Load Balancer Type: Network
  • IP Address Type: IPv4
  • Availability Zones: 3 in Oregon
  • Scheme: Internal
  • DNS Name: k8s By AWS (k8s[string].elb.[region].amazonaws.com - A Record)

S3 Bucket Details:

  • Region: us-west-2 (Oregon)
  • Bucket Versioning: Disabled
  • Encryption Type: Server-side encryption with Amazon S3 managed keys (SSE-S3)
  • Bucket Policy: see below
  • Public Access Block: Enabled
#### Bucket Policy

{
    "Version": "2012-10-17",
    "Id": "AWSLogDeliveryWrite",
    "Statement": [
        {
            "Sid": "AWSLogDeliveryAclCheck",
            "Effect": "Allow",
            "Principal": {
                "Service": "delivery.logs.amazonaws.com"
            },
            "Action": "s3:GetBucketAcl",
            "Resource": "arn:aws:s3:::[destination-bucket]",
            "Condition": {
                "StringEquals": {
                    "aws:SourceAccount": ["My Account ID"]
                },
                "ArnLike": {
                    "aws:SourceArn": ["arn:aws:logs:[oregon]:[My Account ID]:*"]
                }
            }
        },
        {
            "Sid": "AWSLogDeliveryWrite",
            "Effect": "Allow",
            "Principal": {
                "Service": "delivery.logs.amazonaws.com"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::[destination-bucket]/AWSLogs/[My Account ID]/*",
            "Condition": {
                "StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control",
                    "aws:SourceAccount": ["[My Account ID]"]
                },
                "ArnLike": {
                    "aws:SourceArn": ["arn:aws:logs:[oregon]:[My Account ID]:*"]
                }
            }
        }
    ]
}

Enter image description here

Additionally, I have also tested the configuration outlined in Access logs for your Application Load Balancer, but unfortunately, the issue persists.

Bucket Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::[Oregon ID]:root"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::[destination-bucket]/AWSLogs/[My Account ID]/*"
        }
    ]
}

Enter image description here

And also I have also tested a mixed configuration of the Bucket Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::[Oregon ID]:root"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::[destination-bucket]/AWSLogs/[My Account ID]/*"
        },
        {
            "Sid": "AWSLogDeliveryAclCheck",
            "Effect": "Allow",
            "Principal": {
                "Service": "delivery.logs.amazonaws.com"
            },
            "Action": "s3:GetBucketAcl",
            "Resource": "arn:aws:s3:::[destination-bucket]",
            "Condition": {
                "StringEquals": {
                    "aws:SourceAccount": ["My Account ID"]
                },
                "ArnLike": {
                    "aws:SourceArn": ["arn:aws:logs:[oregon]:[My Account ID]:*"]
                }
            }
        },
        {
            "Sid": "AWSLogDeliveryWrite",
            "Effect": "Allow",
            "Principal": {
                "Service": "delivery.logs.amazonaws.com"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::[destination-bucket]/AWSLogs/[My Account ID]/*",
            "Condition": {
                "StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control",
                    "aws:SourceAccount": ["[My Account ID]"]
                },
                "ArnLike": {
                    "aws:SourceArn": ["arn:aws:logs:[oregon]:[My Account ID]:*"]
                }
            }
        }
    ]
}

(2024-10-21 21:44 UTC Time Update)

And we also try this configuration by Gary Mclean (See in the answers). But We get the same error

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::797873946194:root"
      },
      "Action": "s3:PutObject",
      "Resource": "[destination-bucket]/*"
    }
  ]
}

I would greatly appreciate any guidance or advice you could provide to help resolve this issue. Thank you in advance for your assistance!

  • I see the issue. Writing answer

4 Answers
0

There are a few issues with your current configuration that are likely causing the "Access Denied" error for your Network Load Balancer (NLB) access logs:

  1. Principal Service: For NLB access logs, the correct principal service is "logdelivery.elb.amazonaws.com", not "delivery.logs.amazonaws.com" which is used for other AWS services.

  2. Bucket Policy: The bucket policy for NLB access logs should be simpler than what you've provided. It should grant permissions to the Elastic Load Balancing service to write logs to your S3 bucket.

  3. Encryption: While you're using SSE-S3 encryption, which is generally fine, for Network Load Balancer access logs specifically, if you want to use AWS KMS encryption (SSE-KMS), you must use customer managed keys.

Here's what you should do to resolve the issue:

  1. Modify your S3 bucket policy to the following:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "logdelivery.elb.amazonaws.com"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::[your-bucket-name]/AWSLogs/[your-aws-account-id]/*",
"Condition": {
"StringEquals": {
"s3:x-amz-acl": "bucket-owner-full-control"
}
}
}
]
}
  1. Ensure that your S3 bucket is in the same region as your Network Load Balancer.

  2. Make sure the S3 bucket name doesn't include "AWSLogs" in its prefix.

  3. When enabling access logs for your NLB, specify the correct S3 bucket name and prefix (if any) where you want the logs to be stored.

  4. If you're still encountering issues, double-check that the Elastic Load Balancing service has the necessary permissions to write to your S3 bucket.

If you've followed these steps and are still experiencing issues, you may need to contact AWS support for further assistance, as there could be account-specific factors at play.
Sources
Enable access logs for your Network Load Balancer - Elastic Load Balancing
AwsElbLoadBalancerAccessLog - AWS Security Hub
AccessLog - Elastic Load Balancing

profile picture
answered 2 months ago
  • This is ok for Regions available as of August 2022 or later. us-west-2 is pre 2022 and has a different policy per my answer

  • Thank you very much for your prompt response. We have implemented the suggested changes; however, unfortunately, it did not yield the expected results. I have added images of the policies required by the AWS documentation to the original post to provide better context regarding why the policy may seem complex.

    Once again, thank you for your prompt response.

0

You are in the reigon which was available pre 2022 and has a different type of bucket policy. Heres the policy you need on the bucket

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/enable-access-logging.html#attach-bucket-policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::797873946194:root"
      },
      "Action": "s3:PutObject",
      "Resource": "[destination-bucket]/*"
    }
  ]
}

Replace [destination-bucket] with your bucket. The account ID changes depending on which region your ALB is in. I have the account number in the policy for your region.

profile picture
EXPERT
answered 2 months ago
  • Thank you for your swift reply. We try with you version, but We still get the same error message.

0

There are two kinds of modern load balancers that process traffic at the connection or request level:

  1. Network Load Balancers (NLB)
  2. Application Load Balancers (ALB)

You wrote that your load balancer is an ALB but also that the "type" field shows "Network" instead of "Application." That means it is not an ALB. It is an NLB.

NLBs primarily operate at the levels of the TCP and UDP protocols, with the option of performing TLS termination and re-encryption as an added layer on top of TCP connections.

The NLB won't process potential HTTP requests inside the traffic it forwards, and the connection doesn't even need to contain HTTP requests and responses. You can send anything TCP- or UDP-based through an NLB, such as raw database connections (PostgreSQL, MySQL, etc.), LDAP traffic to Active Directory or OpenLDAP servers, or any other TCP-, UDP- or TLS-based traffic. The NLB does have a logging option, but those logs only record TLS encryption details on traffic that the NLB decrypts, not HTTP request level information.

To record HTTP request details, you need to replace your load balancer with an ALB and enable request logging on it: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html.

If you need static IP addresses for the load balancer, you can place an NLB in front of the ALB, so that all end user traffic arrives on the NLB and gets forwarded to the ALB for processing at the HTTP (and optionally TLS encryption) layer.

For more detailed logging, you can also associate a Web Application Firewall v2 web ACL with your ALB. It is able to collect all HTTP request headers and other detailed pieces of information, which aren't included in basic ALB access logs: https://docs.aws.amazon.com/waf/latest/developerguide/logging-fields.html

EXPERT
answered 2 months ago
0

You cant have an AWS service principal in your own account. This is not valid. The service integration delivery.logs.amazonaws.com doesnt support this global condition key:

            "Principal": {
                "Service": "delivery.logs.amazonaws.com"
            },
            "Action": "s3:GetBucketAcl",
            "Resource": "arn:aws:s3:::[destination-bucket]",
            "Condition": {
                "StringEquals": {
                    "aws:SourceAccount": ["My Account ID"]

Lets strip it right back. Try this one.

{
  "Version": "2012-10-17",
  "Statement": [
   {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::797873946194:root"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::[destination-bucket]/*"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "delivery.logs.amazonaws.com"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::[destination-bucket]/*",
            "Condition": {
                "StringEquals": {
                    "s3:x-amz-acl": "bucket-owner-full-control"
                }
            }
        },
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "delivery.logs.amazonaws.com"
            },
            "Action": "s3:GetBucketAcl",
            "Resource": "arn:aws:s3:::[destination-bucket]"
        }
  ]
}
profile picture
EXPERT
answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions