How to specify the endpoint of the source bucket in the copyObject method.

0

For security reasons, the endpoints of source bucket AP_NORTHEAST_2 and destination bucket US_WEST_2 are being applied to Java code. The endpoint of US_WEST_2 was applied to the existing AWS SDK copyObject, but an error occurred that could not be connected to the bucket of source AP_NORTHEAST_2. The US_WEST_2 region and the AP_NORTHEAST_2 region are not connected, so it seems that the endpoint should be applied to the source bucket. It seems that only the destination endpoint can be applied to the copyObject method. What is the solution to this situation?

EX code) AmazonS3Client s3 = getS3Client(); s3.setEndpoint("https://bucket.ex.s3.us-west-2.vpce.amazonaws.com");
s3.setRegion(Region.getRegion(Regions.US_WEST_2)); s3.copyObject(sourceBucketName, file.getKey(), destBucketName, file.getKey());

Error) 2024-05-21 05:14:33 677834128 [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-3] DEBUG o.a.h.i.c.PoolingClientConnectionManager - Connection released: [id: 6686][route: {s}->https://[AP_NORTHEAST_2_BUCKET_NAME].s3-us-west-2.amazonaws.com:443][total kept alive: 0; route allocated: 0 of 50; total allocated: 0 of 50] 2024-05-21 05:14:33 677834128 [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-3] INFO com.amazonaws.http.AmazonHttpClient - Unable to execute HTTP request: Connect timed out

1 Answer
1
Accepted Answer

Resolution

Please note that VPC endpoints don't support cross-Region requests (including copies). If you're using VPC endpoints, your source and destination buckets should be in the same AWS Region as your VPC endpoint.

If the source and destination buckets are in the same AWS Region, and you want to use VPC endpoints for the copy operation, you can create a single S3 client and configure it with the VPC endpoint for that region.

Here's an example of how you can achieve this:

Same Region with VPC Endpoint

     // Example java code 

public class S3CopyExample {
public static void main(String[] args) {
    String sourceBucketName = "source-bucket-name";
    String sourceKey = "source-object-key";
    String destBucketName = "destination-bucket-name";
    String destKey = "destination-object-key";
    String vpcEndpointUrl = "https://vpce-0123456789abcdef.s3.us-west-2.vpce.amazonaws.com";

    // Create an S3 client with the VPC endpoint configuration

    AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
            .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(vpcEndpointUrl, Regions.US_WEST_2.name()))
            .build();
    
    // Copy the object using the VPC endpoint

    CopyObjectRequest copyObjRequest = new CopyObjectRequest(sourceBucketName, sourceKey, destBucketName, destKey);
    s3Client.copyObject(copyObjRequest);
}

}

  • In this example, we create a single AmazonS3Client instance (s3Client) and configure it with the VPC endpoint for the US_WEST_2 region using the withEndpointConfiguration method. The VPC endpoint ("https://vpce-0123456789abcdef.s3.us-west-2.vpce.amazonaws.com") should be replaced with the actual VPC endpoint URL for your VPC and region.

  • Since both the source and destination buckets are in the same region (US_WEST_2), we can use this single s3Client instance to perform the copy operation using the copyObject method.

  • By configuring the S3 client with the VPC endpoint, all requests made through this client will go through the VPC endpoint, ensuring that the traffic stays within the VPC and does not go over the public internet.

Note that when using VPC endpoints, both the source and destination buckets must be in the same AWS Region as the VPC endpoint.

Optional Resolution

If you have VPC endpoints for the S3 buckets, and the source and destination buckets are in different AWS Regions, the best solution would be to use the standard S3 endpoints instead of the VPC endpoints for the cross-region copy operation.

Here's how you can achieve this:

  1. Create two separate S3 clients, one for the source bucket and one for the destination bucket, using the standard S3 endpoints.
  2. Use the copyObject method on the destination bucket client to perform the cross-region copy operation.

Here's an example implementation:

Cross-Region Without VPC Endpoint

// Example java code

public class S3CrossRegionCopyExample {
public static void main(String[] args) {
    String sourceBucketName = "source-bucket-name";
    String sourceKey = "source-object-key";
    String destBucketName = "destination-bucket-name";
    String destKey = "destination-object-key";
    
    // Create a client for the source bucket using the standard endpoint

    AmazonS3 sourceS3Client = AmazonS3ClientBuilder.standard()
            .withRegion(Regions.AP_NORTHEAST_2)
            .build();

    // Create a client for the destination bucket using the standard endpoint

    AmazonS3 destS3Client = AmazonS3ClientBuilder.standard()
            .withRegion(Regions.US_WEST_2)
            .build();

    // Copy the object using the standard endpoints

    CopyObjectRequest copyObjRequest = new CopyObjectRequest(sourceBucketName, sourceKey, destBucketName, destKey);
    destS3Client.copyObject(copyObjRequest);
}

}

  • In this example, both sourceBucketClient and destBucketClient are created using the standard S3 endpoints, which allow cross-region requests. The copyObject method is then called on the destBucketClient to perform the cross-region copy operation.

  • By using the standard S3 endpoints, you can bypass the limitation of VPC endpoints not supporting cross-region requests. However, keep in mind that using the standard S3 endpoints means your traffic will go over the public internet, which may have different security and networking implications compared to using VPC endpoints.

answered 5 months ago
  • Thank you. You've been very helpful to me. I found out what the problem was with adding VPC endpoint settings.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions