Unable to get IAM security credentials from EC2 Instance Metadata Service

I have a C#.NetCore 3.1 app which can upload, list and delete files on S3 via the AWSSDK.S3 (3.7.104.12) and AWSSDK.Extensions.NETCore.Setup (3.7.7) packages. It runs fine when I deploy it to my EC2 with proper AWS credential configured in the ~/.aws/credentials file

I can run in debug mode on Visual Studio on Windows 10 locally.

However when I deployed it on Ubuntu (test server) with the same AWS credential configured as on EC2 I get the error "Unable to get IAM security credentials from EC2 Instance Metadata Service".

Why does this happen and how do I fix it?

My code just calls ListObjects

 var listObjectsRequest = new ListObjectsV2Request
        {
            BucketName = root,
            Prefix = folderWithSlash,
            Delimiter = "/"
        };

        var listObjectsResponse = await s3Client.ListObjectsV2Async(listObjectsRequest);