Problems with Log manager component
Hello,
I have several Endpoints (EP) deployed in production working with the Greengrass framework. The architecture is very simple, one component is for receiving messages, and the other one is for showing messages on a screen. The components communicate through IPC. Also, I have deployed the Log Manager component, provided by AWS, for logging messages to Cloudwatch. The system has worked for 3 months with almost no problems (sometimes I have to restart the Greengrass daemon for the system to work correctly). This week one of the EP has not logged anything at all, although is connected to the internet, since I can ssh it. I have attached the log of the devices, can you help me to diagnose the problem?
2024-07-15T16:00:02.441Z [WARN] (pool-3-thread-23) com.aws.greengrass.tes.CredentialRequestHandler: Encountered error while fetching credentials. {iotCredentialsPath=/role-aliases/GreengrassV2TokenExchangeRoleAlias/credentials}
com.aws.greengrass.deployment.exceptions.AWSIotException: Unable to get response
at com.aws.greengrass.iot.IotCloudHelper.getHttpResponse(IotCloudHelper.java:95)
at com.aws.greengrass.iot.IotCloudHelper.lambda$sendHttpRequest$1(IotCloudHelper.java:80)
at com.aws.greengrass.util.BaseRetryableAccessor.retry(BaseRetryableAccessor.java:32)
at com.aws.greengrass.iot.IotCloudHelper.sendHttpRequest(IotCloudHelper.java:81)
at com.aws.greengrass.tes.CredentialRequestHandler.getCredentialsBypassCache(CredentialRequestHandler.java:206)
at com.aws.greengrass.tes.CredentialRequestHandler.getCredentials(CredentialRequestHandler.java:327)
at com.aws.greengrass.tes.CredentialRequestHandler.getAwsCredentials(CredentialRequestHandler.java:336)
at com.aws.greengrass.tes.LazyCredentialProvider.resolveCredentials(LazyCredentialProvider.java:24)
at software.amazon.awssdk.core.internal.util.MetricUtils.measureDuration(MetricUtils.java:54)
at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.resolveCredentials(AwsCredentialsAuthorizationStrategy.java:100)
at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.addCredentialsToExecutionAttributes(AwsCredentialsAuthorizationStrategy.java:77)
at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:125)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:69)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:78)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
at software.amazon.awssdk.services.cloudwatchlogs.DefaultCloudWatchLogsClient.putLogEvents(DefaultCloudWatchLogsClient.java:2892)
at com.aws.greengrass.logmanager.CloudWatchLogsUploader.uploadLogs(CloudWatchLogsUploader.java:137)
at com.aws.greengrass.logmanager.CloudWatchLogsUploader.lambda$upload$0(CloudWatchLogsUploader.java:62)
at java.base/java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1603)
at com.aws.greengrass.logmanager.CloudWatchLogsUploader.upload(CloudWatchLogsUploader.java:61)
at com.aws.greengrass.logmanager.LogManagerService.lambda$processLogsAndUpload$22(LogManagerService.java:773)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.aws.greengrass.logmanager.LogManagerService.processLogsAndUpload(LogManagerService.java:771)
at com.aws.greengrass.logmanager.LogManagerService.startup(LogManagerService.java:800)
at com.aws.greengrass.lifecyclemanager.Lifecycle.lambda$handleStateTransitionStartingToRunningAsync$9(Lifecycle.java:594)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.net.UnknownHostException: c1j3p7nryqgokl.credentials.iot.us-west-2.amazonaws.com
at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:801)
at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$DelegatingHttpClientConnectionManager.connect(ClientConnectionManagerFactory.java:86)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:254)
at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:104)
at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:231)
at com.aws.greengrass.iot.IotCloudHelper.getHttpResponse(IotCloudHelper.java:88)
... 32 more
2024-07-15T16:00:02.445Z [ERROR] (pool-3-thread-23) com.aws.greengrass.tes.CredentialRequestHandler: Error in retrieving AwsCredentials from TES. {iotCredentialsPath=/role-aliases/GreengrassV2TokenExchangeRoleAlias/credentials, credentialData=Failed to get connection}
2024-07-15T16:00:02.445Z [ERROR] (pool-3-thread-23) com.aws.greengrass.logmanager.CloudWatchLogsUploader: Unable to upload logs for log group /aws/greengrass/UserComponent/us-west-2/com.prod.screen. {}
software.amazon.awssdk.core.exception.SdkClientException: Failed to fetch credentials
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43)
at com.aws.greengrass.tes.LazyCredentialProvider.resolveCredentials(LazyCredentialProvider.java:31)
at software.amazon.awssdk.core.internal.util.MetricUtils.measureDuration(MetricUtils.java:54)
at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.resolveCredentials(AwsCredentialsAuthorizationStrategy.java:100)
at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.addCredentialsToExecutionAttributes(AwsCredentialsAuthorizationStrategy.java:77)
at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:125)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:69)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:78)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
at software.amazon.awssdk.services.cloudwatchlogs.DefaultCloudWatchLogsClient.putLogEvents(DefaultCloudWatchLogsClient.java:2892)
at com.aws.greengrass.logmanager.CloudWatchLogsUploader.uploadLogs(CloudWatchLogsUploader.java:137)
at com.aws.greengrass.logmanager.CloudWatchLogsUploader.lambda$upload$0(CloudWatchLogsUploader.java:62)
at java.base/java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1603)
at com.aws.greengrass.logmanager.CloudWatchLogsUploader.upload(CloudWatchLogsUploader.java:61)
at com.aws.greengrass.logmanager.LogManagerService.lambda$processLogsAndUpload$22(LogManagerService.java:773)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.aws.greengrass.logmanager.LogManagerService.processLogsAndUpload(LogManagerService.java:771)
at com.aws.greengrass.logmanager.LogManagerService.startup(LogManagerService.java:800)
at com.aws.greengrass.lifecyclemanager.Lifecycle.lambda$handleStateTransitionStartingToRunningAsync$9(Lifecycle.java:594)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
2024-07-15T16:00:03.663Z [WARN] (pool-3-thread-23) com.aws.greengrass.tes.CredentialRequestHandler: Encountered error while fetching credentials. {iotCredentialsPath=/role-aliases/GreengrassV2TokenExchangeRoleAlias/credentials}
com.aws.greengrass.deployment.exceptions.AWSIotException: Unable to get response
at com.aws.greengrass.iot.IotCloudHelper.getHttpResponse(IotCloudHelper.java:95)
at com.aws.greengrass.iot.IotCloudHelper.lambda$sendHttpRequest$1(IotCloudHelper.java:80)
at com.aws.greengrass.util.BaseRetryableAccessor.retry(BaseRetryableAccessor.java:32)
at com.aws.greengrass.iot.IotCloudHelper.sendHttpRequest(IotCloudHelper.java:81)
at com.aws.greengrass.tes.CredentialRequestHandler.getCredentialsBypassCache(CredentialRequestHandler.java:206)
at com.aws.greengrass.tes.CredentialRequestHandler.getCredentials(CredentialRequestHandler.java:327)
at com.aws.greengrass.tes.CredentialRequestHandler.getAwsCredentials(CredentialRequestHandler.java:336)
at com.aws.greengrass.tes.LazyCredentialProvider.resolveCredentials(LazyCredentialProvider.java:24)
at software.amazon.awssdk.core.internal.util.MetricUtils.measureDuration(MetricUtils.java:54)
at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.resolveCredentials(AwsCredentialsAuthorizationStrategy.java:100)
at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.addCredentialsToExecutionAttributes(AwsCredentialsAuthorizationStrategy.java:77)
at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:125)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:69)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:78)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
at software.amazon.awssdk.services.cloudwatchlogs.DefaultCloudWatchLogsClient.putLogEvents(DefaultCloudWatchLogsClient.java:2892)
at com.aws.greengrass.logmanager.CloudWatchLogsUploader.uploadLogs(CloudWatchLogsUploader.java:137)
at com.aws.greengrass.logmanager.CloudWatchLogsUploader.lambda$upload$0(CloudWatchLogsUploader.java:62)
at java.base/java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1603)
at com.aws.greengrass.logmanager.CloudWatchLogsUploader.upload(CloudWatchLogsUploader.java:61)
at com.aws.greengrass.logmanager.LogManagerService.lambda$processLogsAndUpload$22(LogManagerService.java:773)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at com.aws.greengrass.logmanager.LogManagerService.processLogsAndUpload(LogManagerService.java:771)
at com.aws.greengrass.logmanager.LogManagerService.startup(LogManagerService.java:800)
at com.aws.greengrass.lifecyclemanager.Lifecycle.lambda$handleStateTransitionStartingToRunningAsync$9(Lifecycle.java:594)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.net.UnknownHostException: c1j3p7nryqgokl.credentials.iot.us-west-2.amazonaws.com
at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:801)
at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)
at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$DelegatingHttpClientConnectionManager.connect(ClientConnectionManagerFactory.java:86)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at
- Newest
- Most votes
- Most comments
Hi. It appears to be a DNS or network issue, not a Greengrass issue:
Caused by: java.net.UnknownHostException: c1j3p7nryqgokl.credentials.iot.us-west-2.amazonaws.com
Greengrass needs to refresh IAM credentials before Log Manager can write to CloudWatch.
Relevant content
- asked 2 years agolg...
- Accepted Answerasked 2 years agolg...
- asked 9 months agolg...
- asked 2 years agolg...
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated a year ago