Problems with Log manager component

0

Hello,

I have several Endpoints (EP) deployed in production working with the Greengrass framework. The architecture is very simple, one component is for receiving messages, and the other one is for showing messages on a screen. The components communicate through IPC. Also, I have deployed the Log Manager component, provided by AWS, for logging messages to Cloudwatch. The system has worked for 3 months with almost no problems (sometimes I have to restart the Greengrass daemon for the system to work correctly). This week one of the EP has not logged anything at all, although is connected to the internet, since I can ssh it. I have attached the log of the devices, can you help me to diagnose the problem?

2024-07-15T16:00:02.441Z [WARN] (pool-3-thread-23) com.aws.greengrass.tes.CredentialRequestHandler: Encountered error while fetching credentials. {iotCredentialsPath=/role-aliases/GreengrassV2TokenExchangeRoleAlias/credentials}
	com.aws.greengrass.deployment.exceptions.AWSIotException: Unable to get response
		at com.aws.greengrass.iot.IotCloudHelper.getHttpResponse(IotCloudHelper.java:95)
		at com.aws.greengrass.iot.IotCloudHelper.lambda$sendHttpRequest$1(IotCloudHelper.java:80)
		at com.aws.greengrass.util.BaseRetryableAccessor.retry(BaseRetryableAccessor.java:32)
		at com.aws.greengrass.iot.IotCloudHelper.sendHttpRequest(IotCloudHelper.java:81)
		at com.aws.greengrass.tes.CredentialRequestHandler.getCredentialsBypassCache(CredentialRequestHandler.java:206)
		at com.aws.greengrass.tes.CredentialRequestHandler.getCredentials(CredentialRequestHandler.java:327)
		at com.aws.greengrass.tes.CredentialRequestHandler.getAwsCredentials(CredentialRequestHandler.java:336)
		at com.aws.greengrass.tes.LazyCredentialProvider.resolveCredentials(LazyCredentialProvider.java:24)
		at software.amazon.awssdk.core.internal.util.MetricUtils.measureDuration(MetricUtils.java:54)
		at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.resolveCredentials(AwsCredentialsAuthorizationStrategy.java:100)
		at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.addCredentialsToExecutionAttributes(AwsCredentialsAuthorizationStrategy.java:77)
		at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:125)
		at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:69)
		at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:78)
		at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179)
		at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
		at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
		at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
		at software.amazon.awssdk.services.cloudwatchlogs.DefaultCloudWatchLogsClient.putLogEvents(DefaultCloudWatchLogsClient.java:2892)
		at com.aws.greengrass.logmanager.CloudWatchLogsUploader.uploadLogs(CloudWatchLogsUploader.java:137)
		at com.aws.greengrass.logmanager.CloudWatchLogsUploader.lambda$upload$0(CloudWatchLogsUploader.java:62)
		at java.base/java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1603)
		at com.aws.greengrass.logmanager.CloudWatchLogsUploader.upload(CloudWatchLogsUploader.java:61)
		at com.aws.greengrass.logmanager.LogManagerService.lambda$processLogsAndUpload$22(LogManagerService.java:773)
		at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
		at com.aws.greengrass.logmanager.LogManagerService.processLogsAndUpload(LogManagerService.java:771)
		at com.aws.greengrass.logmanager.LogManagerService.startup(LogManagerService.java:800)
		at com.aws.greengrass.lifecyclemanager.Lifecycle.lambda$handleStateTransitionStartingToRunningAsync$9(Lifecycle.java:594)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
		at java.base/java.lang.Thread.run(Thread.java:840)
	Caused by: java.net.UnknownHostException: c1j3p7nryqgokl.credentials.iot.us-west-2.amazonaws.com
		at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:801)
		at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
		at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)
		at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
		at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
		at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
		at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$DelegatingHttpClientConnectionManager.connect(ClientConnectionManagerFactory.java:86)
		at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
		at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
		at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
		at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
		at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
		at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:254)
		at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:104)
		at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:231)
		at com.aws.greengrass.iot.IotCloudHelper.getHttpResponse(IotCloudHelper.java:88)
		... 32 more

	2024-07-15T16:00:02.445Z [ERROR] (pool-3-thread-23) com.aws.greengrass.tes.CredentialRequestHandler: Error in retrieving AwsCredentials from TES. {iotCredentialsPath=/role-aliases/GreengrassV2TokenExchangeRoleAlias/credentials, credentialData=Failed to get connection}
	2024-07-15T16:00:02.445Z [ERROR] (pool-3-thread-23) com.aws.greengrass.logmanager.CloudWatchLogsUploader: Unable to upload logs for log group /aws/greengrass/UserComponent/us-west-2/com.prod.screen. {}
	software.amazon.awssdk.core.exception.SdkClientException: Failed to fetch credentials
		at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
		at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43)
		at com.aws.greengrass.tes.LazyCredentialProvider.resolveCredentials(LazyCredentialProvider.java:31)
		at software.amazon.awssdk.core.internal.util.MetricUtils.measureDuration(MetricUtils.java:54)
		at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.resolveCredentials(AwsCredentialsAuthorizationStrategy.java:100)
		at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.addCredentialsToExecutionAttributes(AwsCredentialsAuthorizationStrategy.java:77)
		at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:125)
		at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:69)
		at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:78)
		at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179)
		at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
		at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
		at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
		at software.amazon.awssdk.services.cloudwatchlogs.DefaultCloudWatchLogsClient.putLogEvents(DefaultCloudWatchLogsClient.java:2892)
		at com.aws.greengrass.logmanager.CloudWatchLogsUploader.uploadLogs(CloudWatchLogsUploader.java:137)
		at com.aws.greengrass.logmanager.CloudWatchLogsUploader.lambda$upload$0(CloudWatchLogsUploader.java:62)
		at java.base/java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1603)
		at com.aws.greengrass.logmanager.CloudWatchLogsUploader.upload(CloudWatchLogsUploader.java:61)
		at com.aws.greengrass.logmanager.LogManagerService.lambda$processLogsAndUpload$22(LogManagerService.java:773)
		at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
		at com.aws.greengrass.logmanager.LogManagerService.processLogsAndUpload(LogManagerService.java:771)
		at com.aws.greengrass.logmanager.LogManagerService.startup(LogManagerService.java:800)
		at com.aws.greengrass.lifecyclemanager.Lifecycle.lambda$handleStateTransitionStartingToRunningAsync$9(Lifecycle.java:594)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
		at java.base/java.lang.Thread.run(Thread.java:840)

	2024-07-15T16:00:03.663Z [WARN] (pool-3-thread-23) com.aws.greengrass.tes.CredentialRequestHandler: Encountered error while fetching credentials. {iotCredentialsPath=/role-aliases/GreengrassV2TokenExchangeRoleAlias/credentials}
	com.aws.greengrass.deployment.exceptions.AWSIotException: Unable to get response
		at com.aws.greengrass.iot.IotCloudHelper.getHttpResponse(IotCloudHelper.java:95)
		at com.aws.greengrass.iot.IotCloudHelper.lambda$sendHttpRequest$1(IotCloudHelper.java:80)
		at com.aws.greengrass.util.BaseRetryableAccessor.retry(BaseRetryableAccessor.java:32)
		at com.aws.greengrass.iot.IotCloudHelper.sendHttpRequest(IotCloudHelper.java:81)
		at com.aws.greengrass.tes.CredentialRequestHandler.getCredentialsBypassCache(CredentialRequestHandler.java:206)
		at com.aws.greengrass.tes.CredentialRequestHandler.getCredentials(CredentialRequestHandler.java:327)
		at com.aws.greengrass.tes.CredentialRequestHandler.getAwsCredentials(CredentialRequestHandler.java:336)
		at com.aws.greengrass.tes.LazyCredentialProvider.resolveCredentials(LazyCredentialProvider.java:24)
		at software.amazon.awssdk.core.internal.util.MetricUtils.measureDuration(MetricUtils.java:54)
		at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.resolveCredentials(AwsCredentialsAuthorizationStrategy.java:100)
		at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.addCredentialsToExecutionAttributes(AwsCredentialsAuthorizationStrategy.java:77)
		at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:125)
		at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:69)
		at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:78)
		at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179)
		at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
		at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
		at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
		at software.amazon.awssdk.services.cloudwatchlogs.DefaultCloudWatchLogsClient.putLogEvents(DefaultCloudWatchLogsClient.java:2892)
		at com.aws.greengrass.logmanager.CloudWatchLogsUploader.uploadLogs(CloudWatchLogsUploader.java:137)
		at com.aws.greengrass.logmanager.CloudWatchLogsUploader.lambda$upload$0(CloudWatchLogsUploader.java:62)
		at java.base/java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1603)
		at com.aws.greengrass.logmanager.CloudWatchLogsUploader.upload(CloudWatchLogsUploader.java:61)
		at com.aws.greengrass.logmanager.LogManagerService.lambda$processLogsAndUpload$22(LogManagerService.java:773)
		at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
		at com.aws.greengrass.logmanager.LogManagerService.processLogsAndUpload(LogManagerService.java:771)
		at com.aws.greengrass.logmanager.LogManagerService.startup(LogManagerService.java:800)
		at com.aws.greengrass.lifecyclemanager.Lifecycle.lambda$handleStateTransitionStartingToRunningAsync$9(Lifecycle.java:594)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
		at java.base/java.lang.Thread.run(Thread.java:840)
	Caused by: java.net.UnknownHostException: c1j3p7nryqgokl.credentials.iot.us-west-2.amazonaws.com
		at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:801)
		at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
		at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)
		at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
		at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
		at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
		at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$DelegatingHttpClientConnectionManager.connect(ClientConnectionManagerFactory.java:86)
		at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
		at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
		at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
		at 
asked 3 months ago100 views
1 Answer
0

Hi. It appears to be a DNS or network issue, not a Greengrass issue:

Caused by: java.net.UnknownHostException: c1j3p7nryqgokl.credentials.iot.us-west-2.amazonaws.com

Greengrass needs to refresh IAM credentials before Log Manager can write to CloudWatch.

profile pictureAWS
EXPERT
Greg_B
answered 3 months ago
profile picture
EXPERT
reviewed 3 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions