IoT GreenGrass Provisioning Certficate Error (Bad Endpoint Cert?)

Question

Seeing the below error which we have never encountered:

Provisioning AWS IoT resources for the device with IoT Thing Name: [0-22222-1]...
Error while trying to setup Greengrass Nucleus
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Certificate for  doesn't match any of the subject alternative names: [*.o8791rg889c4o.us-east-1.cs.amazonlightsail.com]
        at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
        at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:223)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:83)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
        at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:196)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:171)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:82)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
        at software.amazon.awssdk.services.iot.DefaultIotClient.getPolicy(DefaultIotClient.java:9158)
        at com.aws.greengrass.easysetup.DeviceProvisioningHelper.createThing(DeviceProvisioningHelper.java:205)
        at com.aws.greengrass.easysetup.GreengrassSetup.provision(GreengrassSetup.java:514)
        at com.aws.greengrass.easysetup.GreengrassSetup.performSetup(GreengrassSetup.java:325)
        at com.aws.greengrass.easysetup.GreengrassSetup.main(GreengrassSetup.java:274)
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Unable to execute HTTP request: Certificate for  doesn't match any of the subject alternative names: [*.o8791rg889c4o.us-east-1.cs.amazonlightsail.com]
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: Unable to execute HTTP request: Certificate for  doesn't match any of the subject alternative names: [*.o8791rg889c4o.us-east-1.cs.amazonlightsail.com]
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 3 failure: Unable to execute HTTP request: Certificate for  doesn't match any of the subject alternative names: [*.o8791rg889c4o.us-east-1.cs.amazonlightsail.com]
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 4 failure: Unable to execute HTTP request: Certificate for  doesn't match any of the subject alternative names: [*.o8791rg889c4o.us-east-1.cs.amazonlightsail.com]
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 5 failure: Unable to execute HTTP request: Certificate for  doesn't match any of the subject alternative names: [*.o8791rg889c4o.us-east-1.cs.amazonlightsail.com]
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 6 failure: Unable to execute HTTP request: Certificate for  doesn't match any of the subject alternative names: [*.o8791rg889c4o.us-east-1.cs.amazonlightsail.com]
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 7 failure: Unable to execute HTTP request: Certificate for  doesn't match any of the subject alternative names: [*.o8791rg889c4o.us-east-1.cs.amazonlightsail.com]
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 8 failure: Unable to execute HTTP request: Certificate for  doesn't match any of the subject alternative names: [*.o8791rg889c4o.us-east-1.cs.amazonlightsail.com]
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 9 failure: Unable to execute HTTP request: Certificate for  doesn't match any of the subject alternative names: [*.o8791rg889c4o.us-east-1.cs.amazonlightsail.com]
        Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 10 failure: Unable to execute HTTP request: Certificate for  doesn't match any of the subject alternative names: [*.o8791rg889c4o.us-east-1.cs.amazonlightsail.com]
Caused by: javax.net.ssl.SSLPeerUnverifiedException: Certificate for  doesn't match any of the subject alternative names: [*.o8791rg889c4o.us-east-1.cs.amazonlightsail.com]
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:507)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:437)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
        at software.amazon.awssdk.http.apache.internal.conn.SdkTlsSocketFactory.connectSocket(SdkTlsSocketFactory.java:77)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
        at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$DelegatingHttpClientConnectionManager.connect(ClientConnectionManagerFactory.java:86)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
        at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
        at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:254)
        at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:104)
        at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:231)
        at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:228)
        at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:67)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:77)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:56)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:39)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:72)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:52)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81)
        ... 26 more

Also seeing the below from the CLI when trying to do a manual setup which suggests and overall SSL cert issue:
aws iot add-thing-to-thing-group --thing-name $deploymentName --thing-group-name $deploymentGroup

SSL validation failed for https://iot.us-east-1.amazonaws.com/thing-groups/addThingToThingGroup hostname 'iot.us-east-1.amazonaws.com' doesn't match '*.o8791rg889c4o.us-east-1.cs.amazonlightsail.com'

Accepted Answer

Hello,

This error looks like you have an issue with your DNS, did you configure any DNS settings or /etc/hosts file for iot.us-east-1.amazonaws.com? It seems like you've configured that to point to `.o8791rg889c4o.us-east-1.cs.amazonlightsail.com` somewhere.

Cheers,

Michael

Answer

To the exact behavior that we were seeing:
* A command like "aws s3 ls" would execute successfully without issues
* The commands around "aws iot" would NOT execute successfully.

For some reason, and it is not something that we ever configured for, the DNS for the IoT endpoint was not resolving correctly. We resolved this issue by statically assigning google DNS servers on the Raspberry Pi that is hosting the IoT process.

Answer

First verify the certificate

* Run aws iot describe-endpoint --endpoint-type iot:Data-ATS.
* Inspect the returned endpointAddress. It should be something like a1234567890abc-ats.iot.us-east-1.amazonaws.com.
* 
* Use nslookup iot.us-east-1.amazonaws.com to verify that the hostname correctly resolves to an AWS IoT IP address.

and also

Verify the IoT policy associated with your device's certificate. It needs appropriate permissions for the actions your device is trying to perform, such as connecting, publishing, subscribing, and receiving messages. You can review and update the IoT policy via the AWS IoT console or CLI​

https://docs.aws.amazon.com/greengrass/v2/developerguide/device-auth.html

IoT GreenGrass Provisioning Certficate Error (Bad Endpoint Cert?)

Relevant content