Greengrass V2 Component install failed - DOCKER_PULL_ERROR
0
I have a set of devices that are deployed in remote locations and move around - sometimes there is an internet connection sometimes not.
I am using the docker application manager and trying to install a component that uses a docker image from ECR.
I received this deployment failed notification:
{
"RecipeFormatVersion": "2020-01-25",
"ComponentName": "myComponent",
"ComponentVersion": "1.2.0",
"ComponentType": "aws.greengrass.generic",
"ComponentDescription": "",
"ComponentPublisher": "",
"ComponentDependencies": {
"aws.greengrass.DockerApplicationManager": {
"VersionRequirement": ">=2.0.0 <2.1.0",
"DependencyType": "HARD"
},
"aws.greengrass.TokenExchangeService": {
"VersionRequirement": ">=2.0.0 <2.1.0",
"DependencyType": "HARD"
}
},
"Manifests": [
{
"Platform": {
"os": "linux"
},
"Lifecycle": {
"Run": {
"Script": "docker run --rm --log-opt max-size=10m --log-opt max-file=5 -v /greengrass/v2:/greengrass/v2 -e AWS_REGION=$AWS_REGION -e SVCUID=$SVCUID -e AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT=$AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT -e AWS_CONTAINER_AUTHORIZATION_TOKEN=$AWS_CONTAINER_AUTHORIZATION_TOKEN -e AWS_CONTAINER_CREDENTIALS_FULL_URI=$AWS_CONTAINER_CREDENTIALS_FULL_URI -e AWS_IOT_THING_NAME=$AWS_IOT_THING_NAME -e IOT_KEY_PATH=/greengrass/v2/privKey.key -e IOT_CERT_PATH=/greengrass/v2/thingCert.crt -e IOT_CA_PATH=$GG_ROOT_CA_PATH -e AWS_IOT_HOST=AWSHOST -e NODE_ENV=prod -v /greengrass/v2/ipc.socket:/greengrass/v2/ipc.socket --network host --privileged --name test region.dkr.ecr.ap-southeast-2.amazonaws.com/dockerImage:1.2.0"
},
"Shutdown": {
"Script": "docker stop"
}
},
"Artifacts": [
{
"Uri": "docker:region.dkr.ecr.ap-southeast-2.amazonaws.com/dockerImage:1.2.0",
"Unarchive": "NONE",
"Permission": {
"Read": "OWNER",
"Execute": "NONE"
}
}
]
}
],
"Lifecycle": {}
}
2023-07-31T05:30:34.288Z [INFO] (pool-2-thread-59) com.aws.greengrass.componentmanager.plugins.docker.DockerImageDownloader: get-ecr-image. task failed and will be retried. {task-attempt=1, componentIdentifier=myComponen, artifactUri=docker:region.dkr.ecr.ap-southeast-2.amazonaws.com/dockerImage:1.2.0}
com.aws.greengrass.componentmanager.plugins.docker.exceptions.ConnectionException: Network issue when docker pull - net/http: TLS handshake timeout
at com.aws.greengrass.componentmanager.plugins.docker.DefaultDockerClient.pullImage(DefaultDockerClient.java:143)
at com.aws.greengrass.componentmanager.plugins.docker.DockerImageDownloader.lambda$performDownloadSteps$2(DockerImageDownloader.java:231)
at com.aws.greengrass.util.RetryUtils.runWithRetry(RetryUtils.java:50)
at com.aws.greengrass.componentmanager.plugins.docker.DockerImageDownloader.lambda$performDownloadSteps$3(DockerImageDownloader.java:230)
at com.aws.greengrass.componentmanager.plugins.docker.DockerImageDownloader.runWithConnectionErrorCheck(DockerImageDownloader.java:294)
at com.aws.greengrass.componentmanager.plugins.docker.DockerImageDownloader.lambda$run$4(DockerImageDownloader.java:283)
at com.aws.greengrass.util.RetryUtils.runWithRetry(RetryUtils.java:50)
at com.aws.greengrass.componentmanager.plugins.docker.DockerImageDownloader.lambda$run$5(DockerImageDownloader.java:283)
at com.aws.greengrass.util.RetryUtils.runWithRetry(RetryUtils.java:50)
at com.aws.greengrass.componentmanager.plugins.docker.DockerImageDownloader.run(DockerImageDownloader.java:280)
at com.aws.greengrass.componentmanager.plugins.docker.DockerImageDownloader.performDownloadSteps(DockerImageDownloader.java:228)
at com.aws.greengrass.componentmanager.plugins.docker.DockerImageDownloader.download(DockerImageDownloader.java:103)
at com.aws.greengrass.componentmanager.ComponentManager.prepareArtifacts(ComponentManager.java:453)
at com.aws.greengrass.componentmanager.ComponentManager.preparePackage(ComponentManager.java:399)
at com.aws.greengrass.componentmanager.ComponentManager.lambda$preparePackages$1(ComponentManager.java:360)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Deployment status: Failed
Error
DOCKER_PULL_ERROR: FAILED_NO_STATE_CHANGE: Failed to download artifact name: 'docker:region.dkr.ecr.ap-southeast-2.amazonaws.com/dockerimage:1.2.0' for component myComponent-1.2.0, reason: Failed to download docker image. Unexpected error while trying to perform docker pull - dial tcp 13.239.162.67:443: connect: network is unreachable
Error code hierarchy
• DEPLOYMENT_FAILURE
• ARTIFACT_DOWNLOAD_ERROR
• DOCKER_ERROR
• DOCKER_PULL_ERROR
I understand that the device was probably offline - however I dont think the deployment should fail - it should be able to handle network loss and try again once it is connected?
Am I missing something in my setup to get the correct functionality?
preguntada hace 9 meses75 visualizacioneslg...
No hay respuestas
- Más nuevo
- Más votos
- Más comentarios
Contenido relevante
- OFICIAL DE AWSActualizada hace 10 meses
- OFICIAL DE AWSActualizada hace 3 años
- OFICIAL DE AWSActualizada hace 2 años
- OFICIAL DE AWSActualizada hace 4 meses
What version of Greengrass Nucleus are you using? Retries were relatively recently added to Docker download.
https://github.com/aws-greengrass/aws-greengrass-nucleus/releases/tag/v2.9.3
currently using v2.10.3
Thank you for letting us know. We do retry on many errors, but this particular network error was not included.
I have a pull request to include it now which can be included in our next 2.11 release. https://github.com/aws-greengrass/aws-greengrass-nucleus/pull/1507
Thanks Michael - great response time! What is the intended behaviour with this update - will it continue to retry with limited network connectivity?
Yes, it will retry forever. Even in the log you posted ("task failed and will be retried"), it shows that it is retrying on one error you had, but it gave up due to a different error type (dial tcp) which it does not currently retry on.