Greengrass devices stuck on "Queued"

Question

I have problems with deployment of greengrassV2 devices, here the procedure I follow:
I install greengrass on a device from command line (linux) this sets up the device that is shown on aws web interface, so far so good. The device has now installed Nucleus and no error is showing on the logs. Mqtt connection is Ok.
At this point as I have a deployment active for this device (for installing aws.greengrass.Cli) but it never executes. I have tried three times creating a virtual machine and installing greengrass but all three devices are stuck on "Queued" without any kind of clue about what is happening.
Trying to resolve the issue I have revised and deleted deployments (is this a problem?).
I have a deploying job on a "Thing group" and I have to say that deployment was working some time ago as I have installed components on devices, but now it does not work anymore.
Logs on greengrass devices do not show errors or warnings, all devices status is reported on the web interface as healty and online.
Thanks for any support.

Here the **greengrass.log**:
```
 [INFO] (main) com.aws.greengrass.util.platforms.Platform: Getting platform instance com.aws.greengrass.util.platforms.unix.linux.LinuxPlatform.. {}
 [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: No ongoing deployment detected. Proceed as default. {}
 [INFO] (main) com.aws.greengrass.config.Configuration: config-loading. Read configuration from a file path. {path=/greengrass/v2/config/config.tlog}
 [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: effective-config-dump-complete. {file=/greengrass/v2/config/effectiveConfig.yaml}
 [INFO] (main) com.aws.greengrass.lifecyclemanager.KernelLifecycle: system-start. Launch Nucleus. {configPath=/greengrass/v2/config, rootPath=/greengrass/v2, version=2.12.1}
 [INFO] (main) software.amazon.awssdk.eventstreamrpc.RpcServer: IpcServer started.... {}
 [INFO] (main) com.aws.greengrass.lifecyclemanager.KernelLifecycle: serviceImplementors: {UpdateSystemPolicyService:class com.aws.greengrass.lifecyclemanager.UpdateSystemPolicyService...}. {}
 [INFO] (main) com.aws.greengrass.security.SecurityService: Register crypto key service provider. {keyType=file}
 [INFO] (main) com.aws.greengrass.security.SecurityService: Register MQTT connection security provider. {keyType=file}
 [INFO] (main) com.aws.greengrass.mqttclient.spool.Spool: Spooler has been configured. {maxSizeInBytes=2621440, storageType=Memory, keepQos0WhenOffline=false}
 [INFO] (main) com.aws.greengrass.mqttclient.spool.Spool: Memory Spooler has been set up. {}
 [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: Attempt to load service from plugins. {serviceName=aws.greengrass.Nucleus}
 [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: Attempt to load service from plugins. {serviceName=main}
 [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: Attempt to load service from plugins. {serviceName=UpdateSystemPolicyService}
 [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: Attempt to load service from plugins. {serviceName=DeploymentService}
 [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: Attempt to load service from plugins. {serviceName=TelemetryAgent}
 [INFO] (main) com.aws.greengrass.lifecyclemanager.Kernel: effective-config-dump-complete. {file=/greengrass/v2/config/effectiveConfig.yaml}
 [INFO] (main) com.aws.greengrass.lifecyclemanager.KernelLifecycle: system-start. {main=services.main:NEW}
 [INFO] (FleetStatusService-lifecycle) com.aws.greengrass.status.FleetStatusService: service-set-state. {serviceName=FleetStatusService, currentState=NEW, newState=INSTALLED}
 [INFO] (DeploymentService-lifecycle) com.aws.greengrass.deployment.DeploymentService: service-set-state. {serviceName=DeploymentService, currentState=NEW, newState=INSTALLED}
 [INFO] (aws.greengrass.Nucleus-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=aws.greengrass.Nucleus, currentState=NEW, newState=INSTALLED}
 [INFO] (TelemetryAgent-lifecycle) com.aws.greengrass.telemetry.TelemetryAgent: service-set-state. {serviceName=TelemetryAgent, currentState=NEW, newState=INSTALLED}
 [INFO] (TelemetryAgent-lifecycle) com.aws.greengrass.telemetry.TelemetryAgent: service-set-state. {serviceName=TelemetryAgent, currentState=INSTALLED, newState=STARTING}
 [INFO] (UpdateSystemPolicyService-lifecycle) com.aws.greengrass.lifecyclemanager.UpdateSystemPolicyService: service-set-state. {serviceName=UpdateSystemPolicyService, currentState=NEW, newState=INSTALLED}
 [INFO] (FleetStatusService-lifecycle) com.aws.greengrass.status.FleetStatusService: service-set-state. {serviceName=FleetStatusService, currentState=INSTALLED, newState=STARTING}
 [INFO] (TelemetryAgent-lifecycle) com.aws.greengrass.telemetry.TelemetryAgent: service-set-state. {serviceName=TelemetryAgent, currentState=STARTING, newState=RUNNING}
 [INFO] (FleetStatusService-lifecycle) com.aws.greengrass.status.FleetStatusService: Stopping backingTask waiting for dependency ready. {serviceName=FleetStatusService, currentState=STARTING}
 [INFO] (FleetStatusService-lifecycle) com.aws.greengrass.status.FleetStatusService: service-set-state. {serviceName=FleetStatusService, currentState=STARTING, newState=RUNNING}
 [INFO] (DeploymentService-lifecycle) com.aws.greengrass.deployment.DeploymentService: service-set-state. {serviceName=DeploymentService, currentState=INSTALLED, newState=STARTING}
 [INFO] (DeploymentService-lifecycle) com.aws.greengrass.deployment.DeploymentService: Stopping backingTask waiting for dependency ready. {serviceName=DeploymentService, currentState=STARTING}
 [INFO] (DeploymentService-lifecycle) com.aws.greengrass.deployment.DeploymentService: service-set-state. {serviceName=DeploymentService, currentState=STARTING, newState=RUNNING}
 [INFO] (aws.greengrass.Nucleus-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=aws.greengrass.Nucleus, currentState=INSTALLED, newState=STARTING}
 [INFO] (aws.greengrass.Nucleus-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: Stopping backingTask waiting for dependency ready. {serviceName=aws.greengrass.Nucleus, currentState=STARTING}
 [INFO] (aws.greengrass.Nucleus-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=aws.greengrass.Nucleus, currentState=STARTING, newState=FINISHED}
 [INFO] (aws.greengrass.Nucleus-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=aws.greengrass.Nucleus, currentState=FINISHED, newState=STOPPING}
 [INFO] (pool-2-thread-11) com.aws.greengrass.lifecyclemanager.GenericExternalService: generic-service-finished. Nothing done. {serviceName=aws.greengrass.Nucleus, currentState=STOPPING}
 [INFO] (pool-2-thread-8) com.aws.greengrass.lifecyclemanager.GenericExternalService: Shutdown initiated. {serviceName=aws.greengrass.Nucleus, currentState=STOPPING}
 [INFO] (pool-2-thread-8) com.aws.greengrass.lifecyclemanager.GenericExternalService: generic-service-shutdown. {serviceName=aws.greengrass.Nucleus, currentState=STOPPING}
 [INFO] (aws.greengrass.Nucleus-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: Stopping backingTask start. {serviceName=aws.greengrass.Nucleus, currentState=STOPPING}
 [INFO] (aws.greengrass.Nucleus-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=aws.greengrass.Nucleus, currentState=STOPPING, newState=FINISHED}
 [INFO] (main-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=main, currentState=NEW, newState=INSTALLED}
 [INFO] (UpdateSystemPolicyService-lifecycle) com.aws.greengrass.lifecyclemanager.UpdateSystemPolicyService: service-set-state. {serviceName=UpdateSystemPolicyService, currentState=INSTALLED, newState=STARTING}
 [INFO] (UpdateSystemPolicyService-lifecycle) com.aws.greengrass.lifecyclemanager.UpdateSystemPolicyService: Stopping backingTask waiting for dependency ready. {serviceName=UpdateSystemPolicyService, currentState=STARTING}
 [INFO] (UpdateSystemPolicyService-lifecycle) com.aws.greengrass.lifecyclemanager.UpdateSystemPolicyService: service-set-state. {serviceName=UpdateSystemPolicyService, currentState=STARTING, newState=RUNNING}
 [INFO] (main-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=main, currentState=INSTALLED, newState=STARTING}
 [INFO] (main-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: Stopping backingTask waiting for dependency ready. {serviceName=main, currentState=STARTING}
 [INFO] (pool-2-thread-11) com.aws.greengrass.lifecyclemanager.GenericExternalService: generic-service-finished. Nothing done. {serviceName=main, currentState=STARTING}
 [INFO] (main-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=main, currentState=STARTING, newState=FINISHED}
 [INFO] (main-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=main, currentState=FINISHED, newState=STOPPING}
 [INFO] (pool-2-thread-11) com.aws.greengrass.lifecyclemanager.GenericExternalService: Shutdown initiated. {serviceName=main, currentState=STOPPING}
 [INFO] (pool-2-thread-11) com.aws.greengrass.lifecyclemanager.GenericExternalService: generic-service-shutdown. {serviceName=main, currentState=STOPPING}
 [INFO] (main-lifecycle) com.aws.greengrass.lifecyclemanager.GenericExternalService: service-set-state. {serviceName=main, currentState=STOPPING, newState=FINISHED}
 [INFO] (main) com.aws.greengrass.status.FleetStatusService: fss-status-update-published. Status update published to FSS. {trigger=NUCLEUS_LAUNCH, serviceName=FleetStatusService, currentState=RUNNING}
 [INFO] (Thread-4) com.aws.greengrass.mqttclient.AwsIotMqtt5Client: Successfully connected to AWS IoT Core. {clientId=X-Gate-tmp-test-1, sessionPresent=false}
```

![Greengrass core device status](/media/postImages/original/IMebiVDgKXQU24iAd8nNz0Sw)

![Thing group](/media/postImages/original/IM_J8demHTR_eLgJ9K6hwqnw)

![Greengrass Deployment](/media/postImages/original/IMEeaIzJ8kSe-q7R5KtHzvGw)

Accepted Answer

Hi

Firstly, I want to thank you for your patience and cooperation in this issue. This is certainly a weird one. From the debug logs, I can see that your device is successfully subscribing to deployment job notifications, however, the device never receives those notifications. Greengrass will not do anything proactively if does not receive the notifications to begin with and that is why your deployment remains in QUEUED state forever.

As a last ditch effort, please confirm that GreengrassV2IoTThingPolicy on your IoT console exists and is configured correctly. You can find this policy under the Policies section in the Security drop down in the navigation panel on the left side of your AWS console. The policy should look something like

```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "iot:Connect",
        "iot:Publish",
        "iot:Subscribe",
        "iot:Receive",
        "greengrass:*"
      ],
      "Resource": "*"
    }
  ]
}
```

If the policy looks good, you should open a ticket to CTI (Category, Type, Item) AWS, IoT, Jobs for further assistance.

Siddhant

Answer

Thanks for bringing to our attention, this certainly does not look right.
Could you please confirm if the logs provided in the snippet provided are complete? If they are, your device does not seem to be subscribing to IoT Jobs as expected and that is causing it to not receive deployments. Could you please try out a couple of things for me:
1. Please look up 'IotJobsHelper' in your logs from all the 4 devices in the target thing group. Please share if you find anything.
2. Please provide the exact command used to install Greengrass on any of your device.
3. Please try to install Greengrass on a fresh device the same way you have been installing so far except this time, enable debug logs by using -Dlog.level=DEBUG in the installation command. Please provide the logs for this simulation as well.

Thanks,
Siddhant

Answer

Hi Siddhant,

> Could you please confirm if the logs provided in the snippet provided are complete?

Yes
> 1. Please look up 'IotJobsHelper' in your logs from all the 4 devices in the target thing group. Please share if you find anything.

- On one device that previously had successful deployment and now is not anymore updating, I found that doing `cat /greengrass/v2/logs/* | grep IotJobs` gives me results for days ago, not giving results for recent logs.
- On another greengrass device I created from scratch recently where I had no succesful deployment I cannot find **IotJobs**.
- In the device that I created from scratch as you requested (point 3 of your questions) I find this:
```
root@host:~# cat /greengrass/v2/logs/* | grep IotJobs
2024-01-31T10:06:06.129Z [DEBUG] (pool-2-thread-6) com.aws.greengrass.deployment.IotJobsHelper: Subscribing to Iot Jobs Topics. {ThingName=GreengrassQuickStartCore-18d5ef74879}
2024-01-31T10:06:06.129Z [DEBUG] (pool-2-thread-6) com.aws.greengrass.deployment.IotJobsHelper: Subscribing to deployment job execution update.. {ThingName=GreengrassQuickStartCore-18d5ef74879}
2024-01-31T10:06:06.537Z [DEBUG] (pool-2-thread-6) com.aws.greengrass.deployment.IotJobsHelper: Subscribed to deployment job execution update.. {ThingName=GreengrassQuickStartCore-18d5ef74879}
2024-01-31T10:06:06.537Z [DEBUG] (pool-2-thread-6) com.aws.greengrass.deployment.IotJobsHelper: Subscribing to deployment job event notifications.. {ThingName=GreengrassQuickStartCore-18d5ef74879}
2024-01-31T10:06:06.596Z [DEBUG] (pool-2-thread-6) com.aws.greengrass.deployment.IotJobsHelper: Subscribed to deployment job event notifications.. {ThingName=GreengrassQuickStartCore-18d5ef74879}
2024-01-31T10:06:06.607Z [DEBUG] (pool-2-thread-6) com.aws.greengrass.deployment.IotJobsHelper: Requesting the next deployment. {ThingName=GreengrassQuickStartCore-18d5ef74879}
```

> 2. Please provide the exact command used to install Greengrass on any of your device.

Normally I take it from management console, like this:
```
sudo -E java -Droot="/greengrass/v2" -Dlog.store=FILE -jar ./GreengrassInstaller/lib/Greengrass.jar --aws-region eu-central-1 --thing-name MyThingName --thing-group-name X-Gate --component-default-user ggc_user:ggc_group --provision true --setup-system-service true
```

> 3. Please try to install Greengrass on a fresh device the same way you have been installing so far except this time, enable debug logs by using -Dlog.level=DEBUG in the installation command. Please provide the logs for this simulation as well.

```
root@host:~# apt install -y openjdk-17-jre-headless

root@host:~# sudo -E java -Droot="/greengrass/v2" -Dlog.store=FILE -Dlog.level=DEBUG -jar ./GreengrassInstaller/lib/Greengrass.jar --aws-region eu-central-1 --thing-name GreengrassQuickStartCore-18d5ef74879 --thing-group-name X-Gate --component-default-user ggc_user:ggc_group --provision true --setup-system-service true
Provisioning AWS IoT resources for the device with IoT Thing Name: [GreengrassQuickStartCore-18d5ef74879]...
Found IoT policy "GreengrassV2IoTThingPolicy", reusing it
Creating keys and certificate...
Attaching policy to certificate...
Creating IoT Thing "GreengrassQuickStartCore-18d5ef74879"...
Attaching certificate to IoT thing...
Successfully provisioned AWS IoT resources for the device with IoT Thing Name: [GreengrassQuickStartCore-18d5ef74879]!
Adding IoT Thing [GreengrassQuickStartCore-18d5ef74879] into Thing Group: [X-Gate]...
IoT Thing Group "X-Gate" already existed, reusing it
Successfully added Thing into Thing Group: [X-Gate]
Setting up resources for aws.greengrass.TokenExchangeService ... 
Attaching TES role policy to IoT thing...
No managed IAM policy found, looking for user defined policy...
IAM policy named "GreengrassV2TokenExchangeRoleAccess" already exists. Please attach it to the IAM role if not already
Configuring Nucleus with provisioned resource details...
Downloading Root CA from "https://www.amazontrust.com/repository/AmazonRootCA1.pem"
Created device configuration
Successfully configured Nucleus with provisioned resource details!
Successfully set up Nucleus as a system service
```

Log of the device [Log of the device](https://www.dropbox.com/scl/fi/yfh3z1pef1uuqob1lrmqp/greengrass.1.log?rlkey=45rnw20ln6nz6bj223p0yeeo4&dl=0)

I have noticed some errors on Mqtt regarding the shadow of the device, is this normal? Or how can be fixed?

```
$aws/things/X-Gate-18b1dfd630b-gci/shadow/get/rejected
January 31, 2024, 11:50:19 (UTC+0100)
{
  "code": 404,
  "message": "No shadow exists with name: 'X-Gate-18b1dfd630b-gci'"
}
$aws/things/X-Gate-18b1dfd630b-gci/shadow/get
January 31, 2024, 11:51:13 (UTC+0100)
{
  "thingName": "X-Gate-18b1dfd630b-gci"
}
```

In the meanwhile the deployment execution is stil **Queued** for all devices.

Answer

Hi

Thanks for reaching out to us. From the looks of things, your Greengrass device is unable to receive the deployments you created on the Greengrass console. Could you please
1. check if your deployment targets the correct thing/thing group?
2. check if your device has a stable network connection?

Thanks,
Siddhant

Answer

Hello,

A deployment stuck on "Queued" means the device has not received the deployment. Based on your scenario, you confirmed the connection is OK by looking at MQTT connection, and could you confirm the following please?
1. When the device status is last updated? 
2. Is the thing a member of your targeted "Thing group"?

For further analysis, could you share us with your `greengrass.log` please? Please do not include any sensitive info.

Thanks

Greengrass devices stuck on "Queued"

Relevant content