[AWS IoT] Stream Manager error while Exporting data from local_streams
I'm facing this issue with Stream Manager (v2.1.12) greengrass component when my RHEL VM is rebooted. On startup (since has an HARD dependency with Token Exchange Service) didn't start correctly and it remains on "STARTING" status causing my client to fail to start. Here the logs from aws.greengrass.StreamManager:
Could anyone support me ?
- Newest
- Most votes
- Most comments
Can you also share why Token Exchange Service is not starting up correctly, may some logs from TES as well would be for insightful
This error indicates that Stream Manager is encountering an issue with sequence numbers being out of bounds, which can happen after an unexpected shutdown like a VM reboot.
The error "InvalidStreamPositionException: the sequence number is out of bounds" typically occurs when there's some data corruption or when the device shuts down unsafely or loses power. In your case, the RHEL VM reboot likely caused this issue.
For Stream Manager, this error can prevent it from starting correctly, keeping it in the "STARTING" status and causing dependent components to fail. While Stream Manager is designed to continually retry exporting data to AWS Cloud (with no maximum limit on retry attempts), in this case it appears to be stuck in the startup phase.
Here are some potential solutions:
-
If you're using an older version of Stream Manager, consider upgrading to the latest version. Note that if you're currently using version 2.0.7, you should not upgrade to versions between 2.0.8 and 2.0.11, but instead go to the latest available version.
-
Check if the Stream Manager's root folder exists and has the correct permissions. Stream Manager requires specific folder permissions to operate correctly.
-
If the issue persists, you might need to reset the Stream Manager's state by removing its stored data (though this could result in data loss for any streams that haven't been successfully exported).
-
Ensure your VM is configured for proper shutdown procedures to prevent corruption during future reboots.
Stream Manager should eventually recover as it continuously retries and the old corrupted data gets overwritten, but these steps might help resolve the issue more quickly.
Sources
AWS Greengrass StreamManager keeps logging "InvalidStreamPositionException: the sequence number is out of bounds" — Am I losing data? | AWS re:Post
Export configurations for supported AWS Cloud destinations - AWS IoT Greengrass
Manage data streams on Greengrass core devices - AWS IoT Greengrass
Troubleshooting AWS IoT Greengrass V2 - AWS IoT Greengrass
Right now we're trying to update the Stream manager component to the latest version of 2.2.1 because in the Changelog we've seen something like a bug related to : "Fixes an issue where stream manager fails to export messages to Kinesis Data Streams destinations."
Relevant content
- asked 10 months ago
Yes we've made a reboot and from TES we don't have any generated log file but from Stream manager log we found those errors :
2025-10-06T14:30:49.787Z [INFO] (Copier) aws.greengrass.StreamManager: stdout. 2025 Oct 06 16:30:49,784 [DEBUG] (pool-7-thread-1) com.amazonaws.request: Received error response: com.amazonaws.services.kinesis.model.AccessDeniedExcepti on: User: arn:aws:sts::672506786861:assumed-role/SC-672506786861-pp-duulp3hilwypi-IamRole-JLwKRDFp9CUo/6042f25161aa1ee0fd63830b3fcbd524f204f3e4cee87775a88526b70029f2b4 is not authorized to perform: kinesis:ListShards on resource: arn: aws:kinesis:eu-west-1:672506786861:stream/opcuadata because no identity-based policy allows the kinesis:ListShards action (Service: AmazonKinesis; Status Code: 400; Error Code: AccessDeniedException; Request ID: c9fef4ec-e3bd-2c8e-a8a 3-203548a19fd3; Proxy: null). {scriptName=services.aws.greengrass.StreamManager.lifecycle.startup.script, serviceName=aws.greengrass.StreamManager, currentState=STARTING}
But it's strange because if I wait about an hour, it starts working again. List.Shard permissions have been handled because I have an IAM that manages all my Greengrass devices. I only want to manage reboots because in those cases Greengrass is installed on RHEL Linux machines that sometimes need updates and security patches.
those are the logs from the beginning of the restarted log file :
2025-10-06T14:40:48.084Z [INFO] (Copier) aws.greengrass.StreamManager: stdout. 2025 Oct 06 16:40:48,084 [DEBUG] (pool-6-thread-2) com.amazonaws.iot.greengrass.streammanager.export.decider.Decider: There are no upload tasks.. {scriptNa me=services.aws.greengrass.StreamManager.lifecycle.startup.script, serviceName=aws.greengrass.StreamManager, currentState=STARTING} 2025-10-06T14:41:47.508Z [INFO] (Copier) aws.greengrass.StreamManager: stdout. 2025 Oct 06 16:41:47,503 [DEBUG] (java-sdk-http-connection-reaper) org.apache.http.impl.conn.PoolingHttpClientConnectionManager: Closing connections idle l onger than 60000 MILLISECONDS. {scriptName=services.aws.greengrass.StreamManager.lifecycle.startup.script, serviceName=aws.greengrass.StreamManager, currentState=STARTING} 2025-10-06T14:41:47.508Z [INFO] (Copier) aws.greengrass.StreamManager: stdout. 2025 Oct 06 16:41:47,503 [DEBUG] (java-sdk-http-connection-reaper) org.apache.http.impl.conn.PoolingHttpClientConnectionManager: Closing connections idle l onger than 60000 MILLISECONDS. {scriptName=services.aws.greengrass.StreamManager.lifecycle.startup.script, serviceName=aws.greengrass.StreamManager, currentState=STARTING} 2025-10-06T14:41:47.508Z [INFO] (Copier) aws.greengrass.StreamManager: stdout. 2025 Oct 06 16:41:47,503 [DEBUG] (java-sdk-http-connection-reaper) org.apache.http.impl.conn.PoolingHttpClientConnectionManager: Closing connections idle l onger than 60000
I'm working with Cristian on this issue and looking better to the log files, after a machine reboot we noticed a strange thing: we rebooted the machine at 09:25:00 but the first line of the log of StreamManager is this one
2025-10-08T10:27:21.342Z [INFO] (pool-3-thread-20) aws.greengrass.StreamManager: shell-runner-start. {scriptName=services.aws.greengrass.StreamManager.lifecycle.startup.script, serviceName=aws.greengrass.StreamManager, currentState=STARTING, command=["java -cp /greengrass/v2/packages/artifacts/aws.greengrass.StreamManager/2.2.1..."]}
as you can see the timestamp is about 30 min ahead. Instead the very next line in the same log file has the right timestamp
2025-10-08T09:25:33.275Z [INFO] (Copier) aws.greengrass.StreamManager: stdout. 2025 Oct 08 11:25:33,272 [36m[DEBUG][m (main) com.amazonaws.iot.greengrass.streammanager.utils.Asserts: Asserting: Scheduled thread pool must have more than 1 thread available to run tasks. {scriptName=services.aws.greengrass.StreamManager.lifecycle.startup.script, serviceName=aws.greengrass.StreamManager, currentState=STARTING}
I suspect the machine (for an unknown reason) starts with a wrong value for its clock and after a while chrony sets the time properly; in the meantime all greengrass components are starting and the clock is changed right under their noses. Could be this one the "real" issue?