Prometheus scrapping

Question

while scrapping and exporting the metrics awsprometheusremotewrite exporter throwing an error like "error   exporterhelper/queued_retry.go:183      Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "name": "awsprometheusremotewrite", "error": "Permanent error: remote write returned HTTP status 400 Bad Request; err = : timestamp too old: 1714982876646 metric: "current_active_requests"
", "dropped_items": 8907}" 
but for in my development env I have installed Prometheus and configured Prometheus with storage block with out of order time window : 1d, is there is any equivalent configuration that I have to make in aws side or in my adot-collector.yml configuration side?

Accepted Answer

Based on the AMP Troubleshooting page, here: https://docs.aws.amazon.com/prometheus/latest/userguide/AMP-troubleshooting.html#AMP-troubleshoot-duplicate-ordering and some other pages I was able to find online from users who have attempted to use this service (https://dev.to/authress/aws-advanced-serverless-prometheus-in-action-j1h), it appears that Amazon Managed Service for Prometheus does not yet support Out-of-order timestamps. So there's no way to configure this option when using AWS-managed Prometheus.

Checking on this official Prometheus page linked from the AWS docs: https://promlabs.com/blog/2022/12/15/understanding-duplicate-samples-and-out-of-order-timestamp-errors-in-prometheus/ and searches of the Prometheus issue tracker, it seems that the project stills considers the feature "experimental" and it may have some outstanding bugs (like https://github.com/prometheus/prometheus/issues/11496) which may be why AWS has not chosen to support it.

Instead, you'll have to fix the root cause of the timestamps at their source, by either overwriting the timestamps or fixing the misconfigurations that led to incorrect timestamps in the first place.

Prometheus scrapping

Relevant content