Recommended setup for Grafana, Thanos, Prometheus, and AWS split accounts

Question

The Thanos documentation says: "Put Prometheus in the same failure domain. This means same network, same datacenter as monitoring services."

But what  if the accounts are split and if Prometheus resides in one account (e.g. in the workloads OU) and Thanos, the connected S3 and Grafana are in another account (e.g. "monitoring" in the infrastructure OU)? Are these then also already considered as different failure domains? 
Should Prometheus also move to the monitoring account? *(I would not consider the latter so favorable because of the transfer performance of the metrics, if that matters at all. Shrug.)*  Would it help to have shared VPCs? What is the recommended setup and organization here? Many thanks for the help!

rePost-User-4210907 · Answer

The guidance provided in the Thanos documentation suggests placing Prometheus within the same failure domain, indicating that it should be situated in the same network and datacenter as the monitoring services. In practical terms, this means having monitoring components such as EKS clusters, EC2 instances, and a Prometheus monitoring cluster in the same failure domain. However, it is possible to have other monitoring elements like Grafana and Thanos (S3) located in different AWS accounts to consolidate monitoring metrics. One recommended approach is to have Grafana in the monitoring account and configure all prometheus clusters as data sources, enabling the consolidation of monitoring data. For more information on how to consolidate cross-account data sources in Amazon managed Grafana, you can refer to the blog mentioned below.

https://aws.amazon.com/blogs/opensource/setting-up-amazon-managed-grafana-cross-account-data-source-using-customer-managed-iam-roles/

Recommended setup for Grafana, Thanos, Prometheus, and AWS split accounts

Relevant content