Self-service Developer Platform architecture on EKS with Crossplane

Question

Hello aws re:Post

I am currently designing a solution that would allow my users to be able to provision workloads on demand through a click-to-deploy interface. In order to minimize costs, I thought of an architecture as follows :

2 K8S clusters (1 on-premise and 1 on AWS)
- On-premise : Developer Platform management cluster
   - nodePool mgmt
      - Crossplane
      - Customer cluster definition
      - Catalog definition
   - nodePool tools
      - APIs
      - Prometheus
      - Grafana
      - ...

- AWS : Customer resources cluster
   - nodePool customerName (one per customer)
      - ns mgmt (created on customer register)
      - ns tenant (one per tenant : production, development, ..)

If you don't know Crossplane, you can look at the documentation here : https://crossplane.io/docs/v1.7/ but it is a bit like AWS Controllers for Kubernetes. It provides infrastructure abstraction as Kubernetes CRDs.

First, I want to get feedback on such an architecture and know if you have better ideas/suggestions.
Secondly, I would like to get an opinion on how to manage clients on the AWS side please.

Thanks in advance for your answers.

Answer

Have you reviewed this project: https://github.com/aws-samples/crossplane-aws-blueprints ? While it may not be exactly what you need, I think it will help you, nonetheless.

Answer

1. It seems your separation between tenant is based on namespaces. Depends on your data security requirements, it could be a weak separation. You do want to review your requirements, potentially implement additional security / policy controls / enforcement, and even consider cluster level separation for larger deployments, refer [here](https://kubernetes.io/blog/2021/04/15/three-tenancy-models-for-kubernetes/); 
2. Not to push for AWS consumption but in many cases from a production deployment perspective you might want to consider putting the control plane of your offering together with your data plane (i.e. both on AWS). This simplifies networking connectivity and potentially help with high availability. Depends on how much data flows between them, you can reduce data transfer cost as well which offset the cluster cost. 
3. From client management perspective, there is not much special to begin with other than the multi-tenancy consideration for a regular Kubernetes cluster(s). To get started, have you looked into this [workshop](https://weaveworks-gitops.awsworkshop.io/80_workshop_8_hybrid_ssp.html) before?

Self-service Developer Platform architecture on EKS with Crossplane

Relevant content