Introduction

Organizations today rely on a combination of relational and non-relational databases to meet their diverse data storage and management needs. If you use Kubernetes to manage containers, then connecting your relational and non-relational databases to your Kubernetes clusters is essential to build scalable and resilient applications.

When you connect Amazon Neptune and Amazon DynamoDB to Amazon EKS, you can build modern, data-driven applications that easily scale. Integrating Amazon EKS with Neptune and DynamoDB allows Kubernetes pods to securely and efficiently access necessary data sources.

In this AWS Support Official article, we provide guidance for integration using a flexible approach to secure access management. We demonstrate how to use AWS Identity and Access Management (IAM) Roles for Service Accounts (IRSA) and Role-Based Access Control (RBAC) mechanisms, following best practices for authentication, connectivity, and performance.

Solution overview

Connect Neptune and Amazon EKS

Neptune DB clusters reside in an Amazon Virtual Private Cloud (Amazon VPC). To access Neptune endpoints, you must configure your Amazon VPC to enable connectivity between Amazon EKS and Neptune.

To access Neptune from Amazon EKS, you must set up an AWS Identity and Access Management (IAM) role with the necessary permissions.

Depending on your requirements and networking preferences, there are a few ways to configure your Amazon VPC to connect between Neptune and Amazon EKS.

Prerequisites to connect Neptune and Amazon EKS

Before you connect Neptune and Amazon EKS, check that you have the following prerequisites:

• An active AWS account.
• Familiarity with using Amazon EKS to manage Kubernetes clusters on AWS.
• Familiarity with using Amazon Neptune to create and manage Neptune database clusters.
• An EKS cluster to deploy your applications.
• A Neptune DB cluster that's in the Available status. The security group of the Neptune DB cluster must have an allow inbound rule on port 8182 from the source security group that's associated with the EKS cluster.
• The latest version of kubectl (v1.29.1). To check your version, run:

kubectl version

• The latest version of eksctl (0.164.0). To check your version, run:

eksctl info

• The latest version of the AWS Command Line Interface (AWS CLI). To check your version, run:

aws –version

• An IAM OpenID Connect (OIDC) provider.

Note: In the following procedure, the Neptune database is named db-neptune-1 and uses the Neptune DB cluster endpoint db-neptune-1.cluster-111122223333.us-west-2.neptune.amazonaws.com/.

Solution implementation

Set the environment variables

Before you use Helm or other command-line tools with your Amazon EKS cluster, you must define specific environment variables that summarize your cluster's details. Use these variables in subsequent commands to make sure that they target the correct cluster and resources.

To set the environment variables, complete the following steps:

1. Confirm that you're operating within the correct cluster context so that subsequent commands are sent to the intended Kubernetes cluster:

kubectl config current-contextD

2. Define the CLUSTER_NAME environment variable for your EKS cluster:

export CLUSTER_NAME=eks-workshop

Note: In the preceding command, replace the example cluster name with your cluster's name.

3. Run the following command to verify that the variable is correctly set:

echo $CLUSTER_NAME

Verify that the output matches your inputs.

Create an IAM role and associate it with your Kubernetes service account

Use IRSA to map your Kubernetes service accounts to IAM roles for fine-grained permissions to your applications that run on EKS. Use eksctl to create and associate an IAM role with a Kubernetes service account in your EKS cluster. Use the AWS managed policy NeptuneReadOnlyAccess to allow reads for all your Neptune clusters.

As a policy best practice, use the AWS managed policies in your AWS account to validate the required permissions for your role and workflows. To modify the permissions for your use case, use IAM Access Analyzer with custom policies. For more information, see AWS managed policies or AWS managed policies for job functions.

To create an IAM role and associate it with your Kubernetes service account, complete the following steps:

1. Create a service account that's associated with NeptuneReadOnlyAccess:

eksctl create iamserviceaccount --name eks-neptune-sa --namespace default --cluster $CLUSTER_NAME --attach-policy-arn arn:aws:iam::aws:policy/NeptuneReadOnlyAccess --approve --override-existing-serviceaccounts

You receive the following output:

2024-02-07 01:12:39 [ℹ] created serviceaccount "default/eks-neptune-sa"

2. Confirm that the eks-neptune-sa service account is correctly set up in the default namespace in your cluster:

kubectl get sa eks-neptune-sa -o yaml

You receive the following output in YAML format:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/eksctl-eks-workshop-addon-iamserviceaccount-d-Role1-Q35yKgdQOlmM
  creationTimestamp: "2024-02-07T01:12:39Z"
  labels:
    app.kubernetes.io/managed-by: eksctl
  name: eks-neptune-sa
  namespace: default
  resourceVersion: "5174750"
  uid: cd6ba2f7-a0f5-40e1-a6f4-4081e0042316

3. In the service account, use the following configuration file to deploy a sample pod:

apiVersion: v1
kind: Pod
metadata:
  name: neptune-util
  namespace: default
spec:
  serviceAccountName: eks-neptune-sa
  containers:
  - name: neptune-util
    image: public.ecr.aws/patrickc/troubleshoot-util
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent

4. Save the configuration file as neptune-util.yaml, and then run the following command:

kubectl apply -f neptune-util.yaml

5. To check the connection between Neptune and Amazon EKS, run the following exec command to use SSH to connect to the pod:

kubectl exec --stdin --tty neptune-util -- /bin/bash

Note: The Neptune DB endpoint is dependent on the AWS Region where you created the Neptune DB cluster.

6. Run the following curl command:

bash-5.1# curl -X POST -d '{"gremlin":"g.V().limit(1)"}' https://db-neptune-1.cluster-111122223333.us-west-2.neptune.amazonaws.com:8182/gremlin

Note: In the preceding command, replace the example Neptune database endpoint with your endpoint.

You receive the following output:

{"requestId":"a4964f2d-12b1-4ed3-8a14-eff511431a0e","status":{"message":"","code":200,"attributes":{"@type":"g:Map","@value":[]}},"result":{"data":{"@type":"g:List","@value":[]},"meta":{"@type":"g:Map","@value":[]}}}

7. Run the following command to exit from the pod:

bash-5.1# exit

Confirm the connectivity between Neptune and Amazon EKS

By default, IAM database authentication is disabled when you create a Neptune DB cluster. To enable IAM database authentication, see Enabling IAM database authentication in Neptune.

To confirm the connectivity between Neptune and Amazon EKS, complete the following steps:

1. Before you use SSH to connect to a running container, run the following command to confirm that the pod is in the Running status.

kubectl get pods

You receive the following output:

NAME READY STATUS RESTARTS AGE
neptune-util 1/1 Running 0 50m

2. To check the connectivity between Neptune and Amazon EKS, run the following exec command to use SSH to connect the pod:

kubectl exec --stdin --tty neptune-util -- /bin/bash

3. Run the following Python command to install the awscurl command line tool:

bash-5.1#pip3 install awscurl

You receive the following output:

Installing collected packages: idna, configparser, configargparse, charset-normalizer, certifi, requests, awscurl
Successfully installed awscurl-0.32 certifi-2024.2.2 charset-normalizer-3.3.2 configargparse-1.7 configparser-6.0.0 idna-3.6 requests-2.31.0

4. Run the following awscurl command:

bash-5.1# awscurl https://db-neptune-1.cluster-111122223333.us-west-2.neptune.amazonaws.com:8182/status --region us-west-2 --service neptune-db

Note: In the preceding command, replace the example Neptune database endpoint with your endpoint.

You receive the following output:

{"status":"healthy","startTime":"Thu Feb 08 01:22:14 UTC 2024","dbEngineVersion":"1.3.0.0.R1","role":"writer","dfeQueryEngine":"viaQueryHint","gremlin":{"version":"tinkerpop-3.6.4"},"sparql":{"version":"sparql-1.1"},"opencypher":{"version":"Neptune-9.0.20190305-1.0"},"labMode":{"ObjectIndex":"disabled","ReadWriteConflictDetection":"enabled"},"features":{"SlowQueryLogs":"disabled","ResultCache":{"status":"disabled"},"IAMAuthentication":"enabled","Streams":"disabled","AuditLog":"disabled"},"settings":{"clusterQueryTimeoutInMs":"120000","SlowQueryLogsThreshold":"5000"}}

Connect DynamoDB and Amazon EKS

Use Amazon EKS to assign IAM permissions to Kubernetes service accounts that allow pod-level access to various AWS services. Because Dynamo DB doesn't use typical credential-based authentication, these IAM permissions make sure that the pods have access to only the necessary DynamoDB resources.

Prerequisites to connect DynamoDB and Amazon EKS

Before you connect DynamoDB and Amazon EKS, check that you have the following prerequisites:

• An active AWS Account.
• Familiarity with DynamoDB to create and manage DynamoDB tables.
• A DynamoDB table that's in the Active status.
• The latest version of kubectl (v1.29.1). To check your version, run: kubectl version.
• The latest version of eksctl (0.164.0). To check your version, run: eksctl info.
• The latest version of the AWS Command Line Interface (AWS CLI). To check your version, run: aws –version.
• An IAM OIDC provider.

Note: In the following procedure, a DynamoDB table was created in the target account and is named "ContainerTable".

Set the environment variables

Complete the following steps:

1. Confirm that you're operating within the correct cluster context so that subsequent commands are sent to the intended Kubernetes cluster:

kubectl config current-context

2. Define the CLUSTER_NAME environment variable for your EKS cluster:

export CLUSTER_NAME=eks-workshop

Note: In the preceding command, replace the example cluster name with your cluster's name.

3. To confirm that the variable is correctly set, run the following command:

echo $CLUSTER_NAME

Verify that the output matches your inputs.

Create an IAM role and associate it with your Kubernetes service account

Use IRSA to map your Kubernetes service accounts to IAM roles for fine-grained permissions to your applications that run on Amazon EKS. Use eksctl to create and associate an IAM role with a Kubernetes service account in your EKS cluster. Use the AWS managed policy AmazonDynamoDBReadOnlyAccess to allow read access for your DynamoDB tables.

When you connect to a DynamoDB table, make sure to follow IAM policy best practices.

To create an IAM role and associate it with your Kubernetes service account, complete the following steps:

1. Run the following command to create a service account that associates with AmazonDynamoDBReadOnlyAccess:

eksctl create iamserviceaccount —name eks-dynamo-sa —namespace default —cluster $CLUSTER_NAME —attach-policy-arn arn:aws:iam::aws:policy/AmazonDynamoDBReadOnlyAccess —approve —override-existing-serviceaccounts

You receive the following response output:

2024-02-21 03:23:57 [ℹ] created serviceaccount "default/eks-dynamo-sa"

2. Run the following command to make sure that the "eks-dynamo-sa" service account is correctly set up in the default namespace in your cluster:

kubectl get sa eks-dynamo-sa -o yaml
You receive the following output in YAML format:
kubectl get sa eks-dynamo-sa -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/eksctl-eks-workshop-addon-iamserviceaccount-d-Role1-jHyvNq1M56Vf
  creationTimestamp: "2024-02-21T03:23:57Z"
  labels:
    app.kubernetes.io/managed-by: eksctl
  name: eks-dynamo-sa
  namespace: default
  resourceVersion: "9164325"
  uid: ae678d9d-b5c2-49c6-803a-a23163569b4b

3. Deploy a sample pod with the following definition saved as dynamo-util.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: python-cli
spec:
  serviceAccountName: eks-dynamo-sa
  containers:
  name: python-cli
    image: python:3.8
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

4. Run the following command to deploy a pod that's named python-cli:

kubectl apply -f dynamo-util.yaml

Confirm the connectivity between Amazon EKS and DynamoDB

Complete the following steps:

1. Before you use SSH to connect to the running container, run the following command to confirm that the pod is in the Running status:

kubectl get pods

You receive the following output:

Name: Ready, Status: RESTARTS AGE
python-cli 1/1 Running 0 50m

2. To check the connectivity between Amazon DynamoDB and Amazon EKS, run the following exec command to use SSH to connect to the pod:

kubectl exec —stdin —tty python-cli — /bin/bash

3. Run the following Python command to install Boto3:

root@python-cli:/# pip3 install boto3

You receive the following output:

Collecting boto3
Downloading boto3-1.34.46-py3-none-any.whl (139 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.3/139.3 kB 3.4 MB/s eta 0:00:00
Collecting s3transfer<0.11.0,>=0.10.0
Downloading s3transfer-0.10.0-py3-none-any.whl (82 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 82.1 kB 18.5 MB/s eta 0:00:00
Collecting botocore<1.35.0,>=1.34.46
Downloading botocore-1.34.46-py3-none-any.whl (12.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.0/12.0 MB 71.6 MB/s eta 0:00:00
Collecting jmespath<2.0.0,>=0.7.1
Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
Collecting urllib3<1.27,>=1.25.4
Downloading urllib3-1.26.18-py2.py3-none-any.whl (143 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 143.8/143.8 kB 31.0 MB/s eta 0:00:00
Collecting python-dateutil<3.0.0,>=2.1
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7 kB 40.0 MB/s eta 0:00:00
Collecting six>=1.5
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: urllib3, six, jmespath, python-dateutil, botocore, s3transfer, boto3
Successfully installed boto3-1.34.46 botocore-1.34.46 jmespath-1.0.1 python-dateutil-2.8.2 s3transfer-0.10.0 six-1.16.0 urllib3-1.26.18
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead. https://pip.pypa.io/warnings/venv

[notice] A new release of pip is available: 23.0.1 → 24.0
[notice] To update, run: pip install —upgrade pip

4. To confirm that the connection is working, run the following example Python code to retrieve the DynamoDB table from the Amazon EKS pod:

root@python-cli:/# cat <<EOF > test_dynamodb
import boto3
dynamodb = boto3.client('dynamodb')
response = dynamodb.list_tables()
print("Tables:", response['TableNames'])
EOF
root@python-cli:/# python test_dynamodb.py

Note: The preceding code creates a DynamoDB client, lists the tables to test the connection, and then prints a list of the tables.

You receive the following output:

Tables: ['ContainerTable']

Cleanup

To avoid unnecessary charges and keep your Kubernetes cluster efficiently running, delete resources that you're no longer using.

To delete the pods, run the following commands:

 kubectl delete -f dynamo-util.yaml
 kubectl delete -f neptune-util.yaml

To delete the service accounts, run the following commands:

kubectl deletes service account eks-dynamo-sa
kubectl delete serviceaccount eks-neptune-sa

To use the DynamoDB console to delete you DynamoDB table, see Deleting the Amazon DynamoDB tables. Or, run the delete-table AWS CLI command.

To use the Neptune console to delete the Neptune database, see Deleting a DB instance in Amazon Neptune. Or, run the delete-db-cluster AWS CLI command.

Conclusion

In this AWS Support Official article, we showed you how to integrate Amazon Neptune and Amazon DynamoDB to get the most out of your Amazon EKS cluster. When you use IRSA and RBAC mechanisms, you're following best practices for access management. You can simplify your application architecture, keep your data accessible, and unlock advanced data management capabilities that let you build and deploy modern, data-driven applications on AWS.

To learn more about how AWS Support can help accelerate your organization’s business outcomes in the cloud leveraging the knowledge and insights of AWS, see AWS Support.

---

About the author

Ramakrishnan Palaninathan

Ram is a Senior Technical Account Manager at AWS. With a deep-rooted foundation in full-stack development and DevOps. He holds expertise in containers, serverless applications, and Cloud Operations.