How do I migrate an EBS volume from gp2 to gp3 in Amazon EKS?

5 minute read

I want to migrate my Amazon Elastic Block Store (Amazon EBS) volume type from General Purpose SSD (gp2) to General Purpose SSD (gp3) in Amazon Elastic Kubernetes Service (Amazon EKS).

Short description

You can use the Amazon EBS Container Storage Interface (CSI) driver to modify Amazon EBS volumes with annotations in PersistentVolumeClaims (PVC). This feature is available in version aws-ebs-csi-driver v1.19.0-eksbuild.2 or later. You can modify the type, iops, and throughput parameters within PVC:

To migrate your EBS volume from gp2 to gp3 in your Amazon EKS cluster, complete the following steps:

Install the Amazon EBS CSI driver in your Amazon EKS cluster.
Activate the volume modification feature in the Amazon EBS CSI driver.
Change the EBS volume type.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Install the volume modification feature in Amazon EBS CSI driver

The volume modification feature is available in Amazon EBS CSI driver version 1.19.0. It's used by Helm chart 2.19.0 or the Amazon EKS managed add-on v1.19.0-eksbuild.2. This feature is implemented as an additional sidecar called volumemodifier. You must activate it manually in Amazon EBS CSI driver. For more information, see Volume modification on the GitHub website.

To activate the volume modification feature in Amazon EKS, configure one of the following methods:

Self-managed add-on configuration with Helm
Amazon EKS managed add-on configuration

Self-managed add-on configuration with Helm

Install the self-managed Amazon EBS CSI driver through the Helm chart, and then set controller.volumeModificationFeature.enabled to true.
Example YAML:

cat values.yaml  
controller:
  volumeModificationFeature:
    enabled: true

Upgrade the configuration. Run the following command:

helm upgrade aws-ebs-csi-driver --namespace kube-system \
    aws-ebs-csi-driver/aws-ebs-csi-driver -f values.yaml

Amazon EKS managed add-on configuration

Create a config file. Run the following command:

cat eks-addon-ebs-csi-driver-config.json 
{
  "controller": {
    "volumeModificationFeature": {
       "enabled": true
    }
  }
}

Update the Amazon EBS CSI driver add-on with the config file. Run the following command:

aws eks update-addon --cluster-name eks-demo \
    --addon-name aws-ebs-csi-driver \
    --service-account-role-arn arn:aws:iam::{AWS_ACCOUNT_ID}:role/AmazonEKS-EBS-CSI-DriverRole \
    --configuration-values 'file://eks-addon-ebs-csi-driver-config.json'

{
    "update": {
         ...
        "status": "InProgress",
        "type": "AddonUpdate",
        "params": [
            ...
            {
                "type": "ConfigurationValues",
                "value": "{\n  \"controller\": {\n    \"volumeModificationFeature\": {\n       \"enabled\": true\n    }\n  }\n}"
            }
        ],
        ...
    }
}

Verify that the additional sidecar volumemodifier is active. Run the following command:

kubectl get pod -n kube-system -l=app=ebs-csi-controller

Example output:

NAME                                  READY   STATUS    RESTARTS   AGE
ebs-csi-controller-5f985b94f4-lk7lv   7/7     Running   0          2m29s
ebs-csi-controller-5f985b94f4-m5nn8   7/7     Running   0          2m29s

Check the container names. Run the following command:

kubectl get pod -n kube-system ebs-csi-controller-5f985b94f4-lk7lv -o jsonpath='{.spec.containers[*].name}{"\n"}'

Example output:

ebs-plugin csi-provisioner csi-attacher csi-snapshotter volumemodifier csi-resizer liveness-probe

Change the EBS volume type

Review the current PVC and PersistentVolume (PV) status. Run the following commands:

kubectl get sc

Example output:

NAME   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2    kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  6h23m

kubectl get pvc ebs-demo-claim

Example output:

NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
ebs-demo-claim   Bound    pvc-10d1d621-22f4-4fb6-a64a-51cdf03a9e30   4Gi        RWO            gp2            <unset>                 18s

kubectl get pv pvc-10d1d621-22f4-4fb6-a64a-51cdf03a9e30

Example output:

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                    STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-10d1d621-22f4-4fb6-a64a-51cdf03a9e30   4Gi        RWO            Delete           Bound    default/ebs-demo-claim   gp2            <unset>                          36s

View the PV details. Run the following command:

kubectl get pv pvc-10d1d621-22f4-4fb6-a64a-51cdf03a9e30 -o yaml

Example output:

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/migrated-to: ebs.csi.aws.com
    pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
    ...
  ...
spec:
  awsElasticBlockStore:
    fsType: ext4
    volumeID: vol-0296e8e6fa4b27670
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: ebs-demo-claim
    namespace: default
    ...
  persistentVolumeReclaimPolicy: Delete
  storageClassName: gp2
  volumeMode: Filesystem
status:
  phase: Bound
  ...

Check the Amazon EBS details. Run the following command:

aws ec2 describe-volumes --volume-ids vol-0296e8e6fa4b27670

Example output:

{
    "Volumes": [
        {
            "AvailabilityZone": "us-east-1a",
            "Size": 4,
            "State": "in-use",
            "VolumeId": "vol-0296e8e6fa4b27670",
            "Iops": 100,
            "VolumeType": "gp2"
            ...
        }
    ]
}

Add the gp3-type annotation. Run the following command:

kubectl annotate pvc ebs-demo-claim ebs.csi.aws.com/volumeType="gp3"

Example output:

persistentvolumeclaim/ebs-demo-claim annotated

Verify the annotations. Run the following commands:

kubectl get pvc ebs-demo-claim -o yaml

Example output:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    ebs.csi.aws.com/volumeType: gp3
    ...
  ...
  name: ebs-demo-claim
  namespace: default

kubectl get pv pvc-10d1d621-22f4-4fb6-a64a-51cdf03a9e30 -o yaml

Example output:

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    ebs.csi.aws.com/volumeType: gp3
    ...
spec:
  awsElasticBlockStore:
    fsType: ext4
    volumeID: vol-0296e8e6fa4b27670
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: ebs-demo-claim
    namespace: default
  ...

Monitor the modification. Run the following command:

kubectl describe pvc ebs-demo-claim

Example output:

Name:          ebs-demo-claim
...
Events:
  Type    Reason                        Age    From                                                                                      Message
  ----    ------                        ----   ----                                                                                      -------
  ...
  Normal  ProvisioningSucceeded         5m20s  ebs.csi.aws.com_ebs-csi-controller-676f998966-bh79n_ec5572b3-c1ec-4b3a-838a-d4b21b8c668f  Successfully provisioned volume pvc-10d1d621-22f4-4fb6-a64a-51cdf03a9e30
  Normal  VolumeModificationStarted     64s    volume-modifier-for-k8s-ebs.csi.aws.com                                                   External modifier is modifying volume pvc-10d1d621-22f4-4fb6-a64a-51cdf03a9e30
  Normal  VolumeModificationSuccessful  57s    volume-modifier-for-k8s-ebs.csi.aws.com

Confirm that the volume type changed to gp3. Run the following command:

aws ec2 describe-volumes --volume-ids vol-0296e8e6fa4b27670

Example output:

{
    "Volumes": [
        {
            "AvailabilityZone": "us-east-1a",
            "Size": 4,
            "State": "in-use",
            "VolumeId": "vol-0296e8e6fa4b27670",
            "Iops": 3000,
            "VolumeType": "gp3",
            "Throughput": 125
            ...
        }
    ]
}

Use volume snapshots to migrate volumes

The Amazon EBS CSI driver provides a CSI volume snapshots feature. You can use snapshots to migrate persistent volumes from the in-tree gp2 StorageClass to the Amazon EBS CSI gp3 StorageClass. For more information, see Volume snapshots on the GitHub website.

To use volume snapshots, complete the following steps:

Install the Amazon EBS CSI driver.
Install the VolumeSnapshotClass, VolumeSnapshotContent, and VolumeSnapshot objects. For more information, see snapshotclass.yaml, volume-snapshot-content.yaml, and snapshot.yaml on the GitHub website.
Use the objects to create a new gp3-type persistent volume from the snapshot. For more information, see Migrating Amazon EKS clusters from gp2 to gp3 EBS volumes.

Related information

Simplifying Amazon EBS volume migration and modification on Kubernetes using the EBS CSI Driver

Topics: Containers Storage
Tags: Amazon Elastic Block Store Amazon Elastic Kubernetes Service
Language: English

AWS OFFICIALUpdated 2 months ago

No comments

Relevant content

Efficient Data Migration and Volume Removal between EBS Volumes on EC2 Instance
Accepted Answer
scholar
asked 2 years ago
EBS gp3 volumes in Severely Degraded state
AWS-User-2242675
asked 4 years ago
Low throughput on EBS gp3 volume
stsk
asked 3 years ago
pods swichover to another AZ in EKS
vijayravi
asked 2 years ago
AWS EKS instance not found when attaching EBS volume
kovacs
asked 3 years ago
How do I use Amazon EBS Multi-Attach to attach the same Volume to multiple workloads in Amazon EKS?
AWS OFFICIALUpdated 2 years ago
What are the differences between the Amazon EBS volume types?
AWS OFFICIALUpdated a year ago
How do I manage EBS volumes to avoid unexpected costs in AWS?
AWS OFFICIALUpdated 23 days ago
How do I troubleshoot issues with my EBS volume mounts in Amazon EKS?
AWS OFFICIALUpdated a year ago
AWS Migration Performance Optimization in AWS MGN and DRS service: A Complete Guide to Staging Area Resource Analysis
SUPPORT ENGINEER
Nikunj G
published 5 months ago