Internal database disk space management in the AWS Transform discovery tool

When you deploy the AWS Transform discovery tool OVA appliance, the internal database is located at /home/discovery/.local/share/DiscoveryTool/discovery_tool_prod.db. When you deploy the tool on Linux, the internal database is installed at the same path in the installing user's home directory. This means that by default, the database is installed in the root partition. When collecting data for a large migration, this can result in the root partition running out of space. Although you can boot into a partition editor like GParted and resize the root partition, you might want to move the database to its own partition instead.

This post covers moving the /home directory of the installation to another mount point. Although the instructions are similar, make sure to follow the Linux instructions if you installed the tool yourself on a Linux distribution. Follow the OVA instructions if you deployed the OVA.

Linux

Follow these instructions if you installed the discovery tool yourself on a Linux distribution. These instructions were tested on Ubuntu Linux.

The installation user is transform in this environment. Make sure to use the user account that is correct in your environment when entering the commands below.

1. Add a new virtual disk to your Linux VM.

2. Verify the new block device is available with lsblk. Here, we find block device sdb available.

   transform@util01:~$ lsblk
   NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
   sda                         8:0    0   25G  0 disk
   ├─sda1                      8:1    0    1G  0 part /boot/efi
   ├─sda2                      8:2    0    2G  0 part /boot
   └─sda3                      8:3    0 21.9G  0 part
     └─ubuntu--vg-ubuntu--lv 252:0    0   11G  0 lvm  /
   sdb                         8:16   0  100G  0 disk
   sr0                        11:0    1 1024M  0 rom

3. Partition the new device

   transform@util01:~$ sudo parted /dev/sdb mklabel gpt
   Information: You may need to update /etc/fstab.
   
   transform@util01:~$ sudo parted /dev/sdb mkpart primary ext4 0% 100%
   Information: You may need to update /etc/fstab.

4. Make a filesystem in the new partition

   transform@util01:~$ sudo mkfs.ext4 /dev/sdb1
   mke2fs 1.47.0 (5-Feb-2023)
   Creating filesystem with 26213888 4k blocks and 6553600 inodes
   Filesystem UUID: c78fdfd3-52f3-4a4c-bc8e-514d1c170c39
   Superblock backups stored on blocks:
           32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
           4096000, 7962624, 11239424, 20480000, 23887872
   
   Allocating group tables: done
   Writing inode tables: done
   Creating journal (131072 blocks): done
   Writing superblocks and filesystem accounting information: done   

5. Stop the AWS Transform discovery tool service

   transform@util01:~$ sudo ./AWS-Transform-discovery-tool.sh stop
   [INFO] Stopping Discovery Tool service...
   [SUCCESS] Service stopped

6. Make a temporary mountpoint

   transform@util01:~$ sudo mkdir /mnt/transform_new
   transform@util01:~$ sudo mount /dev/sdb1 /mnt/transform_new

7. Copy the user home directory to the new mountpoint. The trailing slash is critical in for this command for both source and destination

   transform@util01:~$ sudo rsync -avHAX /home/transform/ /mnt/transform_new/
   sending incremental file list
   created directory /home/transform_new
   transform/
   transform/.bash_history
   transform/.bash_logout
   transform/.bashrc
   transform/.profile
   transform/.sudo_as_admin_successful
   transform/.viminfo
   transform/latest.tar.gz
   transform/replication-installer.sh
   transform/repoint-wp-db.sh
   transform/.cache/
   transform/.cache/motd.legal-displayed
   transform/.local/
   transform/.local/share/
   transform/.local/share/nano/
   
   sent 102,167,479 bytes  received 63,790 bytes  40,892,507.60 bytes/sec
   total size is 101,891,429  speedup is 1.00

8. Verify file copy

   transform@util01:/home$ du -sh /home/transform
   107M    /home/transform
   transform@util01:/home$ sudo du -sh /mnt/transform_new/
   107M    /mnt/transform_new/

9. Move the old home directory and make a new one. Be sure to use the correct account and group to match your environment.

   sudo mv /home/transform /home/transform_old
   sudo mkdir /home/transform
   sudo chown transform:transform /home/transform
   sudo chmod 750 /home/transform

10. Dismount and remove the temp mount point

    sudo umount /mnt/transform_new
    sudo rmdir /mnt/transform_new
    sudo rmdir /home/transform_new

11. Obtain the block ID

    transform@util01:/home$ sudo blkid /dev/sdb1
    /dev/sdb1: UUID="c78fdfd3-52f3-4a4c-bc8e-514d1c170c39" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="primary" PARTUUID="5374b6e5-ca6d-4c43-bcf9-cca3ba217e39"    

12. Update /etc/fstab with the new mountpoint

    transform@util01:/home$ sudo vi /etc/fstab
    
    # /etc/fstab: static file system information.
    #
    # Use 'blkid' to print the universally unique identifier for a
    # device; this may be used with UUID= as a more robust way to name devices
    # that works even if disks are added and removed. See fstab(5).
    #
    # <file system> <mount point>   <type>  <options>       <dump>  <pass>
    # / was on /dev/ubuntu-vg/ubuntu-lv during curtin installation
    /dev/disk/by-id/dm-uuid-LVM-uN2xCBMGJoichsDyijpPVvEBnXyqYhS6HRIIqKydwg9Wji7seNDU3RHGdSNh3gkN / ext4 defaults 0 1
    # /boot was on /dev/sda2 during curtin installation
    /dev/disk/by-uuid/5819b27d-e0a8-4592-a4bb-e9815a8806db /boot ext4 defaults 0 1
    # /boot/efi was on /dev/sda1 during curtin installation
    /dev/disk/by-uuid/B31B-6060 /boot/efi vfat defaults 0 1
    # Disk for AWS Transform database
    /dev/disk/by-uuid/c78fdfd3-52f3-4a4c-bc8e-514d1c170c39 /home/transform ext4 defaults 0 2  

13. Remount the devices in fstab

    transform@util01:/home$ sudo mount -a
    mount: (hint) your fstab has been modified, but systemd still uses the old version; use 'systemctl daemon-reload' to reload.

14. Verify successful mount

    transform@util01:/home$ mount | grep /dev/sdb1
    /dev/sdb1 on /home/transform type ext4 (rw,relatime)

15. Start the AWS Transform discovery tool service

    transform@util01:~$ sudo ./AWS-Transform-discovery-tool.sh start
    [INFO] Starting Discovery Tool service...
    [SUCCESS] Service started successfully
    [INFO] Web interface available at: https://target-ip:5000

16. Once you verify the discovery tool is up and working correctly, delete the old /home directory

    transform@util01:~$  sudo rm -rf /home/transform_old/

OVA deployment on VMware

Follow these instructions if you installed the discovery tool as an OVA. Because the OVA uses IDE disk, you cannot hot-add the new disk.

1. Log in to the VM console of the deployed appliance.

2. The appliance has an alias enablessh built into the discovery account's shell. This changes the default password and enables SSH

   [discovery@discoverytool ~]$ enablessh
   SSH requires password change:
   Changing password for user discovery.
   New password:
   Retype new password:
   passwd: all authentication tokens updated successfully

3. Shut down the VM

   [discovery@discoverytool ~]$ shutdown -h now

4. Add a new virtual disk to IDE:0, not to a SCSI controller

5. Boot the VM

6. Verify the new block device is available with lsblk. Here, we find block device sdb available.

   NAME     MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
   sda        8:0    0   35G  0 disk
   ├─sda1     8:1    0   35G  0 part /
   ├─sda127 259:0    0    1M  0 part
   └─sda128 259:1    0   10M  0 part /boot/efi
   sdb        8:16   0  100G  0 disk

7. Partition the new device

   [discovery@discoverytool ~]$ sudo parted /dev/sdb mklabel gpt
   Information: You may need to update /etc/fstab.
   
   [discovery@discoverytool ~]$ sudo parted /dev/sdb mkpart primary ext4 0% 100%
   Information: You may need to update /etc/fstab.

8. Make a filesystem in the new partition

   [discovery@discoverytool ~]$ sudo mkfs.ext4 /dev/sdb1
   mke2fs 1.46.5 (30-Dec-2021)
   Creating filesystem with 26213888 4k blocks and 6553600 inodes
   Filesystem UUID: 50cca65c-b4de-4597-9a5e-688be1236961
   Superblock backups stored on blocks:
           32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
           4096000, 7962624, 11239424, 20480000, 23887872
   
   Allocating group tables: done
   Writing inode tables: done
   Creating journal (131072 blocks): done
   Writing superblocks and filesystem accounting information: done

9. Stop the AWS Transform discovery tool service

   [discovery@discoverytool ~]$ sudo systemctl stop discovery-tool
   [discovery@discoverytool ~]$

10. Make a temporary mountpoint

    transform@util01:~$ sudo mkdir /mnt/discovery_new
    transform@util01:~$ sudo mount /dev/sdb1 /mnt/discovery_new

11. Copy the user home directory to the new mountpoint. The trailing slash is critical in for this command for both source and destination

    transform@util01:~$ sudo rsync -avHAX /home/discovery/ /mnt/discovery_new/
    sending incremental file list
    ./
    .bash_history
    .bash_logout
    .bash_profile
    .bashrc
    .local/
    .local/share/
    .local/share/DiscoveryTool/
    .local/share/DiscoveryTool/discovery_tool_prod.db
    .local/share/DiscoveryTool/server.crt
    .local/share/DiscoveryTool/server.key
    .local/share/DiscoveryTool/logs/
    .local/share/DiscoveryTool/logs/app.log
    
    sent 908,403 bytes  received 214 bytes  1,817,234.00 bytes/sec
    total size is 907,010  speedup is 1.00

12. Verify file copy

    transform@util01:/home$ du -sh /home/discovery
    908K    /home/discovery
    transform@util01:/home$ sudo du -sh /mnt/discovery_new/
    908K    /mnt/discovery_new/

13. Move the old home directory and make a new one. Be sure to use the correct account and group to match your environment.

    sudo mv /home/discovery /home/discovery_old
    sudo mkdir /home/discovery
    sudo chown discovery:discovery /home/discovery
    sudo chmod 750 /home/discovery

14. Dismount and remove the temp mount point

    sudo umount /mnt/discovery_new
    sudo rmdir /mnt/discovery_new

15. Obtain the block ID

    transform@util01:/home$ sudo blkid /dev/sdb1
    /dev/sdb1: UUID="50cca65c-b4de-4597-9a5e-688be1236961" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="primary" PARTUUID="a9efb1ec-eaaf-4bab-aef3-7e28e3d8c4a0"

16. Update /etc/fstab with the new mountpoint

    [discovery@discoverytool home]$ sudo vi /etc/fstab
    
    #
    UUID=4b7ea247-4d74-4bce-bcf7-ef8fa84bf31b     /           xfs    defaults,noatime  1   1
    UUID=699A-B8A1        /boot/efi       vfat    defaults,noatime,uid=0,gid=0,umask=0077,shortname=winnt,x-systemd.automount 0 2
    # /home/discovery on dedicated VMware virtual disk
    UUID=50cca65c-b4de-4597-9a5e-688be1236961  /home/discovery  ext4  defaults,noatime  0  2

17. Remount the devices in fstab

    [discovery@discoverytool home]$ sudo mount -a

18. Verify successful mount

    [discovery@discoverytool home]$ mount | grep sdb1
    /dev/sdb1 on /home/discovery type ext4 (rw,noatime,seclabel)

19. Start the AWS Transform discovery tool service

    [discovery@discoverytool home]$ sudo systemctl start discovery-tool

20. Once you verify the discovery tool is up and working correctly, delete the old /home directory

    sudo rm -rf /home/discovery_old/