How to SHRINK an Amazon AWS EBS volume safely with no data loss

Note: This guide assumes your volume is based on a Linux file system such as ext4. If you're using a different file system, you may still be able to shrink the volume but your mileage may vary.

To get to the guide, scroll down and skip the prologue.

Prologue

Once my 1 year free trial with Amazaon AWS was over, I found that I'm paying for 30 GB of storage space while I'm actually using only about 5 GB. Hence I decided to shrink it down to about 7 GB or so. My plan is to just monitor the usage and see if I can reduce the usage or expand the volume later as needed.

Searching for guides on Google, the top hit was this blog post. But I found that the instructions are actually terrible. Here are the problems with it:

• The guide tells you to FORMAT the volume with your existing data. YES! That's ridiculous, but it's true. It's this command: sudo mkfs -t ext4 /dev/xvdf. I did it and destroyed my data, but lucky for me, I had made a snapshot. No problem, just restore the snapshot to a volume. But what if you DON'T have a snapshot?
• The guide tells you to use rsync to copy files from your old volume to new one. If you want to know what's wrong with doing that, just check the disk usage (df command) after you're done with it. You'll find that "something is missing". When you want a true clone of a volume, this is NOT good enough. There will be stuff missing like hidden files, metadata, whatnot!
• If your volume is a root volume, forget about your system ever booting from the shrunken volume. Your instance will not even start.

After experimenting with that, I found another blog post in the comments. Well, this kind of pointed me in the right direction, but it is still lacking (hence the reason for my own post). Here is the good and bad about this post:

• Good: No need to mount your volumes. The guide tells you to actually clone the disk or partition to the new one, which is indeed the correct method - you will not miss any data in the process.
• Good: A couple of commands make sure everything is working correctly. This is really good, as you can be confident you're doing the right thing when you see the tests pass.
• Bad: Does not guide you on how to correctly clone a root volume. It just tells you "use fdisk to create one [partition]" (basically the equivalent of "go figure it out yourself") and leaves you hanging there. Without correct guidance to do this, you'll end up with an unbootable volume.
• Bad: Again with the root volume, no guidance whatsoever on cloning the boot loader. If you go look for guides, you're told to install GRUB manually in the new volume using a bunch of complicated commands. The good news is: I have found a much easier way. You can CLONE the existing boot loader, whatever it may be!

That being said, here's a complete guide to actually shrink a root volume. I will follow it with another guide to shrink a normal volume.

Shrinking a Root Volume

1. Before you do anything, stop the instance. Then take a snapshot of the EBS volume you want to shrink. Name it something meaningful (including the description). If you screw up at any step, this is your safeguard for recovery.
   .
2. Create a new EBS volume with the new shrunken size that you want. Please be sure that your new size will be enough to hold your existing data and have some free space. If the original volume is encrypted, make sure the new one is too.
   .
3. Since this is a root volume, you can't use the instance that you boot off of it. You will need to create a new AWS instance (a temporary one, with its own root volume). The easiest way to do this would simply be to select your existing instance and click on "Launch more like this" (in the Actions menu) which basically duplicates the configuration. You can also just create a t2.nano instance with any new version of Ubuntu.
   .
4. Detach the root volume you want to shrink from the instance you boot off of it, then attach it to the new instance as /dev/sdf. Also, attach the new empty volume to the new instance as /dev/sdg.
   .
5. Start the new instance and once it's online, SSH to it. If you had added your own key when creating the instance, you can just use the key to login. Otherwise you may need to enter a password.
   .
6. Execute: sudo ls -l /dev/xv* ... You should see a list of devices. Since this is a root volume, you should see xvdf and xvdf1 but for the new volume, you would see xvdg only. (Note that /dev/s* becomes /dev/xv* in Ubuntu kernel. This is normal - just look at the suffixes like df and dg to figure out the volume.)
   .
7. Execute sudo dd bs=16M if=/dev/xvdf of=/dev/xvdg count=100 ... This will copy the first 1680 MB of data from disk sdf to disk sdg. This is a real low-level clone of the first 1680 MB, which is much more than enough to clone the boot loader and the existing partition information.
   .
   Side note: The goal here is only to clone the boot loader but since we don't know the exact locations, we just copy the first 1680 MB whatever that maybe. The boot loader itself may not be more than a couple of MBs if it even goes beyond 1 MB. Also, the partition information just comes with this clone, but we will actually discard it in the next step.
   .
8. Execute sudo fdisk /dev/xvdg ... This will launch the fdisk utility. Reminder: It's NOT /dev/xvdf.
   .
9. Once you're in the fdisk command line, press p to see the partition information. You will see that there is 1 partition. Take note of the start sector number. You will need it.
   .
10. Press d and delete the existing partition. Then press n to add a new partition and use the same start sector number that you noted as the "First Sector". The rest of the options can be left at default. It looks like this:
    .
    Command (m for help): n[ENTER]
    Partition type:
       p   primary (0 primary, 0 extended, 4 free)
       e   extended
    Select (default p): [ENTER]
    Using default response p
    Partition number (1-4, default 1): [ENTER]
    Using default value 1
    First sector (2048-12582911, default 2048): 16065[ENTER]
    Last sector, +sectors or +size{K,M,G} (16065-12582911, default 12582911): [ENTER]
    Using default value 12582911
    .
    Side note: If you have more than one partition in the original volume, you will need to create as many partitions in the new volume as well, giving each partition enough space to hold the same amount of data. It would be a good idea to launch fdisk on /dev/xvdf and view the partition information of the original volume.
    .
11. Press a and select the new partition to add a boot flag on it.
    .
12. Press w to save the changes and exit fdisk.
    .
13. Now execute sudo ls -l /dev/xv* ... You should see a list of devices. If you did everything correctly, you should be able to see xvdg1 now.
    .
14. Execute sudo e2fsck -f /dev/xvdf1 ... This checks the file system of the old volume and ensures that there are no errors.
    .
15. If there are no errors, execute sudo resize2fs -M -p /dev/xvdf1 ... This is a very interesting command which pushes all the data to the beginning of the partition and resizes it down to fit just the data with no free space at all. The goal is to make sure there is no free space in between data blocks.
    .
    This command can take a while to finish. Keep calm and grab a coffee. If you're worried about session disconnect, execute this using nohup (command) &
    .
16. In the previous command's output, the last line tells you the number of blocks. Each block is sized 4K but when we clone the partition, we are going to do it in 16 MB blocks. So, in order to compute the number of 16 MB, blocks, multiply the number in the last line by 4 / (16 * 1024). Round this number UP (not down) to the nearest integer.
    .
    Example: 1252939 (number in last line) * 4 / (16 * 1024) = 305.893310546875 ... But round this UP to 306 or even 310 (it doesn't matter as long as you don't go below).
    .
17. Execute sudo dd bs=16M if=/dev/xvdf1 of=/dev/xvdg1 count=(PUT THE NUMBER FROM PREVIOUS STEP HERE) ... So for example, you would execute sudo dd bs=16M if=/dev/xvdf1 of=/dev/xvdg1 count=306 ... This will clone the entire partition 1 from the old volume to the new volume.
    .
    This command can take a while to finish, and there won't be any progress display. Keep calm and grab a coffee. You can test if session is still alive by pressing Enter key. If you're worried about session disconnect, execute this using nohup (command) &
    .
18. Execute sudo resize2fs -p /dev/xvdg1 ... This will expand the partition to fill up the remaining space of the volume.
    .
19. Execute sudo e2fsck -f /dev/xvdg1 ... This will check the cloned data for any error. If there are no errors, you should be able to see that the number of files will match the same number from the output of step 14.
    .
20. Now you may log out the temporary instance and stop it (shut it down). Careful, do NOT terminate it. Once it's stopped, detach both the old and new volumes from it.
    .
21. Attach the new volume to the original instance as /dev/sda1 (this is the root volume). Start the instance, and if everything went well, the instance should boot up like nothing ever happened.
    .
22. You can delete the old volume and terminate the temporary instance. Keep the snapshot for a couple of weeks, then delete it once you're sure everything is working fine.

Shrinking a Regular Volume

This procedure is much quicker because there are no partitions to worry about (unless you actually made partitions in the volume, in which case you will need to modify the previous instructions to suit yourself).

1. Before you do anything, stop the instance. Then take a snapshot of the EBS volume you want to shrink. Name it something meaningful (including the description). If you screw up at any step, this is your safeguard for recovery.
   .
2. Create a new EBS volume with the new shrunken size that you want. Please be sure that your new size will be enough to hold your existing data and have some free space. If the original volume is encrypted, make sure the new one is too.
   .
3. At this point, I would suggest creating a new temporary t2.nano instance for the purpose of cloning. However, you can also clone in the same instance where the volume is primarily used - you need to make sure it's not currently in use, and you need to umount (unmount) it.
   .
4. Attach the new empty volume to the chosen instance as /dev/sdg. If you created a new instance, detach the old volume from the primary instance and attach it to the new instance. (I will assume that the old volume is /dev/sdf.)
   .
5. Start the instance if it's not started yet, and once it's online, SSH to it.
   .
6. Execute: sudo ls -l /dev/xv* ... You should see a list of devices: xvdf for the old volume and xvdg for the new volume. (Note that /dev/s* becomes /dev/xv* in Ubuntu kernel. This is normal - just look at the suffixes like df and dg to figure out the volume.)
   .
7. Execute sudo e2fsck -f /dev/xvdf ... This checks the file system of the old volume and ensures that there are no errors.
   .
8. If there are no errors, execute sudo resize2fs -M -p /dev/xvdf ... This is a very interesting command which pushes all the data to the beginning of the partition and resizes it down to fit just the data with no free space at all. The goal is to make sure there is no free space in between data blocks.
   .
   This command can take a while to finish. Keep calm and grab a coffee. If you're worried about session disconnect, execute this using nohup (command) &
   .
9. In the previous command's output, the last line tells you the number of blocks. Each block is sized 4K but when we clone the partition, we are going to do it in 16 MB blocks. So, in order to compute the number of 16 MB, blocks, multiply the number in the last line by 4 / (16 * 1024). Round this number UP (not down) to the nearest integer.
   .
   Example: 1252939 (number in last line) * 4 / (16 * 1024) = 305.893310546875 ... But round this UP to 306 or even 310 (it doesn't matter as long as you don't go below).
   .
10. Execute sudo dd bs=16M if=/dev/xvdf of=/dev/xvdg count=(PUT THE NUMBER FROM PREVIOUS STEP HERE) ... So for example, you would execute sudo dd bs=16M if=/dev/xvdf of=/dev/xvdg count=306 ... This will clone the entire partition 1 from the old volume to the new volume.
    .
    This command can take a while to finish, and there won't be any progress display. Keep calm and grab a coffee. You can test if session is still alive by pressing Enter key. If you're worried about session disconnect, execute this using nohup (command) &
    .
11. Execute sudo resize2fs -p /dev/xvdg ... This will expand the partition to fill up the remaining space of the volume.
    .
12. Execute sudo e2fsck -f /dev/xvdg ... This will check the cloned data for any error. If there are no errors, you should be able to see that the number of files will match the same number from the output of step 7.
    .
13. If you used a temporary instance, you may log out of it and stop it (shut it down). Careful, do NOT terminate it. Once it's stopped, detach both the old and new volumes from it.
    .
14. Attach the new volume to the primary instance using the same name as the old volume (such as /dev/sdb or /dev/sdf). Start the instance (if needed), and if everything went well, the new volume should work exactly as the old one did.
    .
15. You can delete the old volume and terminate the temporary instance (if any). Keep the snapshot for a couple of weeks, then delete it once you're sure everything is working fine.

Expanding a volume

Expanding a volume is very straightforward. You do not need to clone the volume at all. Simply increase the volume size in AWS Dashboard and then use your OS tools to expand the partition to fill up the additional space. There are many guides out there (and you can take some cues from above) but I will not be having a guide for it here.

Good luck!