Thursday, April 13, 2017

How to SHRINK an Amazon AWS EBS volume safely with no data loss

Note: This guide assumes your volume is based on a Linux file system such as ext4. If you're using a different file system, you may still be able to shrink the volume but your mileage may vary.

To get to the guide, scroll down and skip the prologue.


Once my 1 year free trial with Amazaon AWS was over, I found that I'm paying for 30 GB of storage space while I'm actually using only about 5 GB. Hence I decided to shrink it down to about 7 GB or so. My plan is to just monitor the usage and see if I can reduce the usage or expand the volume later as needed.

Searching for guides on Google, the top hit was this blog post. But I found that the instructions are actually terrible. Here are the problems with it:

  • The guide tells you to FORMAT the volume with your existing data. YES! That's ridiculous, but it's true. It's this command: sudo mkfs -t ext4 /dev/xvdf. I did it and destroyed my data, but lucky for me, I had made a snapshot. No problem, just restore the snapshot to a volume. But what if you DON'T have a snapshot?
  • The guide tells you to use rsync to copy files from your old volume to new one. If you want to know what's wrong with doing that, just check the disk usage (df command) after you're done with it. You'll find that "something is missing". When you want a true clone of a volume, this is NOT good enough. There will be stuff missing like hidden files, metadata, whatnot!
  • If your volume is a root volume, forget about your system ever booting from the shrunken volume. Your instance will not even start.
After experimenting with that, I found another blog post in the comments. Well, this kind of pointed me in the right direction, but it is still lacking (hence the reason for my own post). Here is the good and bad about this post:
  • Good: No need to mount your volumes. The guide tells you to actually clone the disk or partition to the new one, which is indeed the correct method - you will not miss any data in the process.
  • Good: A couple of commands make sure everything is working correctly. This is really good, as you can be confident you're doing the right thing when you see the tests pass.
  • Bad: Does not guide you on how to correctly clone a root volume. It just tells you "use fdisk to create one [partition]" (basically the equivalent of "go figure it out yourself") and leaves you hanging there. Without correct guidance to do this, you'll end up with an unbootable volume.
  • Bad: Again with the root volume, no guidance whatsoever on cloning the boot loader. If you go look for guides, you're told to install GRUB manually in the new volume using a bunch of complicated commands. The good news is: I have found a much easier way. You can CLONE the existing boot loader, whatever it may be!
That being said, here's a complete guide to actually shrink a root volume. I will follow it with another guide to shrink a normal volume.

Thursday, February 16, 2017

How do I rename a label for my Facebook Page's messages?

If you have a Facebook Page and you're receiving messages in it, you may have noticed that you can organize chats with labels. A conversation can even have multiple labels applied on it. This should be very useful for you and your fellow page admins to filter and track their categorization as well as their status.

While this is an excellent feature which allows you to create new labels, delete labels that you no longer need, apply colors to labels, filter by labels and even combine two labels into one, Facebook has totally missed out on a very important yet very basic functionality: renaming a label.

Until such time that Facebook realizes what they have missed out and fixes the problem by giving us a one-click rename option for the label management, here's a trick how you can rename an existing label. This trick uses the functionality of combining two labels to create an empty label and combine it with the existing label that you want to rename. Here's how it works:
  1. Open your Page's Messages page. Usually this is
  2. Select a chat. On the right side, in the Labels section, click "Add & Manage".
  3. In the pop-up window, Type in any random name (like 123, abc, asdf, A, anything) and click Add.
  4. In the list, hover over the labels until you see checkboxes next to them. Click the two checkboxes for:
    1. The label you want to rename, and
    2. The label with the random name.
  5. At the top right corner above the list, you will see a new Combine button. Click it.
  6. In the text box below, type in the new name for the label you want to rename. Click the Combine button on the right side of the text box.
That's it, you have renamed the label from a old name to a new name by using the trick above, which is to combine the label with a dummy label. All the conversations with the old label will automatically move to the new label. Since the dummy label would have no conversations attached to it, nothing else is affected.

Of course, the above trick is only needed and useful until such time that Facebook finally implements "Rename" functionality directly in the existing labels. Let me know when that happens!

Wednesday, February 10, 2016

How to change default and internal currency in self-hosted Odoo or OpenERP

If you wish to use Odoo in a single-currency environment, you will find issues changing the currency in the Odoo interface itself. If you have a self-hosted solution of Odoo or OpenERP, you already have unprecedented control over the system, unlike the cloud-based solution from Odoo. This means that you can very easily fix the issue by hacking the database that Odoo is relying on. Even in a multi-currency environment, you can use the same trick to change the global default currency - the fix is very clean for any environment - there is just an extra step to update the currency rate settings.

Even in the latest version of Odoo, we still face the ridiculous bug that Odoo internally ALWAYS uses EUR (Euro) as the internal currency. This has created many problems for many users wanting to use a different currency. Despite being set to a different currency from the Odoo interface, the eCommerce site, reports, accounting, etc are still shown using EUR and the Euro symbol, creating confusion and accounting problems.

Odoo Support has done nothing yet to correct this critical problem in their software which renders their otherwise excellent software for SME business into merely a toy for a hobbyist to play around with. The internet is full of dissatisfied Odoo customers looking for a solution, and many "solutions" revolve around playing with the accounting settings, multi-currency settings, currency rates, etc.

I don't know whether you have this problem or you can fix it if you purchased Odoo cloud service. But if you are installing and using a self-hosted Odoo solution, you can use the following trick to fix the issue at the very heart of Odoo. Since Odoo uses a Postgres database to store all the data, we can connect to the database directly and modify the data inside, thus changing the behavior of Odoo. NOTE that the database is like Windows registry or Linux /etc - drastic careless modification can render your whole Odoo system useless.

Note that you should not use the Odoo interface to change the default currency for the business or otherwise. If you already did so, now would be a good time to change it back to the default currency, which is the Euro (EUR). (Or after the below procedure, change to your target currency).

Log on to your server by SSH or PuTTY, and connect to the Postgres database client using the following command:
sudo -u postgres psql

Connect to your database using the command (replace <database name> with the actual name of your database - to see the list of databases, use \l ):
\c <database name>

Here you will require some knowledge about databases, SQL writing, and Postgres in particular. I will expand the post with some actual usable commands later. But following is the gist of procedure:
  1. In res_currency table, there will be an entry for EUR, and there may be an entry for your currency.
  2. Your fix will be to swap them around, so that the ID for EUR refers to your currency, and the ID for your currency refers to EUR. That way, your currency becomes the global default for Odoo.
  3. You can also fix the symbol of your currency. Note that this will usually require placing the SQL in an UTF-8 file so that your currency symbol can be stored in the table correctly - typing or pasting it directly into the terminal may convert the symbol to question mark or some other gibberish.
  4. If you could not find an entry for your currency, you will need to create it, then do the swapping with EUR.
  5. If you have stored currency rates in a multi-currency environment, you will also need to update res_currency_rate table to swap the ID for EUR and the ID for your new default currency.

Saturday, January 02, 2016

Amazon EC2 instance with LAMP and Git and multiple Git collaborators

Brace yourself, this is going to be --long. I'm not going to explain a lot. I basically followed what was here, but procedures are outdated and can be confusing if you don't know what you're doing.

I'm just going to give an overview of what's different from that page now:
  •  When you launch a new EC2 instance, the wizard is now different. You will no longer have a choice between "Classic Wizard", "Quick Launch Wizard" etc.
  • Select Ubuntu server 14.04 64-bit with HVM (not PV).
  • Remember to add an EBS volume or expand the root volume. I don't recommend expanding root volume, but rather to use another volume to store your data. That way you can wipe you root volume and start over again without data loss. (I almost did, coz I stupidly sudo rm * in an important directory - but I managed to recover.)
  • You won't be creating key pairs in the wizard itself. You will be asked to do so after you click on Launch to start up the instance. The procedure to create key pair is still the same.
  • Remember to convert the downloaded *.pem file to *.ppk if you want to use PuTTY (my favorite SSH client). There are separate guides available for it.
  • WARNING! Half of the commands shown in the guide are to be executed in the local machine, with the other half on the EC2 instance. The guide assumes that your local machine is a Unix/Linux machine, but I'm on Windows. The local machine commands become unnecessary if you're on Windows, and procedures to achieve similar results would be different. For one thing, there's absolutely no notion of "Unix-style permissions" in Windows. If you have a hard time figuring out, here's a rough guide:
    • "Connect to server with your PEM Key using SSH" - commands for local machine. Not needed on Windows as you will use PuTTY or some graphical client.
    • "Install Linux Apache MySQL PHP (LAMP) Server" - commands for EC2 instance. Just run them once connected by PuTTY.
    • "Connecting with SSH without a PEM key" - commands for local machine. I actually skipped this entire section because I don't mind using the PEM key to connect to shell. For Git access, I set up additional user accounts which have password access and are limited to git-shell. More on that below. If you want to do this anyway, you will need to do this on Windows - and yes, you will need to figure out how - or just use PuTTYgen (easy peasy!).
    • "Setup GIT for web deployment and version control" - mostly commands for EC2 instance, until it says "local computer". This section should not be followed ditto as it's showing you how to set up a bare Git repo as well as live Git repo, and to link them so they are updated together. Be sure to understand the section well. The mod_rewrite commands at the end are also for EC2 instance.
  • The guide uses editors like nano and vi. I stick to using vim for all required edits as it's awesome. If you don't know how to use vim, you should learn. Just Google "vim cheatsheet" and you'll get a lot.
  • The added volume will need to be formatted and mounted manually. For this, see the answer here.
Outside the guide, I also set up users for each person I am going to collaborate with (including myself). Here's how I did that (following some parts of this guide, but my users will have write access):
  1. To make things easier, create a group where you will add your Git users. Let's call this group gitusers. So just run sudo addgroup gitusers.
  2. We will create the users one by one and add them to our gitusers group:
    sudo adduser username --ingroup gitusers --shell /usr/bin/git-shell
    • Be sure to replace username with the actual username of each user. Execute once per user.
    • The --shell part is important so that your new user can only do Git stuff, and not SSH into the server with terminal access.
    • Every time you need to add another user, just do this again. To remove a user, you can deluser or just move the user out of the gitusers group.
  3. Allow password-based authentication for your Git users. Note that you can also set up public-private keys as described in "Connecting with SSH without a PEM key", but I chose to go with a password route. To do this, I enable password-based authentication for only the gitusers group.
    1. Edit using sudo vim (or whatever) the file /etc/ssh/sshd_config
    2. At the end of the file, add this: (vim tip: press Shift-G to quickly go there, then press i and start typing)
      Match Group gitusers
              PasswordAuthentication yes
    3. Save (vim tip: press Esc to exit Insert mode, then type :wq Enter)
    4. Execute sudo service ssh restart so that ssh will pick up the changes.
  4. Remember you changed owner to ubuntu for two folders in "Setup GIT for web deployment and version control"? You need to change the group and add group write permissions to the same two folders, like this:
    sudo chgrp -R gitusers /var/git/
    sudo chgrp -R gitusers /var/www/
    sudo chmod -R g+w /var/git/
    sudo chmod -R g+w /var/www/
  5. That's it. If you ever need read-only Git users, just create another group for them and give the group password-based authentication in sshd_config:
    Match Group gitusers,gitreadonly
    Create the users the same way, except change to --ingroup gitreadonly.
  6. When you clone a git repo to a local machine on a command line, connect by ssh as follows:
    git clone ssh://username@hostname-or-ip-address/path/to/the/repository.git
    For a GUI tool, provide repo URL as above (without the git clone part).


Better to install extundelete using sudo apt-get install extundelete so that you can run it if ever you accidentally rm some important file. I accidentally rm'ed the entire /etc/ssh folder contents losing everything, but managed to recover them using this tool.

I also set up and used No-IP service to give my instance a fixed name, so it's reachable by same host name even if I stop and restart it. To have noip2 always start at boot, do sudo crontab -e and add the line:
@reboot sleep 30 && /usr/local/bin/noip2

That's it!


Related Posts with Thumbnails