Linux EC2 to On-Premise migration | a.k.a. The offline rsync method

Setup

This guide assumes that we have one EC2 instance with only the root volume i.e. one mount only. If you have multiple mounts you will need to adjust the steps accordingly and rsync the multiple mounts. We will be migrating the instance from AWS to a VMware clsuter however by adjusting the steps you can use this method to migrate to any other custom environment, even VirtualBox.

1. Preparation

1. Source EC2 to be cloned (I name this SOURCE-VM)
2. Create helper VM in VMware (I name this HELPER-VM)
3. Create helper EC2 VM in AWS (I name this HELER-EC2)
4. Create blank VM at the target VMware Site with a similar structure as the original (I will name this CLONE-VM). Make sure that the disk is about the same size and thin provisioned
5. Take note of the disks location of the CLONE-VM.
   Example:
   [WorkloadDatastore] 2f6ba65a-6af4-6f09-2666-124516e7c87c/debian.vmdk

Gather SOURCE-VM critical system information

cat /etc/*release*
uname -a

# Check if it has a package manager.
# It may be a customized distro and not have a package manager
Execute apt or yum - see if it produces any output

# See what is inside the /boot directorty
ls -la /boot

cat /proc/cpuinfo #Check CPU 
cat /proc/meminfo #Check Memory

#Check mounts and partitions
df -h 
cat /etc/fstab
cat /etc/mtab
blkid

# Check running processes
ps auxwff
netstat -tulnp

2. In AWS

1. Take snapshot of the root volume of the SOURCE-VM to be cloned
2. Attach the disk from the SOURCE-VM snapshot to the helper HELER-EC2
3. Mount the disk into /mnt
4. Verify that the contents from the attached disk match the contents from your SOURCE-VM. They need to contain the entire disk layout.

3. In VMware

Attach the empty disk that you have taken note before from the CLONE-VM to the HELPER-VM as an additional disk. Mount it to /mnt.

NOTE: You don't need to detach the disk from the CLONE-VM in order to attach it to another VM. In VMware the disks can be attached to multiple VMs at once.

On the HELPER-VM

Format the new drive attached from the CLONE-VM
# Identify the new disk
ll /dev/sd*

## /dev/sdd in my case

## This creates the partition and prepares it for ext4 fs
parted
select /dev/sdd
mklabel msdos
mkpart primary ext4 0% 100%
set 1 boot on
quit

## This formats the partition. Be very carefull not to use a wrong partition here, or it will wipe out everything.
mkfs.ext4 /dev/sdd1

## Mount the partition in /mnt
mount /dev/sdd1 /mnt/

4. Initiate data transfer

From the HELPER-EC2 initiate a data transfer of the contents of the root partition of the SOURCE-VM to the disk of the CLONE-VM that is attached to the HELPER-VM

Example in my case:
rsync -avxHAWX --numeric-ids --info=progress2 -e ssh /mnt/ root@192.168.1.57:/mnt

Once the rsync is finished, shutdown the HELPER-VM and detach the additional hard disk from the it.

5. Re-Install the bootloader

We now have almost everything we need for our CLONE-VM to start working, except that it will not boot without a bootloader.

To do that, we need to to a couple of things:

• Remove any existing bootloader from AWS;
• Remove any folders that contain additional options for booting. These options are tailored for use for an EC2 and may cause problems with our VM;
• Reinstall the boatloader (GRUB);
• Check and adjust /etc/fstab to match our new system layout;
• As an additional cosmetic we also need to remove cloud-init.

To do all of this we need a temporary environment to work on. The system in this case was debian so I will use a bootable Ubuntu CD and go to recovery mode. Then we shall execute a chroot environment into our new disk.

The following is an example of commands used to identify the current boot loader. Then I am removing it. Then I remove any existing folders that have additional boot options. Finally I install the new bootloader. If you will do this on a CentOS/RedHad the commands are slightly different. Use Google find them out.

## Find any existing GRUB software and remove all of it.
$ apt list --installed | grep grub*

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

grub-common/now 2.02~beta2-36ubuntu3.15 amd64 [installed,upgradable to: 2.02~beta2-36ubuntu3.17]
grub-gfxpayload-lists/xenial,now 0.7 amd64 [installed,automatic]
grub-legacy-ec2/now 17.1-46-g7acc9e68-0ubuntu1~16.04.1 all [installed,upgradable to: 17.2-35-gf576b2a2-0ubuntu1~16.04.2]
grub-pc/now 2.02~beta2-36ubuntu3.15 amd64 [installed,upgradable to: 2.02~beta2-36ubuntu3.17]
grub-pc-bin/now 2.02~beta2-36ubuntu3.15 amd64 [installed,upgradable to: 2.02~beta2-36ubuntu3.17]
grub2-common/now 2.02~beta2-36ubuntu3.15 amd64 [installed,upgradable to: 2.02~beta2-36ubuntu3.17]

$ apt-get remove grub && apt-get purge grub #The 2nd removes any grub configuration

$ rm -f /etc/default/grub
$ rm -rf /etc/default/grub.d ## This folder has AWS specific configuraiton thus it will cause problems later on and must be removed.

## Reinstall GRUB
$ apt update
$ apt install grub2 os-prober

At this point GRUB will ask you to which drive the boot sectors to be installed on. Choose the whole drive /dev/sda and not the partition /dev/sda1.

After this the system should boot normally.

[end]

Scalable Jenkins with Kubernetes cluster

I wrote a short tutorial on GitHub about creating a scalable Jenkins on a Kubernetes cluster. The main pod is the master which creates slave pods per need and then deletes those pods when they are no longer needed.

You can find the tutorial at the same GitHub repo here: https://github.com/spirit986/skjenkins

VSphere | Advanced snapshot trobuleshooting - Part 3 - Example: Invalid snapshot configuration

Unable to consolidate because of invalid snapshot configuration

This is Part 3 of the short tutorial series:
vSphere | Advanced snapshot troubleshooting

This is much more advanced than our previous example.  You will see these two errors if you attempt to consolidate, clone or migrate the VM:

Detected and invalid snapshot configuration

or

... vmdk was not found

VSphere | Advanced snapshot trobuleshooting - Part 2 - Example: Unable to consolidate because the file is locked

This is Part 2 of the short tutorial series:
vSphere | Advanced snapshot troubleshooting

Virtual Machine consolidation attempts result with bellow error:
Unable to access file since it is locked

VSphere | Advanced snapshot trobuleshooting - Part 1 - The ESXi shell is your friend

I am writing this article as an extension to my previous post A VM is showing disk size of 0B where I will attempt to make a general explanation on how to troubleshoot and successfully solve snapshots and consolidation problems in your vSphere with a number of examples. Because the general article was so big, I decided to split it into three parts. This Part 1, is where I am describing the commands which are used to solve most of the snapshot problems and in the next two parts I will describe the process of troubleshooting via examples.

Linux | Don't reboot after adding a new disk on a VM

It is quite common practice when adding a new drive to a Linux system to reboot the system afterwards so that the server can re-scan its hardware and detect the change. However, every now and then we get to make changes to a server that we don't want to reboot it for some reason. Or maybe you are lazy, like me and don't want to wait 1 min to reboot. In any case, bellow is a way to make your system re-scan the hardware and detect the changes.

Linux | How to search for a particular string in multiple files

Every now and then I find myself into a situation where I have to search for a particular string from a set of files, and every time I have to remind myself again.

As the old proverb says: "The fool remembers, the smart writes down."

That is why I decided to write this down, so that I can come back and find it again.

# grep -rnw '/path/to/somewhere/' -e "pattern"

Here -r or -R is recursive, -n is line number and -w stands match the whole word. We can also add -l (lower-case L) to just give the file name of matching files.

So in my current example, I work on Nagios and I want to find the file containing that check_snmp command i would type:

# grep -rnw '/etc/nagios/objects/' -e 'check_snmp'

Which will produce the output:

/etc/nagios/objects/commands.cfg:147:# 'check_snmp' command definition

/etc/nagios/objects/commands.cfg:149:        command_name    check_snmp

From here we can basically see the file where check_snmp is defined along with the line number of that file.

Extending the command with --include or --exclude parameter we can create neatly commands to also search for files with particular extension. The bellow command will check only .cfg or .conf files.

# grep --include=\*.{cfg,conf} -rnw '/etc/nagios/objects/' -e 'check_snmp'

# grep --exclue=\*.0 -rnw '/etc/nagios/objects/' -e 'check_snmp'

And as always:

RTDM!

# man grep