Linux the darkside migration method | a.k.a. The dd over ssh method

This method is the most difficult but one that almost always works. It uses dd over ssh to dump the entire partition as it is. It is also very useful when you wish to migrate just a single partition or disk. When using this method to migrate the root volume the last step when you boot the VM is uncertain as you will need to observe the errors on the console in order to boot the VM.

This is a super advanced method of migration and I've written far easier ways to migrate. So unless you know what you need to do, consider choosing one of the methods bellow:

• The online rsync method;
• The offline rsync method;

In this example I will be migrating a source VM from AWS EC2 to my home computer on VirtualBox, however by modifying the steps appropriately you can use to to migrate to any other environment.

Prerequisites

1. Support Linux VM at home or Linux OS on the your workstation
2. VMware Workstation
3. VirtualBox

Gather critical system information from the source VM

cat /etc/*release*
uname -a

# Check if it has a package manager.
# It may be a customised distro and not have a package manager
Execute apt or yum - see if it produces any output

# See what is inside the /boot directory
ls -la /boot

cat /proc/cpuinfo #Check CPU 
cat /proc/meminfo #Check Memory

#Check mounts and partitions
df -h 
cat /etc/fstab
cat /etc/mtab
blkid

# Check running processes
ps auxwff
netstat -tulnp

Procedure

1. DD the disks from the original VM
$ ssh root@10.226.65.213 'dd if=/dev/xvda bs=1M | gzip' | gunzip | dd of=xvda.raw

2. Convert the raw disks into VDI VirtualBox disk

.\VBoxManage convertfromraw "F:\VM\mysourcevm\xvda.raw" "F:\VM-VMware\webbuild-linux\xvda.vmdk" --format VDI

3. Build a shell VM clone in VirtualBox  
Create a blank VM on VirtualBox with the same structure as the original and mount the converted disk.

4. Use dark magic to make it working
This step is simply as it is. Once you power on the cloned VM you will need to analyse the errors on the console (if any) to see what could prevent it from booting. You may as well need to reinstall the bootloader or alter the GRUB parameters. From here on you are on your own to make it working. In my experience I've succeeded in this step in 90% of the cases.

Linux migration from anywhere to anywehre | a.k.a. The online rsync method

This tutorial describes migration instruction of Linux machines from anywhere to anywhere (but in this case to AWS EC2). It was shown to me by my friend Sergiu Badan.

1. Check the source operating system distribution and architecture. 

Launch an AMI in EC2 of the same distribution and architecture. Install rsync and screen on both source and destination:
$ yum -y install rsync screen

2. Check disk requirements

If the source distribution has a total disk size of more than the EC2 AMI storage, you'll need to create EBS drives on the EC2 machine.

Example:
The source system has the following disk space distribution:
/home - 39 GB
/usr - 16
/var - 11
/opt - 430
rest - 5

If the destination has a root partition of 8 GB, you will have to append EBS volumes for /home (40), /usr (20), /var (15), /opt (440).

Format the partitions. See example below, for /home partition which will be on device /dev/xvdh.

Do this for all devices:
$ fdisk -l #(see how they are labeled, like /dev/xvdh or /dev/sdh, or whatever).
mkfs.ext4 /dev/xvdh #(put the proper device name here)

# mount to temporary location
$ mount /dev/xvdh /media

# sync all files from mountpoint to the new device
$ rsync -avz /home/ /media/

# edit /etc/fstab and put the new device name and the mount point:
$ cat /etc/mtab | grep xvdh #or whatever the device is

# append the line to /etc/fstab, but replace /media to the mountpoint. Eg:
/dev/xvdh   /home      ext4    defaults        0 0

# remount the partition:
$ mount -o remount,rw /home

3. Create keypairs on the source system. 

Copy the public keys to both source and destination /root/.ssh/authorized_keys.

# on the source:
$ ssh-keygen -t rsa # answer the defaults, put no password.
$ cat /root/.ssh/id_rsa.pub
# append the content to /root/.ssh/authorized_keys on both source and destination server.

# Copy company public key (through which you access the destination server) also to both source and destination in /root/.ssh/authorized_keys.

4. Create a list of files to exclude, on source in /excluded_files

$ vi /excluded_files
# add the following lines
/etc/fstab
/etc/mtab
/etc/sysconfig/network-scripts
/proc
/boot
/sys
/dev
/etc/resolv.conf
/etc/hosts
/etc/conf.d/net
/etc/network/interfaces
/lib/modules
/etc/sysconfig/network
/etc/nsswitch.conf
/etc/lvm
/var/run

5. Execute the first sync

On the source:
$ screen -R rsync
$ ssh root@$destination_ip_address # answer yes to trust the destination key

$ exit # exit from the remote computer

# rsync <options> <excluded files> <source> <destination>
# Copy everything from SOURCE:/ to DESTINATION:/ exclude the excluded_files
# The --delete option will delete any files in the destination directory if 
#  they don't exist in the source directory.
$ rsync -avz --progress --delete --exclude-from=/excluded_files / root@$destination_ip_address:/ 

# to exit the screen press ctrl + a d
# to enter the screen again, type screen -x

6. After first sync finishes, do the second sync

$ screen -x rsync
$ cat /dev/null >/root/.ssh/known_hosts  # wipe the known_hosts file, as on the destination the key has changed

$ ssh root@$destination_ip_address   # answer yes to trust the destination key
$ exit # exit from the remote server

# stop all important services on the source, like mysql, oracle, apache, nginx, sendmail, postfix, after which do the second rsync:
$ rsync -avz --progress --delete --exclude-from=/excluded_files / root@$destination_ip_address:/

7. Finalise

After the rsync finishes, reboot the remote server, and check that it starts successfully. If there are problems, troubleshoot (check startup logs from EC2 console, eventually umount the /var partition and mount it to other server to check the logs).

Linux EC2 to On-Premise migration | a.k.a. The offline rsync method

Setup

This guide assumes that we have one EC2 instance with only the root volume i.e. one mount only. If you have multiple mounts you will need to adjust the steps accordingly and rsync the multiple mounts. We will be migrating the instance from AWS to a VMware clsuter however by adjusting the steps you can use this method to migrate to any other custom environment, even VirtualBox.

1. Preparation

1. Source EC2 to be cloned (I name this SOURCE-VM)
2. Create helper VM in VMware (I name this HELPER-VM)
3. Create helper EC2 VM in AWS (I name this HELER-EC2)
4. Create blank VM at the target VMware Site with a similar structure as the original (I will name this CLONE-VM). Make sure that the disk is about the same size and thin provisioned
5. Take note of the disks location of the CLONE-VM.
   Example:
   [WorkloadDatastore] 2f6ba65a-6af4-6f09-2666-124516e7c87c/debian.vmdk

Gather SOURCE-VM critical system information

cat /etc/*release*
uname -a

# Check if it has a package manager.
# It may be a customized distro and not have a package manager
Execute apt or yum - see if it produces any output

# See what is inside the /boot directorty
ls -la /boot

cat /proc/cpuinfo #Check CPU 
cat /proc/meminfo #Check Memory

#Check mounts and partitions
df -h 
cat /etc/fstab
cat /etc/mtab
blkid

# Check running processes
ps auxwff
netstat -tulnp

2. In AWS

1. Take snapshot of the root volume of the SOURCE-VM to be cloned
2. Attach the disk from the SOURCE-VM snapshot to the helper HELER-EC2
3. Mount the disk into /mnt
4. Verify that the contents from the attached disk match the contents from your SOURCE-VM. They need to contain the entire disk layout.

3. In VMware

Attach the empty disk that you have taken note before from the CLONE-VM to the HELPER-VM as an additional disk. Mount it to /mnt.

NOTE: You don't need to detach the disk from the CLONE-VM in order to attach it to another VM. In VMware the disks can be attached to multiple VMs at once.

On the HELPER-VM

Format the new drive attached from the CLONE-VM
# Identify the new disk
ll /dev/sd*

## /dev/sdd in my case

## This creates the partition and prepares it for ext4 fs
parted
select /dev/sdd
mklabel msdos
mkpart primary ext4 0% 100%
set 1 boot on
quit

## This formats the partition. Be very carefull not to use a wrong partition here, or it will wipe out everything.
mkfs.ext4 /dev/sdd1

## Mount the partition in /mnt
mount /dev/sdd1 /mnt/

4. Initiate data transfer

From the HELPER-EC2 initiate a data transfer of the contents of the root partition of the SOURCE-VM to the disk of the CLONE-VM that is attached to the HELPER-VM

Example in my case:
rsync -avxHAWX --numeric-ids --info=progress2 -e ssh /mnt/ root@192.168.1.57:/mnt

Once the rsync is finished, shutdown the HELPER-VM and detach the additional hard disk from the it.

5. Re-Install the bootloader

We now have almost everything we need for our CLONE-VM to start working, except that it will not boot without a bootloader.

To do that, we need to to a couple of things:

• Remove any existing bootloader from AWS;
• Remove any folders that contain additional options for booting. These options are tailored for use for an EC2 and may cause problems with our VM;
• Reinstall the boatloader (GRUB);
• Check and adjust /etc/fstab to match our new system layout;
• As an additional cosmetic we also need to remove cloud-init.

To do all of this we need a temporary environment to work on. The system in this case was debian so I will use a bootable Ubuntu CD and go to recovery mode. Then we shall execute a chroot environment into our new disk.

The following is an example of commands used to identify the current boot loader. Then I am removing it. Then I remove any existing folders that have additional boot options. Finally I install the new bootloader. If you will do this on a CentOS/RedHad the commands are slightly different. Use Google find them out.

## Find any existing GRUB software and remove all of it.
$ apt list --installed | grep grub*

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

grub-common/now 2.02~beta2-36ubuntu3.15 amd64 [installed,upgradable to: 2.02~beta2-36ubuntu3.17]
grub-gfxpayload-lists/xenial,now 0.7 amd64 [installed,automatic]
grub-legacy-ec2/now 17.1-46-g7acc9e68-0ubuntu1~16.04.1 all [installed,upgradable to: 17.2-35-gf576b2a2-0ubuntu1~16.04.2]
grub-pc/now 2.02~beta2-36ubuntu3.15 amd64 [installed,upgradable to: 2.02~beta2-36ubuntu3.17]
grub-pc-bin/now 2.02~beta2-36ubuntu3.15 amd64 [installed,upgradable to: 2.02~beta2-36ubuntu3.17]
grub2-common/now 2.02~beta2-36ubuntu3.15 amd64 [installed,upgradable to: 2.02~beta2-36ubuntu3.17]

$ apt-get remove grub && apt-get purge grub #The 2nd removes any grub configuration

$ rm -f /etc/default/grub
$ rm -rf /etc/default/grub.d ## This folder has AWS specific configuraiton thus it will cause problems later on and must be removed.

## Reinstall GRUB
$ apt update
$ apt install grub2 os-prober

At this point GRUB will ask you to which drive the boot sectors to be installed on. Choose the whole drive /dev/sda and not the partition /dev/sda1.

After this the system should boot normally.

[end]

Linux | Don't reboot after adding a new disk on a VM

It is quite common practice when adding a new drive to a Linux system to reboot the system afterwards so that the server can re-scan its hardware and detect the change. However, every now and then we get to make changes to a server that we don't want to reboot it for some reason. Or maybe you are lazy, like me and don't want to wait 1 min to reboot. In any case, bellow is a way to make your system re-scan the hardware and detect the changes.

Linux | How to search for a particular string in multiple files

Every now and then I find myself into a situation where I have to search for a particular string from a set of files, and every time I have to remind myself again.

As the old proverb says: "The fool remembers, the smart writes down."

That is why I decided to write this down, so that I can come back and find it again.

# grep -rnw '/path/to/somewhere/' -e "pattern"

Here -r or -R is recursive, -n is line number and -w stands match the whole word. We can also add -l (lower-case L) to just give the file name of matching files.

So in my current example, I work on Nagios and I want to find the file containing that check_snmp command i would type:

# grep -rnw '/etc/nagios/objects/' -e 'check_snmp'

Which will produce the output:

/etc/nagios/objects/commands.cfg:147:# 'check_snmp' command definition

/etc/nagios/objects/commands.cfg:149:        command_name    check_snmp

From here we can basically see the file where check_snmp is defined along with the line number of that file.

Extending the command with --include or --exclude parameter we can create neatly commands to also search for files with particular extension. The bellow command will check only .cfg or .conf files.

# grep --include=\*.{cfg,conf} -rnw '/etc/nagios/objects/' -e 'check_snmp'

# grep --exclue=\*.0 -rnw '/etc/nagios/objects/' -e 'check_snmp'

And as always:

RTDM!

# man grep

Linux | Guidelines for setting up a Linux Server as a VM

These are just plain guidelines for setting up a Linux server as a virtual machine. As time passes I keep seeing the same mistakes from fellow admins, over and over again so I decided to mention some of it.

Feel free to add anything else in the comments or point me out should I am making a mistake somewhere along the way ;)

First things first, you have to decide what will be the use of the VM that you are creating? Will it be a file server, or a web server, or a database server? Perhaps it will be a dedicated VM for a couple of users. No matter your choice, if you are setting up the server as a virtual machine in you infrastructure you have to keep in mind a few things so these are just a few golden rules for setting up a Linux VM so that you don't end up crying later when things go bad.

CentOS 7 | Installation and setup of Icinga2 monitoring service - Part 2 - Setting up the Icinga Web 2 interface

This is part 2 of the tutorial where I'll be setting up the Icinga Web 2 web fronted from where we can log into our Icinga2 server and actually monitor our infrastructure.

If you came here from Google than you can find Part 1 here.

CentOS 7 | Installation and setup of Icinga2 monitoring service - Part 1

Knowing that there is a newer fancier Nagios fork called Icinga I always wanted to try it out, but
I really haven't got the time to do that. Finally when I got the time I realized that online guides for that are missing too. Being a CentOS supporter, I decided that during that process i could write a tutorial for CentOS 7.

This is the part 1 of the tutorial where I'll be going through the steps for preparing the server and setting up the Icinga2 service. In part two I'll be going through the steps for installing and setting up the Icingaweb2 web service which basically is one of the several available Incinga2 front-end applications.

Convert Linux LVM partitions back to normal partitions

There are number of questions online regarding this issue, but I haven't found a concrete solution just untill recently, when I had a task to make a P2V Conversion of a CentoOS Server. If you happen to have a system with a logical volume (LV) but for whatever reasons at a later time you decide that you want to transfer that LV or convert it back to normal partition eg. /dev/sdc1 than I discovered a very easy procedure which can help you to do just that.

You will need a spare drive or a partition with the same size (or bigger) to do that. First make sure that you have your drive ready and with at least one partition table that is the exact same size, or larger then the original LV.

On my system the logical volume is located at: /dev/VolGroup00/LogVol00.
You can use dd to convert the LV back to normal partition.

The procedure to convert the LV to a normal partition is as follows.

# dd if=/dev/VolGroup00/LogVol00 of=/dev/sdc1 bs=4M

Depending on the size of your partitions you will have to wait for some time. Also keep in mind that dd does not produce any output except if something goes wrong. In my case the disks were 230G (the original LV) and 250G (the other disk) respectively, so I've waited for 3 hours at least.

After the process is finished you can mount the /dev/sdc1 partition and you will be able to see all of your files inside. The output of df -h or fdisk -l will show 230G of space instead of 250G as we made a clone of the LV partition.

If you still want to continue to use your old drive where the LV was, than just format that old drive/partition /dev/sdb1 (in my case) and make that same transfer again but in reverse order.

# dd if=/dev/sdc1 of=/dev/sdb1 bs=4M

Virtualizing CentOS 5 physical servers with LVM configuration

Somewhere on the internet was written:

Virtualizing a physical Linux server is a combination of knowledge, experience and luck.

Recently I had a the task for a client to virtualize a CentOS 5 Linux server. After searching on Google about it, I was disappointed to see so many posts of people having problems with it, especially those who had physical Linux servers with LVM configurations. If you research about a bit, you will be surprised on how many production CentOS 5 servers exist which run with their default installations with one large physical volume and one swap partition. The things are even worse if the boot partition is also on that same logical volume.

Unfortunately there are still many forgotten production Linux servers today which run CentOS 5. The things are even worse, because in smaller companies, those servers are set up and forgotten, and even those admins who once administered those servers are gone and not working for that company for years. What happens is when you try to virtualize Linux server with similar configuration described above with the VMware converter, the process usually terminates at 97% with error:

FAILED: An error occurred during the conversion:

'Partition number must be set for the boot volume'

Another common example is also found in this VMware KB article.

When you try to power on this crippled VM you'll end up with one of the two things.

1. The VM will behave as it is an empty VM (just created with no OS)
2. The system will try to boot, you will see the GRUB loading screen, and after a while when the kernel starts you will see a kernel panic message.

Both of this problems are completely solvable. Since you already have the main volume with everything inside (databases, software etc..) you just need to make sure that now you have to repair the  key parts from which every Linux operates boots and operates - the boot partition.

CentOS 7 | Disable and enable the firewall

"Hey Tom, I am just doing this tutorial article about setting up my Apache server but damn I did everything twice and I still can't access my server from outside but the service is working?!"

Every now and then I get these questions from people who just installed their CentOS 7 server and now they are trying to setup their whatever network service.

The pain my friends is the firewall on the CentOS 7 machine that comes installed by default.

There are two solutions for this, both with advantages and disadvantages.