Citrix XenServer 7.1 CentOS VM’s not starting after Kernel update
After I update my CentOS 7 VM kernel, the new kernel stopped booting on my XenServer hypervisor. I’ve been using XenServer version 7.1
Working Kernel version, before yum update was 3.10.0-693
And, after yum update Kernel version is 3.10.0-1127.19.1
This issue is due to the kernel which I updated has the spectre/meltdown patches in. But our XenServer 7 doesn’t. This difference created the kernel booting problem. Interestingly, this issue only affects CentOS VMs. My other Ubuntu servers can just run updates without issue but CentOS VM’s require the host to be updated or they stop working.
To FIX this issue you should update your XenServer to the latest version or else if there are hotfixes available then you should install it.
Link for XenServer 7.1 hotfixes. https://support.citrix.com/article/CTX230788
To upgrade XenServer 7.1 you should have Citrix License first. But a “free” patch was released for 7.2, due to the massive security issue later.
But if you have been running older XenServer’s version, older than 7.1 then you don’t have any choice besides updating XenServer to the latest.
Also, CentOS fixed their booting problem on their new Kernel versions which I will discuss later below.
What is Meltdown and Spectre vulnerabilities?
Someone having access to the VM will be able to read the memory outside the VM itself, and therefore read data from other VMs on this same physical host. An attacker could enter via other application exploits (webserver etc.) on your VMs.
Am I affected?
Meltdown is using a design flaw into Intel CPUs only. This is called by Xen sec team ”SP3” (aka rogue data cache load).
You are impacted only if you are using:
- 64-bits PV type VM (HVM/PVHVM aren’t affected!)
- Intel CPUs (AMD chip design is a bit different and not affected)
- untrusted VMs, ie untrusted users having VM access (even non-root!)
- All XenServer versions are affected
64-bits PV guests are vulnerable because guest and hypervisor share the same address space, but with different privileges. HVM aren’t.
Mitigate Risk:
Migrate all VM’s from PV to HVM or upgrade XenServer to latest or install hotfixes if available.
But if you have been running older version XenServer, older than 7.1 then you have to update your XenServer to resolve this issue.
Back to CentOS 7 booting issue. It seems the booting issue was fixed in the latest CentOS kernel versions. If the XenServer was up-to-date with the hotfixes then the VM’s should not have any problem. But in my case I didn’t update my XenServer, so now I’ve to either install the latest CentOS 7 kernel or downgrade to older previous running one.
Upgrading Kernel Version
Add ELRepo Repository first.
Add ELRepo gpg key to the system.
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
This is important – CentOS will not allow the installation of an unsigned software package. The GPG key provides a digital signature to verify that the software is authentic.
Now add new ELRepo repository.
rpm -Uvh https://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
When it’s complete, check all repository enabled on the system, and make sure ELRepo is on the list.
yum repolist
yum --enablerepo=elrepo-kernel install kernel-ml
kernel-ml – which indicates a mainline release, with a shorter support term but with more frequent updates.
OR INSTALL (Install either ml or lt)
yum --enablerepo=elrepo-kernel install kernel-lt
kernel-lt – signifying a stable long-term support release.
listout all your existing kernel version on your system.
awk -F\' /^menuentry/{print\$2} /etc/grub2.cfg
CentOS Linux (4.4.234-1.el7.elrepo.x86_64) 7 (Core)
CentOS Linux (3.10.0-1127.19.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1127.18.2.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-c848e02bc90943f7a4daa50b09df4) 7 (Core)
We have sucessfully installed CentOS latest kernel version 4.4.234-1. Now set this as a default kernel.
Set Default Kernel Version
First listout all existing kernel versions.
awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg
CentOS Linux (4.4.234-1.el7.elrepo.x86_64) 7 (Core)
CentOS Linux (3.10.0-1127.19.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1127.18.2.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-c848e02bc90943f7a4daa50b09df4) 7 (Core)
NOTE: The first entry is denoted as
Zero. Let us modify the Kernel Version to4.4.234-1which is at line number1but denoted as entry0.
Set default kernel.
grub2-set-default 0
Next, Rebuild grub.cfg file with 'gurb2-mkconfig' command.
grub2-mkconfig -o /boot/grub2/grub.cfg
##### To check default Kernel Version
```yaml
cat /boot/grub2/grubenv |grep saved
saved_entry=CentOS Linux (4.4.234-1.el7.elrepo.x86_64) 7 (Core)
Reboot Server
reboot
Above solution will resolves the booting issue.
If you don’t want to update CentOS kernel to latest one and want to boot from the existing previous working kernel then reboot your system and choose kernel version which was working fine before from grub menu and select enter and follow the below steps.
Set Default Kernel Version but first list-out all existing kernel versions.
awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg
CentOS Linux (4.4.234-1.el7.elrepo.x86_64) 7 (Core)
CentOS Linux (3.10.0-1127.19.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1127.18.2.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)
CentOS Linux (0-rescue-c848e02bc90943f7a4daa50b09df4) 7 (Core)
NOTE: The first entry is denoted as
Zero. So currently the Server is booted to0thentry as per the above command output. Let us modify the Kernel Version to3.10.0-693which is at line number4but denoted as entry3.
Set default kernel.
grub2-set-default 3
Next, Rebuild grub.cfg file with 'gurb2-mkconfig' command.
grub2-mkconfig -o /boot/grub2/grub.cfg
Check default Kernel Version
cat /boot/grub2/grubenv | grep saved
saved_entry=CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)
Reboot Server
reboot
Comments