Skip to content

Done fin’t mess with grub.conf (or you’ll end with kernel panic)

by on January 5, 2015

While installing some SW I had to modify some parameters in various areas:

Modify kernel parameters in the /etc/sysctl.conf file,
Disable the selinux and the iptables
Add some file systems and change their ownership and permission
etc.

Ususaly I do it on one server and copy the script or files to other servers.

Now, I had to do some changes in the /etc/grub.conf file – disable the cstate.
And – instead of running the grubby command, I changed manually on one server (using good old vi) and copied the files to the other servers.

Big mistake.

I got kernel panic and it took me a while to find the cause:

  1. I installed all the servers using the RHEL ISO, (not using PXE install from my cobbler)
  2. The default installation creates an LVM for the system partitions and the default volume group includes the host name
  3. The grub file (which I’m not an expert in it), includes the location of some of the volumes
  4. As the volume includes the host name – it differ between machines
  5. Copying the file from one server to another caused the grub to fail in locating the right image and to go into panic
  6. In one of the servers I had an additional issue:
    1. I’ve created in the RAID controller two volume groups
    2. On all servers, except one, I formatted the disk and created a mountpoint for it during the installation
    3. Doing so, the installation updated the /etc/fstab with the UUID
    4. Not doing so on one machine, I got the disks mixed up during boot time – sdb became sda and vice cersa
    5. this caused the /etc/grub.conf to be missing altogether (as it is a line to /boot/grub/grub.conf, and the /boot partition did not mount because of the mixup)

BTW – resolving the issue was easy: I booted from RHEL iso as recovery and just fixed the grub.conf file to point to the right lvm.

Conclusions:

  1. Do not copy grub file, but use he grubby command
  2. Create an image to be shared across all servers (so, no hostname in any lvm part)
  3. use UUID for mounting disks
  4. insall using PXE

And if anyone reads this and can answer the following issue – I will appriciate it a lot:

I have Dell servers with 10GBE ports “on board” and I have IBM servers with 10GBE ports mounted as PCI-e cards.
How do I make the servers PXE boot from these ports and not from the 1GBE default port?

Dell – model: PowerEdge R720, Network card: Broadcom NetXtreme II BCM57800 1/10 Gigabit Ethernet
IBM – model: x3650 M3, Network card: Emulex OneConnect 10Gb NIC (

and that’s all for now.

Some usefull links:

Advertisements
Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: