Issue – VMware vCenter Appliance 6.x crashed with a damaged LVM

Reading Time: < 1 minute

Hardware VMware Host: Fujitsu PRIMERGY RX600 S6
Hardware VMware Storage: Fujitsu ETERNUS DX90 S2
Hardware VMware vCenter Appliance: VMware vCenter Appliance 6.0
Problem: Lightning strike crashed VMware vCenter Appliance with a damaged LVM

Hello,

a few days a ago I deployed my first VMware vCenter Appliance in Version 6.0. At the weekend we had a lightning strike, which caused a crash of the Appliance. So the following message appears after reboot:

Message: /dev/mapper/core_vg-core: UNEXPECTED INCONSISTENCY: RUN fsck MANUALLY.

vCSA-crashed-LVM-Welcome-Screen

First I found this link: http://www.boche.net/blog/index.php/2016/04/04/vcenter-server-6-appliance-fsck-failed/
So I check /dev/sda3, but that was not the problem, if I would have read it clearly. After a while I realized that the LVM has the problem. So someone give the correct hint to followed these steps: http://blog.grogscave.net/2016/06/recovering-from-vcenter-appliance-disk.html
and that worked and the vCSA was UP and RUNNING.

My steps to success:

  1. Boot the appliance to the command bash (Source: https://kb.vmware.com/kb/2069041– Step “Unlocking a locked out root account”)
  2. When you on the command prompt, load the Device Mapper Module with “modprobe dm_mod
  3. Make a scan of the volume groups with “vgscan” and
  4. Activate the volume groups with “vgchange -ay
  5. Now you can make a e2fsck of the named volume group with “e2fsck -y /dev/mapper/core_vg-core” in my case
  6. CTRL-D to Exit and reboot the appliance
  7. The appliance should now UP and RUNNING

Cheers,

Rolf

 

Leave a Reply