Wednesday, January 28, 2015

Getting VMWare ESXi 5.5 home server and M.2 PCI-E drives

6:29 AM Posted by Grant No comments
I decided more than 6 months ago (read a long time ago) to retire my previous server a Dell Poweredge 2950 III after many years of service, it has been a long road but I have finally gathered the main parts and got the Operating System working.

The story about how I got to this point would usually be fairly mundane except for the pleasure of buying new shiny things that go fast however as technology usually does at the bleeding edge (and far from officially supported) you have problems.

Samsung XP941 PCI-E M.2 SSD and the bleeding edge

These drives and others like it are a new generation of SSD drives that run directly over the PCI Express bus to take advantage of it's significantly higher bandwidth, something SSDs have been choking on for a while now.

While the drive installs onto the Mainboard (which has a small slot directly on it) the problem is that there is no support for the drives and as such mine wouldn't show up as a boot drive.

A visit to the ASUS Support site showed an available BIOS upgrade for my Mainboard (ASUS Z97-M) so after one false start where the drive would detect but was not bootable (fixed in later firmware update) that was taken care of.

VMWare ESXi 5.5 and supporting the unsupported

I downloaded the free version of ESXi 5.5 and wrote it to a spare USB drive with Rufus which made the ISO image bootable, this can be done with either GPT (New) or MBR (Legacy) File Systems of which ESXi supports both however I would suggest GPT.

Booting from USB into the installer seems rather problematic, here are some things you may need to do to avoid going insane like I did:

* Disable Secure boot

Go to the Boot menu then down to Secure Boot and select Other OS to disable it, VMWare installer will boot otherwise.

* Enable CSM (Compatibility Support Module)

I'd suggest setting this to Auto if it's not already, it's function is to provide legacy BIOS support if booting older things that don't support UEFI.

Once this is done check the boot menu for the USB drive and select the Non-UEFI option otherwise you will end seeing "Fatal error: 32 (Syntax)" which is due to VMWare's lack of UEFI support, see here.

Once the installer I ran into another problem of drivers, turns out VMWare are very brutal when it comes to culling drives for hardware and as such support for both my PCI-E NIC and Samsung XP941 were missing.

After some searching I found this man who saved myself and I'm sure many other people in compiling the drivers ourselves, he had recently added support for the Samsung XP941 as well, her deserves massive credit for all his work.

Customising the VMWare install

I grabbed the sata-xahi and net1000 VIB files, these are used to customise the VMWare ISO, conveniently ESXI-Customiser is at the same site and is dead simple to use.


Select the ISO image, select the VIB file to be added to the ISO and specify an output directory and we have our custom ISO!

After writing the image back to the USB drive we were back to VMWare setup however this time the Samsung was showing up, excellent!

VMWare won't boot, why? Because fuck you thats why!

With VMWare installed reboot was performed however attempting to boot the drive resulted in ending up in the UEFI BIOS, without any form of logs I can only assume there was something it did not like about the booting of the drive.

Testing Ubuntu and Windows 8 worked with no problem with both of them installing and booting without any issues so the issue was particular to VMWare, at some stage I had it working on VMWare 6.0 then decided to abandon VMWare for other reasons, suffice to say I forgot what I had done to get it working.

Upon returning to VMWare I came across the issue again, time to dive into UEFI and find out what the problem is, I was using the latest BIOS and latest ESXi 5.5 build so the easy options were out.

EFI booting why you no work?

Without going into details about EFI the very short version is that there is a certain partition called an ESP (EFI System Partition), it's formatted as FPT with a particular partition ID and is FAT32. This partition has a file called bootx64.efi located in \EFI\BOOT\bootx64.efi, this file is searched for by the EFI which uses it to start the boot process.

My Mainboard supports booting to an EFI Shell (DOS / Unix CLI) from a USB drive, all that's required is to grab a copy of shell.efi, rename it to bootx64.efi, copy is to a FAT32 USB drive in the above file structure with this done I was able to get a far better insight into things.

Once into EFI shell I could navigate the ESP, sure enough bootx64.efi was present and if I loaded it directly from shell it would boot VMWare so I knew the file was working.

The issue had to be in the boot options within EFI that pointed it to the file, I'll be honest I didn't dive into how this works but it's likely there is something (Possibly in nstartup.nsh) which controls this.

In this case I just wrote a new boot entry, surprisingly simple to do as well!

Shell > bcfg boot add 0 fs0:\EFI\BOOT\Bootx64.efi "VMWare ESXi"

* bcfg - An older alternative to efibootmgr, my EFI shell only had the older one, attempts to get a newer shell proved pointless, for some reason they would not boot.
* fs0 - This is the Samsung XP941, it can br browsed directly by just typing "fs0:" from the Shell> prompt can be browsed with standard linux commands (e.g ls, cd, mv,rm) and vol can be used to check the name
add 0 - I assume 0 is just for ordering, didn't really look into it that much
"VMWare ESXi" - Just the label in the boot menu

After restarting VMWare now boots without any issue and I'm no longer going insane re-writing USB drives, changing BIOS/UEFI options and trying different partition and file system combinations!

Resources

Arch Linux Wiki (EFI Examples) - Examples of how to modify the EFI boot
Arch Linux Wiki (UEFI) - In depth article with links to EFI shells
Intel Developer zone (EFI Shells and scripting) - Useful command-line reference for EFI commands
Intel Developer zone (UEFI Boot Manager) - Good detail and easy to understand