Recover from an EFI failure on a dedicated server

A few weeks ago, before upgrading our Ubuntu systems, we sort of messed around with our EFI partitions and the impacted servers never came back online on system reboot after the upgrade.

Provisionning a dedicated server from a cloud provider leave you in charge of the whole server, from networking to disk management and booting.

On a desktop computer the EFI partition is indicated in the bios and is not supposed to change once it is set. On our dedicated server, it seems differents. We had no access to the bios but the onboard bios checks each partition to find a valid EFI installation. This is pretty cool. It means you can freely move you EFI partition around and the sever will always find it.

It can be confusing at first, especially when all the disks are shipped with only one small empty EFI partition. You could interprete this state as a mistake or an annoyance and delete all these partitions (except one on your system disk). This is what we did to gain flexibility in how to format the non-system disk. Before deleting any of the EFI partiion, you shall ensure that they are empty, else the server will not reboot. To check this, you can mount the partition and run ls inside.

The following steps will show you how to repair the EFI directory using GRUB.

• First, it is necessary to boot the server on a temporary system. Your cloud provider shall provide you something like a ‘rescue mode’. When activated, you shall be able to log into your machine, for example after receiving creadential for a temporary SSH access, and you can mount its disks.
• Then, find where your root is with lsblk which will print a tree of your disks. Identify on which disk your OS is installed and mount it in /mnt:

  mount /dev/your_root_disk /mnt

• Mount sys, dev and proc filesystems as well:

  mount -t sysfs sys /mnt/sys
  mount -t devtmpfs devtmpfs /mnt/dev
  mount -t proc procfs /mnt/proc

• Mount the desired EFI partition, sda1 here:

  mount /dev/sda1 /mnt/boot/efi

• Chroot in the mounted environment:

  chroot /mnt

• Install grub:

  grub-install --efi-directory=/boot/efi

• Update Grub Config:

  grub-mkconfig -o /boot/grub/grub.cfg

Then cancel the recovery mode and reboot your server and voila, your server should restart normally. Make sure not to delete or corrupt the EFI partition again!