Recover from an EFI failure on a dedicated server

Recover from an EFI failure on a dedicated server

By Grégor JOUET

Apr 16, 2019

Categories: Hack | Tags: Cloud, Infrastructure, Linux

A few weeks ago, before upgrading our Ubuntu systems, we sort of messed around with our EFI partitions and the impacted servers never came back online on system reboot after the upgrade.

Provisionning a dedicated server from a cloud provider leave you in charge of the whole server, from networking to disk management and booting.

On a desktop computer the EFI partition is indicated in the bios and is not supposed to change once it is set. On our dedicated server, it seems differents. We had no access to the bios but the onboard bios checks each partition to find a valid EFI installation. This is pretty cool. It means you can freely move you EFI partition around and the sever will always find it.

It can be confusing at first, especially when all the disks are shipped with only one small empty EFI partition. You could interprete this state as a mistake or an annoyance and delete all these partitions (except one on your system disk). This is what we did to gain flexibility in how to format the non-system disk. Before deleting any of the EFI partiion, you shall ensure that they are empty, else the server will not reboot. To check this, you can mount the partition and run ls inside.

The following steps will show you how to repair the EFI directory using GRUB.

  • First, it is necessary to boot the server on a temporary system. Your cloud provider shall provide you something like a ‘rescue mode’. When activated, you shall be able to log into your machine, for example after receiving creadential for a temporary SSH access, and you can mount its disks.
  • Then, find where your root is with lsblk which will print a tree of your disks. Identify on which disk your OS is installed and mount it in /mnt:

    mount /dev/your_root_disk /mnt
  • Mount sys, dev and proc filesystems as well:

    mount -t sysfs sys /mnt/sys
    mount -t devtmpfs devtmpfs /mnt/dev
    mount -t proc procfs /mnt/proc
  • Mount the desired EFI partition, sda1 here:

    mount /dev/sda1 /mnt/boot/efi
  • Chroot in the mounted environment:

    chroot /mnt
  • Install grub:

    grub-install --efi-directory=/boot/efi
  • Update Grub Config:

    grub-mkconfig -o /boot/grub/grub.cfg

Then cancel the recovery mode and reboot your server and voila, your server should restart normally. Make sure not to delete or corrupt the EFI partition again!

Canada - Morocco - France

International locations

10 rue de la Kasbah
2393 Rabbat
Canada

We are a team of Open Source enthusiasts doing consulting in Big Data, Cloud, DevOps, Data Engineering, Data Science…

We provide our customers with accurate insights on how to leverage technologies to convert their use cases to projects in production, how to reduce their costs and increase the time to market.

If you enjoy reading our publications and have an interest in what we do, contact us and we will be thrilled to cooperate with you.