Azure, Rhel upgrades and ADE - part 1 the concept

Let’s get started…

In this first part, I will explain the current and growing concern I see with my customers as the existing investment of RedHat servers becomes out of date and needs upgrade.

The cloud…the data challenge

If you have been in the Cloud hosting world at all, the growing concern is data custody. What I mean is, how do I control, in someone else’s data center (the very definition of cloud), my data.

Lock it down

The answer, customer managed key or BYOK. (Bring your own key). This ensures that no matter who has access to the data (at a data center level, not o/s level – more on that later), your are in control of who accesses it.

How its done

This is not a new thing, its called multiple things, Bitlocker for windows, dm-crypt or LUKS in Linux. And in Azure its called A.D.E. – or azure disk encryption. Its basically the layer for which the o/s depends on a key to gain access to a vm partition structure.

Now you see it, now you don’t (in reverse)

Before – sda4 looks like it has no content.

After – (see below) – sda4 reveals all the partition goodness that it needs to fulfill its purpose in the linux world.

How does it work?

If you notice above, sda has four partitions. Sda1 contains the efi boot partition that signals this is a linux os disk. Sda2 contains the /boot partition that has all the image files and ‘stuff’ that allows linux to start to boot to get to loading the o/s contents.

Inside that /boot partition is a series of dracut modules which live inside /usr/lib/dracut/modules.d. Since the /usr and other filesystems live in an encrypted space, dracut needs to include (among other things) a module set for running A.D.E. scripts /usr/lib/dracut/modules.d/91adeonline. When you regenerate the kernel with dracut, this is included and includes the magic…

This is not an A.D.E. lecture…

There is far more to that than the above, but in its simplest form, the o/s boots, finds its image, identifies its an A.D.E. encrypted disk, aquires the secret, contacts the keyvault, obtains the key and then …. finally…. the o/s filesystems.

So what’s the problem

If you are like most companies, you probably have years if not decades of investment into the tuning of the o/s. There are claims of servers should be cattle, not pets, but the drive and corresponding discipline to leverage IAC (infrastructure as code) for all aspect of server builds seldom meets the timeline of business delivery. So, we have numerous, specialized builds of linux that (at least in Redhat state) are aging quickly and need a major O/S upgrade to keep current.

Methods of upgrade

Aligning to the speed of business, there is seldom time for patching let alone time for major o/s upgrade and CERTAINLY not the appetite to start from scratch and rebuild. The two options most companies (and their associated professionals) face are the following:

Parallel upgrade:
– Create new server of current o/s
– This requires a new IP address so all the fun of that:
– Firewall rules
– Load balancer settings
– Proxy
– DNS
– Parallel severs gives the owners ‘time’ to migrate
– Cost of parallel workloads,
– If on premise – resource consumption from inventory
– Installation of the product and the painstaking
determination of all the nuances to make it work

In-Place upgrade:
– Leverage the same server name/ip
– Inheritance of all the IP based work already done
– No parallel usage and managing timeframes to convert
– Far less cost to conduct
– Leverage existing configuration
– Does require deletion of existing server
– YOU HAVE TO have a good fallback / recovery plan

In-Place is the clear winner, what is the issue?

Redhat (and other linux versions), has the ability to leverage something called, L.E.A.P.P. Among other things, it requires that the o/s NOT be encrypted. This seems like a no-brainer, but then you look at the fine print for azure disk encryption

If you review the entire list of requirements to even encrypt, its fairly daunting and not without its challenges. Also if you look at the list of things A.D.E. does not support, its somewhat of a ‘why would we do this anyway’.

If you are already A.D.E. this is for you

So, if you or your company has taken the plunge. You might ask, if I want/need the security of customer managed key. What are my options if A.D.E. won’t let me use L.E.A.P.P.?

Enter DES!!

So Microsoft has heard the concerns and provided an alternate approach. Introducing Disk Encryption Set, or more commonly known as DES. This has its own set of complexities, but at a high level this is the change in the boot sequence.

A.D.E. boot

D.E.S. boot

Load E.F.I.

Load Boot/initramfs

Dracut loads ade module

/mnt/azure_bek_disk is mounted (as vfat)

Secret used to encrypt request to keyvault

Keyvault matches secret to key, returns key to o/s

Disk accepts request, shows partition table

Server starts boot sequence

Hardware/hypervisor contacts disk

Disk detected to be in a disk encryption set

Hardware contacts keyvault (has permission to see key)

Disk Encryption set ‘unlocked’

Disk accessed for boot

Load E.F.I. and normal o/s processes are conducted

So back to in place conversion

Ok so D.E.S. seems the way, does L.E.A.P.P. support it? YES…. because the O/S is not aware of encryption, L.E.A.P.P. simply processes it like a normal O/S upgrade.

But now the question, how do you convert an A.D.E. disk to D.E.S. While I searched high and low on Google (including braving Reddit and other non-traditional sources), nothing seemed to reveal itself as a solution….

Always leave ’em wanting more…

If you have gotten this far… thank you. I know its a lot of content, so it seems fitting to stop here, do some real work for a change and let you know that coming soon (1 month or less from today) I will start to build the approach I have been crafting over the last year to achieve the transition from A.D.E. to D.E.S.

Azure, Rhel upgrades and ADE – part 1 the concept