MySQL Encryption: Talking About Keyrings

It has been possible to enable Transparent Data Encryption (TDE) in Percona Server for MySQL /MySQL for a while now, but have you ever wondered how it works under the hood and what kind of implications TDE can have on your server instance? In this blog posts series, we are going to have a look at how TDE works internally. First, we talk about keyrings, as they are required for any encryption to work. Then we explore in detail how encryption in Percona Server for MySQL/MySQL works and what the extra encryption features are that Percona Server for MySQL provides.

MySQL Keyrings

Keyrings are plugins that allow a server to fetch/create/delete keys in a local file (keyring_file) or on a remote server (for example, HashiCorp Vault). All keys are cached locally inside the keyring’s cache to speed up fetching keys. They can be separated into two categories of plugins that use the following:

• Local resource as a backend for storing keys, like local file (we call this resource file-based keyring)
• Remote resource as a backend for storing keys, like Vault server (we call this resource server-based keyring)

The separation is important because depending on the backend, keyrings behave a bit differently, not only when storing/fetching keys but also on startup.

In the case of a file-based keyring, the keyring on startup loads the entire content of the keyring (i.e., key id, key user, key type, together with keys themselves) into the cache.

In the case of server-based keyring (for instance, Vault server), the server loads only a list of the key ids and the key user on the startup so the startup is not slowed by retrieving all of the keys from the server. It is worth mentioning what information is stored in the keyring backend. The keys are lazy-loaded, which means when the first time a server requests a key, the keyring_vault asks the Vault server to send the key. The keyring caches the key in memory to ensure if, in the future, the server can use memory access instead of a TLS connection to the Vault server to retrieve the key.

The record in keyring consist of the following:

• key id – An ID of the key, for instance: INNODBKey-764d382a-7324-11e9-ad8f-9cb6d0d5dc99-1
• key type – The type of key, based on the encryption algorithm used, possible values are: “AES”, “RSA” or “DSA”
• key length – Length is measured in bytes, AES: 16, 24 or 32, RSA 128, 256, 512, and DSA 128, 256 or 384.
• user – Owner of the key. If this key is a system key, such as the Master Key, this field is empty. When the key is created with keyring_udf, this field is the owner of the key.
• key itself

Each key is uniquely identified by pair: key_id, user.

There are also differences when it comes to storing and deleting keys.

The file-based keyring operation should be faster, and the operation is. You may assume the key storage is just a single write of a key to a file, but more tasks are involved. Before any file-based keyring modification, the keyring creates a backup file with the entire content of the keyring and places this backup file next to the keyring file. Let’s say your keyring file is called my_biggest_secrets; the backup is named my_biggest_secrets.backup. Next, the keyring modifies the cache to add or remove a key, and if this task is successful, it dumps (i.e., rewrites the entire content of a keyring file) from the cache into your keyring file. On rare occasions, such as a server crash, you can observe this backup file. The backup file is deleted by keyring next time the keyring is loaded (generally after the server restart).

When storing or deleting a key, the server-based keyring must connect to the server and communicate a “send the key”/”request key deletion” from the server.

Let’s get back to the speed of the server startup. Apart from the keyring itself impacting the startup time, there is also a matter of how many keys must be retrieved from the backend server on startup. Of course, this is especially important for server-based keyrings. On server startup, the server checks what key is needed to decrypt each encrypted table/tablespaces and fetches this key from the keyring. On a “clean” server with Master Key encryption, there should be one Master Key that must be fetched from the keyring. However, there can be more keys required, for instance, when a slave is re-created from master backup, etc. In those cases, it is good to consider the Master Key rotation. I will talk more about that in future blog posts, but I just wanted to outline here that a server that is using multiple Master Keys might startup a bit longer, primarily when server-based keyring is used.

Now let’s talk some more on the keyring_file. When I was developing the keyring_file, the concern was also how to be sure that the keyring_file was not changed under the running server. In 5.7, the check is done based on file stats, which is not a perfect solution and this solution was replaced in 8.0 with SHA256 checksum.

When keyring_file is first started, the file stats and checksum are calculated and remembered by the server, and the changes are only applied if those match. Of course, the checksum is updated as the file gets updated.

We have covered lots of ground on keyrings so far. There is one more important topic, though, that is often forgotten or misunderstood – the per-server separation of keyrings, and why this is essential. What do I mean by that? I mean that each server (let’s say Percona Server) in a cluster should have a separate place on the Vault server where Percona Server should store its keys. Master Keys stored in the keyring have each Percona Server’s GUID embedded into their ids. Why is this important? Imagine you have one Vault Server with keys, and all of the Percona Servers in your cluster are using this one Vault server. The problem seems obvious – if all of the Percona Servers were using Master Keys without unique ids – for instance, id = 1, id = 2, etc. – all the Percona servers in the cluster would be using the same Master Key. What the GUID provides is this per-server separation. Why talk about the per-server separation of keyrings, since there is already a separation with the unique GUID per Percona server? Well, there is one more plugin, keyring_udf. With this plugin, a user of your server can store their own keys inside the Vault server. The problem arises when your user creates a key on, let’s say server1, and then attempts to create a key with the same identifier (key_id) on server2, like this:

--server1:
select keyring_key_store('ROB_1','AES',"123456789012345");
1
--1 means success
--server2:
select keyring_key_store('ROB_1','AES',"543210987654321");
1

Wait. What!? Since both servers use the same Vault server, should not the keyring_key_store fail on the server2? Interesting enough, if you try to do the same on just one server, you will get a failure:

--server1:
select keyring_key_store('ROB_1','AES',"123456789012345");
1
select keyring_key_store('ROB_1','AES',"543210987654321");
0

Right, ROB_1 already exists.

Let’s discuss the second example first. As we discussed earlier – the keyring_vault or any other keyring plugin is caching all of the key ids in memory. So after the new key, ROB_1 is added on server 1 and apart from sending this key to Vault, the key is also added to the keyring’s cache. Now, when we try to add the same key for the second time, keyring_vault checks if this key already exists in the cache and will error out.

The story is different in the first example. Keyring on server1 has its own cache of the keys stored on the Vault server, and server2 has its own cache. After ROB_1 is added to the keyring’s cache on server1 and Vault server, the keyring’s cache on server2 is out of sync. The cache on server2 does not have the ROB_1 key; thus, writes to the keyring_key_store and writes ROB_1 to the Vault server which actually overrides (!) the previous value. Now the key ROB_1 on the Vault server is 543210987654321. Interesting enough, the Vault server does not block such actions and happily overrides the old value.

Now we see why this per-server separation on the Vault server can be significant – in case you allow the use of keyring_udf, and also if you want to store keys in order in your Vault. How can we ensure this separation on the Vault server?

There are two ways of separation on the Vault server. You can create mount points in the Vault server – a mount point per server, or you can use different paths inside the same mount point, with one path per server. It is best to explain those two approaches by examples. So let’s have a look at configuration files. First for mount point separation:

--server1:
vault_url = http://127.0.0.1:8200
secret_mount_point = server1_mount
token = (...)
vault_ca = (...)
--server2:
vault_url = http://127.0.0.1:8200
secret_mount_point = sever2_mount
token = (...)
vault_ca = (...)

We can see that we have server1 is using different mount point than server2. In a path separation the config files would look like the following:

--server1:
vault_url = http://127.0.0.1:8200
secret_mount_point = mount_point/server1
token = (...)
vault_ca = (...)
--server2:
vault_url = http://127.0.0.1:8200
secret_mount_point = mount_point/sever2
token = (...)
vault_ca = (...)

In this case, both servers are using the same secret mount point – the “mount_point,” but different paths. When you create the first secret on server1 in this path – the Vault server automatically creates a “server1” directory. The actions are the same for server2. When you remove the last secret in mount_point/server1 or mount_point/server2, then the Vault server removes these directories also. As we can see in case you use the path separation, you must create only one mount point and modify the configuration files to make servers use separate paths. The mount point can be created with an HTTP request. With CURL it’s:

curl -L -H "X-Vault-Token: TOKEN" –cacert VAULT_CA
--data '{"type":"generic"}' --request POST VAULT_URL/v1/sys/mounts/SECRET_MOUNT_POINT

All of the fields (TOKEN, VAULT_CA, VAULT_URL, SECRET_MOUNT_POINT) correspond to the options from the keyring configuration file. Of course, you can also use the vault binary to do the same. The point is that mount point creation can be automated. I hope you will find this information helpful, and we will see each other in the next blog post of this series.

Thanks,

Robert