Integration of IBM Storage Ceph Object with Vault Server (Agent-Based authentication)

Introduction 

This blog continues our previous discussion on Ceph Object Storage encryption at rest, where we introduced data encryption concepts, discussed various encryption options, and detailed the integration with IBM GKLM for IBM Storage Ceph Object (refer to the blog by Daniel Parkes). 

This document, the second in our series on encryption providers, focuses on integrating HashiCorp Vault with IBM Storage Ceph Object. 

Ceph Vault Integration

Integrating Vault with Ceph enhances security in object storage by managing encryption keys centrally. Ceph Object Gateway works with Vault to retrieve encryption keys when needed, supporting two types of Vault engines:

1. K/V Engine: Stores encryption keys as static data. This is ideal when the same key is required for multiple operations.

2. Transient Engine: Provides one-time keys for single-use encryption or decryption, which enhances security by minimizing the exposure of encryption keys.

Workflow

1. Client: Creates a secret key for a key ID and uploads the object with this key ID.

2. Ceph Object Gateway: Requests the secret key from Vault.

3. Vault: Returns the key, either retrieved (K/V) or generated dynamically (Transient).

4. Encryption: Ceph Object Gateway encrypts the object with the key and stores it in Ceph OSD.

Goal 

This document aims to set up a HashiCorp Vault server and a Vault Agent on RHEL-9.3 to generate and manage encryption keys using Vault's transit engine. These keys would encrypt objects stored in Ceph RGW with AES256 encryption, enhancing data security.

Method

We have used “Agent” based authentication and a “Transit” secret engine for secret/encryption key generation.

Steps Overview

1. Setting Up Vault Server: Install and configure a Vault server to manage encryption keys on RHEL-9.3

2. Utilizing the Transit Engine: Configure Vault's Transit Engine for secret key generation and encryption operations.

3. Configuring Vault Agent: Deploy Vault Agent/s for authentication and automatic retrieval of secrets.

4. Integrating with Ceph RGW: Use the generated keys to encrypt objects in Ceph RGW with AES256 via either SSE-KMS or SSE-s3

Steps in detail

1. Setting Up Vault Server

• Installation of vault

• Create a hashicorp.repo file with the below contents in the folder /etc/yum.repos.d/ as shown in the image.

• Execute “yum install -y vault”

• Check the vault version vault -v (or vault –version)

• Starting the vault server

• Create a config.hcl having details of the host where you want to configure the vault server.

• Start the vault server with the configuration as specified in the config.hcl file

2. Utilizing the Transit Engine

• Test the Vault Server 

• After the vault server is configured, on a separate terminal, ensure it is running on port 8200

• Authenticate via the Vault

• Export VAULT_ADDR and PATH

Note: Every initialized Vault server starts in a sealed state. Vault can access the physical storage from the configuration, but it can't read any of it because it doesn't know how to decrypt it. The process of teaching Vault how to decrypt the data is known as unsealing the Vault

Perform ‘vault operator unseal’ 3 times (as the threshold no. of keys for unsealing the vault is 3). After the vault server is unsealed, we will receive the below output:

• Perform a vault operator init 

• This will provide five keys and a root token. Save them in a secure file since these keys and the root token are frequently used to unseal the vault server and log in.

• Unseal and login to Vault Server.

• Check the vault status, and it shows “Sealed true”

• To unseal the vault server, use any of the three the Unseal keys obtained through the “vault operator init” and perform the “vault operator unseal” command three times, each time giving a unique unseal key.

• The vault server will be unsealed when the vault status shows “sealed false”

• Do vault login with root token obtained before from vault operator init.

• Vault Token policy for Transit engine

• Write a vault token policy for the Transit secret engine at the path transit/keys, which has the CRUD capabilities and is rotated at desired intervals.

• Vault Secret enable for Transit engine. 

• Run the “ vault secrets enable transit” cli to enable the creation of secrets/keys on the transit path.

• Write an encryption key “testKey01” at the transit/ path and enable key rotation after, say, every 1 hour.

• Enable key rotation after every 1 hour.

•  Enable AppRole

• We must enable the approved authentication method to assign a role to the vault token policy created above.

• Now, Create an AppRole and associate it with the vault token policy.

• Fetch the role ID and generate a secret ID

• Generate a Secret ID

3. Configuring Vault Agent

• Login to the client node or the localhost

Run the below commands on the vault agent node

• yum install -y vault

• mkdir -p /usr/local/etc/vault/

• Retrieve the app-role-id and app-secret-id from the vault server and copy at /usr/local/etc/vault/.app-role-id and /usr/local/etc/vault/.app-secret-id respectively

• Create a vault-agent config file.

• Create a /usr/local/etc/vault/agent.hcl file having the below content

• Configure vault agent as a service on the Client node

• In our case, the client node is the ceph node having the RGW daemons

4. Integrating with Ceph RGW

• Enable the below ceph configurations for the Vault to be used as an SSE-KMS for the Ceph. Assuming that the rgw service name is “rgw.1”

• ceph config set client.rgw.1 rgw_crypt_s3_kms_backend vault

• ceph config set client.rgw.1 rgw_crypt_vault_addr http://127.0.0.1:8100  

1. This will be the vault_agent

• ceph config set client.rgw.1 rgw_crypt_vault_auth  agent

• ceph config set client.rgw.1 rgw_crypt_vault_prefix /v1/transit

• ceph config set client.rgw.1 rgw_crypt_vault_secret_engine transit

• ceph config set client.rgw.1 rgw_crypt_require_ssl false

• The image below shows Vault configs for the rgw service with the name “rgw.shared.sec”. 

• Note: These are the configs when we use Valut as a KMS

• Now, upload and download an object with the encrypted key “testKey01” created at the vault server.

                  # s3cmd put file1 s3://mytestbucket1/object-1  --server-side-encryption-kms-id testKey01

                 #  Download of the object will be similar to normal object "s3cmd get  s3://mytestbucket1/object1"