Introduction

Managing NVMe-oF gateways in a Ceph cluster is a critical aspect of ensuring seamless performance and scalability. With the help of a specification file, administrators can easily add or remove gateways, scale resources up or down, and maintain consistent configurations across the cluster.

This blog provides a step-by-step guide on how to efficiently manage NVMe-oF gateways using a specification file, from editing the YAML configuration to applying changes with Ceph orchestration commands. Whether you're scaling up to meet growing demands or scaling down to optimize resource utilization, this process ensures flexibility and performance in your storage infrastructure.

Why Use a Specification File?

A specification file allows you to:

• Manage performance effectively.

• Scale NVMe-oF gateways up or down by modifying node configurations.

• Maintain a standardized deployment configuration for easy updates.

Adding and Removing NVMe-oF Gateways

Scaling NVMe-oF gateways involves two steps:

1. Modifying the specification file to add or remove nodes.

2. Reapplying the updated specification file to implement changes.

Applying the Specification File

After editing the file to add or remove host nodes, use the following command to apply the changes:

ceph orch apply -i NVME_SPEC_FILE

Important Consideration: tgt_cmd_extra_args

When scaling up or scaling down, ensure you use the same tgt_cmd_extra_args: --cpumask= parameter as during the initial deployment. This is crucial for maintaining consistent gateway performance. For additional details, refer to the NVMe-oF Gateway Performance Best Practices documentation.

Example Scenarios: Scaling NVMe-oF Gateways in a spec file

Scenario 1: Removing a Node from a Gateway

To remove host02 from nvmeof_pool01, delete the corresponding entry from the YAML file and reapply it.

Before Removing host02:

service_type: nvmeof

service_id: nvmeof_pool01.group2

service_name: nvmeof.nvmeof_pool01.group2

placement:

  hosts:

    - host01

    - host02

spec:

  pool: nvmeof_pool01

  tgt_cmd_extra_args: --cpumask=0xFF

After Removing host02:

service_type: nvmeof

service_id: nvmeof_pool01.group2

service_name: nvmeof.nvmeof_pool01.group2

placement:

  hosts:

    - host01

spec:

  pool: nvmeof_pool01

  tgt_cmd_extra_args: --cpumask=0xFF

Run the following command to apply the changes:

ceph orch apply -i NVME_SPEC_FILE

Scenario 2: Adding a Node to a Gateway

To add host03, update the YAML file by including the new host and then reapply it.

Updated YAML File After Adding host03:

service_type: nvmeof

service_id: nvmeof_pool01.group2

service_name: nvmeof.nvmeof_pool01.group2

placement:

  hosts:

    - host01

    - host02

    - host03

spec:

  pool: nvmeof_pool01

  tgt_cmd_extra_args: --cpumask=0xFF

Apply the changes using the same command:

ceph orch apply -i NVME_SPEC_FILE

Key Takeaways

• Use a specification file for efficient gateway management.

• Always use consistent tgt_cmd_extra_args parameters for performance integrity.

• Regularly review and update the specification file as needed for scaling.

Conclusion 

By following the above steps, you can manage NVMe-oF gateways efficiently, ensuring optimal performance and scalability.

Let us know if you have any questions or need further assistance in managing gateways in better way!