Key Concepts¶

Volume¶

A Cinder volume is the fundamental resource unit allocated by the Block Storage service. It represents an allocation of persistent, readable, and writable block storage. A Cinder volume can be used as the root disk for a compute instance or as secondary storage that can be attached to or detached from a compute instance. The connection between the consumer and the Cinder service can use iSCSI, NFS, Fibre Channel, or NVMe/TCP storage protocols, depending on the Cinder driver in use.

Cinder volumes are uniquely identified by a UUID assigned at creation. A volume may also have a human-readable name, but this is not guaranteed to be unique within a tenant or deployment.

The actual blocks for a Cinder volume reside on a single Cinder backend. Starting with the Havana release, a Cinder volume can be migrated between storage backends within a deployment. See “Cinder Command Line Interface (CLI)” for a migration example.

The cinder manage command allows importing existing storage objects not managed by Cinder as new Cinder volumes. The operation locates an object in a specified backend and creates the necessary metadata in the Cinder database to manage it like any other Cinder volume. The operation also renames the volume to a name appropriate to the particular Cinder driver in use. The imported object could be a file, LUN, NVMe namespace, or volume, depending on the protocol (iSCSI/FC/NFS/NVMe). This feature is useful for migration scenarios where virtual machines or other data need to be managed by Cinder. See “Cinder Manage Usage” for an example.

The cinder unmanage command allows Cinder to stop managing a specific volume. All related data in the Cinder database is removed, but the backing file, LUN, or storage object is not deleted. This enables transferring the volume to another environment. See “Cinder Unmanage Usage” for an example.

Snapshot¶

A Cinder snapshot is a point-in-time, read-only copy of a Cinder volume. Snapshots can be created from an existing volume that is operational and either attached or detached. A snapshot can serve as the source for a new volume (using the create from snapshot option) or be used to revert a volume to the most recent snapshot (as of the Pike release).

Backend¶

A Cinder backend is the configuration object that represents a single provider of block storage upon which provisioning requests may be fulfilled. A Cinder backend communicates with the storage system through a Cinder driver. Cinder supports multiple backends to be simultaneously configured and managed (even with the same Cinder driver) as of the Grizzly release.

Driver¶

A Cinder driver is a particular implementation of a Cinder backend that maps the abstract APIs and primitives of Cinder to appropriate constructs within the particular storage solution underpinning the Cinder backend.

Volume Type¶

A Cinder volume type is an abstract collection of criteria used to characterize Cinder volumes. They are most commonly used to create a hierarchy of functional capabilities that represent a tiered level of storage services; for example, a cloud administrator might define a premium volume type that indicates a greater level of performance than a basic volume type, which would represent a best-effort level of performance.

The collection of criteria is specified as a list of key/value pairs, which are inspected by the Cinder scheduler when determining which Cinder backend(s) are able to fulfill a provisioning request. Individual Cinder drivers (and subsequently Cinder backends) may advertise arbitrary key/value pairs (also referred to as capabilities) to the Cinder scheduler, which are then compared against volume type definitions when determining which backend will fulfill a provisioning request.

Extra Spec¶

An extra spec is a key/value pair, expressed in the style of key=value. Extra specs are associated with Cinder volume types, so that when users request volumes of a particular volume type, the volumes are created on storage backends that meet the specified criteria.

Quality of Service¶

The Cinder Quality of Service (QoS) support for volumes can be enforced either at the hypervisor or at the storage subsystem (backend), or both.

SolidFire

Within the SolidFire platform, each volume may be configured with minimum, maximum, and burst IOPS values that are strictly enforced within the system. The minimum IOPS provides a guarantee for performance, independent of what other applications on the system are doing. The maximum and burst values control the allocation of performance and deliver consistent performance to workloads.

QoS support for the SolidFire drivers includes the ability to set the following capabilities in the OpenStack Block Storage API cinder.api.contrib.qos_specs_manage qos specs extension module:

Table 4.1a. SolidFire QoS Options

ONTAP

The NetApp ONTAP Cinder driver currently supports QoS by backend QoS specs or via netapp:qos_policy_group assignment using Cinder Extra-Specs. The NetApp Cinder driver accomplishes this by using NetApp QoS policy groups, introduced with ONTAP 8.2, and applying these policy groups to Cinder volumes.

• netapp:qos_policy_group: A Cinder extra-spec, which references an externally provisioned QoS policy group, provides a means to assign a Netapp QoS policy group for a set of Cinder volumes. All Cinder volumes associated with a single QoS policy group share the throughput value restrictions as a group. The ONTAP QoS policy group must be created by the storage administrator on the backend prior to specifying the netapp:qos_policy_group option in a Cinder extra-spec. Whether the policy group is the adaptive type use the Cinder extra-spec netapp:qos_policy_group_is_adaptive. Use the netapp:qos_policy_group option when a Service Level Objective (SLO) needs to be applied to a set of Cinder volumes. For more information on this, see Table 4.11, “NetApp supported Extra Specs for use with Cinder volume Types”.

• QoS Spec: QoS specifications are added as standalone objects that can then be associated with Cinder volume types. A Cinder QoS Spec will create a new NetApp QoS policy group for each Cinder volume. A Cinder QoS spec can specify either non-adaptive (Table 4.1b) or an adaptive (Table 4.1c) QoS throughput values. For the first method, it can be a maximum, a minimum or both QoS, using bytes per second or IOPS. For the adaptive method, it must set the maximum (peak) and the minimum (expected) QoS together, and IOPS values are dynamically set according to the volume size. When deleting a Cinder volume that has a QoS Spec applied, the NetApp QoS policies associated with that Cinder volume will not immediately be deleted. The driver marks the QoS policies for deletion by the NetApp QoS policy reaping job. The NetApp QoS policy reaping job runs every 60 seconds. Refer to NetApp ONTAP documentation for your version of ONTAP to determine NetApp QoS policy group limits. Use the QoS Spec feature when a SLO needs to be applied to a single Cinder volume.

Table 4.1b. NetApp Non-Adaptive Supported Backend QoS Spec Options.

Table 4.1c. NetApp Adaptive Supported Backend QoS Spec Options.

Storage Pools¶

Starting with the Juno release of OpenStack, Cinder introduced the concept of “storage pools.” Backend storage can present one or more logical storage resource pools, and Cinder selects a pool as the storage location when provisioning volumes. In releases prior to Juno, NetApp’s Cinder drivers contained logic to determine which FlexVol volume, volume group, or DDP a Cinder volume would be placed into. With the introduction of pools, all scheduling logic is handled by the Cinder scheduler.

For NetApp’s Cinder drivers, a Cinder pool is a single container. The container mapped to a Cinder pool depends on the storage protocol:

• iSCSI, NVMe/TCP, and Fibre Channel: A Cinder pool is created for every FlexVol volume within the SVM specified by netapp_vserver, or for ONTAP, all FlexVol volumes in the system unless limited by netapp_pool_name_search_pattern.

• NFS: A Cinder pool is created for each junction path from FlexVol or FlexGroup volumes listed in nfs_shares_config.

For more information, see Cinder Scheduling and Resource Pool Selection.

Generic Volume Groups¶

Starting with the Newton release, NetApp supports Generic Volume Groups for ONTAP iSCSI, Fibre Channel, and NFS drivers. SolidFire support was added in the Pike release. Existing consistency group operations will be migrated to use generic volume group operations in future releases. The Consistency Group construct cannot be easily extended for features like consistent group snapshots across multiple volumes. Generic volume groups allow grouping volumes used by the same application, even if they do not support consistent group snapshots. This provides a way to manage and group volumes by a common factor. For more information, see the generic-volume-groups documentation.

Consistency Groups¶

With the Mitaka release of OpenStack, NetApp supports Cinder Consistency Groups when using ONTAP iSCSI/Fibre Channel drivers. With the Newton release of OpenStack, NetApp supports Cinder Consistency Groups when using ONTAP NFS drivers. Consistency group support allows snapshots of multiple volumes in the same consistency group to be taken at the same point-in-time to ensure data consistency. To illustrate the usefulness of consistency groups, consider a bank account database where a transaction log is written to Cinder volume V1 and the account table itself is written to Cinder volume V2. Suppose that $100 is to be transferred from account A to account B via the following sequence of writes:

1. Log start of transaction.

2. Log remove $100 from account A.

3. Log add $100 to account B.

4. Log commit transaction.

5. Update table A to reflect -$100.

6. Update table B to reflect +$100.

Writes 1-4 go to Cinder volume V1 whereas writes 5-6 go to Cinder volume V2. To see that we need to keep write order fidelity in both snapshots of V1 and V2, suppose a snapshot is in progress during writes 1-6, and suppose that the snapshot completes at a point where writes 1-3 and 5 have completed, but not 4 and 6. Because write 4 (log of commit transaction) did not complete, the transaction will be discarded. But write 5 has completed anyways, so a restore from snapshot of the secondary will result in a corrupt account database, one where account A has been debited $100 without account B getting the corresponding credit.

Before using consistency groups, update the policies for the consistency group APIs in /etc/cinder/policy.json. By default, these APIs are disabled. Enable them before running consistency group operations. Example policy entries:

"consistencygroup:create": "group:nobody",
"consistencygroup:delete": "group:nobody",
"consistencygroup:update": "group:nobody",
"consistencygroup:get": "group:nobody",
"consistencygroup:get_all": "group:nobody",
"consistencygroup:create_cgsnapshot" : "group:nobody",
"consistencygroup:delete_cgsnapshot": "group:nobody",
"consistencygroup:get_cgsnapshot": "group:nobody",
"consistencygroup:get_all_cgsnapshots": "group:nobody",

Remove group:nobody to enable these APIs:

"consistencygroup:create": "",
"consistencygroup:delete": "",
"consistencygroup:update": "",
"consistencygroup:get": "",
"consistencygroup:get_all": "",
"consistencygroup:create_cgsnapshot" : "",
"consistencygroup:delete_cgsnapshot": "",
"consistencygroup:get_cgsnapshot": "",
"consistencygroup:get_all_cgsnapshots": "",

Remember to restart the Block Storage API service after changing policies.

The NetApp driver creates consistency group LUN snapshots as thick provisioned. This can be changed on the backend after the snapshot is taken with no effect to Cinder.

Backup and Restore¶

Cinder provides self-service backup and restore operations for Cinder volumes. These operations are performed on individual volumes. A backup operation creates a point-in-time, read-only set of data and metadata that can be used to restore a single Cinder volume to a new or existing volume. Unlike snapshots, backups are stored in a dedicated repository, independent of the original storage pool or backend.

Cinder backup repositories can use an object store (such as Swift) or an NFS shared filesystem. The backup service uses a single repository, regardless of the backends used for storage pools. For example, a FlexVol volume exported from ONTAP using NFS can serve as a backup repository for multi-backend, heterogeneous Cinder deployments.

Tenant-controlled, per-volume backup service complements, but does not replace, administrative backups of the storage pools that hold Cinder volumes. See http://netapp.github.io/openstack/2015/03/12/cinder-backup-restore/ for an approach to administrative backups with ONTAP storage pools.

Disaster Recovery¶

NetApp’s Cinder driver for ONTAP (FC, NFS, iSCSI) supports Cinder’s replication v2.1 specification. This enables replication of an entire backend and allows all replicated volumes across different pools to fail over together. The driver uses ONTAP SnapMirror as the replication transport.

Starting with the Hibiscus release, three SnapMirror policy families are available:

• Asynchronous (MirrorAllSnapshots or Asynchronous) — Replicates data at scheduled intervals. Supports iSCSI, FC, and NFS on ONTAP 9.16.1+. Suitable for disaster recovery scenarios where a small recovery point lag is acceptable.

• AutomatedFailOver — Synchronous replication (zero RPO, near-zero RTO once triggered) using ONTAP SnapMirror Active Sync in asymmetric mode. Supports iSCSI and FC on ONTAP 9.15.1+. REST API required.

• AutomatedFailOverDuplex — Synchronous replication (zero RPO, near-zero RTO) using ONTAP SnapMirror Active Sync in symmetric (stretched) mode. Both sites serve I/O simultaneously and ONTAP fails over automatically, with no administrator action required. Supports iSCSI and FC on ONTAP 9.15.1+. REST API required. See AutomatedFailOverDuplex Policy (SnapMirror Active Sync — Symmetric) for full details.

All policies support grouping all FlexVol volumes in a backend stanza under a single shared Consistency Group (netapp_consistent_replication = True), ensuring crash-consistent failover across all volumes. This is mandatory for AutomatedFailOverDuplex and optional for others.

For full configuration details, prerequisites, and failover/failback procedures, see SnapMirror Replication Policies for Cinder.

SnapMirror Active Sync¶

Starting with the Hibiscus release, the NetApp Cinder driver supports ONTAP SnapMirror Active Sync, which extends synchronous replication to enable storage-level high availability without requiring a MetroCluster configuration.

SnapMirror Active Sync maintains a continuously synchronized copy of data on a secondary ONTAP cluster. The two modes are:

Asymmetric (AutomatedFailOver) — One site serves all I/O (active/standby). In the event of a failure, an administrator triggers failover. Zero data loss (RPO = 0) and near-zero recovery time (RTO in seconds to low minutes)

Symmetric / Stretched (AutomatedFailOverDuplex) — Both sites serve I/O simultaneously. When one site fails, ONTAP automatically redirects all I/O to the surviving site with no operator action required. Zero data loss (RPO = 0) and near-zero recovery time (RTO in seconds to low minutes). This mode is designed for stretched cluster deployments where compute nodes span two physical availability zones.

In symmetric mode, the driver configures host proximity on ONTAP igroup objects to indicate which storage site is closest to each compute host. This is controlled by the netapp_proximal_nodes configuration option, which lists the iSCSI IQNs or FC WWPNs of initiators local to the SVM in each backend stanza.

Revert to Snapshot¶

As of the Pike release, Cinder supports the revert to snapshot feature. This feature can overwrite the current state and data of a volume with the most recent snapshot. A volume cannot be reverted if it was extended after the snapshot was taken.

The SolidFire driver uses an optimized implementation. For ONTAP backends, the feature uses a generic implementation for NFS, iSCSI, FC, and NVMe drivers. From the Xena release, ONTAP NFS, iSCSI, and FC drivers use a safer and faster storage-based approach.

Storage Assisted Migration¶

Starting with the Xena release, Cinder supports storage assisted migration. This feature can migrate ONTAP NFS, iSCSI, and FC drivers within the same cluster. It is non-disruptive within the same SVM and disruptive between different SVMs. For NFS drivers, migration is always disruptive. Scenarios:

1. Within a Storage Virtual Machine (SVM): Non-disruptive for iSCSI and FC drivers. NFS drivers require the volume to be in available status.

2. Between SVMs in the same cluster: Disruptive in all cases; volume must be in available status.

3. Between two different clusters: Not supported for storage assisted; driver falls back to host assisted migration.

Active/Active high availability mode support¶

Starting with the Bobcat release (2023.2), Cinder supports active/active high availability mode for NetApp NFS protocol-based backends. This feature allows failover of ONTAP NFS drivers within or across ONTAP clusters. If a backend is exposed via NFS to Cinder, all volumes in that backend can be automatically failed over to replication targets in the OpenStack cluster when the host is down. For Cinder Active/Active support, NetApp uses the replication functionality built in previous releases.

1. To use Active/Active support, configure the replication target properly for the backend on the primary host. Use the following options in cinder.conf:

replication_device = backend_id:target_cmodeNfs

netapp_replication_aggregate_map = backend_id:target_cmodeNfs,source_aggr_1:destination_aggr_1,source_aggr_2:destination_aggr_2

2. If the backend differs between primary and secondary, configure vserver and cluster peering properly to handle SnapMirror relationships between source and destination.

MCC(MetroCluster Configuration) support¶

Metro cluster is a clustering solution with synchronous replication to deliver continuous availability, immediately duplicating all of the mission-critical data on a transaction-by-transaction basis. MetroCluster configurations enhance the built-in high availability and non disruptive operations of NetApp hardware and ONTAP storage software, providing an additional layer of protection for the entire storage and host environment.

Approach:

After switchover: Update the following parameters in the cinder.conf file for the storage backend stanza and restart the Cinder services.

netapp_server_hostname = <secondary_cluster_host_name_or_ip> netapp_server_hostname = <vserver_name>-mc

For example, if the vserver name on the primary site is vs_iscsi, then after switchover, netapp_server_hostname should be set to vs_iscsi-mc. After making these changes, restart the Cinder service.

After switchback: Revert the netapp_server_hostname parameters to their original values (primary cluster hostname or IP and vserver name) and restart the cinder service.