Deployment Choices: NFS Security¶

How to Enable NFS Security¶

The following checklist provides a reference for enabling NFS security. Be advised that some of the commands, shown in the examples, may differ depending on your environment.

Table 4.16. Checklist to Enable NAS Security

1) Change NAS security options.

Set the nas_secure_file_operations and nas_secure_file_permissions to specify the NAS security mode. Make changes to /etc/cinder/cinder.conf in the backend’s configuration stanza.

[replace-with-nfs-backend]
...
nas_secure_file_operations = true
nas_secure_file_permissions = true
...

2) Determine the cinder user’s UID and GID.

$ id -u cinder
500
$ id -g cinder
510

3) Add users to cinder group.

To have file access, Nova and Glance service users need to belong to the same group as the Cinder user. This step needs to be performed on each node running Nova or Glance services.

...
$ usermod -a -G replace-with-cinder-GID nova
$ usermod -a -G replace-with-cinder-GID glance
...
$ id nova
uid=520(nova) gid=521(nova) groups=510(cinder),...
$ id glance
uid=530(glance) gid=531(glance) groups=510(cinder),...
...

4) Change QEMU configuration.

Certain compute operations (i.e. attaching a volume) require that Libvirt, Qemu, and KVM run as a user belonging to the correct group. Edit the /etc/libvirt/qemu.conf file and make the following changes.

...
#user = "root"
user = "nova"
...
#group = "root"
group = "cinder"
...
#dynamic_ownership = 1
dynamic_ownership = 0
...

5) Edit the Domain setting for idmapd (NFSv4.x only).

Idmapd is the NFSv4 bidirectional ID/name mapping daemon. The domain defined in the /etc/idmapd.conf must match the NFS server domain. The first step is to query ONTAP for the domain. The second step is to then edit the /etc/idmapd.conf file and restart the idmapd service. This step is not necessary if using NFSv3.

...
CDOT:> vserver nfs show -vserver replace-with-vserver-name -fields v4-id-domain
...
vserver  v4-id-domain
-------- ---------------------
replace- nfsv4domain.somewhere.com
...

Edit the /etc/idmapd.conf on the Cinder node:

...
Domain = nfsv4domain.somewhere.com
...

6) Add nova, glance, and cinder users to cinder GID.

If local files are used, then the cluster leverages the unix-user and unix-group tables created for the specified SVM. The nova, glance, and cinder SVM users, if they exist, need to belong to the same cinder GID (510) as used by the cinder service. The nova and glance users can be created with the ONTAP CLI unix-user create command if needed.

...
CDOT:> unix-group create -vserver replace-with-vserver-name -name cinder -id replace-with-cinder-GID
CDOT:> unix-group show -vserver replace-with-vserver-name
...
Vserver        Name                ID
-------------- ------------------- ----------
replace-with-  cinder              510
...
CDOT:> unix-user modify -vserver replace-with-vserver-name -user nova -primary-gid replace-with-cinder-GID
CDOT:> unix-user modify -vserver replace-with-vserver-name -user glance -primary-gid replace-with-cinder-GID
CDOT:> unix-user modify -vserver replace-with-vserver-name -user cinder -primary-gid replace-with-cinder-GID
CDOT:> unix-user show -vserver replace-with-vserver-name
...
               User            User   Group  Full
Vserver        Name            ID     ID     Name
-------------- --------------- ------ ------ --------------------------------
replace-with-  cinder          500    510
replace-with-  nova            501    510
replace-with-  glance          502    510
...

7) Set exported Flexvol owner and group.

Access to a Flexvol can be further restricted by only allowing a specific User ID (UID) and Group ID (GID). The UID must match the cinder UID of the Cinder node. The GID must match the cinder GID of the Cinder node. In this example, the UID is 500 and the GID is 510. These values will be different on your cinder node and must be determined prior to running the following commands.

CDOT:> volume show -vserver replace-with-vserver-name -volume replace-with-volume-name
...
User ID: 0
Group ID: 0
...
CDOT:> volume modify -vserver replace-with-vserver-name -volume replace-with-volume-name -user replace-with-cinder-UID -group replace-with-cinder-GID
CDOT:> volume show -vserver replace-with-vserver-name -volume replace-with-volume-name
...
User ID: 500
Group ID: 510
...

8) Set exported Flexvol permissions.

Access can be further restricted by setting the UNIX permissions on a volume. In this example we set the Flexvol permissions, of the shared volume, to 0755. This step is optional.

CDOT:> volume show -vserver replace-with-vserver-name -volume replace-with-volume-name
...
UNIX Permissions: ---rwxrwxrwx
...
CDOT:> volume modify -vserver replace-with-vserver-name -volume replace-with-volume-name -unix-permissions 0755
CDOT:> volume show -vserver replace-with-vserver-name -volume replace-with-volume-name
...
UNIX Permissions: ---rwxr-xr-x
...

9) Update file permissions to 0660.

Other OpenStack services (i.e. Nova and Glance) need “group” rw privileges in order to access the cinder volumes. This is accomplished by running chmod 0660 on all files in the mount points. Verify that the IP address, of the mount point, matches a LIF IP address of the correct SVM prior to executing the chmod and chown commands. Order of operations are stop Cinder services, run chmod and chown, and unmount mount points.

$ systemctl stop openstack-cinder-{api,scheduler,volume}
$ mount
...
192.168.100.10:/cinder_flexvol_1 on /var/lib/cinder/mnt/69809486d67b39d4baa19744ef3ef90c type nfs (rw,...,addr=192.168.100.10)
192.168.100.10:/cinder_flexvol_2 on /var/lib/cinder/mnt/5821d3908bfae68920f0c7be2dfc0c7b type nfs (rw,...,addr=192.168.100.10)
...
$ cd /var/lib/cinder/mnt/69809486d67b39d4baa19744ef3ef90c
$ chmod -R 0660 *
$ chown -R cinder:cinder
$ cd /var/lib/cinder/mnt/5821d3908bfae68920f0c7be2dfc0c7b
$ chmod -R 0660 *
$ chown -R cinder:cinder
$ cd /var/lib/cinder/mnt
$ sudo umount 69809486d67b39d4baa19744ef3ef90c
$ sudo umount 5821d3908bfae68920f0c7be2dfc0c7b

10) Disable superuser access in export policy.

Disabling superuser access in the export policy is effectively the same as enabling root squash. Any root access from a NFS client (i.e. UID 0) is remapped to the anonymous user, default UID is 65534, when superuser access is disabled. This step also disables set user ID (suid) access. The following example also disables set user ID (suid) and set group ID (sgid) access.

CDOT:> vserver export-policy rule show -vserver replace-with-vserver-name -policyname replace-with-policy-name -fields superuser,allow-suid
...
vserver  policyname ruleindex superuser allow-suid
-------- ---------- --------- --------- ----------
replace- cinder     1         any       true
...
CDOT:> vserver export-policy rule modify -vserver replace-with-vserver-name -policyname replace-with-policy-name -ruleindex replace-with-rule-index -protocol nfs -superuser none -allow-suid false
CDOT:> vserver export-policy rule show -vserver replace-with-vserver-name -policyname replace-with-policy-name -fields superuser,allow-suid
...
vserver  policyname ruleindex superuser allow-suid
-------- ---------- --------- --------- ----------
replace- cinder     1         none      false
...

11) Verify mounts and permissions.

In the previous step we unmounted the NFS mounts to prove that they are mounted properly when the Cinder volume service starts. Verify this by starting Cinder services, examining the Cinder volume service log, creating a new Cinder volume, and listing the volume on the mount point.

$ systemctl start openstack-cinder-{api,scheduler,volume}
$ cinder create --name test-vol-01 1
...
| id                             | 9c989cba-eff6-4847-b5fc-bff2ab5d35da |
...
$ ls -l /var/lib/cinder/mnt/5821d3908bfae68920f0c7be2dfc0c7b/volume-9c989cba-eff6-4847-b5fc-bff2ab5d35da
...
-rw-rw-rw- 1 root root 1073741824 Oct 12 13:15 /var/lib/cinder/mnt/5821d3908bfae68920f0c7be2dfc0c7b/volume-9c989cba-eff6-4847-b5fc-bff2ab5d35da
...

How to Disable NFS Security¶

The following checklist provides a reference for disabling NFS security. Be advised that some of the commands, shown in the examples, may differ depending on your environment.

Table 4.17. Checklist to Disable NFS Security

1) Update NAS Security options.

Set the nas_secure_file_operations and nas_secure_file_permissions to specify the NAS security mode. Make changes to /etc/cinder/cinder.conf in the backend’s configuration stanza.

[nfs_backend]
...
nas_secure_file_operations = false
nas_secure_file_permissions = false
...

2) Enable Superuser access in the export policy.

CDOT:> vserver export-policy rule show -vserver replace-with-vserver-name -policyname replace-with-policy-name -fields superuser
...
vserver  policyname ruleindex superuser
-------- ---------- --------- ---------
replace- cinder     1         none
...
CDOT:> vserver export-policy rule modify -vserver replace-with-vserver-name -policyname replace-with-policy-name -ruleindex replace-with-rule-index -protocol nfs -superuser any
CDOT:> vserver export-policy rule show -vserver replace-with-vserver-name -policyname replace-with-policy-name -fields superuser
...
vserver  policyname ruleindex superuser
-------- ---------- --------- ---------
replace- cinder     1         any
...

3) Update file permissions to 0666.

Other OpenStack services (i.e. Nova and Glance) need “world” rw privileges in order to access the cinder volumes. This is accomplished by running chmod 0666 on all files in the mount points. Verify that the IP address, of the mount point, matches a LIF IP address of the correct SVM prior to executing the chmod and chown commands. Order of operations are stop Cinder services, run chmod, unmount mount points, and start Cinder services.

$ systemctl stop openstack-cinder-{api,scheduler,volume}
$ mount
...
192.168.100.10:/cinder_flexvol_1 on /var/lib/cinder/mnt/69809486d67b39d4baa19744ef3ef90c type nfs (rw,...,addr=192.168.100.10)
192.168.100.10:/cinder_flexvol_2 on /var/lib/cinder/mnt/5821d3908bfae68920f0c7be2dfc0c7b type nfs (rw,...,addr=192.168.100.10)
...
$ cd /var/lib/cinder/mnt/69809486d67b39d4baa19744ef3ef90c
$ chmod -R 0666 *
$ chown -R root:root
$ cd /var/lib/cinder/mnt/5821d3908bfae68920f0c7be2dfc0c7b
$ chmod -R 0666 *
$ chown -R root:root
$ cd /var/lib/cinder/mnt
$ sudo umount 69809486d67b39d4baa19744ef3ef90c
$ sudo umount 5821d3908bfae68920f0c7be2dfc0c7b
$ systemctl start openstack-cinder-{api,scheduler,volume}

4) Delete the .cinderSecureEnvIndicator file if it exists.

The Cinder volume service, under certain conditions, creates the .cinderSecureEnvIndicator file as an indicator that NAS security is enabled.

$ mount
...
192.168.100.10:/cinder_flexvol_1 on /var/lib/cinder/mnt/69809486d67b39d4baa19744ef3ef90c type nfs (rw,...,addr=192.168.100.10)
192.168.100.10:/cinder_flexvol_2 on /var/lib/cinder/mnt/5821d3908bfae68920f0c7be2dfc0c7b type nfs (rw,...,addr=192.168.100.10)
...
$ cd /var/lib/cinder/mnt/69809486d67b39d4baa19744ef3ef90c
$ rm .cinderSecureEnvIndicator
$ cd /var/lib/cinder/mnt/5821d3908bfae68920f0c7be2dfc0c7b
$ rm .cinderSecureEnvIndicator

5) Verify mounts and permissions.

In the previous step we unmounted the NFS mounts to prove that they are mounted properly when the Cinder volume service starts. Verify this by examining the Cinder volume service log, creating a new Cinder volume, and listing the volume on the mount point. It is recommended that Nova services be restarted followed by verification that attaching a volume to a compute instance works.

$ cinder create --name test-vol-01 1
...
| id                             | 9c989cba-eff6-4847-b5fc-bff2ab5d35da |
...
$ ls -l /var/lib/cinder/mnt/5821d3908bfae68920f0c7be2dfc0c7b/volume-9c989cba-eff6-4847-b5fc-bff2ab5d35da
...
-rw-rw-rw- 1 root root 1073741824 Oct 12 13:15 /var/lib/cinder/mnt/5821d3908bfae68920f0c7be2dfc0c7b/volume-9c989cba-eff6-4847-b5fc-bff2ab5d35da
...