=============================================
Attaching virtual persistent memory to guests
=============================================

.. versionadded:: 20.0.0 (Train)

Starting in the 20.0.0 (Train) release, the virtual persistent memory (vPMEM)
feature in Nova allows a deployment using the libvirt compute driver to provide
vPMEMs for instances using physical persistent memory (PMEM) that can provide
virtual devices.

PMEM must be partitioned into `PMEM namespaces`_ for applications to use.
This vPMEM feature only uses PMEM namespaces in ``devdax`` mode as QEMU
`vPMEM backends`_. If you want to dive into related notions, the document
`NVDIMM Linux kernel document`_ is recommended.

To enable vPMEMs, follow the steps below.


Dependencies
------------

The following are required to support the vPMEM feature:

* Persistent Memory Hardware

  One such product is Intel® Optane™ DC Persistent Memory.
  `ipmctl`_ is used to configure it.

* Linux Kernel version >= 4.18 with the following modules loaded:

  ``dax_pmem``, ``nd_pmem``, ``device_dax``, ``nd_btt``

* QEMU version >= 3.1.0

* Libvirt version >= 5.0.0

* `ndctl`_ version >= 62

The vPMEM feature has been verified under the software and hardware listed above.


Configure PMEM namespaces (Compute)
-----------------------------------

#. Create PMEM namespaces as `vPMEM backends`_ using the `ndctl`_ utility.

   For example, to create a 30GiB namespace named ``ns3``:

   .. code-block:: console

      $ sudo ndctl create-namespace -s 30G -m devdax -M mem -n ns3
      {
        "dev":"namespace1.0",
        "mode":"devdax",
        "map":"mem",
        "size":"30.00 GiB (32.21 GB)",
        "uuid":"937e9269-512b-4f65-9ac6-b74b61075c11",
        "raw_uuid":"17760832-a062-4aef-9d3b-95ea32038066",
        "daxregion":{
          "id":1,
          "size":"30.00 GiB (32.21 GB)",
          "align":2097152,
          "devices":[
          {
            "chardev":"dax1.0",
            "size":"30.00 GiB (32.21 GB)"
          }
          ]
        },
        "name":"ns3",
        "numa_node":1
      }

   Then list the available PMEM namespaces on the host:

   .. code-block:: console

      $ ndctl list -X
      [
        {
          ...
          "size":6440353792,
          ...
          "name":"ns0",
          ...
        },
        {
          ...
          "size":6440353792,
          ...
          "name":"ns1",
          ...
        },
        {
          ...
          "size":6440353792,
          ...
          "name":"ns2",
          ...
        },
        {
          ...
          "size":32210157568,
          ...
          "name":"ns3",
          ...
        }
      ]

#. Specify which PMEM namespaces should be available to instances.

   Edit :oslo.config:option:`libvirt.pmem_namespaces`:

   .. code-block:: ini

      [libvirt]
      # pmem_namespaces=$LABEL:$NSNAME[|$NSNAME][,$LABEL:$NSNAME[|$NSNAME]]
      pmem_namespaces = 6GB:ns0|ns1|ns2,LARGE:ns3

   Configured PMEM namespaces must have already been created on the host as
   described above. The conf syntax allows the admin to associate one or more
   namespace ``$NSNAME``\ s with an arbitrary ``$LABEL`` that can subsequently
   be used in a flavor to request one of those namespaces. It is recommended,
   but not required, for namespaces under a single ``$LABEL`` to be the same
   size.

#. Restart the ``nova-compute`` service.

   Nova will invoke `ndctl`_ to identify the configured PMEM namespaces, and
   report vPMEM resources to placement.


Configure a flavor
------------------

Specify a comma-separated list of the ``$LABEL``\ s from
:oslo.config:option:`libvirt.pmem_namespaces` to the flavor's ``hw:pmem``
property. Note that multiple instances of the same label are permitted:

.. code-block:: console

   $ openstack flavor set --property hw:pmem='6GB' my_flavor
   $ openstack flavor set --property hw:pmem='6GB,LARGE' my_flavor_large
   $ openstack flavor set --property hw:pmem='6GB,6GB' m1.medium

.. note:: If a NUMA topology is specified, all vPMEM devices will be put on
          guest NUMA node 0; otherwise nova will generate one NUMA node
          automatically for the guest.

Based on the above examples, an ``openstack server create`` request with
``my_flavor_large`` will spawn an instance with two vPMEMs. One, corresponding
to the ``LARGE`` label, will be ``ns3``; the other, corresponding to the ``6G``
label, will be arbitrarily chosen from ``ns0``, ``ns1``, or ``ns2``.

.. note::

  Using vPMEM inside a virtual machine requires the following:

  * Guest kernel version 4.18 or higher;
  * The ``dax_pmem``, ``nd_pmem``, ``device_dax``, and ``nd_btt`` kernel
    modules;
  * The `ndctl`_ utility.

.. note:: When resizing an instance with vPMEMs, the vPMEM data won't be
          migrated.


Verify inventories and allocations
----------------------------------
This section describes how to check that:

* vPMEM inventories were created correctly in placement, validating the
  `configuration described above <#configure-pmem-namespaces-compute>`_.
* allocations were created correctly in placement for instances spawned from
  `flavors configured with vPMEMs <#configure-a-flavor>`_.

.. note::

  Inventories and allocations related to vPMEM resource classes are on the
  root resource provider related to the compute node.

#. Get the list of resource providers

   .. code-block:: console

      $ openstack resource provider list
      +--------------------------------------+--------+------------+
      | uuid                                 | name   | generation |
      +--------------------------------------+--------+------------+
      | 1bc545f9-891f-4930-ab2b-88a56078f4be | host-1 |         47 |
      | 7d994aef-680d-43d4-9325-a67c807e648e | host-2 |         67 |
      --------------------------------------+---------+------------+

#. Check the inventory of each resource provider to see resource classes

   Each ``$LABEL`` configured in :oslo.config:option:`libvirt.pmem_namespaces`
   is used to generate a resource class named ``CUSTOM_PMEM_NAMESPACE_$LABEL``.
   Nova will report to Placement the number of vPMEM namespaces configured for
   each ``$LABEL``. For example, assuming ``host-1`` was configured as
   described above:

   .. code-block:: console

      $ openstack resource provider inventory list 1bc545f9-891f-4930-ab2b-88a56078f4be
      +-----------------------------+------------------+----------+----------+-----------+----------+--------+
      | resource_class              | allocation_ratio | max_unit | reserved | step_size | min_unit |  total |
      +-----------------------------+------------------+----------+----------+-----------+----------+--------+
      | VCPU                        |             16.0 |       64 |        0 |         1 |        1 |     64 |
      | MEMORY_MB                   |              1.5 |   190604 |      512 |         1 |        1 | 190604 |
      | CUSTOM_PMEM_NAMESPACE_LARGE |              1.0 |        1 |        0 |         1 |        1 |      1 |
      | CUSTOM_PMEM_NAMESPACE_6GB   |              1.0 |        3 |        0 |         1 |        1 |      3 |
      | DISK_GB                     |              1.0 |      439 |        0 |         1 |        1 |    439 |
      +-----------------------------+------------------+----------+----------+-----------+----------+--------+

   Here you can see the vPMEM resource classes prefixed with
   ``CUSTOM_PMEM_NAMESPACE_``. The ``LARGE`` label was configured with one
   namespace (``ns3``), so it has an inventory of ``1``. Since the ``6GB``
   label was configured with three namespaces (``ns0``, ``ns1``, and ``ns2``),
   the ``CUSTOM_PMEM_NAMESPACE_6GB`` inventory has a ``total`` and ``max_unit``
   of ``3``.

#. Check allocations for each server that is using vPMEMs

   .. code-block:: console

      $ openstack server list
      +--------------------------------------+----------------------+--------+-------------------+---------------+-----------------+
      | ID                                   | Name                 | Status | Networks          | Image         | Flavor          |
      +--------------------------------------+----------------------+--------+-------------------+---------------+-----------------+
      | 41d3e139-de5c-40fd-9d82-016b72f2ba1d | server-with-2-vpmems | ACTIVE | private=10.0.0.24 | ubuntu-bionic | my_flavor_large |
      | a616a7f6-b285-4adf-a885-dd8426dd9e6a | server-with-1-vpmem  | ACTIVE | private=10.0.0.13 | ubuntu-bionic | my_flavor       |
      +--------------------------------------+----------------------+--------+-------------------+---------------+-----------------+

      $ openstack resource provider allocation show 41d3e139-de5c-40fd-9d82-016b72f2ba1d
      +--------------------------------------+------------+------------------------------------------------------------------------------------------------------------------------+
      | resource_provider                    | generation | resources                                                                                                              |
      +--------------------------------------+------------+------------------------------------------------------------------------------------------------------------------------+
      | 1bc545f9-891f-4930-ab2b-88a56078f4be |         49 | {u'MEMORY_MB': 32768, u'VCPU': 16, u'DISK_GB': 20, u'CUSTOM_PMEM_NAMESPACE_6GB': 1, u'CUSTOM_PMEM_NAMESPACE_LARGE': 1} |
      +--------------------------------------+------------+------------------------------------------------------------------------------------------------------------------------+

      $ openstack resource provider allocation show a616a7f6-b285-4adf-a885-dd8426dd9e6a
      +--------------------------------------+------------+-----------------------------------------------------------------------------------+
      | resource_provider                    | generation | resources                                                                         |
      +--------------------------------------+------------+-----------------------------------------------------------------------------------+
      | 1bc545f9-891f-4930-ab2b-88a56078f4be |         49 | {u'MEMORY_MB': 8192, u'VCPU': 8, u'DISK_GB': 20, u'CUSTOM_PMEM_NAMESPACE_6GB': 1} |
      +--------------------------------------+------------+-----------------------------------------------------------------------------------+

   In this example, two servers were created. ``server-with-2-vpmems`` used
   ``my_flavor_large`` asking for one ``6GB`` vPMEM and one ``LARGE`` vPMEM.
   ``server-with-1-vpmem`` used ``my_flavor`` asking for a single ``6GB``
   vPMEM.


.. _`PMEM namespaces`: http://pmem.io/ndctl/ndctl-create-namespace.html
.. _`vPMEM backends`: https://github.com/qemu/qemu/blob/19b599f7664b2ebfd0f405fb79c14dd241557452/docs/nvdimm.txt#L145
.. _`NVDIMM Linux kernel document`: https://www.kernel.org/doc/Documentation/nvdimm/nvdimm.txt
.. _`ipmctl`: https://software.intel.com/en-us/articles/quick-start-guide-configure-intel-optane-dc-persistent-memory-on-linux
.. _`ndctl`: http://pmem.io/ndctl/
