Prerequisite data
=================

Naming
------

An item of prerequisite data is identified by two strings, called ``IDEN1``
and ``IDEN2``; together these are the *prerequisite data identifiers* for an
item of prerequisite data.  Typically ``IDEN1`` specifies the context in which
the data is relevant, and ``IDEN2`` identifies the data within its context.
``IDEN2`` is very often the filename in which the prerequisite data will
eventually be stored.  It might also be a human-readable string describing the
purpose of the data.  The following are the valid forms of ``IDEN1``, and
their meanings.

- ``(HOSTNAME . PATH)`` means the data that should be uploaded to ``PATH`` on
  ``HOSTNAME`` (and usually nowhere else, except in the case of, e.g., a
  public key).  ``PATH`` must be absolute, not relative.

- ``("--lisp-system" . SYSTEM)`` means the data is Lisp code which, when
  loaded, defines the packages and symbols contained in the ASDF system
  ``SYSTEM``

- ``("--user-passwd--HOSTNAME" . USER)`` means the data is the password for
  user ``USER`` on ``HOSTNAME``

- ``("--git-snapshot" . NAME)`` means the data is a snapshot of a git repo
  identified by ``NAME``; see ``DATA.GIT-SNAPSHOT``

- ``("--pgp-pubkey" . FINGERPRINT)`` means the/a OpenPGP public key with
  fingerprint FINGERPRINT, ASCII-armoured

- ``("--pgp-seckey" . FINGERPRINT)`` means the/a OpenPGP secret key with
  fingerprint FINGERPRINT, ASCII-armoured

- ``("--luks-passphrase" . VOLUME-LABEL)`` means a LUKS passphrase for volume
  with label ``VOLUME-LABEL``.

- Any other ``IDEN1`` beginning with exactly two hyphens is reserved for
  future use.

- ``(_CONTEXT . ITEM)`` is an arbitrary prerequisite data context named
  ``CONTEXT``; typically ``CONTEXT`` will be a network or grouping name,
  rather than referring to a single host.  ``ITEM`` might be a path or some
  other identifier.  Reserved for consfigs; will not be used by property
  definitions included with Consfigurator, and should not be used by third
  party extensions.

- ``(---CONTEXT . ITEM)`` is, similarly, an arbitrary prerequisite data
  context named ``CONTEXT``.  This form is intended for contexts similar to
  the reserved names beginning with two hyphens: types of information rather
  than site-local network or grouping names.  This form will not be used by
  property definitions included with Consfigurator, but may be used by both
  consfigs and third party extensions.

Any other forms are invalid.  In particular, an ``IDEN1`` that is not a valid
hostname and does not begin with a hyphen or an underscore must not be used.

Mechanics
---------

Properties declare that they need certain pieces of prerequisite data by
adding static informational attributes, and a deployment of those properties
will make an attempt to provide the data.  Properties then either call the
``GET-DATA-STREAM`` function or the ``GET-DATA-STRING`` function, or depend on
the ``DATA-UPLOADED`` property, to get access to the requested data.

A Lisp connection gathers all needed prerequisite data once at the beginning,
and copies it to an on-disk cache inside the home directory of the remote UID
which will run the Lisp image.  A POSIX connection only attempts to obtain
prerequisite data when a property's check function indicates the property is
not already applied.

Sources of prerequisite data
----------------------------

Sources of prerequisite data register two functions.  The second returns
either a string of the prerequisite data itself, or a path to a file
containing the data.  The first returns the latest version number of the data
that source is able to provide -- i.e., the version number of the data that
the second function would return if called.

Consfigurator will call the first function to find out if it needs to call the
first rather than just using its caches.  The first function should return nil
if it can't obtain the prerequisite data on this host, perhaps because it
can't decrypt the store.  If a prerequisite data source wants to effectively
bypass caching and provide fresh data every time Consfigurator deploys the
host, it can use ``GET-UNIVERSAL-TIME`` as its first function.

Versions are compared using ``UIOP:VERSION<`` and ``UIOP:VERSION<=``.

Security issues
---------------

Nothing is done to prevent prerequisite data being swapped out, so ensure your
swap is encrypted.

Certain connection types require storing unencrypted copies of prerequisite
data under ``~/.cache/consfigurator/data``.  Consfigurator only stores data
there when it has to, only the subset of the data that has to be uploaded for
the requested deployment to be successful, and never in the root Lisp.

API reference
-------------

Prerequisite data
~~~~~~~~~~~~~~~~~

Class: ``DATA``
^^^^^^^^^^^^^^^

An item of prerequisite data as provided by a registered prerequisite data
source, or, outside of the root Lisp, as fished out of a local cache of
prerequisite data.

Class: ``STRING-DATA``
^^^^^^^^^^^^^^^^^^^^^^

An item of prerequisite data directly accessible to Lisp.

Class: ``FILE-DATA``
^^^^^^^^^^^^^^^^^^^^

An item of prerequisite data accessible via the filesystem.

Generic function: ``REGISTER-DATA-SOURCE``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(register-data-source type &key)``

Initialise and register a source of prerequisite data in this Lisp image.
Registered data sources are available to all deployments executed from the
root Lisp, regardless of the consfig which defines the host to which
properties are to be applied.  (This could only cause problems if you have
different consfigs with prerequisite data which is identified by the same two
strings, in which case you will need to wrap your deployments with registering
and unregistering data sources.  Usually items of prerequisite data are
identified using things like hostnames, so this shouldn't be necessary.)

Implementations of this function return a pair of functions.

Signals a condition MISSING-DATA-SOURCE when unable to access the data source
(e.g. because can't decrypt it).  This condition is captured and ignored in
all new Lisp images started up by Consfigurator, since prerequisite data
sources are not expected to be available outside of the root Lisp.

Simple error: ``MISSING-DATA-SOURCE``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(missing-data-source)``

Function: ``TRY-REGISTER-DATA-SOURCE``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(try-register-data-source &rest args)``

Register sources of prerequisite data.
This function is typically called in consfigs.  Any relative pathnames in ARGS
will be resolved as paths under the home directory of the user Lisp is running
as, before being passed to implementations of REGISTER-DATA-SOURCE.

Function: ``RESET-DATA-SOURCES``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(reset-data-sources)``

Forget all data sources registered in this Lisp image and items of string
data obtained from data sources by this Lisp image.
This function is typically called at the REPL.

Macro: ``WITH-RESET-DATA-SOURCES``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(with-reset-data-sources &body body)``

Run BODY with initially empty data sources and string data.

This macro is typically used for testing or debugging.

Function: ``GET-DATA-STRING``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(get-data-string iden1 iden2)``

Return the content of an item of prerequisite data as a string.

This function is called by property :APPLY and :UNAPPLY subroutines.

Function: ``GET-DATA-STREAM``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(get-data-stream iden1 iden2)``

Return a stream which will produce the content of an item of prerequisite
data.  The elements of the stream are always octets.  If the item of
prerequisite data was provided by the prerequisite data source as a string, it
will be encoded in UTF-8.

This function is called by property :APPLY and :UNAPPLY subroutines.

Macro: ``WITH-DATA-STREAM``
^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(with-data-stream s &body body)``

Condition class: ``MISSING-DATA``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Function: ``DATA-SOURCE-PROVIDING-P``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(data-source-providing-p iden1 iden2)``

Is there a data source which can provide the item of prerequisite data
identified by IDEN1 and IDEN2?

This function is for implementation of REGISTER-DATA-SOURCE to check for
clashes.  It should not be called by properties.

Function: ``MAYBE-WRITE-REMOTE-FILE-DATA``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(maybe-write-remote-file-data path iden1 iden2 &key mode)``

Wrapper around WRITE-REMOTE-FILE which returns :NO-CHANGE and avoids touching
PATH if PATH's content is already the prerequisite data identified by IDEN1
and IDEN2 and PATH has mode MODE.

Generic function: ``CONNECTION-UPLOAD``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(connection-upload connection data)``

Subroutine to upload an item of prerequisite data to the remote cache.
The default implementation will work for any connection which implements
CONNECTION-WRITE-FILE and CONNECTION-RUN, but connection types which work by
calling CONTINUE-DEPLOY* or CONTINUE-DEPLOY*-PROGRAM will need their own
implementation.

Generic function: ``CONNECTION-CLEAR-DATA-CACHE``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(connection-clear-data-cache connection iden1 iden2)``

Delete all versions of the data identified by IDEN1 and IDEN2 from the remote
cache of CONNECTION.  Called by UPLOAD-ALL-PREREQUISITE-DATA before uploading
new versions of data, to avoid them piling up.

Function: ``UPLOAD-ALL-PREREQUISITE-DATA``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(upload-all-prerequisite-data &optional connection)``

Upload all prerequisite data required by the current deployment to the remote
cache of the current connection hop, or to the remote cache of CONNECTION.

This is called by implementations of ESTABLISH-CONNECTION which call
CONTINUE-DEPLOY* or CONTINUE-DEPLOY*-PROGRAM.

Function: ``DATA-PATHNAME``
^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(data-pathname root &rest segments)``

Function: ``LOCAL-DATA-PATHNAME``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(local-data-pathname &optional iden1 iden2 version)``

Get a pathname where an item of prerequisite data may be cached, ensuring
that parent directories exist.
This is exported for use by prerequisite data sources which work by generating
new files and need somewhere to store them.  It should not be used by
properties, or data sources which return objects referencing existing files.

Note that since prerequisite data sources are queried only in the root Lisp,
but items of prerequisite data are never uploaded to the root Lisp, there is
no risk of clashes between fresly generated files and cached copies of files.

Function: ``REMOTE-DATA-PATHNAME``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(remote-data-pathname &rest args)``

Remote caches
~~~~~~~~~~~~~

Generic function: ``GET-REMOTE-CACHED-PREREQUISITE-DATA``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(get-remote-cached-prerequisite-data connection)``

Return a list of items of prerequisite data in the cache on the remote side
of CONNECTION, where each entry is of the form

.. code-block:: none

  '(iden1 iden2 version).

Local caches
~~~~~~~~~~~~

Function: ``GET-LOCAL-CACHED-PREREQUISITE-DATA``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(get-local-cached-prerequisite-data where)``

Scan a local cache of prerequisite data at WHERE, and return a list of
items of prerequisite data where each entry is of the form

.. code-block:: none

  '(iden1 iden2 version).

This is exported for use by implementations of CONNECTION-UPLOAD, which should
always supply a value for WHERE.

Function: ``GET-HIGHEST-LOCAL-CACHED-PREREQUISITE-DATA``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(get-highest-local-cached-prerequisite-data iden1 iden2)``

Get the highest version of prerequisite data identified by IDEN1 and IDEN2
available in the local cache.

This is exported for use by prerequisite data sources which work by generating
new files and need somewhere to store them.  It should not be used by
properties, or data sources which return objects referencing existing files.

Passphrases
~~~~~~~~~~~

Class: ``WRAPPED-PASSPHRASE``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Function: ``WRAP-PASSPHRASE``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(wrap-passphrase passphrase)``

Make an object which is unprintable by default to contain a passphrase.

Function: ``GET-DATA-PROTECTED-STRING``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(get-data-protected-string iden1 iden2)``

Like GET-DATA-STRING, but wrap the content in an object which is unprintable
by default.  Intended for code which fetches passwords and wants to lessen the
chance of those passwords showing up in the clear in the Lisp debugger.

Variable: ``*DATA-SOURCE-GNUPGHOME*``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Home directory for gnupg when used in a data source.

Because gnupg uses Unix domain sockets internally, this path should be short
enough to avoid the 108 char limit on socket paths.
