Connections
===========

Connection chain specifications
-------------------------------

The normalised form is a list of lists, where the car of each inner list is a
keyword symbol identifying a connection type, and the cdr of each inner list
is arguments to that connection, e.g.::

  ((:ssh :foo foo :bar bar) (:sudo :baz baz :quux quux))

There are two notational simplifications permitted when passing connection
chain specifications to properties, functions and macros.  Firstly, for each
inner list which contains only a single keyword identifying a connection type
and no arguments, this list may be replaced with only the keyword identifying
the connection type, e.g.::

  (:ssh (:sudo :baz baz :quux quux))

Secondly, when there is exactly one connection and it takes no arguments, you
may specify just the keyword identifying the connection type, e.g. ``:ssh``.

Note that if there is a single connection but it takes arguments, you will
need two sets of parentheses, i.e.::

  ((:ssh :foo foo :bar bar))

rather than::

  (:ssh :foo foo :bar bar)

which is invalid.

Defining connection types
-------------------------

The code which establishes connections (i.e., implementations of the
``ESTABLISH-CONNECTION`` generic) is like code in ``:posix`` properties -- it
should restrict its I/O to ``RUN``, ``RUNLINES``, ``READ-REMOTE-FILE`` and
``WRITE-REMOTE-FILE``, functions which access the currently active connection.
This is in order to permit the arbitrary nesting of connections.  If
establishing a connection really does require more I/O, such as in the case of
``:CHROOT.FORK`` connections, code can call ``LISP-CONNECTION-P``, and either
signal an error, or fall back to another connection type.

Connection attributes ("connattrs")
-----------------------------------

Information about hosts which cannot be known without looking at the host, or
for other reasons should not be recorded in consfigs, can be stored as
connection attributes, associated with the current connection.  Typically
property combinators set and unset connattrs, and property ``:APPLY`` and
``:UNAPPLY`` subroutines read them.  They can be used to create context for
the application of properties.  Connection attributes are stored in a plist.
Property combinators use the ``WITH-CONNATTRS`` macro to set them, and
properties use ``GET-CONNATTR`` to read them.

Like hostattrs, connection attributes are identified by keywords for connattrs
which are expected to be used in many contexts, and by other symbols for
connattrs which will be used only among a co-operating group of properties and
property combinators.  However, unlike hostattrs, each connattr need not be a
list to which new items are pushed.

By default the list of connattrs is reset when establishing a new connection
within the context of an existing connection.  However, for some connattrs it
makes sense to propagate them along to the new connection.  For example, a
list of connected hardware of a particular type might still be useful in the
context of a connection which chroots, as /dev might still give access to this
hardware.  Implementations of the ``PROPAGATE-CONNATTR`` generic function can
be used to enable propagation where it makes sense.  Methods can copy and
modify connattrs as appropriate; in the chroot example, paths might be updated
so that they are relative to the new filesystem root.

The propagation of connattrs is currently limited to the establishing of
connections within the same Lisp image; i.e., connection types which start up
new Lisp images never propagate any existing connattrs.

Reserved names for connection attributes
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The semantics of connattrs identified by keywords are documented here.

[list incomplete]

Notes on particular connection types
------------------------------------

``:SUDO``
~~~~~~~~~

Passing the ``:AS`` option to this connection means that Consfigurator will
assume a password is required for all commands, and not passing ``:AS`` means
that Consfigurator will assume a password is not required for any commands.
Consfigurator sends your sudo password on stdin, so if the assumption that a
password is required is violated, your sudo password will end up in the stdin
to whatever command is being run using sudo.  There is no facility for
directly passing in a passphrase; you must use ``:AS`` to obtain passwords
from sources of prerequisite data.  The passphrase will be written to a
private temporary file which is deleted when the ``:SUDO`` connection is torn
down.

If any connection types which start up remote Lisp images occur before a
``:SUDO`` entry in your connection chain, ``ESTABLISH-CONNECTION`` will need
to inform the newly-started remote Lisp image of any sudo passwords needed for
establishing the remaining hops.  Depending on how the connection type feeds
instructions to the remote Lisp image, this may involve writing your sudo
password to a file under ``~/.cache`` on the machine which runs the remote
Lisp image.  At least ``:SBCL`` avoids this by sending your password in on
stdin.  Even with ``:SBCL``, if the Lisp image dumps a copy of itself to disk,
e.g. for the purposes of cronjobs, then your sudo password will be contained
in that saved image.  Typically a ``:SUDO`` connection hop is used before hops
which start up remote Lisp images, so these issues will not arise for most
users.

``:SETUID``
~~~~~~~~~~~

As this connection type subclasses FORK-CONNECTION, it shouldn't leak
root-accessible secrets to a process running under the unprivileged UID.
However, when using the :AS connection type, the unprivileged process will
have access to all the hostattrs of the host.  Potentially, something like
ptrace(2) could be used to extract those.  But hostattrs should not normally
contain any secrets, and at least on Linux, the unprivileged process will not
be ptraceable because it was once privileged.

Connections which fork: ``:CHROOT.FORK``, ``:SETUID``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

These connection types cannot be used as the first hop, i.e., directly out of
the root Lisp.  This is because they must call fork(2), and Consfigurator only
makes this system call in contexts in which there shouldn't ever be more than
one thread (excluding Lisp implementation finaliser threads and the like).
The root Lisp is not such a context, because it is often multithreaded due to
the use of SLIME.  This is, however, not much of a restriction, because
typically the root Lisp is running under a UID which cannot use system calls
like chroot(2) and setuid(2) anyway.  Thus, typical usage on localhost would
be something like::

  (deploy (:sudo :sbcl (:chroot.fork :into "...")) ...)

Connections which use setns(2) to enter containers
--------------------------------------------------

When the current connection is a Lisp-type connection, connection types which
enter Linux containers, such as ``:LXC`` and ``:SYSTEMD-MACHINED``, invoke the
setns(2) system call directly.  The implementation of this is the connection
type ``CONSFIGURATOR.CONNECTION.LINUX-NAMESPACE::SETNS``.  The implementation
of the ``POST-FORK`` generic for that connection type is structured similarly
to the nsenter(1) command from util-linux.  This has the advantage that
``CONSFIGURATOR.CONNECTION.LINUX-NAMESPACE::SETNS`` should be reusable for
implementing connection types which enter other kinds of Linux container; the
container runtime-specific code is limited to determining the PID of the
container's leading process.  However, there are some security implications to
this approach.

Firstly, the current implementation does not join the control group of the
container's leading process, and thus the Consfigurator process running inside
the container is not subject to resource limits applied to the container.  It
might be possible for a process in the container to exploit this to escape its
resource limits.

Secondly, we do not attempt to enter the LSM security context of the
container, such as the container's SELinux execution context or AppArmor
profile.  This is because LSM usage is container runtime-specific.  In the
case of unprivileged containers which make use of user namespaces, however,
failing to enter the LSM security context typically does not breach container
security.  For such containers, employment of an LSM serves as an extra layer
of protection against kernel exploits, not as part of the enforcement of the
container's basic security model.

API reference
-------------

Connections
~~~~~~~~~~~

Generic function: ``ESTABLISH-CONNECTION``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(establish-connection type remaining &key)``

Within the context of the current connection, connect to HOST by
establishing a new connection of type TYPE.
Either returns an object suitable to be the value of *CONNECTION*, or calls
either CONTINUE-DEPLOY* or CONTINUE-DEPLOY*-PROGRAM and returns nil.

Any implementation which calls CONTINUE-DEPLOY*-PROGRAM will need to call
UPLOAD-ALL-PREREQUISITE-DATA.

Generic function: ``CONTINUE-CONNECTION``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(continue-connection connection remaining)``

Called by implementations of ESTABLISH-CONNECTION which return nil.
Calls CONTINUE-DEPLOY* or CONTINUE-DEPLOY*-PROGRAM.

Generic function: ``PREPROCESS-CONNECTION-ARGS``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(preprocess-connection-args type &key)``

Hook to allow connection types to do work in the root Lisp before
Consfigurator begins the attempt to establish the connection chain.  The
return value is used as replacement keyword arguments to the connection.

For an example of usage, see the :SUDO connection type.

Class: ``CONNECTION``
^^^^^^^^^^^^^^^^^^^^^

Class: ``LISP-CONNECTION``
^^^^^^^^^^^^^^^^^^^^^^^^^^

Class: ``POSIX-CONNECTION``
^^^^^^^^^^^^^^^^^^^^^^^^^^^

Function: ``LISP-CONNECTION-P``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(lisp-connection-p)``

Generic function: ``CONNECTION-RUN``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(connection-run connection cmd input)``

Subroutine to run shell commands on the host.

INPUT is a string to send to the shell command's stdin, or a stream which will
be emptied into the shell command's stdin.

Implementations can specialise on both the CONNECTION and INPUT arguments, if
they need to handle streams and strings differently.

Returns (values OUT EXIT) where OUT is either merged stdout and stderr or
stderr followed by stdout, and EXIT is the exit code.  Should not signal any
error condition just because EXIT is non-zero.

Generic function: ``CONNECTION-READ-FILE``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(connection-read-file connection path)``

Subroutine to read the contents of files on the host.

Generic function: ``CONNECTION-READ-AND-REMOVE-FILE``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(connection-read-and-remove-file connection path)``

As READ-REMOTE-FILE and then delete the file.

For some connection types, when latency is high, combining these two
operations is noticeably faster than doing one after the other.  For every use
of RUN we read and delete the file containing the command's stdout, so the
time savings add up.

Generic function: ``CONNECTION-WRITE-FILE``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(connection-write-file connection path content mode)``

Subroutine to replace/create the contents of files on the host.

CONTENT is the new contents of the file or a stream which will produce it.

MODE is the numeric mode that the file should have by the time this function
returns.  Implementations should ensure that CONTENT is not stored on disk
with a mode greater than MODE, and also that if CONTENT is stored on disk
outside of (UIOP:PATHNAME-DIRECTORY-PATHNAME PATH), then it does not
have a mode greater than 700.  It is recommended that implementations write
CONTENT to a temporary file in (UIOP:PATHNAME-DIRECTORY-PATHNAME PATH),
change the mode of that file to MODE, and then rename to PATH.
WITH-REMOTE-TEMPORARY-FILE can be used to do this.

Implementations can specialise on both the CONNECTION and CONTENT arguments,
if they need to handle streams and strings differently.

Generic function: ``CONNECTION-TEAR-DOWN``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(connection-tear-down connection)``

Subroutine to disconnect from the host.

Generic function: ``CONNECTION-CONNATTR``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(connection-connattr connection k)``

Get the connattr identified by K for CONNECTION.

Generic function: ``PROPAGATE-CONNATTR``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(propagate-connattr type connattr connection)``

Possibly propagate CONNATTR, a connattr identified by TYPE, through to the
newly-established CONNECTION.  Implementations should specialise on TYPE and
CONNECTION, not modify any of their arguments, and either return the new
connattr, or nil if nothing should be propagated.

Functions to access the slots of the current connection
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Condition class: ``RUN-FAILED``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Macro: ``WITH-REMOTE-TEMPORARY-FILE``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(with-remote-temporary-file file &body body)``

Execute BODY with FILE containing the path to a freshly created remote file,
which will be cleaned up when BODY is finished.

Function: ``MKSTEMP-CMD``
^^^^^^^^^^^^^^^^^^^^^^^^^

``(mkstemp-cmd &optional template)``

Function: ``MKTEMP``
^^^^^^^^^^^^^^^^^^^^

``(mktemp &key connection directory)``

Make a temporary file on the remote side, in DIRECTORY, defaulting to /tmp.

Macro: ``WITH-REMOTE-CURRENT-DIRECTORY``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(with-remote-current-directory dir &body forms)``

Execute FORMS with the current working directory DIR.
This affects the working directory for commands run using RUN and MRUN, and
the resolution of relative pathnames passed as the first argument of
READ-REMOTE-FILE and WRITE-REMOTE-FILE.  For Lisp-type connections, it
additionally temporarily sets the working directory of the Lisp process using
UIOP:WITH-CURRENT-DIRECTORY.

Function: ``RUN``
^^^^^^^^^^^^^^^^^

``(run &rest args)``

Synchronous execution of shell commands using the current connection.
ARGS can contain keyword-value pairs (and singular keywords) to specify
aspects of this function's behaviour, and remaining elements of ARGS are the
shell command and its parameters, or, as a special case, a single string
specifying the shell command, with any necessary escaping already performed.
It is recommended that all keywords and corresponding values come first,
followed by argument(s) specifying the shell command to execute.

You can additionally supply lists of arguments and these will be spliced into
the resulting list of arguments to be passed to the command.  I.e.
(run "a" (list "b" "c")) is equivalent to (run "a" "b" "c").

Keyword arguments accepted:

  - :FOR-EXIT / :MAY-FAIL -- don't signal an error condition if the command
    does not exit nonzero, usually because it is being called partly or only
    for its exit code

  - :INFORM -- send a copy of the output to *STANDARD-OUTPUT*

  - :INPUT INPUT -- pass the content of the string or stream INPUT on stdin

  - :ENV ENVIRONMENT -- where ENVIRONMENT is a plist specifying environment
    variable names and values, use env(1) to set these variables when running
    the command.  An environment variable value of nil means that the variable
    should be unset.

Returns command's stdout, stderr and exit code, unless :FOR-EXIT, in which
case return only the exit code.

Function: ``MRUN``
^^^^^^^^^^^^^^^^^^

``(mrun &rest args)``

Like RUN but don't separate stdout and stderr ("m" for "merged"; note
that this might mean interleaved or simply concatenated, depending on the
connection chain).

Some (but not all) connection types will want to use this when implementing
ESTABLISH-CONNECTION, CONNECTION-RUN, CONNECTION-WRITE-FILE etc. to avoid the
overhead of splitting the output streams only to immediately recombine them.

Code in property definitions which will not examine command output should
usually use this in preference to RUN for a performance boost; an exception is
when the command sends a lot of text to stdout which might make it harder for
the user to pick out error messages.  Code which examines command output
should use RUN and only examine the stream from which the output to be read is
expected.

Function: ``RUNLINES``
^^^^^^^^^^^^^^^^^^^^^^

``(runlines &rest args)``

Function: ``REMOTE-TEST``
^^^^^^^^^^^^^^^^^^^^^^^^^

``(remote-test &rest args)``

Function: ``REMOTE-MOUNT-POINT-P``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(remote-mount-point-p path)``

Is PATH a mount point?

Uses mountpoint(1) from util-linux, so add a property requiring OS:LINUX or a
subclass to the :HOSTATTRS subroutine of properties calling this.

Function: ``DELETE-REMOTE-TREES``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(delete-remote-trees &rest paths)``

Recursively delete each of PATHS.

Function: ``EMPTY-REMOTE-DIRECTORY``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(empty-remote-directory directory)``

Recursively delete the contents of DIRECTORY, but not DIRECTORY itself.

Function: ``REMOTE-EXISTS-P``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(remote-exists-p &rest paths)``

Does each of PATHS exist?
PATH may be any kind of file, including directories.

Function: ``REMOTE-EXISTS-EVERY-P``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(remote-exists-every-p &rest paths)``

Does each of PATHS exist?
PATH may be any kind of file, including directories.

Function: ``REMOTE-EXISTS-SOME-P``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(remote-exists-some-p &rest paths)``

Do any of PATHS exist?
PATH may be any kind of file, including directories.

Function: ``REMOTE-FILE-STATS``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(remote-file-stats path)``

Get the numeric mode, size in bytes, mtime, owner and group of PATH, or NIL if
it does not exist.

The mtime is only accurate to the nearest UTC day, rounding down, if the file
was modified in the past six months or its mtime is in the future, and only
accurate to the nearest minute, rounding down, otherwise (see the
specification of POSIX ls(1)).

Function: ``REMOTE-LAST-REBOOT``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(remote-last-reboot)``

Get the time of the last reboot, rounded down to the nearest minute.

Function: ``REMOTE-EXECUTABLE-FIND``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(remote-executable-find executable)``

Function: ``READ-REMOTE-FILE``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(read-remote-file path)``

Function: ``WRITE-REMOTE-FILE``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``(write-remote-file path content &key mode)``

Function: ``GET-CONNATTR``
^^^^^^^^^^^^^^^^^^^^^^^^^^

``(get-connattr k)``

Get the connattr identified by K for the current connection.

Macro: ``WITH-CONNATTRS``
^^^^^^^^^^^^^^^^^^^^^^^^^

``(with-connattrs &rest &body forms)``

Execute FORMS with connattrs replaced as specified by CONNATTRS, a plist.
