============================
Installation and development
============================

The ``valve-infra`` container can in turn be run from a host OS, booted directly
using
`boot2container <https://gitlab.freedesktop.org/mupuf/boot2container>`_
or netbooted over the internet using
`iPXE boot server <ipxe-boot-server/README.md>`_.


Requirements
============

The Valve Infra container is meant to be run on the gateway machine of
a CI farm, and comes with the following requirements:

 - **Hardware**: Two network interfaces, one connected to the internet while
   the other one is connected to a switch where all the test machines
   to be exposed by the gateway are connected.
 - **Volumes**: The container requires a volume to store persistent data
   (mounted at ``/mnt/permanent``), and optionally a temporary volume
   which acts as a cache across reboots (mounted at ``/mnt/tmp``).
 - **Container**: The container needs to be run as privileged, and using
   the host network


Building the container
======================

The container image is built using
`rootless podman
<https://github.com/containers/podman/
blob/main/docs/tutorials/rootless_tutorial.md
#etcsubuid-and-etcsubgid-configuration>`_
, and is provisioned using Ansible recipes (see the
``ansible/`` subproject).

The following is a (likely incomplete) list of dependencies for a system running Debian/Ubuntu:

.. code-block:: bash

    $ apt install buildah \
        git \
        jq \
        make \
        podman \
        python \
        qemu-utils \
        qemu-system \
        skopeo \
        socat \
        wget

Note: In order to reduce round-trips to an external registry, a local registry
is automatically started when running certain make targets.

The container image can be built with:

.. code-block:: bash

	$ make valve-infra-container

Build options
-------------

  - ``V=1`` Turn on more verbose logging messages in the build process
  - ``ANSIBLE_EXTRA_ARGS="-vvv ..."`` Pass any custom flags to
    ``ansible-playbook``. Helpful for re-running only tagged roles in
    the ansible build, for example. You can also use this to override variables
    used by the ansible playbook, for example: ``ANSIBLE_EXTRA_ARGS="-e foo=bar"``
  - ``IMAGE_NAME=localhost:8088/my/image``
    The container name to tag the image with. *WARNING:* The image
    will automatically be pushed to the registry that got tagged!
    Defaults to ``localhost:8088/valve-infra/valve-infra-container:latest``.

Once completed, a container image will be generated, for example,

.. code-block:: text

    Successfully tagged localhost:8088/valve-infra/valve-infra-container:latest
    60cc3db9bedd2a11d8d61a4433a2a8e8daf35a59d6229b80c1fdcf9ece73b7ab

Notice that it defaults to a ``localhost`` registry. This is to save on
bandwidth, since the valve-infra container is quite big.

Running the infrastructure
==========================

While not strictly required, enabling nested KVM on the host will dramatically
improve the performance of a DUT VM booted from the VPDU. It can be enabled by
configuring the relevant module parameter:

.. code-block:: bash

    $ cat /etc/modprobe.d/kvm.conf
    options kvm-intel nested=Y
    options kvm-amd nested=1

Nested KVM support can be verified on the host by reading the "nested"
parameter from sysfs:

.. code-block:: bash

    $ cat /sys/module/kvm_*/parameters/nested
    Y

We are now ready to start our virtual gateway machine, which will boot
directly into the container we built in the previous section:

.. code-block:: bash

    $ make vivian [GITLAB_REGISTRATION_TOKEN=...] [GITLAB_URL=https://gitlab.freedesktop.org]

Note 1: The ``GITLAB_REGISTRATION_TOKEN`` should be retrieved from the
``GITLAB_URL`` GitLab instance (`documentation <https://docs.gitlab.com/ee/ci/
runners/register_runner.html#generate-a-registration-token-deprecated>`_).


Note 2: options to vivian can be passed by setting ``VIVIAN_OPTS``, for example:
``$ make VIVIAN_OPTS="--ssh-port=50022" ... vivian``. See ``vivian/vivian -h`` for
the full list of options it supports.

For information on starting virtual DUTs in the VM, see the section on
"Spawning virtual DUTs" below.

The virtual testing recipes will fetch a Linux kernel and a
boot2container ramdisk, and start the system. After the kernel boots
and loads the ramdisk, the ramdisk will then pull the valve-infra
container, and hand control to it.

**N.B:** Due to the stateful nature of the permanent partition in the
VM's tmp disk, it is wise to occasionally delete said disk and check
a fresh boot continues to work as expected. This can be done using ``make clean``.

By default, the qemu console is redirected to telnet on the localhost, and can
be reachable with: ``telnet localhost 4321``

You should now see a QEMU window, which will boot a Linux kernel, then
boot2container will download the container we built and boot into it.
When this is done, you should now see a dashboard which looks mostly green.
The same dashboard should also be present in the terminal you used to connect.


On some distros, you may receive the following error message when vivian tries
to run qemu:

.. code-block:: bash

    failed to create tun device: Operation not permitted
    qemu-system-x86_64: -nic bridge,br=vivianbr0,mac=DE:AD:BE:EF:00:12,model=virtio-net-pci: bridge helper failed

Re-try after enforcing the user ownership on ``/usr/lib/qemu/qemu-bridge-helper``
on your test system:

.. code-block:: bash

    # chmod u+s /usr/lib/qemu/qemu-bridge-helper

There are several options for getting a shell on the gateway:

- create one from the dashboard by pressing ``Ctrl-b c`` in tmux
- connect via SSH using ``make vivian-connect``
- use telnet to connect to the qemu console with ``telnet localhost 4321``

Spawning virtual DUTs
=====================


Right now, our gateway has no idea about any potential machine connected
to its private network interface. Let's boot one!

If the gateway is running in vivian, then virtual DUTs can be booted by using
the `vivian-dut` make target. The `OUTLET` variable can be set to select the
outlet on the VPDU:

.. code-block:: bash

    $ make OUTLET=4 vivian-add-dut

Alternatively, you can use the dashboard to add/start a virtual DUT by
selecting one with the arrow keys and submitting the **DISCOVER** button.

Once a DUT VM has started, output from it is redirected into a log file located
at ``/mnt/tmp/vpdu/dut-log-<mac address>-<date and time stamp>.log``.

This log file is also tail'd in a new tmux window on the gateway's console
dashboard (``CTRL + B`` in tmux then 0..n to select the window by ID).

If all went well, you should now see a machine appear in the "Machines"
column of the dashboard, with the state ``DISCOVERING``. After a short while, it
should transition into ``TRAINING``.

During the ``TRAINING`` step, the machine will be booted in a loop for a set
amount of times in order to test the boot reliability. After the boot loop is
complete, the machine will be marked as ready for testing and its state should
change to ``IDLE``.

Your first virtual test machine is now available for testing both locally, and
on your chosen Gitlab instance.

Running a job on the virtual gateway
====================================

Now that you have at least one machine available, you may run jobs on it using
the following command:

.. code-block:: bash

    $ executorctl -e http://localhost:8000 run -c 10.0.2.2 -t virtio:family:VIRTIO $JOB_FILE

If all went well, congratulations! You seem to have a functional
setup! Most of the steps above are amenable to further configuration.
You are now in a position to play around and modify defaults to your
testing requirements.

Hacking on the infrastructure
=============================

After making changes to Ansible or any other component of the
valve-infra-container, it is recommended to test the changes by either
re-generating the valve-infra-container, or running the ansible playbook
against an already-running vivian instance:

.. code-block:: bash

    $ make vivian-provision

If you want to iterate faster, you may limit the ansible job to some roles
by using the TAGS= parameter:

.. code-block:: bash

    $ make TAGS=dashboard,minio,executor vivian-provision

You can get a full list of available tags by running:

.. code-block:: bash

    $ ansible-playbook --list-tags ansible/gateway.yml


Production deployment
=====================

The valve infrastructure is meant to be netbooted over HTTPS, with the boot
configuration being stored in a git repository for easy of update.

All of the steps are outlined at `iPXE boot server <ipxe-boot-server/README.md>`_.