test_iperf3() is a pretty inscrutable mess of nested background processes.
It has a number of ugly sleeps needed to wait for things to complete.

Rewrite it to be cleaner:
  * Use the construct (a & b & wait) to run 'a' and 'b' in parallel, but
    then wait for them both to complete before continuing
  * This allows us to wait for both the server and client to finish, rather
    than sleeping
  * Use jq to do all the math we need to get the final result, rather than
    jq followed by some complicated 'bc' mangling

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

Currently all the throughput tests are run for 30s.  This is reflected in
both the actual parameters given to the iperf commands, but also in the
matching sleeps in test_iperf3.

Allow this to be adjusted more easily with a new parameter to test_iperf3.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
[sbrivio: Reflect new parameter in comment to test_iperf3()]
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>

These two commands in the DSL to run an iperf client and server are always
used together, and some of the parameters must match between them.  The
iperf3s must also be run more or less immediately after iperf3c, since
iperf3c will run a client in the background after a sleep and requires a
server to be running before it will work.

A bunch of things can be made cleaner if we make a single DSL command that
runs both sides of the test.  For now make the combined command work
exactly like the two commands together did, warts and all.

This does lose the ability for the DSL scripts to give additional options
to the iperf3 server, but we weren't using that anyway.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

On new Ubuntu 22.04 images, stopping systemd-resolved to get the
dhclient script override resolv.conf doesn't work anymore. I
originally used that hack to avoid introducing a delay which is
needed when running it on TCG.

Keep systemd-resolved running instead, and wait for it to be ready
by retrying to resolve a domain a few times before installing
packages, so that we don't add another ugly delay that might
unnecessarily slow down things even further.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>

Performance tests use iperf3(1) with large windows, and these sysctl
entries are needed to run them unmodified.

The passt demo uses perf(1) to report syscall overhead, and that
needs access to hardware performance counters for unprivileged
users.

Reported-by: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

We start getting prompts about restarting outdated services: we're
using daily images but they might have been cached for a while now.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>

Currently in at least some of the testcases we kill qemu processes we're
done with by issuing a Control-C to the tmux panel it's running in.  That
makes things harder as we try to move towards allowing "headless" testing
without tmux.

So, instead always use an explicit kill on a pid derived from a pidfile
for killing qemu.  Note that we don't need to remove the pidfiles
afterwards, because qemu does that itself when terminated.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

The test scripts run with sh -e, which means they will stop if any commands
return an error.  That's generally desirable, because we won't continue
after things are hopeless due to an earlier step failing.

Unfortunately, the tmux setup we run the script in means it's not obvious
where any error messages related to such a failure will go.  Depending on
exactly where the error occurs they might go to the original terminal
hidden behind tmux, or they might go to a tmux panel that's not visible in
the normal layouts.

To make it easier to find such error message, redirect direct output and
errors from the test script itself to a 'script.log' file in the logs
directory.  When in DEBUG=1 mode, additionaly 'set -x' so we log all the
commands we execute to that file.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

For the passt and passt_in_ns tests we have a "shutdown" testcase that
checks for any errors from the passt process we were using (including
valgrind warnings).  Do the same for pasta tests, so that we catch any
error codes from the pasta process.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>

The "valgrind" test cases are designed to pick up errors reported when
passt is running under valgrind.  But what it actually does is just kill
the passt process, then see if it had a non-zero exit code.  That means it
will equally well pick up any other problems which caused passt to exit
with an error status: either something detected within passt or as a result
of passt being killed by an unexpected signal.

The fact that the "valgrind" test is actually responsible for shutting down
the passt process is non-obvious and can lead to problems when selectively
running tests during debugging.

Rename the "valgrind" tests to "shutdown" tests and run it regardless of
whether we're using valgrind or not.  This allows us to remove an ugly
speacial case in the passt_in_ns teardown code.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>