passt/test/run, branch 2022_10_24.c11277b

test: Move slower tests to end of test run

2022-10-15T00:10:36+00:00

The distro and performance tests are by far the slowest part of the passt
testsuite.  Move them to the end of the testsuite run, so that it's easier
to do a quick test during development by letting the other tests run then
interrupting the test runner.

Signed-off-by: David Gibson 
Signed-off-by: Stefano Brivio

test: Add rudimentary support to run selected tests only

2022-10-14T15:38:24+00:00

To keep this simple, only support tests that have corresponding setup
and teardown functions implied by their path. For example:

  ./run passt/ndp

will trigger the 'passt' setup and teardown functions.

This is not really elegant, but it looks robust, and while David is
considering proper alternatives, it should be quite useful.

Signed-off-by: Stefano Brivio 
Reviewed-by: David Gibson

test: Move passt_test_log_pipe to state directory

2022-09-13T09:12:41+00:00

We use this fifo to send messages to the information pane.  Put it in the
state directory so it doesn't need its own cleanup.

Signed-off-by: David Gibson

test: Create common state directories for temporary files

2022-09-13T09:12:41+00:00

The test scripts create a bunch of temporary files to keep track of
internal state.  Some are made in /tmp with individual mktemp calls, some
go in the passt source directory, and some go in $LOGDIR.  This can
sometimes make it messy to clean up after failed test runs.

Start cleaning this up by creating a single "state" directory ($STATEBASE)
in /tmp for all the state or temporary files used by a single test run.
Clean it up automatically in cleanup() - except when DEBUG==1, because
those files can be useful for debugging test script failures.

We create subdirectories under $STATEBASE for each setup function, exposed
as $STATESETUP.  We also create subdirectories for each test script and
expose those to the scripts as __STATEDIR__.

Signed-off-by: David Gibson

test: Actually run cleanup function

2022-09-13T09:12:41+00:00

We install a cleanup() function with 'trap' in order to clean up temporary
files we generate during the tests.  However, we deinstall it after
run_term, which means it won't run in most of the cases where it would be
useful.  Even if "run from_term" exits with an error, that error will be
hidden from the run_term wrapper because it's within a tmux session, so we
will return from run_term normally, uninstall the trap and never clean up.

In fact there's no reason to uninstall the trap at all, it works just as
well on the success exit path as an error exit path.

Signed-off-by: David Gibson

test: Group tests by mode then protocol, rather than the reverse

2022-09-13T09:12:41+00:00

For example, passt/dhcp rather than dhcp/passt.  This is more
consistent with the two_guests and other test groups, and makes some
other cleanups simpler.

Signed-off-by: David Gibson

test: Integration of old-style pane execution and new context execution

2022-09-13T03:32:00+00:00

We're creating a system for tests to more reliably execute commands in
various contexts (e.g. host, guest, namespace).  That transition is going
to happen over a number of steps though, so in the meantime we need to deal
with both the old-style issuing of commands via typing into and screen
scraping tmux panels, and the new-style system for executing commands in
context.

Introduce some transitional helpers which will issue a command via context
if the requested context is initialized, but will otherwise fall back to
the old style tmux panel based method.  Re-implement the various test DSL
commands in terms of these new helpers.

Signed-off-by: David Gibson

test: Log debugging output from test script

2022-08-20T17:07:12+00:00

The test scripts run with sh -e, which means they will stop if any commands
return an error.  That's generally desirable, because we won't continue
after things are hopeless due to an earlier step failing.

Unfortunately, the tmux setup we run the script in means it's not obvious
where any error messages related to such a failure will go.  Depending on
exactly where the error occurs they might go to the original terminal
hidden behind tmux, or they might go to a tmux panel that's not visible in
the normal layouts.

To make it easier to find such error message, redirect direct output and
errors from the test script itself to a 'script.log' file in the logs
directory.  When in DEBUG=1 mode, additionaly 'set -x' so we log all the
commands we execute to that file.

Signed-off-by: David Gibson

test: Use shutdown test for pasta

2022-08-20T17:07:12+00:00

For the passt and passt_in_ns tests we have a "shutdown" testcase that
checks for any errors from the passt process we were using (including
valgrind warnings).  Do the same for pasta tests, so that we catch any
error codes from the pasta process.

Signed-off-by: David Gibson

test: Rename slightly misleading "valgrind" tests

2022-08-20T17:07:12+00:00

The "valgrind" test cases are designed to pick up errors reported when
passt is running under valgrind.  But what it actually does is just kill
the passt process, then see if it had a non-zero exit code.  That means it
will equally well pick up any other problems which caused passt to exit
with an error status: either something detected within passt or as a result
of passt being killed by an unexpected signal.

The fact that the "valgrind" test is actually responsible for shutting down
the passt process is non-obvious and can lead to problems when selectively
running tests during debugging.

Rename the "valgrind" tests to "shutdown" tests and run it regardless of
whether we're using valgrind or not.  This allows us to remove an ugly
speacial case in the passt_in_ns teardown code.

Signed-off-by: David Gibson