aboutgitcodebugslistschat
path: root/test/run
Commit message (Collapse)AuthorAgeFilesLines
* test/lib: Move screen-scraping setup and layout functions to _ugly filesStefano Brivio2022-11-041-0/+2
| | | | | | | | | | | I'm going to add yet another one of those, for which I have no quick solution. It's a regression in some sense, but at least if we make this regression more observable and defined, it should be easier to find a comprehensive solution later, within this or another testing framework. Signed-off-by: Stefano Brivio <sbrivio@redhat.com> Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
* test/run: Temporarily disable distribution tests2022_10_26.e4df8b0Stefano Brivio2022-10-261-0/+4
| | | | | | | | | | | | They're too slow to cope with current release cycles, and they haven't found bugs in months, also because clang-tidy and cppcheck would find most of them earlier. Disable them for the moment. We should pre-install gcc and make in non-x86 images, as those run on my test machine with qemu TCG, and that's the real slow-down here. Then we can re-enable them. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* test: Add log file tests for pasta plus corresponding layout and setupStefano Brivio2022-10-261-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To test log files on a tmpfs mount, we need to unshare the mount namespace, which means using a context for the passt pane is not really practical at the moment, as we can't open a shell there, so we would have to encapsulate all the commands under 'unshare -rUm', plus the "inner" pasta command, running in turn a tcp_rr server. It might be worth fixing this by e.g. detecting we are trying to spawn an interactive shell and adding a special path in the context setup with some form of stdin redirection -- I'm not sure it's doable though. For this reason, add a new layout, using a context only for the host pane, while keeping the old command dispatch mechanism for the passt pane. We also need a new setup function that doesn't start pasta: we want to start and restart it with different options. Further, we need a 'pint' directive, to send an interrupt to the passt pane: add that in lib/test. All the tests before the one involving tmpfs and a detached mount namespace were also tested with the context mechanism. To make an eventual conversion easier, pass tcp_crr directly as a command on pasta's command line where feasible. While at it, fix the comment to the teardown_pasta() function. The new test set can be semi-conveniently run as: ./run pasta_options/log_to_file and it checks basic log creation, size of the log file after flooding it with debug entries, rotations, and basic consistency after rotations, on both an existing filesystem and a tmpfs, chosen as it doesn't support collapsing data ranges via fallocate(), hence triggering the fall-back mechanism for logging rotation. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* test: Move slower tests to end of test runDavid Gibson2022-10-151-10/+10
| | | | | | | | | | The distro and performance tests are by far the slowest part of the passt testsuite. Move them to the end of the testsuite run, so that it's easier to do a quick test during development by letting the other tests run then interrupting the test runner. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* test: Add rudimentary support to run selected tests onlyStefano Brivio2022-10-141-1/+36
| | | | | | | | | | | | | | | To keep this simple, only support tests that have corresponding setup and teardown functions implied by their path. For example: ./run passt/ndp will trigger the 'passt' setup and teardown functions. This is not really elegant, but it looks robust, and while David is considering proper alternatives, it should be quite useful. Signed-off-by: Stefano Brivio <sbrivio@redhat.com> Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
* test: Move passt_test_log_pipe to state directoryDavid Gibson2022-09-131-5/+2
| | | | | | | We use this fifo to send messages to the information pane. Put it in the state directory so it doesn't need its own cleanup. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
* test: Create common state directories for temporary filesDavid Gibson2022-09-131-0/+2
| | | | | | | | | | | | | | | | | | The test scripts create a bunch of temporary files to keep track of internal state. Some are made in /tmp with individual mktemp calls, some go in the passt source directory, and some go in $LOGDIR. This can sometimes make it messy to clean up after failed test runs. Start cleaning this up by creating a single "state" directory ($STATEBASE) in /tmp for all the state or temporary files used by a single test run. Clean it up automatically in cleanup() - except when DEBUG==1, because those files can be useful for debugging test script failures. We create subdirectories under $STATEBASE for each setup function, exposed as $STATESETUP. We also create subdirectories for each test script and expose those to the scripts as __STATEDIR__. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
* test: Actually run cleanup functionDavid Gibson2022-09-131-1/+0
| | | | | | | | | | | | | | We install a cleanup() function with 'trap' in order to clean up temporary files we generate during the tests. However, we deinstall it after run_term, which means it won't run in most of the cases where it would be useful. Even if "run from_term" exits with an error, that error will be hidden from the run_term wrapper because it's within a tmux session, so we will return from run_term normally, uninstall the trap and never clean up. In fact there's no reason to uninstall the trap at all, it works just as well on the success exit path as an error exit path. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
* test: Group tests by mode then protocol, rather than the reverseDavid Gibson2022-09-131-19/+19
| | | | | | | | For example, passt/dhcp rather than dhcp/passt. This is more consistent with the two_guests and other test groups, and makes some other cleanups simpler. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
* test: Integration of old-style pane execution and new context executionDavid Gibson2022-09-131-0/+1
| | | | | | | | | | | | | | | | We're creating a system for tests to more reliably execute commands in various contexts (e.g. host, guest, namespace). That transition is going to happen over a number of steps though, so in the meantime we need to deal with both the old-style issuing of commands via typing into and screen scraping tmux panels, and the new-style system for executing commands in context. Introduce some transitional helpers which will issue a command via context if the requested context is initialized, but will otherwise fall back to the old style tmux panel based method. Re-implement the various test DSL commands in terms of these new helpers. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
* test: Log debugging output from test scriptDavid Gibson2022-08-201-0/+2
| | | | | | | | | | | | | | | | | | | The test scripts run with sh -e, which means they will stop if any commands return an error. That's generally desirable, because we won't continue after things are hopeless due to an earlier step failing. Unfortunately, the tmux setup we run the script in means it's not obvious where any error messages related to such a failure will go. Depending on exactly where the error occurs they might go to the original terminal hidden behind tmux, or they might go to a tmux panel that's not visible in the normal layouts. To make it easier to find such error message, redirect direct output and errors from the test script itself to a 'script.log' file in the logs directory. When in DEBUG=1 mode, additionaly 'set -x' so we log all the commands we execute to that file. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
* test: Use shutdown test for pastaDavid Gibson2022-08-201-0/+1
| | | | | | | | | For the passt and passt_in_ns tests we have a "shutdown" testcase that checks for any errors from the passt process we were using (including valgrind warnings). Do the same for pasta tests, so that we catch any error codes from the pasta process. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
* test: Rename slightly misleading "valgrind" testsDavid Gibson2022-08-201-2/+3
| | | | | | | | | | | | | | | | | | | The "valgrind" test cases are designed to pick up errors reported when passt is running under valgrind. But what it actually does is just kill the passt process, then see if it had a non-zero exit code. That means it will equally well pick up any other problems which caused passt to exit with an error status: either something detected within passt or as a result of passt being killed by an unexpected signal. The fact that the "valgrind" test is actually responsible for shutting down the passt process is non-obvious and can lead to problems when selectively running tests during debugging. Rename the "valgrind" tests to "shutdown" tests and run it regardless of whether we're using valgrind or not. This allows us to remove an ugly speacial case in the passt_in_ns teardown code. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
* test: Split setup/teardown functions for build and distro testsDavid Gibson2022-08-201-0/+4
| | | | | | | | | | | | Currently the build tests and distro tests share a common setup function. That works for now, but changes we want to make will mean they need slightly different setup, so split the setup functions in preparation. Currently, neither build nor distro tests have any teardown function. Again, future changes are going to mean we need to do some teardown, so create some empty for now teardown functions in preparation. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
* test: Split cppcheck and clang-tidy tests into different filesDavid Gibson2022-08-201-1/+2
| | | | | | | | | | Both clang-tidy and cppcheck linting are handled by the same test file, test/build/static_checkers. The two linters are independent of each other though, and each one takes quite a long time. Split them into separate files to make it easier to control which are executed from the top level test script. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
* tests: Explicitly list test files in test/run, remove "onlyfor" supportDavid Gibson2022-07-141-27/+31
| | | | | | | | | | | | | | | | | Currently test/run uses wildcards to run all of the tests in a directory. However, that wildcard list is filtered down by the "onlyfor" directives in the test files... usually to a single file. Therefore, just explicitly list the files we *really* want to run for this test mode. This makes it easier to see at the top level what tests will be executed, and to change that list temporarily while debugging specific failures. This means the "onlyfor" directive no longer has any purpose, and we can remove it. "onlyfor" was also the only used of the $MODE variable, so we can remove that too. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
* tests: Don't automatically traverse directories of test filesDavid Gibson2022-07-141-24/+24
| | | | | | | | | | | | | | | The top level listing control of which tests to run is in test/run, however it uses the test() function which runs an entire directory of test files, filtered by some criteria. This makes it awkward to narrow down to a subset of tests when debugging a specific failure. To make this easier, have test() take an explicit list of test files to run, and have the caller in test/run handle the directory traversal. The construct we use for this is pretty awkward to handle the fact that we're in the source tree root directory rather than test/ at this point in test/run. Later cleanups will improve that. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
* test/run: Return 0 from run(), exit value already reflects failuresStefano Brivio2022-05-191-1/+1
| | | | | | | | | | | There's no need to return non-zero if there have been failures in run(), because the exit value is already determined from the number of failures reported in the log file. Return zero, so that this doesn't cause the script to fail, given we now run it with -e. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tests: Don't ignore errors during scriptDavid Gibson2022-05-191-1/+1
| | | | | | | | | | | | | | | | Most commands issued during the testing scripts aren't explicitly checked for errors. Therefore, if they fail, the shell will just keep on executing. This makes it difficult to figure out where things started going wrong if things fall over. Run the whole script with the set -e mode so that it will exit in the case of any (unchecked) failing command. To make this work we do need to add explicit checks / fallbacks for some commands which we expect to fail. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> [sbrivio: use sh -e instead of setting -e later, so that we don't miss anything before set -e is issued] Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* tests: Remove unused XVFB variableDavid Gibson2022-05-191-3/+0
| | | | | | | The XVFB variable is initialized at the beginning of test/run then never used again. I'm assuming it's a leftover from some ealier iteration. Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
* test, seccomp, Makefile: Switch to valgrind runs for passt functional testsStefano Brivio2022-03-291-0/+9
| | | | | | | | | | | | | | Pass to seccomp.sh a list of additional syscalls valgrind needs as EXTRA_SYSCALLS in a new 'valgrind' make target, and add corresponding support in seccomp.sh itself. In test setup functions, start passt with valgrind, but not for performance tests. Add tests checking that valgrind exits without errors after all the other tests in the group are done. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* demo, ci: Switch to asciinema(1) for terminal recordingsStefano Brivio2022-02-221-8/+8
| | | | | | | | | | For demos, cool-retro-term(1) looked fancier, but several threads of that and ffmpeg(1) are just messing up with performance testing. The CI videos started getting really big as well, and they were difficult to read. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* test: Add demo for Podman with pastaStefano Brivio2022-02-221-0/+8
| | | | | | | | ...showing setup steps, some peculiarities as --net option, and a general side-to-side comparison with slirp4netns(1), including "quick" TCP and UDP throughput and latency benchmarks. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* test: Add distribution tests for several architectures and kernel versionsStefano Brivio2022-01-281-0/+1
| | | | | | | | | | | | | | | The new tests check build and a simple case with pasta sending a short message in both directions (namespace to init, init to namespace). Tests cover a mix of Debian, Fedora, OpenSUSE and Ubuntu combinations on aarch64, i386, ppc64, ppc64le, s390x, x86_64. Builds tested starting from approximately glibc 2.19, gcc 4.7, and actual functionality approximately from 4.4 kernels, glibc 2.25, gcc 4.8, all the way up to current glibc/gcc/kernel versions. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* test: Record CI and demo videos in Xvfb by default, fix demo setup sequenceStefano Brivio2021-09-291-1/+6
| | | | Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
* test: Add CI/demo scriptsStefano Brivio2021-09-271-0/+154
Not really quick, definitely dirty. Signed-off-by: Stefano Brivio <sbrivio@redhat.com>