Too many failed reported in sudo-compliance-tests

[rtczza]

in sudo-rs/test-framework/sudo-compliance-tests run SUDO_UNDER_TEST=theirs cargo test :

test helpers::rsyslogd_works has been running for over 60 seconds
test su::cli::arguments_are_passed_to_shell has been running for over 60 seconds
test su::cli::dash_user_shell_arguments has been running for over 60 seconds
test su::cli::flag_after_positional_argument has been running for over 60 seconds
test su::default_target_is_root has been running for over 60 seconds
test su::does_not_panic_on_io_errors has been running for over 60 seconds
test su::env::vars_in_invoking_users_env_are_preserved has been running for over 60 seconds
test su::env::vars_set_by_su_override_existing_ones has been running for over 60 seconds
test su::env::vars_set_by_su_when_target_is_not_root has been running for over 60 seconds
test su::env::vars_set_by_su_when_target_is_root has been running for over 60 seconds
test su::explicit_target_user has been running for over 60 seconds
test su::flag_command::it_works has been running for over 60 seconds
test su::flag_command::pass_to_shell_via_c_flag has been running for over 60 seconds
test su::flag_command::positional_arguments_are_not_passed_to_command has been running for over 60 seconds
test su::flag_command::when_specified_more_than_once_only_last_value_is_used has been running for over 60 seconds
test su::flag_group::invoking_user_must_be_root has been running for over 60 seconds
test helpers::rsyslogd_works ... FAILED
test su::cli::dash_user_shell_arguments ... FAILED
test su::env::vars_in_invoking_users_env_are_preserved ... FAILED
test su::flag_command::pass_to_shell_via_c_flag ... FAILED
test su::env::vars_set_by_su_override_existing_ones ... FAILED
test su::default_target_is_root ... FAILED
test su::flag_group::invoking_user_must_be_root ... FAILED
test su::flag_command::it_works ... FAILED
test su::flag_command::when_specified_more_than_once_only_last_value_is_used ... FAILED
test su::env::vars_set_by_su_when_target_is_not_root ... FAILED
test su::cli::arguments_are_passed_to_shell ... FAILED
test su::does_not_panic_on_io_errors ... FAILED
...

final results:

test result: FAILED. 3 passed; 645 failed; 60 ignored; 0 measured; 0 filtered out; finished in 164.37s

error: test failed, to rerun pass `--lib`

Is it the wrong way to test? Or missed some steps?

rust version:

rustc 1.72.1 (d5c2e9c34 2023-09-13)

[pvdrz]

I ran the same command on the same folder and got this:

test result: ok. 648 passed; 0 failed; 60 ignored; 0 measured; 0 filtered out; finished in 164.67s

I have rust 1.75 but I doubt that's the issue. It seems there's something wrong with the containers as almost every single test is failing.

[rtczza]

have run the following two commands, but still get the above error.

docker run --rm -it sudo-test-og

docker exec -u 1000:100 -it 70dc3ea36f56  bash

I have always had some doubts about the execution of docker.

Can you provide the commands for the entire testing process? Including how to run the docker you need to use

[japaric]

most likely the shared step of building the docker image is failing and that's causing all tests that rely on the docker image to fail. you can set the SUDO_TEST_VERBOSE_DOCKER_BUILD env var to see the build process of the docker image and figure out what's going wrong there. I suggest you also pass a test filter to only run one test, or as few tests as possible, to reduce the amount of unhelpful text in the output.

$ cd test-framework
$ SUDO_TEST_VERBOSE_DOCKER_BUILD=1 SUDO_UNDER_TEST=theirs cargo t -p sudo-compliance-tests -- insufficient_permissions_to_execute_shell
[+] Building 25.9s (9/9) FINISHED                                                                                   docker-container:mybuilder
 => [internal] booting buildkit                                                                                                           2.4s
 => => starting container buildx_buildkit_mybuilder0                                                                                      2.4s
 => [internal] load build definition from Dockerfile                                                                                      0.1s
 => => transferring dockerfile: 236B                                                                                                      0.0s
 => [internal] load metadata for docker.io/library/debian:bookworm-slim                                                                   2.2s
 => [internal] load .dockerignore                                                                                                         0.1s
 => => transferring context: 2B                                                                                                           0.0s
 => [1/3] FROM docker.io/library/debian:bookworm-slim@sha256:f4a83aa865a2b4a064ff142aa91c713180df9fcb86ce676b5de2981029379c37             0.1s
 => => resolve docker.io/library/debian:bookworm-slim@sha256:f4a83aa865a2b4a064ff142aa91c713180df9fcb86ce676b5de2981029379c37             0.0s
 => [2/3] RUN apt-get update &&     apt-get install -y --no-install-recommends sudo procps sshpass rsyslog &&     rm /etc/sudoers        13.4s
 => [3/3] WORKDIR /tmp                                                                                                                    0.2s
 => exporting to docker image format                                                                                                      6.9s
 => => exporting layers                                                                                                                   3.5s
 => => exporting manifest sha256:79c8abfb0d71d4b95dcf44d5edd32d71f340f8d5f0a12d1af1cda32aeddf8533                                         0.0s
 => => exporting config sha256:d46dfe4cae602bd5d7ee9cbc42ad71f71b83e266cea62b7885d858da19438350                                           0.0s
 => => sending tarball                                                                                                                    3.3s
 => importing to docker                                                                                                                   1.8s
 => => loading layer 19602c60b3cf 196.61kB / 17.25MB                                                                                      1.8s
 => => loading layer 5f70bf18a086 32B / 32B                                                                                               0.3s
test sudo::flag_login::insufficient_permissions_to_execute_shell ... ok

(SUDO_TEST_VERBOSE_DOCKER_BUILD works with both values of SUDO_UNDER_TEST)