base: Honor .gitignore for rsync
#3037
Conversation
cgwalters
requested review from
happz,
janhavlin,
lukaszachy,
psss and
thrix
as code owners
|
I don't quite understand why we need even a logical copy of the context directory at all by default. I would expect tests to just not mutate it. But, if we wanted to be more sure, we could of course try to design a flow where we run tmt in a container, with the context mounted readonly. |
|
Big +1. In Cockpit we are actually in the same boat -- a lot of our developer trees have a ~ 1 GB node_modules/ (most of which isn't needed for running tests) and often also a multi-GB |
cgwalters
changed the title
.gitignore for rsync
There was a problem hiding this comment.
Lovely, thanks @cgwalters ! Neither my previous 👎 nor my 👍 here are authoritative of course, but this is really nice!
|
Love the idea. Could it check for the presence of |
|
This PR doesn't change anything if the directory isn't a git tree. The question is just if anyone should rely on built (i.e. not git-tracked) files to go into the tmt target. IMHO this is just confusing, inefficient, and unpredictable, so I'm all for making this not configurable. (But as always, https://m.xkcd.com/1172/ 😁 ) |
I'm not sure about that, I don't see relevant checks for if the worktree is a git near where |
|
Seems like a good idea, although my experience with various directories in tmt is still limited. I would just propose different $SUBJ, |
|
@happz ping - since you know the tmt code can I please ask you to drive this PR to completion? |
|
Hmm wait, are there two rsyncs that tmt does? Oh wow...there is...we do an rsync of the worktree once into ? Ouch...so we'd clearly need to factor out shared rsync exclusions for these two things. (This said again, the best fix would be to avoid full copies; for the local virt one we could use virtiofs, although that doesn't work for remote VMs) |
Yeah, it's still on the roadmap for 1.35, I just need to find some time to wrap it up & ask folks for review :/ |
|
My additions are in b7e8759. |
|
/packit build |
My main concern was if it picked up recursive $ tree -a ./
./
├── copy_this
├── .gitignore
├── other_value
├── some_folder
│ ├── copy_this_too
│ ├── .gitignore
│ └── ignore_me_too
└── some_valueA
2 directories, 7 files
$ tail .gitignore some_folder/.gitignore
==> .gitignore <==
some_value*
/other_value
==> some_folder/.gitignore <==
ignore_me_too
$ rsync -a --filter=':- .gitignore' ./ ../test_rsync_out/
$ tree -a ../test_rsync_out/
../test_rsync_out/
├── copy_this
├── .gitignore
└── some_folder
├── copy_this_too
└── .gitignore
2 directories, 4 files👍 for such an approach IMO |
I am guessing I may be one of the first people to try using tmt with a Rust project. The default for the `cargo` toolchain is to keep a *lot* of cached incremental data in `target/`. In my case with bootc, it's currently 20G. A plain rsync() of this is *incredibly* inefficient. rsync doesn't even use reflinks if available, though that's a distinct bug. Use `git ls-files` to honor `.gitignore`. Signed-off-by: Colin Walters <[email protected]>
|
/packit build |
I am guessing I may be one of the first people to try using tmt with a Rust project. The default for the `cargo` toolchain is to keep a *lot* of cached incremental data in `target/`. In my case with bootc, it's currently 20G. A plain rsync() of this is *incredibly* inefficient. rsync doesn't even use reflinks if available, though that's a distinct bug. Co-authored-by: Colin Walters <[email protected]> Co-authored-by: Miloš Prchlík <[email protected]>
|
Sorry to resurrect this but my test runs are still failing with this symptom. In current bootc I'm doing The more I think about all of this, the more strongly I feel that what I really want out of tmt is to support a workflow that defines basically:
The problem that we're getting bit by here is tmt is gathering 2. via scraping out of the current working directory. If instead tests were themselves defined to be wrapped in e.g. a container image that was a build artifact, that would help ensure that we weren't leaking unintended contents into the target environment. This would certainly have some overhead; anytime I went to change a test locally I'd have to do e.g. a container build. But we already have that requirement for changes to the code, so keeping thing symmetrical here would help isolate and add consistency. |
|
I feel like I must be the only project using Rust and tmt... |
Probably it is, but I am curious to see how you use it there :). But it being rust should not affect it, unless I am missing some deeper context of the tests. About the issue you've posted before, there is #4029 to move away from rsync for better or worse.
You could use |
Sooo... Like
|
Yes; I think an important aspect here is also that the tests can ergonomically be compiled code, rather than dynamically interpreted scripts. Anyways though...could I work around this by just moving all the tmt stuff into a subdirectory? Let me see... |
A key thing for me is that the `Justfile` should be a one-stop shop for development of the project. It can't have everything but it should answer the basic questions of "how do I build and test this project". This aligns the recently added tmt-on-GHA flow a *bit* more closely with some of that. Biggest is to use the `just build-integration-test-image` as the canonical way to build a container image with our testing stuff in it; which uses our main Dockerfile Other cleanups: - Move tmt/ into tests/tmt/ as a workaround for teemtee/tmt#3037 (comment) - Change the qemu logic to use SMBIOS credentials so we don't have to carry around both a disk image and a SSH key - Change qemu to use `-snapshot` so we can reuse disks - Change the scripts to accept data via argv[1] and not environment - Drop the hardcoded testing directory and use `target/` as a generic build artifact dir Signed-off-by: Colin Walters <[email protected]>
A key thing for me is that the `Justfile` should be a one-stop shop for development of the project. It can't have everything but it should answer the basic questions of "how do I build and test this project". This aligns the recently added tmt-on-GHA flow a *bit* more closely with some of that. Biggest is to use the `just build-integration-test-image` as the canonical way to build a container image with our testing stuff in it; which uses our main Dockerfile Other cleanups: - Move tmt/ into tests/tmt/ as a workaround for teemtee/tmt#3037 (comment) - Change the qemu logic to use SMBIOS credentials so we don't have to carry around both a disk image and a SSH key - Change qemu to use `-snapshot` so we can reuse disks - Change the scripts to accept data via argv[1] and not environment - Drop the hardcoded testing directory and use `target/` as a generic build artifact dir Signed-off-by: Colin Walters <[email protected]>
A key thing for me is that the `Justfile` should be a one-stop shop for development of the project. It can't have everything but it should answer the basic questions of "how do I build and test this project". This aligns the recently added tmt-on-GHA flow a *bit* more closely with some of that. Biggest is to use the `just build-integration-test-image` as the canonical way to build a container image with our testing stuff in it; which uses our main Dockerfile Other cleanups: - Move tmt/ into tests/tmt/ as a workaround for teemtee/tmt#3037 (comment) - Change the qemu logic to use SMBIOS credentials so we don't have to carry around both a disk image and a SSH key - Change qemu to use `-snapshot` so we can reuse disks - Change the scripts to accept data via argv[1] and not environment - Drop the hardcoded testing directory and use `target/` as a generic build artifact dir Signed-off-by: Colin Walters <[email protected]>
A key thing for me is that the `Justfile` should be a one-stop shop for development of the project. It can't have everything but it should answer the basic questions of "how do I build and test this project". This aligns the recently added tmt-on-GHA flow a *bit* more closely with some of that. Biggest is to use the `just build-integration-test-image` as the canonical way to build a container image with our testing stuff in it; which uses our main Dockerfile Other cleanups: - Move tmt/ into tests/tmt/ as a workaround for teemtee/tmt#3037 (comment) - Change the qemu logic to use SMBIOS credentials so we don't have to carry around both a disk image and a SSH key - Change qemu to use `-snapshot` so we can reuse disks - Change the scripts to accept data via argv[1] and not environment - Drop the hardcoded testing directory and use `target/` as a generic build artifact dir Signed-off-by: Colin Walters <[email protected]>
Answer: I can't do that in the obvious way ( Anyways so instead of a conversation on a closed PR I filed #4062 |
A key thing for me is that the `Justfile` should be a one-stop shop for development of the project. It can't have everything but it should answer the basic questions of "how do I build and test this project". This aligns the recently added tmt-on-GHA flow a *bit* more closely with some of that. Biggest is to use the `just build-integration-test-image` as the canonical way to build a container image with our testing stuff in it; which uses our main Dockerfile Other cleanups: - Change test script to move into tests/tmt/ as a workaround for teemtee/tmt#3037 (comment) - Change the qemu logic to use SMBIOS credentials so we don't have to carry around both a disk image and a SSH key - Change qemu to use `-snapshot` so we can reuse disks - Change the scripts to accept data via argv[1] and not environment - Drop the hardcoded testing directory and use `target/` as a generic build artifact dir Signed-off-by: Colin Walters <[email protected]>
A key thing for me is that the `Justfile` should be a one-stop shop for development of the project. It can't have everything but it should answer the basic questions of "how do I build and test this project". This aligns the recently added tmt-on-GHA flow a *bit* more closely with some of that. Biggest is to use the `just build-integration-test-image` as the canonical way to build a container image with our testing stuff in it; which uses our main Dockerfile Other cleanups: - Change test script to move into tests/tmt/ as a workaround for teemtee/tmt#3037 (comment) - Change the qemu logic to use SMBIOS credentials so we don't have to carry around both a disk image and a SSH key - Change qemu to use `-snapshot` so we can reuse disks - Change the scripts to accept data via argv[1] and not environment - Drop the hardcoded testing directory and use `target/` as a generic build artifact dir Signed-off-by: Colin Walters <[email protected]>
A key thing for me is that the `Justfile` should be a one-stop shop for development of the project. It can't have everything but it should answer the basic questions of "how do I build and test this project". This aligns the recently added tmt-on-GHA flow a *bit* more closely with some of that. Biggest is to use the `just build-integration-test-image` as the canonical way to build a container image with our testing stuff in it; which uses our main Dockerfile Other cleanups: - Change test script to move into tests/tmt/ as a workaround for teemtee/tmt#3037 (comment) - Change the qemu logic to use SMBIOS credentials so we don't have to carry around both a disk image and a SSH key - Change qemu to use `-snapshot` so we can reuse disks - Change the scripts to accept data via argv[1] and not environment - Drop the hardcoded testing directory and use `target/` as a generic build artifact dir Signed-off-by: Colin Walters <[email protected]>
A key thing for me is that the `Justfile` should be a one-stop shop for development of the project. It can't have everything but it should answer the basic questions of "how do I build and test this project". This aligns the recently added tmt-on-GHA flow a *bit* more closely with some of that. Biggest is to use the `just build-integration-test-image` as the canonical way to build a container image with our testing stuff in it; which uses our main Dockerfile Other cleanups: - Change test script to move into tests/tmt/ as a workaround for teemtee/tmt#3037 (comment) - Change the qemu logic to use SMBIOS credentials so we don't have to carry around both a disk image and a SSH key - Change qemu to use `-snapshot` so we can reuse disks - Change the scripts to accept data via argv[1] and not environment - Drop the hardcoded testing directory and use `target/` as a generic build artifact dir Signed-off-by: Colin Walters <[email protected]>
A key thing for me is that the `Justfile` should be a one-stop shop for development of the project. It can't have everything but it should answer the basic questions of "how do I build and test this project". This aligns the recently added tmt-on-GHA flow a *bit* more closely with some of that. Biggest is to use the `just build-integration-test-image` as the canonical way to build a container image with our testing stuff in it; which uses our main Dockerfile Other cleanups: - Change test script to move into tests/tmt/ as a workaround for teemtee/tmt#3037 (comment) - Change the qemu logic to use SMBIOS credentials so we don't have to carry around both a disk image and a SSH key - Change qemu to use `-snapshot` so we can reuse disks - Change the scripts to accept data via argv[1] and not environment - Drop the hardcoded testing directory and use `target/` as a generic build artifact dir Signed-off-by: Colin Walters <[email protected]>
A key thing for me is that the `Justfile` should be a one-stop shop for development of the project. It can't have everything but it should answer the basic questions of "how do I build and test this project". This aligns the recently added tmt-on-GHA flow a *bit* more closely with some of that. Biggest is to use the `just build-integration-test-image` as the canonical way to build a container image with our testing stuff in it; which uses our main Dockerfile Other cleanups: - Change test script to move into tests/tmt/ as a workaround for teemtee/tmt#3037 (comment) - Change the qemu logic to use SMBIOS credentials so we don't have to carry around both a disk image and a SSH key - Change qemu to use `-snapshot` so we can reuse disks - Change the scripts to accept data via argv[1] and not environment - Drop the hardcoded testing directory and use `target/` as a generic build artifact dir Signed-off-by: Colin Walters <[email protected]>
A key thing for me is that the `Justfile` should be a one-stop shop for development of the project. It can't have everything but it should answer the basic questions of "how do I build and test this project". This aligns the recently added tmt-on-GHA flow a *bit* more closely with some of that. Biggest is to use the `just build-integration-test-image` as the canonical way to build a container image with our testing stuff in it; which uses our main Dockerfile Other cleanups: - Change test script to move into tests/tmt/ as a workaround for teemtee/tmt#3037 (comment) - Change the qemu logic to use SMBIOS credentials so we don't have to carry around both a disk image and a SSH key - Change qemu to use `-snapshot` so we can reuse disks - Change the scripts to accept data via argv[1] and not environment - Drop the hardcoded testing directory and use `target/` as a generic build artifact dir Signed-off-by: Colin Walters <[email protected]>
A key thing for me is that the `Justfile` should be a one-stop shop for development of the project. It can't have everything but it should answer the basic questions of "how do I build and test this project". This aligns the recently added tmt-on-GHA flow a *bit* more closely with some of that. Biggest is to use the `just build-integration-test-image` as the canonical way to build a container image with our testing stuff in it; which uses our main Dockerfile Other cleanups: - Change test script to move into tests/tmt/ as a workaround for teemtee/tmt#3037 (comment) - Change the qemu logic to use SMBIOS credentials so we don't have to carry around both a disk image and a SSH key - Change qemu to use `-snapshot` so we can reuse disks - Change the scripts to accept data via argv[1] and not environment - Drop the hardcoded testing directory and use `target/` as a generic build artifact dir Signed-off-by: Colin Walters <[email protected]>
A key thing for me is that the `Justfile` should be a one-stop shop for development of the project. It can't have everything but it should answer the basic questions of "how do I build and test this project". This aligns the recently added tmt-on-GHA flow a *bit* more closely with some of that. Biggest is to use the `just build-integration-test-image` as the canonical way to build a container image with our testing stuff in it; which uses our main Dockerfile Other cleanups: - Change test script to move into tests/tmt/ as a workaround for teemtee/tmt#3037 (comment) - Change the qemu logic to use SMBIOS credentials so we don't have to carry around both a disk image and a SSH key - Change qemu to use `-snapshot` so we can reuse disks - Change the scripts to accept data via argv[1] and not environment - Drop the hardcoded testing directory and use `target/` as a generic build artifact dir Signed-off-by: Colin Walters <[email protected]>
A key thing for me is that the `Justfile` should be a one-stop shop for development of the project. It can't have everything but it should answer the basic questions of "how do I build and test this project". This aligns the recently added tmt-on-GHA flow a *bit* more closely with some of that. Biggest is to use the `just build-integration-test-image` as the canonical way to build a container image with our testing stuff in it; which uses our main Dockerfile Other cleanups: - Change test script to move into tests/tmt/ as a workaround for teemtee/tmt#3037 (comment) - Change the qemu logic to use SMBIOS credentials so we don't have to carry around both a disk image and a SSH key - Change qemu to use `-snapshot` so we can reuse disks - Change the scripts to accept data via argv[1] and not environment - Drop the hardcoded testing directory and use `target/` as a generic build artifact dir Signed-off-by: Colin Walters <[email protected]>
A key thing for me is that the `Justfile` should be a one-stop shop for development of the project. It can't have everything but it should answer the basic questions of "how do I build and test this project". This aligns the recently added tmt-on-GHA flow a *bit* more closely with some of that. Biggest is to use the `just build-integration-test-image` as the canonical way to build a container image with our testing stuff in it; which uses our main Dockerfile Other cleanups: - Change test script to move into tests/tmt/ as a workaround for teemtee/tmt#3037 (comment) - Change the qemu logic to use SMBIOS credentials so we don't have to carry around both a disk image and a SSH key - Change qemu to use `-snapshot` so we can reuse disks - Change the scripts to accept data via argv[1] and not environment - Drop the hardcoded testing directory and use `target/` as a generic build artifact dir Signed-off-by: Colin Walters <[email protected]>
A key thing for me is that the `Justfile` should be a one-stop shop for development of the project. It can't have everything but it should answer the basic questions of "how do I build and test this project". This aligns the recently added tmt-on-GHA flow a *bit* more closely with some of that. Biggest is to use the `just build-integration-test-image` as the canonical way to build a container image with our testing stuff in it; which uses our main Dockerfile Other cleanups: - Change test script to move into tests/tmt/ as a workaround for teemtee/tmt#3037 (comment) - Change the qemu logic to use SMBIOS credentials so we don't have to carry around both a disk image and a SSH key - Change qemu to use `-snapshot` so we can reuse disks - Change the scripts to accept data via argv[1] and not environment - Drop the hardcoded testing directory and use `target/` as a generic build artifact dir Signed-off-by: Colin Walters <[email protected]>
6 participants
I am guessing I may be one of the first people to try using tmt
with a Rust project. The default for the
cargotoolchainis to keep a lot of cached incremental data in
target/.In my case with bootc, it's currently 20G.
A plain rsync() of this is incredibly inefficient. rsync doesn't
even use reflinks if available, though that's a distinct bug.
Use
git ls-filesto honor.gitignore.Signed-off-by: Colin Walters [email protected]