Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Another way to test packages in dev stage #1263

Open
frankchen211 opened this issue Mar 28, 2022 · 24 comments · May be fixed by #1305
Open

Another way to test packages in dev stage #1263

frankchen211 opened this issue Mar 28, 2022 · 24 comments · May be fixed by #1305
Labels
enhancement rez-test

Comments

@frankchen211
Copy link

frankchen211 commented Mar 28, 2022 •
edited
Loading

Allow resolving a package from a random specified place instead of paths from rezconfig packages_path for testing purpose before release to production repo.

Let's say we have a package helloworld in d:\dev\helloworld (no version specified at this time) want to test, hope to init env like this:
rez-env helloworld pkg1 pkg2 ... -d d:\dev
helloworld will be resolved to D:\dev\helloworld, pkg1 and pkg2 resolved to a repository path in rez config. or if they are also in D:\dev folder.

helloworld                d:\dev\helloworld
pkg1-3.3                   c:\users\user\packages\pkg1\3.3\                                           (local)
pkg2-1.0.0                c:\users\user\packages\pkg2\1.0.0                                          (local)

or just directly specify the package path:
rez-env helloworld@d:\dev\helloworld pkg1 pkg2 ...

Motivation
Current workflow for testing a package generally is rez-build -i, then rez-test if we have unit test code or open dcc or rez-env xxx -- python to test.
This means you have to copy a package to the local package path , this path also in packages_path in rezconfig.py to search pacakges by default.

Reason why I don't test a packge in this way:
In my studio, all our workstations will be used to render jobs at night, which means if I have some packages are in test stage in local, any production render jobs that using my workstation will use these code, this is not what we are expected. (Some packages we do mirror to local to speed up the load time in production, local path is not always for testing purpose.)

If I want to invite other developers/artists to test my packages(not a formal test in our pipeline), I have to copy to a central repository and give a testing version carefully to avoid affecting production, since these packages are in develop stage, I really don't want to release it to production repository, instead just putting it to a sandbox and tell rez-env to resolve these packages from given paths, required pacakges still can be resolved from formal production repos.
@ColinKennedy
Copy link

ColinKennedy commented Mar 28, 2022 •
edited
Loading

Without getting too deep into pipeline specifics, because testing / deployment is often less a question for how Rez works and more how pipelines choose to adopt Rez (e.g. do artists press a button in a GUI to load Rez, do they write rez-env directly in the command-line, etc). Let's look at how to handle this with Rez, assuming no pipeline.

Assuming you want to develop / test a package called "package_foo"

  • git clone the repository of package_foo
  • cd {repository_root}/path/to/package_foo
  • Bump the major / minor / patch version of package_foo's package.py to a new version
    • Notice: I do not use a testing version. I just bump one of the {major}.{minor}.{patch} tokens. The reason why is because of the next couple steps, below.
  • build to a network location, visible to users and the farm (e.g. /path/to/network)
    • e.g. rez-build --clean --install --prefix=/path/to/network
  • Then I'd create a context which represents "my patched package_foo + all of the other released packages"
REZ_PACKAGES_PATH=/path/to/network:`rez-config release_packages_path` \
    rez-env package_foo==the.version.here some_pakages-1 that_users-2 would_want-1 \
    --output /path/to/network_contexts/context.rxt

This creates a context which is just my package_foo's isolated changes but also includes all of the other released packages which I have not changed.

And then I'd either...

  • Ask users to test out my changes by loading the context, via rez-env --input /path/to/network_contexts/context.rxt
  • Use rez-bundle to produce a bundle for that context.rxt, and then ask users to run tools through that

Although in reality, I actually wouldn't use either option because when I worked Rez at Animal Logic, they had a GUI that in many ways abstracted this workflow. But this is how I would do it in a vanilla Rez environment, assuming no pipeline. Is there something in that suggested workflow I just described that you think needs improvement?

Side note: A key point of this workflow is that "/path/to/network" is a unique path name. Usually a real path would include a ticket number, like /path/to/network/PIPE-1234-my_testing_changes). Because the paths are unique, there's no chance of accidentally mixing your changes with other's work or with your past work. I made a video tutorial related to this topic. It may be good as reference: https://www.youtube.com/watch?v=rPe6JpzE_Nc. It's about automating developer environments and ties into this topic pretty well.

@nerdvegas
Copy link
Contributor

nerdvegas commented Mar 28, 2022 via email

@frankchen211
Copy link
Author

frankchen211 commented Mar 29, 2022 •
edited
Loading

build to a network location, visible to users and the farm (e.g. /path/to/network)

  • e.g. rez-build --clean --install --prefix=/path/to/network
  • Then I'd create a context which represents "my patched package_foo + all of the other released packages"
REZ_PACKAGES_PATH=/path/to/network:`rez-config release_packages_path` \
    rez-env package_foo==the.version.here some_pakages-1 that_users-2 would_want-1 \
    --output /path/to/network_contexts/context.rxt

Your solution here seems add the random new package path /path/to/network to search path and output a context file, which is similar to my proposal - make rez-env directly support this random path:
rez-env package_foo some_pakages-1 that_users-2 would_want-1 -d /path/to/network

Or directly tell rez where to load ackage_foo:
rez-env package_foo@/path/to/network some_pakages-1 that_users-2 would_want-1
In this case, we don't need extra work any more:

  • run rez-build, we don't use this cmd too much since lots of pipeline tools/dev written in python don't need to be build, or plugins need to be built we will use other method to do that, this depends on developers.
  • a script to generate or modify sth like the example here to create a context file

Although in reality, I actually wouldn't use either option because when I worked Rez at Animal Logic, they had a GUI that in many ways abstracted this workflow. But this is how I would do it in a vanilla Rez environment, assuming no pipeline. Is there something in that suggested workflow I just described that you think needs improvement?
Yes we provide GUI to artist to use, but for random tests(before available to production), cli makes it easier and flexible.

Side note: A key point of this workflow is that "/path/to/network" is a unique path name. Usually a real path would include a ticket number, like /path/to/network/PIPE-1234-my_testing_changes). Because the paths are unique, there's no chance of accidentally mixing your changes with other's work or with your past work. I made a video tutorial related to this topic. It may be good as reference: https://www.youtube.com/watch?v=rPe6JpzE_Nc. It's about automating developer environments and ties into this topic pretty well.

Yes, we do use ticket system and branches(ticket nubmer) in our workflow, that's why we dont' use versions until all tests done and merge to master/main branch to trigger deploy process to push to production env, during deploy we give a version.

@frankchen211
Copy link
Author

frankchen211 commented Mar 29, 2022 •
edited
Loading

There's lots of scenarios that we want to test packages, this would make rezconfig complicated and need some extra settings or wrappers in pipeline to differentiate different use cases.

Let's say I have a package that is used in render jobs also, when submit from a test env, a render node should pick that dev package also.

To do so, there's no need more extra work and simple.
rez-env foo bar-1 -d /path/to/sandbox/issue-xxx
Or
rez-env foo@/path/to/sandbox/issue-xxx bar-1

rez-env foo bar-1 -d /path/to/sandbox/issue-xxx will be resovled to foo@/path/to/sandbox/issue-xxx bar-1

then use REZ_USED_REQUEST to init the render job.

@nerdvegas
Copy link
Contributor

nerdvegas commented Mar 29, 2022 via email

@ColinKennedy
Copy link

ColinKennedy commented Mar 29, 2022 •
edited
Loading

@nerdvegas I thought about @frankchen211 situation a bit more and see where he's coming from. I experienced his problem at AL several times. I'll try to explain the problem and how I "solved" it, using the vanilla Rez stuff I'd mentioned earlier.

  • package_foo has a bug
  • I go through the process of bug-fixing, building to /path/to/network etc as mentioned in my previous reply
    • To do this fix, I bumped the patch version of package_foo from 1.2.3 to 1.2.4
  • Another developer comes and releases 1.3.0 to the same release path I was using as a fallback path for non-built packages.

Now I get a chat message from a tester "Colin, your bug fix is gone / doesn't work". It could be that the bug fix actually didn't work. But it's also possible that someone released over top of my change (as in this case)

Since I don't know which situation it is, every time I get a message like that, I have to check if that the user was getting my network-built path first. It's a bit tedious.

There's 2 ways Rez can "solve" this problem

  • Replace the user's rez-env request
  • Upversion your package to something very high

Replace the user's rez-env request

If the deployed / production rez-env request is rez-env package_foo-1.1+<2 another_package-2+ etc-3, I could give the user rez-env package_foo==1.2.4 another_package-2+ etc-3. That way, even if package_foo 1.3.0, 1.4.0, etc is released, there's no chance that Rez will pick those versions. My patch is always chosen.

However, this assumes that you even can modify the request, which may not be feasible. e.g. if your pipeline doesn't have a way of sparsely changing the version of a single package, you're out of luck. Which leads to the other solution...

Upversion your package to something very high

If I change package_foo's version to 1.999.0, it doesn't matter if 1.3.0, 1.4.0, etc come around because my version, which is actually just a bug fix + patch, will always be chosen. This is neither semantic nor reliable, because the request might actually be package_foo-1.1+<1.100.0 (just as an example).

If we had a means of telling Rez "only consider the packages built to this directory but fall back to the other paths for the Rez package families which aren't built", that might be a nice idiom to have.

@nerdvegas
Copy link
Contributor

nerdvegas commented Mar 29, 2022 via email

@ColinKennedy
Copy link

ColinKennedy commented Mar 29, 2022 •
edited
Loading

if you're changing the request, you may as well just specify your testing version explicitly..?

That's assuming that you can apply your change sparsely. If you're trying to make a test environment that's "production + your one change", depending on how your pipeline is set up, the raw Rez request could look like this under the hood.

rez-env package_foo-1.1+<2 another_package-2+ etc-3

My one change, in this case, is "I want to change the package request to a locked patch". But your pipeline doesn't allow you to do that sparsely. So what choice do you have but to copy the whole request and edit the one package.

package_foo-1.1+<2 another_package-2+ etc-3 -> **package_foo==1.2.4** another_package-2+ etc-3
# alternatively you could just append your version to the end of the request
package_foo-1.1+<2 another_package-2+ etc-3 -> package_foo-1.1+<2 another_package-2+ etc-3 **package_foo==1.2.4**

It's the same effect either way - you're either replacing package_foo or adding an extra package_foo with the explicit version. But the problem I'll describe next applies in either approach.

But What If The Original Request Changes

Now another developer comes along and, in the production tool / bundle / w.e introduces a new change to the request. Maybe they add / remove a package or change a version range.

package_foo-1.1+<2 another_package-2+ etc-3 **brand_new_package-1+**

Now my test request, which was **package_foo==1.2.4** another_package-2+ etc-3, doesn't include brand_new_package-1+. My test environment is no longer "production + your one change". It's "out of date production + your one change".

My bit about "sparsely" is that if you don't have a mechanism, whether by GUI, CLI, w/e - to change the one package_foo but inherit the request of the package requests, you end up hard-coding the whole request when setting up a test environment, which isn't great because things can go out of date.

but fall back to the other paths for the Rez package families which aren't built

Sorry, I'll clarify - I never meant to insinuate source / developer packages. I'm always talking about built / installed Rez packages.

"If we had a means of telling Rez "For all built packages in this directory, only consider those versions / family names. For all other paths which may contain built packages (e.g. release_packages_path), resolve those as normal.

The end behavior / goal is - the Rez packages that I built are always used even if there's a later version of a package. And I didn't have to change the request in order to get that behavior (because as mentioned in the paragraph above, modifying the request, as a workflow, has some rough corners)

@ColinKennedy
Copy link

Btw for context, this problem isn't that much of an issue in practice. Just re-releasing a greater version in response to changes is good enough in most cases. I'm mostly just curious if this chat could spawn a better workflow. But if not, that's alright, the current workflow works well enough in most cases.

@frankchen211
Copy link
Author

frankchen211 commented Mar 31, 2022 •
edited
Loading

Currently, syntax used in rez-env is exactly the same as package requirements in a package.py, and I think that's a good thing. By supporting explicit paths like this (ie your 'foo@/path` eg), you either break that correlation, or you allow packages to do this too. If packages can do this, you bet quick fixes referring to temp repo paths will end up in production, and I think that would cause more problems in the long run.

Yes, in my changs, it allows that, but this is only used for quick test from production side, in most cases, production use released versioned packages. Explicit paths may esay to reproduce and debug things?

If your example of -d here means to say, "add this path to REZ_PACKAGES_PATH instead of me having to change my rezconfig", then that's already very similar to the existing --paths option. I'll grant you it is a bit more user friendly though (since you don't have to repeat the rest of the packages path), so if that is what you mean, and if you named it something like --add-pre-paths/--add-post-paths instead, then I wouldn't be opposed to adding that. Thx A

I found env REZ_USED_PACKAGES_PATH will try this if this can ensure env from local to farm is constant.

Thx.

@fnaum
Copy link

fnaum commented Apr 1, 2022 via email

@nerdvegas
Copy link
Contributor

nerdvegas commented Apr 1, 2022 via email

@frankchen211 frankchen211 mentioned this issue Apr 3, 2022
Open
@fnaum
Copy link

fnaum commented Apr 7, 2022 via email

@ColinKennedy
Copy link

@fnaum have you had a chance to check this? Just curious if the code exists and you / AL would be willing to PR it back into Rez or if we'll want to target this as a separate effort.

@fnaum
Copy link

fnaum commented Apr 15, 2022 via email

@hughetop
Copy link

This sounds similar to what we do, but perhaps more complex. I already saw someone suggest saving the resolve to a .rxt file and then loading that to get exactly the same packages and versions.

"If your example of -d here means to say, "add this path to
REZ_PACKAGES_PATH instead of me having to change my rezconfig", then that's
already very similar to the existing --paths option. I'll grant you it is
a bit more user friendly though (since you don't have to repeat the rest of
the packages path), so if that is what you mean, and if you named it
something like --add-pre-paths/--add-post-paths instead, then I
wouldn't be opposed to adding that."

A trick we do is inside a global .cshrc that everyone uses, we define
export REZ_PACKAGES_PATH=~/packages:/some/network/path/int:/some/network/path/ext
which is exactly the same as the packages listed in rezconfig.py, but the benefit is we have that available at runtime. So we can do
rez-env --paths=/some/network/test/area:$REZ_PACKAGES_PATH helloworld-1.2.3
in order to test packages outside of the normal paths defined in rezconfig. You want to specify the package version so it doesn't accidentally pick up newer builds. This avoids the homedir, allows other users (or farm jobs) to test your changes, but doesn't affect any official released packages or the rezzconfig.

@ColinKennedy
Copy link

ColinKennedy commented Apr 29, 2022 •
edited
Loading

That's pretty much what we do as well, it just isn't ideal. As mentioned in previous posts, a package orderer would solve the problem of "what if helloworld 1.2.3.1 gets released outside of /some/network/test/area". You could modify the request to helloworld==1.2.3 to force it to use yours, but then that requires the request to change.

Suggested interface:

rez-env --favor-paths
If specified but no paths provided, use [config.local_packages_path]

rez-env --favor-paths /foo:/bar:/thing
If specified with paths, all Rez package families / versions found in those parts will be preferred over anything else found in config.packages_path.

@ColinKennedy ColinKennedy mentioned this issue May 4, 2022
Open
@ColinKennedy
Copy link

ColinKennedy commented May 5, 2022 •
edited
Loading

@nerdvegas I gave this a shot and got it mostly working. There's a WIP located over at

https://github.com/nerdvegas/rez/compare/master...ColinKennedy:issues/1263-add_favor_paths_package_orderer?expand=1

If you have two paths

/foo
    package_a/
        1.0.0/
/bar
    package_a/
        2.0.0/

REZ_PACKAGES_PATH=/bar:/foo rez-env package_a --favor-paths /foo resolves to 1.0.0 from /foo as expected. However this doesn't 100% solve the problems I'd mentioned in previous posts in this thread.

/foo
    package_a/
        1.0.0/
/bar
    package_a/
        1.0.0/

REZ_PACKAGES_PATH=/bar:/foo rez-env package_a --favor-paths /foo resolves to 1.0.0 from /bar instead of /foo. It seems the reason is related to if key in seen:, from iter_packages.

Since /foo and /bar both have (package_a.name, package_a.version), the iterable which gets passed to FavorPathsOrder.reorder completely omits the package_a 1.0.0 from /foo. So in short, the sort logic works but we cannot sort if the packages aren't ever passed to FavorPathsOrder.reorder. An exact-match package versions still have the potential to resolve from /bar instead of the favored /foo.

The test_conflicting_versions unittest demonstrates the issue. The test should pass, but fails instead, pointing to the "favor_paths_first" folder.

I'm happy to PR this edit in since it is an improvement over what we currently have. However for the exact-match case, the only solution I can think of is to make a new, unique version for package_a in /foo that I know cannot exist in /bar. Like just rez-build a version called 1.0.0-asdf or anything. It's not a great solution. Is there something cleaner that we can do? @fnaum tagging you as well in case you know of a solution.

@halil3d
Copy link

halil3d commented May 6, 2022

@ColinKennedy I'm curious if /foo happened to also be the local_packages_path, would this then work as expected? While this FavorPathsOrder functionality would be nice to allow any path to be favoured, I think if only the local path was favoured for older package versions, this would solve quite a few use cases that have been discussed. I'm not familiar with the specific Animal Logic implementation, but I believe it is specific to the local_packages_path?

@ColinKennedy
Copy link

ColinKennedy commented May 6, 2022 •
edited
Loading

I'd have to double check when I'm on a machine with Rez but I think I tried matching REZ_LOCAL_PACKAGES_PATH and --favor-paths to the same path and it was the same (unwanted) result. Because of the iter_packages behavior as mentioned.

While this FavorPathsOrder functionality would be nice to allow any path to be favoured, I think if only the local path was favoured for older package versions, this would solve quite a few use cases that have been discussed.

Sorry, could you possibly rephrase? I'm not sure I follow. The behavior currently is

  1. rez-env --favor-paths <-- favors local_packages_path
  2. rez-env --favor-paths /blah <-- favors /blah

So you could get the "only favor local_packages_path" behavior by simply not providing a path after --favor-paths, already.

Again, I might've misunderstood. But from your reply, it sounded like the behavior you're looking for is already working. It's been a while since I worked at Animal Logic but I believe theirs only provides functionality (1), where we also can do functionality (2).

@ColinKennedy ColinKennedy linked a pull request May 7, 2022 that will close this issue
Open
@ColinKennedy
Copy link

I made a PR with the suggested edits. Btw to any interested parties - I was thinking we should probably also favor paths as a configuration option (not just as a CLI flag). Would that be alright to add? In this PR or a second one).

The logic would follow as - if provided as CLI flag, always use that. Otherwise use the rez.config.config option, assuming that it's set.

@halil3d
Copy link

halil3d commented May 9, 2022 •
edited
Loading

So you could get the "only favor local_packages_path" behavior by simply not providing a path after --favor-paths, already.

Again, I might've misunderstood. But from your reply, it sounded like the behavior you're looking for is already working. It's been a while since I worked at Animal Logic but I believe theirs only provides functionality (1), where we also can do functionality (2).

Ah perfect, I didn't catch that the default behaviour provided that outcome if no args were passed (1). That is exactly what I was looking to confirm was possible. Hopefully there's a way around the caveat you mention for (2). Thanks for putting up the PR.

@fnaum
Copy link

fnaum commented Jun 2, 2022 via email

@ColinKennedy
Copy link

@fnaum the PR should over at #1305 should have everything. Not sure how to get the PR merged but it's fully tested and ready for whenever anyone wants to review it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement rez-test
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add --favor-paths rez-env flag ColinKennedy/rez
7 participants
@nerdvegas @frankchen211 @fnaum @ColinKennedy @halil3d @JeanChristopheMorinPerso @hughetop