Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cargo publish multiple packages at once #1169

Open
alexcrichton opened this issue Jan 14, 2015 · 38 comments
Open

cargo publish multiple packages at once #1169

alexcrichton opened this issue Jan 14, 2015 · 38 comments
Labels
A-workspaces Area: workspaces call-for-testing Marks issues that require broader testing from the community, e.g. before stabilization. Command-publish E-medium Experience: Medium S-accepted Status: Issue or feature is accepted, and has a team member available to help mentor or review Z-package-workspace Nightly: package-workspace

Comments

@alexcrichton
Copy link
Member

alexcrichton commented Jan 14, 2015

It would be nice to have a flag to cargo publish which publishes all local packages in a DAG fashion.

Non-atomic publish was added in #14433, stabilization is being tracked in #10948

Testing instructions: #1169 (comment)

Notes (edit ehuss):

@wycats
Copy link
Contributor

wycats commented Jan 14, 2015

This would be awesome ;)

@huonw
Copy link
Member

huonw commented Apr 30, 2015

I suspect we may want to have a story with #883 if/when we implement this: I could easily imagine unintentionally publishing crates by not realising that I'm depending on them.

@botev
Copy link

botev commented Jul 30, 2015

Is there any progress on this? I think this would a very good thing to have. One of the nice things about crates is that they are modular, thus reduce the compile time of larger projects. However, if you are developing a large library compile times can get pretty slow (when it hits >2min I start to compile after 50% of the fixing, so it can compile while I'm fixing the other 50% of the issues). In this case you can split the library into sub-crates, where each sub-crate is in some sense standalone, or a build-up over "core" structure. However, each one of them would not make any sense on its own. Publishing each individually does not make any sense also and not packaging it together. If we can have this kind of publish --all this would give as an option to reduce compile times without any drawback from the user and publishing side of the library.

@matthiasbeyer
Copy link
Contributor

Any updates whether this will be implemented and when?

@wycats
Copy link
Contributor

wycats commented Feb 5, 2018

@matthiasbeyer I think this fell through the cracks when we did workspaces. This issue was supposed to track wrapping --all up for publish but I guess it got lost since it was originally opened earlier?

@alexcrichton what's the status?

@alexcrichton
Copy link
Member Author

@wycats AFAIK this has always been in the 'nice to have' category and hasn't progressed to the 'someone has put time into designing this' category.

@matthiasbeyer
Copy link
Contributor

One thing I just thought about: When doing a workspace release, cargo should build everything and after everything is fine, publish all crates at once... if that is even possible. Not like "build & publish the first crate and continue for each crate" but rather "build everything, then publish everything".

@kpcyrd
Copy link

kpcyrd commented Mar 5, 2018

It seems that in a foo-rs, foo-rs/foo-sys project layout, you can't cargo package in foo-rs before foo-rs/foo-sys has been published.

@ufoscout
Copy link

ufoscout commented Jul 31, 2018

I created a simple PR to deal with this annoying issue; it permits to execute cargo publish from a workspace.

One thing I just thought about: When doing a workspace release, cargo should build everything and after everything is fine, publish all crates at once... if that is even possible. Not like "build & publish the first crate and continue for each crate" but rather "build everything, then publish everything".

Unluckily I did not find a way to implement this logic because cargo requires that all the dependencies of a package are in the repository or the package phase (when the tarball is created) fails; consequently, before publishing a package with a "path" dependency, that dependency must be in the repository.

@Nemo157
Copy link
Member

Nemo157 commented Jul 31, 2018

Something else I would want that I don’t think that can handle is only publishing updated packages, if one of the packages already exists at its current version number it should be downloaded and verified that the new package is identical.

@softprops
Copy link

softprops commented Oct 9, 2018

being new to workspaces but not to cargo this feels very much like a paper cut. My finger just got nipped when I tried to publish a new workspace project by following the docs

screen shot 2018-10-09 at 4 45 01 pm

...then realized that order matters. The validation of packages will fail of one of the workspace packages depends on another in the same release but which may not have been published first. In my case it's a very simple ordering but for those new to cargo, something like cargo publish --all could get the ordering right without putting a burden on me at all. Something like that would be a much nicer new user experience, also a much nicer convenience to those that have learned the ordering semantics for the more more manual publishing approach.

@matthiasbeyer
Copy link
Contributor

It would definitively help me with imag where I publish over 50 crates in one release!

@softprops
Copy link

@torkleyy
Copy link
Contributor

This sounds useful, but I'm wondering what the exact behaviour should be. What if you have a path dependency and there's no version specified, should Cargo modify the Cargo.toml for that?

@mitchmindtree
Copy link

mitchmindtree commented Dec 5, 2018

@torkleyy interesting point. It probably should not modify the Cargo.toml at all, but it would be worth considering whether or not cargo should allow for path dependencies but only for crates within the same workspace.

E.g. If building foo and a dependency on bar is specified via a path, perhaps cargo could implicitly publish the new version of foo with the version of bar that was published during the same cargo publish --all? This would also allow for cargo to easily build everything first locally before publishing any of the packages.

Simplified Steps

I'm imagining cargo publish --all should do something like the following:

  1. Check manifests and version validity of all packages before starting to build any.
  2. Create DAG of workspace packages and determine the build order.
  3. Build packages. If any failures occur, bail out of the whole process.
  4. If all manifests and valid and packages are built, publish all packages in the order in which they were built.

It would be worth considerinng if step 4 should be a special "atomic step" recognised by crates.io so that if for some reason the net drops out or there's a crash the user doesn't end up with only half of their packages published.

@Nemo157
Copy link
Member

Nemo157 commented Dec 5, 2018

@mitchmindtree I would like to also see an extra step between 3 and 4 doing full workspace package validation to replace the current pre-publish validation. This does the normal per-package validation steps with 2 changes:

  1. crates.io is queried for an exact version match of any of the crates. If there is an exact version match then the contents are compared to make sure the crate hasn't changed, if they match the crate is removed from the set to publish. (Basically making publishing a crate idempotent so that you can republish a workspace where you have only updated some of the crates.)

  2. during this validation crates can depend on crates that are either available on crates.io or are part of the current set to publish.

@torkleyy
Copy link
Contributor

I started working on this here: https://gitlab.com/torkleyy/cargo-publish-all

@torhovland
Copy link
Contributor

This issue is S-accepted, but it seems to depend on #10948, which isn't. Any chance that can get accepted?

@torhovland
Copy link
Contributor

torhovland commented Aug 6, 2024

Now that #13947 is merged, it's time to look into multi-package publishing:

  • Package everything (including verification) before any publishes
  • Checking if any of the proposed versions are already published
  • Ensuring dependencies are published or will be published as part of this
  • When publishing, we'll also have to do it in-order.

What's a good way to tackle the following?

the biggest open question is how to build the DAG for packaging and then pass that up to publishing.

Would it make sense for packaging to output the packaged order to a file, and let publishing read that? Or should publishing simply re-run the same ordering logic that packaging did, almost like a dry-run packaging?

@epage
Copy link
Contributor

epage commented Aug 6, 2024

For clearer status, writing it out with links to issues.

Would it make sense for packaging to output the packaged order to a file, and let publishing read that? Or should publishing simply re-run the same ordering logic that packaging did, almost like a dry-run packaging?

imo we shouldn't do sideband communication (using the filesystem to pass state from one function to another).

If we could have a single graph for both, that would be ideal as it removes the risk of the two graph traversals disagreeing.

I think ideally, we'd adjust the packaging abstraction so cargo package and cargo publish call into the same graph generation (and registry inferring() code and pass that to a graph packaging function and then publish walks the graph to publish everything.

@epage epage added the Z-package-workspace Nightly: package-workspace label Sep 5, 2024
bors added a commit that referenced this issue Sep 6, 2024
Publish workspace

Adds support for simultaneously publishing multiple (possibly inter-dependent) packages in a workspace, gated by the `-Zpackage-workspace` flag.

Questions to be worked out through stabilization:
- Are we ok stabilizing this and #10948 at the same time?  Currently, they are behind the same flag
- What is the desired behavior for the publish timeout? This PR uploads the crates in batches (depending on the dependency graph), and we only timeout if nothing in the batch is available within the timeout, deferring the rest to the next wait-for-publish. So for example, if you have packages `a`, `b`, `c` then we'll wait up to 60 seconds and if only `a` and `b` were ready in that time, we'll then wait another 60 seconds for `c`.
- What is the desired behavior when some packages in a workspace have `publish = false`? This PR raises an error whenever any of the selected packages has `publish = false`, so it will error on `cargo publish --workspace` in a workspace with an unpublishable package. An alternative interface would implicitly exclude unpublishable packages in this case, but still error out if you explicitly select an unpublishable package with `-p package-name` (see #14356). This PR's behavior is the most conservative one as it can change from an error to implicit excludes later.

This is part of #1169
@epage
Copy link
Contributor

epage commented Oct 23, 2024

Was hoping to do a call for testing but ran into #14721 which breaks some ways of testing this feature.

@foresterre
Copy link

foresterre commented Oct 23, 2024

Are there ways of testing which can already help?

I would absolutely love to have this feature built in.

@epage
Copy link
Contributor

epage commented Oct 23, 2024

Feel free to give it a try in nightly. You don't need more formalized testing instructions to give it a try.

@epage
Copy link
Contributor

epage commented Nov 6, 2024

Tried to test it again and ran into another issues, see #14789

@nyurik
Copy link
Contributor

nyurik commented Nov 21, 2024

UPDATE
The issue bellow is solvable with the nightly toolchain: cargo +nightly -Z package-workspace publish --dry-run -- which handles the unpublished version.

Original post

I ran into a related issue: my workspace has 3 packages that are chain-dependent (varnish -> macros -> sys). The version is the same for all of them, and gets updated at the same time. All 3 packages' Cargo.toml use version.workspace = true, and in their dependencies use the needed varnish-*.workspace = true . The workspace one looks like this:

[workspace.package]
version = "0.2.2"

[workspace.dependencies]
varnish = { path = "./varnish", version = "0.2.2" }
varnish-macros = { path = "./varnish-macros", version = "0.2.2" }
varnish-sys = { path = "./varnish-sys", version = "0.2.2" }

cargo publish --dry-run fails due to failed to select a version for the requirement because the sub-package hasn't been published yet -- because 0.2.2 has not been published yet.

@epage
Copy link
Contributor

epage commented Dec 13, 2024

Call for testing

cargo publish --workspace is now available and we're looking for feedback to see if this can be stabilized!

Basic steps:

Requirements:

  • 1.85.0-nightly (2024-12-01) or newer

Try cargo publish -Zpublic-dependency in your existing workflows and with various combinations of multiple packages being selected (--workspace, --exclude, -p foo -p bar)

Support in third-party release tools:

  • cargo release v0.25.14 as cargo +nightly release -Zworkspace-publish

In particular, consider

  • Publish order is correct?
  • Handling of cycles?
  • Correctly waiting between published packages?
  • --dry-run works?

Note:

Please leave feedback on this issue

@epage epage added the call-for-testing Marks issues that require broader testing from the community, e.g. before stabilization. label Dec 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-workspaces Area: workspaces call-for-testing Marks issues that require broader testing from the community, e.g. before stabilization. Command-publish E-medium Experience: Medium S-accepted Status: Issue or feature is accepted, and has a team member available to help mentor or review Z-package-workspace Nightly: package-workspace
Projects
Status: Big Projects, no backers
Development

No branches or pull requests