Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pubsys: switch to tough async #98

Merged
merged 1 commit into from
Nov 28, 2023
Merged

pubsys: switch to tough async #98

merged 1 commit into from
Nov 28, 2023

Conversation

webern
Copy link
Contributor

@webern webern commented Oct 6, 2023 •
edited
Loading

Issue #, if available:

Description of changes:

Use the new async version of tough and tuftool

Testing

  • make variant
  • make ami
  • make repo
  • verify repo
  • re-sign repo

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@webern webern mentioned this pull request Oct 6, 2023
Merged
bcressey
bcressey reviewed Oct 9, 2023
View reviewed changes
Copy link
Contributor

@bcressey bcressey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the full async conversion, we'll want to look at replacing std with tokio in several spots:

  • std::io
  • std::fs
  • std::process

For example, upload-ova uses a std-process to run govc inside a container. That's not going to do the executor any favors.

tools/pubsys/src/repo.rs Outdated Show resolved Hide resolved
tools/pubsys/src/repo/validate_repo/mod.rs Outdated Show resolved Hide resolved
tools/pubsys/src/repo/validate_repo/mod.rs Outdated Show resolved Hide resolved
tools/pubsys/src/main.rs Show resolved Hide resolved
@webern
Copy link
Contributor Author

webern commented Oct 9, 2023

https://github.com/bottlerocket-os/twoliter/compare/b4249f6622f3dbcc4e98c9c69492c60fd0e8f92b..f2b5060758348e2280932739b55d1b9347b204c9

Rebase

@webern
Copy link
Contributor Author

webern commented Oct 10, 2023

https://github.com/bottlerocket-os/twoliter/compare/f2b5060758348e2280932739b55d1b9347b204c9..d5666cf794c686204a4eba0b971d2398af1de7a5

Fix cargo deny issues (including temporary fixes). Squash to one commit.

@webern
Copy link
Contributor Author

webern commented Oct 10, 2023

https://github.com/bottlerocket-os/twoliter/compare/d5666cf794c686204a4eba0b971d2398af1de7a5..c6f20307ab1e176d586a1e29481667272d7dc80b

Eliminate extra runtime creations and use tokio::fs instead of std::fs.

@webern
Copy link
Contributor Author

webern commented Nov 8, 2023

https://github.com/bottlerocket-os/twoliter/compare/c4ba5003fd3a1e175a4aaf0ccdd7e5141eda855a..6b64a72b5ac06067c30d4f53a1bf7736d481f473

Squash

@webern
Copy link
Contributor Author

webern commented Nov 8, 2023

https://github.com/bottlerocket-os/twoliter/compare/6b64a72b5ac06067c30d4f53a1bf7736d481f473..75e2455eb1e5e9b4c5cdd465812e07551a9f27c0

Switch to published versions.

webern
webern commented Nov 9, 2023
View reviewed changes
tools/pubsys/Cargo.toml Outdated Show resolved Hide resolved
tools/pubsys/src/aws/publish_ami/mod.rs Outdated Show resolved Hide resolved
tools/pubsys/src/main.rs Outdated Show resolved Hide resolved
@webern
Copy link
Contributor Author

webern commented Nov 21, 2023

https://github.com/bottlerocket-os/twoliter/compare/a237b0ccf40525ea64f1a4f1e8d63688b8d24a33..762c65efc913c60ed80343d1b3f8b74a5e335560

Rebase

@webern
Copy link
Contributor Author

webern commented Nov 21, 2023

https://github.com/bottlerocket-os/twoliter/compare/762c65efc913c60ed80343d1b3f8b74a5e335560..f4c94784ad2b70a88ce3185259313f71f15428a6

Small fixups from my own comments.

@webern webern mentioned this pull request Nov 21, 2023
Merged
5 tasks
@webern webern marked this pull request as ready for review November 21, 2023 19:42
@webern webern mentioned this pull request Nov 22, 2023
Closed
7 tasks
@webern webern requested a review from ecpullen November 27, 2023 17:23
zmrow
zmrow approved these changes Nov 27, 2023
View reviewed changes
Copy link
Contributor

@zmrow zmrow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! 🥇

cbgbt
cbgbt reviewed Nov 28, 2023
View reviewed changes
Copy link
Contributor

@cbgbt cbgbt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good to me, nice work. I only have one concern.

tools/pubsys/src/repo/validate_repo/mod.rs Outdated
let _permit = cloned_sem.acquire().await.context(error::SemaphoreSnafu)?;
download_target(cloned_repo, cloned_target).await
};
tasks.spawn(fut);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could spawn an unbounded number of tasks. Before reading this code, my understanding was that if we are using the multi-threaded runtime this could therefor result in an unbounded number of OS threads being spawned. After re-reading the runtime docs it seems like the number of OS threads is bound to the number of CPU cores.

In the past I've used buffered_unordered to limit the number of tasks as well, which seems slightly safer but it is certainly more complex.

ecpullen
ecpullen approved these changes Nov 28, 2023
View reviewed changes
Copy link
Contributor

@ecpullen ecpullen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM other than Sean's comment

@webern
Copy link
Contributor Author

webern commented Nov 28, 2023

https://github.com/bottlerocket-os/twoliter/compare/f4c94784ad2b70a88ce3185259313f71f15428a6..85a8711cfd1408be24168e6477bb03287dc5e36f

Attempt to use buffered_unordered. Will test.

@webern
Copy link
Contributor Author

webern commented Nov 28, 2023

https://github.com/bottlerocket-os/twoliter/compare/85a8711cfd1408be24168e6477bb03287dc5e36f..74bfad10c71d74fd1b16c10fb4d99358dbd2eeb0

Restore MAX_DOWNLOAD_THREADS constant and info log.

@webern
Copy link
Contributor Author

webern commented Nov 28, 2023

https://github.com/bottlerocket-os/twoliter/compare/74bfad10c71d74fd1b16c10fb4d99358dbd2eeb0..784d080a6cae8d22853fd607fa7d88d348d6f4b8

Clippy

cbgbt
cbgbt approved these changes Nov 28, 2023
View reviewed changes
Copy link
Contributor

@cbgbt cbgbt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice 🚀

@webern webern merged commit 302ef13 into bottlerocket-os:develop Nov 28, 2023
1 check passed
@webern webern deleted the tough-async branch November 28, 2023 22:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bcressey bcressey bcressey left review comments

@cbgbt cbgbt cbgbt approved these changes

@zmrow zmrow zmrow approved these changes

@ecpullen ecpullen ecpullen approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

test and use tough async
5 participants
@webern @bcressey @cbgbt @zmrow @ecpullen