Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[2.4] Simulator end run for all clients #2515

Merged
merged 12 commits into from
Apr 19, 2024

Conversation

yhwen
Copy link
Collaborator

@yhwen yhwen commented Apr 18, 2024

Fixes # .

Description

Simulator only runs the END_RUN events for those active running clients when the workflow finishes the run. All those swapped out clients need to be swapped in and re-created to run the END_RUN event handling. Most of the applications the END_RUN event handling are not needed when the client processes are not active. Change the simulator to provide an "end_run_for_all" option. The default is false. Only set it to run END_RUN event handling for all clients when explicitly set.

Also change to use multi-threads to run the END_RUN event handling if needed.

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • Quick tests passed locally by running ./runtest.sh.
  • In-line docstrings updated.
  • Documentation updated.

Copy link
Collaborator

@yanchengnv yanchengnv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The end_run processing logic is very hard to understand. Please improve.

nvflare/private/fed/app/simulator/simulator_runner.py Outdated Show resolved Hide resolved
chesterxgchen
chesterxgchen previously approved these changes Apr 18, 2024
Copy link
Collaborator

@yanchengnv yanchengnv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good now.

@yhwen
Copy link
Collaborator Author

yhwen commented Apr 19, 2024

/build

2 similar comments
@yhwen
Copy link
Collaborator Author

yhwen commented Apr 19, 2024

/build

@SYangster
Copy link
Collaborator

/build

@chesterxgchen chesterxgchen merged commit fe09b25 into NVIDIA:2.4 Apr 19, 2024
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants