Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(datasets) Add a check for same object partitioners #4335

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

adam-narozniak
Copy link
Member

@adam-narozniak adam-narozniak commented Oct 14, 2024

Issue

The partitioners specification as dict of split to the partitioner object should not have the same objects (value of the dict) assigned to different keys (even if the same type of partitioning happens for different splits).

Proposal

This PR adds a check to ensure this correctness (without it, the code "fails" silently, the non-first used split will not be partitioned but the partitions of the first divided split will be returned).

Examples

This would fail silently

iid = IidPartitioner(num_partitions=10)
fds = FederatedDatasets("mnist", partitioners={"train": iid, "test": iid})

With this PR, it would raise an error.

@adam-narozniak adam-narozniak marked this pull request as ready for review November 29, 2024 13:47
Comment on lines +344 to +346
Check if the multiple partitioner objects are not the same Python object, which
is not allowed, as the partitioner objects should be independent (one
partitioner per split).
Copy link
Contributor

@jafermarq jafermarq Dec 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Check if the multiple partitioner objects are not the same Python object, which
is not allowed, as the partitioner objects should be independent (one
partitioner per split).
Check if each partitioner is a different Python object. Using the same partitioner
for different splits is not allowed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is clearer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants