Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Improve StateGroupCache efficiency for wildcard lookups #3383

Merged
merged 1 commit into from
Jun 22, 2018
Merged

Improve StateGroupCache efficiency for wildcard lookups #3383

merged 1 commit into from
Jun 22, 2018

Conversation

richvdh
Copy link
Member

@richvdh richvdh commented Jun 12, 2018

Opening this to remind me to discuss #3380 with erik on his return. It's currently a heinous hack, so needs doing differently whatever we decide.

@richvdh richvdh force-pushed the rav/hacky_speedup_state_group_cache branch from 4cf5f6c to 1ad0a73 Compare June 20, 2018 19:21
@richvdh
Copy link
Member Author

richvdh commented Jun 20, 2018

[I discussed this with erik and we agreed how to make it less of a hack]

@richvdh
Copy link
Member Author

richvdh commented Jun 20, 2018 •
edited
Loading

note that this builds on top of #3382.

@richvdh
Copy link
Member Author

richvdh commented Jun 20, 2018

fixes #3380

@richvdh richvdh changed the title Hackery to disable partial state group caching Improve StateGroupCache efficiency for wildcard lookups Jun 21, 2018
erikjohnston
erikjohnston reviewed Jun 21, 2018
View reviewed changes
Copy link
Member

@erikjohnston erikjohnston left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we removing the concept of full? We still sometimes only pull out subsets?

@erikjohnston erikjohnston assigned richvdh and unassigned erikjohnston Jun 21, 2018
@richvdh
Copy link
Member Author

richvdh commented Jun 22, 2018

Why are we removing the concept of full? We still sometimes only pull out subsets?

it was just redundant. full=True is the same as fetched_keys is not None: you would never set full unless fetched_keys (formerly known as known_absent, which was also a misnomer) was None, and vice versa.

@richvdh
Disable partial state group caching for wildcard lookups
43e02c4 
When _get_state_for_groups is given a wildcard filter, just do a complete
lookup. Hopefully this will give us the best of both worlds by not filling up
the ram if we only need one or two keys, but also making the cache still work
for the federation reader usecase.
@richvdh richvdh force-pushed the rav/hacky_speedup_state_group_cache branch from 5c5ba6a to 43e02c4 Compare June 22, 2018 10:53
@richvdh richvdh assigned erikjohnston and unassigned richvdh Jun 22, 2018
@richvdh richvdh merged commit bb018d0 into develop Jun 22, 2018
@richvdh richvdh deleted the rav/hacky_speedup_state_group_cache branch July 10, 2018 12:56
@ara4n ara4n mentioned this pull request Aug 20, 2018
Closed
@richvdh richvdh mentioned this pull request Aug 21, 2018
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@erikjohnston erikjohnston erikjohnston approved these changes

Assignees

@erikjohnston erikjohnston

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@richvdh @erikjohnston