Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compacting: integrate into Flows system #583

Closed
Tracked by #58
zaychenko-sergei opened this issue Apr 8, 2024 · 0 comments · Fixed by #586
Closed
Tracked by #58

Compacting: integrate into Flows system #583

zaychenko-sergei opened this issue Apr 8, 2024 · 0 comments · Fixed by #586
Assignees
Labels
enhancement New feature or request

Comments

@zaychenko-sergei
Copy link
Contributor

zaychenko-sergei commented Apr 8, 2024

User requirements:

  • it should be possible to launch hard compacting manually for root datasets only
  • it should be possible to override default settings for max slice byte size / number of records in UI
  • we should not allow define a schedule for this type of the flow, as it's a dangerous operation
  • derived datasets that are dependent on the compacted dataset will fail their next update, and that's expected
  • failing transform flows of derived datasets should show a minimal error message explaining what happened

Technical requirements:

  • rename existing compacting to HardCompacting, as we will have SoftCompacting at some future point
  • update GraphQL schema to be able to launch compacting flow:
    • allow compacting-specific flow configuration options (max_slice_size, max_slice_records)
    • for manual triggers probably nothing extra should be done
    • for defining schedules, ensure a correct error is returned when attempted (not allowed for this type of flow)
    • add unit tests
  • connect flow system and compacting service to execute HardCompacting flow
  • compacting service should be configured using options defined in flow configuration, or with defaults, if no config is active
  • propagate compacting result status to flows system up to GraphQL API:
    • extend flow summary/description/... for UI so that enough information is transformed
    • for compacting we are interested how many blocks we had initially, how many we have finally, and probably total number of records
    • add view API unit tests
  • succesful completion of root compacting should initiate "ExecuteTransform" flows of derived datasets:
    • those datasets would fail for now
    • we need to ensure error message is shown up to API level
    • in near future, there will be another type of transforming, which resets previous results and re-calculates derived dataset from scratch, but we are skipping this for now
  • cover flow system logic with solid integration tests ("time diagram" type of tests that is used for other flow system functions now)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants