Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

agent: use touch publications for expanded draft specs #1645

Merged
merged 1 commit into from
Sep 20, 2024

Conversation

psFried
Copy link
Member

@psFried psFried commented Sep 20, 2024

Description:

Touch publications are kind of a perfect fit for expanded specs in user-initiated publications. Expanded specs are never modified by the publication, and it's annoying to add publication specs for things that users haven't intended to publish. Using is_touch: true on expanded specs prevents the creation of unnecessary publication specs. It also avoids unnecessary modification of last_pub_id, so there will be less chance of ExpectPubIdMismatch errors. The touched specs are still validated, which serves the original purpose of spec expansion.

Resolves #1643

Workflow steps:

  • Create a capture into one or more collections, and a materialization.
  • Publish a change to only the collection using flowctl
  • Observe that the last_pub_id of the capture and materialization have not changed, but the last_build_id has
  • No publication_specs are added for the capture and materialization, only the collection

Notes for reviewers:

This is a follow up change from #1629 , which added the "touch" capability. I wanted to break this out as a separate PR just to limit the scope of changes in any one deployment.


This change is Reviewable

Touch publications are kind of a perfect fit for expanded specs in
user-initiated publications. Expanded specs are never modified by the
publication, and it's annoying to add publication specs for things that
users haven't intended to publish. Using `is_touch: true` on expanded
specs prevents the creation of unnecessary publication specs. It also
avoids unnecessary modification of `last_pub_id`, so there will be less
chance of `ExpectPubIdMismatch` errors. The touched specs are still
validated, which serves the original purpose of spec expansion.

Resolves #1643
Copy link
Member

@jgraettinger jgraettinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@psFried psFried merged commit 1edb7f5 into master Sep 20, 2024
3 checks passed
@psFried psFried added the change:planned This is a planned change label Sep 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
change:planned This is a planned change
Projects
None yet
Development

Successfully merging this pull request may close these issues.

agent: touch expanded specs for user-initiated publications
2 participants