-
Notifications
You must be signed in to change notification settings - Fork 5k
[Metricbeat][SQL] Add cursor-based incremental data fetching to SQL module #48722
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 32 commits
Commits
Show all changes
62 commits
Select commit
Hold shift + click to select a range
d83f11f
[Metricbeat][SQL] Add cursor-based incremental data fetching to SQL m…
shmsr 97ce131
Fix remaining Latinisms in cursor doc comments
shmsr 4cf2d92
Improve cursor user-facing docs completeness
shmsr cb515db
Fix CI lint errors: errcheck, goimports, gosec, Vale
shmsr f49d617
Run gofumpt on cursor package files
shmsr bd21c8d
make update
shmsr 0b4fd96
Merge branch 'main' into feat/sql-cursor-incremental-fetch
shmsr 4dc4125
docs update
shmsr 7f616e2
Add query timeout protection for cursor-based fetches
shmsr d08a942
make update
shmsr d6b3ea4
Merge branch 'main' into feat/sql-cursor-incremental-fetch
shmsr 5aeb4fb
Harden SQL cursor registry sharing to prevent file conflicts
shmsr c5b347c
Refactor cursor state key: replace moduleID with direction, document …
shmsr db191c1
Update doc.go state key documentation: add detail, fix DSN tradeoff
shmsr 2ad663e
Update docs
shmsr 5e16beb
Merge branch 'main' into feat/sql-cursor-incremental-fetch
shmsr 99b5051
Merge branch 'main' into feat/sql-cursor-incremental-fetch
shmsr c81c107
Merge branch 'main' into feat/sql-cursor-incremental-fetch
shmsr 07c9c8e
Address review comments
shmsr 1b7c9ad
Fix integration tests
shmsr 0fe7707
Address review comments
shmsr 1e57eb3
Format and add more UT
shmsr 5538725
Merge branch 'main' into sql-cursor-incremental-fetch-oracle-integrat…
stefans-elastic 9cf5d16
oracle integration tests
stefans-elastic f206667
Merge branch 'sql-cursor-incremental-fetch-oracle-integration-tests' …
stefans-elastic 39e24df
Merge branch 'main' into feat/sql-cursor-incremental-fetch
shmsr b70bd8a
[Metricbeat][SQL] Address PR review feedback and improve test robustness
shmsr 9bd7d96
Move test helper newStore to store_test.go
shmsr 56a8e5a
Ensure valid cursor types are checked dynamically
shmsr 8433d42
make update
shmsr 7bf027e
Fix test
shmsr 1ceb89e
Merge branch 'main' into feat/sql-cursor-incremental-fetch
shmsr 6c92252
Apply suggestions from code review
shmsr 53e10a5
Apply suggestions from code review
shmsr 0c1d86d
Address PR feedback and clean up docs/tests
shmsr bbac66d
Address review comments
shmsr 82ac18c
Merge branch 'main' into feat/sql-cursor-incremental-fetch
shmsr c5467fe
Address review comments
shmsr 2464f9b
Update x-pack/metricbeat/module/sql/query/cursor/store_test.go
shmsr 889911d
Address review comments
shmsr 7278b20
Address review comments
shmsr b3d7091
Address review comments
shmsr 0f2684a
Merge branch 'main' into feat/sql-cursor-incremental-fetch
shmsr 018092f
Address review comments
shmsr 432722b
Address review comments
shmsr d487d78
Address review comments
shmsr a07cd54
Fix test
shmsr b18123a
Add better placeholder handling and improve tests
shmsr d1bbbd0
Better tests
shmsr 68476b3
More improvements
shmsr ce40c95
Merge branch 'main' into feat/sql-cursor-incremental-fetch
shmsr f8890a1
Fix CI
shmsr 53e0ad0
Merge branch 'main' into feat/sql-cursor-incremental-fetch
shmsr 10c43db
Update docs
shmsr 80e25bb
Add auto-infer type feature for cursor
shmsr b9fa82c
Introduce state ID
shmsr ceaaec7
Address review comments
shmsr e7ae6d1
Merge branch 'main' into feat/sql-cursor-incremental-fetch
shmsr 540e01b
Fix windows tests and lint checks
shmsr 15bab2d
Merge branch 'main' into feat/sql-cursor-incremental-fetch
shmsr 7a5ff84
Merge branch 'main' into feat/sql-cursor-incremental-fetch
shmsr ce52657
Merge branch 'main' into feat/sql-cursor-incremental-fetch
shmsr File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
49 changes: 49 additions & 0 deletions
49
changelog/fragments/1770358526-add-cursor-based-incremental-fetching-to-sql-module.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| # REQUIRED | ||
| # Kind can be one of: | ||
| # - breaking-change: a change to previously-documented behavior | ||
| # - deprecation: functionality that is being removed in a later release | ||
| # - bug-fix: fixes a problem in a previous version | ||
| # - enhancement: extends functionality but does not break or fix existing behavior | ||
| # - feature: new functionality | ||
| # - known-issue: problems that we are aware of in a given version | ||
| # - security: impacts on the security of a product or a user's deployment. | ||
| # - upgrade: important information for someone upgrading from a prior version | ||
| # - other: does not fit into any of the other categories | ||
| kind: feature | ||
|
|
||
| # REQUIRED for all kinds | ||
| # Change summary; a 80ish characters long description of the change. | ||
| summary: Add cursor-based incremental data fetching to the SQL module query metricset. | ||
|
|
||
| # REQUIRED for breaking-change, deprecation, known-issue | ||
| # Long description; in case the summary is not enough to describe the change | ||
| # this field accommodate a description without length limits. | ||
| description: | | ||
| Add a cursor feature to the SQL query metricset that enables incremental data | ||
| fetching by tracking the last fetched row value. Supports integer, timestamp, | ||
| date, float, and decimal cursor types with ascending and descending scan | ||
| directions. State is persisted via libbeat statestore (memlog backend). | ||
|
|
||
| # REQUIRED for breaking-change, deprecation, known-issue | ||
| # impact: | ||
|
|
||
| # REQUIRED for breaking-change, deprecation, known-issue | ||
| # action: | ||
|
|
||
| # REQUIRED for all kinds | ||
| # Affected component; usually one of "elastic-agent", "fleet-server", "filebeat", "metricbeat", "auditbeat", "all", etc. | ||
| component: metricbeat | ||
|
|
||
| # AUTOMATED | ||
| # OPTIONAL to manually add other PR URLs | ||
| # PR URL: A link the PR that added the changeset. | ||
| # If not present is automatically filled by the tooling finding the PR where this changelog fragment has been added. | ||
| # NOTE: the tooling supports backports, so it's able to fill the original PR number instead of the backport PR number. | ||
| # Please provide it if you are adding a fragment for a different PR. | ||
| pr: https://github.com/elastic/beats/pull/48722 | ||
|
|
||
| # AUTOMATED | ||
| # OPTIONAL to manually add other issue URLs | ||
| # Issue URL; optional; the GitHub issue related to this changeset (either closes or is part of). | ||
| # If not present is automatically filled by the tooling with the issue linked to the PR number. | ||
| # issue: https://github.com/owner/repo/1234 |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.