-
Notifications
You must be signed in to change notification settings - Fork 234
feat: cache warmer implementation #1501
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
StarpTech
merged 65 commits into
main
from
suvij/eng-5580-create-new-optimized-clickhouse-views-for-topn
Jan 17, 2025
Merged
Changes from all commits
Commits
Show all changes
65 commits
Select commit
Hold shift + click to select a range
18fcc79
feat: cache warmer implementation
JivusAyrus 4e6ff40
fix: remove unnecessary code
JivusAyrus bbc7f87
fix: lint
JivusAyrus 0318842
Merge branch 'main' into suvij/eng-5580-create-new-optimized-clickhou…
Noroth 342c55e
feat(router): implement cache warming from CDN source (#1491)
Noroth 8764ff0
fix: path
JivusAyrus e66b761
feat: add cdn as an option for the source of cache operations
JivusAyrus a812141
Merge branch 'main' into suvij/eng-5580-create-new-optimized-clickhou…
Noroth 6598124
Merge branch 'suvij/eng-5580-create-new-optimized-clickhouse-views-fo…
JivusAyrus 822793c
feat: add cache operations ui
JivusAyrus 33887f4
fix: use p90 and order operations by descending order
JivusAyrus 5aa9575
feat: add operation details page
JivusAyrus 0f0271f
chore: remove migrations
JivusAyrus bb92cc9
Merge branch 'main' of github.com:wundergraph/cosmo into suvij/eng-55…
JivusAyrus d339d4d
chore: add migrations
JivusAyrus 4d43501
fix: ci
JivusAyrus 3375630
fix: security bot issues
JivusAyrus 223fbf9
fix: url
JivusAyrus 56ceb02
fix: pr suggestions
JivusAyrus 37a61e7
fix: tests
JivusAyrus 45ea000
feat: add option to enable cache warmer
JivusAyrus 4a0bfde
feat: perform cache warming actions only if its enabled
JivusAyrus 300bdb7
fix: rename rpc to configureCacheWarmer
JivusAyrus 7e393bf
fix: pr suggestion
JivusAyrus ebe4160
Merge branch 'main' of github.com:wundergraph/cosmo into suvij/eng-55…
JivusAyrus 2e60fc2
fix: lint
JivusAyrus 44a135c
fix: ci
JivusAyrus a7ab522
Merge branch 'main' into suvij/eng-5580-create-new-optimized-clickhou…
Noroth 70c220c
fix: cache warmer configuration ui
JivusAyrus 59cd756
fix: operations table ui
JivusAyrus b801cce
fix: fetch the operation content of all the hashes in one query
JivusAyrus 7db6963
fix: merge migrations
JivusAyrus 103a21a
fix: remove gql operations migrations
JivusAyrus ab3a286
fix: revert graphql metrics
JivusAyrus 1c1caf4
fix: cache operations ui
JivusAyrus 5bc7a71
fix: remove cache warmer worker
JivusAyrus 6d44616
fix: check if the org has access to cache warmer
JivusAyrus dd4e392
fix: remove unnecessary fields from the query
JivusAyrus 42bf8f2
chore: add cache warmer to FeatureIds type
JivusAyrus c49c3a7
fix: remoce cache warmer worker
JivusAyrus 12474e6
fix: remove subgraph id condition in the mv
JivusAyrus 1b1bab7
fix: check if the org has access to cache-warmer in the rpcs
JivusAyrus 0f38a20
fix: check if the org has access to cahe warmer
JivusAyrus ed81956
chore: remove obsolte commented code
Noroth c36be50
feat: add getPersistedOperation function
JivusAyrus 6e96fbd
chore: add tests
JivusAyrus d25fdeb
fix: lint
JivusAyrus 9abab21
fix: operations with top planning time query
JivusAyrus 50a1995
fix: cache operations ui
JivusAyrus 51ffe16
fix: message
JivusAyrus 94246e6
fix: handle clickhouse optionally
JivusAyrus 6afcc9c
fix: security bug
JivusAyrus 3f3d235
Merge branch 'main' into suvij/eng-5580-create-new-optimized-clickhou…
JivusAyrus 2fe6910
fix: change the status code to upgrade plan
JivusAyrus c763d86
fix: rpcs
JivusAyrus b924ccd
fix: remove unused imports
JivusAyrus 73e8df8
chore: use protobuf type
Noroth c2620c8
feat: add no cache headers to the cdn
JivusAyrus fcca618
Merge branch 'suvij/eng-5580-create-new-optimized-clickhouse-views-fo…
JivusAyrus b2c43c1
chore: assign types explicitly
Noroth ebbf17c
chore: don't allow pushing PO with incorrect values
Noroth 17d20c4
chore: refactor
StarpTech 3d218e4
chore: refactor
StarpTech 616ace9
chore: update lock
StarpTech ac308b6
chore: fix lint
StarpTech File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| import { existsSync } from 'node:fs'; | ||
| import { readFile } from 'node:fs/promises'; | ||
| import { Command, program } from 'commander'; | ||
| import pc from 'picocolors'; | ||
| import { resolve } from 'pathe'; | ||
| import { EnumStatusCode } from '@wundergraph/cosmo-connect/dist/common/common_pb'; | ||
| import { BaseCommandOptions } from '../../../../../core/types/types.js'; | ||
| import { getBaseHeaders } from '../../../../../core/config.js'; | ||
|
|
||
| export default (opts: BaseCommandOptions) => { | ||
| const command = new Command('push'); | ||
| command.description('Pushes new cache warmer operations to the registry'); | ||
| command.argument( | ||
| '<graph_name>', | ||
| 'The name of the federated graph or monograph on which the check operations are stored.', | ||
| ); | ||
| command.option('-n, --namespace [string]', 'The namespace of the federated graph or monograph.'); | ||
| command.option( | ||
| '-o, --operation-name <operation-name>', | ||
| 'The name of the operation. Only needed when working with multi-operation document.', | ||
| ); | ||
| command.option( | ||
|
StarpTech marked this conversation as resolved.
|
||
| '-p, --persisted-operation-id <persisted-operation-id>', | ||
| 'The id of the persisted operation to be pushed.', | ||
| ); | ||
| command.option( | ||
| '-f, --file <file>', | ||
| 'The file with the operation to push - supports .graphql, .gql. If both the file and the persisted operation id are provided, the persisted operation id will be used.', | ||
| ); | ||
|
|
||
| command.action(async (name, options) => { | ||
| if (!options.file && !options.persistedOperationId) { | ||
| command.error(pc.red('No operation file or the id of persisted operation provided')); | ||
| } | ||
|
|
||
| if (options.persistedOperationId && options.file) { | ||
| command.error(pc.red('Only the persisted operation id or the file can be specified')); | ||
| } | ||
|
|
||
| let operation: string | undefined; | ||
| if (options.file) { | ||
| const operationFile = resolve(options.file); | ||
| if (!existsSync(operationFile)) { | ||
| program.error( | ||
| pc.red( | ||
| pc.bold( | ||
| `The operation file '${pc.bold(operationFile)}' does not exist. Please check the path and try again.`, | ||
| ), | ||
| ), | ||
| ); | ||
| } | ||
|
|
||
| const schemaBuffer = await readFile(operationFile); | ||
| operation = new TextDecoder().decode(schemaBuffer); | ||
| if (operation.trim().length === 0) { | ||
| program.error( | ||
| pc.red(pc.bold(`The schema file '${pc.bold(operationFile)}' is empty. Please provide a valid operation.`)), | ||
| ); | ||
| } | ||
| } | ||
|
|
||
| const result = await opts.client.platform.pushCacheWarmerOperation( | ||
| { | ||
| federatedGraphName: name, | ||
| operationContent: operation, | ||
| operationName: options.operationName, | ||
| operationPersistedId: options.persistedOperationId, | ||
| namespace: options.namespace, | ||
| }, | ||
| { headers: getBaseHeaders() }, | ||
| ); | ||
|
|
||
| if (result.response?.code === EnumStatusCode.OK) { | ||
| console.log(pc.green(`The cache warmer operation was pushed successfully.`)); | ||
| } else { | ||
| console.log(pc.red(`Failed to push the cache warmer operation. Please try again.`)); | ||
| if (result.response?.details) { | ||
| console.error(pc.red(pc.bold(result.response?.details))); | ||
| } | ||
| process.exit(1); | ||
|
JivusAyrus marked this conversation as resolved.
|
||
| } | ||
| }); | ||
| return command; | ||
| }; | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| import { Command } from 'commander'; | ||
| import { BaseCommandOptions } from '../../../../core/types/types.js'; | ||
| import PushCacheOperation from './commands/push.js'; | ||
|
|
||
| export default (opts: BaseCommandOptions) => { | ||
| const schema = new Command('cache'); | ||
| schema.description( | ||
| 'Provides commands for pushing and maintaining router cache warmer operations of a federated graph', | ||
| ); | ||
|
|
||
| schema.addCommand(PushCacheOperation(opts)); | ||
|
|
||
| return schema; | ||
| }; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.