Repository navigation
feat(core): Introduce CredentialDependency entity to track credential dependencies #27151
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 10 commits
24d274a
4231912
222a241
4e3291a
e1d5787
a8b6f9f
bb52f50
de8471c
4646f53
eefaa5c
aee6b9f
5e1d382
3a3ed18
98c88f4
9b9d5b4
59176ed
83698db
1ecd266
3672308
ca233fb
56272bb
08bbbd8
a48a736
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| import { | ||
| Column, | ||
| Entity, | ||
| Index, | ||
| JoinColumn, | ||
| ManyToOne, | ||
| PrimaryGeneratedColumn, | ||
| Relation, | ||
| Unique, | ||
| } from '@n8n/typeorm'; | ||
|
|
||
| import { WithCreatedAt } from './abstract-entity'; | ||
| import type { CredentialsEntity } from './credentials-entity'; | ||
|
|
||
| export type CredentialDependencyType = 'externalSecretProvider'; | ||
|
|
||
| @Entity({ name: 'credential_dependency' }) | ||
| @Unique(['credentialId', 'dependencyType', 'dependencyId']) | ||
| export class CredentialDependency extends WithCreatedAt { | ||
| @PrimaryGeneratedColumn() | ||
| id: number; | ||
|
|
||
| @Column({ length: 36 }) | ||
| @Index() | ||
| credentialId: string; | ||
|
|
||
| @Column({ length: 64 }) | ||
| dependencyType: CredentialDependencyType; | ||
|
|
||
| @Column({ length: 255 }) | ||
| dependencyId: string; | ||
|
|
||
| @ManyToOne('CredentialsEntity', { onDelete: 'CASCADE' }) | ||
| @JoinColumn({ name: 'credentialId' }) | ||
| credential: Relation<CredentialsEntity>; | ||
| } | ||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The utilities in this file might give the impression of duplicate code, that's because it is duplicate code 😂 somehow Usually migrations are meant to be self contained with minimal dependencies to avoid breaking when business logic change over time, if this is not the right way here at n8n I'm happy to amend and reuse the existing utility methods :)
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think it's good that migrations don't import code but rather duplicate it 👍
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @konstantintieber Awesome, thanks for reassuring me 🫶 |
| Original file line number | Diff line number | Diff line change | ||
|---|---|---|---|---|
| @@ -0,0 +1,203 @@ | ||||
| import { Container } from '@n8n/di'; | ||||
| import { Cipher } from 'n8n-core'; | ||||
|
|
||||
| import type { MigrationContext, ReversibleMigration } from '../migration-types'; | ||||
|
|
||||
| const credentialDependencyTable = 'credential_dependency'; | ||||
| const externalSecretProviderDependencyType = 'externalSecretProvider'; | ||||
| const providerKeyPattern = '[A-Za-z0-9]+(?:-[A-Za-z0-9]+)*'; | ||||
|
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sort of a duplicate of
|
||||
|
|
||||
| type CredentialRow = { | ||||
| id: string; | ||||
| data: string; | ||||
| }; | ||||
|
|
||||
| export class CreateCredentialDependencyTable1773000000000 implements ReversibleMigration { | ||||
| private readonly cipher = Container.get(Cipher); | ||||
|
|
||||
| async up({ | ||||
| schemaBuilder: { createTable, column, createIndex }, | ||||
| escape, | ||||
| runQuery, | ||||
| runInBatches, | ||||
| logger, | ||||
| migrationName, | ||||
| }: MigrationContext) { | ||||
| await createTable(credentialDependencyTable) | ||||
| .withColumns( | ||||
| column('id').int.primary.autoGenerate2, | ||||
| column('credentialId').varchar(36).notNull, | ||||
| column('dependencyType').varchar(64).notNull, | ||||
| column('dependencyId').varchar(255).notNull, | ||||
| ) | ||||
| .withForeignKey('credentialId', { | ||||
| tableName: 'credentials_entity', | ||||
| columnName: 'id', | ||||
| onDelete: 'CASCADE', | ||||
| }) | ||||
| .withIndexOn(['credentialId']) | ||||
| .withIndexOn(['dependencyType', 'dependencyId']).withCreatedAt; | ||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good set of indexes you added here 👍
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @konstantintieber True 98c88f4
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No automatic sync process that I'm aware of. All human based 😅 |
||||
|
|
||||
| await createIndex( | ||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why the separate createIndex ? why not another
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That’s an absolutely valid question ✅ The main reason is that I found it hard to clearly spot unique indexes when they’re defined alongside non-unique ones using .withIndexOn(['credentialId'])
.withIndexOn(['dependencyType', 'dependencyId'])
.withIndexOn(['credentialId', 'dependencyType', 'dependencyId'], true);Here, the Since this is a migration file (and not something we expect to revisit or refactor), I preferred to make the unique index explicit. This makes it easier for reviewers and future readers to spot which index enforces uniqueness vs which ones are just for performance. Also… the fact that you pointed it out probably means my plan worked 😂 haha |
||||
| credentialDependencyTable, | ||||
| ['credentialId', 'dependencyType', 'dependencyId'], | ||||
| true, | ||||
| ); | ||||
|
|
||||
| const credentialsTable = escape.tableName('credentials_entity'); | ||||
| const query = `SELECT c.id AS id, c.data AS data FROM ${credentialsTable} c ORDER BY c.id`; | ||||
| const providerIdByKey = await this.loadProviderIdByKey(runQuery, escape); | ||||
|
|
||||
| let processedCount = 0; | ||||
| let insertedCount = 0; | ||||
|
|
||||
| await runInBatches<CredentialRow>(query, async (rows) => { | ||||
| if (rows.length === 0) return; | ||||
|
|
||||
| const batchDependencies = rows.flatMap((row) => { | ||||
| const providerKeys = this.extractProviderKeysFromCredentialData(row.data); | ||||
| const providerIds = providerKeys | ||||
| .map((providerKey) => providerIdByKey.get(providerKey)) | ||||
| .filter((providerId): providerId is string => providerId !== undefined); | ||||
| return providerIds.map((providerId) => ({ | ||||
| credentialId: row.id, | ||||
| dependencyType: externalSecretProviderDependencyType, | ||||
| dependencyId: providerId, | ||||
| })); | ||||
| }); | ||||
|
|
||||
| processedCount += rows.length; | ||||
|
|
||||
| if (batchDependencies.length === 0) { | ||||
| return; | ||||
| } | ||||
|
|
||||
| await this.insertDependencies(batchDependencies, escape, runQuery); | ||||
| insertedCount += batchDependencies.length; | ||||
| }); | ||||
|
|
||||
| logger.info( | ||||
| `[${migrationName}] Backfilled credential dependencies for ${processedCount} credentials. Inserted ${insertedCount} dependencies.`, | ||||
| ); | ||||
| } | ||||
|
|
||||
| async down({ schemaBuilder: { dropTable } }: MigrationContext) { | ||||
| await dropTable(credentialDependencyTable); | ||||
| } | ||||
|
|
||||
| private extractProviderKeysFromCredentialData(encryptedCredentialData: string): string[] { | ||||
| const decrypted = this.tryDecryptCredentialData(encryptedCredentialData); | ||||
| if (decrypted === null) return []; | ||||
| return this.extractProviderKeysFromDecryptedData(decrypted); | ||||
| } | ||||
|
|
||||
| private extractProviderKeysFromDecryptedData(decryptedCredentialData: unknown): string[] { | ||||
| const uniqueKeys = new Set<string>(); | ||||
| const valuesToScan: unknown[] = [decryptedCredentialData]; | ||||
|
|
||||
| while (valuesToScan.length > 0) { | ||||
| const currentValue = valuesToScan.pop(); | ||||
| if (typeof currentValue === 'string') { | ||||
| if (!currentValue.includes('$secrets')) continue; | ||||
| for (const dependencyKey of this.extractProviderKeys(currentValue)) { | ||||
| uniqueKeys.add(dependencyKey); | ||||
| } | ||||
| continue; | ||||
| } | ||||
|
|
||||
| if (Array.isArray(currentValue)) { | ||||
| for (const value of currentValue as unknown[]) { | ||||
| valuesToScan.push(value); | ||||
| } | ||||
| continue; | ||||
| } | ||||
|
|
||||
| if (typeof currentValue === 'object' && currentValue !== null) { | ||||
| for (const value of Object.values(currentValue as Record<string, unknown>)) { | ||||
| valuesToScan.push(value); | ||||
| } | ||||
| } | ||||
| } | ||||
|
|
||||
| return [...uniqueKeys]; | ||||
| } | ||||
|
|
||||
| private tryDecryptCredentialData(encryptedCredentialData: string): unknown { | ||||
| try { | ||||
| const decrypted = this.cipher.decrypt(encryptedCredentialData); | ||||
| return JSON.parse(decrypted) as unknown; | ||||
| } catch { | ||||
| return null; | ||||
| } | ||||
| } | ||||
|
|
||||
| private extractProviderKeys(expression: string): string[] { | ||||
| const providerKeys = new Set<string>(); | ||||
| const expressionBlocks = expression.matchAll(/\{\{(.*?)\}\}/gs); | ||||
|
|
||||
| for (const block of expressionBlocks) { | ||||
| const expressionContent = block[1]; | ||||
|
|
||||
| const dotMatches = expressionContent.matchAll( | ||||
| new RegExp(`\\$secrets\\.(${providerKeyPattern})`, 'g'), | ||||
| ); | ||||
| for (const match of dotMatches) { | ||||
| providerKeys.add(match[1]); | ||||
| } | ||||
|
|
||||
| const bracketMatches = expressionContent.matchAll( | ||||
| new RegExp(`\\$secrets\\[['"](${providerKeyPattern})['"]\\]`, 'g'), | ||||
| ); | ||||
| for (const match of bracketMatches) { | ||||
| providerKeys.add(match[1]); | ||||
| } | ||||
| } | ||||
|
|
||||
| return [...providerKeys]; | ||||
| } | ||||
|
|
||||
| private async insertDependencies( | ||||
| dependencies: Array<{ | ||||
| credentialId: string; | ||||
| dependencyType: string; | ||||
| dependencyId: string; | ||||
| }>, | ||||
| escape: MigrationContext['escape'], | ||||
| runQuery: MigrationContext['runQuery'], | ||||
| ) { | ||||
| const dependencyTable = escape.tableName(credentialDependencyTable); | ||||
| const credentialIdColumn = escape.columnName('credentialId'); | ||||
| const dependencyTypeColumn = escape.columnName('dependencyType'); | ||||
| const dependencyIdColumn = escape.columnName('dependencyId'); | ||||
|
|
||||
| const namedParameters: Record<string, string> = {}; | ||||
| const valuesSql = dependencies | ||||
| .map((dependency, index) => { | ||||
| namedParameters[`credentialId${index}`] = dependency.credentialId; | ||||
| namedParameters[`dependencyType${index}`] = dependency.dependencyType; | ||||
| namedParameters[`dependencyId${index}`] = dependency.dependencyId; | ||||
|
|
||||
| return `(:credentialId${index}, :dependencyType${index}, :dependencyId${index})`; | ||||
| }) | ||||
| .join(', '); | ||||
|
|
||||
| await runQuery( | ||||
| `INSERT INTO ${dependencyTable} (${credentialIdColumn}, ${dependencyTypeColumn}, ${dependencyIdColumn}) VALUES ${valuesSql} ON CONFLICT (${credentialIdColumn}, ${dependencyTypeColumn}, ${dependencyIdColumn}) DO NOTHING;`, | ||||
| namedParameters, | ||||
| ); | ||||
| } | ||||
|
|
||||
| private async loadProviderIdByKey( | ||||
| runQuery: MigrationContext['runQuery'], | ||||
| escape: MigrationContext['escape'], | ||||
| ): Promise<Map<string, string>> { | ||||
| const providerTable = escape.tableName('secrets_provider_connection'); | ||||
| const idColumn = escape.columnName('id'); | ||||
| const providerKeyColumn = escape.columnName('providerKey'); | ||||
| const rows = await runQuery<Array<{ id: number; providerKey: string }>>( | ||||
| `SELECT ${idColumn} AS id, ${providerKeyColumn} AS providerKey FROM ${providerTable}`, | ||||
| ); | ||||
|
|
||||
| return new Map(rows.map(({ id, providerKey }) => [providerKey, id.toString()])); | ||||
| } | ||||
| } | ||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: While I'm fine with the
dependencyId, the existingWorkflowDependencyentity calls thisdependencyKeyUh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That might be specific to the fact that a workflow has more varied kinds of dependencies though (e.g. node type).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Initially I had it as
dependencyKeyto follow the same asworkflowDepdencyand I've given this a thought, and I didn't want people to mistakenly insertproviderKeyinstead ofsecretConnectionId, this is the only reason why I went withdependencyIdrather thandependencyKeyThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good call 👍