Skip to content
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
24d274a
feat(core): Introduce CredentialDependency entity to track credential…
alielkhateeb Mar 17, 2026
4231912
Remove variable related process and migration
alielkhateeb Mar 18, 2026
222a241
adjust the credentials query
alielkhateeb Mar 18, 2026
4e3291a
fix lint
alielkhateeb Mar 18, 2026
e1d5787
Add tests
alielkhateeb Mar 18, 2026
a8b6f9f
Separate credential dependency service
alielkhateeb Mar 19, 2026
bb52f50
combine service methods into one and refactor params
alielkhateeb Mar 19, 2026
de8471c
Fix tests
alielkhateeb Mar 19, 2026
4646f53
Create external secrets utils
alielkhateeb Mar 19, 2026
eefaa5c
Encapsulate credential dependency methods better
alielkhateeb Mar 19, 2026
aee6b9f
Cover the issue with partial provider keys
alielkhateeb Mar 19, 2026
5e1d382
Fix test issues that were leftover due to the incorrect regex
alielkhateeb Mar 19, 2026
3a3ed18
add migration test
alielkhateeb Mar 19, 2026
98c88f4
Add index decorator for consistency
alielkhateeb Mar 19, 2026
9b9d5b4
fix migration test
alielkhateeb Mar 19, 2026
59176ed
Refactor params with more concise and clear intent
alielkhateeb Mar 19, 2026
83698db
extract private method for finding credentials by dependency
alielkhateeb Mar 19, 2026
1ecd266
Delete credential dependency when a connection is deleted
alielkhateeb Mar 19, 2026
3672308
cleanup connection removal related entities and use transactions
alielkhateeb Mar 20, 2026
ca233fb
Refactor dependency delete methods
alielkhateeb Mar 20, 2026
56272bb
trial to fix the migration test
alielkhateeb Mar 20, 2026
08bbbd8
Fix migration test
alielkhateeb Mar 20, 2026
a48a736
Merge branch 'master' into LIGO-230-introduce-credential-dependency-e…
ireneea Mar 23, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions packages/@n8n/db/src/entities/credential-dependency-entity.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
import {
Column,
Entity,
Index,
JoinColumn,
ManyToOne,
PrimaryGeneratedColumn,
Relation,
Unique,
} from '@n8n/typeorm';

import { WithCreatedAt } from './abstract-entity';
import type { CredentialsEntity } from './credentials-entity';

export type CredentialDependencyType = 'externalSecretProvider';

@Entity({ name: 'credential_dependency' })
@Unique(['credentialId', 'dependencyType', 'dependencyId'])
export class CredentialDependency extends WithCreatedAt {
@PrimaryGeneratedColumn()
id: number;

@Column({ length: 36 })
@Index()
credentialId: string;

@Column({ length: 64 })
dependencyType: CredentialDependencyType;

@Column({ length: 255 })
dependencyId: string;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: While I'm fine with the dependencyId, the existing WorkflowDependency entity calls this dependencyKey

@konstantintieber konstantintieber Mar 19, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That might be specific to the fact that a workflow has more varied kinds of dependencies though (e.g. node type).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initially I had it as dependencyKey to follow the same as workflowDepdency and I've given this a thought, and I didn't want people to mistakenly insert providerKey instead of secretConnectionId, this is the only reason why I went with dependencyId rather than dependencyKey

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call 👍


@ManyToOne('CredentialsEntity', { onDelete: 'CASCADE' })
@JoinColumn({ name: 'credentialId' })
credential: Relation<CredentialsEntity>;
}
7 changes: 7 additions & 0 deletions packages/@n8n/db/src/entities/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@ import { ApiKey } from './api-key';
import { AuthIdentity } from './auth-identity';
import { AuthProviderSyncHistory } from './auth-provider-sync-history';
import { BinaryDataFile, SourceTypeSchema, type SourceType } from './binary-data-file';
import {
CredentialDependency,
type CredentialDependencyType,
} from './credential-dependency-entity';
import { CredentialsEntity } from './credentials-entity';
import { ExecutionAnnotation } from './execution-annotation.ee';
import { ExecutionData } from './execution-data';
Expand Down Expand Up @@ -51,6 +55,8 @@ export {
WebhookEntity,
AuthIdentity,
CredentialsEntity,
CredentialDependency,
type CredentialDependencyType,
Folder,
Project,
ProjectRelation,
Expand Down Expand Up @@ -92,6 +98,7 @@ export const entities = {
WebhookEntity,
AuthIdentity,
CredentialsEntity,
CredentialDependency,
Folder,
Project,
ProjectRelation,
Expand Down

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The utilities in this file might give the impression of duplicate code, that's because it is duplicate code 😂 somehow

Usually migrations are meant to be self contained with minimal dependencies to avoid breaking when business logic change over time, if this is not the right way here at n8n I'm happy to amend and reuse the existing utility methods :)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's good that migrations don't import code but rather duplicate it 👍
I wouldn't want the behaviour of the migration to change if we change the util implementation in the future (or at least I wouldn't expect as a dev that changing a function in our code will affect a migration that was built in the past).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@konstantintieber Awesome, thanks for reassuring me 🫶

Original file line number Diff line number Diff line change
@@ -0,0 +1,203 @@
import { Container } from '@n8n/di';
import { Cipher } from 'n8n-core';

import type { MigrationContext, ReversibleMigration } from '../migration-types';

const credentialDependencyTable = 'credential_dependency';
const externalSecretProviderDependencyType = 'externalSecretProvider';
const providerKeyPattern = '[A-Za-z0-9]+(?:-[A-Za-z0-9]+)*';

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sort of a duplicate of

const PROVIDER_KEY_PATTERN = '[A-Za-z0-9]+(?:-[A-Za-z0-9]+)*';
but since migrations should be self-contained I created a new one here


type CredentialRow = {
id: string;
data: string;
};

export class CreateCredentialDependencyTable1773000000000 implements ReversibleMigration {
private readonly cipher = Container.get(Cipher);

async up({
schemaBuilder: { createTable, column, createIndex },
escape,
runQuery,
runInBatches,
logger,
migrationName,
}: MigrationContext) {
await createTable(credentialDependencyTable)
.withColumns(
column('id').int.primary.autoGenerate2,
column('credentialId').varchar(36).notNull,
column('dependencyType').varchar(64).notNull,
column('dependencyId').varchar(255).notNull,
)
.withForeignKey('credentialId', {
tableName: 'credentials_entity',
columnName: 'id',
onDelete: 'CASCADE',
})
.withIndexOn(['credentialId'])
.withIndexOn(['dependencyType', 'dependencyId']).withCreatedAt;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good set of indexes you added here 👍
I think this one here should also be added as decorator on the entity: @Index(['dependencyType', 'dependencyId'])

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@konstantintieber True 98c88f4
I wondered - for my own knowledge - if we have any sort of "sync" process between entities and database state, basically if we ever update the entities and "generate" the migrations?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No automatic sync process that I'm aware of. All human based 😅


await createIndex(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the separate createIndex ? why not another .withIndexOn( ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That’s an absolutely valid question ✅

The main reason is that I found it hard to clearly spot unique indexes when they’re defined alongside non-unique ones using withIndexOn. For example:

.withIndexOn(['credentialId'])
.withIndexOn(['dependencyType', 'dependencyId'])
.withIndexOn(['credentialId', 'dependencyType', 'dependencyId'], true);

Here, the true is quite subtle and easy to miss when scanning the code.

Since this is a migration file (and not something we expect to revisit or refactor), I preferred to make the unique index explicit. This makes it easier for reviewers and future readers to spot which index enforces uniqueness vs which ones are just for performance.

Also… the fact that you pointed it out probably means my plan worked 😂 haha

credentialDependencyTable,
['credentialId', 'dependencyType', 'dependencyId'],
true,
);

const credentialsTable = escape.tableName('credentials_entity');
const query = `SELECT c.id AS id, c.data AS data FROM ${credentialsTable} c ORDER BY c.id`;
const providerIdByKey = await this.loadProviderIdByKey(runQuery, escape);

let processedCount = 0;
let insertedCount = 0;

await runInBatches<CredentialRow>(query, async (rows) => {
if (rows.length === 0) return;

const batchDependencies = rows.flatMap((row) => {
const providerKeys = this.extractProviderKeysFromCredentialData(row.data);
const providerIds = providerKeys
.map((providerKey) => providerIdByKey.get(providerKey))
.filter((providerId): providerId is string => providerId !== undefined);
return providerIds.map((providerId) => ({
credentialId: row.id,
dependencyType: externalSecretProviderDependencyType,
dependencyId: providerId,
}));
});

processedCount += rows.length;

if (batchDependencies.length === 0) {
return;
}

await this.insertDependencies(batchDependencies, escape, runQuery);
insertedCount += batchDependencies.length;
});

logger.info(
`[${migrationName}] Backfilled credential dependencies for ${processedCount} credentials. Inserted ${insertedCount} dependencies.`,
);
}

async down({ schemaBuilder: { dropTable } }: MigrationContext) {
await dropTable(credentialDependencyTable);
}

private extractProviderKeysFromCredentialData(encryptedCredentialData: string): string[] {
const decrypted = this.tryDecryptCredentialData(encryptedCredentialData);
if (decrypted === null) return [];
return this.extractProviderKeysFromDecryptedData(decrypted);
}

private extractProviderKeysFromDecryptedData(decryptedCredentialData: unknown): string[] {
const uniqueKeys = new Set<string>();
const valuesToScan: unknown[] = [decryptedCredentialData];

while (valuesToScan.length > 0) {
const currentValue = valuesToScan.pop();
if (typeof currentValue === 'string') {
if (!currentValue.includes('$secrets')) continue;
for (const dependencyKey of this.extractProviderKeys(currentValue)) {
uniqueKeys.add(dependencyKey);
}
continue;
}

if (Array.isArray(currentValue)) {
for (const value of currentValue as unknown[]) {
valuesToScan.push(value);
}
continue;
}

if (typeof currentValue === 'object' && currentValue !== null) {
for (const value of Object.values(currentValue as Record<string, unknown>)) {
valuesToScan.push(value);
}
}
}

return [...uniqueKeys];
}

private tryDecryptCredentialData(encryptedCredentialData: string): unknown {
try {
const decrypted = this.cipher.decrypt(encryptedCredentialData);
return JSON.parse(decrypted) as unknown;
} catch {
return null;
}
}

private extractProviderKeys(expression: string): string[] {
const providerKeys = new Set<string>();
const expressionBlocks = expression.matchAll(/\{\{(.*?)\}\}/gs);

for (const block of expressionBlocks) {
const expressionContent = block[1];

const dotMatches = expressionContent.matchAll(
new RegExp(`\\$secrets\\.(${providerKeyPattern})`, 'g'),
);
for (const match of dotMatches) {
providerKeys.add(match[1]);
}

const bracketMatches = expressionContent.matchAll(
new RegExp(`\\$secrets\\[['"](${providerKeyPattern})['"]\\]`, 'g'),
);
for (const match of bracketMatches) {
providerKeys.add(match[1]);
}
}

return [...providerKeys];
}

private async insertDependencies(
dependencies: Array<{
credentialId: string;
dependencyType: string;
dependencyId: string;
}>,
escape: MigrationContext['escape'],
runQuery: MigrationContext['runQuery'],
) {
const dependencyTable = escape.tableName(credentialDependencyTable);
const credentialIdColumn = escape.columnName('credentialId');
const dependencyTypeColumn = escape.columnName('dependencyType');
const dependencyIdColumn = escape.columnName('dependencyId');

const namedParameters: Record<string, string> = {};
const valuesSql = dependencies
.map((dependency, index) => {
namedParameters[`credentialId${index}`] = dependency.credentialId;
namedParameters[`dependencyType${index}`] = dependency.dependencyType;
namedParameters[`dependencyId${index}`] = dependency.dependencyId;

return `(:credentialId${index}, :dependencyType${index}, :dependencyId${index})`;
})
.join(', ');

await runQuery(
`INSERT INTO ${dependencyTable} (${credentialIdColumn}, ${dependencyTypeColumn}, ${dependencyIdColumn}) VALUES ${valuesSql} ON CONFLICT (${credentialIdColumn}, ${dependencyTypeColumn}, ${dependencyIdColumn}) DO NOTHING;`,
namedParameters,
);
}

private async loadProviderIdByKey(
runQuery: MigrationContext['runQuery'],
escape: MigrationContext['escape'],
): Promise<Map<string, string>> {
const providerTable = escape.tableName('secrets_provider_connection');
const idColumn = escape.columnName('id');
const providerKeyColumn = escape.columnName('providerKey');
const rows = await runQuery<Array<{ id: number; providerKey: string }>>(
`SELECT ${idColumn} AS id, ${providerKeyColumn} AS providerKey FROM ${providerTable}`,
);

return new Map(rows.map(({ id, providerKey }) => [providerKey, id.toString()]));
}
}
2 changes: 2 additions & 0 deletions packages/@n8n/db/src/migrations/postgresdb/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,7 @@ import { AddSuggestedPromptsToAgentTable1772000000000 } from '../common/17720000
import { AddRoleColumnToProjectSecretsProviderAccess1772619247761 } from '../common/1772619247761-AddRoleColumnToProjectSecretsProviderAccess';
import { ChangeWorkflowPublishedVersionFKsToRestrict1772619247762 } from '../common/1772619247762-ChangeWorkflowPublishedVersionFKsToRestrict';
import { AddTypeToChatHubSessions1772700000000 } from '../common/1772700000000-AddTypeToChatHubSessions';
import { CreateCredentialDependencyTable1773000000000 } from '../common/1773000000000-CreateCredentialDependencyTable';
import type { Migration } from '../migration-types';

export const postgresMigrations: Migration[] = [
Expand Down Expand Up @@ -309,4 +310,5 @@ export const postgresMigrations: Migration[] = [
AddRoleColumnToProjectSecretsProviderAccess1772619247761,
ChangeWorkflowPublishedVersionFKsToRestrict1772619247762,
AddTypeToChatHubSessions1772700000000,
CreateCredentialDependencyTable1773000000000,
];
2 changes: 2 additions & 0 deletions packages/@n8n/db/src/migrations/sqlite/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,7 @@ import { AddSuggestedPromptsToAgentTable1772000000000 } from '../common/17720000
import { AddRoleColumnToProjectSecretsProviderAccess1772619247761 } from '../common/1772619247761-AddRoleColumnToProjectSecretsProviderAccess';
import { ChangeWorkflowPublishedVersionFKsToRestrict1772619247762 } from '../common/1772619247762-ChangeWorkflowPublishedVersionFKsToRestrict';
import { AddTypeToChatHubSessions1772700000000 } from '../common/1772700000000-AddTypeToChatHubSessions';
import { CreateCredentialDependencyTable1773000000000 } from '../common/1773000000000-CreateCredentialDependencyTable';
import type { Migration } from '../migration-types';

const sqliteMigrations: Migration[] = [
Expand Down Expand Up @@ -297,6 +298,7 @@ const sqliteMigrations: Migration[] = [
AddRoleColumnToProjectSecretsProviderAccess1772619247761,
ChangeWorkflowPublishedVersionFKsToRestrict1772619247762,
AddTypeToChatHubSessions1772700000000,
CreateCredentialDependencyTable1773000000000,
];

export { sqliteMigrations };
Loading
Loading