Skip to content

chore: lifecycle date migration#2906

Merged
chronark merged 3 commits intomainfrom
lifecycle-date-migration
Feb 22, 2025
Merged

chore: lifecycle date migration#2906
chronark merged 3 commits intomainfrom
lifecycle-date-migration

Conversation

@chronark
Copy link
Collaborator

@chronark chronark commented Feb 22, 2025

  • chore: add migration column
  • revert: uncommit file

Summary by CodeRabbit

  • New Features

    • Enhanced entity tracking now provides additional timestamp metadata for creation, updates, and deletions across various entities.
    • API responses have been refined with a dedicated precondition failure error type including enhanced documentation.
  • Refactor

    • Streamlined workspace and integration data by removing obsolete properties to improve data retrieval efficiency.

@vercel
Copy link

vercel bot commented Feb 22, 2025

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
dashboard ✅ Ready (Inspect) Visit Preview 💬 Add feedback Feb 22, 2025 10:21am
engineering ✅ Ready (Inspect) Visit Preview 💬 Add feedback Feb 22, 2025 10:21am
play ✅ Ready (Inspect) Visit Preview 💬 Add feedback Feb 22, 2025 10:21am
www ✅ Ready (Inspect) Visit Preview 💬 Add feedback Feb 22, 2025 10:21am

@changeset-bot
Copy link

changeset-bot bot commented Feb 22, 2025

⚠️ No Changeset found

Latest commit: 95322d8

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@github-actions
Copy link
Contributor

github-actions bot commented Feb 22, 2025

Thank you for following the naming conventions for pull request titles! 🙏

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Feb 22, 2025

📝 Walkthrough

Walkthrough

This pull request extends lifecycle tracking across multiple parts of the codebase. It adds three new timestamp properties—createdAtM, updatedAtM, and deletedAtM—to various objects and database schemas (including test harnesses, API routes, and TRPC routers) to complement existing timestamp fields. Additionally, the changes incorporate a shared migration object (lifecycleDatesMigration) into several database table definitions, remove the llmGateways property from certain dashboard queries and type definitions, and add an OpenAPI error schema for precondition failures. The modifications are uniform and do not alter the existing control flow.

Changes

File(s) Change Summary
apps/api/src/pkg/testutil/harness.ts Added createdAtM, updatedAtM, and deletedAtM properties to the Harness class in methods such as createRootKey, optimisticUpsertPermission, optimisticUpsertRole, and createResources.
apps/api/src/routes/v1_migrations_createKey.ts Introduced the new timestamp fields to objects within the roleConnections and permissionConnections arrays.
apps/api/src/routes/v1_ratelimits_limit.ts Added createdAtM, updatedAtM, and deletedAtM fields to the namespace object in the registerV1RatelimitLimit function.
apps/dashboard/app/integrations/vercel/callback/page.tsx Expanded the integration object by adding createdAtM, updatedAtM, and deletedAtM fields.
apps/dashboard/lib/trpc/routers/rbac.ts
apps/dashboard/lib/trpc/routers/rbac/upsertPermission.ts
Added lifecycle timestamp properties to the permission objects handled by the upsert functions.
internal/db/src/schema/apis.ts
internal/db/src/schema/ratelimit.ts
internal/db/src/schema/rbac.ts
internal/db/src/schema/vercel_integration.ts
Incorporated the spread operator (...lifecycleDatesMigration) into various table definitions to include lifecycle date properties.
internal/db/src/schema/workspaces.ts Removed imports and relations for gateways, llmGateways, verificationMonitors, and webhooks from the workspace schema.
apps/dashboard/app/(app)/layout.tsx
apps/dashboard/app/(app)/mobile-sidebar.tsx
Removed the llmGateways property from the workspace query and the type definitions, respectively.
packages/api/src/openapi.d.ts Added the ErrPreconditionFailed error schema (with properties like code, docs, message, and requestId) and updated HTTP 412 responses to return this schema.
apps/api/src/routes/v1_apis_deleteApi.ts
apps/api/src/routes/v1_apis_deleteKeys.ts
apps/api/src/routes/v1_keys_deleteKey.ts
apps/api/src/routes/v1_ratelimits_deleteOverride.ts
apps/dashboard/lib/trpc/routers/api/delete.ts
apps/dashboard/lib/trpc/routers/key/delete.ts
apps/dashboard/lib/trpc/routers/key/deleteRootKey.ts
apps/dashboard/lib/trpc/routers/ratelimit/deleteNamespace.ts
apps/dashboard/lib/trpc/routers/ratelimit/deleteOverride.ts
apps/dashboard/lib/trpc/routers/vercel.ts
Updated deletion logic to include the deletedAtM timestamp field alongside deletedAt in multiple routes and TRPC routers.
apps/dashboard/lib/trpc/routers/workspace/create.ts Introduced createdAtM, updatedAtM, and deletedAtM properties to the workspace object in the createWorkspace procedure.
internal/db/src/schema/keys.ts Updated lifecycle date management by modifying the import and integration of lifecycleDatesMigration and lifecycleDatesV2 in the keys and encryptedKeys table definitions.
internal/db/src/schema/util/lifecycle_dates.ts Added a new constant lifecycleDatesV2 that defines a new structure for lifecycle dates.
tools/migrate/main.ts Restructured the main function to iterate over multiple tables, updating createdAtM, updatedAtM, and deletedAtM fields based on existing records.

Possibly related PRs

  • feat: create identities from external id when migrating keys #2849: The changes in the main PR, which involve adding timestamp properties to various objects in the Harness class, are related to the modifications in the retrieved PR that also introduces new timestamp properties (createdAtM, updatedAtM, and deletedAtM) for objects in the v1_migrations_createKey.ts file, indicating a direct connection at the code level.
  • feat: allow permanently deleting keys #2847: The changes in the main PR, which add new timestamp properties to various objects, are related to the modifications in the retrieved PR that introduce a new parameter for permanent deletion and update the deletion logic, as both involve enhancements to the handling of deletion timestamps in the database.
  • feat: Add default bytes and prefix #2146: The changes in the main PR, which involve adding createdAtM, updatedAtM, and deletedAtM properties to various objects in the Harness class, are related to the modifications in the retrieved PR that also introduce defaultBytes and defaultPrefix properties to the KeyAuth type definitions within the same Harness class. Both PRs enhance the data structure of the Harness class by adding new properties that expand its functionality.

Suggested reviewers

  • mcstepp
  • perkinsjr
  • MichaelUnkey
  • ogzhanolguncu
✨ Finishing Touches
  • 📝 Generate Docstrings (Beta)

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR. (Beta)
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@vercel vercel bot temporarily deployed to Preview – play February 22, 2025 08:55 Inactive
@vercel vercel bot temporarily deployed to Preview – engineering February 22, 2025 08:55 Inactive
@vercel vercel bot temporarily deployed to Preview – www February 22, 2025 08:57 Inactive
@vercel vercel bot temporarily deployed to Preview – dashboard February 22, 2025 08:59 Inactive
@vercel vercel bot temporarily deployed to Preview – play February 22, 2025 10:17 Inactive
@vercel vercel bot temporarily deployed to Preview – engineering February 22, 2025 10:17 Inactive
@vercel vercel bot temporarily deployed to Preview – www February 22, 2025 10:18 Inactive
@vercel vercel bot temporarily deployed to Preview – dashboard February 22, 2025 10:21 Inactive
@chronark chronark changed the title lifecycle date migration chore: lifecycle date migration Feb 22, 2025
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (6)
tools/migrate/main.ts (1)

10-21: Iterate over tables consistently.
The loop enumerates a static list of tables for the migration. If future migrations add new tables requiring the same logic, consider dynamic enumeration to reduce maintenance overhead.

apps/dashboard/lib/trpc/routers/key/deleteRootKey.ts (1)

31-31: Maintain consistent time source.
Using new Date() and Date.now() ensures partial consistency, but be mindful of potential discrepancies (e.g., if you need strict ordering, use a single source of truth).

apps/dashboard/lib/trpc/routers/key/delete.ts (1)

48-48: LGTM! Consider adding type safety for the timestamp fields.

The addition of deletedAtM alongside deletedAt provides both timezone-aware dates and precise chronological ordering capabilities.

Consider defining a shared type for lifecycle dates to ensure consistency:

type LifecycleDates = {
  deletedAt: Date;
  deletedAtM: number;
};
apps/dashboard/lib/trpc/routers/api/delete.ts (1)

48-48: LGTM! Consider extracting the lifecycle date updates into a helper.

The implementation correctly handles both API and associated keys deletion within the same transaction.

Consider creating a helper function to reduce duplication:

const setDeletedTimestamps = {
  deletedAt: new Date(),
  deletedAtM: Date.now()
};

// Usage
.set(setDeletedTimestamps)

Also applies to: 78-78

apps/dashboard/lib/trpc/routers/vercel.ts (1)

528-528: LGTM! Consider using a type-safe approach for the deletion timestamps.

The addition of deletedAtM alongside deletedAt enhances timestamp tracking with millisecond precision.

Consider creating a shared type or interface for deletion timestamps to ensure consistency:

interface DeletionTimestamps {
  deletedAt: Date;
  deletedAtM: number;
}

Then use it in your schema definitions:

const vercelBindings = {
  // ... other fields
  ...deletionTimestamps
}

Also applies to: 585-585

apps/api/src/routes/v1_keys_verifyKey.test.ts (1)

830-830: LGTM! Consider adding a test helper for soft deletion.

The addition of deletedAtM alongside deletedAt consistently implements the new deletion timestamp pattern across test cases.

Consider creating a test helper function to reduce duplication and ensure consistency:

async function softDelete(tx: typeof h.db.primary, table: any, id: string) {
  await tx
    .update(table)
    .set({ deletedAt: new Date(), deletedAtM: Date.now() })
    .where(eq(table.id, id));
}

// Usage in tests:
await softDelete(h.db.primary, schema.keys, key.keyId);
await softDelete(h.db.primary, schema.keyAuth, h.resources.userKeyAuth.id);
await softDelete(h.db.primary, schema.apis, h.resources.userApi.id);

Also applies to: 856-856, 882-882

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 2dc7577 and 95322d8.

📒 Files selected for processing (21)
  • apps/api/src/pkg/testutil/harness.ts (7 hunks)
  • apps/api/src/routes/v1_apis_deleteApi.ts (2 hunks)
  • apps/api/src/routes/v1_apis_deleteKeys.ts (1 hunks)
  • apps/api/src/routes/v1_keys_deleteKey.ts (1 hunks)
  • apps/api/src/routes/v1_keys_verifyKey.test.ts (3 hunks)
  • apps/api/src/routes/v1_migrations_createKey.ts (3 hunks)
  • apps/api/src/routes/v1_ratelimits_deleteOverride.ts (1 hunks)
  • apps/dashboard/app/api/v1/vercel/integration/route.ts (1 hunks)
  • apps/dashboard/lib/trpc/routers/api/delete.ts (2 hunks)
  • apps/dashboard/lib/trpc/routers/key/delete.ts (1 hunks)
  • apps/dashboard/lib/trpc/routers/key/deleteRootKey.ts (1 hunks)
  • apps/dashboard/lib/trpc/routers/ratelimit/deleteNamespace.ts (2 hunks)
  • apps/dashboard/lib/trpc/routers/ratelimit/deleteOverride.ts (1 hunks)
  • apps/dashboard/lib/trpc/routers/rbac/upsertPermission.ts (1 hunks)
  • apps/dashboard/lib/trpc/routers/vercel.ts (2 hunks)
  • apps/dashboard/lib/trpc/routers/workspace/create.ts (1 hunks)
  • internal/db/src/schema/keys.ts (2 hunks)
  • internal/db/src/schema/rbac.ts (5 hunks)
  • internal/db/src/schema/util/lifecycle_dates.ts (1 hunks)
  • internal/db/src/schema/workspaces.ts (2 hunks)
  • tools/migrate/main.ts (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (3)
  • apps/dashboard/lib/trpc/routers/rbac/upsertPermission.ts
  • apps/api/src/pkg/testutil/harness.ts
  • apps/api/src/routes/v1_migrations_createKey.ts
⏰ Context from checks skipped due to timeout of 90000ms (2)
  • GitHub Check: Test Agent Local / test_agent_local
  • GitHub Check: Test API / API Test Local
🔇 Additional comments (21)
apps/dashboard/lib/trpc/routers/workspace/create.ts (1)

51-53: Consider timestamp format consistency.

The workspace object now has two sets of similar timestamp fields with different formats:

  • createdAt/deletedAt using Date objects
  • createdAtM/deletedAtM using milliseconds

This could lead to confusion and potential bugs when handling dates.

Could you clarify:

  1. Why both formats are needed?
  2. If they serve different purposes, should this be documented?
  3. If they're part of a migration, what's the plan to consolidate them?

If these are indeed part of a migration as suggested by the PR title, consider:

-      createdAtM: Date.now(),
+      // TODO: Migration - Remove old timestamp fields (createdAt, deletedAt) once migration is complete
+      createdAtM: Date.now(),
tools/migrate/main.ts (1)

22-28: Check for potential edge cases.
If no rows have createdAtM as null, the code simply logs { count: 0 }. This is expected, but ensure that any downstream logic (like reporting or validation) doesn't assume rows will be processed.

internal/db/src/schema/util/lifecycle_dates.ts (1)

10-16: Clarify default value usage.
Defining both .default(0) and .$defaultFn(() => Date.now()) may introduce ambiguity when the function isn't triggered. Decide if zero is truly acceptable as a fallback, or consider removing .default(0) if you always intend to use the function.

apps/dashboard/app/api/v1/vercel/integration/route.ts (1)

37-37: LGTM! Consistent implementation of lifecycle dates.

The addition of deletedAtM is consistently applied across all removal cases, maintaining data integrity during the deletion process.

Also applies to: 44-44, 48-48

apps/dashboard/lib/trpc/routers/ratelimit/deleteOverride.ts (1)

50-50: LGTM! Well-structured implementation with proper error handling.

The addition of deletedAtM is properly implemented within the transaction, maintaining consistency with other deletion operations.

apps/dashboard/lib/trpc/routers/ratelimit/deleteNamespace.ts (2)

42-42: LGTM! Consistent timestamp tracking added.

The addition of deletedAtM alongside deletedAt enhances timestamp precision while maintaining backward compatibility.


73-73: LGTM! Cascading deletion timestamp tracking.

Consistent implementation of timestamp tracking for related overrides within the same transaction.

apps/api/src/routes/v1_apis_deleteKeys.ts (1)

120-124: LGTM! Proper transaction handling with enhanced timestamp tracking.

The implementation correctly:

  1. Tracks the count of affected keys
  2. Updates both timestamp fields atomically
  3. Maintains proper transaction boundaries
internal/db/src/schema/workspaces.ts (2)

21-21: LGTM! Clean import of lifecycle dates migration.

Proper modularization of the lifecycle dates schema.


110-110: LGTM! Consistent application of lifecycle dates.

The spread operator ensures uniform application of lifecycle date fields across the workspace schema.

apps/api/src/routes/v1_ratelimits_deleteOverride.ts (1)

110-110: LGTM! Consistent timestamp tracking implementation.

The change maintains consistency with other deletion operations across the codebase, properly tracking both date and millisecond precision timestamps within a transaction.

apps/api/src/routes/v1_apis_deleteApi.ts (2)

98-101: LGTM! Soft delete operation enhanced with millisecond precision.

The update operation now includes both deletedAt and deletedAtM fields, providing better timestamp granularity.


128-131: LGTM! Consistent timestamp handling across tables.

The update operation for the keys table follows the same pattern, ensuring consistent timestamp handling.

apps/api/src/routes/v1_keys_deleteKey.ts (1)

134-137: LGTM! Soft delete operation enhanced with millisecond precision.

The update operation now includes both deletedAt and deletedAtM fields, providing better timestamp granularity.

internal/db/src/schema/keys.ts (2)

18-18: LGTM! Import updated to include new lifecycle dates schema.

The import now includes both the migration and v2 lifecycle dates schemas, enabling gradual migration.


152-152: LGTM! Migrated to new lifecycle dates schema.

The encryptedKeys table now uses the new lifecycleDatesV2 schema, while maintaining backward compatibility in other tables.

internal/db/src/schema/rbac.ts (5)

28-32: LGTM! Added millisecond precision timestamps to permissions table.

The implementation correctly handles both creation and update timestamps with millisecond precision.


72-76: LGTM! Added millisecond precision timestamps to keysPermissions table.

The implementation follows the same pattern, ensuring consistent timestamp handling.


115-119: LGTM! Added millisecond precision timestamps to roles table.

The implementation maintains consistency with other tables.


161-165: LGTM! Added millisecond precision timestamps to rolesPermissions table.

The implementation follows the established pattern.


210-214: LGTM! Added millisecond precision timestamps to keysRoles table.

The implementation completes the consistent timestamp handling across all RBAC tables.

Comment on lines +29 to +61
let processed = 0;
let cursor = "";
do {
const rows = await db
.select()
.from(table)
.where(
and(
isNull(table.createdAtM),

gt(table.id, cursor),
),
)
.orderBy(asc(table.id))
.limit(100);

cursor = rows.at(-1)?.id ?? "";
console.info({ cursor, rows: rows.length, processed, count });

await Promise.all(
rows.map(async (row) => {
await db
.update(table)
.set({
createdAtM: new Date(row.createdAt ?? 0).getTime() ?? 0,
updatedAtM: "updatedAt" in row ? row.updatedAt?.getTime() ?? null : null,
deletedAtM: row.deletedAt?.getTime() ?? null,
})
.where(eq(table.id, row.id));
}),
);
processed += rows.length;
} while (cursor);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Optimize bulk updates.
Updating rows individually in a Promise.all can be slow under significant loads. Consider batching updates using a single SQL statement if supported by your schema to reduce overhead.

-      await Promise.all(
-        rows.map(async (row) => {
-          await db
-            .update(table)
-            .set({
-              createdAtM: new Date(row.createdAt ?? 0).getTime() ?? 0,
-              updatedAtM: "updatedAt" in row ? row.updatedAt?.getTime() ?? null : null,
-              deletedAtM: row.deletedAt?.getTime() ?? null,
-            })
-            .where(eq(table.id, row.id));
-        }),
-      );
+      // Example: gather IDs and prepared values, then execute a bulk update
+      const updateData = rows.map((row) => ({
+        id: row.id,
+        createdAtM: new Date(row.createdAt ?? 0).getTime() ?? 0,
+        updatedAtM: "updatedAt" in row ? row.updatedAt?.getTime() ?? null : null,
+        deletedAtM: row.deletedAt?.getTime() ?? null,
+      }));
+
+      // Pseudocode: pass updateData to a single update statement if feasible
+      // attn: Implementation may vary depending on your DB/table structure

Committable suggestion skipped: line range outside the PR's diff.

@chronark chronark merged commit 7e8fea6 into main Feb 22, 2025
29 of 31 checks passed
@chronark chronark deleted the lifecycle-date-migration branch February 22, 2025 10:38
@coderabbitai coderabbitai bot mentioned this pull request Feb 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant