move getvirtual keys to use paginated virtual keys to handle large no of keys

[akshaydeo]

Summary

GetVirtualKeys previously fetched all virtual keys in a single query with GORM preloads. When the number of keys grows large, this triggers PostgreSQL's extended protocol parameter limit, causing the query to fail. This PR fixes the issue by internally paginating through results in bounded pages of 1,000 rows at a time, transparently assembling the full result set before returning it to callers.

Changes

• GetVirtualKeys now delegates to GetVirtualKeysPaginated in a loop using a fixed internal page size (virtualKeyInternalPageSize = 1000), accumulating results until all rows are retrieved.
• A test (TestGetVirtualKeysUsesInternalPagination) was added that creates more than one full internal page of virtual keys and asserts that GetVirtualKeys returns all of them in the correct order.

Type of change

• Bug fix
• Feature
• Refactor
• Documentation
• Chore/CI

Affected areas

• Core (Go)
• Transports (HTTP)
• Providers/Integrations
• Plugins
• UI (React)
• Docs

How to test

go test ./framework/configstore/... -run TestGetVirtualKeysUsesInternalPagination -v
go test ./framework/configstore/...

The new test creates 1,005 virtual keys (one full internal page plus five extras) and verifies that GetVirtualKeys returns all 1,005 rows with the first and last IDs in the expected order.

Screenshots/Recordings

N/A

Breaking changes

• Yes
• No

Related issues

Security considerations

No auth, secrets, or PII implications. The change only affects how rows are batched internally during a database read.

Checklist

• I read docs/contributing/README.md and followed the guidelines
• I added/updated tests where appropriate
• I updated documentation where needed
• I verified builds succeed (Go and UI)
• I verified the CI pipeline passes locally if applicable

Summary by CodeRabbit

Release Notes

• New Features

  • Added encryption_key configuration option for enhanced security.

• Improvements

  • Optimized database handling for large datasets using efficient pagination.
  • Resolved potential service limitations with PostgreSQL queries.

• Tests

  • Added pagination coverage tests.

• Chores

  • Updated and cleaned up dependencies.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 441359ae-1b7d-4294-ba90-f645e5490219

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

This PR implements internal pagination for virtual key retrieval to address PostgreSQL parameter limits, adds an encryption key to a configuration example, and normalizes a go.mod dependency. The main change introduces bounded paging in GetVirtualKeys with a deterministic ordering and a supporting test that validates correct multi-page behavior.

Changes

Virtual Keys Pagination, Config, and Dependencies

Layer / File(s)	Summary
Virtual Keys Pagination with Internal Paging framework/configstore/rdb.go, framework/configstore/rdb_test.go	GetVirtualKeys now uses offset-based pagination with a bounded page size to avoid PostgreSQL extended-protocol parameter limits. A new getVirtualKeysPage helper fetches pages with deterministic ordering (created_at, id) and preloads relationships. New test validates all keys are returned across multiple pages with correct boundary ordering.
Configuration Example Update examples/configs/withconfigstorelogsstorepostgres/config.json	The PostgreSQL configuration example now includes a top-level encryption_key field.
Go Module Dependency Cleanup transports/go.mod	Normalized the indirect github.com/mattn/go-sqlite3 v1.14.32 dependency line, removing a prior malformed merged line artifact.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Suggested reviewers

• danpiths

Poem

A rabbit hops through pages bright,
Virtual keys now paginated right,
PostgreSQL limits fade from sight,
Config encryption locks up tight,
Dependencies cleaned with all their might! 🐇✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title directly describes the main change: moving GetVirtualKeys to use pagination to handle large numbers of keys, which aligns with the core problem and solution in the PR.
Description check	✅ Passed	The PR description provides all major required sections including Summary, Changes, Type of change, Affected areas, How to test, Breaking changes, Security considerations, and a completed Checklist.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 06-01-move_getvirtual_keys_to_use_paginated_virtual_keys_to_handle_large_no_of_keys

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[akshaydeo]

• move getvirtual keys to use paginated virtual keys to handle large no of keys #3957 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[greptile-apps]

Confidence Score: 5/5

The change is safe to merge: it touches only the internal loading path for virtual keys, introduces no schema changes, and the new keyset cursor correctly handles the degenerate case where all rows share the same created_at timestamp.

The keyset pagination logic is correct — (created_at, id) is a stable total order (primary key guarantees uniqueness), the cursor condition mirrors the ORDER BY exactly, and the early-exit on a short page avoids a final empty-result round trip. The test deliberately exercises the worst-case tie-break scenario. No correctness bugs were found.

No files require special attention, though a future migration adding a composite index on (created_at, id) would improve keyset-scan efficiency for very large virtual-key tables.

Important Files Changed

Filename	Overview
framework/configstore/rdb.go	Replaces the single-query GetVirtualKeys with a keyset-cursor pagination loop (1000 rows/page) via the new private getVirtualKeysPage; no extra COUNT queries; cursor correctness relies on the total order of (created_at, id).
framework/configstore/rdb_test.go	Adds TestGetVirtualKeysUsesInternalPagination that inserts 1005 keys (one full internal page + 5) with a shared CreatedAt, then asserts GetVirtualKeys returns all 1005 in the expected order.

_{Reviews (6): Last reviewed commit: "1.5.7 changelogs (#3907)" | Re-trigger Greptile}

[akshaydeo]

@coderabbitai full review

[coderabbitai]

✅ Actions performed

Full review triggered.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@examples/configs/withconfigstorelogsstorepostgres/config.json`:
- Line 3: Replace the hard-coded "encryption_key" value in config.json with an
environment-backed reference: stop embedding the literal under the
"encryption_key" property and instead read from an env var (e.g.
BIFROST_ENCRYPTION_KEY). Update the configuration consumer (where
"encryption_key" is parsed/used) to resolve process.env.BIFROST_ENCRYPTION_KEY
at runtime and fail fast with a clear error if it's missing; keep the property
name "encryption_key" in the JSON but document that its value must come from the
environment in examples and tests.

In `@framework/configstore/rdb.go`:
- Around line 2574-2603: The current GetVirtualKeys loop using offset-based
pagination is unstable under concurrent writes; replace it with keyset
pagination: change GetVirtualKeys to iterate by passing a cursor (lastCreatedAt,
lastID) into getVirtualKeysPage and have getVirtualKeysPage query with the same
ORDER BY ("governance_virtual_keys.created_at ASC, governance_virtual_keys.id
ASC") and a WHERE clause equivalent to (created_at, id) > (?, ?) to fetch the
next page, keeping Limit but removing Offset; handle the initial empty cursor
case, stop when fewer than limit rows returned, and ensure
preloadVirtualKeyBaseRelations is applied and edge cases where created_at ties
are disambiguated by id are covered so traversal is mutation-safe.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 23ffeb0b-d42b-4169-a76c-ede339817ad8

📥 Commits

Reviewing files that changed from the base of the PR and between 270c965 and 5850388.

📒 Files selected for processing (4)

• examples/configs/withconfigstorelogsstorepostgres/config.json
• framework/configstore/rdb.go
• framework/configstore/rdb_test.go
• transports/go.mod

[akshaydeo]

Merge activity

• Jun 1, 5:15 PM UTC: A user started a stack merge that includes this pull request via Graphite.
• Jun 1, 5:16 PM UTC: @akshaydeo merged this pull request with Graphite.